Build RESTful Microservices with AWS Lambda and API Gateway

Build RESTful Microservices with AWS Lambda and API Gateway

Amazon API Gateway allows us to design RESTful interfaces and connect them to our favorite backend. We can design our own resources structure, add dynamic routing parameters, and develop custom authorizations logic. Each API resource can be configured independently, while each stage can have specific cache, throttling, and logging configurations.

This approach is particularly useful when we consider that each request and response can be attached to a custom mapping template, in order to perform custom data manipulation or improve API backward compatibility.

In this post, we will see how to define a simple API and how to connect it to AWS Lambda. This provides a nice way to obtain a scalable backend for modern web applications or mobile apps. We will configure custom stages, protect resources with an API key, and explain how to best connect API Gateway stages with AWS Lambda versions and aliases. We will learn about AWS Lambda’s basic configuration, monitoring, and versioning as you progress through the post.

Throughout this post, we will configuring API Gateway and Lambda to reach this end-state:

Quick Overview:

  • Understand the basics of RESTful APIs
  • Implement REST APIs using Amazon API Gateway
  • Enable desirable API features in API Gateway including caching, throttling, CORS, usage plans, and API key access
  • Create serverless API backends using AWS Lambda functions
  • Implement best practices for integrating Lambda backends in API Gateway

Understanding RESTful APIs

The RESTful approach makes the development of modern web applications much more flexible and maintainable. The new tendency is to build more complex clients with client-side frameworks such as AngularJS, ReactJS, PolymerJS, etc. This way, your web app can easily be distributed as a set of static assets - HTML, JavaScript and CSS files - which will load dynamic content via API.

This new architectural pattern allows you to separate business logic from your presentation layer(s). At the same time, your services will be easier to scale and reuse, eventually by more than one client, including your own mobile apps too.

In the AWS world, a typical configuration looks similar to the following:

  • Static website hosted on Amazon S3 and distributed via Amazon CloudFront.
  • RESTful API implemented with AWS Lambda and HTTP endpoints exposed via Amazon API Gateway.
  • Dynamic data stored in DynamoDB, RDS or other database as a service alternative.

This is how you’d build a completely serverless web application, meaning that you won’t need to manage, patch or maintain any server during your development and deployment workflow.

What is REST?: REST stands for Representational state transfer and is meant to be an architectural reference for developing modern and user-friendly web services.

Instead of defining custom methods and protocols such as SOAP or WSDL, REST is based on HTTP as the transport protocol. HTTP is used to exchange textual representations of web resources across different systems, using predefined methods such as GET, POST, PUT, PATCH, DELETE, etc.

The standard representation format is JSON, which is also the most convenient format to develop modern web applications since it’s natively supported by JavaScript.

The level of abstraction provided by a RESTful API should guarantee a uniform interface and a set of stateless interactions: this means that all the information necessary to process a request must be included in the request itself (i.e. URL, headers, query string or body). Furthermore, each resource should be eventually cachable by the client, based on the particular use case.

How does Amazon API Gateway help?: With Amazon API Gateway, you can define resources, map them to custom models, specify which methods are available (i.e. GET, POST, etc.) and eventually bind each method to a particular Lambda function. Alternatively, you can attach more than one method to one single Lambda function. This way, you will maintain fewer functions and partially avoid the cold-start Lambda issue.

Defining New API Gateway Resources

  • Create a RESTful Application Programming Interface using Amazon API Gateway
  • Define a model for the API
  • Define resources for the model
  • Add a method for clients to access the resources

Here we will design a very simple API to read a list of items and retrieve the details of a given item by ID. Therefore, you will define two HTTP endpoints:

  • /items/
  • /items/{ID}/

These routes are compliant with the RESTful design principles: the first endpoint lets you retrieve the full list of items, while the second one corresponds to the detail of a single item given its ID. As you can notice, the second route is defined in terms of a dynamic parameter, which is part of the URL itself. Of course, you could achieve the very same result with a query string parameter (i.e. /items/?ID=XXX), but this pattern is generally discouraged.

Step 1: Create your API

In order to keep your data structures well defined and your interface documented and maintainable, you can create new API Gateway Models.

Models are useful to map your API endpoints’ payload via mapping templates.

They are not strictly required, as you can manually define your mapping templates, but having a model will let you generate strongly-typed SDKs for your API resources and better validate your payloads.

Technically, you can define a model with a JSON Schema. For this post, you will define this simple model:

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "Item Schema",
    "type": "object",
    "properties": {
        "ID": {"type" : "number"},
        "name": {"type" : "string"},
        "price": {"type" : "number"}
    }
}

Our items will have a unique numeric ID, a string identifying its name and a numeric price.

In the AWS Console, click Services, enter API Gateway in the search bar, and select the API Gateway search result. In the Choose an API type form, select the REST API. In the Create new API form, select New API and enter the following values, then click Create API:

The API is now created and you can define a model for it.

Step 2: Define your Model

Click APIs > ItemsAPI > Models in the left navigation panel and click Create to define a new API Gateway model:

Create another model for items

You should now see both Item and Items beneath Models in the left navigation. You are now ready to create API resources and methods.

Step 3: Create Resources and Methods

Click on Resources in the left navigation pane, then Actions > Create Resource.

Resources are the key abstraction for information in a RESTful API. You will create resources for lists of items and individual items.

In the New Child Resource form, enter the following values and then click Create Resource:

The /items resource is now selected in the Resources panel. Notice there are no methods created yet. You will add a GET method to retrieve a list of items.

Click on Actions > Create Method and select GET from the drop-down menu. Then click the check mark to confirm. In /items - GET - Setup form set the Integration type to Mock and click the Save button.

The mock integration allows you to define API resources and methods whenever you don’t have a real backend implementation yet. The advantage of this is that you can now use the Items model previously created to generate fake data. For example, this would allow you to provide a temporary API endpoint to your frontend/mobile developer, even before you have a real implementation.

In the Method Execution diagram, click on Integration Response:

Click the right arrow to expand the 200 response status, expand the Mapping Templates section and click on application/json. Here you can automatically generate a mapping template by selecting the Items model from the drop-down menu in the Generate template field:

AWS generates a mapping template with random fake data, based on your JSON Schema. Given that the model defines a list, the template generates a foreach loop, which can be customized before saving.

Overwrite the generated template with the following sample code to generate a list of 3 Item objects, with different IDs, prices and names:

[
#foreach($ID in [1, 2, 3])
    #set($price = $ID * 1.5)
    #set($name = "foo" + $ID)
    {
        "ID" : $ID,
        "name" : "$name",
        "price" : $price
    }
    #if($foreach.hasNext),#end
#end
]

At the top of the panel, click on <- Method Execution to return to the diagram. Click TEST to open the Method Test form and then click the Test button:

The method returns the template you defined.

You will now define an API resource for individual items with the /items/{ID} path.

Select /items in the Resources panel and click Actions > Create Resource.

You only need to insert the {ID} part of the path since /items is automatically added when creating a new resource with /items selected in the resource panel. This makes the Item resource is a child of the Items resource. The {ID} path variable makes the path dynamic. This way, API Gateway will treat any value as a dynamic ID and pass it to the backend implementation, which will return the corresponding data.

Create a GET method with a Mock integration. Modify the Integration Response Mapping Templates to the following:

This template extracts the dynamic ID parameter, casts it to an integer and returns an Item object with fake generated data. Test the new /items/{ID} resource as you did previously. This time you need to enter a value under {ID} before clicking the Test button:

If you leave the ID field blank, an “Internal Server Error” is returned. Any integer value should return a valid JSON response. You will replace the simple Mock implementation with a real implementation in the following steps. For now, you can proceed to deploy the API with the mock backends.

Step 4: Deploy the API

Click Actions > Deploy API and select [New Stage] from the Deployment stage drop-down menu. Enter the following values and then click Deploy:

The API is now deployed and ready to serve requests at the /items and /items/{ID} endpoints in the dev stage/environment.

Repeat the Deploy API action from the Resources panel action again, this time creating the prod (production) stage.

Summary: In this step, you have deployed an API Gateway API in dev and prod stages by:

  • Creating a model
  • Creating API resources
  • Creating API resource methods using a mock backend integration
  • Deploying the API to each stage

In the next steps, you will create and implement a Lambda function as a backend of the two HTTP endpoints you have defined.

Create an AWS Lambda Function Backend

You designed and created an API Gateway resource in the previous section, but it doesn’t have a real implementation yet. We designed and created two API endpoints using the AWS API Gateway. Now those two endpoints display a list of items or a specific item from a published URL endpoint for us. Our endpoints currently display mock data only using the AWS Gateway’s mock data setting. Now we are going to create a new Lambda function that we can connect up to both of these two endpoints.

In this section, you will create a new Lambda function that will handle both endpoints.

This approach has the advantage of reducing the number of functions you need to maintain (and therefore the amount of code), besides partially solving the cold-start issue of AWS Lambda. The cold-start issue with Lambda is due to the initial startup phase of your functions code, which might take up to a few seconds (worst case). By having fewer functions, you increase the likelihood of keeping them warm even if you don’t have a very high load.

The Lambda function you will create will dynamically check whether an Item ID or the entire list has been requested and behave accordingly. Since you already created an API, you will not use the simplified flow to create new Lambda-backed API Gateway resources. This flow is recommended whenever you start from scratch with Lambda and you don’t need much in the way of API Gateway configuration, which is the norm in most simple use cases.

Create a lambda function and create three tests.

First create the lambda function and select an appropriate IAM role for it.

Scroll down to the Function code section, and overwrite the contents of the lambda_function.py file with the following code:

# static list of items
items = [
  {"ID": 1, "name": "Pen", "price": 2.5},
  {"ID": 2, "name": "Pencil", "price": 1.5},
  {"ID": 3, "name": "Highlighter", "price": 3.0},
  {"ID": 4, "name": "Ruler", "price": 5.0},
]

def lambda_handler(event, context):
  print("Event: %s" % event) # log event data
  ID = event.get("ID") # extract ID

  # list case
  if not ID:
    return items

  # ID case
  found = [item for item in items if item["ID"] == ID]
  if found:
    return next(iter(found))

  # nothing was found
  raise Exception("NotFoundError")

The function is very simple, without any specific dependency or infrastructure requirement.

The logic is quite straightforward:

  • if no ID is given, return the whole list
  • if the ID exists, return the corresponding Item
  • otherwise, raise an error

The other Lambda configuration fields can be left at their defaults.

Create test cases

Create some test cases for this function:

In the Configure test event form, enter the following values into the form:

Hit Test, and you should see the execution results at the top:

Repeat the test process to test two more code paths for the following cases:

  • An object such as {“ID”: 1}, which will return only one item.
  • An object such as {“ID”: 5}, which will return an NotFoundError.

Please note that in a real-world scenario you’d normally perform these operations on a database (DynamoDB, RDS, Firebase, etc.). In the simple scenario, a static list of objects in main memory is more than enough. Also, keep in mind that the code will run on multiple machines (i.e. containers) and we will not be able to update the in-memory list consistently. In fact, only GET methods are defined and no transformation on the Items are allowed.

In this step, you created and tested the Lambda function that will serve as the backend for your API in API Gateway. Before you can use it, it is best to understand versioning so you can use different versions of the function in the dev and stage environments. The next step will show you how to work with function versions.

Versioning and Aliasing the Lambda Function

In this step, you will create different versions of the Lambda function corresponding with the dev and prod stages you created in API Gateway.

Each Lambda function has a default $LATEST version, which is the one you can always work on and edit. Once your code is stable enough or whenever your code changes significantly, you can create a new version. A simple incremental number will be assigned to the new version and you will be able to use and test any version. As you can imagine, this is a useful mechanism to keep track of your functions history and eventually rollback to previous versions.

Using versions is helpful, but sometimes even that is not flexible enough. For example, you can only bind your API Gateway resources and methods to a specific version of your Lambda function: whenever you create a new Lambda Version, you’d need to update your API Gateway configuration as well. Fortunately, AWS provides the concept of alias to improve this situation.

An alias is a useful abstraction that allows you to refer to a function version without actually using a version number. For example, you may want to create a prod alias and connect it to your API Gateway production stage. You could do the same with a dev alias, bound to your development stage.

The best practice would be always referencing to aliases when configuring your API Gateway backend integrations. By default, API Gateway will point to the $LATEST version, but you can always configure it to use a specific version or alias.

The following are examples of how the version and alias mappings work:

  • ItemsFunction -> $LATEST Version
  • ItemsFunction:1 -> Version 1
  • ItemsFunction:prod -> prod alias -> Version 1

You will now create a new version (this gets created when you hit publish) and two new aliases for the Lambda function.

The dev stage will always work with the latest version of the function. By binding the dev alias to our $LATEST version, you will be able to quickly implement changes and test them without explicitly publishing new versions. Of course, you can always adapt this configuration to your own needs.

You usually don’t want to be so aggressive in automatically releasing the latest function to production. You will alias prod to a specific version instead.

Note: When no alias is selected (the Unqualified alias is selected), the menu is labeled Qualifiers. You can use the Qualifiers menu to switch between versions of the function.

Configuring API Gateway Backend

Next you will update the API Gateway Integration Request from Mock to AWS Lambda. You will map both API endpoints to the same Lambda function. Remember, when you simply select the Lambda Function name, API Gateway will use its $LATEST version. Using $LATEST is generally not recommended since you always want to bind your API Resources to stable Lambda aliases (dev, stage, etc.).

At this point, you will be prompted with these questions:

This is required because, by default, nobody can invoke your Lambda functions unless you grant the right IAM permission. If you configure API Gateway via the Console, this operation will be automatically performed by AWS.

Now do the same to the prod stage, by editing and click the ok button next to it. It will give you the same two prompts.

Now repeat the same for the /items/{ID}/item URL for both dev and prod stages.

Click Integration Request and set the following values:

  • Integration type: Lambda Function
  • Use Lambda Proxy integration: unchecked
  • Lambda Region: us-west-2
  • Lambda Function: ItemsFunction:dev
  • Mapping Templates:
    • Request body passthrough: When there are no templates defined

This time you need to configure the Mapping Templates because the resource includes a dynamic {ID} variable. You will map the ID path parameter to a Lambda event parameter, via a custom mapping template.

Click on application/json and scroll down to the template editor and overwrite the default contents with the following and click Save below the editor:

{
  "ID": $input.params("ID")
}

At this point, all the API Gateway Resources can access both Lambda Aliases and we won’t need to manually grant any additional permission. You now need to re-deploy the API into the prod stage.

Click Actions > Deploy API and select the prod Deployment stage before clicking Deploy.

If everything went fine, the same endpoint will finally return real data from the Lambda function, instead of fake data.

In theory, you’d need to deploy the dev-configured Resources into the dev Stage and the prod-configured resources into the prod Stage. For now, you can just deploy the same configuration into both stages (dev and prod) since you will completely change the backend configuration in the next step, to make it more dynamic and flexible.

Click Actions > Deploy API and select the dev Deployment stage before clicking Deploy.

Always remember that you can test your API integration in the AWS Console before deploying it (return to Method Execution and click TEST). This way you will also be shown the corresponding API Gateway logs for debugging purposes.

Summary: In this step, you configured the API backend integration to use the Lambda function you created. You deployed the updated API to both the dev and prod stages.

Following Best Practices for Versions, Aliases and Stages

You have previously created the following resources:

  • Lambda function (ItemsFunction)
  • Lambda Versions ($LATEST and 4)
  • Lambda Aliases (dev and prod)
  • API Gateway Stages (dev and prod)
  • API Gateway Resources (ItemsList and Item)

Both the API endpoints are attached to the same Lambda function and they currently use its version 4. This configuration is not very robust and doesn’t allow you to update your Lambda function and see the updates in your dev environment.

The recommended best practice is connecting your API Gateway stages to the corresponding Lambda Alias, so that you can easily manage new functionalities, testing, bug fixing, rollbacks, etc. You can achieve this setup by creating a new stage variable and then use this variable as a Lambda alias in your API Gateway backend configuration. Stage Variables can be used to configure the request integration of your resources so that you don’t have to modify them every time you want to re-deploy your API.

Repeat the same for prod stage, by selecting the prod stage and create a lambdaAlias stage variable with the value of prod.

Now you can update the API Gateway Resources’ configuration to use the stage variable.

For both the /items GET method and the /items/{ID} GET method, modify the Integration Request Lambda Function to ItemsFunction:${stageVariables.lambdaAlias}. Click OK to the Add Permission to Lambda Function modals that popup.

You can grant the permission with the AWS CLI, as recommended by the modal. For the current setup, you don’t need to grant any additional permission because you already granted API Gateway access to the dev and prod aliases that are referenced by the stage variables.

Re-deploy your dev and prod stages before proceeding (Actions > Deploy API).

To test the API, you can no longer use the Test button in the Method Execution diagram because the method now depends on being executed in the context of a specific stage.

Click on Stages > prod and copy the Invoke URL:

Paste it in the browser by appending items

The list of items is returned, confirming that the stage variable for the prod stage is configured correctly.

In this step, you configured stage variables to reference the Lambda function used for the API Gateway backend integration. Now that API Gateway is correctly configured you are free to update the Lambda function code and play with its alias mapping without coming back to API Gateway.

Creating API Keys and Usage Plans

The API Resources are still open, meaning that no authorization is required. Usually, you will need to secure your API and eventually have a granular way to restrict their access.

In this step, you will see the most basic authorization method via API Keys, managed by AWS. Please note that API Keys are intended to track API consumers and define custom throttling and rate limiting.

In the API Gateway Console, you can create or import API Keys and associate them with your API Gateway stages. AWS also offers the concept of usage plans, which allow you to manage throttling and quotas in a more granular way. Technically, each API Key must be bound to one or more Usage Plans, which can be bound to one or more API stages. This way, you can control and monitor each API Key’s usage and cluster similar keys together based on your own needs.

In the API Gateway Console, select Usage Plans in the left navigation panel and click Create:

You will enable throttling and a quota in a later step.

In the Associated API Stages form, click Add API Stage and set the following values before clicking the checkmark icon:

Click Add API Stage again and associate the plan to the ItemsAPI prod stage as well.

In most situations, you would create a different plan for each stage so that you can also have independent API Keys. Within the scope of this post, you can simply bind both stages to the same usage plan.

Click Next and Done to finish creating the usage plan.

Select API Keys in the left navigation panel and click on Actions > Create API Key.

In the Create API Key form, enter the following values before clicking Save:

Once the API Key is created, you can bind it to the two API Gateway stages.

Click on Add to Usage Plan and enter CloudAcademyPlan as the plan name before clicking the checkmark icon:

Click Show next to API Key in the upper section of the panel to reveal the API Key (40-char alphanumeric string):

You will use this API Key as a custom HTTP header later on. But first, you need to update the API Gateway to require API Keys.

For both the /items and /items/{ID} resource GET methods, click Method Request in the diagram and set API Key Required to true:

Re-deploy the API to both stages (Actions > Deploy API).

You won’t be able to simply GET the resources in your browser anymore. In that case, since the endpoint is not open anymore, you’d receive a {“message”: “Forbidden”} response.

Run curl -H "x-api-key: <api-key>" https://bx0tcb7amj.execute-api.us-west-2.amazonaws.com/prod/items

(base) shravan-aws# curl -H "x-api-key: your-api-key" https://bx0tcb7amj.execute-api.us-west-2.amazonaws.com/prod/items
[{"ID": 1, "name": "Pen", "price": 2.5}, {"ID": 2, "name": "Pencil", "price": 1.5}, {"ID": 3, "name": "Highlighter", "price": 3.0}, {"ID": 4, "name": "Ruler", "price": 5.0}]
(base) shravan-aws#

Enabling CORS on API Gateway Resources

CORS stands for Cross-origin resource sharing. It is a set of standard HTTP headers used to restrict the access of web resources from other domains. In the web app security model, this is called same-origin policy, and it’s supposed to avoid cross-site scripting (XSS) attacks. Browsers will not fetch the requested resources unless the corresponding server attaches the required HTTP headers in the OPTIONS call (or GET method).

The API Gateway will not automatically add these headers, but it provides a very user-friendly wizard. In this step, you will see how to enable CORS on our API Gateway resources. Keep in mind that you need to enable it on each resource.

Select the /items API Resource and click on Actions > Enable CORS:

This starts a wizard that allows you to configure which headers to serve and which headers are allowed. A minimal CORS integration would require at least the Access-Control-Allow-Origin header. If its value is *, every domain will be allowed to fetch this resource. The API Gateway will automatically allow GET and OPTIONS methods, plus all the default headers used by AWS services (Content-Type, X-Amz-Date, X-Api-Key, X-Amz-Security-Token and Authorization). This will guarantee that you can use API Keys, Basic Authentication and IAM tokens.

Click Enable CORS and replace existing CORS headers.

An additional confirmation popup will follow, with a list of all the required operations that will be executed for you:

Note that there is not too much magic going on here. The API Gateway will simply configure itself to enable CORS on the specified resource, by creating a new OPTIONS method and all the required Method Response and Integration Response options.

Note: The OPTIONS method will be invoked by the browser as a preflight request, before the actual GET request. If the OPTIONS call doesn’t succeed, browsers won’t even issue your GET request.

In this step, you used the API Gateway wizard for enabling CORS on both of your API resources. CORS enhances the security of web apps.

Enabling API Gateway Caching and Throttling

You can configure throttling and caching independently for each API Gateway stage. For example, you may want to avoid throttling and caching on your dev stage, and enable them on your prod stage.

Please note that although throttling is free, caching is provided by a custom instance whose size and cost depend on the selected cache capacity (from 500MB to 237GB). Optionally you can encrypt your cache data, set up a time-to-live (TTL) in seconds and require authorization for cache invalidation requests.

You will configure caching and throttling for the prod stage in this step.

In case you have special customers and you want to limit their access to your API, you can configure the throttling and daily/weekly/monthly quota on the corresponding Usage Plan.

Select Usage Plans > CloudAcademyPlan > Details > Edit and observe the Throttling and Quota sections:

Check Enable quota and enter a quota of 1000 requests per Day before clicking Save:

This configuration will ensure that each individual user (as identified by unique API keys) won’t consume your stage/account capacity. Of course, you can monitor and update these configurations at any time, even after creating a Usage Plan or an API Key.

In this step, you enabled API caching and throttling on the prod stage. You also enabled a quota at the usage plan-level to limit the number of requests authorized by an API Key per day.

Cleaning up API Resources and Lambda Functions

Delete the following resources in the given order to avoid conflicts:

  • API Key (API Keys > Delete API Key)
  • API Stages (Stages > dev/prod > Delete Stage)
  • Usage Plan (Actions > Delete Usage Plan)
  • API Resources (Actions > Delete API)
  • Lambda Function (Actions > Delete function)

Please note that you won’t be able to delete Usage Plans if at least one API Stage is bound to it. That’s why you’ll need to delete stages first.

Alternatively, you can just delete the API altogether with its stages. You will need to confirm the API name in order to delete every API resource, models, stages, stage variables, etc.

By deleting the Lambda Function, you will also delete all the related versions and aliases.