Infrastructure as Code using AWS CloudFormation

Introduction

What is (Cloud) Infrastructure?: Cloud services are broadly categorized as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). Cloud is a collection of geographically distributed data centers that are logically grouped into regions and availability-zones.

The IaaS allows a user to provision Virtual Machines (VMs), set up networks (VPC), create subnets, and associate necessary security features. VMs can be attached to storage volumes, different networks or servers. All the resources mentioned above are referred to as Infrastructure.

The diagram below shows various cloud infrastructure resources that we will learn to provision on the AWS cloud using the AWS command line and cloud formation tools. These are the primary resources that are required to build a web app architecture.

cloudformation

Configuring and managing various infrastructure resources on AWS cloud using CloudFormation

Roadmap

Our main goal is to learn to provision cloud-infrastructure resources using (re-usable) scripts. We will use AWS as a cloud service provider. By the end of this blog-post-series, you will know:

  1. How to make use of the Cloud Formation tool to provision and manage cloud infrastructure.
  2. Create diagrams for web application architecture that represents how different resources can be placed in the cloud. Infrastructure Diagrams will take the help of LucidChart to draw diagrams.
  3. Provision networking components such as VPN, subnets, route tables, and internet gateway as a part of your cloud infrastructure. Networking Infrastructure describes all the networking components mentioned above.
  4. Deploy servers with the necessary level of security (firewall rules). Servers and Security Groups demonstrate the placement of servers (VMs) in private subnets and managing them using a jump-box.
  5. Attach the storage volumes to your servers. Storage and Databases describes configuring the S3 service.

Goal of this post

  • Describe Infrastructure as a Code (IaaC) as one of the best practices used in the DevOps model.
  • Configure basic settings to start using AWS services as an Identity and Access Management (IAM) user.
  • Explain the fundamentals of Cloud Formation.
  • Contrast the manual vs. automated (script-based) provisioning of Elastic Compute Cloud (EC2) instances in a Virtual Private Cloud (VPC).
  • Utilize the AWS command-line tool - CLI for necessary activities, such as configuring a VPC, or creating an IAM user.

What is DevOps and Why DevOps?

DevOps is the combination of industry best practices, and set of tools that improve an organization’s ability to:

  • Increase the speed of software delivery
  • Increases the speed of software evolution
  • Have better reliability of the software
  • Have scalability using automation,
  • Improved collaboration among teams.

In other words, these tools enable a company to showcase industry best practices in software development.

Cloud DevOps

So, what exactly is Cloud DevOps trying to solve? There are 3 main issues that DevOps tries to fix:

  • Unpredictable deployments - Operational team is not aware of the nuances of the code, so there is some unpredictability because they are the ones who didn’t write the code.
  • Mismatched environments (development doesn’t match production): This happens when the development environment is different than your production environment.
  • Configuration Drift - Caused by manual changes.

DevOps is a set of best practices and tools.

One of the benefits of using DevOps is that it allows predictable deployments using automated scripts. In the DevOps model, development and operations teams are merged into a single team. These DevOps teams use a few tools and best practices that deploy and manage configuration changes to servers.

The most important practices are:

  • Continuous Integration / Continuous Delivery (CI/CD) - new features are automatically deployed with all the required dependencies.
  • Infrastructure as Code (IaaC) - configuration and management of cloud infrastructure using re-usable scripts.

Other prevalent practices are:

  • Microservices
  • Monitoring and Logging
  • Communication and Collaboration

CloudFormation is a tool in AWS for managing, configuring and deploying infrastructure (push code along with the necessary server configurations).

Setup Environment

IAM setup

In order to be able to interact with AWS without a console, we need a set of API keys. Search for IAM, click on Add Users, give it a name, we will not be using this to login to the console, so just request programmatic access.

cloudformation2

Next, we are going to decide what this user can do by attaching them a policy. Click on “Attach existing policies directly”. In our case, we want the user to be able to create infrastructure resources like creating servers, networks etc, so we will assign this user to AdminstratorAccess.

cloudformation3

Dev and Prod user accounts:

In practice, Dev and DevOps members may have separate user accounts for the dev environment as opposed to the production environment. This makes it easier for developers by giving them wider privileges in the dev environment that would normally only be reserved for DevOps members in the production environment.

Setup Visual Studio Code

Download Visual Studio Code, create a new repo in github and clone it locally. Open the project folder. Setup the Vim Plugin and run this defaults write com.microsoft.VSCode ApplePressAndHoldEnabled -bool false from terminal. This command will allow you to move freely in VSCode.

CloudFormation

What is CloudFormation?:

  • CloudFormation is a declarative language, not an imperative language.
  • CloudFormation handles resource dependencies so that you don’t have to specify which resource to start up before another. There are cases where you can specify that a resource depends on another resource, but ideally, you’ll let CloudFormation take care of dependencies.
  • VPC is the smallest unit of resource.

  • Declarative languages: These languages specify what you want, without requiring you to specify how to get it. An example of a popular declarative language is SQL.

  • Imperative languages: These languages use statements to change the state of the program.

YAML and JSON: YAML and JSON file formats are both supported in CloudFormation, but YAML is the industry preferred version that’s used for AWS and other cloud providers (Azure, Google Cloud Platform).

An important note about YAML files: the whitespace indentation matters! We recommend that you use four white spaces for each indentation.

Having scripts specific for networking and other scripts specific to EC2 Servers or Databases keep your scripts small, and easily shared across teams with different skill sets, such as database administrators and network experts.

CloudFormation Scripts:

  • Name: A name you want to give to the resource (does this have to be unique across all resource types?)
  • Type: Specifies the actual hardware resource that you’re deploying.
  • Properties: Specifies configuration options for your resource. Think of these as all the drop-down menus and checkbox options that you would see in the AWS console if you were to request the resource manually.
  • Stack: A stack is a group of resources. These are the resources that you want to deploy, and that are specified in the YAML file.

Best practices: Coding best practice: Create separate files to organize your code. You can either create separate files for similar resources or create files for each developer who uses those resources.

Documentation for CloudFormation syntax: You don’t need to memorize the code that you need for each resource. You can find sample code in the documentation for CloudFormation for examples of how to write your CloudFormation scripts.

Here’s an example:

Description: >
    Shravan Kuchkula
    This template deploys a VPC
Resources:
    skvpc:
        Type: AWS::EC2::VPC
        Properties:
            CidrBlock: 10.0.0.0/16
            EnableDnsHostnames: true

How to run CloudFormation?

Now it is time for us to run the script and create a VPC. But before that, let’s just login to the aws console and check what we currently have:

cloudformation4

(base) shravan-cloudformation$ pwd
/Users/shravan/projects/cloudformation
(base) shravan-cloudformation$ ls -rtl
total 16
-rw-r--r--  1 shravan  staff   53 Mar  2 10:40 README.md
-rw-r--r--  1 shravan  staff  208 Mar  2 11:12 testcfn.yml

(base) shravan-cloudformation$ aws cloudformation create-stack --stack-name myfirststack --region us-west-2 --template-body file://testcfn.yml
{
    "StackId": "arn:aws:cloudformation:us-west-2:506140549518:stack/myfirststack/1998aed0-5ca4-11ea-b0e6-029123b1cc42"
}
(base) shravan-cloudformation$

Validate this by logging into the console:

cloudformation5

The most important thing to check are the events, which details the steps taken to create these resources:

cloudformation6

Alternatively you can create a shell script with the create-stack command as shown below:

aws cloudformation create-stack
--stack-name $1
--template-body file://$2  
--parameters file://$3
--capabilities "CAPABILITY_IAM" "CAPABILITY_NAMED_IAM"
--region=us-west-2

Where $1, $2, and $3 can be replaced with the actual values. Note the --parameters and --capabilities options that we will learn in the upcoming posts.

Delete the VPC: Run this command to delete the VPC that you just created. aws cloudformation delete-stack --stack-name myfirststack --region us-west-2

cloudformation7

Conclusion

In this post, we learnt what DevOps is at a high-level and the advantages of DevOps. The two pillars of DevOps are: CI/CD and IaaC. There are tools that will enable us to implement these best practices. We dived into learning how to setup our AWS account first, then setup our development environment, next we learnt CloudFormation tool as a way to implement IaaC and finally we learnt how to create a simple resource using CloudFormation and later destroyed it.