Docker on AWS EC2

Docker on AWS

Docker has taken DevOps by storm and ignited interest in containers. Containers offer many of the benefits of virtual machines but in a much more efficient, less resource-intensive system. Containers allow you to package up an application in an isolated environment that can be executed across machines in a reproducible manner. No more “well, it worked on my machine” syndrome when you leverage Docker for your application development and deployment.

In this post, you will get up and running with Docker on Linux using an AWS virtual machine.

You will work with images from the public Docker registry, run a handful of containers, and create your own image from which to create containers.

Upon completion of this post you will be able to:

  • Install Docker on Linux using an AWS virtual machine
  • Add a user to the Docker group
  • Find and use images from the public Docker Registry
  • Build your own images using Dockerfiles

aws_docker

Connect to the EC2 instance

Launch and connect to an EC2 instance.

(base) shravan-Downloads# ssh -i 481730208893.pem ec2-user@54.213.31.218
The authenticity of host '54.213.31.218 (54.213.31.218)' can't be established.
ECDSA key fingerprint is SHA256:uH8q9rXZB7bjEe0o4g1YH80Lw9hlfkUT0NuJxa4RKUM.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '54.213.31.218' (ECDSA) to the list of known hosts.

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
35 package(s) needed for security, out of 109 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-10-0-0-217 ~]#

Install Docker

You will install Docker using the yum package manager that is available on Amazon Linux. Docker comes in two flavors: Community Edition (CE) and Enterprise Edition (EE). The community edition is open source and available free of charge. It includes the core Docker functionality and was previously called “Docker Engine.” The enterprise edition requires a subscription and includes a support package, a certification program to create trusted containers and extensions, and other features. You will be installing the community edition. You will also see how to add your user to the docker group to use the Docker commands without elevating to root permissions.

  1. Enter the following to install Docker: sudo yum -y install docker
  2. Enter the command below to start Docker as a service: sudo systemctl start docker
  3. Verify Docker is running by entering: sudo docker info. This will output system-wide information about Docker. The information will resemble the following:
[ec2-user@ip-10-0-0-217 ~]# sudo docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.6-ce
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ff48f57fc83a8c44cf4ad5d672424a98ba37ded6
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.14.123-111.109.amzn2.x86_64
 Operating System: Amazon Linux 2
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 983.7MiB
 Name: ip-10-0-0-217.us-west-2.compute.internal
 ID: MZL6:ZYZS:SREY:JZRK:BBJM:IGMB:SWHX:KFO6:QVZQ:ZBZD:I4AL:354B
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[ec2-user@ip-10-0-0-217 ~]#

You can see some useful information, such as the number of containers, and the version of the Docker server. Docker adopts a client-server architecture, so the server doesn’t have to be running on the same host as the client. In your case, you are using the Docker command line interface (CLI) client to connect to the server, called the Docker daemon.

So far, you added the official Docker CE repositories to the yum package manager and installed Docker CE. You also learned your first Docker command to print system-wide information relevant to Docker.

Using Docker without Root Permission on Linux

So far you used root permissions to run a Docker command. In this step you will learn how to use Docker commands without elevating to root permissions. This may not always be what you want and should only be performed for trusted users. This is because the actions are equivalent to granting root permissions to a user. It will be convenient in the following steps to not have to enter sudo for every command.

Try running docker info as ec2-user, you will get the following

[ec2-user@ip-10-0-0-219 ~]# docker info
Client:
 Debug Mode: false

Server:
ERROR: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/info: dial unix /var/run/docker.sock: connect: permission denied
errors pretty printing info
[ec2-user@ip-10-0-0-219 ~]#

By default, the Docker daemon will reject requests from users that aren’t part of the docker group. If you encounter this message in your travels, you can either use root permission or add your user to the docker group.

Verify the docker group exists by searching for it in the groups file: grep docker /etc/group or getent group docker

[ec2-user@ip-10-0-0-219 ~]# getent group docker
docker:x:993:
[ec2-user@ip-10-0-0-219 ~]#

If you don’t see a line beginning with “docker:”, you will need to add the group yourself by entering: sudo groupadd docker. However, in our case, we don’t need to.

Next, add your user (ec2-user) to docker group: sudo gpasswd -a <dollar-sign>USER docker. The groups of the currently logged in user is cached, you can verify this by entering groups.

You can login again to have your groups updated by entering: newgrp docker. Note: It is convenient to not have to terminate your current ssh session by using newgrp, but terminating the ssh session and logging in again will work just as well.

[ec2-user@ip-10-0-0-219 ~]# sudo gpasswd -a <dollar-sign>USER docker
Adding user ec2-user to group docker

[ec2-user@ip-10-0-0-219 ~]# groups
ec2-user adm wheel systemd-journal

[ec2-user@ip-10-0-0-219 ~]# newgrp docker
[ec2-user@ip-10-0-0-219 ~]# groups
docker adm wheel systemd-journal ec2-user
[ec2-user@ip-10-0-0-219 ~]#

Verify that your user can successfully issue Docker commands by entering: docker info

Getting Docker Help from the Command Line

Here you will learn about how Docker commands are organized and how to get help for commands on the command line. If you are familiar with using the Linux command line or the git version control system, you will find many of the commands familiar in name and action. Even if you aren’t familiar, the commands are organized in an intuitive way that makes learning to use Docker commands as painless as practicable.

The commands are organized into common commands and a more exhaustive list grouped around the management of a specific component of Docker. You use each with a different syntax. For a common command, the usage is:

docker command-name [options]

and the usage for a management group command is:

docker management-group command-name [options]

Most of the common commands can be accessed in the short form or by specifying the management grouping they also fall under. Some commands are only available by specifying the management group. You will get the hang of this and review a couple of the main concepts in the following instructions.

The management commands and common commands are highlighted in the image below:

aws_docker2

To see all of the commands grouped under system, enter: docker system --help

[ec2-user@ip-10-0-0-93 ~]# docker system --help

Usage:	docker system COMMAND

Manage Docker

Commands:
  df          Show docker disk usage
  events      Get real time events from the server
  info        Display system-wide information
  prune       Remove unused data

Run 'docker system COMMAND --help' for more information on a command.
[ec2-user@ip-10-0-0-93 ~]#

Notice the info command previously entered. events is a common command, but df and prune are only available through the system management command. You can use df to see the disk usage for Docker and prune to clean up unused data.

docker images: To view the commands grouped with images, enter: docker images --help

aws_docker3

Images are read-only snapshots that containers can be created from. Images are built up in layers, one image built on top of another. Because each layer is read-only, they can be identified with cryptographic hash values computed from the bytes of the data. Layers can be shared between images as another benefit of being read-only. You can, and will later, build your own images. The build command accomplishes that. When you build your own image, you will select a base image to build on top of with your custom application. A pull can be used to pull or download an image to your server from an image registry, while push can upload an image to a registry. Read through the other command descriptions so you are aware of what else is available.

docker containers: To view the commands grouped with containers, enter: docker container --help

aws_docker4

A container is another core concept in Docker. Containers run applications or services, almost always just one per container. Containers run on top of an image. In terms of storage, a container is like another layer on top of the image, but the layer is writable instead of read-only. You can have many containers using the same image. Each will use the same read-only image and have its own writable layer on top. Containers can be used to create images using the commit command, essentially converting the writable layer to a read-only layer in an image.

The relationship between images and containers aside, run is used to run a command on top of an image. A container can be stopped and started again. ls is used to list containers. It is aliased to ps and list as well. Read through the other commands in the list to see what else is available for working with containers.

Running your first Docker Container

As you learned in the previous Lab Step, containers run a command on top of an image. There are many available images to choose from. Images are collected in repositories. A repository can have many versions of an image. Tags are used to manage versions. You will use images from repositories inside the default Docker Registry, Docker Hub. Docker Hub hosts official images as well as community-contributed images. Docker Hub offers free and paid accounts. You get an unlimited amount of public repositories and one free private repository with the free account.

In this Lab Step, you will run your first container. You will learn how to search Docker Hub, pull images, and work with containers running on top of the images. You will get practice with many common commands when working with Docker.

Enter the following to see how easy it is to get a container running: docker run hello-world

aws_docker5

There are two sections to the output: output from Docker as it prepares to run the container and the output from the command running in the container.

Take a look at the Docker output first:

  1. In the first line, Docker is telling you that it couldn’t find the image you specified, hello-world, on the Docker Daemon’s local host. The latest portion after the colon (:) is a tag. The tag identifies which version of the image to use. By default, it looks for the latest version.
  2. In the next line, it notifies you that it automatically pulled the image. You could manually perform that task using the command docker pull hello-world. The library/hello-world is the repository it’s pulling from inside the Docker Hub registry. library is the account name for official Docker images. In general, images will come from repositories identified using the pattern account/repository.
  3. The last three lines confirm the pull completed and the image has been downloaded.

The container output is the output that gets written by the command that runs in the container. Read the output to learn more about how it was produced. You ask, but what was the command? You only specified the image name, hello-world. The image provides a default command in this case which prints the output you see.

Nginx: Try running a more complex container with some options specified:

docker run --name web-server -d -p 8080:80 nginx:1.12

This runs the nginx web server in a container using the official nginx image. You will see output similar to:

[ec2-user@ip-10-0-0-93 ~]# docker run --name web-server -d -p 8080:80 nginx:1.12
Unable to find image 'nginx:1.12' locally
1.12: Pulling from library/nginx
f2aa67a397c4: Pull complete
e3eaf3d87fe0: Pull complete
38cb13c1e4c9: Pull complete
Digest: sha256:72daaf46f11cc753c4eab981cbf869919bd1fee3d2170a2adeac12400f494728
Status: Downloaded newer image for nginx:1.12
b743cc41052b983b2789569e1ac2b307a20c14197fdfa95dcdecb4c3e573a1ba
[ec2-user@ip-10-0-0-93 ~]#

This time you specified the tag 1.12 indicating you want version 1.12 of nginx instead of the default latest version. There are three Pull complete messages this time, indicating the image has three layers. The last line is the id of the running container. The meanings of the command options are:

  • --name container_name: Label the container container_name. In the command above, the container is labeled web-server. This is much more manageable than the id.
  • -d: Detach the container by running it in the background and print its container id. Without this, the shell would be attached to the running container command and you wouldn’t have the shell returned to you to enter more commands.
  • -p host_port:container_port: Publish the container’s port number container_port to the host’s port number host_port. This connects the host’s port 8080 to the container port 80 (http) in the nginx command.

You again used the default command in the image, which runs the web server in this case.

[ec2-user@ip-10-0-0-93 ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
b743cc41052b        nginx:1.12          "nginx -g 'daemon of…"   3 minutes ago       Up 3 minutes        0.0.0.0:8080->80/tcp   web-server
[ec2-user@ip-10-0-0-93 ~]#

Stop nginix server: To stop the nginx server, enter: docker stop web-server

[ec2-user@ip-10-0-0-93 ~]# docker stop web-server
web-server
[ec2-user@ip-10-0-0-93 ~]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[ec2-user@ip-10-0-0-93 ~]#

start nginx:

[ec2-user@ip-10-0-0-93 ~]# docker start web-server
web-server
[ec2-user@ip-10-0-0-93 ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
b743cc41052b        nginx:1.12          "nginx -g 'daemon of…"   6 minutes ago       Up 3 seconds        0.0.0.0:8080->80/tcp   web-server
[ec2-user@ip-10-0-0-93 ~]#

This is different from re-running the original docker run command, which would make a second container running the same command instead of using the stopped container.

view logs:

ec2-user@ip-10-0-0-93 ~]# docker logs web-server
172.17.0.1 - - [23/Apr/2020:02:43:56 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.61.1" "-"
[ec2-user@ip-10-0-0-93 ~]#

Docker logs messages written to standard output and standard error. In the case of nginx, it writes a line for each request that it receives.

run commands in a running container: You can run other commands in a running container. For example, to get a bash shell in the container enter: docker exec -it web-server /bin/bash

This indicates you are at a shell prompt in the container using the root container user. The -it options tell Docker to handle your keyboard events in the container. Enter some commands to inspect the container environment, such as ls and cat /etc/nginx/nginx.conf. When finished, enter exit to return to the VM ssh shell. Your shell prompt should change to confirm you are no longer in the container bash shell.

list files in the container’s /etc/nginx directory: docker exec web-server ls /etc/nginx

[ec2-user@ip-10-0-0-93 ~]# docker exec web-server ls /etc/nginx
conf.d
fastcgi_params
koi-utf
koi-win
mime.types
modules
nginx.conf
scgi_params
uwsgi_params
win-utf
[ec2-user@ip-10-0-0-93 ~]#

This runs the ls command and returns to the ssh shell prompt without using a container shell to execute the command. What commands are supported depends on the layers in the container’s image. Up until now you have used two images but don’t know how to find more.

Search for images:

[ec2-user@ip-10-0-0-93 ~]# docker search airflow
NAME                                DESCRIPTION                                     STARS               OFFICIAL            AUTOMATED
apache/airflow                      Apache Airflow                                  103
astronomerio/airflow                Airflow                                         24                                      [OK]
bitnami/airflow                     Bitnami Docker Image for Apache Airflow         6
camil/airflow                       Airflow is a system to programmatically auth…   5                                       [OK]
astronomerio/airflow-saas                                                           4
bitnami/airflow-scheduler           Bitnami Docker Image for Apache Airflow Sche…   4
bitnami/airflow-worker              Bitnami Docker Image for Apache Airflow Work…   2
godatadriven/airflow                Apache Airflow                                  2                                       [OK]
pbweb/airflow-prometheus-exporter   Export airflow metrics (collected from mysql…   2
airflowdocker/example-tasks                                                         1
groupmeridian/airflow               Apache Airflow build for Openshift              1                                       [OK]
airflowdocker/aws-autoscaler                                                        1
airflowci/incubator-airflow-ci      A Docker container used for running Docker C…   1                                       [OK]
datagovsg/airflow-pipeline          An Airflow setup that aims to work well with…   1                                       [OK]
yieldr/airflow                      Airflow                                         1                                       [OK]
lowess/airflow-demo                 Airflow Docker image with Example DAGs          0
duanconsulting/airflow-etl                                                          0
hmike96/airflow                                                                     0
guangie88/airflow-pipeline          Airflow + Spark (k8s) + Hadoop + SQLAlchemy …   0
fahad0000/airflow-example           Airflow example                                 0                                       [OK]
terragontech/airflow                Terragons version of the Airflow Image         0
thajib/airflow                                                                      0
x10232/airflow                                                                      0
flowminder/airflow                  FM airflow container                            0
usagemeter/airflow                  Airflow used for vCloud Usage Insight           0
[ec2-user@ip-10-0-0-93 ~]#

Creating a Docker Image

Docker containers run on top of images. You have seen how to use images on Docker’s public registry, the Docker Hub. There are many different images available. It is worth trying to find existing images when you can. Inevitably, you will need to create your own images when making your own applications. In that case, you will still want to invest time into finding the right base layer to add your own layer(s) on top of.

There are a couple of ways to make images. You can use docker commit to create an image from a container’s changes. The changes may come from using exec to open a shell in the container like in the previous Lab Step. The other method is using a Dockerfile. A Dockerfile is easier to maintain, easier to repeatedly create images from, and distributions easier. You will create a Dockerfile in this Lab Step. Just know that it is possible to create equivalent images using commits.

Dockerfiles specify a sequence of instructions. Instructions can install software, expose network ports, set the default command for running a container using the image, and other tasks. Instructions can really handle anything required to configure the application. Many of the instructions add layers. It is usually a good idea to keep the number of layers to a reasonable number. There is overhead with each layer, and the total number of layers in an image is limited. When the Dockerfile is ready, you can create the image using the docker build command.

You will see how all of this comes together by creating an image of a Python Flask web app that randomly chooses what type of Cloud Academy content you should look at next. The choice of Python as the example app is arbitrary. You will not focus on the specifics of the programming language. You should be able to repeat the process for any other programming language by following a similar process. Whatever programming language or framework you are working with, you should consult the Docker Hub documentation for the image, as it will usually include advice on how to structure your Dockerfile.

Install git: sudo yum -y install git You will clone a code repository with the Flask app using Git.

[ec2-user@ip-10-0-0-192 ~]# git clone https://github.com/cloudacademy/flask-content-advisor.git
Cloning into 'flask-content-advisor'...
remote: Enumerating objects: 25, done.
remote: Total 25 (delta 0), reused 0 (delta 0), pack-reused 25
Unpacking objects: 100% (25/25), done.

[ec2-user@ip-10-0-0-192 ~]# pwd
/home/ec2-user
[ec2-user@ip-10-0-0-192 ~]# ls
flask-content-advisor
[ec2-user@ip-10-0-0-192 ~]# cd flask-content-advisor

[ec2-user@ip-10-0-0-192 flask-content-advisor]# ls
README.md  requirements.txt  src

Create a Dockerfile: From the same directory, create a Dockerfile. aws_docker6

  • FROM sets the base layer image
  • COPY . . copies all of the files in the code repository into the container’s /usr/src/app directory
  • RUN executes a command in a new layer at the top of the image
  • EXPOSE only indicates what port the container will be listening on, it doesn’t automatically open the port on the container
  • CMD sets the default command to run when a container is made from the image

There are more instruction types, but this lab will only focus on those mentioned. After completing the lab, you can review all of the instructions available in Dockerfiles at the Dockerfile reference web page.

Build the docker image: docker build -t flask-content-advisor:latest . The -t tells Docker to tag the image with the name flask-content-advisor and tag latest. The . at the end tells Docker to look for a Dockerfile in the current directory. Docker will report what it’s doing to build the image. Each instruction has its own step. Steps one and four take longer than the others. Step one needs to pull several layers for the Python 3 base layer image and Step four downloads code dependencies for the Flask web application framework. Notice that each Step ends with a notice that an intermediate container was removed. Because layers are read-only, Docker needs to create a container for each instruction. When the instruction is complete, Docker commits it to a layer in the image and discards the container.

[ec2-user@ip-10-0-0-192 flask-content-advisor]# docker build -t flask-content-advisor:latest .
Sending build context to Docker daemon  90.11kB
Step 1/6 : FROM python:3
3: Pulling from library/python
90fe46dd8199: Pull complete
35a4f1977689: Pull complete
bbc37f14aded: Pull complete
74e27dc593d4: Pull complete
4352dcff7819: Pull complete
deb569b08de6: Pull complete
98fd06fa8c53: Pull complete
7b9cc4fdefe6: Pull complete
e8e1fd64f499: Pull complete
Digest: sha256:a361f3d856a57ad373ab972a9ab87733610fea91ddd92d46b5784b42a30c80c9
Status: Downloaded newer image for python:3
 ---> a6be143418fc
Step 2/6 : WORKDIR /usr/src/app
 ---> Running in fbcdf6a5c9a8
Removing intermediate container fbcdf6a5c9a8
 ---> a2c5203767e2
Step 3/6 : COPY . .
 ---> 682e99d5d203
Step 4/6 : RUN pip install --no-cache-dir -r requirements.txt
 ---> Running in 852510ad6c63
Collecting flask==0.12.2
  Downloading Flask-0.12.2-py2.py3-none-any.whl (83 kB)
Collecting itsdangerous>=0.21
  Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting Werkzeug>=0.7
  Downloading Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
Collecting click>=2.0
  Downloading click-7.1.1-py2.py3-none-any.whl (82 kB)
Collecting Jinja2>=2.4
  Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting MarkupSafe>=0.23
  Downloading MarkupSafe-1.1.1-cp38-cp38-manylinux1_x86_64.whl (32 kB)
Installing collected packages: itsdangerous, Werkzeug, click, MarkupSafe, Jinja2, flask
Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 Werkzeug-1.0.1 click-7.1.1 flask-0.12.2 itsdangerous-1.1.0
Removing intermediate container 852510ad6c63
 ---> 4484fd6a45d1
Step 5/6 : EXPOSE 5000
 ---> Running in 65cb672c6235
Removing intermediate container 65cb672c6235
 ---> 5e675ad2b813
Step 6/6 : CMD [ "python", "./src/app.py" ]
 ---> Running in 13abb5b30695
Removing intermediate container 13abb5b30695
 ---> 74be63a85da6
Successfully built 74be63a85da6
Successfully tagged flask-content-advisor:latest
[ec2-user@ip-10-0-0-192 flask-content-advisor]#

Record your VM’s public IP address: curl ipecho.net/plain; echo .You will need your IP to test that the web app is available with your browser. The echo is only to put the shell prompt on a new line.

[ec2-user@ip-10-0-0-192 flask-content-advisor]# curl ipecho.net/plain; echo
54.188.175.199

Run the container: Now you can run a container using the image you just built: docker run --name advisor -p 80:5000 flask-content-advisor

ec2-user@ip-10-0-0-192 flask-content-advisor]# docker run --name advisor -p 80:5000 flask-content-advisor
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 204-454-399

This runs a container named advisor and maps the container’s port 5000 to the host’s port 80 (http). This time you didn’t include -d to run in detached mode. That is why you see output and you don’t have the shell prompt returned to you. If you did run with -d, you could get the same information from docker logs.