Docker on AWS
Docker has taken DevOps by storm and ignited interest in containers. Containers offer many of the benefits of virtual machines but in a much more efficient, less resource-intensive system. Containers allow you to package up an application in an isolated environment that can be executed across machines in a reproducible manner. No more “well, it worked on my machine” syndrome when you leverage Docker for your application development and deployment.
In this post, you will get up and running with Docker on Linux using an AWS virtual machine.
You will work with images from the public Docker registry, run a handful of containers, and create your own image from which to create containers.
Upon completion of this post you will be able to:
- Install Docker on Linux using an AWS virtual machine
- Add a user to the Docker group
- Find and use images from the public Docker Registry
- Build your own images using Dockerfiles
Connect to the EC2 instance
Launch and connect to an EC2 instance.
(base) shravan-Downloads# ssh -i 481730208893.pem firstname.lastname@example.org The authenticity of host '18.104.22.168 (22.214.171.124)' can't be established. ECDSA key fingerprint is SHA256:uH8q9rXZB7bjEe0o4g1YH80Lw9hlfkUT0NuJxa4RKUM. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '126.96.36.199' (ECDSA) to the list of known hosts. __| __|_ ) _| ( / Amazon Linux 2 AMI ___|\___|___| https://aws.amazon.com/amazon-linux-2/ 35 package(s) needed for security, out of 109 available Run "sudo yum update" to apply all updates. [ec2-user@ip-10-0-0-217 ~]#
You will install Docker using the yum package manager that is available on Amazon Linux. Docker comes in two flavors: Community Edition (CE) and Enterprise Edition (EE). The community edition is open source and available free of charge. It includes the core Docker functionality and was previously called “Docker Engine.” The enterprise edition requires a subscription and includes a support package, a certification program to create trusted containers and extensions, and other features. You will be installing the community edition. You will also see how to add your user to the docker group to use the Docker commands without elevating to root permissions.
- Enter the following to install Docker:
sudo yum -y install docker
- Enter the command below to start Docker as a service:
sudo systemctl start docker
- Verify Docker is running by entering:
sudo docker info. This will output system-wide information about Docker. The information will resemble the following:
[ec2-user@ip-10-0-0-217 ~]# sudo docker info Client: Debug Mode: false Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 19.03.6-ce Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: ff48f57fc83a8c44cf4ad5d672424a98ba37ded6 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 4.14.123-111.109.amzn2.x86_64 Operating System: Amazon Linux 2 OSType: linux Architecture: x86_64 CPUs: 1 Total Memory: 983.7MiB Name: ip-10-0-0-217.us-west-2.compute.internal ID: MZL6:ZYZS:SREY:JZRK:BBJM:IGMB:SWHX:KFO6:QVZQ:ZBZD:I4AL:354B Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false [ec2-user@ip-10-0-0-217 ~]#
You can see some useful information, such as the number of containers, and the version of the Docker server. Docker adopts a client-server architecture, so the server doesn’t have to be running on the same host as the client. In your case, you are using the Docker command line interface (CLI) client to connect to the server, called the Docker daemon.
So far, you added the official Docker CE repositories to the yum package manager and installed Docker CE. You also learned your first Docker command to print system-wide information relevant to Docker.
Using Docker without Root Permission on Linux
So far you used root permissions to run a Docker command. In this step you will learn how to use Docker commands without elevating to root permissions. This may not always be what you want and should only be performed for trusted users. This is because the actions are equivalent to granting root permissions to a user. It will be convenient in the following steps to not have to enter sudo for every command.
docker info as
ec2-user, you will get the following
[ec2-user@ip-10-0-0-219 ~]# docker info Client: Debug Mode: false Server: ERROR: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/info: dial unix /var/run/docker.sock: connect: permission denied errors pretty printing info [ec2-user@ip-10-0-0-219 ~]#
By default, the Docker daemon will reject requests from users that aren’t part of the docker group. If you encounter this message in your travels, you can either use root permission or add your user to the docker group.
Verify the docker group exists by searching for it in the groups file:
grep docker /etc/group or
getent group docker
[ec2-user@ip-10-0-0-219 ~]# getent group docker docker:x:993: [ec2-user@ip-10-0-0-219 ~]#
If you don’t see a line beginning with “docker:”, you will need to add the group yourself by entering:
sudo groupadd docker. However, in our case, we don’t need to.
Next, add your user (ec2-user) to docker group:
sudo gpasswd -a <dollar-sign>USER docker. The groups of the currently logged in user is cached, you can verify this by entering
You can login again to have your groups updated by entering:
newgrp docker. Note: It is convenient to not have to terminate your current ssh session by using newgrp, but terminating the ssh session and logging in again will work just as well.
[ec2-user@ip-10-0-0-219 ~]# sudo gpasswd -a <dollar-sign>USER docker Adding user ec2-user to group docker [ec2-user@ip-10-0-0-219 ~]# groups ec2-user adm wheel systemd-journal [ec2-user@ip-10-0-0-219 ~]# newgrp docker [ec2-user@ip-10-0-0-219 ~]# groups docker adm wheel systemd-journal ec2-user [ec2-user@ip-10-0-0-219 ~]#
Verify that your user can successfully issue Docker commands by entering:
Getting Docker Help from the Command Line
Here you will learn about how Docker commands are organized and how to get help for commands on the command line. If you are familiar with using the Linux command line or the git version control system, you will find many of the commands familiar in name and action. Even if you aren’t familiar, the commands are organized in an intuitive way that makes learning to use Docker commands as painless as practicable.
The commands are organized into common commands and a more exhaustive list grouped around the management of a specific component of Docker. You use each with a different syntax. For a common command, the usage is:
docker command-name [options]
and the usage for a management group command is:
docker management-group command-name [options]
Most of the common commands can be accessed in the short form or by specifying the management grouping they also fall under. Some commands are only available by specifying the management group. You will get the hang of this and review a couple of the main concepts in the following instructions.
The management commands and common commands are highlighted in the image below:
To see all of the commands grouped under system, enter:
docker system --help
[ec2-user@ip-10-0-0-93 ~]# docker system --help Usage: docker system COMMAND Manage Docker Commands: df Show docker disk usage events Get real time events from the server info Display system-wide information prune Remove unused data Run 'docker system COMMAND --help' for more information on a command. [ec2-user@ip-10-0-0-93 ~]#
Notice the info command previously entered.
events is a common command, but
prune are only available through the system management command. You can use
df to see the disk usage for Docker and prune to clean up unused data.
docker images: To view the commands grouped with images, enter:
docker images --help
Images are read-only snapshots that containers can be created from. Images are built up in layers, one image built on top of another. Because each layer is read-only, they can be identified with cryptographic hash values computed from the bytes of the data. Layers can be shared between images as another benefit of being read-only. You can, and will later, build your own images. The build command accomplishes that. When you build your own image, you will select a base image to build on top of with your custom application. A pull can be used to pull or download an image to your server from an image registry, while push can upload an image to a registry. Read through the other command descriptions so you are aware of what else is available.
docker containers: To view the commands grouped with containers, enter:
docker container --help
A container is another core concept in Docker. Containers run applications or services, almost always just one per container. Containers run on top of an image. In terms of storage, a container is like another layer on top of the image, but the layer is writable instead of read-only. You can have many containers using the same image. Each will use the same read-only image and have its own writable layer on top. Containers can be used to create images using the commit command, essentially converting the writable layer to a read-only layer in an image.
The relationship between images and containers aside, run is used to run a command on top of an image. A container can be stopped and started again. ls is used to list containers. It is aliased to ps and list as well. Read through the other commands in the list to see what else is available for working with containers.
Running your first Docker Container
As you learned in the previous Lab Step, containers run a command on top of an image. There are many available images to choose from. Images are collected in repositories. A repository can have many versions of an image. Tags are used to manage versions. You will use images from repositories inside the default Docker Registry, Docker Hub. Docker Hub hosts official images as well as community-contributed images. Docker Hub offers free and paid accounts. You get an unlimited amount of public repositories and one free private repository with the free account.
In this Lab Step, you will run your first container. You will learn how to search Docker Hub, pull images, and work with containers running on top of the images. You will get practice with many common commands when working with Docker.
Enter the following to see how easy it is to get a container running:
docker run hello-world
There are two sections to the output: output from Docker as it prepares to run the container and the output from the command running in the container.
Take a look at the Docker output first:
- In the first line, Docker is telling you that it couldn’t find the image you specified, hello-world, on the Docker Daemon’s local host. The latest portion after the colon (:) is a tag. The tag identifies which version of the image to use. By default, it looks for the latest version.
- In the next line, it notifies you that it automatically pulled the image. You could manually perform that task using the command docker pull hello-world. The library/hello-world is the repository it’s pulling from inside the Docker Hub registry. library is the account name for official Docker images. In general, images will come from repositories identified using the pattern account/repository.
- The last three lines confirm the pull completed and the image has been downloaded.
The container output is the output that gets written by the command that runs in the container. Read the output to learn more about how it was produced. You ask, but what was the command? You only specified the image name, hello-world. The image provides a default command in this case which prints the output you see.
Nginx: Try running a more complex container with some options specified:
docker run --name web-server -d -p 8080:80 nginx:1.12
This runs the nginx web server in a container using the official nginx image. You will see output similar to:
[ec2-user@ip-10-0-0-93 ~]# docker run --name web-server -d -p 8080:80 nginx:1.12 Unable to find image 'nginx:1.12' locally 1.12: Pulling from library/nginx f2aa67a397c4: Pull complete e3eaf3d87fe0: Pull complete 38cb13c1e4c9: Pull complete Digest: sha256:72daaf46f11cc753c4eab981cbf869919bd1fee3d2170a2adeac12400f494728 Status: Downloaded newer image for nginx:1.12 b743cc41052b983b2789569e1ac2b307a20c14197fdfa95dcdecb4c3e573a1ba [ec2-user@ip-10-0-0-93 ~]#
This time you specified the tag
1.12 indicating you want version
1.12 of nginx instead of the default latest version. There are three Pull complete messages this time, indicating the image has three layers. The last line is the id of the running container. The meanings of the command options are:
--name container_name: Label the container container_name. In the command above, the container is labeled web-server. This is much more manageable than the id.
-d: Detach the container by running it in the background and print its container id. Without this, the shell would be attached to the running container command and you wouldn’t have the shell returned to you to enter more commands.
-p host_port:container_port: Publish the container’s port number container_port to the host’s port number host_port. This connects the host’s port 8080 to the container port 80 (http) in the nginx command.
You again used the default command in the image, which runs the web server in this case.
[ec2-user@ip-10-0-0-93 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b743cc41052b nginx:1.12 "nginx -g 'daemon of…" 3 minutes ago Up 3 minutes 0.0.0.0:8080->80/tcp web-server [ec2-user@ip-10-0-0-93 ~]#
Stop nginix server: To stop the nginx server, enter:
docker stop web-server
[ec2-user@ip-10-0-0-93 ~]# docker stop web-server web-server [ec2-user@ip-10-0-0-93 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES [ec2-user@ip-10-0-0-93 ~]#
[ec2-user@ip-10-0-0-93 ~]# docker start web-server web-server [ec2-user@ip-10-0-0-93 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b743cc41052b nginx:1.12 "nginx -g 'daemon of…" 6 minutes ago Up 3 seconds 0.0.0.0:8080->80/tcp web-server [ec2-user@ip-10-0-0-93 ~]#
This is different from re-running the original docker run command, which would make a second container running the same command instead of using the stopped container.
ec2-user@ip-10-0-0-93 ~]# docker logs web-server 172.17.0.1 - - [23/Apr/2020:02:43:56 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.61.1" "-" [ec2-user@ip-10-0-0-93 ~]#
Docker logs messages written to standard output and standard error. In the case of nginx, it writes a line for each request that it receives.
run commands in a running container: You can run other commands in a running container. For example, to get a bash shell in the container enter:
docker exec -it web-server /bin/bash
This indicates you are at a shell prompt in the container using the root container user. The -it options tell Docker to handle your keyboard events in the container. Enter some commands to inspect the container environment, such as ls and
cat /etc/nginx/nginx.conf. When finished, enter exit to return to the VM ssh shell. Your shell prompt should change to confirm you are no longer in the container bash shell.
list files in the container’s
docker exec web-server ls /etc/nginx
[ec2-user@ip-10-0-0-93 ~]# docker exec web-server ls /etc/nginx conf.d fastcgi_params koi-utf koi-win mime.types modules nginx.conf scgi_params uwsgi_params win-utf [ec2-user@ip-10-0-0-93 ~]#
This runs the ls command and returns to the ssh shell prompt without using a container shell to execute the command. What commands are supported depends on the layers in the container’s image. Up until now you have used two images but don’t know how to find more.
Search for images:
[ec2-user@ip-10-0-0-93 ~]# docker search airflow NAME DESCRIPTION STARS OFFICIAL AUTOMATED apache/airflow Apache Airflow 103 astronomerio/airflow Airflow 24 [OK] bitnami/airflow Bitnami Docker Image for Apache Airflow 6 camil/airflow Airflow is a system to programmatically auth… 5 [OK] astronomerio/airflow-saas 4 bitnami/airflow-scheduler Bitnami Docker Image for Apache Airflow Sche… 4 bitnami/airflow-worker Bitnami Docker Image for Apache Airflow Work… 2 godatadriven/airflow Apache Airflow 2 [OK] pbweb/airflow-prometheus-exporter Export airflow metrics (collected from mysql… 2 airflowdocker/example-tasks 1 groupmeridian/airflow Apache Airflow build for Openshift 1 [OK] airflowdocker/aws-autoscaler 1 airflowci/incubator-airflow-ci A Docker container used for running Docker C… 1 [OK] datagovsg/airflow-pipeline An Airflow setup that aims to work well with… 1 [OK] yieldr/airflow Airflow 1 [OK] lowess/airflow-demo Airflow Docker image with Example DAGs 0 duanconsulting/airflow-etl 0 hmike96/airflow 0 guangie88/airflow-pipeline Airflow + Spark (k8s) + Hadoop + SQLAlchemy … 0 fahad0000/airflow-example Airflow example 0 [OK] terragontech/airflow Terragons version of the Airflow Image 0 thajib/airflow 0 x10232/airflow 0 flowminder/airflow FM airflow container 0 usagemeter/airflow Airflow used for vCloud Usage Insight 0 [ec2-user@ip-10-0-0-93 ~]#
Creating a Docker Image
Docker containers run on top of images. You have seen how to use images on Docker’s public registry, the Docker Hub. There are many different images available. It is worth trying to find existing images when you can. Inevitably, you will need to create your own images when making your own applications. In that case, you will still want to invest time into finding the right base layer to add your own layer(s) on top of.
There are a couple of ways to make images. You can use docker commit to create an image from a container’s changes. The changes may come from using exec to open a shell in the container like in the previous Lab Step. The other method is using a Dockerfile. A Dockerfile is easier to maintain, easier to repeatedly create images from, and distributions easier. You will create a Dockerfile in this Lab Step. Just know that it is possible to create equivalent images using commits.
Dockerfiles specify a sequence of instructions. Instructions can install software, expose network ports, set the default command for running a container using the image, and other tasks. Instructions can really handle anything required to configure the application. Many of the instructions add layers. It is usually a good idea to keep the number of layers to a reasonable number. There is overhead with each layer, and the total number of layers in an image is limited. When the Dockerfile is ready, you can create the image using the docker build command.
You will see how all of this comes together by creating an image of a Python Flask web app that randomly chooses what type of Cloud Academy content you should look at next. The choice of Python as the example app is arbitrary. You will not focus on the specifics of the programming language. You should be able to repeat the process for any other programming language by following a similar process. Whatever programming language or framework you are working with, you should consult the Docker Hub documentation for the image, as it will usually include advice on how to structure your Dockerfile.
sudo yum -y install git You will clone a code repository with the Flask app using Git.
[ec2-user@ip-10-0-0-192 ~]# git clone https://github.com/cloudacademy/flask-content-advisor.git Cloning into 'flask-content-advisor'... remote: Enumerating objects: 25, done. remote: Total 25 (delta 0), reused 0 (delta 0), pack-reused 25 Unpacking objects: 100% (25/25), done. [ec2-user@ip-10-0-0-192 ~]# pwd /home/ec2-user [ec2-user@ip-10-0-0-192 ~]# ls flask-content-advisor [ec2-user@ip-10-0-0-192 ~]# cd flask-content-advisor [ec2-user@ip-10-0-0-192 flask-content-advisor]# ls README.md requirements.txt src
Create a Dockerfile: From the same directory, create a
- FROM sets the base layer image
- COPY . . copies all of the files in the code repository into the container’s /usr/src/app directory
- RUN executes a command in a new layer at the top of the image
- EXPOSE only indicates what port the container will be listening on, it doesn’t automatically open the port on the container
- CMD sets the default command to run when a container is made from the image
There are more instruction types, but this lab will only focus on those mentioned. After completing the lab, you can review all of the instructions available in Dockerfiles at the Dockerfile reference web page.
Build the docker image:
docker build -t flask-content-advisor:latest . The
-t tells Docker to tag the image with the name
flask-content-advisor and tag latest. The
. at the end tells Docker to look for a Dockerfile in the current directory. Docker will report what it’s doing to build the image. Each instruction has its own step. Steps one and four take longer than the others. Step one needs to pull several layers for the Python 3 base layer image and Step four downloads code dependencies for the Flask web application framework. Notice that each Step ends with a notice that an intermediate container was removed. Because layers are read-only, Docker needs to create a container for each instruction. When the instruction is complete, Docker commits it to a layer in the image and discards the container.
[ec2-user@ip-10-0-0-192 flask-content-advisor]# docker build -t flask-content-advisor:latest . Sending build context to Docker daemon 90.11kB Step 1/6 : FROM python:3 3: Pulling from library/python 90fe46dd8199: Pull complete 35a4f1977689: Pull complete bbc37f14aded: Pull complete 74e27dc593d4: Pull complete 4352dcff7819: Pull complete deb569b08de6: Pull complete 98fd06fa8c53: Pull complete 7b9cc4fdefe6: Pull complete e8e1fd64f499: Pull complete Digest: sha256:a361f3d856a57ad373ab972a9ab87733610fea91ddd92d46b5784b42a30c80c9 Status: Downloaded newer image for python:3 ---> a6be143418fc Step 2/6 : WORKDIR /usr/src/app ---> Running in fbcdf6a5c9a8 Removing intermediate container fbcdf6a5c9a8 ---> a2c5203767e2 Step 3/6 : COPY . . ---> 682e99d5d203 Step 4/6 : RUN pip install --no-cache-dir -r requirements.txt ---> Running in 852510ad6c63 Collecting flask==0.12.2 Downloading Flask-0.12.2-py2.py3-none-any.whl (83 kB) Collecting itsdangerous>=0.21 Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB) Collecting Werkzeug>=0.7 Downloading Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB) Collecting click>=2.0 Downloading click-7.1.1-py2.py3-none-any.whl (82 kB) Collecting Jinja2>=2.4 Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB) Collecting MarkupSafe>=0.23 Downloading MarkupSafe-1.1.1-cp38-cp38-manylinux1_x86_64.whl (32 kB) Installing collected packages: itsdangerous, Werkzeug, click, MarkupSafe, Jinja2, flask Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 Werkzeug-1.0.1 click-7.1.1 flask-0.12.2 itsdangerous-1.1.0 Removing intermediate container 852510ad6c63 ---> 4484fd6a45d1 Step 5/6 : EXPOSE 5000 ---> Running in 65cb672c6235 Removing intermediate container 65cb672c6235 ---> 5e675ad2b813 Step 6/6 : CMD [ "python", "./src/app.py" ] ---> Running in 13abb5b30695 Removing intermediate container 13abb5b30695 ---> 74be63a85da6 Successfully built 74be63a85da6 Successfully tagged flask-content-advisor:latest [ec2-user@ip-10-0-0-192 flask-content-advisor]#
Record your VM’s public IP address:
curl ipecho.net/plain; echo .You will need your IP to test that the web app is available with your browser. The echo is only to put the shell prompt on a new line.
[ec2-user@ip-10-0-0-192 flask-content-advisor]# curl ipecho.net/plain; echo 188.8.131.52
Run the container: Now you can run a container using the image you just built:
docker run --name advisor -p 80:5000 flask-content-advisor
ec2-user@ip-10-0-0-192 flask-content-advisor]# docker run --name advisor -p 80:5000 flask-content-advisor * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit) * Restarting with stat * Debugger is active! * Debugger PIN: 204-454-399
This runs a container named advisor and maps the container’s port 5000 to the host’s port 80 (http). This time you didn’t include -d to run in detached mode. That is why you see output and you don’t have the shell prompt returned to you. If you did run with -d, you could get the same information from docker logs.