A practical guide to understanding Kubernetes pods

What are pods?

Pods are the basic building block in kubernetes. Pods contain one or more containers. We’ll stick with one container per pod in this lesson and talk about multi-container pods later. All pods share a container network that allows any pod to communicate with any other pod, regardless of the nodes the pods are running on.

Each pod gets a single IP address in the container network.

Kubernetes does all the heavy-lifting to make that happen. You get to work with the simple abstraction of all pods can communicate with each other and each pod has one IP address.

pods

Because pods include containers, the declaration of a pod includes all of the properties that you would expect for running containers, for example with docker run, you need to specify the Container image, any ports you want to publish to allow access into the container, you can choose the restart policy to determine if a pod should automatically restart when its container fails, you can specify limits on the CPU and Memory consumption. There are many more properties that are specific to pods and kubernetes, we will encounter them in the subsequent posts.

pods1

Manifest file

All of the desired properties are written to a manifest file.

Manifest files are used to describe all kinds of resources in Kubernetes, not only pods.

Based on the kind of resource the manifest describes, you will configure different properties. The configuration specific to each kind of resource is referred to as its specification or spec.

pods3

The spec contains resource-specific properties.

The manifests are sent to the Kubernetes API server where the necessary actions are taken to realize what is described in the manifest. You will use kubectl to send the manifest to the API server. One way of doing this is with the kubectl create command.

pods2

For pod manifests, the API server will take the following actions:

  • selects a node with available resources for all of the pod’s containers,
  • schedules the pod to that node,
  • the node then downloads the pod’s container images And runs the containers.

There are more steps involved but that is enough to get the idea.

pods4

Why not use kubectl subcommands instead of manifest file?

As mentioned before that kubectl also provides subcommands to directly create resources without using manifests. It’s usually a good idea to stick with manifests for several reasons:

  • You can check in your manifests in to source control systems to track their history and roll back when needed
  • It makes it easy to share your work so it can be created in other clusters
  • It’s also easier to work with compared to stringing together sequences of commands with many options to achieve the same result

We will stick with manifests instead of getting into sub-commands.

Demo: Deploy Nginx webserver running in a Kubernetes Pod

Now we’re ready to see all of this in action using kubectl and a Kubernetes cluster. Our goal will be to deploy an Nginx web server running in a Kubernetes pod.

Login to the bastion host and run the following kubectl commands. Quick reminder of the environment

k15

ubuntu@ip-10-0-128-5:~# kubectl get nodes
NAME                                        STATUS   ROLES    AGE   VERSION
ip-10-0-0-100.us-west-2.compute.internal    Ready    master   82m   v1.15.0
ip-10-0-0-203.us-west-2.compute.internal    Ready    <none>   80m   v1.15.0
ip-10-0-21-114.us-west-2.compute.internal   Ready    <none>   80m   v1.15.0

The output shows that our Kubernetes cluster is comprised of one master and two worker nodes.

ubuntu@ip-10-0-128-5:~# kubectl get pods
No resources found.
ubuntu@ip-10-0-128-5:~#

The output tells us that no pod resources were found in the default namespace in the cluster. If it wasn’t able to connect to the API server, you would have seen an error message, so everything looks good.

Let’s start with a minimal example of a Pod manifest to get a taste for manifests. Shown here is a simple manifest file that we will use to create our first pod.

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mycontainer
    image: nginx:latest

This manifest declares a pod with one container that uses the nginx:latest image.

All manifests have the same top-level keys: apiVersion, kind, and metadata followed by the spec.

  • apiVersion: Kubernetes supports multiple api versions. V1 is the core api version containing many of the most common resources such as pods and nodes.
  • kind: kind indicates what the resource is.
  • metadata: Metadata includes information relevant to the resource and can help identify resources. The minimum amount of metadata is a name which is set to mypod. Names must be unique within a Kubernetes namespace.
  • spec: spec is the specification for the declared kind and must match what is expected by the defined api version, for example the spec can change between the beta and the generally available API version of a resource. The spec is essentially where all of the meat goes. You can refer to the official api docs for complete info on all versions and supported fields. I’ll explain the ones that we need for this demo but know that there are more left to discover. The pod spec defines the containers in the pod. The minimum required field is a single container which must declare its image and name. This pod only has a single container but the YAML is a list allowing you to specify more than one.

So, create the pod using the 1.1-basic_pod.yaml as our manifest file. Run: kubectl create -f 1.1-basic_pod.yaml The -f option tells the create command you are using a manifest to declare the resources you want created.

ubuntu@ip-10-0-128-5:~/src# kubectl create -f 1.1-basic_pod.yaml
pod/mypod created
ubuntu@ip-10-0-128-5:~/src#

List the pods kubectl get pods, kubectl shows the name, number of ready containers, the pod’s state, restarts and also the age of all the pod in the cluster. Note that this pod only has one container.

ubuntu@ip-10-0-128-5:~/src# kubectl get pods
NAME    READY   STATUS    RESTARTS   AGE
mypod   1/1     Running   0          3m46s
ubuntu@ip-10-0-128-5:~/src#

To see more information about this pod, use the kubectl describe pod mypod command. Describe takes a resource kind, just like get, and to narrow in on a specific resource of that kind, we add the name, which you can also do with get.

ubuntu@ip-10-0-128-5:~/src# kubectl describe pod mypod
Name:         mypod
Namespace:    default
Priority:     0
Node:         ip-10-0-21-114.us-west-2.compute.internal/10.0.21.114
Start Time:   Sat, 25 Apr 2020 16:59:03 +0000
Labels:       <none>
Annotations:  <none>
Status:       Running
IP:           192.168.41.65
Containers:
  mycontainer:
    Container ID:   docker://e6a649e94e24e14bb39bd866addd1e1deecbc550978c42bda58a8d7823c370a9
    Image:          nginx:latest
    Image ID:       docker-pullable://nginx@sha256:86ae264c3f4acb99b2dee4d0098c40cb8c46dcf9e1148f05d3a51c4df6758c12
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Sat, 25 Apr 2020 16:59:10 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-4729q (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-4729q:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-4729q
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                                                Message
  ----    ------     ----  ----                                                -------
  Normal  Scheduled  20m   default-scheduler                                   Successfully assigned default/mypod to ip-10-0-21-114.us-west-2.compute.internal
  Normal  Pulling    20m   kubelet, ip-10-0-21-114.us-west-2.compute.internal  Pulling image "nginx:latest"
  Normal  Pulled     20m   kubelet, ip-10-0-21-114.us-west-2.compute.internal  Successfully pulled image "nginx:latest"
  Normal  Created    20m   kubelet, ip-10-0-21-114.us-west-2.compute.internal  Created container mycontainer
  Normal  Started    20m   kubelet, ip-10-0-21-114.us-west-2.compute.internal  Started container mycontainer
ubuntu@ip-10-0-128-5:~/src#

As you can see there is a lot more information than what get provided. The name, namespace, and the node running the pod are given at the top along with other metadata. Also note the Pod is assigned an IP. No matter how many containers we included, there would be only one IP. In the containers section we can see the image and whether or not the container is ready. You can also see the port and container port are both none. Ports are part of the container spec but Kubernetes assigned default values for us. Just like with docker you need to tell kubernetes which port to publish if you want it to be accessible. We’ll have to go back and declare a port after this otherwise nothing is going to reach the web server.

At the bottom is the Events section. It lists the most recent events related to the resource. You can see the steps Kubernetes took to start the pod from scheduling, pulling the container image to starting the container. The events section is shared by most kinds of resources when you use describe and is very helpful for debugging.

Publish the port

Let’s tell Kubernetes which port to publish to allow access to the webserver. In the below manifest file we can see the ports mapping is added and the containerPort field is set to 80 for HTTP. Kubernetes will use TCP as the protocol by default and will assign an available host port automatically so we don’t need to declare anything more.

pods5

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mycontainer
    image: nginx:latest
    ports:
      - containerPort: 80

Kubernetes can apply certain changes to different kinds of resources on the fly. Unfortunately Kubernetes can not update ports on a running pod. So we need to delete the pod and recreate it.

ubuntu@ip-10-0-128-5:~/src# kubectl delete pod mypod
pod "mypod" deleted
ubuntu@ip-10-0-128-5:~/src#

You can also specify the -f with the manifest file and kubernetes would delete all the resources declared in that file. It is a bit clunky to have to delete and then recreate whenever a pod spec changes, but just bare with me for now. We’ll see a seamless way to manage such changes in later posts. Now we can create the new pod.

ubuntu@ip-10-0-128-5:~/src# kubectl describe pod mypod | more
Name:         mypod
Namespace:    default
Priority:     0
Node:         ip-10-0-0-203.us-west-2.compute.internal/10.0.0.203
Start Time:   Sat, 25 Apr 2020 17:36:26 +0000
Labels:       <none>
Annotations:  <none>
Status:       Running
IP:           192.168.31.129
Containers:
  mycontainer:
    Container ID:   docker://3d0d8cd2fac946b3c66c78a64baff7360c11db6586d8398c90d62de05837914e
    Image:          nginx:latest
    Image ID:       docker-pullable://nginx@sha256:86ae264c3f4acb99b2dee4d0098c40cb8c46dcf9e1148f05d3a51c4df6758c12
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sat, 25 Apr 2020 17:36:38 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-4729q (ro)

We can see port 80 is given as the port. So now you may think to try to send a request to port 80 on that noted IP 192.168.31.129. But it still doesn’t work. Why do you think that is? Well, the pod’s Ip is on the container network.

The bastion-host instance is not part of the container network so it won’t work. If however, you sent the request from a container in a Kubernetes pod, the request would succeed since pods can communicate with all other pods by default. We’ll see how we can access the web server from the bastion-host instance in the next post where we cover Services.

Labels

In the describe section you might have seen the labels field was set to none. kubectl describe pod mypod | more

Labels are key-value pairs that identify resource attributes, for example the application tier whether it is front-end or backend, or a region such as us-east or us-west. In addition to providing meaningful identifying information, labels are used to make selections in Kubernetes. For example, you could tell kubectl to get only resources in the us-west region.

apiVersion: v1
kind: Pod
metadata:
  name: mypod
  labels:
    app: webserver
spec:
  containers:
  - name: mycontainer
    image: nginx:latest
    ports:
      - containerPort: 80

In the above manifest a label is added to identify the type of app the pod is part of. We’re using nginx as a web server so the label value is webserver. You can have multiple labels but one is enough for this example.

Quality of Service Classes

The last point I want to make in this post relates to how kubernetes can schedule pods based on their resource requests. The pods that we have seen so far didn’t set any resource request. That makes it easier to schedule them because the scheduler doesn’t need to find nodes with the requested amounts of resources. It will just schedule them onto any node that isn’t under pressure or starved for resources. However, these pods will be the first to be evicted if a node becomes under pressure and needs to free up resources.

That’s called BestEffort quality of service which was displayed in the describe output. BestEffort pods can also create resource contention with other pods on the same node and usually it is a good idea to set resource requests.

apiVersion: v1
kind: Pod
metadata:
  name: mypod
  labels:
    app: webserver
spec:
  containers:
  - name: mycontainer
    image: nginx:latest
    resources:
      requests:
        memory: "128Mi" # 128Mi = 128 mebibytes
        cpu: "500m"     # 500m = 500 milliCPUs (1/2 CPU)
      limits:
        memory: "128Mi"
        cpu: "500m"
    ports:
      - containerPort: 80

In the above manifest yaml file I have set a resource request and limit for the pod’s container. Request sets the minimum required resources to schedule the pod onto a node and the limit is the maximum amount of resources you want the node to ever give the pod. You can set resource requests and limits for each container. There is also support for requesting amounts local disk by using the ephemeral-storage.

When we create this pod using kubectl create -f manifest.yaml

The pod will be guaranteed the resources you requested or it won’t be scheduled until those resources are available.

Describe the pod to see the QoS

ubuntu@ip-10-0-128-5:~/src# kubectl describe pod mypod | more
Name:         mypod
Namespace:    default
Priority:     0
Node:         ip-10-0-21-114.us-west-2.compute.internal/10.0.21.114
Start Time:   Sat, 25 Apr 2020 17:47:00 +0000
Labels:       app=webserver
Annotations:  <none>
Status:       Running
IP:           192.168.41.66
Containers:
  mycontainer:
    Container ID:   docker://d227c121d3dc880953fbe3e9a570327fd8a0b17d8aaa1cfd95105ddba913a0d7
    Image:          nginx:latest
    Image ID:       docker-pullable://nginx@sha256:86ae264c3f4acb99b2dee4d0098c40cb8c46dcf9e1148f05d3a51c4df6758c12
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sat, 25 Apr 2020 17:47:04 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:        500m
      memory:     128Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-4729q (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-4729q:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-4729q
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:

In the describe output you can see this is in the guaranteed quality of service. You need to do some benchmarking to configure a reasonable request and limit but the effort is well worth it to ensure your pods have the resources they need and to best utilize the resources in the cluster, which is one of the reasons you are using containers in the first place. For the rest of this course we will use best effort pods since we won’t have specific resource requirements in mind. This is not something you should do in production environments.

Summary

We covered a lot in this post, so let’s review what we covered.

  • Pods are the basic building block in Kubernetes and contain one or more containers.
  • You declare pods and other resources in manifest files. All manifests share an api version, kind and metadata.
  • Metadata must include a name but labels are usually a good idea to help organize resources. Manifests also include a spec to configure the unique parts of each resource kind.
  • Pod specs include the list of containers which must specify a container name and image. It is often useful to set resource requests and limits. We will see more fields of pod specs in later posts.

In the next post I will make the web server running in the pod accessible from the bastion host VM through something called Services.