Kubernetes Init Containers

Need for Init Containers

Sometimes you need to perform some tasks or check some prerequisites before a main application container starts. Some examples include:

  • waiting for a service to be created,
  • downloading files the application depends on,
  • or dynamically deciding which port the application should use.

The code that performs those tasks could be crammed into the main application image but it is better to keep a clean separation between the main application and supporting functionality to keep the smallest footprint you can for the images. However, the tasks are closely linked to the main application (same pod) and are required to run before the main application starts. Kubernetes provides init containers as a way to run these tasks that are required to complete before the main container starts.

Pods may declare any number of init containers. They run in a sequence in the order they are declared. Each init container must run to completion before the following init container begins. Once all of the init containers have completed the main containers in the pod can start. Init containers use different images from the containers in a pod.

Benefits of Init Containers

This provides some benefits:

  • They can contain utilities that are not desirable to include in the actual application image for security reasons.
  • They can also contain utilities or custom code for setup that is not present in the application image. For example there is no need to include utilities like Sed Awk or dig in an application image if they are only used for setup.
  • Init containers also provide an easy way to block or delay the startup of an application until some preconditions are met. They are similar to readiness probes in this sense but only run at pod startup and can perform other useful work.

All these features together make init containers a vital part of the Kubernetes tool box.

There is one important thing to understand about init containers. They run every time a pod is created. This means they will run once for every replica in a deployment. If a pod restarts, say due to a failed liveness probe, the init containers would run again as part of the restart. Thus you have to assume that init containers run at least once. This usually means init containers should be idempotent; meaning running it more than once has no additional effect.

Demo

To start off, lets start by creating the namespace probes, we will be making use of the manifests from the probes post as our starting point.

Create namespace, data-tier and app-tier

ubuntu@ip-10-0-128-5:~/src# kubectl create -f 7.1-namespace.yaml
namespace/probes created
ubuntu@ip-10-0-128-5:~/src# kubectl create -f 7.2-data_tier.yaml -n probes
service/data-tier created
deployment.apps/data-tier created
ubuntu@ip-10-0-128-5:~/src# kubectl create -f 7.3-app_tier.yaml -n probes
service/app-tier created
deployment.apps/app-tier created
ubuntu@ip-10-0-128-5:~/src# kubectl get -n probes deployments.
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
app-tier    1/1     1            1           2m43s
data-tier   1/1     1            1           2m54s
ubuntu@ip-10-0-128-5:~/src#

Declare the init container in the app-tier manifest

Let’s add an init container to our app-tier that will wait for Redis before starting any application servers.

apiVersion: v1
kind: Service
metadata:
  name: app-tier
  labels:
    app: microservices
spec:
  ports:
  - port: 8080
  selector:
    tier: app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-tier
  labels:
    app: microservices
    tier: app
spec:
  replicas: 1
  selector:
    matchLabels:
      tier: app
  template:
    metadata:
      labels:
        app: microservices
        tier: app
    spec:
      containers:
      - name: server
        image: lrakai/microservices:server-v1
        ports:
          - containerPort: 8080
            name: server
        env:
          - name: REDIS_URL
            # Environment variable service discovery
            # Naming pattern:
            #   IP address: <all_caps_service_name>_SERVICE_HOST
            #   Port: <all_caps_service_name>_SERVICE_PORT
            #   Named Port: <all_caps_service_name>_SERVICE_PORT_<all_caps_port_name>
            value: redis://$(DATA_TIER_SERVICE_HOST):$(DATA_TIER_SERVICE_PORT_REDIS)
            # In multi-container example value was
            # value: redis://localhost:6379
          - name: DEBUG
            value: express:*
        livenessProbe:
          httpGet:
            path: /probe/liveness
            port: server
          initialDelaySeconds: 5
        readinessProbe:
          httpGet:
            path: /probe/readiness
            port: server
          initialDelaySeconds: 3
      initContainers:
        - name: await-redis
          image: lrakai/microservices:server-v1
          env:
          - name: REDIS_URL
            value: redis://$(DATA_TIER_SERVICE_HOST):$(DATA_TIER_SERVICE_PORT_REDIS)
          command:
            - npm
            - run-script
            - await-redis

We see that initContainers have the same fields as regular containers in a pod spec. The one exception is initContainers do not support readiness probes because they must run to completion before the state of the pod can be considered ready. You will receive an error if you try to include a readiness probe in an initContainer.

You can see that the fields are the same as what we have seen with regular containers. I’ve used the same image as the main application for simplicity and it has everything we need in it already. The command field is used to override the image’s default entrypoint command. For this init container we want to run a script that waits for a successful connection with redis. The script is already included in the image and is executed with the NPM run-script await-redis command. This command will block until a connection is established with the configured redis url provided as an environment variable.

Apply the changes to existing deployments

Now let’s apply the changes to the existing deployment. After that, describe the deployment’s pod

ubuntu@ip-10-0-128-5:~/src# kubectl apply -f 8.1-app_tier.yaml -n probes
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
service/app-tier configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
deployment.apps/app-tier configured
ubuntu@ip-10-0-128-5:~/src# kubectl describe pod -n probes app-tier-6bf4d544c-kt9vd
Name:           app-tier-6bf4d544c-kt9vd
Namespace:      probes
Priority:       0
Node:           ip-10-0-18-215.us-west-2.compute.internal/10.0.18.215
Start Time:     Tue, 05 May 2020 18:26:40 +0000
Labels:         app=microservices
                pod-template-hash=6bf4d544c
                tier=app
Annotations:    <none>
Status:         Running
IP:             192.168.70.194
Controlled By:  ReplicaSet/app-tier-6bf4d544c
Init Containers:
  await-redis:
    Container ID:  docker://70833194504bd680bef2f272ebfa1b85ec4a9f0342ce29ab5a5f5cb737dc3c61
    Image:         lrakai/microservices:server-v1
    Image ID:      docker-pullable://lrakai/microservices@sha256:9e3e3c45bb9d950fe7a38ce5e4e63ace2b6ca9ba8e09240f138c5df39d7b7587
    Port:          <none>
    Host Port:     <none>
    Command:
      npm
      run-script
      await-redis
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 05 May 2020 18:26:46 +0000
      Finished:     Tue, 05 May 2020 18:26:46 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      REDIS_URL:  redis://$(DATA_TIER_SERVICE_HOST):$(DATA_TIER_SERVICE_PORT_REDIS)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-66l8q (ro)
Containers:
  server:
    Container ID:   docker://b9fe1afb2de7b4e0d7a29270fdb373c72a296c4f9e95dad293bd861459720fab
    Image:          lrakai/microservices:server-v1
    Image ID:       docker-pullable://lrakai/microservices@sha256:9e3e3c45bb9d950fe7a38ce5e4e63ace2b6ca9ba8e09240f138c5df39d7b7587
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 05 May 2020 18:26:48 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:server/probe/liveness delay=5s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:server/probe/readiness delay=3s timeout=1s period=10s #success=1 #failure=3
    Environment:
      REDIS_URL:  redis://$(DATA_TIER_SERVICE_HOST):$(DATA_TIER_SERVICE_PORT_REDIS)
      DEBUG:      express:*
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-66l8q (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-66l8q:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-66l8q
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                                                Message
  ----    ------     ----  ----                                                -------
  Normal  Scheduled  36s   default-scheduler                                   Successfully assigned probes/app-tier-6bf4d544c-kt9vd to ip-10-0-18-215.us-west-2.compute.internal
  Normal  Pulling    35s   kubelet, ip-10-0-18-215.us-west-2.compute.internal  Pulling image "lrakai/microservices:server-v1"
  Normal  Pulled     31s   kubelet, ip-10-0-18-215.us-west-2.compute.internal  Successfully pulled image "lrakai/microservices:server-v1"
  Normal  Created    31s   kubelet, ip-10-0-18-215.us-west-2.compute.internal  Created container await-redis
  Normal  Started    30s   kubelet, ip-10-0-18-215.us-west-2.compute.internal  Started container await-redis
  Normal  Pulled     29s   kubelet, ip-10-0-18-215.us-west-2.compute.internal  Container image "lrakai/microservices:server-v1" already present on machine
  Normal  Created    29s   kubelet, ip-10-0-18-215.us-west-2.compute.internal  Created container server
  Normal  Started    28s   kubelet, ip-10-0-18-215.us-west-2.compute.internal  Started container server
ubuntu@ip-10-0-128-5:~/src#

And observe the event log now shows the entire lifecycle with init containers. The await-redis init container runs to completion before the server container is created. You can also view the logs of init containers using the usual logs command and specifying the name of the init container as the last argument after the pod name to retrieve logs for a given init container. This is specifically important when debugging an init container failure which prevents the main containers from ever being created.

View Init Container logs to debug pod startup issues

ubuntu@ip-10-0-128-5:~/src# kubectl logs -n probes app-tier-6bf4d544c-kt9vd await-redis
npm info it worked if it ends with ok
npm info using npm@3.10.10
npm info using node@v6.11.0
npm info lifecycle server@1.0.0~preawait-redis: server@1.0.0
npm info lifecycle server@1.0.0~await-redis: server@1.0.0

> server@1.0.0 await-redis /usr/src/app
> node await.js

Connection ok
npm info lifecycle server@1.0.0~postawait-redis: server@1.0.0
npm info ok
ubuntu@ip-10-0-128-5:~/src#

Conclusion

This concludes our tour of init containers. They give you another mechanism for controlling the lifecycle of pods. You can use them to perform some tasks before the main containers have an opportunity to start. This can be useful for checking preconditions such checking that the depended upon services are created or preparing the depended upon files. The files use case requires knowledge of another Kubernetes concept to pull of, namely: volumes which can be used to share files between containers. We will see this next.