Pause, Resume And Scale Kubernetes Deployments - Alibaba Cloud

Home » How To Stop All Kubernetes Deployments » Pause, Resume And Scale Kubernetes Deployments - Alibaba Cloud

Maybe your like

By Alwyn Botha, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud's incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.

Pausing, resuming and scaling Deployments are very easy to understand.

We have several commands to show the status of these 3 actions. Unfortunately we will be using the same commands repeatedly so that you can see over time which commands work best for which specific need.

This tutorial consists of 1-2 sentences of descriptions followed by a command to run. Then follows one short snippet of comments and one more command and so on. Each of those steps must be done for the tutorial to work.

Kubectl Rollout Pause

Kubernetes enable you to pause a Deployment. You can then make adjustments to the Deployment and resume it.

Deployments do not need to be paused to make a change. Use pause to pause a Deployment so that you can calmly make several changes ( that are kept in a queue till resume is ordered ).

We start with one normal, working Deployment :

nano myDeployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: busybox-deployment labels: app: busybox spec: replicas: 10 strategy: type: RollingUpdate selector: matchLabels: app: busybox template: metadata: labels: app: busybox spec: containers: - name: busybox image: busybox imagePullPolicy: IfNotPresent command: ['sh', '-c', 'echo Container is Running ; sleep 3600'] terminationGracePeriodSeconds: 0

Create the Deployment

kubectl create -f myDeployment.yaml deployment.apps/mydeployment created

Just 2 seconds later the Deployment is complete. ( busybox image already on server, starting such a simple container is very fast - 10 replicas started up simultaneously ... adding to the fast speed. )

kubectl get deploy NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox-deployment 10 10 10 0 2s

Check the rollout status:

kubectl rollout status deployment.v1.apps/busybox-deployment deployment "busybox-deployment" successfully rolled out

Describe the Deployment :

kubectl describe deployment.extensions/busybox-deployment Name: busybox-deployment CreationTimestamp: Sun, 20 Jan 2019 10:16:11 +0200 Labels: app=busybox Annotations: deployment.kubernetes.io/revision: 1 Selector: app=busybox Replicas: 10 desired | 10 updated | 10 total | 10 available | 0 unavailable StrategyType: RollingUpdate RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app=busybox Containers: busybox: Image: busybox Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets: <none> NewReplicaSet: busybox-deployment-5d6889b44 (10/10 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 26s deployment-controller Scaled up replica set busybox-deployment-5d6889b44 to 10

The previous 3 commands showed a perfectly running 10 Pods Deployment.

Now we pause this Deployment and observe its status.

kubectl rollout pause deployment busybox-deployment deployment.extensions/busybox-deployment paused

get deploy does not actually show this Deployment as paused.

kubectl get deploy NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox-deployment 10 10 10 10 2m12s

kubectl rollout status does not actually show this Deployment as paused.

kubectl rollout status deployment.v1.apps/busybox-deployment deployment "busybox-deployment" successfully rolled out

This is a disappointment ... rollout status deployment does NOT show Deployment as being paused.

It shows ROLLOUT status, not DEPLOYMENT status.

Describe deployment :

kubectl describe deployment.extensions/busybox-deployment Name: busybox-deployment CreationTimestamp: Sun, 20 Jan 2019 10:16:11 +0200 Labels: app=busybox Annotations: deployment.kubernetes.io/revision: 1 Selector: app=busybox Replicas: 10 desired | 10 updated | 10 total | 10 available | 0 unavailable StrategyType: RollingUpdate RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app=busybox Containers: busybox: Image: busybox Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing Unknown DeploymentPaused OldReplicaSets: <none> NewReplicaSet: busybox-deployment-5d6889b44 (10/10 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 2m28s deployment-controller Scaled up replica set busybox-deployment-5d6889b44 to 10

ONLY one line shows that this Deployment is now paused:

Progressing Unknown DeploymentPaused

IMPORTANT: ONLY use kubectl describe to reliably find paused Deployments.

Make Changes to Paused Deployment

While a Deployment is paused we can make changes to its Pods.

Change all our replicated Pods to use busybox:1.30-glibc image.

--record the fact that we are using a new image in the Deployment annotations.

kubectl set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-glibc --record deployment.apps/busybox-deployment image updated

This specific change is not applied immediately ( Deployment is currently paused ).

While a Deployment is paused we can make changes to its number of replicas.

kubectl scale deployment.v1.apps/busybox-deployment --replicas=5 deployment.apps/busybox-deployment scaled

Changes to number of replicas are implemented immediately.

kubectl get deploy NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox-deployment 5 5 0 5 3m44s

Important: note the up-to-date column ... none of the available / running Pods are up to date. They are all using the busybox image they used at the exact pause command time.

Show list of Pods - they still run old busybox image.

kubectl get pods --show-labels NAME READY STATUS RESTARTS AGE LABELS busybox-deployment-5d6889b44-78tq8 1/1 Running 0 4m59s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-7xwds 1/1 Running 0 4m59s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-gjgc5 1/1 Running 0 4m59s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-j47xp 1/1 Running 0 4m59s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-kbh7t 1/1 Running 0 4m59s app=busybox,pod-template-hash=5d6889b44

Kubectl Rollout Resume

We need to resume our rollout so that a new set of Pods can be created using the image we requested : busybox:1.30-glibc

kubectl rollout resume deployment busybox-deployment deployment.extensions/busybox-deployment resumed

3 seconds later: 3 new Pods at bottom are being created.

kubectl get pods --show-labels NAME READY STATUS RESTARTS AGE LABELS busybox-deployment-5d6889b44-78tq8 1/1 Running 0 5m52s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-7xwds 1/1 Running 0 5m52s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-gjgc5 1/1 Running 0 5m52s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-kbh7t 1/1 Running 0 5m52s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-f54b89b4-8xpv2 0/1 ContainerCreating 0 3s app=busybox,pod-template-hash=f54b89b4 busybox-deployment-f54b89b4-lgbxv 0/1 ContainerCreating 0 3s app=busybox,pod-template-hash=f54b89b4 busybox-deployment-f54b89b4-x64m5 0/1 ContainerCreating 0 3s app=busybox,pod-template-hash=f54b89b4

another 10 seconds later ...

3 Pods in middle of list are already new.

Pod at the top of list is old. Very easy to determine old versus new: old Pods have older age.

( Freshly created new Pods have young age )

kubectl get pods --show-labels NAME READY STATUS RESTARTS AGE LABELS busybox-deployment-5d6889b44-7xwds 1/1 Running 0 6m5s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-f54b89b4-8xpv2 0/1 ContainerCreating 0 16s app=busybox,pod-template-hash=f54b89b4 busybox-deployment-f54b89b4-cbxt4 1/1 Running 0 3s app=busybox,pod-template-hash=f54b89b4 busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 16s app=busybox,pod-template-hash=f54b89b4 busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 4s app=busybox,pod-template-hash=f54b89b4 busybox-deployment-f54b89b4-x64m5 0/1 ContainerCreating 0 16s app=busybox,pod-template-hash=f54b89b4

another 10 seconds later ...

All Pods are new ... short running age is the easy giveaway hint.

kubectl get pods --show-labels NAME READY STATUS RESTARTS AGE LABELS busybox-deployment-f54b89b4-8xpv2 1/1 Running 0 26s app=busybox,pod-template-hash=f54b89b4 busybox-deployment-f54b89b4-cbxt4 1/1 Running 0 13s app=busybox,pod-template-hash=f54b89b4 busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 26s app=busybox,pod-template-hash=f54b89b4 busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 14s app=busybox,pod-template-hash=f54b89b4 busybox-deployment-f54b89b4-x64m5 1/1 Running 0 26s app=busybox,pod-template-hash=f54b89b4

Display Deployment status ... all 5 Pods up to date.

kubectl get deploy NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox-deployment 5 5 5 5 7m4s

Note the last column. Overall age of this Deployment is 7 minutes. Only last 30 seconds used the new busybox image.

Show the rollout history of the Deployment :

kubectl rollout history deployment.v1.apps/busybox-deployment deployment.apps/busybox-deployment REVISION CHANGE-CAUSE 1 <none> 2 kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-glibc --record=true

We see our new busybox image listed as revision 2.

Scaling Paused Deployments

Deployments can be scaled while they are running.

Scaling in this context means changing the number of running identical Pod replicas.

Below is a demo that you can freely scale a paused Deployment :

Current state for reference:

kubectl get deploy NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox-deployment 5 5 5 5 27m

We have 5 running Pods.

kubectl get pods NAME READY STATUS RESTARTS AGE busybox-deployment-f54b89b4-8xpv2 1/1 Running 0 25m busybox-deployment-f54b89b4-cbxt4 1/1 Running 0 25m busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 25m busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 25m busybox-deployment-f54b89b4-x64m5 1/1 Running 0 25m

Pause Deployment :

kubectl rollout pause deployment busybox-deployment deployment.extensions/busybox-deployment paused

Get list of Pods:

IMPORTANT: Pausing a Deployment DOES NOT pause its containers.

All Pods still running.

kubectl get pods NAME READY STATUS RESTARTS AGE busybox-deployment-f54b89b4-8xpv2 1/1 Running 0 25m busybox-deployment-f54b89b4-cbxt4 1/1 Running 0 25m busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 25m busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 25m busybox-deployment-f54b89b4-x64m5 1/1 Running 0 25m

Change replicas to 2:

kubectl scale deployment.v1.apps/busybox-deployment --replicas=2 deployment.apps/busybox-deployment scaled

Investigate change ... only 2 Pods now running:

kubectl get pods NAME READY STATUS RESTARTS AGE busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 26m busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 25m

Change replicas to 6:

kubectl scale deployment.v1.apps/busybox-deployment --replicas=6 deployment.apps/busybox-deployment scaled

Investigate change ... 6 Pods now running:

Four new Pods have an age of only 4 seconds.

kubectl get pods NAME READY STATUS RESTARTS AGE busybox-deployment-f54b89b4-4fw4b 1/1 Running 0 4s busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 26m busybox-deployment-f54b89b4-n952j 1/1 Running 0 4s busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 26m busybox-deployment-f54b89b4-t569g 1/1 Running 0 4s busybox-deployment-f54b89b4-xlwr8 1/1 Running 0 4s

Let's change the Pods to all use another different busybox image: busybox:1.30-uclibc

Important: right now the Deployment is paused. It only takes note of the newly desired image. It does not roll out the change immediately - it is waiting for the kubectl rollout resume command.

kubectl set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-uclibc --record deployment.apps/busybox-deployment image updated

Show list of Pods. They are unaware a change is coming. Still running as before.

kubectl get pods NAME READY STATUS RESTARTS AGE busybox-deployment-f54b89b4-4fw4b 1/1 Running 0 40s busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 26m busybox-deployment-f54b89b4-n952j 1/1 Running 0 40s busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 26m busybox-deployment-f54b89b4-t569g 1/1 Running 0 40s busybox-deployment-f54b89b4-xlwr8 1/1 Running 0 40s

The Deployment status shows ZERO up to date Pods. It is aware a change is pending while Deployment is in a paused state.

kubectl get deploy NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox-deployment 6 6 0 6 32m

Rollout history does not show new busybox:1.30-uclibc image pending.

It only shows history of previous and running Deployments status.

kubectl rollout history deployment.v1.apps/busybox-deployment deployment.apps/busybox-deployment REVISION CHANGE-CAUSE 1 <none> 2 kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-glibc --record=true

Let's resume the Deployment :

kubectl rollout resume deployment busybox-deployment deployment.extensions/busybox-deployment resumed

5 seconds later ... 3 new Pods being created.

kubectl get pods NAME READY STATUS RESTARTS AGE busybox-deployment-6fc48bf75f-92pkn 0/1 ContainerCreating 0 5s busybox-deployment-6fc48bf75f-qv8wv 0/1 ContainerCreating 0 5s busybox-deployment-6fc48bf75f-wgrv5 0/1 ContainerCreating 0 5s busybox-deployment-f54b89b4-4fw4b 1/1 Running 0 3m1s busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 29m busybox-deployment-f54b89b4-n952j 1/1 Running 0 3m1s busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 29m busybox-deployment-f54b89b4-xlwr8 1/1 Running 0 3m1s

another 5 seconds later:

  • 3 new Pods running
  • 3 new Pods being created
  • 2 old Pods running ( at bottom ) waiting for termination
kubectl get pods NAME READY STATUS RESTARTS AGE busybox-deployment-6fc48bf75f-74wlt 1/1 Running 0 4s busybox-deployment-6fc48bf75f-92pkn 0/1 ContainerCreating 0 10s busybox-deployment-6fc48bf75f-qv8wv 1/1 Running 0 10s busybox-deployment-6fc48bf75f-wgrv5 0/1 ContainerCreating 0 10s busybox-deployment-6fc48bf75f-xcv5c 0/1 ContainerCreating 0 2s busybox-deployment-6fc48bf75f-xqmzc 1/1 Running 0 3s busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 29m busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 29m

A few seconds later shows all Pods new ( young AGE )

kubectl get pods NAME READY STATUS RESTARTS AGE busybox-deployment-6fc48bf75f-74wlt 1/1 Running 0 15s busybox-deployment-6fc48bf75f-92pkn 1/1 Running 0 21s busybox-deployment-6fc48bf75f-qv8wv 1/1 Running 0 21s busybox-deployment-6fc48bf75f-wgrv5 1/1 Running 0 21s busybox-deployment-6fc48bf75f-xcv5c 1/1 Running 0 13s busybox-deployment-6fc48bf75f-xqmzc 1/1 Running 0 14s

All Pods up to date with desired new busybox image:

kubectl get deploy NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox-deployment 6 6 6 6 35m

History now shows new busybox image as revision 3.

kubectl rollout history deployment.v1.apps/busybox-deployment deployment.apps/busybox-deployment REVISION CHANGE-CAUSE 1 <none> 2 kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-glibc --record=true 3 kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-uclibc --record=true

Describe Deployment :

kubectl describe deployment.extensions/busybox-deployment Name: busybox-deployment CreationTimestamp: Sun, 20 Jan 2019 10:16:11 +0200 Labels: app=busybox Annotations: deployment.kubernetes.io/revision: 3 kubernetes.io/change-cause: kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-uclibc --record=true Selector: app=busybox Replicas: 6 desired | 6 updated | 6 total | 6 available | 0 unavailable StrategyType: RollingUpdate RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: app=busybox Containers: busybox: Image: busybox:1.30-uclibc Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets: <none> NewReplicaSet: busybox-deployment-6fc48bf75f (6/6 replicas created)

Note the annotation that shows which image is being used.

Delete Pod.

kubectl delete -f myDeployment.yaml --force --grace-period=0 deployment.apps "busybox-deployment" force deleted

Kubectl Rollout Undo

Deployments allow you to roll back to any previous revision.

By default it keeps a list of your last 10 revisions.

Let's create our normal test set of 10 Pod replicas :

( First few steps should be boring familiar by now )

nano myDeployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: mydeployment labels: app: busybox spec: replicas: 10 strategy: type: RollingUpdate selector: matchLabels: app: busybox template: metadata: labels: app: busybox spec: containers: - name: busybox image: busybox imagePullPolicy: IfNotPresent command: ['sh', '-c', 'echo Container 1 is Running ; sleep 3600'] terminationGracePeriodSeconds: 0

Create the Deployment

kubectl create -f myDeployment.yaml deployment.apps/mydeployment created

Show history:

kubectl rollout history deployment.v1.apps/busybox-deployment deployment.apps/busybox-deployment REVISION CHANGE-CAUSE 1 <none>

Show the Pods:

kubectl get pods --show-labels NAME READY STATUS RESTARTS AGE LABELS busybox-deployment-5d6889b44-2sb8m 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-6dtgr 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-6t94r 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-7j6bg 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-8s5lq 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-dn62f 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-gpsrv 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-rxbqx 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-tvxrn 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-vt7dd 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44

Show version of busybox running ... to compare against later.

Remember that ... BusyBox v1.30.0

kubectl exec busybox-deployment-5d6889b44-2sb8m -i -t -- /bin/sh / # busybox | head -1 BusyBox v1.30.0 (2018-12-31 18:16:17 UTC) multi-call binary. / # exit

Change all Pods to use busybox:1.29.3

kubectl set image deployment.v1.apps/busybox-deployment busybox=busybox:1.29.3 --record deployment.apps/busybox-deployment image updated

Investigate Pods during rollout process

kubectl get pods --show-labels NAME READY STATUS RESTARTS AGE LABELS busybox-deployment-5d6889b44-6dtgr 1/1 Running 0 3m6s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-755956dfff-65vvk 0/1 ContainerCreating 0 4s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-gmlmm 1/1 Running 0 8s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-j2w84 0/1 ContainerCreating 0 4s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-lcjz2 1/1 Running 0 8s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-lfvzn 0/1 ContainerCreating 0 4s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-m4vzz 1/1 Running 0 5s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-mzw75 1/1 Running 0 8s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-njtm2 1/1 Running 0 8s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-nw6nh 1/1 Running 0 8s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-pzbmt 1/1 Running 0 5s app=busybox,pod-template-hash=755956dfff

Seconds later ... complete set of 10 fresh new Pods with young age.

kubectl get pods --show-labels NAME READY STATUS RESTARTS AGE LABELS busybox-deployment-755956dfff-65vvk 1/1 Running 0 14s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-gmlmm 1/1 Running 0 18s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-j2w84 1/1 Running 0 14s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-lcjz2 1/1 Running 0 18s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-lfvzn 1/1 Running 0 14s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-m4vzz 1/1 Running 0 15s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-mzw75 1/1 Running 0 18s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-njtm2 1/1 Running 0 18s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-nw6nh 1/1 Running 0 18s app=busybox,pod-template-hash=755956dfff busybox-deployment-755956dfff-pzbmt 1/1 Running 0 15s app=busybox,pod-template-hash=755956dfff

Check to make sure the Pods are using version 29.3 busybox:

kubectl exec busybox-deployment-755956dfff-65vvk -i -t -- /bin/sh / # busybox | head -1 BusyBox v1.29.3 (2018-12-24 21:25:20 UTC) multi-call binary. / # exit

Show history. New busybox is revision 2.

kubectl rollout history deployment.v1.apps/busybox-deployment deployment.apps/busybox-deployment REVISION CHANGE-CAUSE 1 <none> 2 kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.29.3 --record=true

Describe shows annotation as expected.

kubectl describe deployment.extensions/busybox-deployment Name: busybox-deployment Namespace: default CreationTimestamp: Sat, 19 Jan 2019 09:10:49 +0200 Labels: app=busybox Annotations: deployment.kubernetes.io/revision: 2 kubernetes.io/change-cause: kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.29.3 --record=true

NOW we undo this latest Deployment.

We undo deployment to revision 1 via flag : --to-revision=1

kubectl rollout undo deployment busybox-deployment --to-revision=1 deployment.extensions/busybox-deployment rolled back

Follow state of undo rollout

kubectl get deploy NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox-deployment 10 13 8 8 10m

8 Pods up to date in a few seconds.

kubectl get deploy NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox-deployment 10 11 10 8 10m

Still only 8 Pods up to date.

kubectl get deploy NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox-deployment 10 10 10 10 10m

Seconds later ... all 10 Pods back to the previous version busybox.

kubectl get pods --show-labels NAME READY STATUS RESTARTS AGE LABELS busybox-deployment-5d6889b44-4frh6 1/1 Running 0 2m28s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-4kfrf 1/1 Running 0 2m28s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-59qsz 1/1 Running 0 2m26s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-6xmkx 1/1 Running 0 2m26s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-77qnl 1/1 Running 0 2m25s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-7c9xb 1/1 Running 0 2m28s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-8pg76 1/1 Running 0 2m26s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-bjc5s 1/1 Running 0 2m24s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-jnqf5 1/1 Running 0 2m28s app=busybox,pod-template-hash=5d6889b44 busybox-deployment-5d6889b44-z5xql 1/1 Running 0 2m28s app=busybox,pod-template-hash=5d6889b44

Check Pods are using busybox version 1.30 ... the number we had to remember.

kubectl exec busybox-deployment-5d6889b44-4frh6 -i -t -- /bin/sh / # busybox | head -1 BusyBox v1.30.0 (2018-12-31 18:16:17 UTC) multi-call binary. / # exit

Success. kubectl rollout undo deployment correctly rolled back to revision 1.

Delete Deployment

kubectl delete -f myDeployment.yaml --force --grace-period=0 deployment.apps "busybox-deployment" force deleted

Deployment Internals

Let's create a Deployment and explore its internals.

nano myDeployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: busybox-deployment labels: app: busybox spec: replicas: 5 strategy: type: RollingUpdate selector: matchLabels: app: busybox template: metadata: labels: app: busybox spec: containers: - name: busybox image: busybox imagePullPolicy: IfNotPresent command: ['sh', '-c', 'echo Container is Running ; sleep 3600'] terminationGracePeriodSeconds: 0

Create the Deployment

kubectl create -f myDeployment.yaml pod/mydeployment created

List Deployment status :

kubectl get deploy NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox-deployment 5 5 5 5 5s

A Deployment actually manages a ReplicaSet that runs under its command.

kubectl get rs NAME DESIRED CURRENT READY AGE busybox-deployment-1111111111 5 5 5 14s

rs is ReplicaSet abbreviation for command line use.

ReplicaSet name starts with name of Deployment. Kubernetes adds a 10 digit random number.

kubectl get pods NAME READY STATUS RESTARTS AGE busybox-deployment-1111111111-6v8qt 1/1 Running 0 19s busybox-deployment-1111111111-fr4q7 1/1 Running 0 19s busybox-deployment-1111111111-p8h2l 1/1 Running 0 19s busybox-deployment-1111111111-pr85v 1/1 Running 0 19s busybox-deployment-1111111111-wt7hh 1/1 Running 0 19s

These Pods run under control of the ReplicaSet. Kubernetes adds 5 digit random number to individual Pod names.

Actual old and new Pods were similar to :

busybox-deployment-568495f8b6-rv8swand busybox-deployment-8659fbc5bf-6v8qt

I changed it to 1111.... and 3333... so it is easy to differentiate in lists below.

  • 1111... old, Pods created first
  • 3333... new, current fresh Pods

Let's scale to 7 replicas:

  • Deployment gets command to scale to 7 replicas
  • Deployment sends this command to the ReplicaSet
  • ReplicaSet receive command, counts 5 replicas, need 7 so create 2 more Pods.
kubectl scale deployment.v1.apps/busybox-deployment --replicas=7 deployment.apps/busybox-deployment scaled

Deployment now shows 7 running replicas.

kubectl get deploy NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox-deployment 7 7 7 7 113s ReplicaSets now show 7 running replicas. kubectl get rs NAME DESIRED CURRENT READY AGE busybox-deployment-1111111111 7 7 7 116s Detail of 7 running Pods. kubectl get pods NAME READY STATUS RESTARTS AGE busybox-deployment-1111111111-6v8qt 1/1 Running 0 2m busybox-deployment-1111111111-7bkvx 1/1 Running 0 11s busybox-deployment-1111111111-fr4q7 1/1 Running 0 2m busybox-deployment-1111111111-hj426 1/1 Running 0 11s busybox-deployment-1111111111-p8h2l 1/1 Running 0 2m busybox-deployment-1111111111-pr85v 1/1 Running 0 2m busybox-deployment-1111111111-wt7hh 1/1 Running 0 2m

Let's use this busybox image: busybox:1.30-uclibc

  • Deployment gets command : kubectl set image ...
  • Deployment sends this command to the ReplicaSet
  • ReplicaSet receive command, determines all Pods are running wrong version of busybox
  • A new set of Pods in the series busybox-deployment-3333333333 gets created
  • Some old busybox-deployment-1111111111 Pods are kept running during rolling update

maxUnavailable determines how many old Pods may be unavailable during update

maxSurge determines how many NEW Pods may be created in addition to already running Pods

kubectl set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-uclibc --record deployment.apps/busybox-deployment image updated

Investigate status of rolling update:

1111... series keep on doing useful work while update is in progress3333... series ... some new Pods already running3333... series ... some new Pods in process of being created: ContainerCreating

kubectl get pods NAME READY STATUS RESTARTS AGE busybox-deployment-3333333333-2sqcs 0/1 ContainerCreating 0 1s busybox-deployment-3333333333-69xq9 1/1 Running 0 3s busybox-deployment-3333333333-6ph4z 1/1 Running 0 3s busybox-deployment-3333333333-8trbx 0/1 ContainerCreating 0 1s busybox-deployment-3333333333-fmqwv 0/1 ContainerCreating 0 1s busybox-deployment-3333333333-rv8sw 1/1 Running 0 3s busybox-deployment-1111111111-6v8qt 1/1 Running 0 3m36s busybox-deployment-1111111111-fr4q7 1/1 Running 0 3m36s busybox-deployment-1111111111-p8h2l 1/1 Running 0 3m36s

A few seconds later and the rolling update is complete. All old 111... Pods deleted.

kubectl get pods NAME READY STATUS RESTARTS AGE busybox-deployment-3333333333-2sqcs 1/1 Running 0 5s busybox-deployment-3333333333-69xq9 1/1 Running 0 7s busybox-deployment-3333333333-6ph4z 1/1 Running 0 7s busybox-deployment-3333333333-8trbx 1/1 Running 0 5s busybox-deployment-3333333333-fmqwv 1/1 Running 0 5s busybox-deployment-3333333333-r2cgc 1/1 Running 0 4s busybox-deployment-3333333333-rv8sw 1/1 Running 0 7s

7 new Pods up-to-date and available.

kubectl get deploy NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE busybox-deployment 7 7 7 7 5m8s

List of ReplicaSets.

Note the old ReplicaSet is kept in case we need to roll back a Deployment.

kubectl get rs NAME DESIRED CURRENT READY AGE busybox-deployment-3333333333 7 7 7 98s busybox-deployment-1111111111 0 0 0 5m11s

From https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#revision-history-limit

All old ReplicaSets will be kept by default, consuming resources in etcd and crowding the output of kubectl get rs, if this field is not set.

etcd is the 'database' / key-value data store Kubernetes use to keep track of all its API objects.

etcd keeps at least 3 types of information about all Kubernetes API objects:

  • meta data ( data about data ) https://en.wikipedia.org/wiki/Metadata
  • Kubernetes uses metadata to keep track of objects and to note object interrelationships ( see below )
  • our desired spec ... details of the spec we need for our objects
  • the actual status of this specific object

This can be clearly and neatly seen when we run :

kubectl get rs/busybox-deployment-568495f8b6 -o yaml apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: annotations: deployment.kubernetes.io/desired-replicas: "7" deployment.kubernetes.io/max-replicas: "9" deployment.kubernetes.io/revision: "2" kubernetes.io/change-cause: kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-uclibc --record=true creationTimestamp: "2019-01-21T07:09:11Z" generation: 6 labels: app: busybox pod-template-hash: 568495f8b6 name: busybox-deployment-568495f8b6 namespace: default ownerReferences: - apiVersion: apps/v1 blockOwnerDeletion: true controller: true kind: Deployment name: busybox-deployment uid: f510ce02-1d4a-11e9-8596-0800270102d2 resourceVersion: "602067" selfLink: /apis/extensions/v1beta1/namespaces/default/replicasets/busybox-deployment-568495f8b6 uid: 73b2480b-1d4b-11e9-8596-0800270102d2 spec: replicas: 7 selector: matchLabels: app: busybox pod-template-hash: 568495f8b6 template: metadata: creationTimestamp: null labels: app: busybox pod-template-hash: 568495f8b6 spec: containers: - command: - sh - -c - echo Container is Running ; sleep 3600 image: busybox:1.30-uclibc imagePullPolicy: IfNotPresent name: busybox resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 0 status: availableReplicas: 7 fullyLabeledReplicas: 7 observedGeneration: 6 readyReplicas: 7 replicas: 7

Kubernetes Deployment course complete ... delete Deployment :

kubectl delete -f myDeployment.yaml --force --grace-period=0 pod "mydeployment" force deleted

Deleting a Deployment deletes the Deployment itself, underlying ReplicaSets and all its Pods.

Your Turn

Experiment with all the Deployment settings: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#writing-a-deployment-spec

Determine the correct settings for all your production Pods and their controlling Deployments.

Tag » How To Stop All Kubernetes Deployments