Investigate performing tenant update without waking up idled services

[aslakknutsen]

No description provided.

[surajssd]

# there is a dc 'testing-idle'
$ oc get dc
NAME           REVISION   DESIRED   CURRENT   TRIGGERED BY
testing-idle   2          0         0         config,image(testing-idle:latest)

# currently no pods are running for that dc
$ oc get pods
NAME                   READY     STATUS      RESTARTS   AGE
testing-idle-1-build   0/1       Completed   0          9m
 

 
# See the ports defined in the dc
$ oc get dc testing-idle -o=jsonpath='{.spec.template.spec.containers[0].ports}'
[map[containerPort:8080 protocol:TCP] map[containerPort:8443 protocol:TCP] 
map[containerPort:8000 protocol:TCP]]
  
# so i edited the dc to add new port
$ oc edit dc testing-idle
deploymentconfig "testing-idle" edited

# which looks like following
$ oc get dc testing-idle -o=jsonpath='{.spec.template.spec.containers[0].ports}'
[map[containerPort:8080 protocol:TCP] map[containerPort:8443 protocol:TCP]
map[protocol:TCP containerPort:8000] map[containerPort:9000 protocol:TCP]]

# this has triggered a new deployment, see that revision has changed to 3
$ oc get dc
NAME           REVISION   DESIRED   CURRENT   TRIGGERED BY
testing-idle   3          0         0         config,image(testing-idle:latest)

# still no new pod
$ oc get pods
NAME                   READY     STATUS      RESTARTS   AGE
testing-idle-1-build   0/1       Completed   0          18m

so we have successfully updated the config without unidling the deploymentconfig.

[surajssd]

$ oc version
oc v3.7.2+282e43f
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server xxxx
openshift v3.7.23
kubernetes v1.7.6+a08f5eeb62

[surajssd]

@aslakknutsen as we saw it is not unidling the service even on the dc update, what changes need to go out on the tenant side?

[aslakknutsen]

Try changing the Image

[surajssd]

@aslakknutsen

# image name is 'fabric8-wit:SNAPSHOT-PR-2048-5'
$ oc get dc
NAME          REVISION   DESIRED   CURRENT   TRIGGERED BY
fabric8-wit   1          0         0         config,image(fabric8-wit:SNAPSHOT-PR-2048-5)


$ oc get pods
No resources found.


$ oc get dc fabric8-wit -o=jsonpath='{.spec.template.spec.containers[0].image}'
docker.io/fabric8/fabric8-wit@sha256:47f45a7e5849e4050855483bc812a9805428d77f0f6ffda82b84a6aa7677de21
 

# changed the image name to 'fabric8-wit:SNAPSHOT-PR-2071-5'
$ oc set image dc/fabric8-wit fabric8-wit=docker.io/fabric8/fabric8-wit:SNAPSHOT-PR-2071-5
deploymentconfig "fabric8-wit" image updated

# image name is changed
$ oc get dc fabric8-wit -o=jsonpath='{.spec.template.spec.containers[0].image}'
docker.io/fabric8/fabric8-wit:SNAPSHOT-PR-2071-5

# pod revision updated
$ oc get dc
NAME          REVISION   DESIRED   CURRENT   TRIGGERED BY
fabric8-wit   2          0         0         config,image(fabric8-wit:SNAPSHOT-PR-2048-5)


# no pods created still
$ oc get pods
No resources found.

[surajssd]

$ cat dc.yaml 
apiVersion: v1
kind: DeploymentConfig
metadata:
  labels:
    run: wit
  name: wit
spec:
  replicas: 1
  selector:
    run: wit
  strategy:
    resources: {}
  template:
    metadata:
      labels:
        run: wit
    spec:
      containers:
      - image: fabric8/fabric8-wit:SNAPSHOT-PR-2048-5
        name: wit

# create above deploymentconfig
$ oc apply -f dc.yaml 
deploymentconfig "wit" created

# which created the pod
$ oc get pods
NAME          READY     STATUS    RESTARTS   AGE
wit-1-svcs5   1/1       Running   0          1m

# exposing the dc with service
$ oc expose dc wit --port 8080
service "wit" exposed

# idling the service
 $ oc idle wit
The service "testing-idle/wit" has been marked as idled 
The service will unidle DeploymentConfig "testing-idle/wit" to 1 replicas once it receives traffic 
DeploymentConfig "testing-idle/wit" has been idled

# no pods running
$ oc get pods
No resources found.

$ oc get dc
NAME      REVISION   DESIRED   CURRENT   TRIGGERED BY
wit       1          0         0         config

edited the file to have new image

$ git diff
diff --git a/dc.yaml b/dc.yaml
index d8fc777..8cca768 100644
--- a/dc.yaml
+++ b/dc.yaml
@@ -16,5 +16,5 @@ spec:
         run: wit
     spec:
       containers:
-      - image: fabric8/fabric8-wit:SNAPSHOT-PR-2048-5
+      - image: fabric8/fabric8-wit:SNAPSHOT-PR-2071-5
         name: wit

# applied the changed configs, with changed image
$ oc apply -f dc.yaml 
deploymentconfig "wit" configured

$ oc get pods
NAME          READY     STATUS    RESTARTS   AGE
wit-2-bckzr   1/1       Running   0          35s

It is running because in the configuration we have replicas: 1 which is setting the value to 1.

Now the proble here is:

• We created a resource using oc apply from a file declaratively
• Then used the command oc idle to change the value of one field replicas, from replicas: 1 to replicas: 0. This value has changed only on cluster. But the declarative config still says, replicas: 1. Now our cluster config and local config is out of sync.
• Now when we make changes to config, let's say to the image name. Now we have cluster config and local config out of sync in two places the image name and replicas.
• When you do oc apply the cluster takes it as truth value. So the replicas value in file is still 1 that is why the idled service is unidled again.

We are mixing the declarative and imperative way to manipulating resources. So the possible fix to the problem is to use only one way to do things. Only use the declarative way i.e. to idle change the replica value to 0 and then apply the config. This way we can maintain single source of truth.

for idling do this

$ git diff
diff --git a/dc.yaml b/dc.yaml
index d8fc777..a65d0ef 100644
--- a/dc.yaml
+++ b/dc.yaml
@@ -5,7 +5,7 @@ metadata:
     run: wit
   name: wit
 spec:
-  replicas: 1
+  replicas: 0
   selector:
     run: wit
   strategy:

$ oc apply -f dc.yaml 
deploymentconfig "wit" configured

$ oc apply -f dc.yaml 
deploymentconfig "wit" configured

the svc is idled and the configs are consistent, make as much updates you would like to the config locally, and apply again, to unidle set the value of replicas to 1.