Recovering from deployment pod that failed without reporting state

[sosiouxme]

I deployed a router, it was supposed to be broken but not quite like this.

$ osc get pods
POD                    IP        CONTAINER(S)   IMAGE(S)                           HOST                             LABELS    STATUS       CREATED     MESSAGE
deploy-router-1wlv0i                                                               ip-10-51-163-209/10.51.163.209   <none>    Failed       2 minutes   
                                 deployment     openshift/origin-deployer:v0.5.1                                              Terminated   2 minutes   exit code 255
$ docker ps -a
CONTAINER ID        IMAGE                              COMMAND                CREATED             STATUS                       PORTS               NAMES
db929a127597        openshift/origin-deployer:v0.5.1   "/usr/bin/openshift-   2 minutes ago       Exited (255) 2 minutes ago                       k8s_deployment.3ea4efa8_deploy-router-1wlv0i_default_dff46974-fea2-11e4-95d6-22000b3280a3_e3f01fb3   
23992ab347b7        openshift/origin-pod:v0.5.1        "/pod"                 2 minutes ago       Exited (0) 2 minutes ago                         k8s_POD.d6d6c430_deploy-router-1wlv0i_default_dff46974-fea2-11e4-95d6-22000b3280a3_2f922d59          
a98bcb10a0b5        openshift/origin-release:latest    "/bin/sh -c 'tar mxz   23 hours ago        Exited (0) 23 hours ago                          naughty_albattani                                                                                    
$ docker logs db929a127597
F0520 03:47:26.039993       1 deployer.go:66] Failed to decode DeploymentConfig from controller: couldn't get version/kind: unexpected end of JSON input

Alright, not very helpful, but the point is it's broken. What do I do now?

$ osc deploy router
router #1 deployment pending on update. Run osc deploy --latest to deploy now.

$ osc deploy router --latest
Error: #1 is already in progress ()
$ osc get deployment
NAME      STATUS    CAUSE

It's not pending, it's failed. I should be able to just run another one.

Other than delete everything I honestly don't know how to proceed. Could there be some better directions on this?

[ashcrow]

I hit a very similar problem. I deleted the service and the deploymentConfig to get the status removed. I'm interested to see how to fix this as well!

[pweil-]

@sosiouxme - this is due to an incompatibility in older deployer images.

If you could, can you please:

1. Check to see if you have images in docker that have both a docker.io prefix and a non-docker.io prefix
2. Delete the openshift/origin-deployer images
3. Redeploy - you likely have to delete the existing deployer. @abhgupta - is that in line with the work you've been doing for cancelling existing deployments?

/cc @akram who also ran in to this

[sosiouxme]

@pweil why does it matter what images are in use? Sometimes things will go wrong with the deployer, that's just reality, but what should a user do when the deployer has failed in some way? There should be some better way to clear/reset/override deployment state than to nuke everything.

[pweil-]

@sosiouxme Agreed. There was a forward compatibility issue with an annotation in the deployer image that resulted in the decode error you're seeing. (ie. using newer code against an older image)

@abhgupta was working on something for cancelling in flight deployments, I'll let him comment on that piece.

[sosiouxme]

Alright thanks. I'm not so concerned about why the deploy failed - I can think of a lot of ways that could happen and some just can't be prevented. More interested in what the user is supposed to do to recover.

[abhgupta]

In your specific case, as @pweil- pointed out, it was an incompatibility issue that was introduced in the latest code (with the older images). This is not something that is expected to happen and is not user-serviceable. We fixed it by cutting a new version and the users (internal/dev) were required to delete older images from their systems in order to move forward with the deployment.

More generally, if a user faces an issue with deployments, now there are multiple options.

The user can kick off a new deployment, if there is no pending deployment

osc deploy <config-name> --latest

The user can "retry" the last deployment, if it is in failed state

osc deploy <config-name> --retry

The user can cancel an in-progress deployment at any time (See #2468)

osc deploy <config-name> --cancel

[sosiouxme]

@abhgupta thanks, I think --cancel will get what we need. Could it be mentioned when --latest or --retry refuse due to a pending deploy?

I added a layer ENTRYPOINT ["/usr/bin/openshift-BORK"] on origin-deployer and used that to recreate. Once it fails:

# osc get pods
POD        IP           CONTAINER(S)   IMAGE(S)                           HOST                             LABELS    STATUS    CREATED      MESSAGE
router-1   172.17.0.7                                                     ip-10-136-62-105/10.136.62.105   <none>    Pending   40 minutes   
                        deployment     openshift/origin-deployer:v0.5.2                                              Waiting                API error (404): Cannot start container 472ad7e8b6cbb50a8e18b593d6c18e5d1b3faa0e1d1aeb87deda968ab47eabba: [8] System error: exec: "/usr/bin/openshift-BORK": stat /usr/bin/openshift-BORK: no such file or directory

# osc deploy router
router #1 deployment pending 40 minutes ago
# osc deploy --retry router
error: #1 is Pending; only failed deployments can be retried
# osc deploy --latest router
error: #1 is already in progress (Pending)

All of those would be good places to mention that if you don't want to wait (forever) for that pending deployment, then use --cancel.

[abhgupta]

Good points - will incorporate these.

Also, another important thing to note is that deployer pods now have 6 hours to complete the deployment task and will be killed after that, if still pending/running. This aims to address "stuck" deployments and builds on top on the work that Derek did for introducing ActiveDeadlineSeconds for a pod. This duration cannot be set by a user as of now for the deployer pod and we will expose it via the master configuration later.

Note that this specifically addresses cases where a deployer pod is "stuck" and is pending/running forever.

See #2402

[pweil-]

@abhgupta you want to put this one in your name? Anything to make my list shorter 😄