Support reason parameter on pod delete

[bgrant0607]

Forked from #137. /cc @dchen1107

Client-initiated terminations are triggered by DELETE via the API. As discussed in #137, I'd like clients to be able to pass in arbitrary termination reasons, to facilitate ecosystem management extensions, as well as user-provided reasons.

I suggest a URL parameter on DELETE: ?reason=MYREASON.

The apiserver would record this as the termination reason, regardless what Kubelet reported when the termination actually occurred.

Once we generate termination events, the reason should be visible there.

We plan to pass termination reasons to lifecycle hooks (#1445). This means we'll need a way to pass the reason to Kubelet. I suggest implementing a "deletion dance", which we've also talked about doing in the apiserver: rather than actually deleting objects, we should first put them into a deleting/terminated state, so that clients can observe their final state. The reason could be passed this way.

Of course, objects eventually need to be deleted, and there's always a chance that some clients may not have observed the final state. We still need to handle that, but we should ensure it doesn't happen often. Think of this as a declarative graceful termination.

See also #1370.

[thockin]

I have a problem with DELETE not meaning delete. Specifically, after I
call DELETE /pods/foo, I should be able to immediately create a new pod
named foo.

So either we need a STOP verb that is different than DELETE, or we need
DELETE to move a resource to a new collection (pods -> exPods?) for later
GC.

On Fri, Sep 26, 2014 at 11:29 AM, bgrant0607 notifications@github.com
wrote:

Forked from #137
#137. /cc
@dchen1107 https://github.com/dchen1107

Client-initiated terminations are triggered by DELETE via the API. As
discussed in #137
#137, I'd like
clients to be able to pass in arbitrary termination reasons, to facilitate
ecosystem management extensions, as well as user-provided reasons.

I suggest a URL parameter on DELETE: ?reason=MYREASON.

The apiserver would record this as the termination reason, regardless what
Kubelet reported when the termination actually occurred.

Once we generate termination events, the reason should be visible there.

We plan to pass termination reasons to lifecycle hooks (#1445
#1445). This
means we'll need a way to pass the reason to Kubelet. I suggest
implementing a "deletion dance", which we've also talked about doing in the
apiserver: rather than actually deleting objects, we should first put them
into a deleting/terminated state, so that clients can observe their final
state. The reason could be passed this way.

Of course, objects eventually need to be deleted, and there's always a
chance that some clients may not have observed the final state. We still
need to handle that, but we should ensure it doesn't happen often. Think of
this as a declarative graceful termination.

Reply to this email directly or view it on GitHub
#1462.

[bgrant0607]

@thockin Just as with self-terminating pods, it's useful for the state of stopped pods to not become immediately inaccessible, as you wrote in https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/pod-states.md.

Some information could be communicated using events, as discussed in #1370. Do we want the kubelets to subscribe to events? Regardless, I don't think we want to copy the full final current state into events.

Delete could make the name available without immediately eliminating the data. The deleted object could be made accessible via a different index key, such as UID.

Or DELETE with no reason could result in immediate removal.

A stop verb (POST objectpath/stop) does have a certain appeal. That could, for example, be used for graceful termination of replication controllers (gracefully kill all controlled pods) or services (allow traffic to drain). I just want to retain enough uniformity in the API to facilitate use of general patterns, and someone would need to investigate/consider the consequences to HTTP infrastructure (caches, etc.).

[thockin]

On Fri, Sep 26, 2014 at 11:54 AM, bgrant0607 notifications@github.com wrote:

@thockin Just as with self-terminating pods, it's useful for the state of stopped pods to not become immediately inaccessible, as you wrote in https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/pod-states.md.

I agree with the sentiment of this issue, I just wanted to call out
the issue I saw

Some information could be communicated using events, as discussed in #1370. Do we want the kubelets to subscribe to events? Regardless, I don't think we want to copy the full final current state into events.

Delete could make the name available without immediately eliminating the data. The deleted object could be made accessible via a different index key, such as UID.

That's more or less what I called "exPod" - it's no longer at the same
REST path. I guess we could always index by both name and UID, and
let UID linger after name has been released

Or DELETE with no reason could result in immediate removal.

A stop verb (POST objectpath/stop) does have a certain appeal. That could, for example, be used for graceful termination of replication controllers (gracefully kill all controlled pods) or services (allow traffic to drain). I just want to retain enough uniformity in the API to facilitate use of general patterns, and someone would need to investigate/consider the consequences to HTTP infrastructure (caches, etc.).

I agree explicit STOP might be useful, but I don't have enough REST
experience to say whether STOP + DELETE is a better or worse pattern
that DELETE + TTL-in-secondary-index

[bgrant0607]

I'd actually do STOP + TTL, at least in the apiserver. DELETE would only be necessary in the case that the client immediately needed to reuse the name. It could issue the DELETE after GETting whatever final state it wanted, if any.

[thockin]

To be clear - my gut feeling is that DELETE should release the Name of the
resource, but that the resource should still be available by a different
index, probably UID. I presume it would stay in the "Terminated" state at
this point.

Specifically:

precondition /pods/foo exists

precondition /podsByUid/1234-56-78-9abcdef0 is an alias to /pods/foo (or

vice-versa)

DELETE /pods/foo

postcondition: /pods/foo does not exist

postcondition: /podsByUid/1234-56-78-9abcdef0 exists for some TTL

DELETE /podsByUid/1234-56-78-9abcdef0

not sure if that should work or not

Additionally, if a pod goes "terminated" and stays there for the TTL, it
should be removed, regardless of whether it has been explicitly DELETEd or
not.

On Fri, Sep 26, 2014 at 2:22 PM, bgrant0607 notifications@github.com
wrote:

I'd actually do STOP + TTL, at least in the apiserver. DELETE would only
be necessary in the case that the client immediately needed to reuse the
name. It could issue the DELETE after GETting whatever final state it
wanted, if any.

Reply to this email directly or view it on GitHub
#1462 (comment)
.

[dchen1107]

Thanks for forking this out. I will attack this once I have done with what I am currently working on.

Haven't read through all discussion yet, but agreed with this in general. Also I agreed @thockin's comment above that DELETE should release the Name of the resource since NAME is space-uniqueness, and the same NAME of the resource might be reused to create the resource right after the deletion.

Ok, then is there a human-friendly issue for only keeping UID for deleted resource object?

[bgrant0607]

I couldn't find an issue, but I recently discussed adding a uid precondition with @lavalamp, which would prevent deleting the wrong object (in a delete-create-delete race) without using a uid-based URL path, in which case we'd only need to support GET and LIST by uids.

[smarterclayton]

For auditability purposes name probably needs to remain in some form but in a separate space. We will never be guaranteed that every use of a resource will record name and uid - an auditor has to be able to work backwards from a name in a URL and a time to an audit record probably.

Moving to the other issue