CAT: align timings of eks alb ingress deregistration, Linkerd removal, and pod termination

[yolabingo]

Parent Issue

Allow transparent rolling restarts of a dotCMS cluster to logged-in, backend users #28532

Task

When terminating a dotCMS pod in aws eks we want to give the application time to finish in-flight requests. When k8s initiates a pod shutdown a few things happen.

The ALB Ingress default behavior needs no modification. When using kubernetes.io/ingress.class: al the pod is deregistered from the ALB TG. The deregistered target pod may finish in-flight requests but won't receive new requests.

In k8s we need to set timeouts so the Pod runs and Linkerd maintains its connection to the Pod for a brief period:
pod preStop < linkerd 'wait before exit' < terminationGracePeriodSeconds

example:

apiVersion: apps/v1
kind: StatefulSet
spec:
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
        config.alpha.linkerd.io/proxy-wait-before-exit-seconds: '30'
    spec:
      terminationGracePeriodSeconds: 40
      containers:
      - name: dotcms
        image: dotcms/dotcms:trunk
        lifecycle:
          preStop:
            exec:
              command:
              - sleep
              - '20'

Proposed Objective

Cloud Engineering

Proposed Priority

Priority 3 - Average

[yolabingo]

Completed - creating issue for visibility and posterity