Pods remain in Completing state, inconsistent with specified lifecycle policy

[kye308]

What happened: Pods in a job remain in Running even though another task with policy CompleteJob on TaskSucceeded has completed.

What you expected to happen: Pods are completed after the task has completed.

How to reproduce it (as minimally and precisely as possible):

vcjob yaml used

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: vc14-test-1
spec:
  minAvailable: 3
  plugins:
    env: []
    svc: []
  policies:
  - action: RestartTask
    events:
    - PodEvicted
  - action: TerminateJob
    events:
    - PodFailed
  queue: default
  tasks:
  - name: driver
    policies:
    - action: CompleteJob
      events:
      - TaskCompleted
    replicas: 1
    template:
      spec:
        containers:
        - args:
          - -c
          - "echo Driver Started!; sleep 20; echo Still working...; sleep 20; echo Job Finished!" 
          command:
            - /bin/sh
          image: ubuntu:20.04
          name: container-0
          volumeMounts:
          - mountPath: /dev/shm
            name: bigger-shm
        restartPolicy: Never
        terminationGracePeriodSeconds: 90
        volumes:
        - emptyDir:
            medium: Memory
          name: bigger-shm
  - name: worker
    replicas: 2
    template:
      spec:
        containers:
        - args:
          - -c
          - "echo worker Started!; sleep 450; echo Still working...; sleep 450; echo Job Finished!" 
          command:
            - /bin/sh
          image: ubuntu:20.04
          name: container-0
          volumeMounts:
          - mountPath: /dev/shm
            name: bigger-shm
        restartPolicy: Never
        terminationGracePeriodSeconds: 90
        volumes:
        - emptyDir:
            medium: Memory
          name: bigger-shm

vcjob status

status:
  minAvailable: 3
  running: 2
  runningDuration: 45.576421357s
  state:
    lastTransitionTime: "2022-01-13T22:16:09Z"
    phase: Completing
  succeeded: 1
  taskStatusCount:
    driver:
      phase:
        Succeeded: 1
    worker:
      phase:
        Running: 2
  version: 3

Pods finally were completed after almost 24 hours.

Anything else we need to know?:
This was previously working on version 1.3.0. Pods would be marked completed within 10 minutes.

Environment:

• Volcano Version: 1.4.0

[kye308]

for this job the pods reached completed state after about an hour

[kye308]

controller logs: https://gist.github.com/kye308/d1fe5526cf08b2e2cdf44e0323e4770a

[kye308]

scheduler logs: https://gist.github.com/kye308/a1115373a9821d0794254b672f0761ea

[hwdef]

/cc

[hwdef]

#1746

Please use the volcano version after this commit

[kye308]

@hwdef we are using the v1.4.0 tag which appears to include this change already. Is my understanding correct?

[hwdef]

Yes, 1.4.0 includes this commit, may be caused by other reasons.

[william-wang]

@hwdef @shinytang6 We need to try to reproduce the issue based on kye308's input in our envrionment.

[kye308]

Hi all,

I was able to fix this issue by cherry-picking #1719. Would it be possible to cherry-pick that change into the v1.4.0 release?

[william-wang]

@kye308 Sure. It's reasonable to cherry-pick it to v1.4.0.

[kye308]

@william-wang do you have any estimate on when this can be included in the v1.4.0 release on dockerhub?

[william-wang]

@kye308 The bugfix has been merged to release-1.4 branch. We planed to update the image on dockerhub this week.

[kye308]

@william-wang 👍 thanks for the update

[kye308]

@william-wang any updates on the dockerhub image? would it include this change as well: #2026

[stale]

Hello 👋 Looks like there was no activity on this issue for last 90 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for 60 days, this issue will be closed (we can always reopen an issue if we need!).

[stale]

Closing for now as there was no activity for last 60 days after marked as stale, let us know if you need this to be reopened! 🤗