Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Flaky] Job E2E Test: Test Job PVCs #275

Closed
k82cn opened this issue Jul 2, 2019 · 6 comments
Closed

[Flaky] Job E2E Test: Test Job PVCs #275

k82cn opened this issue Jul 2, 2019 · 6 comments
Assignees
@TommyLike
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/high

Comments

@k82cn
Copy link
Member

k82cn commented Jul 2, 2019

/kind bug

Meet this failure several times, it's better to get it fixed :)

• Failure [67.221 seconds]
Job E2E Test: Test Job PVCs
/home/travis/gopath/src/volcano.sh/volcano/test/e2e/job_controlled_resource.go:28
  Generate PVC name if not specified [It]
  /home/travis/gopath/src/volcano.sh/volcano/test/e2e/job_controlled_resource.go:29
  Expected error:
      <*errors.errorString | 0xc0002ed7b0>: {
          s: "[Wait time out]: expected job 'job-pvc-name-empty' to have 1 ready pods, actual got 0",
      }
      [Wait time out]: expected job 'job-pvc-name-empty' to have 1 ready pods, actual got 0
  not to have occurred
  /home/travis/gopath/src/volcano.sh/volcano/test/e2e/job_controlled_resource.go:61

xref https://travis-ci.com/volcano-sh/volcano/jobs/212475065
@volcano-sh-bot volcano-sh-bot added the kind/bug Categorizes issue or PR as related to a bug. label Jul 2, 2019
@k82cn
Copy link
Member Author

k82cn commented Jul 2, 2019

/priority high

@TommyLike
Copy link
Contributor

/assign

@TommyLike
Copy link
Contributor

Container get killed due to the null pointer issue, Part of the log output as below:

error: invalid memory address or nil pointer dereference)
2019-07-03T02:30:09.103573733Z stderr F /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
2019-07-03T02:30:09.103578792Z stderr F /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
2019-07-03T02:30:09.103583234Z stderr F /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
2019-07-03T02:30:09.103589131Z stderr F /usr/local/go/src/runtime/asm_amd64.s:522
2019-07-03T02:30:09.103595906Z stderr F /usr/local/go/src/runtime/panic.go:513
2019-07-03T02:30:09.10360179Z stderr F /usr/local/go/src/runtime/panic.go:82
2019-07-03T02:30:09.103607499Z stderr F /usr/local/go/src/runtime/signal_unix.go:390
2019-07-03T02:30:09.103612392Z stderr F /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/client-go/tools/reference/ref.go:50
2019-07-03T02:30:09.103618034Z stderr F /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/client-go/tools/record/event.go:257
2019-07-03T02:30:09.103623151Z stderr F /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/client-go/tools/record/event.go:257
2019-07-03T02:30:09.103628347Z stderr F /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/client-go/tools/record/event.go:287
2019-07-03T02:30:09.103633491Z stderr F /root/go_projects/src/volcano.sh/volcano/pkg/controllers/job/job_controller_actions.go:177
2019-07-03T02:30:09.103638517Z stderr F /root/go_projects/src/volcano.sh/volcano/pkg/controllers/job/job_controller.go:188
2019-07-03T02:30:09.103645663Z stderr F /root/go_projects/src/volcano.sh/volcano/pkg/controllers/job/state/pending.go:71
2019-07-03T02:30:09.103652113Z stderr F /root/go_projects/src/volcano.sh/volcano/pkg/controllers/job/job_controller.go:258
2019-07-03T02:30:09.103657705Z stderr F /root/go_projects/src/volcano.sh/volcano/pkg/controllers/job/job_controller.go:219
2019-07-03T02:30:09.103673244Z stderr F /root/go_projects/src/volcano.sh/volcano/pkg/controllers/job/job_controller.go:208
2019-07-03T02:30:09.103678923Z stderr F /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
2019-07-03T02:30:09.10368433Z stderr F /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
2019-07-03T02:30:09.103689893Z stderr F /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
2019-07-03T02:30:09.103694414Z stderr F /usr/local/go/src/runtime/asm_amd64.s:1333
2019-07-03T02:30:09.105394599Z stderr F panic: runtime error: invalid memory address or nil pointer dereference [recovered]
2019-07-03T02:30:09.105409214Z stderr F   panic: runtime error: invalid memory address or nil pointer dereference
2019-07-03T02:30:09.105415653Z stderr F [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xeefb35]
2019-07-03T02:30:09.105419426Z stderr F
2019-07-03T02:30:09.105423925Z stderr F goroutine 262 [running]:
2019-07-03T02:30:09.105428189Z stderr F volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
2019-07-03T02:30:09.105432863Z stderr F   /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x108
2019-07-03T02:30:09.105437912Z stderr F panic(0x1193460, 0x1f5ceb0)
2019-07-03T02:30:09.105441552Z stderr F   /usr/local/go/src/runtime/panic.go:513 +0x1b9
2019-07-03T02:30:09.105446488Z stderr F volcano.sh/volcano/pkg/apis/batch/v1alpha1.(*Job).GetObjectKind(0x0, 0x0, 0x0)
2019-07-03T02:30:09.105450537Z stderr F   <autogenerated>:1 +0x5
2019-07-03T02:30:09.105454476Z stderr F volcano.sh/volcano/vendor/k8s.io/client-go/tools/reference.GetReference(0xc0002432d0, 0x1478ec0, 0x0, 0x1d, 0x7fe6dd0999f8, 0x452ad0)
2019-07-03T02:30:09.105465796Z stderr F   /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/client-go/tools/reference/ref.go:50 +0x73
2019-07-03T02:30:09.10547028Z stderr F volcano.sh/volcano/vendor/k8s.io/client-go/tools/record.(*recorderImpl).generateEvent(0xc0004fc400, 0x1478ec0, 0x0, 0x0, 0xbf3f22ac461b746d, 0x818d309341, 0x1f6eee0, 0x1303c98, 0x7, 0x130449a, ...)
2019-07-03T02:30:09.105478025Z stderr F   /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/client-go/tools/record/event.go:257 +0x5d
2019-07-03T02:30:09.105483042Z stderr F volcano.sh/volcano/vendor/k8s.io/client-go/tools/record.(*recorderImpl).Event(0xc0004fc400, 0x1478ec0, 0x0, 0x1303c98, 0x7, 0x130449a, 0x8, 0xc00026a000, 0xc3)
2019-07-03T02:30:09.105487133Z stderr F   /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/client-go/tools/record/event.go:287 +0xc2
2019-07-03T02:30:09.105490983Z stderr F volcano.sh/volcano/pkg/controllers/job.(*Controller).createJob(0xc0000f96c0, 0xc00087c4b0, 0x1385000, 0x0, 0x0)
2019-07-03T02:30:09.105495029Z stderr F   /root/go_projects/src/volcano.sh/volcano/pkg/controllers/job/job_controller_actions.go:177 +0x85b
2019-07-03T02:30:09.105498613Z stderr F volcano.sh/volcano/pkg/controllers/job.(*Controller).createJob-fm(0xc00087c4b0, 0x1385000, 0xc000871bd8, 0x5)
2019-07-03T02:30:09.105502842Z stderr F   /root/go_projects/src/volcano.sh/volcano/pkg/controllers/job/job_controller.go:188 +0x3e
2019-07-03T02:30:09.10550646Z stderr F volcano.sh/volcano/pkg/controllers/job/state.(*pendingState).Execute(0xc000388160, 0x1303c0c, 0x7, 0xc000871bd8, 0x5)
2019-07-03T02:30:09.105510408Z stderr F   /root/go_projects/src/volcano.sh/volcano/pkg/controllers/job/state/pending.go:71 +0x93
2019-07-03T02:30:09.105514093Z stderr F volcano.sh/volcano/pkg/controllers/job.(*Controller).processNextReq(0xc0000f96c0, 0xc000475100)
2019-07-03T02:30:09.105517759Z stderr F   /root/go_projects/src/volcano.sh/volcano/pkg/controllers/job/job_controller.go:258 +0x719
2019-07-03T02:30:09.105628737Z stderr F volcano.sh/volcano/pkg/controllers/job.(*Controller).worker(0xc0000f96c0)
2019-07-03T02:30:09.105636899Z stderr F   /root/go_projects/src/volcano.sh/volcano/pkg/controllers/job/job_controller.go:219 +0x2b
2019-07-03T02:30:09.105648041Z stderr F volcano.sh/volcano/pkg/controllers/job.(*Controller).worker-fm()
2019-07-03T02:30:09.105652235Z stderr F   /root/go_projects/src/volcano.sh/volcano/pkg/controllers/job/job_controller.go:208 +0x2a
2019-07-03T02:30:09.105660345Z stderr F volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000642520)
2019-07-03T02:30:09.105663751Z stderr F   /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
2019-07-03T02:30:09.105667107Z stderr F volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000642520, 0x0, 0x0, 0x1, 0x0)
2019-07-03T02:30:09.105671471Z stderr F   /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbe
2019-07-03T02:30:09.10567527Z stderr F volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc000642520, 0x0, 0x0)
2019-07-03T02:30:09.105678804Z stderr F   /root/go_projects/src/volcano.sh/volcano/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
2019-07-03T02:30:09.105682231Z stderr F created by volcano.sh/volcano/pkg/controllers/job.(*Controller).Run
2019-07-03T02:30:09.10568607Z stderr F  /root/go_projects/src/volcano.sh/volcano/pkg/controllers/job/job_controller.go:208 +0x37c

@TommyLike
Copy link
Contributor

Should have been fixed via #277 and #283

kevin-wangzefeng pushed a commit that referenced this issue Jul 4, 2019
@k82cn
Merge pull request #275 from k82cn/ka_273_1
3b9ecd6 
Updated client-go to 1.11.
@k82cn
Copy link
Member Author

k82cn commented Jul 4, 2019

/close

Seems better after #277 and #283

@volcano-sh-bot
Copy link
Contributor

@k82cn: Closing this issue.

In response to this:

/close

Seems better after #277 and #283

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@TommyLike TommyLike

Labels
kind/bug Categorizes issue or PR as related to a bug. priority/high
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@TommyLike @k82cn @volcano-sh-bot