Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[flake test] TestStartStop/group/old-k8s-version/serial/Pause - "sudo systemctl disable kubelet" failed #11615

Open
ilya-zuyev opened this issue Jun 9, 2021 · 5 comments
Labels
area/testing kind/flake Categorizes issue or PR as related to a flaky test. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/backlog Higher priority than priority/awaiting-more-evidence.

Comments

@ilya-zuyev
Copy link
Contributor

There is at least one test flaking with error like below. It looks like it's related to minikube stop operation and kubelet service status.

	I0609 01:41:59.357566  373430 ssh_runner.go:149] Run: sudo systemctl disable kubelet
	I0609 01:41:59.480954  373430 retry.go:31] will retry after 276.165072ms: kubelet disable: sudo systemctl disable kubelet: Process exited with status 1
	stdout:
	
	stderr:
	Synchronizing state of kubelet.service with SysV service script with /lib/systemd/systemd-sysv-install.
	Executing: /lib/systemd/systemd-sysv-install disable kubelet
	update-rc.d: error: kubelet Default-Start contains no runlevels, aborting.
	I0609 01:41:59.757382  373430 ssh_runner.go:149] Run: sudo systemctl disable kubelet
	I0609 01:41:59.875735  373430 retry.go:31] will retry after 540.190908ms: kubelet disable: sudo systemctl disable kubelet: Process exited with status 1
	stdout:
	
	stderr:
	Synchronizing state of kubelet.service with SysV service script with /lib/systemd/systemd-sysv-install.
	Executing: /lib/systemd/systemd-sysv-install disable kubelet
	update-rc.d: error: kubelet Default-Start contains no runlevels, aborting.

For the full log please see https://storage.googleapis.com/minikube-builds/logs/11610/64a4182/Docker_Linux.html#fail_TestStartStop%2fgroup%2fold-k8s-version%2fserial%2fPause

@ilya-zuyev ilya-zuyev added area/testing kind/flake Categorizes issue or PR as related to a flaky test. labels Jun 9, 2021
@sharifelgamal
Copy link
Collaborator

update-rc.d: error: kubelet Default-Start contains no runlevels, aborting. makes it seems like the kubelet systemd service somehow got into a very bad state

@ilya-zuyev
Copy link
Contributor Author

https://github.com/systemd/systemd/blob/8156422c8f0d94d3444043282f01551084271b22/src/systemctl/systemctl-sysv-compat.c#L183

somehow systemd thinks kubelet has a sysV part and tries to use it to stop the unit. Setting SYSTEMCTL_SKIP_SYSV env for systemctl stop should help

@spowelljr spowelljr added the priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. label Jun 15, 2021
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 13, 2021
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 13, 2021
@sharifelgamal sharifelgamal added lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. labels Oct 20, 2021
@sharifelgamal
Copy link
Collaborator

The Pause tests continue to flake, I'm upping the priority of this.

@sharifelgamal sharifelgamal added this to the 1.25.0-candidate milestone Oct 20, 2021
@spowelljr spowelljr modified the milestones: 1.25.0, 1.26.0-candidate Nov 8, 2021
@spowelljr spowelljr added priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. and removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Dec 29, 2021
@spowelljr spowelljr added priority/backlog Higher priority than priority/awaiting-more-evidence. and removed priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels May 4, 2022
@spowelljr spowelljr modified the milestones: 1.26.0, 1.27.0-candidate Jun 24, 2022
@spowelljr spowelljr modified the milestones: 1.27.0-previous, 1.29.0 Nov 28, 2022
@spowelljr spowelljr modified the milestones: 1.31.0, 1.32.0 Jul 19, 2023
@spowelljr spowelljr removed this from the 1.32.0 milestone Jul 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/testing kind/flake Categorizes issue or PR as related to a flaky test. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/backlog Higher priority than priority/awaiting-more-evidence.
Projects
None yet
Development

No branches or pull requests

5 participants