Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Our ksonnet sometimes throws "SIGSEGV: segmentation violation" failing deployment #241

Closed
Ark-kun opened this issue Nov 13, 2018 · 5 comments · Fixed by #1005
Closed

Our ksonnet sometimes throws "SIGSEGV: segmentation violation" failing deployment #241

Ark-kun opened this issue Nov 13, 2018 · 5 comments · Fixed by #1005
Assignees
@IronPan @jinchihe
Labels
area/kfctl help wanted The community is welcome to contribute. kind/bug priority/p0

Comments

@Ark-kun
Copy link
Contributor

Ark-kun commented Nov 13, 2018

https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/kubeflow_pipelines/232/presubmit-e2e-test-gce-minikube/129/

KSonnet issue: ksonnet/ksonnet#883
Resolved in https://github.com/ksonnet/ksonnet-lib/releases/tag/v0.1.12

deploy-ml-pipeline:	Configure ksonnet ...
deploy-ml-pipeline:	Configure ksonnet completed successfully
deploy-ml-pipeline:	Initialize a ksonnet APP ...
deploy-ml-pipeline:	level=info msg="Using context \"default\" from kubeconfig file \"/root/.kube/config\""
deploy-ml-pipeline:	level=info msg="Creating environment \"default\" with namespace \"default\", pointing to cluster at address \"https://10.96.0.1:443\""
deploy-ml-pipeline:	panic: runtime error: invalid memory address or nil pointer dereference
deploy-ml-pipeline:	[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x456452]
deploy-ml-pipeline:	goroutine 1 [running]:
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet.parsePaths(0xc4206d0800, 0x6, 0x1, 0xc)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet/paths.go:33 +0x1f1
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet.NewCatalog(0xc4206d0800, 0xc420874d90, 0x1, 0x1, 0x40, 0x0, 0x0)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet/catalog.go:76 +0x1de
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet.GenerateLib(0xc4213ce030, 0xe, 0x0, 0x0, 0x0)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/vendor/github.com/ksonnet/ksonnet-lib/ksonnet-gen/ksonnet/ksonnet.go:25 +0xce
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/pkg/util/kslib.astBuilder(0xc4213ce030, 0xe, 0xc420cde000, 0x3f0dc0, 0x3ffe00, 0x0, 0x0, 0x0)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/pkg/util/kslib/ksonnet.go:87 +0x40
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/pkg/util/kslib.Ksonnet(0xc420cde000, 0x3f0dc0, 0x3ffe00, 0x0, 0x0, 0x0)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/pkg/util/kslib/ksonnet.go:83 +0x3fd
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/pkg/lib.(*Manager).GenerateLibData(0xc4200a08c0, 0x1, 0x1dc69a0, 0x2a15f78)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/pkg/lib/lib.go:83 +0x8a
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/pkg/app.updateLibData(0x1dc69a0, 0x2a15f78, 0xc4214b6780, 0xf, 0xc421317380, 0x14, 0x1, 0xc421575288, 0x43b4ab, 0xc421660848, ...)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/pkg/app/app.go:128 +0xba
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/pkg/app.(*App010).AddEnvironment(0xc42000c380, 0x187146a, 0x7, 0xc4214b6780, 0xf, 0xc4204e9e00, 0x0, 0x1a4, 0x0)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/pkg/app/app010.go:62 +0x427
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/pkg/env.(*creator).Create(0xc42034f830, 0xc42034f830, 0x200)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/pkg/env/create.go:117 +0x979
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/pkg/env.Create(0x1dcb1a0, 0xc42000c380, 0xc420646bc0, 0x15, 0x187146a, 0x7, 0x187146a, 0x7, 0xc4214b6780, 0xf, ...)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/pkg/env/create.go:42 +0x157
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/pkg/appinit.(*initApp).Run(0xc420875980, 0x7fd8937ca000, 0x0)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/pkg/appinit/init.go:88 +0x452
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/pkg/appinit.Init(0x1dc69a0, 0x2a15f78, 0x7fff81075c7f, 0xf, 0xc42020d920, 0x10, 0x0, 0x0, 0xc4214b6780, 0xf, ...)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/pkg/appinit/init.go:33 +0x14a
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/pkg/actions.(*Init).Run(0xc42034e090, 0xc42034e090, 0x0)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/pkg/actions/init.go:106 +0x121
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/pkg/actions.RunInit(0xc4212945a0, 0xc420229cb0, 0xe)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/pkg/actions/init.go:37 +0x5c
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/pkg/clicmd.runAction(0xe, 0xc4212945a0, 0x188a651, 0x17)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/pkg/clicmd/actions.go:113 +0x66
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/pkg/clicmd.glob..func17(0x29d5da0, 0xc42075e410, 0x1, 0x1, 0x0, 0x0)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/pkg/clicmd/init.go:107 +0x743
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra.(*Command).execute(0x29d5da0, 0xc42075e3e0, 0x1, 0x1, 0x29d5da0, 0xc42075e3e0)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra/command.go:753 +0x468
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x29d96a0, 0x13, 0x18b9332, 0x30)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra/command.go:843 +0x30a
deploy-ml-pipeline:	github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra.(*Command).Execute(0x29d96a0, 0xc420040118, 0x0)
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/vendor/github.com/spf13/cobra/command.go:791 +0x2b
deploy-ml-pipeline:	main.main()
deploy-ml-pipeline:		/Users/bryan/go/src/github.com/ksonnet/ksonnet/cmd/ks/main.go:35 +0x85
deploy-ml-pipeline:	Initialized ksonnet APP completed successfully
@eLco eLco mentioned this issue Nov 14, 2018
Closed
@eLco
Copy link

eLco commented Nov 14, 2018

Maybe it is possible to add ksonnet flags like --api-spec for example to bootstraper image flags to have more freedom.

@vicaire vicaire assigned Ark-kun and unassigned vicaire Nov 15, 2018
@Ark-kun Ark-kun removed their assignment Nov 21, 2018
@jinchihe
Copy link
Member

jinchihe commented Mar 20, 2019 •
edited
Loading

I also met the problem today, and that can be reproduced 100%, the deployment has been blocked.

Initializing KSonnet app...
+ echo 'Initializing KSonnet app...'
+ ks init tf-serving-app
level=info msg="Using context \"kubeflow\" from kubeconfig file \"/root/.kube/config\""
level=info msg="Creating environment \"default\" with namespace \"default\", pointing to \"version:v1.12.4\" cluster at address \"
https://kubernetes.default
\""
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x456452]

From the ksonnet/ksonnet#883, seems the problem has been fixed in the ks v0.13.1. I tested, that's passed with the new ks. I think we need to upgrade ks in deployer component.

@jinchihe jinchihe mentioned this issue Mar 20, 2019
Merged
@vicaire
Copy link
Contributor

vicaire commented Mar 27, 2019

Assuming that this has been resolved since it has been idle for a long time. Please re-open if it is not the case.

@vicaire vicaire closed this as completed Mar 27, 2019
@jinchihe
Copy link
Member

@vicaire I met the problem recently, and block the following steps. I created one PR #1005 to fix this, but has not merged yet. Thanks.

@vicaire
Copy link
Contributor

vicaire commented Mar 27, 2019

Thanks @jinchihe. Apologies for closing this.

@vicaire vicaire reopened this Mar 27, 2019
@vicaire vicaire added help wanted The community is welcome to contribute. priority/p0 kind/bug area/kfctl labels Mar 27, 2019
Linchin pushed a commit to Linchin/pipelines that referenced this issue Apr 11, 2023
@rileyjbauer @k8s-ci-robot
Improve readability of priority/p1 and status/icebox labels (kubeflow…
e66a0e6 
…#241)

The mentioned labels are very close to the threshold that github uses to determine whether to use black or white foreground text.

By darkening them slightly, the foreground switches to white, and the labels become more readable.
magdalenakuhn17 pushed a commit to magdalenakuhn17/pipelines that referenced this issue Oct 22, 2023
@rakelkar @k8s-ci-robot
add cmpopts dependency (kubeflow#241)
e3af5c8 
* add cmpopts dependency

* Revert "add cmpopts dependency"

This reverts commit 388d12cc3aca81abad6e840c18a00d55b49841d2.

* dep ensure -add github.com/google/go-cmp/cmp/cmpopts@v0.3.0

* Revert "dep ensure -add github.com/google/go-cmp/cmp/cmpopts@v0.3.0"

This reverts commit 8b5e21e10d322bc20761b578a53c5d89bbd1cfa9.

* dep ensure -add github.com/google/go-cmp/cmp/cmpopts@6f77996f0c42f7b84e5a2b252227263f93432e9b
(use specific version to match knative)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@IronPan IronPan

@jinchihe jinchihe

Labels
area/kfctl help wanted The community is welcome to contribute. kind/bug priority/p0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

upgrade ks in deployer to fix one block issue jinchihe/pipelines
5 participants
@eLco @Ark-kun @IronPan @jinchihe @vicaire