Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Controller crash with SecretTemplate that references other generated passwords #90

Open
petewall opened this issue Jul 9, 2022 · 3 comments
Open

Controller crash with SecretTemplate that references other generated passwords #90

petewall opened this issue Jul 9, 2022 · 3 comments
Labels
bug This issue describes a defect or unexpected behavior carvel-accepted This issue should be considered for future work and that the triage process has been completed documentation This issue indicates a change to the docs should be considered priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.

Comments

@petewall
Copy link

petewall commented Jul 9, 2022

What steps did you take:
I'm trying to create a secret that references other generated secrets. The first secrets are created quickly, but the controller pod crashes when trying to resolve the fourth.

What happened:
The controller pod crashed

What did you expect:
The fourth secret to resolve, using data generated from the previous three

Anything else you would like to add:
Here are the inputs that I was using:

---
apiVersion: secretgen.k14s.io/v1alpha1
kind: SSHKey
metadata:
  name: worker-key
  namespace: concourse
spec: {}
---
apiVersion: secretgen.k14s.io/v1alpha1
kind: SSHKey
metadata:
  name: tsa-host-key
  namespace: concourse
spec: {}
---
apiVersion: secretgen.k14s.io/v1alpha1
kind: RSAKey
metadata:
  name: session-signing-key
  namespace: concourse
spec: {}
---
# Source: concourse/templates/web-secrets.yaml
apiVersion: secretgen.carvel.dev/v1alpha1
kind: SecretTemplate
metadata:
  name: concourse-web
  namespace: concourse
spec:
  inputResources:
  - name: session-signing-key
    ref:
      apiVersion: v1
      kind: Secret
      name: session-signing-key
  - name: tsa-host-key
    ref:
      apiVersion: v1
      kind: Secret
      name: tsa-host-key
  - name: worker-key
    ref:
      apiVersion: v1
      kind: Secret
      name: worker-key
template:
  data:
    host-key: $(.tsa-host-key.data.privateKey)
    session-signing-key: $(.session-signing-key.data.privateKey)
    worker-key-pub: $(.worker-key.data.authorizedKey)
    local-users: "dGVzdDp0ZXN0"

The list of secrets:

$ kubectl get secret -n concourse
NAME                  TYPE                     DATA   AGE
session-signing-key   Opaque                   2      15m
worker-key            kubernetes.io/ssh-auth   2      15m
tsa-host-key          kubernetes.io/ssh-auth   2      15m
$ kubectl get secrettemplate -n concourse
NAME            DESCRIPTION   AGE
concourse-web   Reconciling   15m

And here are the controller pod logs:

pwall@Personal-MBP:~/src/petewall/cluster/deployments/concourse $ kubectl logs -n secretgen-controller secretgen-controller-667f6f9d67-zlt5m
{"level":"info","ts":1657380006.8805838,"logger":"sg.entrypoint","msg":"secretgen-controller","version":"0.10.3"}
{"level":"info","ts":1657380006.8806255,"logger":"sg.entrypoint","msg":"setting up manager"}
{"level":"info","ts":1657380007.2324445,"logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1657380007.2326853,"logger":"sg.entrypoint","msg":"setting up controllers"}
{"level":"info","ts":1657380007.2337837,"logger":"sg.entrypoint","msg":"starting manager"}
{"level":"info","ts":1657380007.2339766,"msg":"Starting metrics server","path":"/metrics"}
{"level":"info","ts":1657380007.2340772,"logger":"controller.sg-cert","msg":"Starting EventSource","source":"kind source: *v1alpha1.Certificate"}
{"level":"info","ts":1657380007.234097,"logger":"controller.sg-cert","msg":"Starting Controller"}
{"level":"info","ts":1657380007.2342255,"logger":"controller.sg-password","msg":"Starting EventSource","source":"kind source: *v1alpha1.Password"}
{"level":"info","ts":1657380007.2342393,"logger":"controller.sg-password","msg":"Starting Controller"}
{"level":"info","ts":1657380007.2342494,"logger":"controller.sg-secret","msg":"Starting EventSource","source":"kind source: *v1.Secret"}
{"level":"info","ts":1657380007.2342887,"logger":"controller.sg-secret","msg":"Starting EventSource","source":"kind source: *v1alpha1.SecretExport"}
{"level":"info","ts":1657380007.2342992,"logger":"controller.sg-secret","msg":"Starting EventSource","source":"kind source: *v1.Namespace"}
{"level":"info","ts":1657380007.2343037,"logger":"controller.sg-secret","msg":"Starting Controller"}
{"level":"info","ts":1657380007.2344172,"logger":"controller.sg-rsakey","msg":"Starting EventSource","source":"kind source: *v1alpha1.RSAKey"}
{"level":"info","ts":1657380007.2344322,"logger":"controller.sg-rsakey","msg":"Starting Controller"}
{"level":"info","ts":1657380007.2345197,"logger":"controller.sg-sshkey","msg":"Starting EventSource","source":"kind source: *v1alpha1.SSHKey"}
{"level":"info","ts":1657380007.2345417,"logger":"controller.sg-sshkey","msg":"Starting Controller"}
{"level":"info","ts":1657380007.234648,"logger":"controller.sg-template","msg":"Starting EventSource","source":"kind source: *v1.Secret"}
{"level":"info","ts":1657380007.2346628,"logger":"controller.sg-template","msg":"Starting EventSource","source":"kind source: *v1.Secret"}
{"level":"info","ts":1657380007.23467,"logger":"controller.sg-template","msg":"Starting EventSource","source":"kind source: *v1alpha1.SecretTemplate"}
{"level":"info","ts":1657380007.2346745,"logger":"controller.sg-template","msg":"Starting Controller"}
{"level":"info","ts":1657380007.2347345,"logger":"controller.sg-secexp","msg":"Starting EventSource","source":"kind source: *v1alpha1.SecretExport"}
{"level":"info","ts":1657380007.234757,"logger":"controller.sg-secexp","msg":"Starting EventSource","source":"kind source: *v1.Secret"}
{"level":"info","ts":1657380007.2347615,"logger":"controller.sg-secexp","msg":"Starting Controller"}
{"level":"info","ts":1657380007.2348795,"logger":"controller.sg-secimp","msg":"Starting EventSource","source":"kind source: *v1alpha1.SecretImport"}
{"level":"info","ts":1657380007.2348936,"logger":"controller.sg-secimp","msg":"Starting EventSource","source":"kind source: *v1.Secret"}
{"level":"info","ts":1657380007.234897,"logger":"controller.sg-secimp","msg":"Starting EventSource","source":"kind source: *v1alpha1.SecretExport"}
{"level":"info","ts":1657380007.2349007,"logger":"controller.sg-secimp","msg":"Starting EventSource","source":"kind source: *v1.Namespace"}
{"level":"info","ts":1657380007.2349072,"logger":"controller.sg-secimp","msg":"Starting Controller"}
{"level":"info","ts":1657380007.3345132,"logger":"controller.sg-password","msg":"Starting workers","worker count":1}
{"level":"info","ts":1657380007.3346202,"logger":"controller.sg-cert","msg":"Starting workers","worker count":1}
{"level":"info","ts":1657380007.334624,"logger":"controller.sg-rsakey","msg":"Starting workers","worker count":1}
{"level":"info","ts":1657380007.3347082,"logger":"controller.sg-sshkey","msg":"Starting workers","worker count":1}
{"level":"info","ts":1657380007.3352313,"logger":"controller.sg-secret","msg":"Starting workers","worker count":1}
{"level":"info","ts":1657380007.3354335,"logger":"sg.secret","msg":"Reconciling","request":"network/ddclient"}
{"level":"info","ts":1657380007.3355248,"logger":"controller.sg-template","msg":"Starting workers","worker count":1}
{"level":"info","ts":1657380007.3355968,"logger":"controller.sg-secexp","msg":"Starting workers","worker count":1}
{"level":"info","ts":1657380007.3356595,"logger":"sg.secret","msg":"Reconciling","request":"cert-manager/cert-manager-webhook-ca"}
{"level":"info","ts":1657380007.3357017,"logger":"controller.sg-secimp","msg":"Starting workers","worker count":1}
{"level":"info","ts":1657380007.3356607,"logger":"sg.template","msg":"reconciling","request":"concourse/concourse-web"}
{"level":"info","ts":1657380007.335777,"logger":"sg.secexp","msg":"Reconciling","request":"cert-manager/letsencrypt-prod"}
{"level":"info","ts":1657380007.3357766,"logger":"sg.secret","msg":"Reconciling","request":"cert-manager/letsencrypt-prod"}
{"level":"info","ts":1657380007.3359263,"logger":"sg.secexp","msg":"Reconciling","request":"ghost/ghost-tls"}
{"level":"info","ts":1657380007.3359742,"logger":"sg.secexp","msg":"Reconciling","request":"concourse/session-signing-key"}
{"level":"info","ts":1657380007.3359776,"logger":"sg.secret","msg":"Reconciling","request":"ghost/ghost-tls"}
{"level":"info","ts":1657380007.3360217,"logger":"sg.secexp","msg":"Reconciling","request":"concourse/tsa-host-key"}
{"level":"info","ts":1657380007.3360746,"logger":"sg.secexp","msg":"Reconciling","request":"cert-manager/letsencrypt-staging"}
{"level":"info","ts":1657380007.336126,"logger":"sg.secret","msg":"Reconciling","request":"concourse/session-signing-key"}
{"level":"info","ts":1657380007.3361483,"logger":"sg.secexp","msg":"Reconciling","request":"concourse/worker-key"}
{"level":"info","ts":1657380007.3362415,"logger":"sg.secret","msg":"Reconciling","request":"concourse/tsa-host-key"}
{"level":"info","ts":1657380007.3363152,"logger":"sg.secret","msg":"Reconciling","request":"cert-manager/letsencrypt-staging"}
{"level":"info","ts":1657380007.336336,"logger":"sg.secexp","msg":"Reconciling","request":"network/ddclient"}
{"level":"info","ts":1657380007.336398,"logger":"sg.secret","msg":"Reconciling","request":"concourse/worker-key"}
{"level":"info","ts":1657380007.3364034,"logger":"sg.secexp","msg":"Reconciling","request":"cert-manager/cert-manager-webhook-ca"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x133637d]

goroutine 392 [running]:
github.com/vmware-tanzu/carvel-secretgen-controller/pkg/generator.evaluateTemplate(_, _)
        github.com/vmware-tanzu/carvel-secretgen-controller/pkg/generator/secret_template_reconciler.go:280 +0x3d
github.com/vmware-tanzu/carvel-secretgen-controller/pkg/generator.(*SecretTemplateReconciler).reconcile(0xc0005bea00, {0x18f4ec8, 0xc00057a570}, 0xc000497380)
        github.com/vmware-tanzu/carvel-secretgen-controller/pkg/generator/secret_template_reconciler.go:131 +0x8d
github.com/vmware-tanzu/carvel-secretgen-controller/pkg/generator.(*SecretTemplateReconciler).Reconcile(0xc0005bea00, {0x18f4ec8, 0xc00057a570}, {{{0xc0005a1d50?, 0x15d69c0?}, {0xc0005a1d30?, 0xc000046800?}}})
        github.com/vmware-tanzu/carvel-secretgen-controller/pkg/generator/secret_template_reconciler.go:121 +0x36c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc000433ae0, {0x18f4ec8, 0xc00057a4e0}, {{{0xc0005a1d50?, 0x15d69c0?}, {0xc0005a1d30?, 0xc0006bc5c0?}}})
        sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:114 +0x222
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000433ae0, {0x18f4e20, 0xc000046040}, {0x155fa40?, 0xc000490180?})
        sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:311 +0x2e9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000433ae0, {0x18f4e20, 0xc000046040})
        sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:266 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:227 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
        sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:223 +0x309

Environment:

  • secretgen-controller version (execute kubectl get deployment -n secretgen-controller secretgen-controller -o yaml and the annotation is kbld.k14s.io/images):
    kbld.k14s.io/images: |
      - origins:
        - local:
            path: /home/runner/work/carvel-secretgen-controller/carvel-secretgen-controller
        - git:
            dirty: true
            remoteURL: https://github.com/vmware-tanzu/carvel-secretgen-controller
            sha: 7cf938231129673564646d851015d08630307efe
            tags:
            - v0.10.3
        url: ghcr.io/vmware-tanzu/carvel-secretgen-controller@sha256:00466d6beb98fdd8aed61642013ea0ba538bb496e84745c8c2e1871fdc54b1a9
  • Kubernetes version (use kubectl version)
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.0-2+59bbb3530b6769", GitCommit:"59bbb3530b6769e4935a05ac0e13c9910c79253e", GitTreeState:"clean", BuildDate:"2022-05-13T06:41:13Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.
@petewall petewall added bug This issue describes a defect or unexpected behavior carvel-triage This issue has not yet been reviewed for validity labels Jul 9, 2022
@petewall
Copy link
Author

petewall commented Jul 9, 2022

(I'll be changing local-users, so no worries about leaking the default test:test)

@petewall
Copy link
Author

petewall commented Jul 9, 2022

OK, figured out the crash. I mis-indented the SecretTemplate definition so template was parallel to spec and not inside it. Indenting that prevented the crash. Still probably shouldn't crash, so I'm going to leave this issue open.

Another maybe issue (let me know if you'd like me to open a new one), the key I needed to use to index the SSH and RSA secrets were not the same as is in the docs:
https://github.com/vmware-tanzu/carvel-secretgen-controller/blob/develop/docs/ssh_key.md
says I can use privateKey, but when it's generated, the secret uses ssh-privatekey.

@joe-kimmel-vmw
Copy link
Contributor

Thanks @petewall ! I agree it sounds like we should update our docs, and likely there's a better path than crashing to handle the error case you're describing - appreciate your feedback!

@joe-kimmel-vmw joe-kimmel-vmw added documentation This issue indicates a change to the docs should be considered carvel-accepted This issue should be considered for future work and that the triage process has been completed priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. and removed carvel-triage This issue has not yet been reviewed for validity labels Jul 12, 2022
@aaronshurley aaronshurley moved this to To Triage in Carvel Jul 25, 2022
@github-project-automation github-project-automation bot moved this to To Triage in Carvel Feb 14, 2023
@neil-hickey neil-hickey moved this from To Triage to Unprioritized in Carvel Feb 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug This issue describes a defect or unexpected behavior carvel-accepted This issue should be considered for future work and that the triage process has been completed documentation This issue indicates a change to the docs should be considered priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
Carvel
Status: Unprioritized
Milestone
No milestone
Development

No branches or pull requests
2 participants
@petewall @joe-kimmel-vmw