TLS: Support for user provided domains

[cjellick]

This is NOT for on-acorn.io domains

This is for the case where user supplies their own cluster domain, or even more likely creates an app with the following publish flag:

-p myapp.mydomain.com:<app deatils>

Supporting this means installing and managing cert-manager and auto-creating the Certificate CRs if the app wants TLS

[cjellick]

We arent going to install cert-manager.
User expresses they want TLS. If we find the Certificate CR, then we program it

[andyjeffries]

What's the proposal then for how cert-manager should be installed? Just manually in to the cluster?

I don't know your design goals, but this was feeling to me like "bring me a cluster, give me access, deploy your apps with easy commands". Darren just said in the Webinar:

This feels like "No knowledge of Kubernetes YAML required, FOOTNOTE: unless you want to use your own domain name and use TLS with it". Hopefully it's reconsidered, having Acorn install cert-manager for you unless already installed/available.

[andyjeffries]

Ahhh, I took the wording of the issue to be almost like a "we won't be doing this", but I guess it's just pending, based on:

In the future, Acorn will provide built-in mechanisms to automatically provide certificates for each endpoint. Today, adding a certificate must follow the manual approach.

-- https://docs.acorn.io/running/certificates

[vincent99]

@andyjeffries There's a balance we're trying to find the right line on between batteries-included (you just want a cert, I don't know or care about CertManager) and tolerable-for-cluster-operator-to-install. The more stuff we add to someone's cluster the more they worry about trying it, compatibility between versions issues, should it be cleaned up on uninstall, etc.

CertManager does a lot of stuff and we don't even actually need so one option we're considering is not requiring it at all by doing ACME dances in the controller instead. That's straightforward enough for single name http-01 challenges.

The part CertManager does do that's really useful is taking creds and talking to all the provider APIs for dns-01/wildcards. But we can maybe do a one-time setup _acme_challenge.yourdomain.com CNAME [your-cluster].on-acorn.io yourself step, and then with that we can manage a *.yourdomain.com wildcard without having any creds or provider-specific implementation.

[andyjeffries]

Not requiring (or self-embedding the relevant parts of) cert-manager would be cool.

I definitely lean to the "I have a cluster, but want acorn to manage it as a kind-of/lightweight PAAS for me" camp.

I know about Kubernetes (as former CTO of Civo.com), but I don't want to deal with the minutia of it in my new job as it's not necessary for me.

[cjellick]

Right now, our implementation plans in this area are as follows:

1. Supply a wildcard cert for the on-acorn.io domain associated with the cluster. This will not use cert-manager. It will just use existing go-libraries in our acorn controller
2. For easy cert-manager integration (for those that already have it), allow for the propogation of annotations from the acorn containers to the ingress we create. This solves a lot of use cases, one of which is automatic cert creation through cert-manager: https://cert-manager.io/docs/usage/ingress/#supported-annotations
3. Revisit that last paragraph of Vince's and other possible solutions to see what we want to do.

[denist-huma]

Hello there.
Acorn is not finding a TLS secret!?
I just tried several times it is not working.

Acorn will automatically inspect each certificate in the Acorn namespace for one that can be used with my-app.example.com. If
no TLS secret is found with that FQDN, it will be exposed on HTTP only.

acorn install --cluster-domain sbx.huma.com --ingress-class-name nginx --acorn-dns disabled
acorn run -n todo-app-acorn --publish todo-app-acorn-1.sbx.huma.com:server .

I have a TLS secrets that external secrets operator created from GCP SM.

$ k get secret sbx-huma-com-tls-secret -n todo-app-acorn-cd495f1f-791
NAME                      TYPE                DATA   AGE
sbx-huma-com-tls-secret   kubernetes.io/tls   2      10m

1. there is no .spec.tls section in the ingress
2. I use external-dns so I need to put an annotation. "external-dns.alpha.kubernetes.io/hostname": "todo-app-acorn-1.sbx.huma.com" annotation

$ k get ing -n todo-app-acorn-cd495f1f-791 -o yaml
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      acorn.io/targets: '{"server.todo-app-acorn.sbx.huma.com":{"port":8090,"service":"server"},"todo-app-acorn-1.sbx.huma.com":{"port":8090,"service":"server"}}'
      apply.acorn.io/applied: H4sIAAAAAAAA/8RTwW7bOhD8lYc9S4rkZ7sRgR7anooCRQ+5VTmsyLUkhCIJcuU6EPjvBaUkjW0kPfa2JmeGs+PRDCMxKmQEMQMaYxl5sCYsP6X1phjsDaPviAMImBsI5I/kC7bK5uhcvqJCeyr6acRC2rEBMTfgrOcGxG1Zl9nKGiQ1INIMGaBz+rF4ecL+MuTz7vgAAgbD5A3qP7fHKvvv22DUx0/OfTWB0cg3NQyOBALO/b0HDg5lYrwLDFObS2uYTgwCIGYgPS1R3Q0jBcbRgTCT1hlobEmfB5h8vOnrEnTl5xkwosGOVBLxE72+eUr3+Y31L7pepcfQgwCJNW1UXWO5rwllTWpf3R7KUm52/2+Uwr0qy3q726YtLxVfGzxfJZdqW+8O1SH/UFeJGhzJlMJgOk8hfNEYwvdVznSDOUEGftIUQPycobeBryWrs15BBj2zS5oOuV+JLcoHMiodPqWQxkvbqYzL+TS25NdWxhizRQgE3MA63j26xPvh6TCcIN7HmL14+3vx/4nB+xQ1I09L5bRF9Rl1+kA8iDnG+DsAAP//e5YZReMDAAA
      apply.acorn.io/owner-gvk: internal.acorn.io/v1, Kind=AppInstance
      apply.acorn.io/owner-name: todo-app-acorn
      apply.acorn.io/owner-namespace: acorn
      apply.acorn.io/owner-sub-context: ""
    creationTimestamp: "2022-08-22T20:05:18Z"
    generation: 1
    labels:
      acorn.io/app-name: todo-app-acorn
      acorn.io/app-namespace: acorn
      acorn.io/managed: "true"
      acorn.io/service-name: server
      apply.acorn.io/hash: ca9e2d99a069eac9ed618f00c2532dda6d009454
    name: server
    namespace: todo-app-acorn-cd495f1f-791
    resourceVersion: "21480806"
    uid: 42579c35-6344-4d36-86f8-bc803ae75259
  spec:
    ingressClassName: nginx
    rules:
    - host: todo-app-acorn-1.sbx.huma.com
      http:
        paths:
        - backend:
            service:
              name: server
              port:
                number: 8090
          path: /
          pathType: Prefix
    - host: server.todo-app-acorn.sbx.huma.com
      http:
        paths:
        - backend:
            service:
              name: server
              port:
                number: 8090
          path: /
          pathType: Prefix

[cjellick]

I have a TLS secrets that external secrets operator created from GCP SM.
$ k get secret sbx-huma-com-tls-secret -n todo-app-acorn-cd495f1f-791

It looks like you are creating the secret in the generated namepsace that acorn creates. For this auot-detect feature to work, you want to add the secret to the user-facing namespace (acorn by default).

I use external-dns so I need to put an annotation. "external-dns.alpha.kubernetes.io/hostname": "todo-app-acorn-1.sbx.huma.com" annotation

I didn't realize external-dns relied on an annotation being on the ingress. I am currently working a feature to support the addition of arbitrary user-defined labels and annotations to be added to the resources created by acorn. Should be out in the next week or two.

[denist-huma]

I didn't realize external-dns relied on an annotation being on the ingress. I am currently working a feature to support the addition of arbitrary user-defined labels and annotations to be added to the resources created by acorn. Should be out in the next week or two.

Here is link to its docs
Annotate the Service with your desired external DNS name. Make sure to change example.org to your domain.

kubectl annotate service nginx "external-dns.alpha.kubernetes.io/hostname=nginx.example.org."

It looks like you are creating the secret in the generated namepsace that acorn creates. For this auot-detect feature to work, you want to add the secret to the user-facing namespace (acorn by default).

IMHO not intuitive at all. When I feed my TLS secrets to my operator, I create secrets in each namespace, because this allows to cover tenant per namespace case.

[cjellick]

Annotate the Service with your desired external DNS name. Make sure to change example.org to your domain.

Roger that. Hopefully when the annotation feature lands, it will meet your needs.

IMHO not intuitive at all. When I feed my TLS secrets to my operator, I create secrets in each namespace, because this allows to cover tenant per namespace case.

I hear you. That just hits a chicken-and-egg problem of adding the secrets after acorn has created and deployed everything. We have another feature in the queue for being to tell acorn to create its k8s resources in a pre-defined/pre-created namespace rather than a generated one. When that feature lands, you'd be able to drop the tls secret into namespace X and then deploy deploy there.

That said, as the feature stands now, I think you can still might be able to achieve what you desire. You can have a user-facing "acorn" namespace per tenant, rather than just using the default acorn namespace. You'd need setup some RBAC to ensure each tenant can only create acorn Apps in their namespace and no others.