Stash CAKeyPair in a Kube Secret

[matoszz]

Once the Machines + Cluster have been created, the Cluster object should derive it's caKeyPair values from a kube secret

[deitch]

Hey @mateo1647 ,

It has been a little while, so refreshing on this (and really need to move it over from v1a1 to v1a3, but that is a whole separate issue).

If I remember it correctly, the CA key/cert are not actually part of the cluster-api spec (at least as far as v1a1), but just part of the ProviderSpec. CAPI actually doesn't care about it at all, since CAPI only goes so far as creating the cluster and nodes; how you bootstrap the nodes is a provider-specific implementation.

We use the CA cert (passed in the cluster part of provider spec, or created in the cluster actuator if blank) to pass in the userdata to all of the master nodes, as they will need it as part of bootstrap to launch masters on the cluster (as masters need to sign certs).

I don't object to stashing the CAKeyPair in a k8s secret, but what do we gain? Each master needs it before joining the cluster.

Granted, we could use kubeadm tokens, or fixed tokens, and ca bootstrapping, and that probably would be a good thing. Also much of it has changed with v1a2 and v1a3.

So, yes, no objection, but what is the benefit in v1a1?

[matoszz]

Benefit in v1a1? Little to none. I posted the issue while I was working through bootstrapping a cluster using a declarative Gitops model via Argo because I noticed the SSH keys and certs weren't referenced via kube secret. Even though I had to delete the packet nodes I think the old manifests / cluster are still visible in my setup: https://argocd.k8s.halyard.app/applications/packet-cluster?node=cluster.k8s.io%2FCluster%2Fcluster-api-provider-packet-system%2Fpacket-2rjup%2F0

In v2 and v3 it's still debatable as to "what you'd gain" - personally, I think its valuable because many folks (like myself) have used / implemented / are advocates for a non-ephemeral management cluster which controls other bootstrapped clusters declaratively and in that management cluster you don't want access to your other clusters to be readily available. I'd consider this a feature / nice to have vs. any sort of actual issue with the ProviderSpec implemented here.

[deitch]

declarative Gitops model via Argo

Yes, definitely! This is what we call "good stuff" :-)

I think its valuable because many folks (like myself) have used / implemented / are advocates for a non-ephemeral management cluster which controls other bootstrapped clusters declaratively and in that management cluster you don't want access to your other clusters to be readily available.

Now I am confused. I first thought you were suggesting it be a k8s secret in the management cluster (FWIW, gardener and few others do this, although the original CAPI view was, IIRC, that the management cluster is temporary and you pivot off of it).

Even with a persistent management cluster, I still don't understand:

• putting it as a k8s secret in the deployed (i.e. non-management) cluster doesn't help much, since, as above, it is needed for new master nodes that have not yet joined the cluster.
• putting it as a k8s secret in the management cluster doesn't help much, since it is in the cluster config... oh, wait, it is not, as the config is "", which means, "generate a new one".

Currently the flow is the following (abridged):

1. Create config files, leave CA in providerSpec blank
2. Bootstrap cluster (ephemeral or persistent) sees it is blank, generates key/cert; or takes the one provided in the config
3. Bootstrap cluster adds it to the config in its cluster, and adds it to the userdata for master nodes passed to Packet API
4. Master node gets the userdata, uses the info to generate certs, etc.
5. Data lingers inside the bootstrap cluster (as long as it exists) as part of the config for future master nodes deployed

What would the future flow look like?

[matoszz]

Apologies for any confusion - what I'm suggesting is using k8s secrets in the management cluster, not in the "workload" cluster.

Usage of k8s secrets in the "workload" or provisioned cluster would be a part of down-stream automation and shouldn't be in scope of this type of provider, IMO.

[deitch]

Apologies for any confusion

None necessary! :-)

is using k8s secrets in the management cluster

Instead of storing it in the updated config? Sure, that makes sense, even as is.

[matoszz]

Correct! Instead of storing it in the updated config. Today, in the flow, you create the cluster with empty fields in the cluster and machine specs:

(cluster spec)

    providerSpec:
      value:
        apiVersion: "packetprovider/v1alpha1"
        kind: "PacketClusterProviderSpec"
        projectID: 
        caKeyPair:
          cert: ""
          key: ""

(machine spec)

    providerSpec:
      value:
        apiVersion: "packetprovider/v1alpha1"
        kind: "PacketMachineProviderSpec"
        roles:
        - Node
        OS: ubuntu_18_04
        facility:
        - ewr1
        machineType: c1.small.x86
        billingCycle: hourly
        projectID: 
        sshKeys:

The cert + key, and sshKeys fields are updated post-creation. If you look at my live manifest (you have to scroll down slightly) the cert + key and sshKeys are visible in plain text in the updated manifests for the created clusters.

It would be great if you simply created a few additional secrets within the provider-components.yaml for both the cert + key, and SSH keys, and then subsequently referenced/updated those throughout the provisioning. It should look close to what you're already doing with the cluster-api-provider-packet-credentials:

apiVersion: v1
kind: Secret
metadata:
  name: cluster-api-provider-packet-credentials
  namespace: cluster-api-provider-packet-system
stringData:
  apiKey: 
  projectID: 
type: Opaque

EX:(for the cluster spec):

apiVersion: v1
kind: Secret
metadata:
  name: cluster-api-provider-packet-generated-certificates
  namespace: cluster-api-provider-packet-system
stringData:
  cert: 
  key: 
type: Opaque

Would need to modify the cluster spec here

EX (for the machine spec):

apiVersion: v1
kind: Secret
metadata:
  name: cluster-api-provider-packet-machine-keys
  namespace: cluster-api-provider-packet-system
stringData:
  sshKeys: 
type: Opaque

And modify the machine spec here

[deitch]

That is about the path I am going down. It is a bit of a pain, in that you need to actively manage the secrets, so you need to use the k8s api, which is fine. The cluster provider calls pass in the clusterGetter but not a generic client to use to get a secretGetter, but it is doable.

[deitch]

Why ssh keys? These are all public keys anyways.

[deitch]

@mateo1647 please take a look at #12 for the design.

[matoszz]

@deitch Hey avi - apologies for the delay in response; I was thinking of using a secret for the SSH keys in case there were multiple. A secret would be useful to store an array of keys. If you can iterate on multiple keys within the spec, that would also work, but might be more difficult to manage.

[deitch]

Multiple keys. That makes sense. I would sooner change the spec, since it is in our control. Open a separate issue for it, "support passing multiple ssh keys to a node", please.

It will have to wait, though; too much is creaking because of v1alpha1; it just is time to move to v1a3.

[matoszz]

v1a3 > everything! :)