Karpenter instance profile usage

[niallthomson]

What feature/behavior/change do you want?

The Karpenter integration creates a node role and associated instance profile that can be used with a provisioner. However, the name of the instance profile created is not explicitly documented.

Additionally, it would be great if this could be optionally or by default set as the default instance profile. This could be activated by a flag and pass in using the helm chart values:

aws:
  # -- The default instance profile to use when launching nodes on AWS
  defaultInstanceProfile: ""

Why do you want this feature?

Users need to figure this out for themselves, which makes getting Karpenter running more difficult.

[Skarlso]

Hi!

Thanks for the issue.

For the record, the document says:

It will create all the necessary prerequisites outlined in Karpenter's Getting Started section including installing Karpenter itself using Helm.

Which includes the created node role that the Getting Started is talking about. :)

But, alas, I do agree that it is useful to expose that setting to eksctl config too and is a relatively easy thing to do.

[niallthomson]

I agree that the documentation is accurate, but I'd just say its missing some details. For example, the IAM instance profile name (eksctl-KarpenterNodeInstanceProfile-$CLUSTER_NAME) is different from the Karpenter Getting Started page (KarpenterNodeInstanceProfile-${CLUSTER_NAME}).

These are minor things, overall the integration is fantastic it could just have some little quality of life enhancements :)

[Skarlso]

Ah, right. The problem there is, that we prefix everything with eksctl and that is definitely missing from the doc. :/ We need the eksctl prefix because of the IAM permissions that we have configured for resources that begin wth that prefix.

[niallthomson]

I'm happy to submit a PR that introduces a flag for eksctl that sets the default instance profile, for example:

karpenter:
   setDefaultInstanceProfile: true

[Skarlso]

No need for the set otherwise LGTM. Please don't forget to update the test which checks if the helm value is passed in correctly. Thanks! :)

[Skarlso]

@niallthomson also, thank you that would be very welcomed! :)

[afolarin]

For example, the IAM instance profile name (eksctl-KarpenterNodeInstanceProfile-$CLUSTER_NAME)

Came here to post precisely this.

[Skarlso]

@aclevername @cPu1 We talked about this, remember by any chance? I know we have policies in place to name things, but this is a deviation from the doc for Karpenter. I'm wondering if we could do anything about that, like not use the prefix, since we updated the minimum IAM requirements anyways....

Or, the custom field might solve that. WDYT?

[niallthomson]

I'm not sure you necessarily want to set the precedent of matching another projects installation docs, otherwise if those docs change its not been worth it anyway. My 2 cents would be to adding to the eksctl docs to clearly specify the IAM instance profile and role thats created.

Also, the Karpenter docs currently set aws.defaultInstanceProfile during the Helm chart installation. Should this new flag be enabled by default and documented as such? I could go either way, my WIP code has it off by default.

[afolarin]

I think it is sufficient (for now) to update the docs https://eksctl.io/usage/eksctl-karpenter/ to specify that

1. defaultInstanceProfile is not created during Helm install (superficially you might expect it to be following the Helm Get Started)
2. This is definitely threw me a bit:

It will create all the necessary prerequisites outlined in Karpenter's Getting Started section including installing Karpenter itself using Helm.

as it was not immediately obvious no instance profile would be created and without at e.g. spec.provider.instanceProfile (with the eksctl- prefixed profile) Karpenter would have no capacity to launch nodes.

[Skarlso]

@afolarin

Hi!

as it was not immediately obvious no instance profile would be created and without at e.g. spec.provider.instanceProfile (with the eksctl- prefixed profile) Karpenter would have no capacity to launch nodes.

I'm not following this one, sorry. Out of the blue, without any other necessary thing, you can install Karpetner right now, add a provisioner and it will work. What other thing did you expect to happen?

as it was not immediately obvious no instance profile would be created and without

Specifically this one ^^ It does create and use an instance profile.

[afolarin]

Out of the blue, without any other necessary thing, you can install Karpetner right now, add a provisioner and it will work.

yes....if you are aware a step is missing from the Get Started Karpenter docs.

What other thing did you expect to happen?

If the Karpenter Getting Started steps (where default instance profile is added) are followed, then one might expect the default instance profile to be set if "necessary prerequisites" are indeed provided. So without knowledge that instance profiles must be provided the workflow of setting up with eksctl Karpenter then adding an Provisioner (for example the example default Provisioner given in the Inflate deployment example) fails and you will only see that when you look at the logs

2022-02-04T22:24:55.322Z	ERROR	controller.provisioning	Could not launch node, launching instances, getting launch template configs, getting launch templates, neither spec.provider.instanceProfile nor --aws-default-instance-profile is specified	{"commit": "62c4546", "provisioner": "default"}