create a local NVMe RAID-0

[FernandoMiguel]

What I'd like:
AL now supports creating RAID of instance store volumes
That work was done in this PR
awslabs/amazon-eks-ami#1171

Could bottlerocket implement the same or similar approach so that EKS nodes benefit from that feature?

[arnaldo2792]

Hey @FernandoMiguel, could you please have a look at this discussion? There are instructions there to create a RAID-0 array for containerd, using bootstrap containers. Will this be sufficient to support your use case?

[FernandoMiguel]

@arnaldo2792 i rather prefer the native way of al2 to having to modify bottlerocket boot myself with no future garantes it will work.
Hence looking for a similar approach to be implemented into bottlerocket so we can take benefit of instance store disks.

[stevehipwell]

@arnaldo2792 I think having an OCI image able to run arbitrary scripts, as requested in #2010, would make it easier to manually implement this as a short term solution. But ideally an API configuration to enable this would be my preference.

CC @bwagner5

[zmrow]

Thanks for bringing this to our attention @stevehipwell and @FernandoMiguel ! We aren't currently working on this, but will follow up here if/when we start in on it.

[denraf]

The problem with the boot container is that karpenter is not aware of the ephemeral ssd storage on an instance type like c6id.

[bryantbiggs]

The problem with the boot container is that karpenter is not aware of the ephemeral ssd storage on an instance type like c6id.

Sure it is, you can query that from the AWS EC2 API

[denraf]

The problem with the boot container is that karpenter is not aware of the ephemeral ssd storage on an instance type like c6id.

Sure it is, you can query that from the AWS EC2 API

Unfortunately not.
E.g. c6id.2xlarge
https://karpenter.sh/preview/concepts/instance-types/#c6id2xlarge
ephemeral-storage: 17G

Draft PR to resolve it for AL2: aws/karpenter-provider-aws#3827
In that PR it is stated work is needed in bottlerocket to support that.
Which brings us back to this issue.

[bryantbiggs]

I think you are mixing "native support in Karpenter" and "Karpenter is not aware of the ephemeral ssd storage"

You can enable this today with Karpenter, its just not as smooth as if its handled natively - https://github.com/awslabs/data-on-eks/blob/8a95736c1e7c1823c3c5944ffc469857db8ec6c9/analytics/terraform/spark-k8s-operator/karpenter-provisioners/spark-compute-optimized-provisioner.yaml#L74-L78

[denraf]

That's for AL2, not for bottlerocket.

[bryantbiggs]

I believe this should be resolved on the next release with bottlerocket-os/bottlerocket-core-kit#15

[stevehipwell]

Can this issue be closed if it's been released?

[yeazelm]

As of Bottlerocket 1.22.0, users can specify bootstrap commands to format ephemeral disks on first boot and mount several key directories on that newly created filesystem.

For k8s variants, this involves passing the following user-data:

[settings.bootstrap-commands.000-format-ephemeral-storage]
commands = [[ "apiclient", "ephemeral-storage", "init"],
["apiclient", "ephemeral-storage", "bind", "--dirs", "/var/lib/containerd", "/var/lib/kubelet", "/var/log/pods"]]
essential = true
mode = "always"

And for ECS variants, this involves passing the following user-data:

[settings.bootstrap-commands.000-format-ephemeral-storage]
commands = [[ "apiclient", "ephemeral-storage", "init"],
["apiclient", "ephemeral-storage", "bind", "--dirs", "/var/lib/containerd", "/var/lib/docker", "/var/log/ecs"]]
essential = true
mode = "always"

This is safe to pass to instances that have zero, one, or many local ephemeral disks and will format the local storage if needed, then mount that storage, and finally bind mount the provided directories on this storage. If there are no local disks, it will safely exit without doing anything to the existing mounts.

[yeazelm]

https://bottlerocket.dev/en/os/1.22.x/concepts/bootstrap-commands/#configure-ephemeral-disks documents how to use bootstrap commands to create a local NVMe RAID0 disk. Closing this ticket now that the docs are available.