allow adding to environment of pods in charts or autotune GOMAXPROCS

[juliantaylor]

The charts should allow adding values to the environment of each container.

This would be useful to tune go runtime parameters like GOMAXPROCS or its memory settings.
In particular GOMAXPROCS tuning is necessary in kubernetes deployments as golang defaults to the number of available host cores,not the available resources of the container.
On hardware with high cpu counts this leads to golang starting a lot of threads leading to increased memory usage, management overheads and latencies and potential issues when cpu frequency scaling is enabled as each thread receives little load.

There are go modules to automatically configure values e.g. https://github.com/uber-go/automaxprocs which could be an alternative.

Being able to set environment variables of each container in a pod via helm chart values. One setting for all containers of a pod may be enough though.

[nixpanic]

Making this optional with a parameter would be nice. The example at https://pkg.go.dev/go.uber.org/automaxprocs@v1.6.0/maxprocs suggests that it is possible to call the Set() method only when it needs to be enabled.

@juliantaylor, is this something you want to take on?

[juliantaylor]

I could provide an MR to allow setting environment variables of the pods in the chart, but adding autotuning may be a bit harder for me as I am not that familiar with golang. What would be best file location to start look at adding the automaxprocs?

I also overlooked that many images of the chart are external kubernetes-csi images which have the same problem, so I'll need to go file issues there too.
An advantage of just updating the charts to set the variable would be that these external components could be adjusted as well without waiting for updates on their side.

But the memory usage of the actually ceph-csi components the pods still reduces a good 30-50% (~300mb to ~200mb) with GOMAXPROCS=2 on our 80-256 thread machines. cpu usage also reduces a good bit but that is already very low to begin with, so less significant.

[nixpanic]

The reduction in resource consumption is pretty impressive! I am happy to do the Go coding for you, also for the Kubernetes CSI components. If you have similar results for the other containers, please report issues in their projects too. Feel free to leave links to those issues here, and I'll take a look at them as well.

My preference would be to have a commandline parameter -automaxprocs which can be used to enable the feature. By default we would not enable it (yet), so the parameter should then be passed in the Helm charts or other deployment mechanisms. Once it is stable, we can enable the option by default (and keep -automaxprocs=false to disable it).

[juliantaylor]

just for reference, prometheus for example uses automaxprocs by default since 3.0, optional before.
https://github.com/prometheus/prometheus/blob/main/cmd/prometheus/main.go#L787

https://github.com/prometheus/prometheus/releases/tag/v3.0.0

[nixpanic]

I haven't tested it yet, but there is a build with a new -automaxprocs commandline parameter at

quay.io/nixpanic/cephcsi:automaxprocs

The iamge is built from my automaxprocs branch.

[juliantaylor]

thanks, ill try to test it in the next few days

I have filed an issue for the csi node-driver-registrar which I think is run in the rbdplugin daemonset and thus relatively worthwhile to adjust kubernetes-csi/node-driver-registrar#494

the other csi components I'm not sure about, they only run in the provisioner which is a deployment but the gains being around 50% relative but absolute still less than 20-30mb per comtainer may be a little low to be worthwhile.

[juliantaylor]

An issue with the autoconfiguring is that the chart overrides /sys with the hosts /sys for the provisioner pods csi-rbdplugin container so the pod actually cannot see its cpu quotas and automaxprocs will not work.

But for the other containers its works, but note the chart also does not allow adding the automaxprocs parameter via values.

I would also recommend to enable logging for the automaxprocs so you can see what is happening.

[nixpanic]

An issue with the autoconfiguring is that the chart overrides /sys with the hosts /sys for the provisioner pods csi-rbdplugin container so the pod actually cannot see its cpu quotas and automaxprocs will not work.

Ah, of course! I was already wondering why I didn't see a noticeable different.

I would also recommend to enable logging for the automaxprocs so you can see what is happening.

For Ceph-CSI we really would need to set the GOMAXPROCS environment variable, I guess. Unless we can specify a more appropriate /sys subdirectory.

[nixpanic]

An issue with the autoconfiguring is that the chart overrides /sys with the hosts /sys for the provisioner pods csi-rbdplugin container so the pod actually cannot see its cpu quotas and automaxprocs will not work.

Ah, of course! I was already wondering why I didn't see a noticeable different.

I am actually not sure this is correct. Ceph-CSI does modify the PID-limit for its node-plugin, but it does so after calling maxprocs.Set(). In an updated quay.io/nixpanic/cephcsi:automaxprocs container-image, the logging for maxprocs.Set() should be included now. That should make it easier to verify what is getting configured.