Allow setting the shared memory (shm) size for the API container

[ninkin]

Hello.

I'm looking for the way to set shm size of cortex workers.

According to the aws documentation(https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_linuxparameters), it seems possible to set shared memory size of containers.

I've implemented pytorch dataloader (num_workers > 0) to use prefetch feature to get images from s3. Because it takes much time to download the files.

btw only I got was this message, "error: unexpected bus error encountered in worker. this might be caused by insufficient shared memory (shm)".

Is it possible to set shm size or any workarounds?

Many thanks in advance.

[vishalbollu]

It may be possible that there isn't enough memory or disk space. Try increasing the memory allocation (using larger instance types in your cluster configuration and setting compute.mem to a higher value in your API spec). To increase disk space, you can set the instance_volume_size to a higher value.

After doing some cursory research it looks like the pytorch workers use shared memory (as you've mentioned). At this point Cortex doesn't expose a way to update shared memory. Cortex uses Kubernetes under the hood so we will have to use a different strategy to update the shared memory. I found a way to increase the shared memory size here. https://stackoverflow.com/questions/46085748/define-size-for-dev-shm-on-container-engine/46434614#46434614.

Here is a very high level guess as to how I would try to increase the shm size:

1. setup kubectl and configure it to point to your cluster https://docs.cortex.dev/guides/kubectl-setup
2. Deploy your API
3. If you are deploying a Batch API, run kubectl edit job api-<your api name here> and if you are deploying a Realtime API run kubectl edit deployment api-<your api name here>. It will open up an editor to edit the relevant resource. Navigate to the volume and volumeMounts section and update those sections according to the config specified in stackoverflow post above.

Let us know if this works for you.

[ninkin]

Thanks for answers.

I think the timing possible to modify job config is right after the job submission.
Unfortunately, the api initialization is quite fast, so the api pod doesn't wait for me to modify job config using kubectl(my hands are so dumb). It starts itself and is killed by its shared memory exception.

If it is not possible to set shm size for now, could you consider that as feature request for future?

[vishalbollu]

Yes this will be considered as a feature request.

Oh that's right. Also I think kubernetes jobs are meant to be immutable so kubectl edit may not even work on a job spec.

[deliahu]

@ninkin this has now been released in v0.26

[ninkin]

Great Thanks!