[k8s] Allow configuring /dev/shm size limit

[romilbhardwaj]

Many production environments require limits on the size of /dev/shm partition allocated to a pod. This PR adds support for this configuration through pod_config in ~/.sky/config.yaml.

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Manual tests - sky launch with config.yaml set and inspect pod with kubectl describe pods

config.yaml used for testing:

kubernetes:
  ports: ingress
  pod_config:
    spec:
      containers:
        - volumeMounts: # Custom volume mounts for the pod
            - mountPath: /foo
              name: example-volume
              readOnly: true
      volumes:
        - name: example-volume
          hostPath:
            path: /tmp
            type: Directory
        - name: dshm          # SkyPilot creates a /dev/shm volume for each pod called dshm. You can override the sizeLimits/volume type here.
          emptyDir:
            medium: Memory
            sizeLimit: 3Gi

[Michaelvll]

Thank you for adding this @romilbhardwaj! Question: could we mention why we the use case of this and the best practice for a user to specify this field, e.g. if they encounter some specific errors, they should change this field?

[Michaelvll]

Could we mention the use case for specifying this in the comment?

Also, it seems a user could specify this directly in the pod_config above, is it necessary to have a separate field in the config for this?

[romilbhardwaj]

Added a comment. This is required by one of our users, since their production environment uses an AdmissionController that denies any pods which do not have a specific limit on /dev/shm size.

Using pod_config is possible, but requires the user to understand our pod template and use the exact fields ( (name: dshm, ...) when overriding the pod_config.

[Michaelvll]

Thanks for the explanation! How frequent this field will be used? I am wondering if it is not that frequently used, we can, instead, just have the full config in the pod_config in this doc, and have the comment there.
I kind of feeling including two ways to change the same config might be a bit confusing.

[romilbhardwaj]

Ahh good point. I thought more about it and indeed it likely won't be used that frequently. I added a comment to our config.yaml docs and updated the PR description.

[Michaelvll]

Thanks for updating this PR @romilbhardwaj! LGTM.

[Michaelvll]

Is /dev/shm a convention name that people would know? Just wondering if we should elaborate a bit for what is this for.

[romilbhardwaj]

/dev/shm is generally well understood as a shared memory, and explaining it here might add more clutter. It's usage depends on the application (e.g., Ray stores GCS state there, pytorch uses it for IPC), so it's hard to generalize. Folks running docker containers would usually be familiar with this, so might be fine to leave it be?

[Michaelvll]

nit: using filter might be easier to read. not feeling strongly, please feel free to ignore.

Suggested change

	destination_volume = next(
	(v for v in destination[key]
	if v.get('name') == new_volume_name), None)
	destination_volume = next(filter(lambda v: v.get('name') == new_volume_name, destination[key]), None)

[romilbhardwaj]

the generator might be more pythonic :)