Log retrieval/streaming support for Pods dispatched to Slurm

[twelho]

Hi! What happens if you request logs kubect logs [-f] for a Pod that has been delegated to the bridge scheduler (Slurm side)? I crawled through the code but was unable to find a definitive answer. My guess is that this isn't currently handled, meaning the log request will get directed to kubelet, which then reports either an error or no logs since it doesn't "manage" that Pod. Unfortunately, I don't have an environment to test this myself at the moment.

[kannon92]

When you do kubectl logs on a pod, you don't need the logs of the scheduler.

This should work without issue. Slurm-Bridge is a scheduler plugin and kubelet will still execute the pod so kubectl logs should work.

[twelho]

Actually, let me rephrase the answer. From the architecture it seems there's a path where the user creates a Pod spec, submits it to the API server, which can then be picked up by the Slinky Bridge and scheduled in the Slurm environment. What happens in this case? Is the Pod also picked up by the kubelet on the same node? Can the user retreive the stdout and stderr of their Slurm job in any way, such as through kubectl logs?

[kannon92]

Can the user retreive the stdout and stderr of their Slurm job in any way, such as through kubectl logs?

kubernetes/enhancements#3288

The feature is in alpha so I don't think its actually possible to get separate streams of these logs via kubectl logs.

What happens in this case? Is the Pod also picked up by the kubelet on the same node?

I am mostly a bystander on this project and I was trying this out via Kind. I am not qualified to answer this question.

[SkylerMalinowski]

Kubernetes and Slurm manage the launch of their own workload, hence pods are launched by kubelet and jobs are launched by slurmd. That means that each workload manager handles logging for their workload. Workload logging is not piped/streamed or otherwise shared between workload managers.

From a technical standpoint, when you submit a pod intended to be scheduled by slurm-bridge, an external job is created in Slurm to represent the pod(s) from Kubernetes. Slurm external jobs are treated like regular jobs but without launch, hence no execution of steps, hence no output file(s).

[twelho]

Thanks for the answer, that clears it up!

[kannon92]

Kubernetes and Slurm manage the launch of their own workload, hence pods are launched by kubelet and jobs are launched by slurmd. That means that each workload manager handles logging for their workload. Workload logging is not piped/streamed or otherwise shared between workload managers.

From a technical standpoint, when you submit a pod intended to be scheduled by slurm-bridge, an external job is created in Slurm to represent the pod(s) from Kubernetes. Slurm external jobs are treated like regular jobs but without lunch, hence no execution of steps, hence no output file(s).

This confused the heck out of me..

If I submit a Kubernetes Job which in turns creates pods, this would be managed by kubelet but slurm is aware of that workload being schedueld?

And I am able to use kubectl logs on kubernetes objects?

[SkylerMalinowski]

Let me clarify nomenclature. The same term can mean different things in Slurm vs Kubernetes.

When I say pod, I mean a Kubernetes pod — Kubernetes' fundamental workload unit. Pods are created by users, or controllers for builtin resources or custom resources.

When I say job, I mean a Slurm job — Slurm's fundamental workload unit. Jobs are created by users, or derived from an array.

If I submit a Kubernetes Job which in turn creates pods, this would be managed by kubelet but slurm is aware of that workload being scheduled?

Kubernetes batch/v1 job will create pods. Those pods (if intended) are scheduled on Kubernetes by Slurm via slurm-bridge (external job used to represent the pod(s)). Once a scheduling decision has been made, the pod is launched by the kubelet. Slurm is only aware of the nodes under its control, as configured.

And I am able to use kubectl logs on kubernetes objects?

Yes.

[kannon92]

Thanks for clearing that up. That makes sense to me.

[twelho]

Kubernetes batch/v1 job will create pods. Those pods (if intended) are scheduled on Kubernetes by Slurm via slurm-bridge (external job used to represent the pod(s)). Once a scheduling decision has been made, the pod is launched by the kubelet. Slurm is only aware of the nodes under its control, as configured.

With this context, Slurm is aware of Pods executed by Kubelet through the "phantom" Slurm job acting as a placeholder for them (one the bridge makes the scheduling decision). However, just to confirm, does the opposite also apply? That is, is a "phantom" no-op Pod created in Kubernetes for Slurm jobs executed by slurmd on the node?

This implies that Slurm has complete visibility into all Slurm and Kubernetes workloads, but does Kubernetes have it as well? Or does the Kubernetes environment not see / can't interact with the Slurm side in any way (other than plumbing scontrol etc. into a container or using the REST API externally)?

[SkylerMalinowski]

The visibility model is not direct. Meaning that Slurm is not directly communicating to Kubernetes and vice versa. The slurm-bridge represents the Kubernetes state by representing pods as jobs. Because Slurm's scope is limited, it only needs to be the source of truth for the resources it manages.

Kubernetes does not have reciprocated visibility into Slurm because it does not need to. For nodes that the slurm-bridge scheduler is supposed to use, a node taint is applied (automatically). This seems to be the preferred methodology as Kubernetes lacks any formal process to denote certain nodes as being exclusively owned by a scheduler (primary, secondary, or otherwise).

Because those tainted nodes are being managed by slurm-bridge (Slurm by proxy), Slurm will decide if Kubernetes or Slurm workload lands on those nodes. Without DRA drivers, we cannot mix both workloads on the same node (yet).

Kubernetes and Slurm lack deep integration where tooling from one natively works on the other (e.g. you cannot sbatch to Kubernetes; you cannot kubectl logs to Slurm). However, their REST APIs is the interface that slurm-bridge can interact with (for both workload management systems) to facilitate coherency.

[twelho]

Alright, thanks for the extended answer! This helped a lot in piecing together the interaction.