Research: spark-k8s job orchestration

[adwk67]

As a user I want to have a plan/outline/overview of how to orchestrate spark jobs running in k8s. This will involve looking at Airflow (i.e. airflow/spark-k8s interplay) and Yunikorn or Furiko.

This is a research ticket: as a result we will able to

• understand pros/cons
• be able to estimate implementation effort
• define specific implementation ticket(s)

[adwk67]

See stackabletech/airflow-operator#155

[adwk67]

Yunikorn

• a resource scheduler rather than a job scheduler
• very non-invasive: install the Yunikorn components and you are good to go
• it overrides the default k8s resource scheduler whenever the admission-controller is installed
• it comes with a UI that provides an overview for running, queued and completed jobs
• default config files can be found here: https://github.com/apache/yunikorn-core/tree/master/config

Furiko

• https://github.com/furiko-io/furiko is a job scheduler
• is in its infancy (< 5 committers, 3 releases, first commit 2022-03-24)
• it currently only supports running a task for a Pod https://furiko.io/docs/execution/job/task-executor#task-executor-list though more are on the roadmap https://furiko.io/docs/contributing/roadmap

Since neither of the above are viable for running & scheduling spark jobs, we have....

Airflow

• see [Merged by Bors] - Documentation: Job scheduling airflow-operator#155 for documentation on how to call/apply custom resources from airflow running in-cluster

[lfrancke]

Thank you!
Can you elaborate a bit on the differences between job and resource scheduler?
Can I implement "queue" like things using Yunikorn? To separate workloads from e.g. different departments? (they get 40% of the cluster, we get 20% and the rest is for evreyone else)

[adwk67]

In the context of this issue, I take job scheduling to refer to the act of submitting a job to run at a particular time/frequency/schedule, fairly independent of available resources. Resource scheduling refers to how or where that job is run, once it has been submitted. The issue description refers to orchestration, which covers both I suppose, but I take it to primarily to refer to how I invoke jobs. With airflow, jobs are scheduled using airflow mechanisms (triggers, cron-type schedules etc.), and are then run by Kubernetes, using the Kubernetes scheduler by default.

Yunikorn transparently overrides/replaces the Kubernetes scheduler: Yunikorn's admission controller runs in a separate pod and runs a mutation and validation webhook to override the scheduler name, amongst other things. The resource is inspected and then assigned to a queue based on e.g. the user, namespace, application name etc. In this way loads can be prioritized or assigned to particular nodes based on their metadata. In Yunikorn parlance this is referred to as applying placement rules.

[lfrancke]

Okay! In this case the customer was referring to the resource scheduling part of things.
So Yunikorn would be a solution for us. Thank you!