SparkJob resource in Kubernetes

[foxish]

We are planning to make changes to the way external resources are represented in Kubernetes. This includes TPRs which we planned on using for representing a SparkJob.

I'm using this issue as a place to collect thoughts about what items we want a spark job to surface up to Kubernetes.

A preliminary list includes the following:

• Spark-submit command line parameters (container images, cpu, memory, service account etc)
• Number of executors requested
• Number of executors active
• Job submission timestamp
• Job completion timestamp (if completed)
• URLs of driver and executor UIs
• Exit Status and Exit Reason (if completed)

Do we have other items we want Kubernetes to know about?

cc/ @erikerlandson @mccheah @ash211 @ssuchter

[foxish]

This is prerequisite to #22

[ash211]

URLs of driver and executor UIs are most valuable to me.

An additional point to note is that when a job is running the Spark UI is served out of those services, but after those services stop they also stop serving the Spark UI. Often you want to look at a Spark UI after its job has finished. So Spark supports writing event logs out to a separate location (spark.eventLog.dir) and running a separate service called the Spark History Server that recreates most of the Spark UI for offline use. This is documented at http://spark.apache.org/docs/latest/monitoring.html#viewing-after-the-fact

For SparkJobs after they've stopped, I'd want to also have the URL of the job in the Spark History Server as a link.

[kimoonkim]

+1 for Spark web UI URL.

As to spark.eventLog.dir, this is typically a dir on HDFS, as the example on the documentation page shows. So this might not work right away:

spark.eventLog.dir hdfs://namenode/shared/spark-logs

[ssuchter]

Number of executors active - that's interesting because it might mutate frequently - is that something that can be supported by this Kubernetes external resource (frequently mutating items)

If so, it'd be pretty snazzy to also include number of tasks active and what stage number (min/max) are running right now. (I suggest min & max because the full list might be an unbounded set, and that way you can just show two scalars).

URLs are really important, as Andrew pointed out.

Interesting thing for this whole project is to make the Spark History server work. Do we need an external HDFS for that? Or will some other external storage work?

The other thing that occurs to me a-la the first bullet is the main Spark class name that we're trying to run. (What the main class name in the user's code) For example 'org.apache.spark.examples.SparkPi'.

I also bet that the user will want to know what Kubernetes namespace this was run in, but that seems like it might be known elsewhere from this external resource spec.

[foxish]

Given that we have some time now, we should add TPRs now rather than later. I'm marking this as alpha.
The lack of the TPR installed will not block a user from submitting jobs, but if the endpoint is present, we will create such an object and possibly update it.

A slew of fixes for kubectl to work with TPR are coming in k8s 1.6, and to have it around from the beginning would be a good idea.
I'll incorporate the above items and we can do some rounds of review.

Most of the effort is just porting it over from https://github.com/foxish/spark

cc/ @iyanuobidele

[iyanuobidele]

@foxish

+1
I am on board with this. I believe you discussed this with the team at the latter part of the sig meeting.

I like the idea - Whenever the upstream fixes come, the capability will be there for cluster admins to leverage.

I'll take this on, since we might need to make a few changes to the SparkJob abstraction.

[foxish]

Sounds good. Thanks @iyanuobidele!