Make resource manager pluggable

[hectcastro]

Right now, the resource manager referenced in spark.js targets YARN specifically. It would be great to have support for both YARN and Mesos.

[ryan-williams]

Good idea! I've never actually run Spark on Mesos so someone who has will probably be better suited to taking this on.

[hectcastro]

Going to mess with it soon because I want to take advantage of all of the work already done here. Just wanted to put it out there in case anyone else already started down the path.

Thanks, @ryan-williams!

[ryan-williams]

excellent, thanks for your help

[seregasheypak]

Hi, we are not using YARN currently, what could be an approach to gather apps ids from master or spark-history server?

[ryan-williams]

Good question @seregasheypak. It seems like it should be straightforward to make a request to Spark history server instead of YARN RM, and scrape the results.

Alternatively, once Spark-4435 goes in (which seems like will be soon), maybe there will be a JSON endpoint that you can get a more structured form of that data from?

I'd start with just scraping the history server (unless there's already a JSON endpoint that will work).

Let me know if you are interested in working on that, otherwise I'll see if I can get to it.

[seregasheypak]

Got it, will report back in few days.