[SPARK-6343] Doc driver-worker network reqs

[parente]

Attempt at making the driver-worker networking requirement more explicit and up-front in the documentation (see https://issues.apache.org/jira/browse/SPARK-6343).

Update cluster overview diagram to show connections from workers to driver. Add a bullet below about how driver listens / accepts connections from workers.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

LGTM. What do you think @mateiz @pwendell @rxin ?
Also you don't need to include a PPTX file.

[parente]

The original diagram source is in the docs/img/ folder already.

https://github.com/apache/spark/blob/master/docs/img/cluster-overview.pptx

Would you like me to remove it as part of the PR?

[srowen]

Oh I looked past that since it looked like a new file in Github. Yes leave it.

[JoshRosen]

LGTM; this is a nice improvement over the old diagram.

[pwendell]

Great to have an improvement here. One thing I don't understand, there are two curved arrows from the SparkContext to the Executors/Workers. However, in the upper arrow it is connected to the boundary of the Worker box and not the boundary of the Executor box. Is this intentional?

I believe in theory, the process running the SparkContext should not need to directly communicate with the Worker nodes. It should only directly communicate with the Executor process. So maybe they should both touch the boundary of the Executor box.

[WangTaoTheTonic]

I have same question as Patrick too.

[parente]

@pwendell @WangTaoTheTonic Good catch. I misconnected the upper line. It was not intentional.

I've updated the PR with the diagram fix.

[pwendell]

Great LGTM - @WangTaoTheTonic does that look okay to you?

[WangTaoTheTonic]

LGTM +1