[SPARK-16856] [WEBUI] [CORE] Link the application's executor page to the master's UI #14461

nblintao · 2016-08-02T16:34:12Z

What changes were proposed in this pull request?

Added a "Back to Master" link on Executor Page of Application Detail UI, if the application is run in standalone mode.

The links link to the main page of Master UI.

How was this patch tested?

Run Spark as a "local cluster" by ./bin/spark-shell --master=local-cluster[2,1,1024].

Then view the Executor Page on Application Detail UI (http://192.168.1.104:4040/executors/). Users could go to the Master Page by the link "Back to Master" on that page.

SparkQA · 2016-08-02T17:00:04Z

Test build #63129 has finished for PR 14461 at commit f35afa2.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class RegisteredApplication(appId: String, master: RpcEndpointRef,

ajbozarth · 2016-08-02T18:00:30Z

I take some time to check out and run this later today, but on a quick pass the code looks ok. I have some issues with the actual pr though. The link on the application summary page is redundant, clicking the spark logo goes back to the master summary already. As for the Executors page, I'll have to test it, but I think there are quite a few situations where a admin might not want a user to access the master UI, even in a standalone cluster.

ajbozarth · 2016-08-05T22:33:49Z

I forgot to comment, but I did check this out and it runs well, I still think the link on the app summary is redundant though.

SparkQA · 2016-08-06T01:49:36Z

Test build #63302 has finished for PR 14461 at commit 0192b37.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

# Conflicts: # project/MimaExcludes.scala

nblintao · 2016-08-06T01:55:18Z

@ajbozarth Thanks for pointing out that. I have removed the link on the application summary page and reverted related changes.

nblintao · 2016-08-06T01:57:43Z

retest it please

SparkQA · 2016-08-06T01:58:55Z

Test build #63303 has finished for PR 14461 at commit 4564bc5.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class ShuffleIndexInformation
- public class ShuffleIndexRecord
- case class Least(children: Seq[Expression]) extends Expression
- case class Greatest(children: Seq[Expression]) extends Expression
- case class CreateTable(tableDesc: CatalogTable, mode: SaveMode, query: Option[LogicalPlan])
- case class PreprocessDDL(conf: SQLConf) extends Rule[LogicalPlan]

SparkQA · 2016-08-06T02:33:46Z

Test build #63306 has finished for PR 14461 at commit e3707db.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

nblintao · 2016-08-06T02:46:45Z

[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/project/MimaExcludes.scala:50: ')' expected but '.' found.
[error]       ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.scheduler.SparkListenerApplicationStart.apply")
[error]                     ^
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/project/MimaExcludes.scala:51: ';' expected but '.' found.
[error]       ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.scheduler.SparkListenerApplicationStart.copy")
[error]                     ^

I think I have fixed the conflict correctly. Why's that?

ajbozarth · 2016-08-06T02:50:08Z

The excludes are a list, you forgot to add commas between them

nblintao · 2016-08-06T02:52:17Z

Oh yes. Thanks :)

SparkQA · 2016-08-06T04:51:13Z

Test build #63308 has finished for PR 14461 at commit 76e68eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ajbozarth · 2016-10-13T21:59:35Z

@nblintao I'd like to take another look at this, mind rebasing?

nblintao · 2016-12-29T02:55:02Z

@ajbozarth I finally have a chance to rebase it in the winter break. Could you please have a look? Thanks!

SparkQA · 2016-12-29T05:11:09Z

Test build #70698 has finished for PR 14461 at commit d598e55.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ajbozarth · 2017-01-03T18:37:53Z

Thanks for rebasing, I'll check this out sometime this week

ajbozarth · 2017-01-18T00:05:04Z

@nblintao did you try this code on different cluster types? Like local, standalone, yarn, mesos?

ajbozarth · 2017-01-18T00:05:30Z

Also @srowen and @vanzin could you take a look at this one too?

ajbozarth · 2017-01-19T01:32:37Z

I did some testing and this properly shows up on local and standalone and doesn't show the link on yarn. So this LGTM

vanzin · 2017-01-25T22:47:04Z

core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala

@@ -126,7 +126,8 @@ case class SparkListenerApplicationStart(
    time: Long,
    sparkUser: String,
    appAttemptId: Option[String],
-    driverLogs: Option[Map[String, String]] = None) extends SparkListenerEvent
+    driverLogs: Option[Map[String, String]] = None,
+    masterWebUiUrl: Option[String] = None) extends SparkListenerEvent


Do you need this? It seems the standalone registration messages already take care of this.

If you want this, are you intentionally not saving this in event logs? The way it is, this is only ever useful when the application is starting but hasn't yet registered with the master. That's a very short period of time and probably not worth it.

If you add code to save to event logs, I'd rename this to be more cluster manager-agnostic. e.g. "clusterWebUiUrl".

vanzin · 2017-01-25T22:47:32Z

core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala

@@ -50,6 +50,10 @@ private[ui] class ExecutorsPage(
  private val listener = parent.listener

  def render(request: HttpServletRequest): Seq[Node] = {
+    val masterWebUiUrl = listener.masterWebUiUrl


I think this information makes way more sense in the "Environment" tab.

HyukjinKwon · 2017-02-09T13:56:26Z

(gentle ping @nblintao)

nblintao · 2017-02-09T18:39:26Z

@HyukjinKwon Thanks for reminding. I will have a look asap. Too many things to do recently.

HyukjinKwon · 2017-05-11T13:01:39Z

(gentle ping @nblintao)

nblintao · 2017-05-11T15:46:25Z

@HyukjinKwon Thanks for reminding. I am finals this week. I will fix this after 15th.

link application page to master page

f35afa2

nblintao changed the title ~~[SPARK-16856] [WEBUI] [CORE] Link application summary page and detail page to the master page~~ [SPARK-16856] [WEBUI] [CORE] Link the application's executor page to the master's UI Aug 6, 2016

nblintao added 2 commits August 6, 2016 09:41

Remove the link on the application summary page

1f56f46

Update MimaExcludes

0192b37

Merge branch 'master' into linkMasterPage

4564bc5

# Conflicts: # project/MimaExcludes.scala

remove blanks

e3707db

update MimaExcludes

76e68eb

nblintao added 2 commits December 28, 2016 20:09

Merge remote-tracking branch 'apache/master'

8743898

resolve conflicts

d598e55

ajbozarth mentioned this pull request Jan 18, 2017

[SPARK-16520] [WEBUI] Link executors to corresponding worker pages #14204

Closed

vanzin reviewed Jan 25, 2017

View reviewed changes

HyukjinKwon mentioned this pull request Jun 7, 2017

[INFRA] Close stale PRs #18223

Closed

asfgit closed this in b771fed Jun 8, 2017

[SPARK-16856] [WEBUI] [CORE] Link the application's executor page to the master's UI #14461

[SPARK-16856] [WEBUI] [CORE] Link the application's executor page to the master's UI #14461

Uh oh!

Conversation

nblintao commented Aug 2, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Aug 2, 2016

Uh oh!

ajbozarth commented Aug 2, 2016

Uh oh!

ajbozarth commented Aug 5, 2016

Uh oh!

SparkQA commented Aug 6, 2016

Uh oh!

nblintao commented Aug 6, 2016

Uh oh!

nblintao commented Aug 6, 2016

Uh oh!

SparkQA commented Aug 6, 2016

Uh oh!

SparkQA commented Aug 6, 2016

Uh oh!

nblintao commented Aug 6, 2016

Uh oh!

ajbozarth commented Aug 6, 2016

Uh oh!

nblintao commented Aug 6, 2016

Uh oh!

SparkQA commented Aug 6, 2016

Uh oh!

ajbozarth commented Oct 13, 2016

Uh oh!

nblintao commented Dec 29, 2016

Uh oh!

SparkQA commented Dec 29, 2016

Uh oh!

ajbozarth commented Jan 3, 2017

Uh oh!

ajbozarth commented Jan 18, 2017

Uh oh!

ajbozarth commented Jan 18, 2017

Uh oh!

ajbozarth commented Jan 19, 2017

Uh oh!

vanzin Jan 25, 2017

Choose a reason for hiding this comment

Uh oh!

vanzin Jan 25, 2017

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Feb 9, 2017

Uh oh!

nblintao commented Feb 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented May 11, 2017

Uh oh!

nblintao commented May 11, 2017

Uh oh!

Uh oh!

nblintao commented Aug 2, 2016 •

edited

Loading

nblintao commented Feb 9, 2017 •

edited

Loading