[SPARK-7298] Harmonize style of new visualizations #5942

mateiz · 2015-05-06T14:29:30Z

Colors on the timeline now match the rest of the UI
The expandable buttons to show timeline view, DAG, etc are now more visible
Timeline text is smaller
DAG visualization text and colors are more consistent throughout
Fix some JavaScript style issues
Various small fixes throughout (e.g. inconsistent capitalization, some confusing names, HTML escaping, etc)

AmplabJenkins · 2015-05-06T14:32:10Z

Merged build triggered.

AmplabJenkins · 2015-05-06T14:32:17Z

Merged build started.

SparkQA · 2015-05-06T14:33:59Z

Test build #31979 has started for PR 5942 at commit 49fd0eb.

mateiz · 2015-05-06T14:37:22Z

core/src/main/scala/org/apache/spark/ui/jobs/AllStagesPage.scala

-              <strong>Scheduling Mode: </strong>
-              {listener.schedulingMode.map(_.toString).getOrElse("Unknown")}
-            </li>
-            {


I removed this because this info is already on the homepage

SparkQA · 2015-05-06T16:16:02Z

Test build #31979 has finished for PR 5942 at commit 49fd0eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-06T16:16:08Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-06T16:16:08Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31979/
Test PASSed.

tdas · 2015-05-07T01:44:18Z

@mateiz Could you post before and after screenshots :)

tdas · 2015-05-07T01:45:02Z

@zsxwing you should take a look at the changes to the streaming UI

tdas · 2015-05-07T01:52:08Z

also pinging @andrewor14

duncanfinney · 2015-05-07T04:03:28Z

core/src/main/scala/org/apache/spark/ui/UIUtils.scala

@@ -352,7 +353,7 @@ private[spark] object UIUtils extends Logging {
    <div>
      <span class="expand-dag-viz" onclick={s"toggleDagViz($forJob);"}>
        <span class="expand-dag-viz-arrow arrow-closed"></span>
-        <strong>DAG visualization</strong>
+        <a>DAG Visualization</a>


Small typo. This shouldn't be an <a />

No, why would it be? </a> is proper HTML syntax.

AmplabJenkins · 2015-05-07T17:52:11Z

Merged build triggered.

AmplabJenkins · 2015-05-07T17:52:21Z

Merged build started.

SparkQA · 2015-05-07T17:54:06Z

Test build #32128 has started for PR 5942 at commit be03a58.

mateiz · 2015-05-07T17:55:27Z

Here are some pics of what it looks like:

andrewor14 · 2015-05-07T19:15:15Z

core/src/main/scala/org/apache/spark/ui/UIUtils.scala

+    <link rel="stylesheet" href={prependBaseUri("/static/webui.css")}
+          type="text/css" />
+    <link rel="stylesheet" href={prependBaseUri("/static/timeline-view.css")}
+          type="text/css"></link>


small nit: this can be /> to be consistent with the other lines.

andrewor14 · 2015-05-07T19:18:05Z

@mateiz The new colors look great! One thing that would be good to add is synchronize hovers over the two graphs on the job page. E.g. if I mouseover a stage on the timeline, it should light up in the DAG and vice versa. I can add this separately if you prefer.

SparkQA · 2015-05-07T19:34:08Z

Test build #32128 has finished for PR 5942 at commit be03a58.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-07T19:34:13Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-07T19:34:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32128/
Test PASSed.

mateiz · 2015-05-07T21:17:11Z

Yeah let's add that separately. BTW I will take a bit of time to tweak the DAG view as well, and after that I think I'm done with this PR.

AmplabJenkins · 2015-05-08T03:42:10Z

Build triggered.

AmplabJenkins · 2015-05-08T03:42:20Z

Build started.

andrewor14 · 2015-05-08T04:04:21Z

DAG parts LGTM.

mateiz · 2015-05-08T04:21:53Z

Cool, thanks! I've reduced the spacing too with your hint.

AmplabJenkins · 2015-05-08T04:22:12Z

Merged build triggered.

AmplabJenkins · 2015-05-08T04:22:21Z

Merged build started.

SparkQA · 2015-05-08T04:22:40Z

Test build #32196 has started for PR 5942 at commit 4c5a364.

SparkQA · 2015-05-08T05:31:57Z

Test build #32189 has finished for PR 5942 at commit 43dcbe3.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ElementwiseProduct extends UnaryTransformer[Vector, Vector, ElementwiseProduct]
- class ElementwiseProduct(val scalingVector: Vector) extends VectorTransformer
- trait Star extends NamedExpression with trees.LeafNode[Expression]
- trait CaseWhenLike extends Expression
- case class CaseWhen(branches: Seq[Expression]) extends CaseWhenLike
- case class CaseKeyWhen(key: Expression, branches: Seq[Expression]) extends CaseWhenLike
- case class CreateTableAsSelect(

AmplabJenkins · 2015-05-08T05:32:02Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-08T05:32:02Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32189/
Test PASSed.

SparkQA · 2015-05-08T05:32:29Z

Test build #32196 has finished for PR 5942 at commit 4c5a364.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ElementwiseProduct extends UnaryTransformer[Vector, Vector, ElementwiseProduct]
- class ElementwiseProduct(val scalingVector: Vector) extends VectorTransformer
- trait Star extends NamedExpression with trees.LeafNode[Expression]
- trait CaseWhenLike extends Expression
- case class CaseWhen(branches: Seq[Expression]) extends CaseWhenLike
- case class CaseKeyWhen(key: Expression, branches: Seq[Expression]) extends CaseWhenLike
- case class CreateTableAsSelect(

AmplabJenkins · 2015-05-08T05:32:34Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-08T05:32:35Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32196/
Test FAILed.

AmplabJenkins · 2015-05-08T14:57:12Z

Merged build triggered.

AmplabJenkins · 2015-05-08T14:57:21Z

Merged build started.

SparkQA · 2015-05-08T14:57:36Z

Test build #32239 has started for PR 5942 at commit def38d0.

pwendell · 2015-05-08T15:17:45Z

This looks cool Matei. I did some basic playing around and found a case where it looks like the job level DAG is broken. Not sure if it was added by this patch though (ping @andrewor14 as well):

val df = sqlContext.jsonFile("examples/src/main/resources/people.json")
df: org.apache.spark.sql.DataFrame = [age: bigint, name: string]
scala> df.count()
res0: Long = 3

mateiz · 2015-05-08T15:32:41Z

Yeah that seems to be a bug in the JS for drawing DAGs that might've been there before (look at the JS console during it, you will see an error printed).

pwendell · 2015-05-08T15:35:22Z

Sounds good - the changes in this patch LGTM.

On Fri, May 8, 2015 at 4:33 PM, Matei Zaharia notifications@github.com
wrote:

Yeah that seems to be a bug in the JS for drawing DAGs that might've been
there before (look at the JS console during it, you will see an error
printed).

—
Reply to this email directly or view it on GitHub
#5942 (comment).

SparkQA · 2015-05-08T16:38:05Z

Test build #32239 has finished for PR 5942 at commit def38d0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-08T16:38:11Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-08T16:38:11Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32239/
Test PASSed.

zsxwing · 2015-05-08T16:59:28Z

@pwendell , @andrewor14 is fixing it in #6002

andrewor14 · 2015-05-08T17:28:14Z

Thanks @zsxwing. Yes it's currently broken for SQL. There are a few upcoming fixes:

[SPARK-7466] DAG visualization: fix orphan nodes #6002 fixes this general bug, which is also a problem in Streaming and some MLlib jobs
[SPARK-7469] [SQL] DAG visualization: show SQL query operators #5999 makes the graph display SQL operators rather than low level Spark primitives.

- Colors on the timeline now match the rest of the UI - The expandable buttons to show timeline view, DAG, etc are now more visible - Timeline text is smaller - DAG visualization text and colors are more consistent throughout - Fix some JavaScript style issues - Various small fixes throughout (e.g. inconsistent capitalization, some confusing names, HTML escaping, etc) Author: Matei Zaharia <matei@databricks.com> Closes #5942 from mateiz/ui and squashes the following commits: def38d0 [Matei Zaharia] Add some tooltips 4c5a364 [Matei Zaharia] Reduce stage and rank separation slightly 43dcbe3 [Matei Zaharia] Some updates to DAG fac734a [Matei Zaharia] tweaks 6a6705d [Matei Zaharia] More fixes 67629f5 [Matei Zaharia] Various small tweaks (cherry picked from commit a1ec08f) Signed-off-by: Matei Zaharia <matei@databricks.com>

- Colors on the timeline now match the rest of the UI - The expandable buttons to show timeline view, DAG, etc are now more visible - Timeline text is smaller - DAG visualization text and colors are more consistent throughout - Fix some JavaScript style issues - Various small fixes throughout (e.g. inconsistent capitalization, some confusing names, HTML escaping, etc) Author: Matei Zaharia <matei@databricks.com> Closes apache#5942 from mateiz/ui and squashes the following commits: def38d0 [Matei Zaharia] Add some tooltips 4c5a364 [Matei Zaharia] Reduce stage and rank separation slightly 43dcbe3 [Matei Zaharia] Some updates to DAG fac734a [Matei Zaharia] tweaks 6a6705d [Matei Zaharia] More fixes 67629f5 [Matei Zaharia] Various small tweaks

mateiz reviewed May 6, 2015
View reviewed changes

duncanfinney reviewed May 7, 2015
View reviewed changes

mateiz force-pushed the ui branch from 49fd0eb to be03a58 Compare May 7, 2015 17:49

andrewor14 reviewed May 7, 2015
View reviewed changes

mateiz force-pushed the ui branch from 728eedd to 8a552bb Compare May 8, 2015 03:38

mateiz added 3 commits May 7, 2015 23:40

Various small tweaks

67629f5

More fixes

6a6705d

tweaks

fac734a

Some updates to DAG

43dcbe3

Reduce stage and rank separation slightly

4c5a364

Add some tooltips

def38d0

asfgit closed this in a1ec08f May 8, 2015

[SPARK-7298] Harmonize style of new visualizations #5942

[SPARK-7298] Harmonize style of new visualizations #5942

Conversation

mateiz commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

SparkQA commented May 6, 2015

mateiz May 6, 2015

Choose a reason for hiding this comment

SparkQA commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

tdas commented May 7, 2015

tdas commented May 7, 2015

tdas commented May 7, 2015

duncanfinney May 7, 2015

Choose a reason for hiding this comment

mateiz May 7, 2015

Choose a reason for hiding this comment

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

mateiz commented May 7, 2015

andrewor14 May 7, 2015

Choose a reason for hiding this comment

andrewor14 commented May 7, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

mateiz commented May 7, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

andrewor14 commented May 8, 2015

mateiz commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

SparkQA commented May 8, 2015

SparkQA commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

SparkQA commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

SparkQA commented May 8, 2015

pwendell commented May 8, 2015

mateiz commented May 8, 2015

pwendell commented May 8, 2015

SparkQA commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

zsxwing commented May 8, 2015

andrewor14 commented May 8, 2015