Skip to content

Commit 540afc2

Browse files
shahidki31srowen
authored andcommitted
[SPARK-26109][WEBUI] Duration in the task summary metrics table and the task table are different
## What changes were proposed in this pull request? Task summary table displays the summary of the task table in the stage page. However, the 'Duration' metrics of 'task summary' table and 'task table' are not matching. The reason is because, in the 'task summary' we display 'executorRunTime' as the duration, and in the 'task table' the actual duration of the task. Except duration metrics, all other metrics are properly displaying in the task summary. In Spark2.2, used to show 'executorRunTime' as duration in the 'taskTable'. That is why, in summary metrics also the 'exeuctorRunTime' shows as the duration. So, we need to show 'executorRunTime' as the duration in the tasks table to follow the same behaviour as the previous versions of spark. ## How was this patch tested? Before patch: ![screenshot from 2018-11-19 04-32-06](https://user-images.githubusercontent.com/23054875/48679263-1e4fff80-ebb4-11e8-9ed5-16d892039e01.png) After patch: ![screenshot from 2018-11-19 04-37-39](https://user-images.githubusercontent.com/23054875/48679343-e39a9700-ebb4-11e8-8df9-9dc3a28d4bce.png) Closes apache#23081 from shahidki31/duratinSummary. Authored-by: Shahid <shahidki31@gmail.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>
1 parent a480a62 commit 540afc2

File tree

1 file changed

+4
-2
lines changed

1 file changed

+4
-2
lines changed

core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -843,7 +843,7 @@ private[ui] class TaskPagedTable(
843843
</div>
844844
</td>
845845
<td>{UIUtils.formatDate(task.launchTime)}</td>
846-
<td>{formatDuration(task.duration)}</td>
846+
<td>{formatDuration(task.taskMetrics.map(_.executorRunTime))}</td>
847847
<td class={TaskDetailsClassNames.SCHEDULER_DELAY}>
848848
{UIUtils.formatDuration(AppStatusUtils.schedulerDelay(task))}
849849
</td>
@@ -996,7 +996,9 @@ private[ui] object ApiHelper {
996996
HEADER_EXECUTOR -> TaskIndexNames.EXECUTOR,
997997
HEADER_HOST -> TaskIndexNames.HOST,
998998
HEADER_LAUNCH_TIME -> TaskIndexNames.LAUNCH_TIME,
999-
HEADER_DURATION -> TaskIndexNames.DURATION,
999+
// SPARK-26109: Duration of task as executorRunTime to make it consistent with the
1000+
// aggregated tasks summary metrics table and the previous versions of Spark.
1001+
HEADER_DURATION -> TaskIndexNames.EXEC_RUN_TIME,
10001002
HEADER_SCHEDULER_DELAY -> TaskIndexNames.SCHEDULER_DELAY,
10011003
HEADER_DESER_TIME -> TaskIndexNames.DESER_TIME,
10021004
HEADER_GC_TIME -> TaskIndexNames.GC_TIME,

0 commit comments

Comments
 (0)