Skip to content

MAPREDUCE-7279. Adds RM host name to history server web page #2028

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: trunk
Choose a base branch
from
+93 −2
Open

MAPREDUCE-7279. Adds RM host name to history server web page #2028

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions ...ient-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1588,6 +1588,19 @@ protected class MetaInfo {
this.jobIndexInfo =
new JobIndexInfo(-1, -1, user, jobName, jobId, -1, -1, null,
queueName);

if (getConfig().getBoolean(
JHAdminConfig.MR_HISTORY_APPEND_RM_HOST_TO_HISTORY_FILE_NAME_ENABLED,
JHAdminConfig.DEFAULT_MR_HISTORY_APPEND_RM_HOST_TO_HISTORY_FILE_NAME_ENABLED)) {

String hostName = getConfig().get(YarnConfiguration.RM_HOSTNAME, "");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't config the yarn.resourcemanager.address in client, how do we get the right address? And if enabled the federation feature, how can we ensure this?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't config it, it will not show up on the UI.

Could you tell me what federation feature you are referring to?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I means the YARN Federation feature. As I know in the federation cluster has only one JobHistory server, but maybe has more YARN physical clusters.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that YARN federation feature can have multiple RMs.
I created a high availability cluster to check my changes, and noticed that one of the RMs (job-cluster-m-0) was listed as the host.

Is it safe to assume that YARN federation will behave similarly? Please note that I used Hadoop 3.2.1 to test this.

Copy link
Member

@jiwq jiwq Sep 9, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so. In federation cluster, the app can be routed to any physical cluster.

if (!hostName.isEmpty()) {
this.jobIndexInfo.setResourceManagerHost(hostName);
} else {
LOG.warn("Could not retreive a valid YARN host name");
}
}

this.jobSummary = new JobSummary();
this.flushTimer = new Timer("FlushTimer", true);
this.forcedJobStateOnShutDown = forcedJobStateOnShutDown;
Expand Down
1 change: 1 addition & 0 deletions ...hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ public interface Job {
boolean isUber();
String getUserName();
String getQueueName();
String getResourceManagerHost();

/**
* @return a path to where the config file for this job is located.
Expand Down
7 changes: 6 additions & 1 deletion ...preduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1162,7 +1162,12 @@ public String getUserName() {
public String getQueueName() {
return queueName;
}


@Override
public String getResourceManagerHost() {
throw new UnsupportedOperationException();
}

@Override
public void setQueueName(String queueName) {
this.queueName = queueName;
Expand Down
5 changes: 5 additions & 0 deletions ...adoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -605,6 +605,11 @@ public String getQueueName() {
return "mockqueue";
}

@Override
public String getResourceManagerHost() {
return "mockresourcemanager";
}

@Override
public Path getConfFile() {
return configFile;
Expand Down
5 changes: 5 additions & 0 deletions ...ce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -490,6 +490,11 @@ public String getQueueName() {
throw new UnsupportedOperationException("Not supported yet.");
}

@Override
public String getResourceManagerHost() {
throw new UnsupportedOperationException("Not supported yet.");
}

@Override
public int getTotalMaps() {
return mapTasks.size();
Expand Down
18 changes: 18 additions & 0 deletions ...nt-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ public class FileNameIndexUtils {
private static final int JOB_STATUS_INDEX = 7;
private static final int QUEUE_NAME_INDEX = 8;
private static final int JOB_START_TIME_INDEX = 9;
private static final int RESOURCE_MANAGER_INDEX = 10;

/**
* Constructs the job history file name from the JobIndexInfo.
Expand Down Expand Up @@ -116,6 +117,13 @@ public static String getDoneFileName(JobIndexInfo indexInfo,
sb.append(encodeJobHistoryFileName(
String.valueOf(indexInfo.getJobStartTime())));

String resourceManagerName = getResourceManagerHost(indexInfo);
if (resourceManagerName != null) {
sb.append(DELIMITER);
sb.append(escapeDelimiters(encodeJobHistoryFileName(
getResourceManagerHost(indexInfo))));
}

sb.append(encodeJobHistoryFileName(
JobHistoryUtils.JOB_HISTORY_FILE_EXTENSION));
return sb.toString();
Expand Down Expand Up @@ -198,6 +206,12 @@ public static JobIndexInfo getIndexInfo(String jhFileName)
LOG.warn("Unable to parse start time from job history file "
+ jhFileName + " : " + e);
}

if (jobDetails.length > RESOURCE_MANAGER_INDEX) {
indexInfo.setResourceManagerHost(decodeJobHistoryFileName(jobDetails[RESOURCE_MANAGER_INDEX]));
} else {
LOG.warn("Could not parse cluster name from job history file " + jhFileName );
}
} catch (IndexOutOfBoundsException e) {
LOG.warn("Parsing job history file with partial data encoded into name: "
+ jhFileName);
Expand Down Expand Up @@ -292,6 +306,10 @@ private static String getQueueName(JobIndexInfo indexInfo) {
return getNonEmptyString(indexInfo.getQueueName());
}

private static String getResourceManagerHost(JobIndexInfo indexInfo) {
return getNonEmptyString(indexInfo.getResourceManagerHost());
}

//TODO Maybe handle default values for longs and integers here?

private static String getNonEmptyString(String in) {
Expand Down
7 changes: 6 additions & 1 deletion ...-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,12 @@ public class JHAdminConfig {
MR_HISTORY_PREFIX + "intermediate-user-done-dir.permissions";
public static final short
DEFAULT_MR_HISTORY_INTERMEDIATE_USER_DONE_DIR_PERMISSIONS = 0770;


public static final String MR_HISTORY_APPEND_RM_HOST_TO_HISTORY_FILE_NAME_ENABLED =
MR_HISTORY_PREFIX + "append-rm-host-to-history-file-name.enabled";
public static final boolean DEFAULT_MR_HISTORY_APPEND_RM_HOST_TO_HISTORY_FILE_NAME_ENABLED =
true;

/** Size of the job list cache.*/
public static final String MR_HISTORY_JOBLIST_CACHE_SIZE =
MR_HISTORY_PREFIX + "joblist.cache.size";
Expand Down
8 changes: 8 additions & 0 deletions ...e-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ public class JobIndexInfo {
private int numReduces;
private String jobStatus;
private long jobStartTime;
private String resourceManagerHost;

public JobIndexInfo() {
}
Expand All @@ -59,6 +60,7 @@ public JobIndexInfo(long submitTime, long finishTime, String user,
this.jobStatus = jobStatus;
this.jobStartTime = -1;
this.queueName = queueName;
this.resourceManagerHost = null;
}

public long getSubmitTime() {
Expand Down Expand Up @@ -121,6 +123,12 @@ public long getJobStartTime() {
public void setJobStartTime(long lTime) {
this.jobStartTime = lTime;
}
public String getResourceManagerHost() {
return resourceManagerHost;
}
public void setResourceManagerHost(String resourceManagerHost) {
this.resourceManagerHost = resourceManagerHost;
}

@Override
public String toString() {
Expand Down
5 changes: 5 additions & 0 deletions ...oop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,6 +416,11 @@ public String getQueueName() {
return jobInfo.getJobQueueName();
}

@Override
public String getResourceManagerHost() {
throw new UnsupportedOperationException();
}

@Override
public int getTotalMaps() {
return (int) jobInfo.getTotalMaps();
Expand Down
5 changes: 5 additions & 0 deletions ...adoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,11 @@ public String getQueueName() {
return jobIndexInfo.getQueueName();
}

@Override
public String getResourceManagerHost() {
return jobIndexInfo.getResourceManagerHost();
}

@Override
public JobState getState() {
JobState js = null;
Expand Down
5 changes: 5 additions & 0 deletions ...doop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/UnparsedJob.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,11 @@ public String getQueueName() {
return jobIndexInfo.getQueueName();
}

@Override
public String getResourceManagerHost() {
return jobIndexInfo.getResourceManagerHost();
}

@Override
public Path getConfFile() {
return jhfInfo.getConfFile();
Expand Down
10 changes: 10 additions & 0 deletions ...preduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobsBlock.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ public class HsJobsBlock extends HtmlBlock {
th(".id", "Job ID").
th(".name", "Name").
th("User").
th(".resourceManagerHost", "Resource Manager Host").
th("Queue").
th(".state", "State").
th("Maps Total").
Expand All @@ -96,6 +97,11 @@ public class HsJobsBlock extends HtmlBlock {
.checkAccess(ugi, JobACL.VIEW_JOB, job.getUserName(), null)) {
continue;
}

String resourceManagerHost = "";
if (j.getResourceManagerHost() != null) {
resourceManagerHost = j.getResourceManagerHost();
}
jobsTableData.append("[\"")
.append(dateFormat.format(new Date(job.getSubmitTime()))).append("\",\"")
.append(job.getFormattedStartTimeStr(dateFormat)).append("\",\"")
Expand All @@ -106,6 +112,8 @@ public class HsJobsBlock extends HtmlBlock {
job.getName()))).append("\",\"")
.append(StringEscapeUtils.escapeEcmaScript(StringEscapeUtils.escapeHtml4(
job.getUserName()))).append("\",\"")
.append(StringEscapeUtils.escapeEcmaScript(StringEscapeUtils.escapeHtml4(
resourceManagerHost))).append("\",\"")
.append(StringEscapeUtils.escapeEcmaScript(StringEscapeUtils.escapeHtml4(
job.getQueueName()))).append("\",\"")
.append(job.getState()).append("\",\"")
Expand Down Expand Up @@ -140,6 +148,8 @@ public class HsJobsBlock extends HtmlBlock {
.$name("name").$value("Name").__().__().
th().input("search_init").$type(InputType.text)
.$name("user").$value("User").__().__().
th().input("search_init").$type(InputType.text)
.$name("resourceManagerHost").$value("Resource Manager Host").__().__().
th().input("search_init").$type(InputType.text)
.$name("queue").$value("Queue").__().__().
th().input("search_init").$type(InputType.text)
Expand Down
1 change: 1 addition & 0 deletions ...apreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestBlocks.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -398,6 +398,7 @@ private Job getJob() {
when(job.getReport()).thenReturn(report);
when(job.getName()).thenReturn("JobName");
when(job.getUserName()).thenReturn("UserName");
when(job.getResourceManagerHost()).thenReturn("ResourceManagerHost");
when(job.getQueueName()).thenReturn("QueueName");
when(job.getState()).thenReturn(JobState.SUCCEEDED);
when(job.getTotalMaps()).thenReturn(3);
Expand Down
5 changes: 5 additions & 0 deletions ...ient-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesAcls.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -378,6 +378,11 @@ public String getQueueName() {
return mockJob.getQueueName();
}

@Override
public String getResourceManagerHost() {
return mockJob.getResourceManagerHost();
}

@Override
public Path getConfFile() {
return new Path("/some/path/to/conf");
Expand Down