[SPARK-18662] Move resource managers to separate directory #16092

foxish · 2016-12-01T00:53:35Z

What changes were proposed in this pull request?

Moves yarn and mesos scheduler backends to resource-managers/ sub-directory (in preparation for https://issues.apache.org/jira/browse/SPARK-18278)
Corresponding change in top-level pom.xml.

Ref: #16061 (comment)

How was this patch tested?

Manual tests

/cc @rxin

foxish · 2016-12-01T00:58:19Z

/cc @tnachen

JoshRosen · 2016-12-01T02:57:29Z

There's a few other places that will need updating, including the source_file_regexes in the test runner configuration (modules.py):

spark/dev/sparktestsupport/modules.py

Line 472 in 0a81121

"yarn/",

and

spark/dev/sparktestsupport/modules.py

Line 488 in 0a81121

source_file_regexes=["mesos/"],

JoshRosen · 2016-12-01T02:59:17Z

Jenkins, this is ok to test

SparkQA · 2016-12-01T03:04:16Z

Test build #69448 has finished for PR 16092 at commit c6c7dc2.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2016-12-01T09:34:23Z

Hm, is this really necessary? I know it's just a move but for 2-3 modules, does it matter enough?

tnachen · 2016-12-01T09:37:30Z

Changed suggested by @rxin

ash211 · 2016-12-01T19:41:43Z

With a k8s backend on the way I do think it adds a nice organization for these 3 clearly grouped modules

foxish · 2016-12-01T22:25:30Z

@JoshRosen Thanks for reviewing.
Updated. PTAL.

SparkQA · 2016-12-01T23:26:41Z

Test build #69499 has finished for PR 16092 at commit 6763a80.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

foxish · 2016-12-01T23:30:36Z

The test failure appears to be an unrelated flake.

SparkQA · 2016-12-03T04:04:35Z

Test build #3465 has finished for PR 16092 at commit 6763a80.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2016-12-06T01:48:30Z

@foxish this needs to be rebased.

foxish · 2016-12-06T16:48:15Z

@vanzin Done. Thanks!

SparkQA · 2016-12-06T19:19:56Z

Test build #69735 has finished for PR 16092 at commit 3e78591.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2016-12-06T21:54:50Z

dev/sparktestsupport/modules.py

@@ -469,7 +469,7 @@ def __hash__(self):
    name="yarn",
    dependencies=[],
    source_file_regexes=[
-        "yarn/",
+        "resource-managers/yarn/",


You also need to update L477 (and probably a similar change in the mesos entry).

@vanzin If I understand correctly, the sbt test goals can stay the same?
I can see them defined the same way for other modules such as sketch (as sketch/test), but it is nested under common/.

Oh. Nevermind, I thought those were file paths like these. So fine the way they are.

@vanzin also rebased after recent changes to HEAD. Is this okay to merge?

vanzin · 2016-12-07T00:22:58Z

LGTM, merging to master.

foxish · 2016-12-07T00:27:52Z

Thanks @vanzin :)

SparkQA · 2016-12-07T02:38:34Z

Test build #69756 has finished for PR 16092 at commit 05c478c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

## What changes were proposed in this pull request? * Moves yarn and mesos scheduler backends to resource-managers/ sub-directory (in preparation for https://issues.apache.org/jira/browse/SPARK-18278) * Corresponding change in top-level pom.xml. Ref: apache#16061 (comment) ## How was this patch tested? * Manual tests /cc rxin Author: Anirudh <ramanathana@google.com> Closes apache#16092 from foxish/fix-scheduler-structure-2.

…n directory. ## What changes were proposed in this pull request? apache#16092 moves YARN resource manager related code to resource-managers/yarn directory. The test case ```YarnSchedulerBackendSuite``` was added after that but with the wrong place. I move it to correct directory in this PR. ## How was this patch tested? Existing test. Author: Yanbo Liang <ybliang8@gmail.com> Closes apache#16595 from yanboliang/yarn.

## What changes were proposed in this pull request? * Moves yarn and mesos scheduler backends to resource-managers/ sub-directory (in preparation for https://issues.apache.org/jira/browse/SPARK-18278) * Corresponding change in top-level pom.xml. Ref: apache#16061 (comment) ## How was this patch tested? * Manual tests /cc rxin Author: Anirudh <ramanathana@google.com> Closes apache#16092 from foxish/fix-scheduler-structure-2.

…n directory. ## What changes were proposed in this pull request? apache#16092 moves YARN resource manager related code to resource-managers/yarn directory. The test case ```YarnSchedulerBackendSuite``` was added after that but with the wrong place. I move it to correct directory in this PR. ## How was this patch tested? Existing test. Author: Yanbo Liang <ybliang8@gmail.com> Closes apache#16595 from yanboliang/yarn.

mgummelt · 2017-02-22T19:42:11Z

@foxish @vanzin I wish you would have cc'd me on this. If you're going to do something significant to Mesos code, please cc me.

foxish · 2017-02-22T21:05:43Z

Will cc you on future issues. Apologies if this disrupted your workflow. It was meant to be a quick change only in code structure.

foxish force-pushed the fix-scheduler-structure-2 branch from 6763a80 to 3e78591 Compare December 6, 2016 16:47

vanzin reviewed Dec 6, 2016

View reviewed changes

foxish added 2 commits December 6, 2016 16:12

Moved cluster schedulers.

60bb878

Update source-file regexes and parent references.

05c478c

foxish force-pushed the fix-scheduler-structure-2 branch from 3e78591 to 05c478c Compare December 7, 2016 00:12

asfgit closed this in 81e5619 Dec 7, 2016

foxish deleted the fix-scheduler-structure-2 branch December 7, 2016 00:30

foxish restored the fix-scheduler-structure-2 branch December 7, 2016 00:30

yanboliang mentioned this pull request Jan 16, 2017

[Minor][YARN] Move YarnSchedulerBackendSuite to resource-managers/yarn directory. #16595

Closed

foxish deleted the fix-scheduler-structure-2 branch February 22, 2017 21:05

[SPARK-18662] Move resource managers to separate directory #16092

[SPARK-18662] Move resource managers to separate directory #16092

Uh oh!

Conversation

foxish commented Dec 1, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

foxish commented Dec 1, 2016

Uh oh!

JoshRosen commented Dec 1, 2016

Uh oh!

JoshRosen commented Dec 1, 2016

Uh oh!

SparkQA commented Dec 1, 2016

Uh oh!

srowen commented Dec 1, 2016

Uh oh!

tnachen commented Dec 1, 2016

Uh oh!

ash211 commented Dec 1, 2016

Uh oh!

foxish commented Dec 1, 2016

Uh oh!

SparkQA commented Dec 1, 2016

Uh oh!

foxish commented Dec 1, 2016

Uh oh!

SparkQA commented Dec 3, 2016

Uh oh!

vanzin commented Dec 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

foxish commented Dec 6, 2016

Uh oh!

SparkQA commented Dec 6, 2016

Uh oh!

vanzin Dec 6, 2016

Choose a reason for hiding this comment

Uh oh!

foxish Dec 7, 2016

Choose a reason for hiding this comment

Uh oh!

vanzin Dec 7, 2016

Choose a reason for hiding this comment

Uh oh!

foxish Dec 7, 2016

Choose a reason for hiding this comment

Uh oh!

vanzin commented Dec 7, 2016

Uh oh!

foxish commented Dec 7, 2016

Uh oh!

SparkQA commented Dec 7, 2016

Uh oh!

mgummelt commented Feb 22, 2017

Uh oh!

foxish commented Feb 22, 2017

Uh oh!

Uh oh!

vanzin commented Dec 6, 2016 •

edited

Loading