Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-7945][CORE]Do trim to values in properties file #6496

Closed
wants to merge 3 commits into from

Conversation

WangTaoTheTonic
Copy link
Contributor

https://issues.apache.org/jira/browse/SPARK-7945

Now applications submited by org.apache.spark.launcher.Main read properties file without doing trim to values in it.
If user left a space after a value(say spark.driver.extraClassPath) then it probably affect global functions(like some jar could not be included in the classpath), so we should do it like Utils.getPropertiesFromFile.

@srowen
Copy link
Member

srowen commented May 29, 2015

Yeah, the way Properties parses is a little weird, in that whitespace doesn't matter around keys and values except at the end of the value. On the one hand I'm reluctant to make Spark's use of Properties nonstandard, but I also can't think of a case where trailing whitespace is valuable to preserve.

This only affects one usage of Properties though which seems inconsistent, so I wonder if this is just causing more confusion. It's also not directly addressing the stated problem, which is theoretical, that this is causing JARs to not load.

@WangTaoTheTonic
Copy link
Contributor Author

What more confusion would this cause? And I cann't think of another solution to address this perfectly so did same as Utils.getPropertiesFromFile.

BTW the problem this causes really hurts us as whitespace behind some value is really hard to be detected.

@srowen
Copy link
Member

srowen commented May 29, 2015

You expect Properties syntax but get something slightly different? A pretty theoretical problem, but that's what I meant.

I suppose we can't use Utils.getPropertiesFromFile because is in the launcher, but are these the only places Properties are parsed this way? then at least this change would make Spark consistent.

I suppose if Utils is already doing this then the cat's out of the bag, so no point in restricting this change to the identified problem I guess.

@SparkQA
Copy link

SparkQA commented May 29, 2015

Test build #33732 has finished for PR 6496 at commit 2c053a1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@WangTaoTheTonic
Copy link
Contributor Author

If you look at this change #3916, then will find actually we did use Utils.getPropertiesFromFile before.

I think it is a miss by carelessness. @vanzin Could you help check this either?

@vanzin
Copy link
Contributor

vanzin commented May 29, 2015

Yeah, it's a theoretical problem, but to be consistent with the rest of the code, I guess it doesn't hurt.

@@ -295,7 +295,11 @@ Properties loadPropertiesFile() throws IOException {
FileInputStream fd = null;
try {
fd = new FileInputStream(propsFile);
props.load(new InputStreamReader(fd, "UTF-8"));
Properties rawProps = new Properties();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is way more complicated than it needs to be. Just do:

for (Map.Entry<Object, Object> e : props.entrySet()) {
  e.setValue(e.getValue().toString().trim());
}

@WangTaoTheTonic
Copy link
Contributor Author

I'm at home now and commit this without compiling locally, hope it all be good.

@SparkQA
Copy link

SparkQA commented May 29, 2015

Test build #33742 has finished for PR 6496 at commit bb41b4b.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@WangTaoTheTonic
Copy link
Contributor Author

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented May 29, 2015

Test build #33745 has finished for PR 6496 at commit bb41b4b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@WangTaoTheTonic
Copy link
Contributor Author

I've done a test with current code, it worked out fine.

@srowen
Copy link
Member

srowen commented May 30, 2015

LGTM

@asfgit asfgit closed this in 9d8aadb May 30, 2015
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
https://issues.apache.org/jira/browse/SPARK-7945

Now applications submited by org.apache.spark.launcher.Main read properties file without doing trim to values in it.
If user left a space after a value(say spark.driver.extraClassPath) then it probably affect global functions(like some jar could not be included in the classpath), so we should do it like Utils.getPropertiesFromFile.

Author: WangTaoTheTonic <wangtao111@huawei.com>
Author: Tao Wang <wangtao111@huawei.com>

Closes apache#6496 from WangTaoTheTonic/SPARK-7945 and squashes the following commits:

bb41b4b [Tao Wang] indent 4 to 2
6dd1cf2 [WangTaoTheTonic] use a simpler way
2c053a1 [WangTaoTheTonic] Do trim to values in properties file
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
https://issues.apache.org/jira/browse/SPARK-7945

Now applications submited by org.apache.spark.launcher.Main read properties file without doing trim to values in it.
If user left a space after a value(say spark.driver.extraClassPath) then it probably affect global functions(like some jar could not be included in the classpath), so we should do it like Utils.getPropertiesFromFile.

Author: WangTaoTheTonic <wangtao111@huawei.com>
Author: Tao Wang <wangtao111@huawei.com>

Closes apache#6496 from WangTaoTheTonic/SPARK-7945 and squashes the following commits:

bb41b4b [Tao Wang] indent 4 to 2
6dd1cf2 [WangTaoTheTonic] use a simpler way
2c053a1 [WangTaoTheTonic] Do trim to values in properties file
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants