SPARK-3480 - Throws out Not a valid command 'yarn-alpha/scalastyle' in dev/scalastyle for sbt build tool during 'Running Scala style checks'

[jameszhouyi]

Symptom:
Run ./dev/run-tests and dump outputs as following:
....
Running Scala style checks
Scalastyle checks failed at following occurrences:
[error] Expected ID character
[error] Not a valid command: yarn-alpha
[error] Expected project ID
[error] Expected configuration
[error] Expected ':' (if selecting a configuration)
[error] Expected key
[error] Not a valid key: yarn-alpha
[error] yarn-alpha/scalastyle
[error] ^

Possible Cause:
I checked the dev/scalastyle, found that there are 2 parameters 'yarn-alpha/scalastyle' and 'yarn/scalastyle' separately,like
echo -e "q\n" | sbt/sbt -Pyarn -Phadoop-0.23 -Dhadoop.version=0.23.9 yarn-alpha/scalastyle \

scalastyle.txt

echo -e "q\n" | sbt/sbt -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 yarn/scalastyle \

scalastyle.txt

From above error message, sbt seems to complain them due to '/' separator. So it can be run through after I manually modified original ones to 'yarn-alpha:scalastyle' and 'yarn:scalastyle'..

[SparkQA]

Can one of the admins verify this patch?

[andrewor14]

I am not able to reproduce this on the latest master. Would you mind closing this @jameszhouyi ?

[jameszhouyi]

The issue found in apache/spark branch-1.1

[jameszhouyi]

Hi @andrewor14
I also tested this on apache/spark master, there is no errors as like yours. But the issue exist on branch-1.1. I am not sure why causes this inconsistent behavior on master v.s. branch-1.1. Could you please confirm this ?

[andrewor14]

Oh I see, this is against branch-1.1. Let me verify.

[andrewor14]

Hi @jameszhouyi I just dev/scalastyle against the latest branch-1.1 and there were no issues. Can you confirm?

[jameszhouyi]

Hi @andrewor14 , I just got the the latest branch-1.1 and run again, but still run across the issue like below details, please kindly review below (I ran it on CentOS6.4).

$ git status
On branch branch-1.1
$ ./dev/run-tests
Scalastyle checks failed at following occurrences:
[error] Expected ID character
[error] Not a valid command: yarn-alpha
[error] Expected project ID
[error] Expected configuration
[error] Expected ':' (if selecting a configuration)
[error] Expected key
[error] Not a valid key: yarn-alpha
[error] yarn-alpha/scalastyle
[error] ^

[srowen]

I also can't reproduce this on branch-1.1, running scalastyle or run-tests. I tried it on OS X and Debian. Is this likely to be some local change, something already fixed?