Skip to content

[SPARK-27544][PYTHON][TEST][BRANCH-2.4] Fix Python test script to work on Scala-2.12 build #24439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-27544][PYTHON][TEST][BRANCH-2.4] Fix Python test script to work on Scala-2.12 build #24439

wants to merge 1 commit into from

Conversation

dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Apr 23, 2019 •
edited
Loading

What changes were proposed in this pull request?

Since SPARK-27274 deprecated Scala-2.11 at Spark 2.4.1, we need to test Scala-2.12 more. This PR aims to fix the Python test script on Scala-2.12 build in branch-2.4.

BEFORE

$ dev/change-scala-version.sh 2.12

$ build/sbt -Pscala-2.12 package

$ python/run-tests.py --python-executables python2.7 --modules pyspark-sql
Traceback (most recent call last):
  File "python/run-tests.py", line 70, in <module>
    raise Exception("Cannot find assembly build directory, please build Spark first.")
Exception: Cannot find assembly build directory, please build Spark first.

AFTER

$ python/run-tests.py --python-executables python2.7 --modules pyspark-sql
Running PySpark tests. Output is in /Users/dongjoon/APACHE/spark/python/unit-tests.log
Will test against the following Python executables: ['python2.7']
Will test the following Python modules: ['pyspark-sql']
Starting test(python2.7): pyspark.sql.tests
...

How was this patch tested?

Manually do the above procedure because Jenkins doesn't test Scala-2.12 in branch-2.4.

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-27544][PYTHON][TEST] Fix Python test script to work on Scala-2.12 build [SPARK-27544][PYTHON][TEST][BRANCH-2.4] Fix Python test script to work on Scala-2.12 build Apr 23, 2019
@dongjoon-hyun
Copy link
Member Author

cc @srowen and @HyukjinKwon

felixcheung
felixcheung reviewed Apr 23, 2019
View reviewed changes
python/run-tests.py
# Later, add back 2.12 to this list:
# for scala in ["2.11", "2.12"]:
for scala in ["2.11"]:
for scala in ["2.11", "2.12"]:
build_dir = os.path.join(SPARK_HOME, "assembly", "target", "scala-" + scala)
if os.path.isdir(build_dir):
SPARK_DIST_CLASSPATH = os.path.join(build_dir, "jars", "*")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't this going to add both scala jar 2.11 and 2.12 into class path at the same time?

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Apr 23, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for review, @felixcheung . We assume that the Spark developers don't build both profiles without cleanup. This is a standard Spark way used since Spark 2.0.0.

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Apr 23, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, as we see here, although we officially deprecated Scala-2.11 at Spark 2.4.1, it seems that we don't test Scala-2.12 actually during Spark 2.4.2 vote process. The community is still on Scala-2.11. It's time to switch branch-2.4 to Scala-2.12 if possible.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on this. For next 2.4.3, we should have branch-2.4 default to Scala 2.12 as we will remove Scala 2.11 support in Spark 3.0

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @dbtsai . +1 for that

@SparkQA
Copy link

SparkQA commented Apr 23, 2019

Test build #104825 has finished for PR 24439 at commit 5eba36a.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

Retest this please.

@SparkQA
Copy link

SparkQA commented Apr 23, 2019

Test build #104827 has finished for PR 24439 at commit 5eba36a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

Merged to branch-2.4. Thank you for review, @srowen .

dongjoon-hyun added a commit that referenced this pull request Apr 23, 2019
@dongjoon-hyun
[SPARK-27544][PYTHON][TEST][BRANCH-2.4] Fix Python test script to wor…
b615f22 
…k on Scala-2.12 build

## What changes were proposed in this pull request?

Since [SPARK-27274](https://issues.apache.org/jira/browse/SPARK-27274) deprecated Scala-2.11 at Spark 2.4.1, we need to test Scala-2.12 more. This PR aims to fix the Python test script on Scala-2.12 build in `branch-2.4`.

**BEFORE**
```
$ dev/change-scala-version.sh 2.12

$ build/sbt -Pscala-2.12 package

$ python/run-tests.py --python-executables python2.7 --modules pyspark-sql
Traceback (most recent call last):
  File "python/run-tests.py", line 70, in <module>
    raise Exception("Cannot find assembly build directory, please build Spark first.")
Exception: Cannot find assembly build directory, please build Spark first.
```

**AFTER**
```
$ python/run-tests.py --python-executables python2.7 --modules pyspark-sql
Running PySpark tests. Output is in /Users/dongjoon/APACHE/spark/python/unit-tests.log
Will test against the following Python executables: ['python2.7']
Will test the following Python modules: ['pyspark-sql']
Starting test(python2.7): pyspark.sql.tests
...
```

## How was this patch tested?

Manually do the above procedure because Jenkins doesn't test Scala-2.12 in `branch-2.4`.

Closes #24439 from dongjoon-hyun/SPARK-27544.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
@dongjoon-hyun dongjoon-hyun deleted the SPARK-27544 branch April 23, 2019 14:09
HyukjinKwon
HyukjinKwon reviewed Apr 23, 2019
View reviewed changes
Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Late lgtm

kai-chi pushed a commit to kai-chi/spark that referenced this pull request Jul 23, 2019
@dongjoon-hyun @kai-chi
[SPARK-27544][PYTHON][TEST][BRANCH-2.4] Fix Python test script to wor…
4a8d06b 
…k on Scala-2.12 build

## What changes were proposed in this pull request?

Since [SPARK-27274](https://issues.apache.org/jira/browse/SPARK-27274) deprecated Scala-2.11 at Spark 2.4.1, we need to test Scala-2.12 more. This PR aims to fix the Python test script on Scala-2.12 build in `branch-2.4`.

**BEFORE**
```
$ dev/change-scala-version.sh 2.12

$ build/sbt -Pscala-2.12 package

$ python/run-tests.py --python-executables python2.7 --modules pyspark-sql
Traceback (most recent call last):
  File "python/run-tests.py", line 70, in <module>
    raise Exception("Cannot find assembly build directory, please build Spark first.")
Exception: Cannot find assembly build directory, please build Spark first.
```

**AFTER**
```
$ python/run-tests.py --python-executables python2.7 --modules pyspark-sql
Running PySpark tests. Output is in /Users/dongjoon/APACHE/spark/python/unit-tests.log
Will test against the following Python executables: ['python2.7']
Will test the following Python modules: ['pyspark-sql']
Starting test(python2.7): pyspark.sql.tests
...
```

## How was this patch tested?

Manually do the above procedure because Jenkins doesn't test Scala-2.12 in `branch-2.4`.

Closes apache#24439 from dongjoon-hyun/SPARK-27544.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
kai-chi pushed a commit to kai-chi/spark that referenced this pull request Jul 25, 2019
@dongjoon-hyun @kai-chi
[SPARK-27544][PYTHON][TEST][BRANCH-2.4] Fix Python test script to wor…
e5f352a 
…k on Scala-2.12 build

## What changes were proposed in this pull request?

Since [SPARK-27274](https://issues.apache.org/jira/browse/SPARK-27274) deprecated Scala-2.11 at Spark 2.4.1, we need to test Scala-2.12 more. This PR aims to fix the Python test script on Scala-2.12 build in `branch-2.4`.

**BEFORE**
```
$ dev/change-scala-version.sh 2.12

$ build/sbt -Pscala-2.12 package

$ python/run-tests.py --python-executables python2.7 --modules pyspark-sql
Traceback (most recent call last):
  File "python/run-tests.py", line 70, in <module>
    raise Exception("Cannot find assembly build directory, please build Spark first.")
Exception: Cannot find assembly build directory, please build Spark first.
```

**AFTER**
```
$ python/run-tests.py --python-executables python2.7 --modules pyspark-sql
Running PySpark tests. Output is in /Users/dongjoon/APACHE/spark/python/unit-tests.log
Will test against the following Python executables: ['python2.7']
Will test the following Python modules: ['pyspark-sql']
Starting test(python2.7): pyspark.sql.tests
...
```

## How was this patch tested?

Manually do the above procedure because Jenkins doesn't test Scala-2.12 in `branch-2.4`.

Closes apache#24439 from dongjoon-hyun/SPARK-27544.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
kai-chi pushed a commit to kai-chi/spark that referenced this pull request Aug 1, 2019
@dongjoon-hyun @kai-chi
[SPARK-27544][PYTHON][TEST][BRANCH-2.4] Fix Python test script to wor…
2da67de 
…k on Scala-2.12 build

## What changes were proposed in this pull request?

Since [SPARK-27274](https://issues.apache.org/jira/browse/SPARK-27274) deprecated Scala-2.11 at Spark 2.4.1, we need to test Scala-2.12 more. This PR aims to fix the Python test script on Scala-2.12 build in `branch-2.4`.

**BEFORE**
```
$ dev/change-scala-version.sh 2.12

$ build/sbt -Pscala-2.12 package

$ python/run-tests.py --python-executables python2.7 --modules pyspark-sql
Traceback (most recent call last):
  File "python/run-tests.py", line 70, in <module>
    raise Exception("Cannot find assembly build directory, please build Spark first.")
Exception: Cannot find assembly build directory, please build Spark first.
```

**AFTER**
```
$ python/run-tests.py --python-executables python2.7 --modules pyspark-sql
Running PySpark tests. Output is in /Users/dongjoon/APACHE/spark/python/unit-tests.log
Will test against the following Python executables: ['python2.7']
Will test the following Python modules: ['pyspark-sql']
Starting test(python2.7): pyspark.sql.tests
...
```

## How was this patch tested?

Manually do the above procedure because Jenkins doesn't test Scala-2.12 in `branch-2.4`.

Closes apache#24439 from dongjoon-hyun/SPARK-27544.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dbtsai dbtsai dbtsai left review comments

@HyukjinKwon HyukjinKwon HyukjinKwon left review comments

@felixcheung felixcheung felixcheung left review comments

@srowen srowen srowen approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@dongjoon-hyun @SparkQA @srowen @dbtsai @HyukjinKwon @felixcheung