Skip to content

Commit b1fd03a

Browse files
HyukjinKwonHyukjinKwon
committed
[SPARK-32245][INFRA] Run Spark tests in Github Actions
This PR aims to run the Spark tests in Github Actions. To briefly explain the main idea: - Reuse `dev/run-tests.py` with SBT build - Reuse the modules in `dev/sparktestsupport/modules.py` to test each module - Pass the modules to test into `dev/run-tests.py` directly via `TEST_ONLY_MODULES` environment variable. For example, `pyspark-sql,core,sql,hive`. - `dev/run-tests.py` _does not_ take the dependent modules into account but solely the specified modules to test. Another thing to note might be `SlowHiveTest` annotation. Running the tests in Hive modules takes too much so the slow tests are extracted and it runs as a separate job. It was extracted from the actual elapsed time in Jenkins: ![Screen Shot 2020-07-09 at 7 48 13 PM](https://user-images.githubusercontent.com/6477701/87050238-f6098e80-c238-11ea-9c4a-ab505af61381.png) So, Hive tests are separated into to jobs. One is slow test cases, and the other one is the other test cases. _Note that_ the current GitHub Actions build virtually copies what the default PR builder on Jenkins does (without other profiles such as JDK 11, Hadoop 2, etc.). The only exception is Kinesis https://github.com/apache/spark/pull/29057/files#diff-04eb107ee163a50b61281ca08f4e4c7bR23 Last week and onwards, the Jenkins machines became very unstable for many reasons: - Apparently, the machines became extremely slow. Almost all tests can't pass. - One machine (worker 4) started to have the corrupt `.m2` which fails the build. - Documentation build fails time to time for an unknown reason in Jenkins machine specifically. This is disabled for now at apache#29017. - Almost all PRs are basically blocked by this instability currently. The advantages of using Github Actions: - To avoid depending on few persons who can access to the cluster. - To reduce the elapsed time in the build - we could split the tests (e.g., SQL, ML, CORE), and run them in parallel so the total build time will significantly reduce. - To control the environment more flexibly. - Other contributors can test and propose to fix Github Actions configurations so we can distribute this build management cost. Note that: - The current build in Jenkins takes _more than 7 hours_. With Github actions it takes _less than 2 hours_ - We can now control the environments especially for Python easily. - The test and build look more stable than the Jenkins'. No, dev-only change. Tested at #4 Closes apache#29057 from HyukjinKwon/migrate-to-github-actions. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 34544d6 commit b1fd03a

File tree

19 files changed

+430
-159
lines changed

19 files changed

+430
-159
lines changed

‎.github/workflows/branch-2.4.yml‎

Lines changed: 0 additions & 104 deletions
This file was deleted.

‎.github/workflows/master.yml‎

Lines changed: 213 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
name: master
2+
3+
on:
4+
pull_request:
5+
branches:
6+
- branch-2.4
7+
8+
jobs:
9+
# TODO(SPARK-32248): Recover JDK 11 builds
10+
# Build: build Spark and run the tests for specified modules.
11+
build:
12+
name: "Build modules: ${{ matrix.modules }} ${{ matrix.comment }} (JDK ${{ matrix.java }}, ${{ matrix.hadoop }})"
13+
runs-on: ubuntu-latest
14+
strategy:
15+
fail-fast: false
16+
matrix:
17+
java:
18+
- 1.8
19+
hadoop:
20+
- hadoop2.6
21+
# TODO(SPARK-32246): We don't test 'streaming-kinesis-asl' for now.
22+
# Kinesis tests depends on external Amazon kinesis service.
23+
# Note that the modules below are from sparktestsupport/modules.py.
24+
modules:
25+
- |-
26+
core, unsafe, kvstore, avro,
27+
network_common, network_shuffle, repl, launcher
28+
examples, sketch, graphx
29+
- |-
30+
catalyst, hive-thriftserver
31+
- |-
32+
streaming, sql-kafka-0-10, streaming-kafka-0-10,
33+
mllib-local, mllib,
34+
yarn, mesos, kubernetes, hadoop-cloud, spark-ganglia-lgpl,
35+
streaming-flume, streaming-flume-sink, streaming-kafka-0-8
36+
- |-
37+
pyspark-sql, pyspark-mllib
38+
- |-
39+
pyspark-core, pyspark-streaming, pyspark-ml
40+
- |-
41+
sparkr
42+
- >-
43+
sql
44+
# Here, we split Hive and SQL tests into some of slow ones and the rest of them.
45+
included-tags: [""]
46+
excluded-tags: [""]
47+
comment: [""]
48+
include:
49+
# Hive tests
50+
- modules: hive
51+
java: 1.8
52+
hadoop: hadoop2.6
53+
included-tags: org.apache.spark.tags.SlowHiveTest
54+
comment: "- slow tests"
55+
- modules: hive
56+
java: 1.8
57+
hadoop: hadoop2.6
58+
excluded-tags: org.apache.spark.tags.SlowHiveTest
59+
comment: "- other tests"
60+
env:
61+
TEST_ONLY_MODULES: ${{ matrix.modules }}
62+
TEST_ONLY_EXCLUDED_TAGS: ${{ matrix.excluded-tags }}
63+
TEST_ONLY_INCLUDED_TAGS: ${{ matrix.included-tags }}
64+
HADOOP_PROFILE: ${{ matrix.hadoop }}
65+
# GitHub Actions' default miniconda to use in pip packaging test.
66+
CONDA_PREFIX: /usr/share/miniconda
67+
ARROW_PRE_0_15_IPC_FORMAT: 1
68+
steps:
69+
- name: Checkout Spark repository
70+
uses: actions/checkout@v2
71+
# Cache local repositories. Note that GitHub Actions cache has a 2G limit.
72+
- name: Cache Scala, SBT, Maven and Zinc
73+
uses: actions/cache@v1
74+
with:
75+
path: build
76+
key: build-${{ hashFiles('**/pom.xml') }}
77+
restore-keys: |
78+
build-
79+
- name: Cache Maven local repository
80+
uses: actions/cache@v2
81+
with:
82+
path: ~/.m2/repository
83+
key: ${{ matrix.java }}-${{ matrix.hadoop }}-maven-${{ hashFiles('**/pom.xml') }}
84+
restore-keys: |
85+
${{ matrix.java }}-${{ matrix.hadoop }}-maven-
86+
- name: Cache Ivy local repository
87+
uses: actions/cache@v2
88+
with:
89+
path: ~/.ivy2/cache
90+
key: ${{ matrix.java }}-${{ matrix.hadoop }}-ivy-${{ hashFiles('**/pom.xml') }}-${{ hashFiles('**/plugins.sbt') }}
91+
restore-keys: |
92+
${{ matrix.java }}-${{ matrix.hadoop }}-ivy-
93+
- name: Install JDK ${{ matrix.java }}
94+
uses: actions/setup-java@v1
95+
with:
96+
java-version: ${{ matrix.java }}
97+
# PySpark
98+
- name: Install PyPy3
99+
# SQL component also has Python related tests, for example, IntegratedUDFTestUtils.
100+
# Note that order of Python installations here matters because default python3 is
101+
# overridden by pypy3.
102+
uses: actions/setup-python@v2
103+
if: contains(matrix.modules, 'pyspark') || (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-'))
104+
with:
105+
python-version: pypy3
106+
architecture: x64
107+
- name: Install Python 3.6
108+
uses: actions/setup-python@v2
109+
if: contains(matrix.modules, 'pyspark') || (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-'))
110+
with:
111+
python-version: 3.6
112+
architecture: x64
113+
- name: Install Python 2.7
114+
uses: actions/setup-python@v2
115+
if: contains(matrix.modules, 'pyspark') || (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-'))
116+
with:
117+
python-version: 2.7
118+
architecture: x64
119+
- name: Install Python packages
120+
if: contains(matrix.modules, 'pyspark') || (contains(matrix.modules, 'sql') && !contains(matrix.modules, 'sql-'))
121+
# PyArrow is not supported in PyPy yet, see ARROW-2651.
122+
# TODO(SPARK-32247): scipy installation with PyPy fails for an unknown reason.
123+
run: |
124+
python3 -m pip install numpy pyarrow pandas scipy
125+
python3 -m pip list
126+
python2 -m pip install numpy pandas scipy
127+
python2 -m pip list
128+
pypy3 -m pip install numpy pandas
129+
pypy3 -m pip list
130+
# SparkR
131+
- name: Install R 3.6
132+
uses: r-lib/actions/setup-r@v1
133+
if: contains(matrix.modules, 'sparkr')
134+
with:
135+
r-version: 3.6
136+
- name: Install R packages
137+
if: contains(matrix.modules, 'sparkr')
138+
run: |
139+
sudo apt-get install -y libcurl4-openssl-dev
140+
sudo Rscript -e "install.packages(c('knitr', 'rmarkdown', 'testthat', 'devtools', 'e1071', 'survival', 'arrow', 'roxygen2'), repos='https://cloud.r-project.org/')"
141+
# Show installed packages in R.
142+
sudo Rscript -e 'pkg_list <- as.data.frame(installed.packages()[, c(1,3:4)]); pkg_list[is.na(pkg_list$Priority), 1:2, drop = FALSE]'
143+
# Run the tests.
144+
- name: "Run tests: ${{ matrix.modules }}"
145+
run: |
146+
# Hive tests become flaky when running in parallel as it's too intensive.
147+
if [[ "$TEST_ONLY_MODULES" == "hive" ]]; then export SERIAL_SBT_TESTS=1; fi
148+
mkdir -p ~/.m2
149+
./dev/run-tests --parallelism 2
150+
rm -rf ~/.m2/repository/org/apache/spark
151+
152+
# Static analysis, and documentation build
153+
lint:
154+
name: Linters, licenses, dependencies and documentation generation
155+
runs-on: ubuntu-latest
156+
steps:
157+
- name: Checkout Spark repository
158+
uses: actions/checkout@v2
159+
- name: Cache Maven local repository
160+
uses: actions/cache@v2
161+
with:
162+
path: ~/.m2/repository
163+
key: docs-maven-repo-${{ hashFiles('**/pom.xml') }}
164+
restore-keys: |
165+
docs-maven-
166+
- name: Install JDK 1.8
167+
uses: actions/setup-java@v1
168+
with:
169+
java-version: 1.8
170+
- name: Install Python 3.6
171+
uses: actions/setup-python@v2
172+
with:
173+
python-version: 3.6
174+
architecture: x64
175+
- name: Install Python linter dependencies
176+
run: |
177+
pip3 install flake8 sphinx numpy
178+
- name: Install R 3.6
179+
uses: r-lib/actions/setup-r@v1
180+
with:
181+
r-version: 3.6
182+
- name: Install R linter dependencies and SparkR
183+
run: |
184+
sudo apt-get install -y libcurl4-openssl-dev
185+
sudo Rscript -e "install.packages(c('devtools'), repos='https://cloud.r-project.org/')"
186+
sudo Rscript -e "devtools::install_github('jimhester/lintr@v2.0.0')"
187+
./R/install-dev.sh
188+
- name: Install Ruby 2.7 for documentation generation
189+
uses: actions/setup-ruby@v1
190+
with:
191+
ruby-version: 2.7
192+
- name: Install dependencies for documentation generation
193+
run: |
194+
sudo apt-get install -y libcurl4-openssl-dev pandoc
195+
pip install sphinx mkdocs numpy
196+
gem install jekyll jekyll-redirect-from pygments.rb
197+
sudo Rscript -e "install.packages(c('devtools', 'testthat', 'knitr', 'rmarkdown', 'roxygen2'), repos='https://cloud.r-project.org/')"
198+
- name: Scala linter
199+
run: ./dev/lint-scala
200+
- name: Java linter
201+
run: ./dev/lint-java
202+
- name: Python linter
203+
run: ./dev/lint-python
204+
- name: R linter
205+
run: ./dev/lint-r
206+
- name: License test
207+
run: ./dev/check-license
208+
- name: Dependencies test
209+
run: ./dev/test-dependencies.sh
210+
- name: Run documentation build
211+
run: |
212+
cd docs
213+
jekyll build

‎common/tags/src/test/java/org/apache/spark/tags/SlowHiveTest.java‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
package org.apache.spark.tags;
19+
20+
import org.scalatest.TagAnnotation;
21+
22+
import java.lang.annotation.ElementType;
23+
import java.lang.annotation.Retention;
24+
import java.lang.annotation.RetentionPolicy;
25+
import java.lang.annotation.Target;
26+
27+
@TagAnnotation
28+
@Retention(RetentionPolicy.RUNTIME)
29+
@Target({ElementType.METHOD, ElementType.TYPE})
30+
public @interface SlowHiveTest { }

‎core/src/test/scala/org/apache/spark/scheduler/BarrierTaskContextSuite.scala‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,8 @@ class BarrierTaskContextSuite extends SparkFunSuite with LocalSparkContext {
153153
assert(error.contains("within 1 second(s)"))
154154
}
155155

156-
test("SPARK-31485: barrier stage should fail if only partial tasks are launched") {
156+
// Disabled as it is flaky in GitHub Actions.
157+
ignore("SPARK-31485: barrier stage should fail if only partial tasks are launched") {
157158
val conf = new SparkConf()
158159
.setMaster("local-cluster[2, 1, 1024]")
159160
.setAppName("test-cluster")

‎dev/run-pip-tests‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ fi
6868
PYSPARK_VERSION=$(python3 -c "exec(open('python/pyspark/version.py').read());print(__version__)")
6969
PYSPARK_DIST="$FWDIR/python/dist/pyspark-$PYSPARK_VERSION.tar.gz"
7070
# The pip install options we use for all the pip commands
71-
PIP_OPTIONS="--upgrade --no-cache-dir --force-reinstall "
71+
PIP_OPTIONS="--user --upgrade --no-cache-dir --force-reinstall "
7272
# Test both regular user and edit/dev install modes.
7373
PIP_COMMANDS=("pip install $PIP_OPTIONS $PYSPARK_DIST"
7474
"pip install $PIP_OPTIONS -e python/")
@@ -81,8 +81,12 @@ for python in "${PYTHON_EXECS[@]}"; do
8181
VIRTUALENV_PATH="$VIRTUALENV_BASE"/$python
8282
rm -rf "$VIRTUALENV_PATH"
8383
if [ -n "$USE_CONDA" ]; then
84+
if [ -f "$CONDA_PREFIX/etc/profile.d/conda.sh" ]; then
85+
# See also https://github.com/conda/conda/issues/7980
86+
source "$CONDA_PREFIX/etc/profile.d/conda.sh"
87+
fi
8488
conda create -y -p "$VIRTUALENV_PATH" python=$python numpy pandas pip setuptools
85-
source activate "$VIRTUALENV_PATH"
89+
conda activate "$VIRTUALENV_PATH" || (echo "Falling back to 'source activate'" && source activate "$VIRTUALENV_PATH")
8690
else
8791
mkdir -p "$VIRTUALENV_PATH"
8892
virtualenv --python=$python "$VIRTUALENV_PATH"
@@ -115,6 +119,7 @@ for python in "${PYTHON_EXECS[@]}"; do
115119
cd /
116120

117121
echo "Run basic sanity check on pip installed version with spark-submit"
122+
export PATH="$(python3 -m site --user-base)/bin:$PATH"
118123
spark-submit "$FWDIR"/dev/pip-sanity-check.py
119124
echo "Run basic sanity check with import based"
120125
python "$FWDIR"/dev/pip-sanity-check.py
@@ -125,7 +130,7 @@ for python in "${PYTHON_EXECS[@]}"; do
125130

126131
# conda / virtualenv environments need to be deactivated differently
127132
if [ -n "$USE_CONDA" ]; then
128-
source deactivate
133+
conda deactivate || (echo "Falling back to 'source deactivate'" && source deactivate)
129134
else
130135
deactivate
131136
fi

0 commit comments

Comments
 (0)