ARROW-18390: [CI][Python] Update spark test modules to match spark master #14715

@jorisvandenbossche what do you think about this to fix the nightly failures? I am not sure in which version of spark this will be released, I suppose if we decide to add newer versions of spark to the CI matrix that contain this change we can update the if to match the newer versions but this currently fixes the issue.

kou · 2022-11-23T19:36:52Z

@kiszk You may want to review this as an Apache Spark committer.

kiszk · 2022-11-24T02:51:45Z

@raulcd Thank you. Almost looks good to me.

Can we enhance the decision logic of SPARK_VERSION? If version is (<= 2 or) <= 3.3, we use the current package names. Otherwise, we use the new package names. As a result, we could avoid further changes for a while.

jorisvandenbossche · 2022-11-24T08:52:04Z

I would personally also be fine with just merging this as is and handling updating it once it start failing again when those test changes are released, but if there is an easy way to check the actual spark version, that sounds good as well.

raulcd · 2022-11-24T10:19:11Z

@github-actions crossbow submit test-conda-python--spark-

github-actions · 2022-11-24T10:19:54Z

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/3539688563

AlenkaF · 2022-11-24T10:48:38Z

@github-actions crossbow submit test-conda-python--spark-

github-actions · 2022-11-24T10:51:13Z

Revision: dde4e85

Submitted crossbow builds: ursacomputing/crossbow @ actions-e62a5efe73

Task	Status
test-conda-python-3.7-spark-v3.1.2
test-conda-python-3.8-spark-v3.2.0
test-conda-python-3.9-spark-master

AlenkaF · 2022-11-24T14:01:55Z

@github-actions crossbow submit test-conda-python--spark-

github-actions · 2022-11-24T14:04:24Z

Revision: dad621d

Submitted crossbow builds: ursacomputing/crossbow @ actions-f76e4f71d1

Task	Status
test-conda-python-3.7-spark-v3.1.2
test-conda-python-3.8-spark-v3.2.0
test-conda-python-3.9-spark-master

…igger bot tasks (#14727) I've been added as triager but now if I try to trigger crossbow tasks they fail with permissions issues: #14715 (comment) ``` Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in. The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/3539688563 ``` This should fix the issue Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>

kiszk · 2022-11-24T14:43:55Z

nit: if [ ${old_test_modules} == "true" ]; then -> if [ "${old_test_modules}" == "true" ]; then

kiszk · 2022-11-24T18:37:16Z

@github-actions crossbow submit test-conda-python--spark-

github-actions · 2022-11-24T18:38:32Z

Unable to match any tasks for `test-conda-python--spark-`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/3542999264

raulcd · 2022-11-24T18:46:27Z

@github-actions crossbow submit test-conda-python--spark-

raulcd · 2022-11-24T18:48:31Z

@kiszk GitHub markdown is removing the required *, should be: @github-actions crossbow submit test-conda-python-*-spark-*. I might have permissions again :) trying

github-actions · 2022-11-24T18:49:03Z

Revision: 72089d5

Submitted crossbow builds: ursacomputing/crossbow @ actions-f0e59191d5

Task	Status
test-conda-python-3.7-spark-v3.1.2
test-conda-python-3.8-spark-v3.2.0
test-conda-python-3.9-spark-master

kiszk · 2022-11-24T19:13:10Z

@raulcd The problem seems to be a format of SPARK_VERSION.

https://github.com/ursacomputing/crossbow/actions/runs/3539915127/jobs/5942357685#step:5:7657 implies SPARK_VERSION is v3.1.2, not 3.1.2. On the other hand, the current integration_spark.sh assumes that the format is 3.1.2.

IMHO, we should take a major version (e.g. 3.) by using ${SPARK_VERSION:1:2}.

ci/scripts/integration_spark.sh

raulcd · 2022-11-25T12:19:44Z

@github-actions crossbow submit test-conda-python--spark-

github-actions · 2022-11-25T12:22:28Z

Revision: 2a87421

Submitted crossbow builds: ursacomputing/crossbow @ actions-cf8c73ef1b

Task	Status
test-conda-python-3.7-spark-v3.1.2
test-conda-python-3.8-spark-v3.2.0
test-conda-python-3.9-spark-master

raulcd · 2022-11-25T12:23:47Z

@raulcd The problem seems to be a format of SPARK_VERSION.

@kiszk you are correct. I got confused by the other check on the script which seems to be wrong then:

if [ "${SPARK_VERSION:0:2}" == "2." ]; then
  # https://github.com/apache/spark/blob/master/docs/sql-pyspark-pandas-with-arrow.md#compatibility-setting-for-pyarrow--0150-and-spark-23x-24x
  export ARROW_PRE_0_15_IPC_FORMAT=1
fi

I've updated to:


if [ "${SPARK_VERSION:1:2}" == "2." ]; then

even though it was not used as we don't have any nightly test for versions lower than 3.1.2.

raulcd · 2022-11-25T14:36:46Z

@kiszk @kou this should be ready now, see the last builds from crossbow. Thanks for your help.

kou

+1

ursabot · 2022-11-26T20:41:43Z

Benchmark runs are scheduled for baseline = 2078af7 and contender = 8a93741. 8a93741 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.81% ⬆️0.5%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.73% ⬆️1.08%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 8a937413 ec2-t3-xlarge-us-east-2
[Finished] 8a937413 test-mac-arm
[Finished] 8a937413 ursa-i9-9960x
[Finished] 8a937413 ursa-thinkcentre-m75q
[Finished] 2078af7c ec2-t3-xlarge-us-east-2
[Finished] 2078af7c test-mac-arm
[Finished] 2078af7c ursa-i9-9960x
[Finished] 2078af7c ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ARROW-18390: [CI][Python] Remove testing missing module from spark ma…

6562e07

…ster

Fix all the modules based on spark test modules refactoring

9bb13f6

raulcd changed the title ~~ARROW-18390: [CI][Python] Remove testing missing module from spark master~~ ARROW-18390: [CI][Python] Update spark test modules to match spark master Nov 23, 2022

Add logic to use old modules only for Spark <= 2.x or <= 3.3.x

dde4e85

raulcd mentioned this pull request Nov 24, 2022

MINOR: [Archery] Add collaborators to list of roles with access to trigger bot tasks #14727

Merged

Fix conditions

dad621d

Fix review comment

72089d5

kou reviewed Nov 25, 2022

View reviewed changes

ci/scripts/integration_spark.sh Outdated Show resolved Hide resolved

Fix SPARK_VERSION check as suggested on review

2a87421

kiszk approved these changes Nov 25, 2022

View reviewed changes

kou approved these changes Nov 25, 2022

View reviewed changes

kou merged commit 8a93741 into apache:master Nov 25, 2022

asfimport mentioned this pull request Nov 26, 2022

[CI][Python] Nightly python test for spark master missing test module #33554

Closed

ARROW-18390: [CI][Python] Update spark test modules to match spark master #14715

ARROW-18390: [CI][Python] Update spark test modules to match spark master #14715

Uh oh!

Conversation

raulcd commented Nov 23, 2022

Uh oh!

github-actions bot commented Nov 23, 2022

Uh oh!

raulcd commented Nov 23, 2022

Uh oh!

github-actions bot commented Nov 23, 2022

Uh oh!

raulcd commented Nov 23, 2022

Uh oh!

raulcd commented Nov 23, 2022

Uh oh!

github-actions bot commented Nov 23, 2022

Uh oh!

raulcd commented Nov 23, 2022

Uh oh!

kou commented Nov 23, 2022

Uh oh!

kiszk commented Nov 24, 2022

Uh oh!

jorisvandenbossche commented Nov 24, 2022

Uh oh!

raulcd commented Nov 24, 2022

Uh oh!

github-actions bot commented Nov 24, 2022

Uh oh!

AlenkaF commented Nov 24, 2022

Uh oh!

github-actions bot commented Nov 24, 2022

Uh oh!

AlenkaF commented Nov 24, 2022

Uh oh!

github-actions bot commented Nov 24, 2022

Uh oh!

kiszk commented Nov 24, 2022

Uh oh!

kiszk commented Nov 24, 2022

Uh oh!

github-actions bot commented Nov 24, 2022

Uh oh!

raulcd commented Nov 24, 2022

Uh oh!

raulcd commented Nov 24, 2022

Uh oh!

github-actions bot commented Nov 24, 2022

Uh oh!

kiszk commented Nov 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

raulcd commented Nov 25, 2022

Uh oh!

github-actions bot commented Nov 25, 2022

Uh oh!

raulcd commented Nov 25, 2022

Uh oh!

raulcd commented Nov 25, 2022

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

ursabot commented Nov 26, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

kiszk commented Nov 24, 2022 •

edited

Loading