[SPARK-40801][BUILD][3.2] Upgrade Apache commons-text to 1.10

[bjornjorgensen]

What changes were proposed in this pull request?

Upgrade Apache commons-text from 1.6 to 1.10.0

Why are the changes needed?

CVE-2022-42889
this is a 9.8 CRITICAL

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA

[bjornjorgensen]

@wangyum

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

This is probably fine; a bigger upgrade from 1.6 but I don't have reason to believe it's a problem in 3.2 and test pass

[srowen]

Hm, the test seems to be stuck

[bjornjorgensen]

@srowen Yes, the same thing happened with this one too.. #38098

Right now it's failing the time tests.
And the python test is failing. But that is not because of this upgrade.

[bjornjorgensen]

@srowen I did rerun the failed tests now and now they pass.

But there are 2 python tests that don't pass,
-The python3.9 -m black command was not found
and
-ImportError: Missing optional dependency 'Jinja2'

[srowen]

Huh, that also seems unrelated. Let's hold onto this for a day or two and then rerun if needed

[dongjoon-hyun]

cc @sunchao since he is the release manager of Apache Spark 3.2.3.

[bjornjorgensen]

@srowen two days have passed.

[srowen]

The test still shows 'pending', hm. It sounds like you saw some possibly-unrelated tests failing. Ideally we'd see the tests pass first. Can you re-set or re-run the tests?

[bjornjorgensen]

ok, I re-run the tests now.

[bjornjorgensen]

@srowen tests have been re-run but it's the same result.
But it's the same with everyone else's PR, for a long time.
Like this one

[bjornjorgensen]

@xinrong-meng There are two tests that don't work for branch 3.2
Those are both python tests, can you have a look at them?

[vitas]

Could you make the same patch for 3.1 branch?

[dongjoon-hyun]

Could you make the same patch for 3.1 branch?

No, Apache Spark 3.1 reached EOL last month because the first release was March 2, 2021, @vitas .

[itholic]

Looks good when the test pass

[sithmein]

Is Spark actually affected by the problem in the StringLookup class or is this dependency update only to get rid of all the alarms raised by code scanners?

[srowen]

The latter.

[srowen]

Hm, maybe we can try the tests 1 more time. I'm inclined to merge for the 3.2 RC as I think the test failures are unrelated.

[bjornjorgensen]

I can restart the tests, but everyone else have the same problem in branch3.2

[bjornjorgensen]

https://github.com/bjornjorgensen/spark/actions/runs/3304235420

[bjornjorgensen]

@srowen Will you merge this?

[srowen]

Well, I don't know if it passes - it probably does, but other errors are preventing it. I also don't know if this even affects Spark. If any other committer would endorse the 'override' I'd merge.

[bjornjorgensen]

Yes, there are 2 python tests that fail, but the tests for java are running and those are OK. 
For this PR there are 2 @wangyum and @dongjoon-hyun that have approved this. Do you need any more? 

[dongjoon-hyun]

We have been waiting for the release manager's approval, @bjornjorgensen , because the release preparation was already started. So, it would be great that @sunchao reviews and merges this as a part of 3.2.3 release preparation.

• [SPARK-40801][BUILD][3.2] Upgrade Apache commons-text to 1.10 #38352 (comment)

[sunchao]

I'm actually holding 3.2.3 for this PR. Once it's merged I'll start the release process.

[srowen]

OK, merged to 3.2

[bjornjorgensen]

Thanks everyone :)

[bsikander]

@sunchao thank you for you efforts. When can we expect the release of 3.2.3?

[sunchao]

I'm going to start working on it this week.

[fryz]

Apologies if this is discussed elsewhere (or an FAQ that I wasn't able to find), but is there somewhere I can subscribe to this release and get notified when it's ready to pick up? Thanks!

[bjornjorgensen]

@fryz It will be posted at dev@spark.apache.org and user@spark.apache.org

[bsikander]

@sunchao @bjornjorgensen any update on this release?
As internal alarms are going off continuously, i am desperately looking for the release.

[sunchao]

@bsikander again, pls check dev@spark.apache.org - it's being voted.

[dongjoon-hyun]

+1 for @sunchao 's comment.
To @bsikander , it would be great if you can participate [VOTE] Release Spark 3.2.3 (RC1).

[srowen]

I'm curious what the urgency is @bsikander - do you have a theory that this even affects Spark? this is a 'just in case' and 'to silence automated warnings' kind of update as far as I can see

[bsikander]

@srowen i also don't see any affect on Spark. My goal is to silence the warnings as soon as possible. That is why i am waiting for the release :)

[bsikander]

@bjornjorgensen I noticed that you updated pyspark-notebook docker file in docker-stacks. I am looking for pyspark-notebook docker image with this fix inside. Is there a pyspark-notebook image available with spark 3.2.3?