[SPARK-28921][BUILD][K8S] Upgrade kubernetes client to 4.4.2

[andygrove]

What changes were proposed in this pull request?

Upgrade kubernetes client from 4.1.2 to 4.4.2

Why are the changes needed?

To fix compatibility issue with EKS since Amazon rolled out some security patches over the past week; 1.15.3, 1.14.6, 1.13.10, 1.12.10, and 1.11.10.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass the Jenkins and manually test on EKS.

[srowen]

@andygrove any possible incompatibilities or gotchas that might arise from the update? I suspect it's fine.

[SparkQA]

Test build #4848 has finished for PR 25640 at commit db8db6f.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Oh, you'll need to run ./dev/test-dependencies.sh --replace-manifest

[andygrove]

@andygrove any possible incompatibilities or gotchas that might arise from the update? I suspect it's fine.

I suspect it is fine too. kubernetes client 4.4.2 has jackson 2.9.9 instead of jackson 2.9.8 with the 4.1.2 release, but hopefully that's a non issue

[srowen]

Should be fine. We're on 2.9.9.x in Spark.

[dongjoon-hyun]

Retest this please.

[dongjoon-hyun]

+1, LGTM. (Pending Jenkins. I also triggered JDK11 test to make it sure).
Thank you so much for the fix, @andygrove .

[dongjoon-hyun]

Also, if possible, please specify the version number of EKS which you are mentioning, @andygrove . latest becomes obsolete very soon.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/15008/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/15008/

[SparkQA]

Test build #4855 has finished for PR 25640 at commit 9a29d0b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #109989 has finished for PR 25640 at commit 9a29d0b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

cc @skonto , too.

[andygrove]

Also, if possible, please specify the version number of EKS which you are mentioning, @andygrove . latest becomes obsolete very soon.

I updated the JIRA title. So far, this has been reported against versions 1.15.3, 1.14.6, 1.13.10, 1.12.10, and 1.11.10.

[felixcheung]

looks reasonable to me

[dongjoon-hyun]

Hi, @andygrove . For Spark testing environments, it looks good. I'd like to verify this PR in EKS since you reported this for that. When I check the EKS today, it seems to show 1.11, 1.12, and 1.13 without the details. How did you test this PR manually, @andygrove ? Could you describe some into the PR description for the other reviewers?

BTW, to understand your procedure, I visited your interesting blog article. I have one question about your article; Why do you think this is the existing Apache Spark's 2.3/2.4 regression? Those releases work with the old K8S releases and still do. Did I miss something? In Apache Arrow project, do you think some 3rd party library's future release is considered as the existing Apache Arrow library's regression? For me, it's just unsupported version issue, isn't it?

• https://andygrove.io/2019/08/apache-spark-regressions-eks/

According to your last sentence, I'm waiting for your update on that blog.

I’ll update this post after the weekend once I’ve had time to verify how well this works.

[andygrove]

@dongjoon-hyun The regression is with EKS, not with Spark ... I guess I could have picked a better title for the blog post. I'll update that.

Once connected to the EKS cluster, I run kubectl version to see the detailed version information.

The initial testing I'm doing is just running our automated integration test suite for the service that my team works on, and this causes Spark jobs to run in EKS. Basically we're just running spark jobs using spark-submit though, and making sure they run to completion.

It's Labor day in the US today so I won't be back at work until tomorrow, but we have tests passing now because I followed the instructions in the blog post and built a release based on my PR against 2.3 and it got past the issue.

[dongjoon-hyun]

Got it. Thank you for the detailed explanation, @andygrove . I'll verify on EKS and merge your PR today~

[dongjoon-hyun]

This PR is tested with EKS.

$ kubectl version --short
Client Version: v1.15.3
Server Version: v1.13.10-eks-5ac0f1

$ aws ecr list-images --repository-name spark
{
    "imageIds": [
        {
            "imageDigest": "sha256:c92d634507aa8336c79cb094ba69083d9cb50f4a6f09259e3b0cb4b6bf1c5214",
            "imageTag": "PR-25640"
        },
        {
            "imageDigest": "sha256:a2a48304453c147ec2f049ea0b6c4dbadb625a0c8d76d4c1eb4f7cb3f134890c",
            "imageTag": "latest"
        }
    ]
}

$ echo $K8S_MASTER
https://9310EC45A37C51BCCF6BC12CDBFCBB61.sk1.us-west-2.eks.amazonaws.com

$ echo $IMAGE
095589911305.dkr.ecr.us-west-2.amazonaws.com/spark:PR-25640

bin/spark-submit \
  --master k8s://$K8S_MASTER \
  --deploy-mode cluster \
  --name spark-pi \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.executor.instances=1 \
  --conf spark.kubernetes.container.image=$IMAGE \
  local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0-SNAPSHOT.jar

Thank you, @andygrove , @srowen , @felixcheung .

Merged to master.

[srowen]

Oops, thanks @skonto -- see #25664

We are all OK to back port to 2.4 too?

[dongjoon-hyun]

Thanks, @skonto and @srowen !