Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve getMasterURL() to add [] to IPv6 if needed #1825

Conversation

LittleWat
Copy link
Contributor

@LittleWat LittleWat commented Aug 30, 2023

Resolves #1344

Spark 3.4 supports IPv6:

So I want to make the operator also support IPv6.

I can confirm that this can submit the following spark job that uses spark 3.4 both in IPv6-only and IPv4 environments.
(Although it is necessary to add the following environment variables to the operator in the IPv6-only environment... for the workaround of the okhttp3 bug

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-on-k8s-spark-operator
spec:
  template:
    spec:
      containers:
      - name: spark-operator
        env:
        - name: _JAVA_OPTIONS
          value: "-Djava.net.preferIPv6Addresses=true"
        - name: KUBERNETES_DISABLE_HOSTNAME_VERIFICATION
          value: "true"

The sample I tested:

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"sparkoperator.k8s.io/v1beta2","kind":"SparkApplication","metadata":{"annotations":{},"name":"spark-pi","namespace":"data-platform-demo"},"spec":{"driver":{"coreLimit":"1200m","cores":1,"labels":{"version":"3.4.1"},"memory":"512m","serviceAccount":"spark"},"executor":{"cores":1,"instances":1,"labels":{"version":"3.4.1"},"memory":"512m"},"image":"apache/spark:3.4.1","imagePullPolicy":"Always","mainApplicationFile":"local:///opt/spark/examples/jars/spark-examples_2.12-3.4.1.jar","mainClass":"org.apache.spark.examples.SparkPi","mode":"cluster","restartPolicy":{"type":"Never"},"sparkVersion":"3.4.1","type":"Scala"}}
  creationTimestamp: "2023-08-31T05:19:57Z"
  generation: 5
  name: spark-pi
  namespace: data-platform-demo
  resourceVersion: "219867269"
  uid: fcfc6341-e0df-4679-a1cb-e0983f3a8423
spec:
  driver:
    coreLimit: 1200m
    cores: 1
    labels:
      version: 3.4.1
    memory: 512m
    serviceAccount: spark
  executor:
    cores: 1
    instances: 1
    labels:
      version: 3.4.1
    memory: 512m
  image: apache/spark:3.4.1
  imagePullPolicy: Always
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.4.1.jar
  mainClass: org.apache.spark.examples.SparkPi
  mode: cluster
  restartPolicy:
    type: Never
  sparkVersion: 3.4.1
  type: Scala
status:
  applicationState:
    state: COMPLETED
  driverInfo:
    podName: spark-pi-driver
    webUIAddress: fd85:464b:2e72::f610:0
    webUIPort: 4040
    webUIServiceName: spark-pi-ui-svc
  executionAttempts: 1
  executorState:
    spark-pi-88a60e8a4a219c23-exec-1: COMPLETED
  lastSubmissionAttemptTime: "2023-08-31T05:47:00Z"
  sparkApplicationId: spark-c0bd0af55802492494682041bbdd22d5
  submissionAttempts: 1
  submissionID: 329a09b9-139a-4e55-b034-c7ebeda9e82e
  terminationTime: "2023-08-31T05:47:23Z"

Resolves kubeflow#1344

Spark 3.4 supports IPv6:
- apache/spark#36868

So I want to make the operator support IPv6.

I can confirm that this can submit the spark-job in IPv6-only environment.

Although it is necessary to add the following environment variables to the operator

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-on-k8s-spark-operator
spec:
  template:
    spec:
      containers:
      - name: spark-operator
        env:
        - name: _JAVA_OPTIONS
          value: "-Djava.net.preferIPv6Addresses=true"
        - name: KUBERNETES_DISABLE_HOSTNAME_VERIFICATION
          value: "true"

```
@LittleWat LittleWat marked this pull request as ready for review September 6, 2023 07:23
@liyinan926 liyinan926 merged commit 463ed1e into kubeflow:master Oct 26, 2023
peter-mcclonski pushed a commit to TechnologyBrewery/spark-on-k8s-operator that referenced this pull request Apr 16, 2024
Resolves kubeflow#1344

Spark 3.4 supports IPv6:
- apache/spark#36868

So I want to make the operator support IPv6.

I can confirm that this can submit the spark-job in IPv6-only environment.

Although it is necessary to add the following environment variables to the operator

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-on-k8s-spark-operator
spec:
  template:
    spec:
      containers:
      - name: spark-operator
        env:
        - name: _JAVA_OPTIONS
          value: "-Djava.net.preferIPv6Addresses=true"
        - name: KUBERNETES_DISABLE_HOSTNAME_VERIFICATION
          value: "true"

```

Signed-off-by: Peter McClonski <mcclonski_peter@bah.com>
sigmarkarl pushed a commit to spotinst/spark-on-k8s-operator that referenced this pull request Aug 7, 2024
Resolves kubeflow#1344

Spark 3.4 supports IPv6:
- apache/spark#36868

So I want to make the operator support IPv6.

I can confirm that this can submit the spark-job in IPv6-only environment.

Although it is necessary to add the following environment variables to the operator

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-on-k8s-spark-operator
spec:
  template:
    spec:
      containers:
      - name: spark-operator
        env:
        - name: _JAVA_OPTIONS
          value: "-Djava.net.preferIPv6Addresses=true"
        - name: KUBERNETES_DISABLE_HOSTNAME_VERIFICATION
          value: "true"

```
jbhalodia-slack pushed a commit to jbhalodia-slack/spark-operator that referenced this pull request Oct 4, 2024
Resolves kubeflow#1344

Spark 3.4 supports IPv6:
- apache/spark#36868

So I want to make the operator support IPv6.

I can confirm that this can submit the spark-job in IPv6-only environment.

Although it is necessary to add the following environment variables to the operator

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-on-k8s-spark-operator
spec:
  template:
    spec:
      containers:
      - name: spark-operator
        env:
        - name: _JAVA_OPTIONS
          value: "-Djava.net.preferIPv6Addresses=true"
        - name: KUBERNETES_DISABLE_HOSTNAME_VERIFICATION
          value: "true"

```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

sparksubmit operator fails in ipv6+istio environment
2 participants