SparkKubernetesSensor Cannot Attach Log When There Are Sidecars in the Driver Pod

[joshuayeung]

Apache Airflow Provider(s)

cncf-kubernetes

Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes==3.0.0

Apache Airflow version

2.2.5 (latest released)

Operating System

Debian GNU/Linux 10 (buster)

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

What happened

When using SparkKubernetesSensor with attach_log=True, it cannot get the log correctly with the below error:

 Make sure timeToLiveSeconds is set on your SparkApplication spec.
 underlying exception: (400)
 Reason: Bad Request
 HTTP response headers: HTTPHeaderDict({'Audit-Id': '29ac5abb-452d-4411-a420-8d74155e187d', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 20 Apr 2022 08:42:04 GMT', 'Content-Length': '259'})
 HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"a container name must be specified for pod spark-pi-0.4753748373914717-1-driver, choose one of: [istio-init istio-proxy spark-kubernetes-driver]","reason":"BadRequest","code":400}\n'

It is because no container is specified when calling kubernetes hook.get_pod_logs

airflow/airflow/providers/cncf/kubernetes/sensors/spark_kubernetes.py

Line 85 in 501a3c3

for line in self.hook.get_pod_logs(driver_pod_name, namespace=namespace):

What you think should happen instead

It should get the log of container spark-kubernetes-driver

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[joshuayeung]

@bbenzikry Do you know if the container name is always spark-kubernetes-driver in the driver pod?
If yes, can we just hardcode the container name or we can get the info from k8s hook response dynamically?

[bbenzikry]

@joshuayeung Yeah, it's derived from
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/74ea1c86c9a595e6c1c8a0c2c40a50bf89288ede/pkg/config/constants.go#L306

From what I remember while testing, there was some assurance of container ordering ( that is why the container name was omitted ) - even while using istio sidecars.
Can you tell me a bit about the setup / versions ( k8s, istio, operator, operator CRDs )?

Practically I can't be sure I'll have the time to submit a PR for this in the next few weeks - so feel free to open one to handle this.

Just hope the constant doesn't change in upcoming versions of the operator :)

[hanna-liashchuk]

The PR is open, feel free to review :)