Helm chart sets the incorrect execution_api_server_url when the base_url contains a subpath

[LucasRoesler]

Official Helm Chart version

1.16.0 (latest released)

Apache Airflow version

3.0.1

Kubernetes Version

1.32.3

Helm Chart configuration

ingress:
  apiServer:
    enabled: true
    hosts:
      - name: airflow.example.com
    path: "/dataplatform"

extraEnv: |
  - name: AIRFLOW__CORE__LOAD_EXAMPLES
    value: True

config:
  api:
    base_url: http://airflow.example.com/dataplatform/

Docker Image customizations

None

What happened

If you set the base_url to a value includes a path prefix, tasks will fail with very hard to debug messages.

For example, the Web UI will show this error message

When I first looked into the worker logs, I would see a 404 error but it was missing a lot of information, it didn't contain the host information so I wasn't really sure which component was failing. After enabling the debug logs, it shows a bit more information, like this and see that it was the api server returning a 404 error

That is easily confirmed in the api-server logs, which showed

"PATCH /execution/task-instances/01973b8f-b80d-7818-bb88-4f127d7384ce/run HTTP/1.1" 404 Not Found

It took me a long time to figure out a workaround. I never found the right search terms for the Github issues, until after I found the solution: the execution_api_server_url value was incorrect. It was set to

http://airflow-api-server:8080/execution/

but it needed to be

http://airflow-api-server:8080/dataplatform/execution/

What you think should happen instead

The Helm chart should support configuring the execution_api_server_url when a base url is set. There are other components that already did this for Airflow 2, e.g. in the webserver component, it adapated the health check to include the base url path

path: {{ if .Values.config.webserver.base_url }}{{- with urlParse (tpl .Values.config.webserver.base_url .) }}{{ .path }}{{ end }}{{ end }}/health

A similar approach can be used when building the configmap.yaml, for example

  {{- if semverCompare ">=3.0.0" .Values.airflowVersion -}}
    {{- $basePath := "" -}}
    {{- $config := merge .Values.config ( dict  "core" dict )}}
    {{- if not (hasKey $config.core "execution_api_server_url") -}}
      {{-  if $config.api.base_url -}}
        {{- with urlParse $config.api.base_url }}{{ $basePath = (trimSuffix "/" .path) }}{{ end }}
      {{- end -}}
      {{- $_ := set $config.core "execution_api_server_url" (printf "http://%s-api-server:%d%s/execution/" (include "airflow.fullname" .) (int .Values.ports.apiServer) $basePath)  -}}
    {{- end -}}
  {{- end -}}

How to reproduce

I have reproduced this in 2 different clusters by deploying helm with the provided yaml values from above. Then attempt to execute one of the example DAGs.

Anything else

I think this is partially related to the following issues:

• [3.0.1] Distributed DAG Processing - Worker PATCH to /api/v2/task-instances/{id}/run fails with 405 Method Not Allowed #51235
• Task fails and cannot read logs. Invalid URL 'http://:8793/log/...': No host supplied #42136

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Towerhint]

I have the same issue hopefully airflow can fix this issue or add some additional comments on the config file to avoid this (because it is hard to debug)

Version: Airflow 3.1.5