Liveness probe fails, causing scheduler and triggerer restarts every 5 minutes #20644
Description
Official Helm Chart version
1.3.0 (latest released)
Apache Airflow version
2.2.3 (latest released)
Kubernetes Version
version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.4-3+adc4115d990346", GitCommit:"adc4115d990346b87714cc4f033d225711bf744d", GitTreeState:"clean", BuildDate:"2021-11-17T22:03:17Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
Helm Chart configuration
The only values I provided were the image and the tag
helm repo add apache-airflow https://airflow.apache.org
helm repo update
cat <<- 'EOF' | helm install airflow \
--namespace tmp \
apache-airflow/airflow \
-f -
defaultAirflowRepository: apache/airflow
defaultAirflowTag: 2.2.3
EOF
Docker Image customisations
No customizations, just using apache/airflow:2.2.3
What happened
I deploy airflow and just leave it alone, and every 5 minutes, I get events like this one:
LAST SEEN TYPE REASON OBJECT MESSAGE
51s Normal Killing pod/airflow-scheduler-84b9f855b8-2b9cj Container scheduler failed liveness probe, will be restarted
Same story for the triggerer. The weird thing is that when I run the liveness probe code manually, it returns 0 every time (see the gist for an example of this), so I'm not sure why k8s is getting the idea that the probe command fails.
What you expected to happen
Maybe a restart or two as part of the deployment process, but after that I expected there to be no periodic resets.
How to reproduce
I made a gist with a script that replicates this, and also its output: https://gist.github.com/MatrixManAtYrService/97179963d5b543218d704a4594ca2720
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct