Connection Close immediately on nginx reloading, leading to RST

[realzhuangxinyu]

What happened:
My enviroment is: clinet->F5->K8s ingress-nginx->server
Sometimes HTTP 502 occurs.
Since both the F5 and ingress-nginx access logs lack the corresponding records, I believe the 502 error was not caused by a server anomaly.
I noticed that the time difference between the occurrence and the ingress-nginx log entry “NGINX load triggered due to a change in configuration” is always within 3 seconds.
By actively modifying any ingress to trigger an nginx reload, I was indeed able to reproduce the issue with some probability. By capturing packets on the F5 side and periodically collecting Conntrack information on the ingress nodes, I observed the following information:
i) A persistent connection is maintained between F5 and the K8s ingress-nginx, and the connection where the 502 error occurred remains active.
ii) At 16:51:19.220, ingress-nginx logs “Backend successfully reloaded.”
iii) At 16:51:19.221, ingress-nginx logs type: ‘Normal’ reason: ‘RELOAD’ NGINX reload triggered due to a change in configuration.
iv) At 16:51:20.368, ingress-nginx responds to a request from F5 on this connection with a status code 200.
v) At 16:51:20.518, F5 sends a POST request to ingress-nginx using this connection.
vi) At 16:51:20.525, ingress-nginx responds to this request with an RST packet.
vii)Conntrack information shows that at 16:51:19, this connection was in the ESTABLISHED state, and at 16:51:20, this connection was in the CLOSE state.

What you expected to happen:
ingress-nginx responds without RST.

ingress-nginx responded with an RST.

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):

NGINX Ingress controller
Release: v1.1.1
Build: a17181e
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.9

Kubernetes version (use kubectl version):

(Openshift Container Platform)Server Version: 4.12.55
Kubernetes Version: v1.25.16+bd92d70

Environment:

• Cloud provider or hardware configuration: RedHat

• OS (e.g. from /etc/os-release):Red Hat Enterprise Linux CoreOS 412.86.202403280709-0 (Ootpa)

• Kernel (e.g. uname -a):Linux 4.18.0-372.98.1.el8_6.x86_64 Basic structure  #1 SMP Tue Mar 26 00:49:18 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

• How was the ingress-nginx-controller installed: deployment

• Current State of the controller: seems healthy

• Current state of ingress object, if applicable: seems healthy

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[longwuyuan]

Version 1.1.1 of this controller is not supported anymore