Skip to content

fix(chart): close client connections per response to stop 400 on aborted uploads#106

Merged
ServerSideHannes merged 1 commit into
mainfrom
fix/frontproxy-httpclose-upload-desync-400
Jul 2, 2026
Merged

fix(chart): close client connections per response to stop 400 on aborted uploads#106
ServerSideHannes merged 1 commit into
mainfrom
fix/frontproxy-httpclose-upload-desync-400

Conversation

@ServerSideHannes

Copy link
Copy Markdown
Owner

Problem

barman multipart PUT uploads (CloudNativePG base backups) intermittently fail against s3proxy-python-frontproxy with HAProxy's built-in error:

urllib3.exceptions.HeaderParsingError: [MissingHeaderBodySeparatorDefect()],
unparsed data: 'HTTP/1.1 400 Bad request\r\nContent-length: 90\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n'

The Content-length: 90 + Connection: close + text/html page is HAProxy 3.4's own built-in 400 Bad request — not from the Python backend or real S3.

Root cause

The frontproxy runs mode http with default client keep-alive. During a backup, many concurrent multipart PUTs are in flight. When a backend pod is killed mid-body (KEDA scale-down churn + option redispatch, maxconn 20/pod), HAProxy sends the response before the request body is fully read. On the kept-alive client connection, the leftover upload-body bytes are then parsed as the next request line — not valid HTTP — so HAProxy emits its built-in 400 and closes. barman retries (backups limp through, noisily).

Fix

Add option httpclose so the client connection is closed after each response and a poisoned connection is never reused.

  • option http-server-close would not help — the desync is on the client-facing connection, which http-server-close keeps alive.
  • We use plain HTTP/1.1 (no alpn h2), so the known httpclose+h2 400 caveat does not apply.
  • Cost is negligible: plaintext internal HTTP, large sequential parts; per-part TCP setup is cheap and bounded by maxconn 4096.

Note: the underlying mid-upload aborts (backend pod churn under load) are a separate track (KEDA/memory tuning already in flight). This PR stops the client-visible 400 symptom.

…ted uploads

barman multipart PUT uploads intermittently fail with HAProxy's built-in
"400 Bad request". When a backend pod is killed mid-body (KEDA churn +
option redispatch), HAProxy responds before the request body is fully read;
on a kept-alive client connection the leftover body bytes are then parsed as
the next request line, which is not valid HTTP, so HAProxy emits a 400 and
the client fails the part.

Add option httpclose so a poisoned client connection is never reused.
http-server-close would not help: the desync is on the client-facing side.
@ServerSideHannes ServerSideHannes merged commit 7010a1f into main Jul 2, 2026
4 checks passed
@ServerSideHannes ServerSideHannes deleted the fix/frontproxy-httpclose-upload-desync-400 branch July 2, 2026 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant