Repository navigation
caddyhttp: don't flush a hijacked response - #8164
Open
jotikrishna wants to merge 2 commits into
Open
jotikrishna wants to merge 2 commits into
jotikrishna wants to merge 2 commits into
Conversation
jotikrishna
force-pushed
the
fix/hijacked-response-flush-panic
branch
from
October 7, 2026 04:27
3b769ef to
dc70754
Compare
Since caddyserver#7945, Server.ServeHTTP flushes the response in a deferred func when an HTTP/1.x request that is being access-logged took at least write_timeout. The deferred func already skips http.ErrHijacked, on the assumption that Flush reports a hijacked response that way. It does not. (*http.response).Hijack releases the bufio.Writer, and (*http.response).FlushError calls w.w.Flush() without checking for the hijack. Only Write and WriteHeader consult hijacked() and return ErrHijacked. So when the handler hijacked the connection, as reverse_proxy does when it passes through a 101 Switching Protocols upgrade, and the upgraded stream outlives write_timeout, the deferred flush reaches a nil bufio.Writer and panics with an invalid memory address. net/http recovers it and it fires only after the upgraded stream ended, so clients never notice, but every such connection ends with an error-level stack trace in the logs. responseRecorder already tracks the hijack for DetachAfterHijack and WriteResponse, so report it from FlushError too, which is what the deferred func expects. Fixes caddyserver#8151
jotikrishna
force-pushed
the
fix/hijacked-response-flush-panic
branch
from
October 7, 2026 04:28
dc70754 to
e0c6801
Compare
WeidiDeng
requested changes
Oct 7, 2026
| func (rr *responseRecorder) FlushError() error { | ||
| // Hijack releases the underlying bufio.Writer, and net/http's FlushError | ||
| // dereferences it without checking for the hijack, so flushing after an | ||
| // upgrade panics with a nil pointer dereference. Write and WriteHeader |
Member
There was a problem hiding this comment.
WriteHeader doesn't return anything.
| // hijackableRecorder is an http.ResponseWriter that implements http.Hijacker the | ||
| // way net/http does. This matters: after a successful Hijack the real | ||
| // (*http.response) has released its bufio.Writer (w.w = nil) and requires | ||
| // Write/WriteHeader to return ErrHijacked. FlushError, however, dereferences |
|
|
||
| // Flush mirrors stdlib (*http.response).FlushError: it does not consult the | ||
| // hijack state, which is exactly how the regression in #8151 becomes a panic. | ||
| func (h *hijackableRecorder) Flush() { |
Member
There was a problem hiding this comment.
Implement FlushError instead of Flush.
Address review feedback from WeidiDeng on caddyserver#8164: - The comment claimed Write and WriteHeader already report the hijack, but WriteHeader does not return anything: after a hijack it is a silent no-op. Only Write returns ErrHijacked. Reword to say what each actually does. - The test fake implemented Flush; net/http responses implement FlushError, which is what http.NewResponseController prefers, so implement FlushError instead to mirror the real interface shape.
WeidiDeng
requested changes
Oct 7, 2026
| // upgrade panics with a nil pointer dereference. Write and WriteHeader | ||
| // already report the hijack; FlushError has to do the same. See #8151. | ||
| // Hijack releases the underlying bufio.Writer. net/http's Write returns | ||
| // ErrHijacked from then on and WriteHeader silently does nothing, but |
Member
There was a problem hiding this comment.
WriteHeader has an error log for it.
| // that writer without checking for the hijack, so flushing afterwards is not | ||
| // merely wrong, it is a nil pointer dereference. | ||
| // (*http.response) has released its bufio.Writer (w.w = nil). Write returns | ||
| // ErrHijacked from then on and WriteHeader silently does nothing, but |
steadytao
requested changes
Oct 8, 2026
steadytao
left a comment
Member
There was a problem hiding this comment.
Standard Caddy header to the test file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Since #7945,
Server.ServeHTTPflushes the response from a deferred func when an HTTP/1.x request that is being access-logged took at leastwrite_timeout. The deferred func already skipshttp.ErrHijacked, on the assumption thatFlushreports a hijacked response that way.It does not. When the handler hijacked the connection - as
reverse_proxydoes when it passes through a101 Switching Protocolsupgrade - and the upgraded stream outliveswrite_timeout, that deferred flush panics:Root Cause
(*http.response).Hijackreleases thebufio.Writerand setsw.w = nil.(*http.response).FlushErrorcallsw.w.Flush()without checkinghijacked(); onlyWriteandWriteHeaderconsult it and returnhttp.ErrHijacked.So
Flushcannot report a hijack the way the deferred func expects, and the flush dereferences a nil writer.net/httprecovers the panic, and because it fires only after the upgraded stream has ended, clients never notice - but every upgraded connection that outliveswrite_timeoutends with an error-level stack trace in the logs. Reported in production against headscale behindreverse_proxy(/ts2021and/derp, both HTTP/1.1 upgrades) withwrite 30s, so practically every client session produced one.Solution
responseRecorderalready tracks the hijack: #7649 added thehijackedfield and uses it inWriteResponseandDetachAfterHijack.FlushErrorsimply did not consult it.Return
http.ErrHijackedfromFlushErrorwhen the response was hijacked. The deferred flush inServeHTTPalready handles that error, so the loop closes with no change to callers:Seven insertions in one file. The guard belongs on the recorder because the recorder is the layer that performed the hijack, so it is the layer that can answer whether the response was hijacked.
Worth noting for reviewers: #7649 also sets
rr.stream = trueinsideHijack, so theif rr.streambranch is now always taken for a hijacked streaming response. That commit made this path deterministic rather than incidental.Testing
Environment: Go 1.27.0 windows/amd64, base
c1acb2f(currentmaster).Baseline before any edit:
go test ./modules/caddyhttp/->ok 7.939s, with no fix and no test file present. Clean, so the failures below are attributable to the new tests rather than pre-existing breakage.Regression test fails before the fix (guard removed, everything else identical):
The panic string matches the one in the issue exactly.
All four tests pass after the fix:
TestServerServeHTTPDeferredFlushAfterHijackdrives the realServer.ServeHTTPpath with a handler that writes101, hijacks, and then outliveswrite_timeout, which is the scenario requested below. The two other tests are controls: a non-hijacked streamed response must still flush exactly once, and a buffered response must still suppress the flush (#6144).go test ./modules/caddyhttp/go test ./modules/caddyhttp/...go build ./...go vet ./modules/caddyhttp/...gofmt -lon both filesThe test uses an
httptest-style fake that replicates net/http's hijack semantics (aFlushafterHijackfaults, exactly as stdlib does) rather than a real socket, so it is deterministic and needs no network. Noncor external process is involved.Not run, for transparency:
golangci-lintis not installed in my environment, so the repository's own lint config has not been executed against this change.Impact
Upgraded connections that outlive
write_timeoutstop logging an error-level panic trace. The access-log entry itself is unchanged (status: 101, same duration) - only the spurious stack trace disappears.Compatibility
No API change. No behaviour change for non-hijacked responses. No dependency change.
References
Fixes #8151
Maintainer direction this responds to
Assistance disclosure
AI was used to write this change and its tests; I reviewed the diff and ran every check listed above myself. The issue's own Assistance Disclosure notes the reporter reached the same root cause with AI assistance - this is an independent implementation from a different contributor, disclosed here so the overlap is visible to reviewers rather than something they have to discover.