Skip to content

caddyhttp: don't flush a hijacked response - #8164

Open
jotikrishna wants to merge 2 commits into
caddyserver:masterfrom
jotikrishna:fix/hijacked-response-flush-panic
Open

jotikrishna wants to merge 2 commits into
caddyserver:masterfrom
jotikrishna:fix/hijacked-response-flush-panic

Conversation

@jotikrishna

Copy link
Copy Markdown

Problem

Since #7945, Server.ServeHTTP flushes the response from a deferred func when an HTTP/1.x request that is being access-logged took at least write_timeout. The deferred func already skips http.ErrHijacked, on the assumption that Flush reports a hijacked response that way.

It does not. When the handler hijacked the connection - as reverse_proxy does when it passes through a 101 Switching Protocols upgrade - and the upgraded stream outlives write_timeout, that deferred flush panics:

{"level":"error","logger":"http.stdlib","msg":"http: panic serving 127.0.0.1:56866: runtime error: invalid memory address or nil pointer dereference
...
bufio.(*Writer).Flush(0x1078443d8?)
	net/http/server.go:1728
net/http.(*response).FlushError(0x637fdd514000)
	net/http/responsecontroller.go:52
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*responseRecorder).FlushError(...)
	modules/caddyhttp/responsewriter.go:244
net/http.(*ResponseController).Flush(0x1421bcc2e?)
	net/http/responsecontroller.go:52
github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP.func1()
	modules/caddyhttp/server.go:692

Root Cause

(*http.response).Hijack releases the bufio.Writer and sets w.w = nil. (*http.response).FlushError calls w.w.Flush() without checking hijacked(); only Write and WriteHeader consult it and return http.ErrHijacked.

So Flush cannot report a hijack the way the deferred func expects, and the flush dereferences a nil writer. net/http recovers the panic, and because it fires only after the upgraded stream has ended, clients never notice - but every upgraded connection that outlives write_timeout ends with an error-level stack trace in the logs. Reported in production against headscale behind reverse_proxy (/ts2021 and /derp, both HTTP/1.1 upgrades) with write 30s, so practically every client session produced one.

Solution

responseRecorder already tracks the hijack: #7649 added the hijacked field and uses it in WriteResponse and DetachAfterHijack. FlushError simply did not consult it.

Return http.ErrHijacked from FlushError when the response was hijacked. The deferred flush in ServeHTTP already handles that error, so the loop closes with no change to callers:

func (rr *responseRecorder) FlushError() error {
	// Hijack releases the underlying bufio.Writer, and net/http's FlushError
	// dereferences it without checking for the hijack, so flushing after an
	// upgrade panics with a nil pointer dereference. Write and WriteHeader
	// already report the hijack; FlushError has to do the same. See #8151.
	if rr.hijacked {
		return http.ErrHijacked
	}
	if rr.stream {
		//nolint:bodyclose
		return http.NewResponseController(rr.ResponseWriterWrapper).Flush()
	}
	return nil
}

Seven insertions in one file. The guard belongs on the recorder because the recorder is the layer that performed the hijack, so it is the layer that can answer whether the response was hijacked.

Worth noting for reviewers: #7649 also sets rr.stream = true inside Hijack, so the if rr.stream branch is now always taken for a hijacked streaming response. That commit made this path deterministic rather than incidental.

Testing

Environment: Go 1.27.0 windows/amd64, base c1acb2f (current master).

Baseline before any edit: go test ./modules/caddyhttp/ -> ok 7.939s, with no fix and no test file present. Clean, so the failures below are attributable to the new tests rather than pre-existing breakage.

Regression test fails before the fix (guard removed, everything else identical):

--- FAIL: TestResponseRecorderFlushAfterHijack (0.00s)
    flushing a hijacked response panicked: runtime error: invalid memory address or nil pointer dereference
--- FAIL: TestServerServeHTTPDeferredFlushAfterHijack (0.01s)
    ServeHTTP panicked after an upgraded connection outlived write_timeout: ...

The panic string matches the one in the issue exactly.

All four tests pass after the fix:

--- PASS: TestResponseRecorderFlushAfterHijack (0.00s)
--- PASS: TestResponseRecorderFlushWithoutHijack (0.00s)
--- PASS: TestResponseRecorderBufferedFlushUnaffected (0.00s)
--- PASS: TestServerServeHTTPDeferredFlushAfterHijack (0.01s)
ok  	github.com/caddyserver/caddy/v2/modules/caddyhttp	0.391s

TestServerServeHTTPDeferredFlushAfterHijack drives the real Server.ServeHTTP path with a handler that writes 101, hijacks, and then outlives write_timeout, which is the scenario requested below. The two other tests are controls: a non-hijacked streamed response must still flush exactly once, and a buffered response must still suppress the flush (#6144).

Check Command Result
Package suite go test ./modules/caddyhttp/ ok 7.887s
Subsystem suite go test ./modules/caddyhttp/... ok, 14/14 packages
Build go build ./... exit 0
Vet go vet ./modules/caddyhttp/... exit 0
Formatting gofmt -l on both files clean

The test uses an httptest-style fake that replicates net/http's hijack semantics (a Flush after Hijack faults, exactly as stdlib does) rather than a real socket, so it is deterministic and needs no network. No nc or external process is involved.

Not run, for transparency: golangci-lint is not installed in my environment, so the repository's own lint config has not been executed against this change.

Impact

Upgraded connections that outlive write_timeout stop logging an error-level panic trace. The access-log entry itself is unchanged (status: 101, same duration) - only the spurious stack trace disappears.

Compatibility

No API change. No behaviour change for non-hijacked responses. No dependency change.

References

Fixes #8151


Maintainer direction this responds to

steadytao (MEMBER, 2026-10-05): "The recorder-level hijack state looks like the right fix. Please open a PR with the patch and a regression covering an upgraded connection which outlives write_timeout."

Assistance disclosure

AI was used to write this change and its tests; I reviewed the diff and ran every check listed above myself. The issue's own Assistance Disclosure notes the reporter reached the same root cause with AI assistance - this is an independent implementation from a different contributor, disclosed here so the overlap is visible to reviewers rather than something they have to discover.

@CLAassistant

CLAassistant commented Oct 7, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@jotikrishna
jotikrishna force-pushed the fix/hijacked-response-flush-panic branch from 3b769ef to dc70754 Compare October 7, 2026 04:27
Since caddyserver#7945, Server.ServeHTTP flushes the response in a deferred func when an
HTTP/1.x request that is being access-logged took at least write_timeout. The
deferred func already skips http.ErrHijacked, on the assumption that Flush
reports a hijacked response that way.

It does not. (*http.response).Hijack releases the bufio.Writer, and
(*http.response).FlushError calls w.w.Flush() without checking for the hijack.
Only Write and WriteHeader consult hijacked() and return ErrHijacked.

So when the handler hijacked the connection, as reverse_proxy does when it
passes through a 101 Switching Protocols upgrade, and the upgraded stream
outlives write_timeout, the deferred flush reaches a nil bufio.Writer and
panics with an invalid memory address. net/http recovers it and it fires only
after the upgraded stream ended, so clients never notice, but every such
connection ends with an error-level stack trace in the logs.

responseRecorder already tracks the hijack for DetachAfterHijack and
WriteResponse, so report it from FlushError too, which is what the deferred
func expects.

Fixes caddyserver#8151
@jotikrishna
jotikrishna force-pushed the fix/hijacked-response-flush-panic branch from dc70754 to e0c6801 Compare October 7, 2026 04:28
@francislavoie francislavoie added the bug 🐞 Something isn't working label Oct 7, 2026
@francislavoie francislavoie added this to the v2.12.0 milestone Oct 7, 2026
Comment thread modules/caddyhttp/responsewriter.go Outdated
func (rr *responseRecorder) FlushError() error {
// Hijack releases the underlying bufio.Writer, and net/http's FlushError
// dereferences it without checking for the hijack, so flushing after an
// upgrade panics with a nil pointer dereference. Write and WriteHeader

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WriteHeader doesn't return anything.

// hijackableRecorder is an http.ResponseWriter that implements http.Hijacker the
// way net/http does. This matters: after a successful Hijack the real
// (*http.response) has released its bufio.Writer (w.w = nil) and requires
// Write/WriteHeader to return ErrHijacked. FlushError, however, dereferences

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same as above.


// Flush mirrors stdlib (*http.response).FlushError: it does not consult the
// hijack state, which is exactly how the regression in #8151 becomes a panic.
func (h *hijackableRecorder) Flush() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implement FlushError instead of Flush.

Address review feedback from WeidiDeng on caddyserver#8164:

- The comment claimed Write and WriteHeader already report the hijack, but
  WriteHeader does not return anything: after a hijack it is a silent no-op.
  Only Write returns ErrHijacked. Reword to say what each actually does.
- The test fake implemented Flush; net/http responses implement FlushError,
  which is what http.NewResponseController prefers, so implement FlushError
  instead to mirror the real interface shape.
// upgrade panics with a nil pointer dereference. Write and WriteHeader
// already report the hijack; FlushError has to do the same. See #8151.
// Hijack releases the underlying bufio.Writer. net/http's Write returns
// ErrHijacked from then on and WriteHeader silently does nothing, but

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WriteHeader has an error log for it.

// that writer without checking for the hijack, so flushing afterwards is not
// merely wrong, it is a nil pointer dereference.
// (*http.response) has released its bufio.Writer (w.w = nil). Write returns
// ErrHijacked from then on and WriteHeader silently does nothing, but

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same as above.

@steadytao steadytao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Standard Caddy header to the test file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug 🐞 Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Panic in deferred write-timeout flush when an upgraded (hijacked) connection outlives write_timeout

5 participants