spike: choose transport timeouts and pool sizes for the backend proxies

[santosral]

User story

As a developer working on teacher-workspace, I want measured values for the backend proxies' transport settings, so the implement issue can be built against real numbers instead of guesses.

Background

Both proxies set only Rewrite and ErrorHandler, so ReverseProxy.Transport is nil and every proxied request runs on http.DefaultTransport. That default is built for a general-purpose client, not for a reverse proxy that sends all its traffic to a small fixed set of hosts. Two problems follow from it, and neither can be fixed without picking numbers we do not have yet.

The deployment this is being sized for

Every number below is chosen against this, so it is stated first rather than left to be discovered:

• The call is internal, on the same subnet. The server opens a TCP connection straight to the backend process.
• Nothing sits in between. No load balancer.
• No TLS on the backend hop. Plain HTTP.

What DefaultTransport sets today

This is the whole of DefaultTransport, from transport.go in Go 1.26.5:

var DefaultTransport RoundTripper = &Transport{
	Proxy: ProxyFromEnvironment,
	DialContext: defaultTransportDialContext(&net.Dialer{
		Timeout:   30 * time.Second,
		KeepAlive: 30 * time.Second,
	}),
	ForceAttemptHTTP2:     true,
	MaxIdleConns:          100,
	IdleConnTimeout:       90 * time.Second,
	TLSHandshakeTimeout:   10 * time.Second,
	ExpectContinueTimeout: 1 * time.Second,
}

What matters here is as much what is missing as what is there. The fields it leaves at their zero value:

Field	Zero value means	Effect on us
ResponseHeaderTimeout	no limit	the wait for the reply to start never ends
MaxIdleConnsPerHost	falls back to 2	past 2 in flight, a new connection per request
MaxConnsPerHost	no limit	nothing ever waits for a free connection
ReadBufferSize, WriteBufferSize	4KB each	not yet known whether 4KB suits our bodies

Two of the values it does set are irrelevant to us. TLSHandshakeTimeout never runs, because the backend hop is plain HTTP. MaxIdleConns of 100 is a global cap we will never approach with two backends.

Where each setting applies

The call path through ReverseProxy.ServeHTTP. Steps 4 and 6 have no limit today. Steps 1 and 7 are the pool.

flowchart TB
    A["ReverseProxy.ServeHTTP<br/>p.Transport is nil, so http.DefaultTransport is used"]
    B["outreq := req.Clone(ctx)<br/>ctx is req.Context.<br/>A deadline set here is the only thing<br/>that can reach step 6."]
    C["p.Rewrite<br/>our proxyRewriter"]
    D["transport.RoundTrip(outreq)"]
    S1["1. get a connection<br/>reuse a kept one, or dial<br/>MaxIdleConnsPerHost 2, MaxConnsPerHost 0"]
    S2["2. dial, only when none is free<br/>Dialer.Timeout 30s"]
    S3["3. write the request<br/>ExpectContinueTimeout 1s,<br/>only with an Expect header"]
    S4["4. wait for the reply to start<br/>ResponseHeaderTimeout NOT SET"]
    ERR["p.ErrorHandler<br/>our proxyErrorHandler<br/>502 Bad Gateway, logged"]
    S5["5. rw.WriteHeader<br/>status and headers sent to the browser<br/>no 502 is possible after this"]
    S6["6. p.copyResponse<br/>read the reply body, stream it out<br/>NO TRANSPORT SETTING COVERS THIS"]
    PANIC["panic http.ErrAbortHandler<br/>http.Server recovers it and logs nothing"]
    S7["7. res.Body.Close<br/>connection kept for the next request,<br/>IdleConnTimeout 90s, or closed"]

    A --> B --> C --> D --> S1
    S1 -->|"none free"| S2
    S1 -->|"one free"| S3
    S2 --> S3
    S2 -->|"dial fails or times out"| ERR
    S3 --> S4
    S4 -->|"times out or errors"| ERR
    S4 -->|"headers arrive"| S5
    S5 --> S6
    S6 -->|"read fails or stalls"| PANIC
    S6 -->|"body copied"| S7

Loading

#	Stage	Setting	Today	What it does
1	Get a connection	Transport.MaxIdleConnsPerHost	2, the default	past 2 in flight, no kept connection is free
2	Dial the backend	Dialer.Timeout	30s	502 via ErrorHandler when it expires
3	Send the request	Transport.ExpectContinueTimeout	1s, only with an Expect header	the body is sent anyway when it expires
4	Wait for the reply to start	Transport.ResponseHeaderTimeout	not set	nothing. Waits until the browser gives up
5	Send status and headers	none	n/a	after this a 502 is impossible
6	Read the reply body	none	no limit	nothing. Waits until the browser gives up
7	Keep or close the connection	Transport.IdleConnTimeout	90s	how long an unused connection is kept

Everything from step 1 to step 4 fails into p.ErrorHandler, which is our proxyErrorHandler, so the caller gets a 502 and we log it. Step 6 has no such path: it panics with http.ErrAbortHandler, which http.Server recovers while suppressing the log (reverseproxy.go, server.go). So the two unbounded steps need different fixes and give different results.

The http.Server timeouts are not in this table because they bound the browser side, not the backend side. WriteTimeout (30s) is the one people expect to cover this. It does not: it only limits writes to the browser, and the handler is stuck reading from the backend.

Two facts that narrow the options

• Transport.ResponseHeaderTimeout bounds the wait for reply headers only. Its doc says that time "does not include the time to read the response body". So it fixes step 4 and cannot fix step 6.
• ReverseProxy.ServeHTTP builds the outbound request as outreq := req.Clone(ctx) from req.Context(). A deadline on the inbound context reaches the backend call and the body read. That is the only lever for step 6.

Prior claims

AC3 of #40 covers a backend that cannot be reached. A closed port fails at once and returns a 502 either way. It does not cover a backend that is up but not answering.

#58 reported this as done (add-1 in the 2026-08-03 comment on #40). It is not. Transport, ResponseHeaderTimeout, TimeoutHandler and context.WithTimeout appear nowhere under server/.

Acceptance criteria

A recommended value for every setting is delivered

• Given the settings listed under Spike question
• When the spike concludes
• Then a written recommendation names a value for each one, or records a decision to leave it at the default
• And each value cites the measurement or constraint it came from, not a rule of thumb
• And it states which settings become TW_* environment variables and which are fixed in code
• And it accounts for all 25 exported fields on http.Transport, each with a chosen value or a one-line reason it stays at its default, so nothing is left decided by accident

Out of scope

• Retries, circuit breaking and rate limiting.

Spike question

Which of http.Transport's settings do we need to set on the backend proxies, and what value should each one take?

#	Field	Type	Default
1	Proxy	func(*Request) (*url.URL, error)	ProxyFromEnvironment
2	DialContext	func(...) (net.Conn, error)	30s dial, 30s TCP keepalive
3	Dial	func(...) (net.Conn, error)	nil
4	DisableKeepAlives	bool	false
5	DisableCompression	bool	false
6	MaxIdleConns	int	100
7	MaxIdleConnsPerHost	int	2, from the zero value
8	MaxConnsPerHost	int	0, no limit
9	IdleConnTimeout	time.Duration	90s
10	ResponseHeaderTimeout	time.Duration	unset, no limit
11	ExpectContinueTimeout	time.Duration	1s
12	MaxResponseHeaderBytes	int64	10MB
13	WriteBufferSize	int	4KB, from the zero value
14	ReadBufferSize	int	4KB, from the zero value
15	ForceAttemptHTTP2	bool	true
16	HTTP2	*HTTP2Config	nil
17	Protocols	*Protocols	nil, so HTTP/1.1

---

🤖 Generated with create-issue

[tfx-currents]

---

🌊 Managed by Currents · Task 20927bb9-0da2-413b-9441-e591c62a4419

{"metadata":{"sourceType":"github"},"rank":"a0","taskId":"20927bb9-0da2-413b-9441-e591c62a4419","version":1}