Stop a blocked event loop from failing completed connections

Provider SDK defaults assume a caller whose event loop is responsive. An
inspect runner's is not: long CPU-bound sections hold it for seconds at a time,
and a 5s connect deadline that expires during one of those blocks fails a
connection the kernel had already completed, because anyio's cancellation lands
on the first checkpoint after the loop resumes. The result is a run peppered
with `APIConnectionError`, which inspect classifies as transient and retries,
so the symptom is a slow eval rather than an obvious failure.

Provider HTTP clients now allow 60s for connection setup rather than 5s, keep a
keepalive pool as large as the connection limit, retry connection establishment
once, and carry the TCP keepalive socket options the Anthropic SDK sets on its
own. A partial raise of the deadline buys nothing: a 26s block failed 75% of
connects at 5s and 76.7% at 15s, and 0% at 30s. `_util/http_defaults` holds the
rationale and the measurements behind each value.

This reaches the OpenAI, OpenAI-compatible, Anthropic and Groq providers,
Google under trio, and the image fetches that run on the eval loop beside a
generate. It deliberately does not reach the rest, which are not exposed to the
problem rather than being overlooked: Bedrock, SageMaker and Azure AI have
their own connection deadlines, already at or above 60s; Google on asyncio
passes `aiohttp.ClientTimeout(total=...)` per request, which carries no connect
deadline at all; and Grok speaks gRPC, establishing connections off the event
loop where the block cannot reach them.

Three things had to be handled to get the deadline as far as the socket. SDKs
stamp their own scalar timeout into `request.extensions` after the client is
built, and httpx expands a bare float to all four deadlines, so a budget such
as `-M client_timeout=30` or the 900s `service_tier="flex"` budget would
silently become the connect deadline too; a request event hook now floors it,
raising a short deadline without shortening a deliberately long one. Supplying
a transport turns off httpx's own proxy discovery, so the proxy mounts are
rebuilt with the same settings, which is what keeps a proxied deployment from
losing the keepalive probes the SDK client would have carried. And httpx
applies `verify`, `cert`, `trust_env`, `http1` and `http2` only to a transport
it builds itself, so those are forwarded rather than silently dropped.

Six `INSPECT_HTTP_*` variables tune these without a release, documented with
their reach in `docs/models-concurrency.qmd`. A provider's own default is a
starting point rather than a refusal to be configured: Groq keeps its tighter
request budget and uncapped pool until an operator sets the variable, which
matters most there because its pool is the unbounded one. An unusable value
keeps the provider's default rather than falling back to the global one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 16d6e6f)

Author: rasmusfaber

CHANGELOG.md

@@ -4,6 +4,7 @@
 - Fixed duplicated task prompt and redundant context sent to the model when using `CompactionAuto` with providers that support native compaction. (#4804)
 - Transcripts now show which compaction strategy actually ran, and flag when `CompactionAuto` fell back from native to summary compaction. (#4804)
 - Logging: Reduced memory usage and event-loop stalls when finalizing long samples with realtime logging; summary-only hooks can opt out of full-sample materialization via `Hooks.needs_full_sample`. (#4879)
+- Models: OpenAI, OpenAI-compatible, Anthropic and Groq now allow 60s rather than 5s for connection setup, so a busy event loop is much less likely to cause `APIConnectionError`. (METR/inspect_ai `faber/http-connect-defaults`, not yet filed upstream)
 
 ## 0.3.258 (11 August 2026)
 

docs/models-concurrency.qmd

@@ -179,6 +179,39 @@ Increasing the max connections might yield better performance due to higher para
 Since it can be difficult to tune this value (especially across different times of day), you are generally much better off using [Adaptive Connections](#adaptive-connections) which will dynamically find the maximum throughput that can be supported.
 
 
+## HTTP Client Settings
+
+The settings above govern how many requests Inspect has in flight. Separately, the HTTP client underneath each provider has its own connection settings, tunable by environment variable so a deployment can adjust them without a release.
+
+The defaults differ from the provider SDKs' in one respect that matters for long-running evals. SDK defaults assume a responsive event loop, but an eval's loop is held for seconds at a time by CPU-bound work, and a connect deadline that expires during one of those pauses fails a connection the kernel had already completed. Inspect therefore allows 60s for connection setup rather than the 5s most SDKs use, and retries connection establishment once.
+
+| Variable | Default | Effect |
+|-----------------|-----------------|--------------------------------------|
+| `INSPECT_HTTP_CONNECT_TIMEOUT` | 60 | Seconds allowed to establish a connection. |
+| `INSPECT_HTTP_REQUEST_TIMEOUT` | 600 | Seconds allowed for the rest of a request. |
+| `INSPECT_HTTP_POOL_CONNECTIONS` | 1000 | Sockets in the HTTP connection pool. Distinct from `--max-connections`, which bounds requests in flight. |
+| `INSPECT_HTTP_POOL_KEEPALIVE_CONNECTIONS` | the connection pool size | Idle connections kept for reuse. |
+| `INSPECT_HTTP_KEEPALIVE_EXPIRY` | 5 | Seconds an idle connection is kept. |
+| `INSPECT_HTTP_CONNECT_RETRIES` | 1 | Retries of connection establishment. Only the connect is retried, never the request, so this is safe for non-idempotent calls. Unavailable through a proxy, which httpx does not retry. |
+
+Two of these interact with settings a provider may already have:
+
+- A request budget you set yourself, via `-M client_timeout`, `-M timeout` or `service_tier="flex"`, is kept. `INSPECT_HTTP_CONNECT_TIMEOUT` acts as a floor alongside it: it raises a connect deadline shorter than the floor, and leaves a longer one alone.
+
+- A provider whose SDK default is deliberately different keeps that default until you set the variable. Groq, for example, uses a tighter request budget and an unbounded connection pool; setting `INSPECT_HTTP_REQUEST_TIMEOUT` or `INSPECT_HTTP_POOL_CONNECTIONS` overrides it.
+
+These settings reach the OpenAI, OpenAI-compatible (including vLLM, SGLang, Ollama, OpenRouter, Together and the other `openai-api/` services), Anthropic and Groq providers.
+
+Google under Trio gets the pool and connect-retry settings but not the deadlines: it applies its own overall budget to every request, which is already far longer than the connect floor.
+
+They do not reach the remaining providers, which are on other HTTP stacks and are not exposed to the problem these defaults address:
+
+- Bedrock and SageMaker have their own connection deadline, already 60s, set per model with `-M connect_timeout`. Azure AI likewise uses its own, which is longer still.
+
+- Google on asyncio applies a single overall budget with no separate connection deadline, so there is none to outlast. (Under Trio it uses a different HTTP stack, which is why the settings apply there.)
+
+- Grok speaks gRPC and establishes connections off the event loop, where the pause that motivates these defaults cannot affect it.
+
 ## Learning More
 
 - [Parallelism](parallelism.qmd): running multiple tasks or models in parallel, sandbox container concurrency, and writing parallel custom code.

src/inspect_ai/_util/http_defaults.py

@@ -0,0 +1,238 @@
+"""Shared HTTP client defaults for provider SDK clients.
+
+The SDK defaults are tuned for a caller whose event loop is responsive. An
+inspect runner's loop is not: long CPU-bound sections (payload transforms,
+transcript scans) hold it for seconds at a time, and a connect deadline that
+expires during one of those blocks fails a connection that the kernel already
+completed — anyio's cancellation lands on the first checkpoint after the loop
+resumes, so a finished connect is no defence.
+
+Three defaults follow, all overridable by environment variable:
+
+* ``connect`` at 60s rather than 5s. A partial raise buys nothing (a 26s block
+  failed 75% at 5s and 76.7% at 15s, 0% at 30s), and the deadline has to clear
+  *several* blocks: a connect that misses its window waits out whole blocks, so
+  failure latency quantises to multiples of the block length (a 3s block
+  against a 5s deadline failed 77% of the time, median 6.1s). The cost is that
+  a dead endpoint takes 120s to detect, since the retry below gives it a second
+  full deadline; httpcore charges the deadline separately to the TCP connect
+  and the TLS handshake, so a slow-then-stalling endpoint can reach 4x, and an
+  SDK that retries multiplies that again.
+
+* ``max_keepalive_connections`` at the connection limit rather than 100. Above
+  the cap every connection is destroyed the moment it goes idle, stepping the
+  new-connection rate ~20x once concurrency passes ~120, and only new
+  connections run the connect path at all.
+
+* One connection-establishment retry. httpcore retries the connect only and
+  never re-sends the request, so it is safe for non-idempotent calls. It needs
+  the loop running again by the time it fires, so it rescues a lightly loaded
+  caller and does little for a saturated one.
+
+The transport also carries the TCP keepalive socket options the Anthropic SDK
+sets on its own, since they live on the transport and supplying one drops them.
+Supplying a transport likewise turns off httpx's environment proxy discovery,
+so the proxy mounts are rebuilt here and given the same settings.
+
+``keepalive_expiry`` stays at httpx's 5s default: raising it past an
+intermediary's idle timeout (an ALB commonly defaults to 60s) trades these
+failures for stale-connection ones. Note this leaves the
+``max_keepalive_connections`` raise inert whenever blocks outlast the expiry,
+which is the case that motivated it (at a 6s block, 100 versus 1000 changed
+nothing, both failing every request). A deployment that knows its own topology
+should raise the expiry alongside the cap.
+"""
+
+from __future__ import annotations
+
+import os
+import socket
+from typing import Any, overload
+
+import httpx
+
+
+def _no_environment_proxies() -> dict[str, str | None]:
+    return {}
+
+
+try:
+    # Private to httpx, but replicating its NO_PROXY rules would drift; the
+    # anthropic SDK vendors a copy for the same reason. Degrade to no mounts
+    # rather than failing to import if httpx moves it.
+    from httpx._utils import get_environment_proxies as _get_environment_proxies
+except ImportError:  # pragma: no cover - httpx moved its internals
+    _get_environment_proxies = _no_environment_proxies
+
+DEFAULT_CONNECT_TIMEOUT = 60.0
+DEFAULT_REQUEST_TIMEOUT = 600.0
+DEFAULT_POOL_CONNECTIONS = 1000
+DEFAULT_KEEPALIVE_EXPIRY = 5.0
+DEFAULT_CONNECT_RETRIES = 1
+
+CONNECT_TIMEOUT_ENV = "INSPECT_HTTP_CONNECT_TIMEOUT"
+REQUEST_TIMEOUT_ENV = "INSPECT_HTTP_REQUEST_TIMEOUT"
+POOL_CONNECTIONS_ENV = "INSPECT_HTTP_POOL_CONNECTIONS"
+POOL_KEEPALIVE_CONNECTIONS_ENV = "INSPECT_HTTP_POOL_KEEPALIVE_CONNECTIONS"
+KEEPALIVE_EXPIRY_ENV = "INSPECT_HTTP_KEEPALIVE_EXPIRY"
+CONNECT_RETRIES_ENV = "INSPECT_HTTP_CONNECT_RETRIES"
+
+
+def _env_float(name: str, fallback: float) -> float:
+    """A non-negative float from the environment, `fallback` on unset or junk."""
+    raw = os.environ.get(name)
+    if raw is None or not raw.strip():
+        return fallback
+    try:
+        value = float(raw)
+    except ValueError:
+        return fallback
+    return value if value >= 0 else fallback
+
+
+@overload
+def _env_int(name: str, fallback: int) -> int: ...
+
+
+@overload
+def _env_int(name: str, fallback: None) -> int | None: ...
+
+
+def _env_int(name: str, fallback: int | None) -> int | None:
+    """A non-negative int from the environment, `fallback` on unset or junk."""
+    raw = os.environ.get(name)
+    if raw is None or not raw.strip():
+        return fallback
+    try:
+        value = int(raw)
+    except ValueError:
+        return fallback
+    return value if value >= 0 else fallback
+
+
+def connect_timeout() -> float:
+    return _env_float(CONNECT_TIMEOUT_ENV, DEFAULT_CONNECT_TIMEOUT)
+
+
+def default_timeout(
+    request_timeout: float = DEFAULT_REQUEST_TIMEOUT,
+) -> httpx.Timeout:
+    """Timeouts, with the environment overriding `request_timeout`.
+
+    The argument is only the fallback, so a provider whose SDK has a budget of
+    its own can pass it and keep it until an operator sets the variable.
+    """
+    return httpx.Timeout(
+        timeout=_env_float(REQUEST_TIMEOUT_ENV, request_timeout),
+        connect=connect_timeout(),
+    )
+
+
+def default_limits(
+    max_connections: int | None = DEFAULT_POOL_CONNECTIONS,
+) -> httpx.Limits:
+    """Pool limits, with the environment overriding `max_connections`.
+
+    The argument is only the fallback, so a provider whose pool is deliberately
+    uncapped can pass None and keep it until an operator sets the variable.
+    """
+    connections = _env_int(POOL_CONNECTIONS_ENV, max_connections)
+    return httpx.Limits(
+        max_connections=connections,
+        # Tracks the pool size unless set outright, so lowering the limit
+        # cannot leave a keepalive cap above it.
+        max_keepalive_connections=_env_int(POOL_KEEPALIVE_CONNECTIONS_ENV, connections),
+        keepalive_expiry=_env_float(KEEPALIVE_EXPIRY_ENV, DEFAULT_KEEPALIVE_EXPIRY),
+    )
+
+
+def _default_socket_options() -> list[tuple[int, int, int]]:
+    """TCP keepalive options matching the ones the Anthropic SDK installs."""
+    options: list[tuple[int, int, int]] = [
+        (socket.SOL_SOCKET, socket.SO_KEEPALIVE, True)
+    ]
+    # Not every knob exists on every platform.
+    for name, value in (
+        ("TCP_KEEPINTVL", 60),
+        ("TCP_KEEPCNT", 5),
+        ("TCP_KEEPIDLE", 60),
+    ):
+        option = getattr(socket, name, None)
+        if isinstance(option, int):
+            options.append((socket.IPPROTO_TCP, option, value))
+    return options
+
+
+async def _floor_connect_timeout(request: httpx.Request) -> None:
+    """Stop a scalar SDK timeout from shortening the connect deadline.
+
+    SDKs stamp their own per-request timeout over whatever the client was built
+    with, and httpx expands a bare float to all four phases, so an overall
+    budget such as `-M client_timeout=30` would become the connect deadline
+    too. Only ever raises it, leaving a longer deadline and a `None` (no limit)
+    alone.
+    """
+    extension = request.extensions.get("timeout")
+    if not isinstance(extension, dict):
+        return
+    # Safe to mutate: httpx.Timeout.as_dict() builds this fresh per request.
+    timeout: dict[str, float | None] = extension
+    connect = timeout.get("connect")
+    floor = connect_timeout()
+    if connect is not None and connect < floor:
+        timeout["connect"] = floor
+
+
+def _transport_kwargs(
+    limits: httpx.Limits, overrides: dict[str, Any]
+) -> dict[str, Any]:
+    """Settings httpx would apply itself if it were building the transport."""
+    kwargs: dict[str, Any] = {
+        "retries": _env_int(CONNECT_RETRIES_ENV, DEFAULT_CONNECT_RETRIES),
+        "limits": limits,
+        "socket_options": _default_socket_options(),
+    }
+    # httpx applies these only to a transport it builds itself, so a caller
+    # passing verify=... would otherwise have it silently dropped.
+    kwargs.update(
+        {
+            arg: overrides[arg]
+            for arg in ("verify", "cert", "trust_env", "http1", "http2")
+            if arg in overrides
+        }
+    )
+    return kwargs
+
+
+def default_client_kwargs(**overrides: Any) -> dict[str, Any]:
+    """`httpx.AsyncClient` kwargs carrying these defaults; caller overrides win."""
+    kwargs = dict(overrides)
+    # `limits` resolves first because httpx applies `limits=` only to a
+    # transport it builds itself, so the transport has to bake them in.
+    limits = kwargs.setdefault("limits", default_limits())
+    kwargs.setdefault("timeout", default_timeout())
+    kwargs.setdefault("follow_redirects", True)
+
+    if "transport" not in kwargs:
+        transport_kwargs = _transport_kwargs(limits, kwargs)
+        # Supplying a transport turns off httpx's environment proxy discovery,
+        # so rebuild the mounts with the same settings rather than losing
+        # either the proxy or the retry and keepalive options.
+        mounts: dict[str, httpx.AsyncBaseTransport | None] = {
+            key: None
+            if url is None
+            else httpx.AsyncHTTPTransport(proxy=httpx.Proxy(url), **transport_kwargs)
+            for key, url in _get_environment_proxies().items()
+        }
+        mounts.update(kwargs.get("mounts") or {})
+        kwargs["mounts"] = mounts
+        kwargs["transport"] = httpx.AsyncHTTPTransport(**transport_kwargs)
+
+    hooks = dict(kwargs.get("event_hooks") or {})
+    hooks["request"] = [_floor_connect_timeout, *hooks.get("request", [])]
+    kwargs["event_hooks"] = hooks
+    return kwargs
+
+
+def default_async_client(**overrides: Any) -> httpx.AsyncClient:
+    return httpx.AsyncClient(**default_client_kwargs(**overrides))

src/inspect_ai/_util/images.py

@@ -5,9 +5,8 @@
 from typing import Awaitable, Callable, Iterator
 from urllib.parse import urlparse
 
-import httpx
-
 from .file import file as open_file
+from .http_defaults import default_async_client
 from .url import (
     data_uri_mime_type,
     data_uri_to_base64,
@@ -103,7 +102,9 @@ async def file_as_data(file: str) -> tuple[bytes, str]:
 
         # handle url or file
         if is_http_url(file):
-            async with httpx.AsyncClient() as client:
+            # Fetched on the eval loop alongside model requests, so it needs
+            # the same connect deadline.
+            async with default_async_client() as client:
                 file_bytes = (await client.get(file)).content
         else:
             with open_file(file, "rb") as f:

src/inspect_ai/model/_openai.py

@@ -59,6 +59,7 @@
     is_retryable_http_status,
     parse_retry_after_from_exception,
 )
+from inspect_ai._util.http_defaults import default_client_kwargs
 from inspect_ai._util.images import file_as_data_uri
 from inspect_ai._util.url import is_http_url
 from inspect_ai.model._call_tools import parse_tool_call
@@ -1222,30 +1223,14 @@ def openai_media_filter(key: JsonValue | None, value: JsonValue) -> JsonValue:
     return value
 
 
-# httpx-native equivalents of openai's DEFAULT_TIMEOUT / DEFAULT_CONNECTION_LIMITS
-# (same values). Do NOT import those from `openai`: openai >= 3.0 is built on
-# httpx2, and its httpx2-typed constants silently corrupt the timeout config of
-# our legacy `httpx.AsyncClient`, failing every request with APIConnectionError.
-DEFAULT_TIMEOUT = httpx.Timeout(timeout=600, connect=5.0)
-DEFAULT_CONNECTION_LIMITS = httpx.Limits(
-    max_connections=1000, max_keepalive_connections=100
-)
-
-
 class OpenAIAsyncHttpxClient(httpx.AsyncClient):
-    """Custom async client that uses OpenAI's default settings.
-
-    This ensures proper proxy support and follows OpenAI's recommended configuration.
-    OpenAI has already incorporated timeout improvements for reasoning models in their
-    default transport, so we don't need custom socket options.
+    """Async client carrying the shared HTTP defaults (see `_util.http_defaults`).
 
+    Do NOT source these from `openai`: openai >= 3.0 is built on httpx2, and
+    its httpx2-typed `DEFAULT_TIMEOUT` / `DEFAULT_CONNECTION_LIMITS` silently
+    corrupt the timeout config of this legacy `httpx.AsyncClient`, failing
+    every request with APIConnectionError.
     """
 
     def __init__(self, **kwargs: Any) -> None:
-        # Use OpenAI's default settings which handle proxies correctly
-        # https://github.com/openai/openai-python/commit/347363ed67a6a1611346427bb9ebe4becce53f7e
-        kwargs.setdefault("timeout", DEFAULT_TIMEOUT)
-        kwargs.setdefault("limits", DEFAULT_CONNECTION_LIMITS)
-        kwargs.setdefault("follow_redirects", True)
-
-        super().__init__(**kwargs)
+        super().__init__(**default_client_kwargs(**kwargs))