Problem
On every CONNECT, ext_proc returns all of the actor's HTTPS SNI patterns for
the dialed port (SNIRules(port) in
cmd/atenet/internal/router/egress/egress.go). Envoy attaches that list to the
tunnel twice: as the CONNECT stream's dynamic metadata, and as a JSON
filter-state string (manifests/ate-install/atenet-egress.yaml, CONNECT-leg
set_filter_state).
The list is read only once, when the Rust listener filter matches the
ClientHello's SNI. Every request after that is checked by ext_proc against its
own per-actor cache. Yet both copies stay in memory for the tunnel's whole life.
So every HTTPS tunnel carries the actor's entire host list, and gateway memory
grows with open tunnels × hosts in the policy. HTTP tunnels carry nothing,
because no HTTPS rule covers port 80.
Memory cost
Measured on one gateway replica with 10,000 open HTTPS tunnels:
| Hosts in the policy |
Envoy memory per tunnel |
vs. a wildcard policy |
| 1 (wildcard) |
89 KiB |
1× |
| 100 |
135 KiB |
1.5× |
| 1,000 |
555 KiB |
6.2× |
HTTP stays at about 83 KiB whatever the policy size.
At 1,000 actors × 1,000 open connections × a 1,000-host policy (1M tunnels),
the gateway needs about 530 GiB of Envoy memory comparing with about 85 GiB for HTTP tunnels.
Problem
On every CONNECT, ext_proc returns all of the actor's HTTPS SNI patterns for
the dialed port (
SNIRules(port)incmd/atenet/internal/router/egress/egress.go). Envoy attaches that list to thetunnel twice: as the CONNECT stream's dynamic metadata, and as a JSON
filter-state string (
manifests/ate-install/atenet-egress.yaml, CONNECT-legset_filter_state).The list is read only once, when the Rust listener filter matches the
ClientHello's SNI. Every request after that is checked by ext_proc against its
own per-actor cache. Yet both copies stay in memory for the tunnel's whole life.
So every HTTPS tunnel carries the actor's entire host list, and gateway memory
grows with open tunnels × hosts in the policy. HTTP tunnels carry nothing,
because no HTTPS rule covers port 80.
Memory cost
Measured on one gateway replica with 10,000 open HTTPS tunnels:
HTTP stays at about 83 KiB whatever the policy size.
At 1,000 actors × 1,000 open connections × a 1,000-host policy (1M tunnels),
the gateway needs about 530 GiB of Envoy memory comparing with about 85 GiB for HTTP tunnels.