atunnel uses less memory for idle egress connections by chuangw6 · Pull Request #2908 · agent-substrate/substrate

Chuang Wang (chuangw6) · 2026-10-19T21:01:23Z

Each idle egress tunnel held about 157 KiB and 5 goroutines: a 32 KiB
io.Copy buffer per direction in copyBothWays, and a 32 KiB buffer held
by the HTTP/2 stream's reader goroutine while it waited. At a 1 GiB
limit, atunnel was OOM-killed at around 6,000 idle connections, which
breaks egress for every actor on the worker.

Changes:

copyBothWays waits for data with a 512 B buffer and borrows a pooled
32 KiB buffer only while data is moving.
h2StreamConn.WriteTo reads the response body directly, so the proxy
doesn't start a reader goroutine per stream. With a read deadline
set, it falls back to Read.

benchmarking/atunnel, 1 CPU, 1 GiB:

	before	after
EgressIdleConns, per connection	157 KiB, 5.1 goroutines	29 KiB, 4.0 goroutines
EgressIdleConns, 8,000 connections	OOM-killed	226 MiB
EgressTCP (average of 3 runs)	311 MB/s	302 MB/s
EgressHTTP at 8,000 req/s	1.00 core	0.83 core

EgressTCP is about 3% slower because bulk copies do a few extra small
reads. Most of the remaining 29 KiB is net/http's 16 KiB HTTP/2 send
buffer, which this doesn't touch.

Tests pass
Appropriate changes to documentation are included in the PR

Each egress tunnel held about 157 KiB while open, even with no data moving: a 32 KiB io.Copy buffer per direction, plus a 32 KiB buffer held by the HTTP/2 stream's reader goroutine. With a 1 GiB limit, about 6,000 idle connections were enough to OOM-kill atunnel. copyBothWays now waits for data with a 512 B buffer and borrows a pooled 32 KiB buffer only while data is moving. h2StreamConn.WriteTo reads the response body directly, so the reader goroutine and its buffer are gone. With a read deadline set, it falls back to Read.

Bowei Du (bowei) · 2026-10-19T21:18:43Z

Can we see how the numbers change if we have a 512 byte, 1k, 2k byte buffers?

I'm curious whether or not this makes an impact.

Chuang Wang (chuangw6) · 2026-10-19T21:57:39Z

Bowei Du (@bowei) Ran 512 B, 1 KiB, 2 KiB and 4 KiB (same setup, 1 CPU / 1 GiB):

wait buffer	idle, per conn (8k held)	EgressHTTP 8k req/s	EgressTCP, 3 runs (MB/s)
512 B	27.4 KiB	0.85 core	299, 348, 385
1 KiB	28.3 KiB	0.83 core	306, 297, 301
2 KiB	30.3 KiB	0.82 core	298, 359, 288
4 KiB	34.3 KiB	0.83 core	298, 338, 301

Idle memory grows by about 2 KiB per extra KiB (one buffer per
direction). HTTP CPU is flat, and the TCP spread between runs is bigger
than any difference between sizes. So size doesn't seem to matter much
for throughput. I'd keep 512 B, but 1 KiB is fine too if we want a
rounder number.

Lior Lieberman (LiorLieberman)

LGTM. Thanks!

…strate#3072) EgressIdleConns couldn't get past 64,512 connections. All of them came from the egress actor in one sandbox, and atunnel's redirect sends every connection to the same listener, so only the source port tells them apart. That's a per-actor limit, not a per-worker one. The worker now runs an egress actor in every sandbox, and EgressIdleConns splits its connections across them. The other egress scenarios still use actor-0's, so their numbers stay comparable with earlier runs. Also adds a 100k run, with atunnel at 4 CPUs / 16 GiB so neither limit is reached: | idle connections | per connection (after GC) | goroutines/conn | atunnel cgroup memory | CPU opening 200 conns/s | |---|---|---|---|---| | 25,000 | 26.9 KiB | 4.0 | 1.3 GiB | 0.15 cores | | 50,000 | 27.1 KiB | 4.0 | 2.6 GiB | 0.14 cores | | 100,000 | 26.8 KiB | 4.0 | 5.2 GiB | 0.19 cores | Memory per connection stays flat. Before agent-substrate#2908 it was about 157 KiB. The CPU is what opening new connections costs (about 1 ms each); holding the idle ones costs almost none. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR

Chuang Wang (chuangw6) requested review from Lior Lieberman (LiorLieberman) and Bowei Du (bowei) October 19, 2026 21:05

Lior Lieberman (LiorLieberman) previously approved these changes Oct 20, 2026

View reviewed changes