sandboxd: pluggable metadata sources backed by PostgreSQL (TenantSource, ConfigSource, Catalog) for 10,000 nodes and 1M users #291

Summary

sandboxd keeps all of its control state in node-local files: config.json, pools.json and tenants.json. A mesh shares only placement hints and a digest of the settings that must match. That design fits one mesh of hundreds of nodes (see the Scale section of docs/cluster.md). This issue covers how the state splits when the target is 10,000 nodes serving 1,000,000 users, and introduces PostgreSQL as an optional, strongly consistent source behind three interfaces:

• TenantSource
• ConfigSource
• Catalog

Each interface hides a cache over its source. When no PostgreSQL is configured, it falls back to today's file and gossip implementation with identical behavior.

Not in scope: changing the claim hot path, replicating claims, or putting per-request events into PostgreSQL.

Scale assumptions

Measured numbers this analysis relies on:

• Gossip state per push/pull:
  • 0.63 MB at 1,000 nodes and 6.27 MB at 10,000, each with 5 templates per node;
  • 352.77 MB at 10,000 nodes with 500 templates per node;
  • memberlist refuses state over 20 MiB and stretches push/pull to 10 s at 10,000 nodes (mesh: delta gossip and a compact template summary, so one cell scales past the full-state push/pull #290).
• Tenant set held in memory (sandboxd: manage tenants at runtime through /v1/tenants, persisted like pools.json #286):
  • auth is 102 ns at 10,000 tenants;
  • each change rebuilds and fsyncs the whole set, about 21 ms and 17.6 MB allocated at 10,000 tenants, growing linearly with N.

Today: every piece of state

config.json (read at boot; the reloadable subset after #288)

Pools (config.json pools[], <data_dir>/pools.json)

Tenants (config.json tenants[], <data_dir>/tenants.json)

Runtime state that is not configuration

Where each piece belongs at 10,000 nodes and 1 M users

Placement rules:

1. Node bootstrap stays in the local file. That covers node identity, paths, attachment, engine knobs and the new metadata store DSN. None of it is shared.
2. Desired state goes to PostgreSQL, read through a cache. That covers cell config, pool definitions and targets, tenants, and the template and checkpoint catalog.
3. Fast-moving placement stays in gossip, within one cell: warm counts and membership.
4. High-volume events never go to PostgreSQL: usage and audit stream to a log pipeline.
5. Secret values never go to PostgreSQL: secrets[] defines names and headers; values come from the node environment, a secret manager or the claim env.

Tenants at 1 M users

• Memory: a node caches only tenants active on it. That is tens to hundreds per node, against 1 M rows held on every node with the current set.
• Cold lookups: 1 M users × 1–3 nodes a day is about 3 M lookups a day, or roughly 35 per second averaged and a few hundred per second at peak. An indexed PK lookup on a replica handles that easily.
• Writes: each sign-up is one row insert plus one notify. Today it is an O(N) rebuild and fsync on every node.
• Revocation:
  • A notify (LISTEN/NOTIFY through a per-cell listener, or a version poll) evicts the entry everywhere within seconds.
  • Operations that extend a claim's life (renew, fork) may check the source directly to make revocation immediate.
  • The removed-tenant seals from sandboxd: manage tenants at runtime through /v1/tenants, persisted like pools.json #286 keep working, with "absent from the set" replaced by "revoked or deleted in the source".
• Negative cache:
  • An unknown token must not reach PG on every request. Cache misses for a short TTL, and cap the source lookups per second per node.
  • Without both, a stream of random tokens loads the database.
• The alternative stays open: one tenant per control plane, with users in claim_ref and metadata, needs none of this. TenantSource makes per-user tenants viable; it does not require them.

Pools at 10,000 nodes

• One template release becomes a single cell-document change that every node in the cell applies through the reload path. Goldens are rebuilt where the stamp changed (sandboxd: reload config-owned settings at runtime (per-pool egress/warmup/storage, new template keys) without a restart #288). Today the same release is an edit and a restart on every node.
• Targets move from per-node pools.json to a cell-level desired state. An autoscaler writes one row per cell instead of fanning out to every node, and PUT /v1/pools becomes a write to the source.

Config distribution at 10,000 nodes

• Poll load: every node polls one version row on a replica, with jitter. At a 5 s interval that is 2,000 trivial reads per second fleet-wide.
• Fetch on change: the document (kilobytes) is fetched only when the version moves.
• Applied version: each node writes its applied version and the outcome to node_status. Convergence is a query, and a node that refused a reload names the reason there.
• Connection count: 10,000 nodes must not hold direct connections. Put PgBouncer in front, route reads to replicas, and allow a separate DSN for reads and writes.

Interfaces

They follow the repository layout: the interface sits in the parent package, each implementation in a subpackage, and config.json picks one:

{"meta_store": {"kind": "pg", "dsn_env": "SANDBOXD_PG_DSN", "read_dsn_env": "SANDBOXD_PG_READ_DSN"}}

meta_store is read at boot only. When it is absent, every source is the file or gossip implementation, and behavior is unchanged.

• TenantSource
  • Methods: resolve by token SHA-256 (hot path, cached); put, delete and replace; and a paginated list, because GET /v1/tenants cannot return 1 M rows at once.
  • It owns invalidation, the negative cache and the lookup rate cap.
• ConfigSource
  • Methods: return the current cell document with its version, notify on a version change, and record the applied version or the refusal.
  • The file implementation is the local file plus SIGHUP or POST /v1/config/reload. That is what sandboxd: reload config-owned settings at runtime (per-pool egress/warmup/storage, new template keys) without a restart #288 builds: its ReloadConfig(ctx, *config.Config) is already the seam.
• Catalog
  • Methods: put, delete, and find the owners of a template or checkpoint by key and scope.
  • The fallback implementation is today's gossip plus probe.

Each interface lands in the same PR as its PostgreSQL implementation. That PR also moves today's code behind it unchanged and adds one contract test suite that runs against both implementations. The PostgreSQL leg runs only when SANDBOXD_TEST_PG_URL is set and is reported as skipped otherwise.

Phases

1. Catalog for templates and checkpoints. Templates leave the gossip, which takes the template part of mesh: delta gossip and a compact template summary, so one cell scales past the full-state push/pull #290 and the root-scope key, and checkpoint owner probes become one lookup. This is the prerequisite for cells.
2. TenantSource: cached resolve, notify invalidation, negative cache, paginated listing, fleet-wide quota leases.
3. ConfigSource: a versioned cell document applied through the sandboxd: reload config-owned settings at runtime (per-pool egress/warmup/storage, new template keys) without a restart #288 reload, pool definitions and targets moved into it, and node_status.
4. Sandbox directory and the usage/audit pipeline.

Cell routing (user to cell) sits in front of sandboxd, in the control plane, and is outside this issue. sandboxd only has to keep each mesh within its envelope.

Acceptance per phase

• Hot path:
  • claim and per-request latency are unchanged with PostgreSQL configured, on a cache hit, measured A/B in one run;
  • a cold tenant costs one lookup;
  • there is no PG round trip on the claim path.
• PG outage:
  • with the primary and replicas stopped, running claims, exec and egress keep working;
  • cached tenants authenticate and uncached ones fail closed;
  • writes return a clear error;
  • recovery needs no restart.
• Hardware: a two-cell testbed (two meshes) sharing one PostgreSQL, running the cluster regression plus one leg per phase: a tenant added in one cell authenticates in the other, a template promoted in one is found from the other, and a config version applies across a cell with node_status converging.

Open questions

1. Which PostgreSQL schema owns migrations: a sandboxd migrate subcommand, or the data-source team's own tooling?
2. Should tenant quota be fleet-wide (leases) or stay per node?
3. How strict should revocation be: the notify interval for everything, or a direct source check on renew and fork as proposed above?
4. Pool targets: a cell-level target split by sandboxd, or per-node-class targets set by the autoscaler?