feat(core): the warm pool module: one budget, a keep rule that fits, and an off switch

[V3RON]

Closes #368

Status

Implement: done (14 policy, 17 converger, engine and unit tests green; e2e files touched pass) Review: round 5, 0 open Mutate: 3 alive, equivalent (below) Hardware: n/a Gate: pending

Done when

• At the running cap with simlock lease --class phone waiting, simlock release of a fitting iPhone grants that device to the waiter and simlock events shows no device.provisioned for it — lease-engine.test.ts "a release at the running cap with a class request waiting grants the released device and provisions nothing" (the only device.provisioned is the seed's)
• With warmPool.enabled: false, a released fake iOS device is shutdown and a released fake Android device is shutdown after its device.reclaimed with strategy snapshot — engine test "with warmPool disabled …" and e2e/lease-lifecycle.test.ts "with warmPool.enabled false …"
• With maxRunning lowered below the number of ready devices and the daemon restarted, the excess devices are shutdown with initiator warm-pool after daemon.started — engine tests "shuts down one of three ready devices …" and "shuts down an unleased ready device over maxRunning at startup …"
• simlock config shows warmPool.enabled: true by default — config.test.ts "keeps the warm pool on by default …"

Assumptions

• Assumption: a boot reserves with tryReserveBoot (a running slot and the RAM), not tryReserveRewarm — a shut-down device holds no running slot, and tryReserveRewarm checks RAM only; the acquisition path boots a shut-down device the same way.
• Assumption: a pass proposes no budget shutdown while a device is being handed to a lease (a boot claim on a ready device) — that device counts as running and as reserved until the grant, so the budget reads one over for a moment; the grant triggers the next pass.
• Assumption: a boot for a waiting request is proposed for one device, for the request at the head of the queue.
• Assumption: a recently released device is one whose lastLeaseEndedAt is less than idle.shutdownAfterMs ago, strictly.
• Assumption: a request whose device work is in flight (processing) is not waiting, but still holds one free slot on its platform from the speculative "recently released" boots, as does a waiting request no idle device serves — otherwise the pool boots back a device the waiter just evicted, in a loop.
• Assumption: a failed boot or shutdown is not retried for the same device for one tick (30 s) — its reservation release triggers the next pass at once and a broken boot would loop.
• Assumption: the pool does nothing while an operator reset (nuke) holds acquisition closed, and stops mid-pass when one begins.
• Assumption: ReleaseCoordinator's post-claim notification also requests a pass: device.reclaimed fires while the reclaim claim is still held, so the pass it triggers sees the device as busy.
• Assumption: engine.settle() also waits for the pool's running pass, so tests and graceful stop see the settled pool.
• Assumption: e2e/doctor-drift.test.ts accepts ready for the device doctor --fix marked shut down, and a makeReady for only that device — the pool boots a device released a moment ago back; the fix is still proved by its device.shutdown event with initiator doctor.
• Assumption: none left here for head-only boots: the spec now says the pool boots only for the queue head, in flight or not (a request behind it holds a slot but gets no boot).
• Assumption: a nuke waits for the pool's in-flight pass after closing acquisition, so a boot already running cannot outlive the reset.
• Assumption: a device that is shutdown while an operator reset (nuke) holds acquisition closed is never booted back as "recently released", until it leaves shutdown (the mark is in memory; the forced release stamps lastLeaseEndedAt = now, which would otherwise undo a nuke without --delete-devices).

Notes

• The A released device that fits a waiting class request is shut down and another booted #350 engine test already passed on main after Reclaim is its own coordinator and commits what the driver returns #373 (a shut-down device that serves the waiter is booted by the planner at the cap); it is kept as the guard that the pool does not undo it.
• Alive mutants, all equivalent: converger.ts #again = false initial value (reset at the start of every drain); lease-engine.ts ...(logger === undefined ? {} : { logger }) (an absent logger is the same as logger: undefined); src/http/test-fakes.ts enabled: true (a fixture value no test reads).
• Behaviour to look at in review: a device a person stopped outside Simlock and doctor --fix marked shutdown is booted back if it was released within idle.shutdownAfterMs.

Review

Spec review: 10 blocking, 4 fixed, 11 notes. Code review: 8 blocking, 6 fixed, 13 notes.
Mutate: 3 mutants, 3 alive.

Rejected:

• spec: Assumption 1 (tryReserveBoot, not tryReserveRewarm) drops ADR 0017's full-size RAM pricing — same canBoot check and boot entry, plus a running slot; stricter, no spec outcome changes.
• spec: Assumption 5 counts an in-flight provision or boot twice — it only delays a speculative boot to the next trigger; no limit is exceeded. Kept as a note.
• spec: booting only for the queue head decides behaviour nobody decided — the maintainer decided it on The warm pool module: one budget, a keep rule that fits, and an off switch #368 (head only; per-platform lines in A request waits only behind requests for the same platform #387).
• code: a request arriving during a pool warm boot provisions or evicts instead of waiting (ADR 0017 §6) — out of scope here; A reserve of running slots, and a request waits for a device on its way #369 delivers it.

Written by an agent.

[V3RON]

Review notes

Not blocking, not verified. Each is one reviewer's claim.

• spec: src/core/warm-pool/policy.ts:164 a provision or boot in flight counts in reserved and again as unserved, so one fewer recently released device is booted back while it runs.
• code: src/core/domain.ts:93 the WaitingDemand.inFlight doc ("a slot coming that is not yet reserved") is false for a provision or boot in flight.
• spec: src/core/warm-pool/converger.ts:156 the pool skips kick() when the lifecycle declines a shutdown or boot.
• spec+code: src/core/lease-engine.ts:257 the release coordinator's notifyAvailability calls the pool's pass(), a trigger outside the spec's list (ADR 0017 §3, architecture rule 15).
• spec: src/core/lease-acquisition-coordinator.ts:565 waitingDemand() also lists processing waiters with inFlight, unlike the spec's shape.
• spec: e2e/doctor-drift.test.ts accepts ready for deviceA, so a regression where doctor --fix boots it would pass.
• code: src/core/warm-pool/converger.ts:82 a trigger can be lost between #drain's last #again check and the .finally clearing #running; the 30 s tick recovers it.
• code: src/core/acquisition-planner.ts:87 a request during a pool bootWarm provisions or evicts instead of waiting; docs/CONFIGURATION.md:26 understates it (left to A reserve of running slots, and a request waits for a device on its way #369).
• code: src/core/warm-pool/converger.ts:146 a device whose warm boot keeps failing is retried every 30 s for up to idle.shutdownAfterMs.
• code: src/core/lease-engine.ts:255 daemon stop right after a release boots the device back and waits for it — filed as daemon stop right after a release boots the released simulator back and waits for it #388.
• code: src/core/warm-pool/converger.ts:171 the pool acts on devices whose driver was refused at discovery — filed as The warm pool retries a shutdown on a refused platform's device every 30 s #389.
• code: src/core/warm-pool/converger.ts:125 boot proposals are computed at pass start, so a later pool boot can take a slot freed by a failed one before a queued request's decision runs.
• code: src/core/warm-pool/converger.ts:236 the device.reclaimed trigger never lets the pool act on the reclaimed device (claim still held).

Written by an agent.