Skip to content

WorkerPool can exceed maxWorkers while existing workers retire #142

Open

Description

@RichardHoekstra

Reproduction

Configure two workers with two lifetime calls per worker. Queue eight calls behind a shared barrier. The original pool allocates four workers while the limit is two; releasing the barrier lets them drain.

Expected behavior

Retiring and closing threads should occupy their pool slots until physical exit. New allocation should wait when all slots are occupied.

Proposed fix

A tested local candidate retains those slots until exit. It also makes pool close permanent, rejecting queued/future calls, and evicts disposed OpenAPI services from their cache.

The limit violation reproduces on the official 6.2 image for GitHub main a91a7a28. The earlier candidate passed retirement, collection and selected compatibility checks on 6.1; it still needs updating to the current base.

Decision needed

Is WorkerPool.close() permanent disposal or a reusable reset? The fix and cache behavior need to follow that contract.

This reproduction shows excess allocation during retirement, not a permanent leak. The candidate bounds threads per pool; active calls and queued bytes remain unbounded.

Activity

  1. unilynx commented on Oct 27, 2026

    @unilynx
    Member

    i'm pretty sure the intent was to have new workers available as fast as possible and not have to worry about slow timeouts and other things that might keep old workers alive. That does risk a memory starvation - but the other side of the medal is a starvation of executors

    @blexrob can you weigh in?

  2. RichardHoekstra commented on Oct 27, 2026

    @RichardHoekstra
    ContributorAuthor

    Runnable reproduction: fixture and instructions, with a disposable Docker runner.

    From that checkout:

    python3 whtree/modules/platform/scripts/repro/run.py --docker 142

    On the pinned official 6.2 image for a91a7a28: 4 worker allocations with maxWorkers=2, 8 started calls and 8 completed calls after releasing the barrier. The fixture closes the pool and waits for worker exit.

    This records the allocation behavior discussed above; the availability-versus-capacity policy remains open. No proposed pool implementation is installed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions