Skip to content

fix(test): prevent concurrent scenario image rebuilds - #435

Draft
Nan Liu (liunan-ms) wants to merge 1 commit into
mainfrom
liunan/tests-fix
Draft

Nan Liu (liunan-ms) wants to merge 1 commit into
mainfrom
liunan/tests-fix

Conversation

@liunan-ms

@liunan-ms Nan Liu (liunan-ms) commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Failure

The scenario tests (example) keep failing in TestCustomSourceRegenerationDetectsStaleHash during scenario container initialization:

create container: Error response from daemon: No such image: github.com/azldev/azldev-test/scenario/container:latest

The log also shows multiple builds of the same :latest tag. This is a container-harness failure, before the test can validate the custom-source behavior. The unit-test and E2E jobs in that workflow passed. The branch's investigation also identified AlreadyExists failures associated with concurrent image rebuilds.

Root cause

The existing sync.Once only guarded a best-effort warm-up container for the default Dockerfile. Subsequent container requests still supplied FromDockerfile, so they could rebuild the same image rather than reuse the warmed image.

Parallel tests and separate test-package processes shared fixed image tags (latest and Dockerfile-derived variants). The process-local warm-up could not prevent those builds from racing to replace a tag while another test was creating a container. Alternate Dockerfiles also bypassed the default-image warm-up. Build failures were not cached as results for subsequent callers.

Fix

  • Replace the warm-up with a concurrency-safe image cache keyed by Dockerfile.
  • Use sync.OnceValues to build each Dockerfile once per test process and cache both the image reference and any build error.
  • Let testcontainers generate unique image tags, isolating builds from other test processes.
  • Create containers using the cached Image reference instead of rebuilding through FromDockerfile.
  • Report build failures explicitly, including the initiating caller's captured build log, and return the cached error to subsequent callers.
  • Remove the obsolete warm-up helper and fixed-tag logic.
  • Add regressions for 16 concurrent callers sharing one build, independent Dockerfile caches, and cached build failures.
  • Document the container harness's image reuse and error behavior in the developer testing guide.

Validation

  • mage unit - passed, rerun on this branch before opening the PR; includes the new image-cache regression tests.
  • The existing commit records successful race-detector unit tests, quality checks, and container-harness scenarios. Those checks were not rerun while preparing this PR.
  • mage all - the existing commit records failure due to no space left on device in other scenarios. The full suite was not rerun while preparing this PR because the local filesystem remains at 100% reported utilization, with approximately 5.7 GiB available. Full scenario validation remains outstanding; this PR is intentionally a draft.

The image warm-up did not prevent later container requests from
rebuilding the shared latest tag. Parallel tests and package processes
could replace that tag while another test was creating a container,
causing AlreadyExists and No such image failures.

Build each Dockerfile once per test process with a generated unique tag
and create containers from the cached image reference. Cache build errors
so every caller receives the failure without attempting another build.
Add regressions for concurrent callers, separate Dockerfiles, and failed
builds, and document the container harness behavior.

Validation: unit tests, race-detector unit tests, quality checks, and
container harness scenarios passed. The full mage all run encountered
no-space-left-on-device failures in other scenarios.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 14, 2026 05:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The provider caches configuration before reaper timeout overrides are set, risking premature startup failures, and uniquely tagged images lack process-level cleanup, causing disk usage to grow across test runs.

2 open issues
What changed in this PR

This PR updates the scenario container harness to prevent concurrent image rebuilds and shared-tag races across test processes.

Changes:

  • Cache each Dockerfile’s image reference or build error once per process, using unique generated tags.
  • Create containers from cached images instead of rebuilding them for each request.
  • Add cache regression tests and document image reuse and failure behavior.
File Description
scenario/​internal/​containertest/​containers.go Uses cached image references and removes fixed-tag and warm-up logic.
scenario/​internal/​containertest/​containers_internal_test.go Removes obsolete fixed-tag assertions.
scenario/​internal/​containertest/​containercache.go Adds synchronized, per-Dockerfile caching of images and build errors.
scenario/​internal/​containertest/​containercache_internal_test.go Tests concurrent callers, separate Dockerfiles, and cached failures.
docs/​developer/​how-to/​testing.md Documents process-local image reuse and build-error reporting.

Review effort: Balanced


// Run a simple echo command to cache the container image
_, err := runCmdInContainerImpl(t, collateral, []string{"echo", "ready"}, NoTimeout)
provider, err := testcontainers.NewDockerProvider()
Context: dockerPath,
Dockerfile: dockerfile,
Repo: containerURI,
KeepImage: true,
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants