Repository navigation
fix(test): prevent concurrent scenario image rebuilds - #435
Draft
Nan Liu (liunan-ms) wants to merge 1 commit into
Draft
Nan Liu (liunan-ms) wants to merge 1 commit into
Nan Liu (liunan-ms) wants to merge 1 commit into
Conversation
The image warm-up did not prevent later container requests from rebuilding the shared latest tag. Parallel tests and package processes could replace that tag while another test was creating a container, causing AlreadyExists and No such image failures. Build each Dockerfile once per test process with a generated unique tag and create containers from the cached image reference. Cache build errors so every caller receives the failure without attempting another build. Add regressions for concurrent callers, separate Dockerfiles, and failed builds, and document the container harness behavior. Validation: unit tests, race-detector unit tests, quality checks, and container harness scenarios passed. The full mage all run encountered no-space-left-on-device failures in other scenarios. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The provider caches configuration before reaper timeout overrides are set, risking premature startup failures, and uniquely tagged images lack process-level cleanup, causing disk usage to grow across test runs.
2 open issues
What changed in this PR
This PR updates the scenario container harness to prevent concurrent image rebuilds and shared-tag races across test processes.
Changes:
- Cache each Dockerfile’s image reference or build error once per process, using unique generated tags.
- Create containers from cached images instead of rebuilding them for each request.
- Add cache regression tests and document image reuse and failure behavior.
| File | Description |
|---|---|
scenario/internal/containertest/containers.go |
Uses cached image references and removes fixed-tag and warm-up logic. |
scenario/internal/containertest/containers_internal_test.go |
Removes obsolete fixed-tag assertions. |
scenario/internal/containertest/containercache.go |
Adds synchronized, per-Dockerfile caching of images and build errors. |
scenario/internal/containertest/containercache_internal_test.go |
Tests concurrent callers, separate Dockerfiles, and cached failures. |
docs/developer/how-to/testing.md |
Documents process-local image reuse and build-error reporting. |
Review effort: Balanced
|
|
||
| // Run a simple echo command to cache the container image | ||
| _, err := runCmdInContainerImpl(t, collateral, []string{"echo", "ready"}, NoTimeout) | ||
| provider, err := testcontainers.NewDockerProvider() |
| Context: dockerPath, | ||
| Dockerfile: dockerfile, | ||
| Repo: containerURI, | ||
| KeepImage: true, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Failure
The scenario tests (example) keep failing in
TestCustomSourceRegenerationDetectsStaleHashduring scenario container initialization:The log also shows multiple builds of the same
:latesttag. This is a container-harness failure, before the test can validate the custom-source behavior. The unit-test and E2E jobs in that workflow passed. The branch's investigation also identifiedAlreadyExistsfailures associated with concurrent image rebuilds.Root cause
The existing
sync.Onceonly guarded a best-effort warm-up container for the default Dockerfile. Subsequent container requests still suppliedFromDockerfile, so they could rebuild the same image rather than reuse the warmed image.Parallel tests and separate test-package processes shared fixed image tags (
latestand Dockerfile-derived variants). The process-local warm-up could not prevent those builds from racing to replace a tag while another test was creating a container. Alternate Dockerfiles also bypassed the default-image warm-up. Build failures were not cached as results for subsequent callers.Fix
sync.OnceValuesto build each Dockerfile once per test process and cache both the image reference and any build error.Imagereference instead of rebuilding throughFromDockerfile.Validation
mage unit- passed, rerun on this branch before opening the PR; includes the new image-cache regression tests.mage all- the existing commit records failure due tono space left on devicein other scenarios. The full suite was not rerun while preparing this PR because the local filesystem remains at 100% reported utilization, with approximately 5.7 GiB available. Full scenario validation remains outstanding; this PR is intentionally a draft.