Skip to content

Pull MinIO from a registry that still serves it - #1559

Merged
dimitri-yatsenko merged 1 commit into
masterfrom
fix/minio-test-image
Oct 1, 2026
Merged

dimitri-yatsenko merged 1 commit into
masterfrom
fix/minio-test-image

Conversation

@dimitri-yatsenko

Copy link
Copy Markdown
Member

Unblocks CI on #1555, #1556 and #1558. This is not a flake and re-running will not clear it — I said otherwise on #1555 before diagnosing it, and that was wrong.

What happened

MinIO withdrew its public images:

Oct 2025 stopped publishing free community builds
Feb 2026 archived the open-source repository
2026-09-11 deleted minio/minio from Docker Hub — the whole repository, every tag
2026-09-24 quay.io/minio/minio began requiring authentication

Verified directly rather than inferred: while authenticated to Docker Hub and with datajoint/mysql:8.0 pulling fine, every minio/minio tag returns pull access denied ... repository does not exist, the Hub API returns object not found for the repository itself, and quay.io/minio/minio returns 401 while a control public Quay image pulls normally.

Pinning a tag does not help — the repository is gone, not the tag.

Why it surfaced only now

tests/conftest.py hardcoded minio/minio:latest. Anyone with that image already in their Docker cache — I had a copy from Sept 2025 — keeps passing locally while CI, which starts clean, fails at collection on all five object-storage suites: test_gc, test_npy_codec, test_codecs, test_attach, test_object. That is the whole of the 33 errors on the release PRs.

The fix

cgr.dev/chainguard/minio:latest — public, no credentials, drop-in for testcontainers' MinioContainer. Now read from DJ_TEST_MINIO_IMAGE, matching the existing DJ_TEST_MYSQL_IMAGE, so a deployment can point at its own mirror without editing tests.

It is also the safer image on its own merits. The last community build is a year stale (2025-09-07) and carries unpatched high-severity CVEs with no upstream fix coming; the Chainguard build is rebuilt continuously — the copy I tested was built 2026-09-30.

Verification

Deleted the local MinIO images first, so the pull is cold exactly as in CI:

118 passed  —  test_gc, test_npy_codec, test_codecs, test_attach, test_object

What this does not settle

Whether the test suite should depend on MinIO at all, given the project's direction. Candidates that pull publicly today are LocalStack, SeaweedFS and adobe/s3mock. Worth noting before anyone reaches for the lightest option: DataJoint's storage layer goes through fsspec/s3fs, not boto3, and the GC suite leans on prefix-listing semantics — the area where a thin mock is most likely to diverge. That evaluation belongs in its own issue, not on the eve of a release.

Sources for the timeline: MinIO removal reporting, Quay anonymous-pull change.

MinIO withdrew its public images. `minio/minio` on Docker Hub was deleted on
2026-09-11 -- the whole repository, not just `latest`, so pinning a tag does
not help -- and `quay.io/minio/minio` began requiring authentication on
2026-09-24. Free community builds stopped in October 2025 and the open-source
repository was archived in February 2026, so this is a withdrawal rather than
an outage: re-running a job will not clear it.

Every object-storage suite fails at collection in CI as a result -- test_gc,
test_npy_codec, test_codecs, test_attach, test_object -- while passing locally
for anyone with the old image still in their Docker cache, which is why this
did not show up until CI ran on a clean machine.

Switches the fixture to Chainguard's build, which is public, needs no
credentials, and is drop-in for testcontainers' MinioContainer. It is also the
safer choice on its own merits: the last community image is a year stale and
carries unpatched high-severity CVEs, while the Chainguard build is rebuilt
continuously.

The image is now read from DJ_TEST_MINIO_IMAGE, matching DJ_TEST_MYSQL_IMAGE,
so a deployment can point at its own mirror without editing tests.

Verified with the local MinIO cache deleted, so the pull is cold as it is in
CI: 118 passed across the five object-storage suites.
@dimitri-yatsenko

Copy link
Copy Markdown
Member Author

Follow-up filed as #1560 (v2.4): whether to keep depending on MinIO in the test fixture at all.

This PR stays the right fix for the release — it changes no test code and is verified cold. #1560 carries the evaluation, including the constraint that decides it: the storage layer goes through fsspec/s3fs rather than boto3, and gc.py walks stores recursively, so isdir and walk ride on s3fs's emulation of directories over list_objects_v2. A thin mock that diverges there will not raise — it will return a different key set and let a GC bug through green.

@dimitri-yatsenko
dimitri-yatsenko merged commit f80d7aa into master Oct 1, 2026
17 checks passed
@dimitri-yatsenko
dimitri-yatsenko deleted the fix/minio-test-image branch October 1, 2026 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants