[botdeniska]

Contribution Licensing (no CLA, inbound=outbound, DCO required)

This pull request contributes to PGSTY SILO (pgsty/silo). Code contributions
are accepted under AGPL-3.0-or-later, the same license as the server.
This project does not use a CLA or require a separate Apache-2.0 license grant.
By submitting this pull request I represent that I have the right to contribute
the code changes under this repository's
GNU Affero General Public License v3.0 or later
and retain copyright in my original work. Existing copyright and license
notices remain intact; separately licensed material keeps its applicable terms.
Every commit must carry a DCO Signed-off-by trailer
(git commit -s) certifying the
Developer Certificate of Origin — see
CONTRIBUTING.md.

Description

Fixes #254

getCRCMeta builds the checksum headers for replication requests and drops x-amz-checksum-type (cmd/bucket-replication.go:4246):

for k, v := range cs {
	if k == xhttp.AmzChecksumType {
		continue
	}
	cktype := hash.ChecksumStringToType(k)
	if cktype.IsSet() {
		meta[cktype.Key()] = v
		meta[xhttp.AmzChecksumAlgo] = cktype.String()
	}
}

putReplicationOpts passes these headers to the target's NewMultipartUpload. A multipart object with a FULL_OBJECT CRC32 or CRC32C checksum is therefore replicated into a COMPOSITE upload. replicateObjectWithMultipart completes it with the whole-object CRC, and the target's composite check (cmd/erasure-multipart.go:1586) rejects that with BadDigest: The CRC32 you specified did not match the calculated checksum.. The version stays FAILED and never reaches the target. CRC64NVME is not affected because the target defaults it to FULL_OBJECT.

The skip comes from upstream minio#21535 (4ea6f3b06); before it, the type was forwarded with every checksum. c859041 and 7e079ff do not cover this case: they act on completions of uploads the target already treats as FULL_OBJECT, and replication never sends the type.

With this change getCRCMeta adds x-amz-checksum-type: FULL_OBJECT when the checksum is multipart, a mergeable CRC (CanMerge) and of type FULL_OBJECT. The target then creates a FULL_OBJECT upload. The completion carries the type too, which passes the explicit type check from 7e079ff. Part requests, single-part objects, COMPOSITE checksums and SHA1/SHA256 keep the headers they get today.

Motivation and Context

Since c859041 (#31), SILO accepts FULL_OBJECT CRC32/CRC32C multipart uploads from SDK clients. With replication configured, these objects stay FAILED on the source and never reach the target. Every retry, including the one a HeadObject on a FAILED version queues, sends the whole object again.

Why only multipart FULL_OBJECT CRCs. Forwarding the type for every checksum, as before minio#21535, would send FULL_OBJECT with single-part SHA1/SHA256 checksums, and the target rejects that combination (getContentChecksum, internal/hash/checksum.go:766). Single-part objects and COMPOSITE uploads replicate correctly without the type. CanMerge limits the change to CRC32, CRC32C and CRC64NVME; for CRC64NVME the header only repeats the target's default.

Server-side copies. putReplicationOpts already replicates an object with a single PutObject when the object is not multipart but has a FULL_OBJECT checksum (cmd/bucket-replication.go:883). With the type dropped, that check never matched. A CopyObject of a multipart FULL_OBJECT object is such an object: it keeps the multipart checksum but has a plain ETag. On main it is replicated as a one-part multipart upload and fails with the same BadDigest. With this change the check works as written, and the copy is replicated with a single PUT.

How to test this PR?

go test ./cmd -count=1 -v -run '^(TestGetCRCMetaChecksumType|TestReplicateObjectChecksumTypes)$'

The tests are in the new file cmd/bucket-replication-checksum_test.go.

• TestGetCRCMetaChecksumType checks the headers getCRCMeta builds for single-part objects (SHA256, CRC32, CRC64NVME), multipart COMPOSITE objects (CRC32C, SHA256) and multipart FULL_OBJECT objects (CRC32, CRC32C, CRC64NVME), and for a part request.
• TestReplicateObjectChecksumTypes uploads objects through the S3 API of a test server and replicates them into a second bucket the way replicateObject does: putReplicationOpts, then replicateObjectWithMultipart or PutObject. It checks the replication mode and compares the replica's checksum with the source. The cases are single PUTs (SHA256, CRC32), multipart COMPOSITE (CRC32C, SHA256), multipart FULL_OBJECT (CRC32, CRC32C, CRC64NVME) and a CopyObject of the FULL_OBJECT CRC32 object.

Without the change:

• TestGetCRCMetaChecksumType fails on the three multipart FULL_OBJECT cases because x-amz-checksum-type is missing.
• TestReplicateObjectChecksumTypes fails on mp-FULL_OBJECT-CRC32 and mp-FULL_OBJECT-CRC32C with replication failed: The CRC32 you specified did not match the calculated checksum., and on copy-FULL_OBJECT-CRC32 with multipart replication: got true, want false.

With the change both tests pass.

Run locally on Go 1.27.3 (darwin/arm64), on main b91e1c6f2:

• make verifiers (golangci-lint v2.15.0: 0 issues, check-gen, rebrand-guard), go vet ./cmd, make build
• go test ./cmd -count=1 -run 'Replicat|Checksum|CRC|Multipart'
• go test ./cmd -count=1 -timeout 60m: ok, 274 s

Manual check with two single-drive servers and mc admin replicate add, as in the issue. Objects are uploaded to the first site with aws CLI v2 (2 × 5 MiB parts for multipart). Each cell shows the source replication status; COMPLETED also means that the replica has the same checksum and checksum type as the source.

object	main → main	fix → fix	fix → main	main → fix
multipart FULL_OBJECT CRC32	FAILED, no replica	COMPLETED	COMPLETED	FAILED, no replica
multipart FULL_OBJECT CRC32C	FAILED, no replica	COMPLETED	COMPLETED	FAILED, no replica
CopyObject of the FULL_OBJECT CRC32 object	FAILED, no replica	COMPLETED	COMPLETED	FAILED, no replica
multipart FULL_OBJECT CRC64NVME	COMPLETED	COMPLETED	COMPLETED	COMPLETED
multipart COMPOSITE CRC32	COMPLETED	COMPLETED	COMPLETED	COMPLETED
multipart COMPOSITE SHA256	COMPLETED	COMPLETED	COMPLETED	COMPLETED
single PUT CRC32	COMPLETED	COMPLETED	COMPLETED	COMPLETED

The columns are source → target builds. In the failing cells the target's mc admin trace --errors shows 400 s3.CompleteMultipartUpload for each attempt.

Compatibility impact

No API, configuration, storage metadata, metric, route or dependency changes.

• Replication request headers. For objects whose checksum is a multipart FULL_OBJECT CRC32, CRC32C or CRC64NVME, NewMultipartUpload and CompleteMultipartUpload to the target now carry x-amz-checksum-type: FULL_OBJECT. A copy of such an object, which is not multipart, is replicated with one PutObject carrying the same headers instead of a one-part multipart upload. Part requests and all other objects are unchanged.
• SSE-C objects are unchanged: their checksum is sent encrypted in X-Minio-Replication-Ssec-Crc, and getCRCMeta cannot decrypt it without the client key.
• Mixed versions. Only the source needs the change. A fixed source replicates these objects to an unfixed target; an unfixed source still fails against a fixed target.
• Batch replication builds its options through putReplicationOpts as well (batchReplicationOpts), so it sends the same headers. I did not test it separately.
• Rollback brings the failure back for new attempts. Versions that were already replicated stay as they are.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Optimization (provides speedup with no functional changes)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• All commits are signed off (git commit -s) per the DCO
• Fixes a regression (If yes, please add commit-id or PR # here): upstream 4ea6f3b06 (fix: invalid checksum on site replication with conforming checksum types minio/minio#21535)
• Unit tests added/updated
• make verifiers passes
• Relevant package tests and make build pass
• Compatibility and rollback impact documented
• Internal documentation updated (not needed)
• Public documentation update opened in pgsty/silo.pgsty.com, if needed (not needed)