[kolsys]

ClickHouse rejects ALTER TABLE ... FREEZE for a table whose storage policy
has a plain_rewritable disk, wherever its parts are
(checkAlterPartitionIsPossible checks every disk of the policy). Until now
create handled only tables with all parts on plain disks and rejected the
common hot/cold layout, a local hot volume with a plain_rewritable cold
volume, with "parts on both plain and non-plain disks".

What changes

create snapshots such a table without FREEZE:

• SYSTEM SYNC REPLICA when sync_replicated_tables asks for it, then
  SYSTEM STOP MERGES and SYSTEM STOP MOVES for the table, started again
  once the table is in the backup. A merge would replace the parts under the
  snapshot, a TTL or manual move between the volumes would remove them, so
  moves are now stopped for every table on a storage policy with plain disks,
  not only for the mixed ones. The backup user needs the SYSTEM MERGES and
  SYSTEM MOVES grants.
• One system.parts query lists the active parts, --partitions and
  freeze_by_part_where apply to it the way they apply to FREEZE PARTITION.
• The parts of the local disks are hardlinked into <disk>/shadow/<uuid>/...,
  the layout FREEZE produces, so MoveShadowToBackup and the rest of the
  pipeline work unchanged. delete-on-destroy.txt, txn_version.txt and
  metadata_version.txt are left out as FREEZE leaves them out. Part paths
  are matched against the raw system.disks.path and accessed through the
  disk_mapping-resolved disk path, so a sidecar with disk_mapping works.
• The parts of the plain disks are copied on the bucket level as before,
  a plain disk in skip_disks is skipped.
• UNFREEZE is rejected for such a table as FREEZE is, so the shadow is
  removed directly, best effort, a leftover is cleaned by cleanOwnFreezes
  at the end of create.
• ignore_not_exists_error_during_freeze applies: a table dropped between
  the listing and the snapshot gives an empty backup of it instead of a
  failed create. A part is skipped only when it is gone as a whole
  (DROP PARTITION), a file missing inside a live part is an error.

Limitations

Only local disks, and encrypted disks over them, are supported next to
plain disks. Parts on an object disk with local metadata (s3, azure,
their cache and encrypted wrappers, web, hdfs) are rejected with an
error naming the disk and its type: their blobs are protected by the
hardlink counter inside the metadata files, which only FREEZE maintains.
A cache wrapper over an object disk is listed by GetDisks under the
underlying disk name while system.parts reports the cache name, the error
message points at this case instead of a missing disk.

restore of the plain disk parts still needs ClickHouse 25.11+
(SYSTEM DROP DISK METADATA CACHE), create works from 24.8.

Tests

• TestPartPathOnDisk and TestFilterPlainPolicyParts (unit) cover the
  disk_mapping path matching and the disk type guard.
• TestHardlinkPart (unit) covers the FREEZE-like snapshot of a part with
  projections and excluded files, HardlinkPart no longer chowns the
  temporary shadow directories so the test runs as root too.
• TestPlainRewritableHybridS3 (integration, s3_plain_rewritable_hybrid
  policy added to dynamic_settings.sh) runs create_remote with and
  without the plain disk in skip_disks on every version from 24.8, checks
  that merges and moves work again afterwards and that the backup metadata
  lists the parts of both disks, and verifies the restored row counts on
  25.11+.

[Slach]

CI failures in run 40681309512 (commit 64e2616) and a follow-up commit e20c634.

26.3 cascade (TestAzure, TestRBAC, TestKafkaKeeperOffsets, … all on env project2): TestPlainRewritableHybridS3 left /var/lib/clickhouse/shadow owned by root, every following FREEZE failed with shadow/increment.txt: Permission denied. Already fixed by prepareShadowDir in ba05fb6.

24.8 / 25.3: TestPlainRewritableHybridS3 fails on MOVE PARTITION ... TO VOLUME 'cold' with code 344. Before 25.4 MergeTreeData::checkAlterPartitionIsPossible rejects every partition command except DROP for a table whose storage policy has a disk without hardlinks, wherever the parts are. ClickHouse lifted this for ATTACH/MOVE in ClickHouse/ClickHouse@b2a1123c13e (first in v25.4.1, not backported to 25.3). That also covers ATTACH PART, so the default restore of such a table on < 25.4 fails even with the plain disk in skip_disks (only local parts in the backup).

I checked it with a standalone script (minio + clickhouse-server, policy hot=default / cold=s3_plain_rewritable, local parts only, create with CLICKHOUSE_SKIP_DISKS=disk_plain, then restore --rm):

ClickHouse	default restore (ATTACH PART)	restore_as_attach: true
24.8.14	code 344	OK
25.3.14	code 344	OK
25.8.24	OK	OK
26.3.34	OK	OK

e20c634:

• restoreDataRegularByParts returns an error before copying to detached when ATTACH PART would run on < 25.4 for a table on a plain policy, and the error suggests clickhouse.restore_as_attach: true.
• TestPlainRewritableHybridS3 now requires 25.4, because its setup needs MOVE PARTITION to the plain volume.

Known leftover: with restore --rm on < 25.4 the table is still dropped and recreated empty during the schema stage, before this check runs. Moving the check before the schema restore would be a separate change.

[coveralls]

Coverage Report for CI Build 40737963885

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Coverage decreased (-2.5%) to 68.036%

Details

• Coverage decreased (-2.5%) from the base build.
• Patch coverage: 46 uncovered changes across 4 files (125 of 171 lines covered, 73.1%).
• 752 coverage regressions across 22 files.

Uncovered Changes

File	Changed	Covered	%
pkg/backup/create.go	123	85	69.11%
pkg/backup/restore.go	11	7	63.64%
pkg/clickhouse/clickhouse.go	16	14	87.5%
pkg/filesystemhelper/filesystemhelper.go	17	15	88.24%
Total (5 files)	171	125	73.1%

Coverage Regressions

752 previously-covered lines in 22 files lost coverage.

Top 10 Files by Coverage Loss	Lines Losing Coverage	Coverage
pkg/backup/restore_cloud.go	240	33.84%
pkg/storage/cos.go	161	3.56%
pkg/server/server.go	104	57.45%
pkg/storage/gcs.go	86	44.11%
pkg/storage/s3.go	31	61.71%
pkg/storage/general.go	26	67.04%
cmd/clickhouse-backup/main.go	22	77.73%
pkg/backup/backuper.go	19	76.28%
pkg/backup/restore_remote.go	12	56.09%
pkg/backup/delete.go	9	74.44%

---

Coverage Stats

Relevant Lines:	29658
Covered Lines:	20178
Line Coverage:	68.04%
Coverage Strength:	26716.95 hits per line

---

💛 - Coveralls