Repository navigation
Conversation
The ThinLTO link of `clickhouse` and `clickhouse-keeper` dominates the release build: ~82 min of a ~117 min `amd_release` build on a 16-core, 62 GB builder. lld can cache the optimized native object of every module (`--thinlto-cache-dir`), keyed by the module's bitcode hash, its import set and the LTO configuration, and reuse it on the next link when nothing in that key changed. Only the parallel backend phase is cached; the single-threaded thin-link and the final native link still run in full. Measured on the same builder class with the production `-g` configuration (sha e3f85bc, same commit built twice): 114.5 min cold, 65.5 min warm. Forty-six commits of churn on top of a warm cache cost about 3 min, since a header change only invalidates modules whose emitted IR changed. The cache lives on an attached volume at `/mnt/clickhouse_thinlto`. praktika gains `--volume-if-exists=HOST:CONTAINER` in a job's docker settings, which mounts the host path only on runners that have it rather than creating an empty directory everywhere as `--volume` does, so the build job enables the cache by the directory's presence and builders without the volume run unchanged. lld prunes inside the linker after each link's backend has run, removing the least recently used entries until the cache fits the size target, so the policy is size-only (`prune_after=0s`, lld's own default is 7 days) at 90% of the volume. The volume must be mounted `relatime` so that hits refresh access times. One `amd_release` `-g` population is ~15 GB. The build job also prints the output of its `sccache --show-stats` and cache stats commands, which `Shell.check` otherwise discards. Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
Each ThinLTO link spends most of its time in a single-threaded thin-link phase, so two links overlap almost entirely. Measured on a 16-core, 62 GB builder with zram: 114.5 -> 83 min cold and 65.5 -> 48.5 min warm (with the ThinLTO cache), with a combined memory peak of ~48 GB. `limit_jobs.cmake` caps ThinLTO links from a budget of ~100 GB per `-g` link and applied that cap even to an explicit `PARALLEL_LINK_JOBS`, so a 62 GB builder could never link both binaries at once. The cap now applies only to the auto-calculated value, and the build job sets two link jobs for every ThinLTO flavour. Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
Cache does not belong in meta label since builder instances are also used for some tests
strtgbb
added
cicd
On the builder volume the root-owned `lost+found` directory is unreadable for the build user, so `du` failed and the `&&` chain stopped before the entry count and `df`. Exclude it from `du`, count only lld's own `llvmcache-*` entries, and run the three commands independently. https://github.com/Altinity/ClickHouse/actions/runs/40545556921/job/122623854938 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
amd and arm cache entries never share keys, so a build only benefits from a volume where a build for the same target ran recently. With one volume name, the scaler hands each build whichever detached volume it finds first, and a master push runs two amd and two arm ThinLTO builds at once. Give each target its own meta label, `altinity-thinlto-cache-amd` and `altinity-thinlto-cache-arm`, which the runner controller expands into the volume (`clickhouse_thinlto_amd`, `clickhouse_thinlto_arm`) and its location, so neither size nor location is fixed in the repository. The build job mounts whichever of the per-target volumes is present, or the single `/mnt/clickhouse_thinlto` directory that static builders keep on their own disk for both targets. The 180 GB cap in the cache policy stays, since those builders hold both targets' entries. `arm_release` now runs on `BUILDER_ARM`, and `amd_cfi` and the amd cache warmup build on `BUILDER_AMD`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
strtgbb
force-pushed
the
ci/antalya-26.8/thinlto-link-cache
branch
from
October 28, 2026 13:18
511923f to
0d56da2
Compare
Most runners are in Hetzner, so every job that downloads the submodule archive from AWS S3 pays egress. Keep the archive in a Hetzner bucket instead. Buckets listed in `S3_EXTERNAL_BUCKETS` are served by a boto3 client with their own endpoint and credentials. `S3.put` and `S3.head_object` use it for those buckets, and keep using the AWS CLI for every other bucket. With `public_read`, reads are unsigned, so community PRs can use the archive without credentials; they never create it. A missing credential is an error, so ambient AWS credentials are never sent to the Hetzner endpoint. The archive is public, so `_prepare_submodule_cache` refuses to upload it if the clone token ended up in a git config file. The workflows export `HETZNER_S3_ACCESS_KEY` and `HETZNER_S3_SECRET_KEY` from GitHub secrets. The bucket is `altinity-builds-cache` in hel1. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The first CI run failed with a 403 on the existence check for the submodule archive. The bucket is public for object reads, but answers a HEAD of a missing key with 403 to callers who cannot list it. The config job read the bucket unsigned because the bucket is marked `public_read`, even though it has the keys. Reads are now unsigned only for jobs without the bucket's keys. Those jobs treat a 403 from `head_object` as a miss; jobs with the keys use a signed client, so a 403 is still an error for them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The Altinity PR workflow has full-suite sanitizer jobs that use CAS storage (`Stateless tests (amd_asan_ubsan, cas s3 storage, ...)`), which the test rejected as unexpected full-suite jobs. It failed in every CI run, including without other changes. Related: https://github.com/Altinity/ClickHouse/actions/runs/41066953900 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ubmodule-cache' into ci/antalya-26.8/thinlto-link-cache
Attached volumes tied the cache to whichever builder a job landed on, and the runner scaler never reused a volume for builders made from a snapshot image. The cache now travels as one archive per branch, build type and debug-info flavour in the Hetzner bucket from the submodule cache: `altinity-builds-cache/ci_ch_cache/thinlto/<branch>/<build type>-<g|g0>.tar.zst`. Every ThinLTO build that links restores the archive before `ninja`. PR builds use their base branch's archive. Pushes and internal PRs then upload it again, so `-g` archives come from `MasterCI` and `-g0` ones from PR builds; community PRs read anonymously and write nothing, like the shared compiler cache. The cache-warmup builds only compile, so they skip it. The upload keeps exactly the entries the build used: restored entries get an access time of zero, so under `relatime` the next read sets it, and the entries still at zero are dropped before compressing with `zstd --long`. That keeps an archive at one build's working set, about 15 GB raw for `-g`, and lld's own pruning is off. A restore that fails leaves an empty cache rather than a partly extracted one, and an archive over 20 GB (`-g`) or 5 GB (`-g0`) is not uploaded. This removes the praktika `--volume-if-exists` setting, the cache mounts and the per-target builder labels, and puts `amd_cfi` and the cache-warmup builds back on `ARM_LARGE`. Related: #2740 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`S3.copy_file_to_s3` uploaded the ThinLTO cache archive, then raised `KeyError: 'altinity-builds-cache'` while building the link it returns, which failed the release builds after a successful upload. https://github.com/Altinity/ClickHouse/actions/runs/41109373051/job/124345996580 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Significantly reduce release build time by optimizing the link phase.
Changelog category (leave one):
CI/CD Options
Exclude tests:
Regression jobs to run: