[FridrichMethod]

Summary

Sherlock flagged jobs whose Boltz workers started far more compute threads than the job's CPUs (one report: 52 threads on a small CPU request). This adds an opt-in, per-task numerical thread default to shk submit:

• --cpu-thread-budget [N] (N defaults to 1 and must be 1 through --cpus) or the project key cpu_thread_budget; an explicit CLI value overrides the configured one.
• The frozen argv becomes env NAME=VALUE ... sbatch ..., so the values are set on the sbatch process and reach the job through the existing --export=ALL. The job script stays byte-identical; the remote submit.sh and preflight.sh already run "$@", so no remote program changes.
• Variables: OMP_NUM_THREADS, OMP_THREAD_LIMIT, MKL_NUM_THREADS, OPENBLAS_NUM_THREADS, BLIS_NUM_THREADS, VECLIB_MAXIMUM_THREADS, NUMEXPR_NUM_THREADS, NUMEXPR_MAX_THREADS set to N, plus OMP_MAX_ACTIVE_LEVELS=1, OMP_DYNAMIC=FALSE, MKL_DYNAMIC=FALSE, each also as APPTAINERENV_ and SINGULARITYENV_ for clean-environment containers. Apptainer only warns when the two prefixed forms differ, so identical values stay silent.
• The staged spec records cpu_thread_policy (mode, budget per task, environment, enforcement scope).
• Without either setting, argv and spec are exactly as before.

This is a launch-environment default, not an OS-thread limit or a cgroup guarantee; docs/cpu-threads.md states the limits (PyTorch inter-op threads, DataLoader workers and runtime setters remain the application's job).

Commits:

• feat: add opt-in numerical CPU thread budgets for submissions (the original change, rebased onto 78bb33b; it was already running from the dotfiles submodule checkout on lab-ubuntu).
• fix: name both sources in the CPU thread budget error: a configured budget above --cpus used to blame the flag the user did not pass.
• docs: list the CPU thread budget in the README: config table row (unknown keys are errors, so the key must be documented), submit synopsis and docs list.
• test: refuse a configured budget above --cpus: covers the config-only path the review found untested.

Review notes

• Nothing in src/ or bin/ assumes argv[0] == "sbatch"; argv is only forwarded (submit.sh, preflight.sh, control.py) or printed (dry-run). No code reads the staged spec JSON back.
• External controllers or SubmitLine verifiers that require argv[0] == "sbatch" must accept the env prefix before enabling the option; the doc says so.
• Known, accepted: --cpu is no longer an unambiguous abbreviation of --cpus (argparse allow_abbrev); there is no CLI switch to turn a configured budget off, only an explicit lower N.
• Rules files are unchanged, since they mention only released CLI. __version__ stays 0.5.0 and CHANGELOG is untouched: test_release_metadata.test_one_version pins the first CHANGELOG heading to __version__, so the entry belongs to the release commit that bumps the version.

Test plan

• pre-commit run --all-files
• SHK_REQUIRE_ALL=1 python3 -B tests/run_suite.py with SHK_BUILD_PYTHON on Python 3.14 and 3.11, in a temporary HOME. Only test_executor.RealOpenSshTests.test_inaccessible_control_path_is_local_denial fails, and it fails the same way on main on this workstation (environment-specific, OpenSSH 9.6p1).
• GitHub CI (lint, tests on 3.11 and 3.14)

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.