Skip to content

Faster CI: batched test compilation, sanitizer builds in their own jobs - #238

Merged
petlenz merged 1 commit into
mainfrom
build/batched-tests
Oct 30, 2026
Merged

petlenz merged 1 commit into
mainfrom
build/batched-tests

Conversation

@petlenz

@petlenz petlenz commented Oct 29, 2026

Copy link
Copy Markdown
Member

CI took 45–55 minutes per PR, almost all of it compiling: the tests themselves run in about 30 s. This keeps ASan+UBSan on every PR for both compilers, and makes it cheaper.

Where the time went

  • numsim_add_test builds each of the ~78 test files twice: plain, and an ASan+UBSan copy. The sanitizer copy compiles about 1.9× slower.
  • For one test file, the optimisation level makes no difference: -O0, -O1 and -O3 each take about 29 s under ASan.
  • A precompiled header saves about 1.5 s of those 29 s.
  • -ftime-report: 66 % of the time is code generation for instantiated templates (callgraph functions expansion alone is 48 %). Every test file instantiates the same engine, tmech and gtest templates again.
  • Compiling 4 test files as one unit took 56 s, against 117 s for the 4 separately.

Change

  1. Batched tests. add_numsim_test collects the single-file tests. Six at a time go into one generated translation unit (tests/batch_tests.cmake):

    • each file sits in its own namespace;
    • every #include of the batch is hoisted above the namespaces, so the inner ones are no-ops;
    • no test file changes, and the add_numsim_test(...) lines stay as they are, so open PRs that add tests merge without conflicts.

    Built standalone, as before:

    • tests with several sources (test_read_check_layout);
    • tests that need their own flags or check include hygiene (test_material_conformance, test_host_neutral_includes);
    • tests that define C entry points (NUMSIM_MATERIALS_DEFINE_* or extern "C"), whose symbols ignore namespaces. This is detected from the source.

    -DNUMSIM_MATERIALS_BATCH_TESTS=OFF gives one executable per test again, for TDD on one file. NUMSIM_MATERIALS_TEST_BATCH_SIZE sets the batch size (default 6).

  2. CI.

    • The matrix is {g++-14, clang++-19} × {plain, asan}: four parallel jobs.
    • The ASan jobs build only the sanitizer executables (numsim_materials_asan_tests) and run ctest -R _asan$.
    • The plain jobs build with NUMSIM_SANITIZER_TESTS=OFF and also run the report tests.
    • ctest runs with -j$(nproc).

Checks

  • Local gate, g++-14 -O1, sanitizers on, -j4: 1486 tests, 0 warnings, the same count as main. Wall time 25 min → 11 min.
  • No two test files share a full test name, and no fixture suite spans two files, so the batched executables register the same tests as before.
  • Peak compiler memory: the heaviest ASan batch (with test_mean_field_local) takes 3.6 GB. The runners have 16 GB for 4 parallel compiles. If CI shows memory pressure, the batch size goes down.

Merge order

Independent of the epic stack. #228 also edits ci.yml: whichever merges second needs a small conflict resolution, which I will do.

Single-file tests are compiled six to a translation unit, each file in its
own namespace, so shared templates are instantiated once per batch instead
of once per file. Tests with C entry points or their own flags stay
standalone. NUMSIM_MATERIALS_BATCH_TESTS=OFF restores one executable per
test.

CI runs plain and ASan+UBSan builds of each compiler as separate jobs.
@petlenz

petlenz commented Oct 29, 2026

Copy link
Copy Markdown
Member Author

First CI run:

Job Time Tests
clang++-19 5 min 751
g++-14 8 min 751
clang++-19, asan 13 min 735
g++-14, asan 16 min 735

Wall clock 16 min, down from 45–55 min. 751 + 735 = 1486: the same tests as before. The plain jobs also run the report tests. No memory problems with batch size 6.

@petlenz
petlenz merged commit bccf12a into main Oct 30, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant