Improve CI by clairemerker · Pull Request #226 · MeteoSwiss/evalml

clairemerker · 2026-07-23T12:17:10Z

See Testing documentation and CI/CD setup.

Disentangle inference testing from core-functionality testing (reports, meteograms, animations). Core logic is checked fast on every push; expensive inference is replayed from frozen reference data where possible, with full end-to-end runs kept separate.

Tiers

0. Unit — GitHub Actions on every push; locally pytest.

1. longtest (per-PR; cscs-ci run longtest or ci/run-integration-coverage.sh longtest tests/integration)
Baseline or frozen inference fixtures only — no GPU/MLflow, so it stays quick.

test_scoremaps_small — ICON-CH2-CTRL, 2025-04-01, KENDA-CH1 truth, T_2M/TOT_PREC6, Switzerland
test_replay_dryrun — fixture-replay DAG wiring + missing/checksum guards
test_meteogram_small — Varda + ICON-CH1/2-CTRL, 2024-08-01, SwissMetNet, GVE/SAE (skipped: fixture stale after the TOT_PREC fix, needs re-capture with servicedepl MLflow access)
test_dashboard_small — ICON-CH1-CTRL + INCA (WIP, disabled)
test_mec_small — Varda forecaster, 2025-08-01, MEC (no ffv2) (needs sarus)

2. heavytest (weekly cron; cscs-ci run heavytest or .../heavytest tests/integration)
Full workflow incl. real inference (test_configs.py).

forecasters-ich1.yaml — forecaster eval, 2025-03-01→02, ICON-CH2-EPS, KENDA-CH1
varda-single-1.0.yaml — forecaster + downscaler, 4 seasons, INCA/ICON-CH2-CTRL/ICON-CH1-CTRL, SwissMetNet, scorecards

Mechanisms

Uses cscs-ci and https://github.com/MeteoSwiss/varda-cscs-ci-workflows
Tests can be triggered per comment in the PR or locally.
Fixture capture/replay (evalml capture-fixture, src/evalml/fixtures.py) with per-GRIB checksums in MANIFEST.yaml.
Golden-reference metric checks (re-bless with EVALML_UPDATE_REFERENCE=1).
Subprocess-aware coverage across the snakemake/SLURM chain.
uv env pinned to a managed Python.

Todo / missing tests

config/ lives alongside tests/integration/configs/ currently (duplicated): team to decide final location
mean & median baselines in longtest
longtest: test_animation_small.py (ICON-CH2-CTRL, 2 lead times)
heavytest: aifs-single.yaml, forecasters-ich1-oper.yaml, forecasters-ich1-oper-fixed.yaml, forecasters-ich1_mec_ffv2.yaml
consolidate CI docs into main README
re-capture meteogram fixture (servicedepl MLflow) and un-skip

lclanzi · 2026-07-23T12:28:25Z

cscs-ci run

jonasbhend · 2026-07-23T13:59:28Z

cscs-ci run

Co-authored-by: Mary McGlohon <Mary.McGlohon@meteoswiss.ch>

Adds an integration test that runs the full evalml experiment pipeline and asserts that key verification metrics match reference values from a known-good run. --------- Co-authored-by: Luca Lanzilao <llanzila@balfrin-ln004.cscs.ch> Co-authored-by: Luca Lanzilao <llanzila@balfrin-ln003.cscs.ch>

This PR implements minor changes to the existing meteogram integration test. ### Summary of changes * rename test and config to meteogram_small * change date to 2024-08-01 (official start of ICON archive) * additional parameters to cover all use cases (derived, aggregated and instantaneous)

lclanzi · 2026-07-24T14:57:27Z

cscs-ci run

lclanzi · 2026-07-27T07:27:23Z

cscs-ci run

lclanzi · 2026-07-27T09:15:41Z

cscs-ci run

lclanzi · 2026-07-27T09:19:17Z

cscs-ci run

Under the service account, uv needs to be installed for every CI run since there is no global installation. --------- Co-authored-by: Mary McGlohon <Mary.McGlohon@meteoswiss.ch> Co-authored-by: clairemerker <34312518+clairemerker@users.noreply.github.com>

Two different pipelines: - longtest: intergration tests - heavytest: heavy integration tests (full end to end) The idea is to trigger heavytest weekly or nighly using the CI cron, and longtest in PRs using the comment call.

Fix truth in config test move configs to test folder --------- Co-authored-by: Luca Lanzilao <llanzila@balfrin-ln004.cscs.ch>

lclanzi · 2026-07-27T13:10:53Z

cscs-ci run heavytest

… testing-hackathon

lclanzi · 2026-07-27T13:29:40Z

cscs-ci run heavytest

lclanzi · 2026-07-27T14:36:27Z

cscs-ci run heavytest

marymcglo · 2026-09-21T17:28:42Z

cscs-ci run heavytest

marymcglo · 2026-10-13T15:07:40Z

cscs-ci run longtest

Resolve conflict in workflow/rules/inference.smk by keeping the unset ECCODES_ECKIT_GEO fix from main.

- Restore lead_time to 120h in sgm-multidataset ich1-oper config - Pin uv version to 0.11.32 in CI install - Add 'cancel' to summarize_logs error keywords

- Fix snakefmt indentation of account_flag params block in inference.smk - Apply ruff formatting to summarize_logs error keyword set

Apply trailing-whitespace, end-of-file, and ruff-format fixes to existing files flagged by pre-commit run --all-files.

clairemerker · 2026-10-21T20:32:46Z

cscs-ci run longtest

clairemerker · 2026-10-21T20:32:58Z

cscs-ci run heavytest

CI runners resolve a system Python 3.12 built without the sqlite3 extension, which crashes pytest-cov (coverage imports sqlite3 at startup). Pin python-preference = only-managed so uv always uses a python-build-standalone interpreter, which bundles sqlite3.

clairemerker · 2026-10-21T22:27:09Z

cscs-ci run longtest

The spatial-interpolation tolerance fix (#271) merged from main slightly changes the matched-cell set: N_total drops 2295960 -> 2284728 (-0.49%) and stat means shift ~1-2%, while min/max extremes are unchanged and all field invariants (RMSE>=MAE, RMSE**2==STDE**2+BIAS**2) and the not-grey PNG checks still pass. Recorded with EVALML_UPDATE_REFERENCE=1 on linard.

The TOT_PREC units fix merged from main changed the forecaster config, so its run_id hash moved (864a->f361) and the frozen fixture no longer matches. Re-capturing needs a real inference run with servicedepl mlflow access; skip the test until then so longtest is green.

clairemerker · 2026-10-22T21:11:31Z

cscs-ci run longtest

The file is written by the evalml CLI before launching snakemake, but is absent when the workflow is invoked directly (e.g. snakemake -n in the replay dry-run tests), which crashed DAG building at parse time. It only feeds the startup banner, so fall back to an empty string when missing.

clairemerker · 2026-10-23T06:44:19Z

cscs-ci run longtest

The script is CI-only; co-locate it with the other CI assets and update the longtest/heavytest job references. Removes the now-empty tools/ dir.

Co-locate run-integration-coverage.sh and coverage-integration.cfg with the test suite they drive; update the longtest/heavytest job invocations, the CI README, and the scripts' internal config path.

Replace the local runner and setup fragments (.baremetal-runner-balfrin, .sh_preamble, .install_uv, .install_evalml) with the reusable blocks from varda-cscs-ci-workflows, included remotely. cscs.yml now defines a single .evalml-job base (shared runner config, .mch-* setup before_script, and the log-summary artifact); longtest/heavytest only set their marker and Slurm time limit. Cluster selection is delegated to the shared cluster.yml (currently linard).

clairemerker · 2026-10-23T09:57:27Z

cscs-ci run longtest

The PR moved the production configs into tests/integration/configs/, which breaks existing workflows that reference config/. Restore config/ from main so those keep working; the test suite continues to use its own copies under tests/integration/configs/. Intentionally duplicated until the team decides on a single location.

Move the contents of tests/integration/README.md (inference fixtures, MEC caveats, fixture versioning, regenerating expected metrics) into the Testing documentation section of the top-level README, and repoint the test/config references at it. Removes the separate tests/integration/README.md.

Move the CI/CD admin/setup and trigger docs into a CI/CD setup subsection of the main README (images repathed to ci/images/), and remove ci/README.md so all documentation lives in one place.

minor change

9ede394

enable longtests @cscs

87b3b3b

clairemerker and others added 6 commits July 24, 2026 09:12

Merge branch 'main' into testing-hackathon

b0a21e0

increase time limit for integration tests

a5893b3

fix lint error (#230)

4bfd1bc

Co-authored-by: Mary McGlohon <Mary.McGlohon@meteoswiss.ch>

Merge branch 'main' into testing-hackathon

5f26b34

clairemerker added 3 commits July 24, 2026 17:05

linting

9ca7883

exclude longtest and heavytest from default pytest run

b6b0c75

fix syntax

e29d474

marymcglo and others added 5 commits July 27, 2026 13:42

edit config test and move configs to test folder

221160f

split CI into two pipelines (#237)

f1092ef

Two different pipelines: - longtest: intergration tests - heavytest: heavy integration tests (full end to end) The idea is to trigger heavytest weekly or nighly using the CI cron, and longtest in PRs using the comment call.

reduce verbose

0f444a8

edit config test and move configs to test folder (#238)

c1131c3

Fix truth in config test move configs to test folder --------- Co-authored-by: Luca Lanzilao <llanzila@balfrin-ln004.cscs.ch>

Merge branch 'testing-hackathon' of github.com:MeteoSwiss/evalml into…

98b625b

… testing-hackathon

Luca Lanzilao added 4 commits July 27, 2026 16:17

allow more characters in stderr

dd61117

Merge branch 'main' into testing-hackathon

80ad071

fix linting

4a37e40

fix unit test

cf9b967

Mary McGlohon added 3 commits October 13, 2026 16:55

update test runner to use linard

f4c1ca1

Disable MEC test when sarus not available (e.g. on linard)

8e838d2

Merge remote-tracking branch 'origin/main' into testing-hackathon

3a1dc77

Mary McGlohon and others added 5 commits October 14, 2026 15:21

disabling dashboard test (which is probably not yet complete)

0383f50

Merge branch 'main' into testing-hackathon

e59d3df

Resolve conflict in workflow/rules/inference.smk by keeping the unset ECCODES_ECKIT_GEO fix from main.

Address review comments on CI PR

16595d6

- Restore lead_time to 120h in sgm-multidataset ich1-oper config - Pin uv version to 0.11.32 in CI install - Add 'cancel' to summarize_logs error keywords

Fix linting

cbf0961

- Fix snakefmt indentation of account_flag params block in inference.smk - Apply ruff formatting to summarize_logs error keyword set

Fix pre-commit failures across repo

cb55c37

Apply trailing-whitespace, end-of-file, and ruff-format fixes to existing files flagged by pre-commit run --all-files.

clairemerker added 2 commits October 22, 2026 08:56

clairemerker added 7 commits October 23, 2026 08:58

Apply suggestion from @clairemerker

d04e887

Apply suggestion from @clairemerker

028b510

Apply suggestion from @clairemerker

4d2e9f2

Move summarize_logs.py into ci/ and drop tools/

47fdaff

The script is CI-only; co-locate it with the other CI assets and update the longtest/heavytest job references. Removes the now-empty tools/ dir.

Move integration coverage runner and config into tests/

3edcbdc

Co-locate run-integration-coverage.sh and coverage-integration.cfg with the test suite they drive; update the longtest/heavytest job invocations, the CI README, and the scripts' internal config path.

remove reference to linard

aea0f8f

clairemerker added 3 commits October 23, 2026 12:29

Fold ci/README.md into the main README

af1fa2a

Move the CI/CD admin/setup and trigger docs into a CI/CD setup subsection of the main README (images repathed to ci/images/), and remove ci/README.md so all documentation lives in one place.

Conversation

clairemerker commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tiers

Mechanisms

Todo / missing tests

Uh oh!

lclanzi commented Jul 23, 2026

Uh oh!

jonasbhend commented Jul 23, 2026

Uh oh!

lclanzi commented Jul 24, 2026

Uh oh!

lclanzi commented Jul 27, 2026

Uh oh!

lclanzi commented Jul 27, 2026

Uh oh!

lclanzi commented Jul 27, 2026

Uh oh!

lclanzi commented Jul 27, 2026

Uh oh!

lclanzi commented Jul 27, 2026

Uh oh!

lclanzi commented Jul 27, 2026

Uh oh!

marymcglo commented Sep 21, 2026

Uh oh!

marymcglo commented Oct 13, 2026

Uh oh!

clairemerker commented Oct 21, 2026

Uh oh!

clairemerker commented Oct 21, 2026

Uh oh!

clairemerker commented Oct 21, 2026

Uh oh!

clairemerker commented Oct 22, 2026

Uh oh!

clairemerker commented Oct 23, 2026

Uh oh!

clairemerker commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

clairemerker commented Jul 23, 2026 •

edited

Loading