Skip to content

ci: test oga-hipep models per family, and all nightly on an STX Halo - #1063

Open
andrew180917 wants to merge 1 commit into
mainfrom
ci/hipep-all-models
Open

andrew180917 wants to merge 1 commit into
mainfrom
ci/hipep-all-models

Conversation

@andrew180917

Copy link
Copy Markdown
Collaborator

Why

Test Windows HIPEP tested only qwen3.5-oga-hipep:9b. The other 13 oga-hipep models (4 to 124 GB) ran in no CI job, and testing them on the shared NPU runners would hold those runners for hours. xconucstrhalo86, a new STX Halo, is for this job alone.

What changed

  • Machines: Test Windows HIPEP runs one job per kind of machine in the new .github/hipep-machines.json, which .github/script/ci/hipep_machines.py (with tests) turns into the matrix. Checks: Test Windows HIPEP (stx-halo) and Test Windows HIPEP (npu-runners); reports: windows-hipep-<id>.
    • stx-halo (xconucstrhalo86, labels oga-hipep and stx-halo): no NPU job asks for these labels, so it runs beside the NPU tests. Nightly it tests all 14 oga-hipep models (about 410 GB). On a PR it tests the pr tier's models and those whose catalog entries the PR added or edited, with 360 instead of 120 minutes when there are any.
    • npu-runners (the shared NPU runners that also carry oga-hipep): the pr tier's models only, nightly too, up to 32 GB. It holds the runner as an NPU test job does.
    • Repository variable FLM_HIPEP_MACHINES runs other kinds, or none, without a commit.
  • Models (npu_test/plan.py, PR_GPU): the install and pr tiers test one model per family. Always qwen3.8-oga-hipep:27b and gemma4-it-oga-hipep:26b-a4b (about 33 GB). qwen2.5-it-oga-hipep:14b, gemma3-oga-hipep:4b, gemma4-12b-oga-hipep:12b and gpt-oss-oga-hipep:120b only when the change touches their family.
  • Change detection (changes.py): a frontend change triggers HIPEP for every family with an oga-hipep entry, read from the catalog (was qwen3.5 only). The new output hipep-models lists the GPU entries the change added or edited.
  • Model cache (npu_test/model_cache.py): after a pull verifies a model, a stamp in $FLM_MODEL_PATH/npu-test-verified keeps its catalog pins and each file's size and mtime. Later runs skip the pull, and its re-hash of every file, while both are unchanged.
  • run.py gets --changed-models and --max-model-gb (a larger model is a setup/size SKIP), passed through new npu-test action inputs. "Show the GPU" also prints system memory and dedicated GPU memory.
  • NPU test jobs and their model picks are unchanged.

Admin setup to enable xconucstrhalo86

  1. Before this PR's CI runs, set the repository variable FLM_HIPEP_MACHINES (Settings > Secrets and variables > Actions > Variables) to the kinds that already have a runner:

    • npu-runners until xconucstrhalo86 is online;
    • stx-halo until a shared runner carries oga-hipep;
    • none if neither has one.

    A HIPEP job that no runner takes waits a day before it fails, and holds up the run's verdict. If it is already queued, cancel and re-run after setting the variable.

  2. Shared NPU runners: add the label oga-hipep (Settings > Actions > Runners) only to runners that meet all of these:

    • the GPU can use 32 GB or more (dedicated GPU memory plus half of system memory, e.g. a 64 GB STX or Krackan, or an STX Halo);
    • a current NPU driver;
    • room in FLM_MODEL_PATH for the pr tier's models.

    Never add stx-halo to a shared runner.

  3. xconucstrhalo86:

    • Machine: Windows 11; NPU driver 32.0.203.311 or later (flm.exe loads XRT from it); an AMD GPU driver that hip-ep supports (validated: 32.0.31041.1004); Python 3.9+ on PATH; 96 GB dedicated GPU memory in the BIOS (the 124 GB model peaked at 78 GiB locally). Turn off driver updates through Windows Update, and run nothing else on the GPU.
    • Runner: register one runner as a service, set up like the NPU runners, for this repository (or in the org runner group the NPU runners use). For example: config.cmd --url https://github.com/ROCm/FastFlowLM --token <token> --name xconucstrhalo86 --labels oga-hipep,stx-halo --runasservice. self-hosted, Windows and X64 are added by default. Do not add npu or sweep, or NPU jobs would take it.
    • In the runner folder's .env, set FLM_MODEL_PATH to an NVMe folder with 500 GB free that the service account can write, then restart the service. Without it the job uses a machine-wide FLM_MODEL_PATH left by an flm install, or %USERPROFILE%\.flm.
    • Network: access to huggingface.co (models) and github.com releases (hip-ep, about 137 MiB, downloaded on first use).
    • Optional: pre-pull the 14 models with the same FLM_MODEL_PATH (flm pull <model>). A CI pull gets 90 minutes per model.
  4. Once both kinds have runners, delete FLM_HIPEP_MACHINES. Set it to npu-runners whenever xconucstrhalo86 is down.

Details: .github/script/npu_test/README.md, section "HIPEP machines".

Tests run

  • test_hipep_machines.py: 13/13 pass.
  • test_changes.py: 24/25 pass. The failure, test_table_matches_catalog (r2t2 missing from the family table), fails on main too.
  • Model picks on the current catalog:
    • pr tier: 2 models (33.4 GB); each touched family adds its model;
    • full tier: 14 models (409 GB);
    • skipped on npu-runners (over 32 GB): gpt-oss 120b, qwen3.5 122b-a10b, qwen3.8-flash-next 125b-a6b;
    • NPU picks are identical to main on the install, pr and full tiers.
  • changes.py:
    • editing modeling_gemma4_12b.cpp runs HIPEP with family gemma4-12b;
    • editing oga_engine.cpp runs HIPEP on the two pr-tier models;
    • editing modeling_llama3.cpp runs no HIPEP.
  • Local STX Halo (Radeon 8060S, 96 GB dedicated):
    • model_cache.py finds every file in the real model folders and stamps a model after its pull;
    • "Show the GPU" prints 32 GB of system memory and 96 GB dedicated.
  • actionlint: clean. zizmor: no new findings, 0 high.

Known issues

  • gemma4-12b-oga-hipep:12b (feat(oga): serve gemma4-12b on oga-hipep with shared KV and images #1052): flm serve aborts (0xC0000409) at the first prefill with hip-ep 0.5.3. ORT reports that the preallocated present.0.key is {1,8,32768,256} while the run computes {1,8,32789,256} (local run, main's CI package from fc13a07). It is tested on PRs only when touched; expect it to fail nightly until fixed.
  • gemma4-it-oga-hipep:12b: its tool-call round trip fails on OGA 0.19 (local runs), so 26b-a4b stands for Gemma 4 on PRs. Expect it to fail nightly.

Unverified

  • First run on GitHub: xconucstrhalo86 is not registered yet, and no shared runner carries oga-hipep.

Test Windows HIPEP runs one job per kind of machine in
.github/hipep-machines.json (FLM_HIPEP_MACHINES overrides it):
stx-halo (xconucstrhalo86, labels oga-hipep and stx-halo, beside the NPU
jobs) tests all 14 models nightly; npu-runners (the shared NPU runners
also labelled oga-hipep) tests the pr tier's models up to 32 GB.

The pr tier tests qwen3.8 27b and gemma4 26b-a4b instead of qwen3.5 9b,
plus qwen2.5 14b, gemma3 4b, gemma4-12b and gpt-oss 120b when the
change touches their family, and on stx-halo, with 360 minutes, the
models whose catalog entries the change added or edited (changes.py now
reads the GPU families and entries from the catalog).
npu_test/model_cache.py skips pulling a model verified on the runner
before whose pins and files are unchanged.
@github-actions

Copy link
Copy Markdown
Security scan found no critical or high findings on lines changed by this pull request

9 medium, low, or unrated finding(s) are not listed; the scan run has them.

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

This branch was successfully deployed

1 active deployment
npu-approval — 1fd7deca Deployed Nov 17, 2026 by andrew180917 via npu-gate #449
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant