Repository navigation
ci: test oga-hipep models per family, and all nightly on an STX Halo - #1063
Open
andrew180917 wants to merge 1 commit into
Open
andrew180917 wants to merge 1 commit into
andrew180917 wants to merge 1 commit into
Conversation
Test Windows HIPEP runs one job per kind of machine in .github/hipep-machines.json (FLM_HIPEP_MACHINES overrides it): stx-halo (xconucstrhalo86, labels oga-hipep and stx-halo, beside the NPU jobs) tests all 14 models nightly; npu-runners (the shared NPU runners also labelled oga-hipep) tests the pr tier's models up to 32 GB. The pr tier tests qwen3.8 27b and gemma4 26b-a4b instead of qwen3.5 9b, plus qwen2.5 14b, gemma3 4b, gemma4-12b and gpt-oss 120b when the change touches their family, and on stx-halo, with 360 minutes, the models whose catalog entries the change added or edited (changes.py now reads the GPU families and entries from the catalog). npu_test/model_cache.py skips pulling a model verified on the runner before whose pins and files are unchanged.
Security scan found no critical or high findings on lines changed by this pull request9 medium, low, or unrated finding(s) are not listed; the scan run has them. View the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Test Windows HIPEP tested only
qwen3.5-oga-hipep:9b. The other 13 oga-hipep models (4 to 124 GB) ran in no CI job, and testing them on the shared NPU runners would hold those runners for hours. xconucstrhalo86, a new STX Halo, is for this job alone.What changed
.github/hipep-machines.json, which.github/script/ci/hipep_machines.py(with tests) turns into the matrix. Checks:Test Windows HIPEP (stx-halo)andTest Windows HIPEP (npu-runners); reports:windows-hipep-<id>.stx-halo(xconucstrhalo86, labelsoga-hipepandstx-halo): no NPU job asks for these labels, so it runs beside the NPU tests. Nightly it tests all 14 oga-hipep models (about 410 GB). On a PR it tests the pr tier's models and those whose catalog entries the PR added or edited, with 360 instead of 120 minutes when there are any.npu-runners(the shared NPU runners that also carryoga-hipep): the pr tier's models only, nightly too, up to 32 GB. It holds the runner as an NPU test job does.FLM_HIPEP_MACHINESruns other kinds, ornone, without a commit.npu_test/plan.py,PR_GPU): the install and pr tiers test one model per family. Alwaysqwen3.8-oga-hipep:27bandgemma4-it-oga-hipep:26b-a4b(about 33 GB).qwen2.5-it-oga-hipep:14b,gemma3-oga-hipep:4b,gemma4-12b-oga-hipep:12bandgpt-oss-oga-hipep:120bonly when the change touches their family.changes.py): a frontend change triggers HIPEP for every family with an oga-hipep entry, read from the catalog (was qwen3.5 only). The new outputhipep-modelslists the GPU entries the change added or edited.npu_test/model_cache.py): after a pull verifies a model, a stamp in$FLM_MODEL_PATH/npu-test-verifiedkeeps its catalog pins and each file's size and mtime. Later runs skip the pull, and its re-hash of every file, while both are unchanged.run.pygets--changed-modelsand--max-model-gb(a larger model is asetup/sizeSKIP), passed through newnpu-testaction inputs. "Show the GPU" also prints system memory and dedicated GPU memory.Admin setup to enable xconucstrhalo86
Before this PR's CI runs, set the repository variable
FLM_HIPEP_MACHINES(Settings > Secrets and variables > Actions > Variables) to the kinds that already have a runner:npu-runnersuntil xconucstrhalo86 is online;stx-halountil a shared runner carriesoga-hipep;noneif neither has one.A HIPEP job that no runner takes waits a day before it fails, and holds up the run's verdict. If it is already queued, cancel and re-run after setting the variable.
Shared NPU runners: add the label
oga-hipep(Settings > Actions > Runners) only to runners that meet all of these:FLM_MODEL_PATHfor the pr tier's models.Never add
stx-haloto a shared runner.xconucstrhalo86:
PATH; 96 GB dedicated GPU memory in the BIOS (the 124 GB model peaked at 78 GiB locally). Turn off driver updates through Windows Update, and run nothing else on the GPU.config.cmd --url https://github.com/ROCm/FastFlowLM --token <token> --name xconucstrhalo86 --labels oga-hipep,stx-halo --runasservice.self-hosted,WindowsandX64are added by default. Do not addnpuorsweep, or NPU jobs would take it..env, setFLM_MODEL_PATHto an NVMe folder with 500 GB free that the service account can write, then restart the service. Without it the job uses a machine-wideFLM_MODEL_PATHleft by an flm install, or%USERPROFILE%\.flm.FLM_MODEL_PATH(flm pull <model>). A CI pull gets 90 minutes per model.Once both kinds have runners, delete
FLM_HIPEP_MACHINES. Set it tonpu-runnerswhenever xconucstrhalo86 is down.Details:
.github/script/npu_test/README.md, section "HIPEP machines".Tests run
test_hipep_machines.py: 13/13 pass.test_changes.py: 24/25 pass. The failure,test_table_matches_catalog(r2t2 missing from the family table), fails on main too.npu-runners(over 32 GB): gpt-oss 120b, qwen3.5 122b-a10b, qwen3.8-flash-next 125b-a6b;changes.py:modeling_gemma4_12b.cppruns HIPEP with family gemma4-12b;oga_engine.cppruns HIPEP on the two pr-tier models;modeling_llama3.cppruns no HIPEP.model_cache.pyfinds every file in the real model folders and stamps a model after its pull;Known issues
gemma4-12b-oga-hipep:12b(feat(oga): serve gemma4-12b on oga-hipep with shared KV and images #1052):flm serveaborts (0xC0000409) at the first prefill with hip-ep 0.5.3. ORT reports that the preallocatedpresent.0.keyis {1,8,32768,256} while the run computes {1,8,32789,256} (local run, main's CI package from fc13a07). It is tested on PRs only when touched; expect it to fail nightly until fixed.gemma4-it-oga-hipep:12b: its tool-call round trip fails on OGA 0.19 (local runs), so26b-a4bstands for Gemma 4 on PRs. Expect it to fail nightly.Unverified
oga-hipep.