Skip to content

Windows CUDA b10826: llama-server hangs at load_model on PQ2_0 (AVX-512 repack path; --no-repack loads in 22s, Zen 4 / CUDA 12.4) #412

Open

Description

@ahmadgamal15-art

Build: release prism-b10826-c9c34bc, asset llama-prism-b10826-c9c34bc-bin-win-cuda-12.4-x64.zip (+cudart zip). Zip sha256 independently recomputed = publisher digest (d96396b2…94fe), 72/72 files byte-identical on extraction. --version runs clean, so this is not the #241 silent-exit class.

Host: Windows 11, AMD Ryzen 7 7800X3D (Zen 4, AVX-512 present via VEX encoding), RTX 4070 Ti SUPER 16 GB, CUDA 12.4 runtime per the release asset.

Symptom: llama-server.exe -m Ternary-Bonsai-2-27B-PQ2_0.gguf … starts, binds its port, then freezes permanently at srv load_model: loading model. Process CPU time frozen (~4.6 s and not growing), GPU 0 %, no WER crash event, never reaches loader metadata logging. Killed after 5 min twice (console and file-redirect variants — rules out the blocked-console-writer class).

Decisive A/B (same binary, same model, single flag flipped):

run command result
default llama-server.exe -m …PQ2_0.gguf --port 8899 hangs at load_model: loading model (forever; 5+ min, twice)
repack off llama-server.exe -m …PQ2_0.gguf --no-repack -ngl 0 --port 8899 srv llama_server: model loaded + listening on :8899 at t=22.3 s

So the hang is in the PQ2 CPU weight-repack path at load time. b10826 ships the arch-split ggml-cpu-* AVX-512 variants in the CUDA zip (fix #271 present); on this Zen 4 host the AVX-512 variant is runtime-selected and the repack deadlocks/hangs — consistent with the documented AVX-512 CPU-backend defect family (#180, #204, #219, and Bonsai-demo #182 where sidestepping the AVX-512 CPU DLLs was the proven workaround). Our working prism-b10735 fork build ships a single AVX2-only ggml-cpu.dll and never enters that code, which is the load-bearing baseline-vs-candidate difference.

Workaround confirmed on this host: --no-repack (flag verified present in b10826's llama-common.dll) — model loads and serves. Note GGML_CPU_DISABLE_AVX512 is not honored in this fork, and the #182 workaround (rename/sideline the 4 AVX-512 CPU DLLs) should also work but we have not needed it after --no-repack.

Asked: either make the PQ2 repack safe in the AVX-512 variant (or fall back to AVX2 for repack when the variant is Zen-class), or document --no-repack (or a CPU-feature guard) in the b10826 release notes for Zen 4 Windows users.

Environment details on request; happy to re-run any probe with a debug-instrumented build.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions