From 2bd7697cfc2719915456eabd4143605cfe4d4fbc Mon Sep 17 00:00:00 2001 From: Chi Zhang Date: Sat, 14 Nov 2026 16:18:19 -0800 Subject: [PATCH 1/7] feat(core): ryzenai-corelib 0.70.0, get_npu replaces npu_usable and testing_npu_arch Moves the core backend to ryzenai-corelib 0.70.0 (corelib main at 39c30eab). What changed in corelib 0.61 to 0.70 that reaches FastFlowLM: - 0.70, NPU query: ryzenai_corelib_npu_usable and the testing hook ryzenai_corelib_testing_npu_arch are gone. ryzenai_corelib_get_npu (none / aiem / aie2p) and ryzenai_corelib_npu_name replace them, so corelib_api.hpp no longer includes corelib/testing.h. - The runtime's readiness check now asks for aiem. The old npu_usable was false on AIE2P, whose kernels corelib runs only behind an xclbin FastFlowLM never gives it; get_npu now answers aie2p there, so checking for "any NPU" would have offered the core backend on Strix. - NpuArch names the enum with npu_name, "" for none, as before, so dev04 / aie2p platform detection is unchanged. - 0.61 / 0.68, model structs: model_metadata gains max_draft_tokens and audio_token_id; session_options gains draft_tokens. FastFlowLM zero-initialises both at their full sizeof, so no draft is asked for. - 0.62 / 0.67, matmul: the 4-bit layout is mladf_v4 (was mladf) beside AIE2P's mladf_v2, and matmul_bf16_kernel_info gains format. FastFlowLM names neither; the fake corelib fills format. - 0.68: ssmlp_bf16_weights_desc gains ungated; unused here. - Model libraries and their names are unchanged. The weights loader's switch from bad_argument to unsupported for a wrong tensor type does not matter here: FastFlowLM does not branch on corelib's status codes. The fake corelib answers get_npu from one enum and npu_name as corelib spells it; the tests refuse none and aie2p, and read "" for none. On an AIE4 (Medusa) test machine, qwen3-core 0.6b/1.7b/4b/8b, phi4-mini-it-core:4b, qwen3.5-core 4b/9b and llama3.2-core:1b each load, pass the readiness check through get_npu, report no fallback and answer correctly, at decode rates within a few percent of corelib 0.59's. Co-authored-by: Cursor --- docs/core-setup-linux.md | 6 +++--- docs/docs/models/phi.md | 4 ++-- docs/docs/models/qwen.md | 2 +- src/common/core/corelib_runtime.cpp | 17 ++++++++++++----- src/common/models/README.md | 4 ++-- src/include/core/corelib_api.hpp | 12 ++++-------- src/test/phi4_core/fake_corelib.cpp | 24 +++++++++++++++--------- src/test/phi4_core/fake_corelib.hpp | 6 ++---- src/test/phi4_core/test_corelib_api.cpp | 20 ++++++++++++-------- 9 files changed, 53 insertions(+), 42 deletions(-) diff --git a/docs/core-setup-linux.md b/docs/core-setup-linux.md index 8d9b64263..cda2cbfe6 100644 --- a/docs/core-setup-linux.md +++ b/docs/core-setup-linux.md @@ -9,7 +9,7 @@ Written against the working install on this machine. Paths use `$P` for the dependency prefix (here `/home/alfxu/ddprefix`); substitute your own. > **Status:** the core backend has been run on hardware on Windows only. The -> steps below are updated for corelib 0.60.0, but they have not been run end +> steps below are updated for corelib 0.70.0, but they have not been run end > to end on Linux at this revision. The [Known issues](#known-issues) are from > an earlier Linux bring-up. @@ -52,14 +52,14 @@ loaded from, so those libraries must sit beside it; nothing else is needed at run time, no kernel store and no environment variable. **Version must match exactly.** `src/include/core/corelib_api.hpp` has a -`#error` for anything other than corelib **0.60.0**: +`#error` for anything other than corelib **0.70.0**: ```bash grep -E 'CORELIB_VERSION_(MAJOR|MINOR|PATCH)' \ $P/install-corelib/include/ryzenai/corelib/base.h ``` -Must print `0`, `60`, `0`. (`BUILD-NOTES.md` still says 0.4.0 in its opening +Must print `0`, `70`, `0`. (`BUILD-NOTES.md` still says 0.4.0 in its opening line — that text is stale.) ### Verify corelib before going further diff --git a/docs/docs/models/phi.md b/docs/docs/models/phi.md index 9fbba3dcf..7d59c8496 100644 --- a/docs/docs/models/phi.md +++ b/docs/docs/models/phi.md @@ -75,9 +75,9 @@ The `src/test/phi4_core` suite is still Windows-only and is not configured on Li ### Pointing FastFlowLM at the runtime -A core build (`-DFLM_ENABLE_CORE=ON`) **links corelib in**. corelib keeps its NPU device to itself and opens a hardware context per stream on it; FastFlowLM opens its own device for every other family, and a core model shares no buffer with those. The core backend is offered only where corelib reports a usable NPU (`ryzenai_corelib_npu_usable`): elsewhere corelib would run a session on the GPU or the host. Point the build at the library with `RYZENAI_CORELIB_INCLUDE_DIR` and `RYZENAI_CORELIB_LIBRARY`. `FLM_CORE_CORELIB_PATH` selects a shared library (`.dll` on Windows, `.so` elsewhere) only in the older dynamically loading configuration; in a statically linked core build it is ignored, and `flm` says so if it is set. +A core build (`-DFLM_ENABLE_CORE=ON`) **links corelib in**. corelib keeps its NPU device to itself and opens a hardware context per stream on it; FastFlowLM opens its own device for every other family, and a core model shares no buffer with those. The core backend is offered only where corelib reports an AIEM NPU (`ryzenai_corelib_get_npu`): elsewhere, AIE2P included, corelib would run a session on the GPU or the host. Point the build at the library with `RYZENAI_CORELIB_INCLUDE_DIR` and `RYZENAI_CORELIB_LIBRARY`. `FLM_CORE_CORELIB_PATH` selects a shared library (`.dll` on Windows, `.so` elsewhere) only in the older dynamically loading configuration; in a statically linked core build it is ignored, and `flm` says so if it is set. -The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.60.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build installs `ryzenai_corelib` plus one `ryzenai_corelib_model_` library per model, each carrying that model's kernels; corelib loads them only from its own directory, so they must sit beside it (a Windows core build copies them next to `flm.exe`). Each engine enables its model's kernel bundle on the stream it opens; no kernel store or environment variable is involved. Corelib's own dependency directory must be reachable on `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux). +The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.70.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build installs `ryzenai_corelib` plus one `ryzenai_corelib_model_` library per model, each carrying that model's kernels; corelib loads them only from its own directory, so they must sit beside it (a Windows core build copies them next to `flm.exe`). Each engine enables its model's kernel bundle on the stream it opens; no kernel store or environment variable is involved. Corelib's own dependency directory must be reachable on `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux). ```powershell flm pull phi4-mini-it-core:4b diff --git a/docs/docs/models/qwen.md b/docs/docs/models/qwen.md index 22fd08256..b23acf7c0 100644 --- a/docs/docs/models/qwen.md +++ b/docs/docs/models/qwen.md @@ -111,7 +111,7 @@ flm run qwen3:8b - **Backend:** `core` — one engine for all four sizes. The GGUF's own metadata selects the size and is cross-checked against it. - **Source format:** GGUF, read directly by corelib. With a Hugging Face token that can read AMD's private repositories (`HF_TOKEN`, or `hf auth login`), `flm pull` fetches AMD's 4-bit file, which packs exactly as it is; without one, a tag below on an AMD file cannot be pulled, as corelib's own runner has no public fallback either. Only Qwen3-0.6B and Llama, which AMD ships no 4-bit file of, pull a public `Q8_0`, requantized to 4-bit groups on every load, which is lossy; corelib refuses `Q8_0` for Qwen3.5, Qwen3.6, Gemma 4 and Phi-4-mini. - **Context:** each tag's limit is the most corelib runs that model at; the default is lower where corelib or our own runs found answers going wrong past it. Over-capacity requests are rejected with HTTP 400 before any work reaches the device. -- **Availability:** developer build against corelib 0.60.0 — see [Phi-4 on the core backend](phi.md). +- **Availability:** developer build against corelib 0.70.0 — see [Phi-4 on the core backend](phi.md). The same corelib path runs every `-core` tag: diff --git a/src/common/core/corelib_runtime.cpp b/src/common/core/corelib_runtime.cpp index ccf32c220..ea812beb9 100644 --- a/src/common/core/corelib_runtime.cpp +++ b/src/common/core/corelib_runtime.cpp @@ -44,9 +44,14 @@ std::shared_ptr CorelibRuntime::CreateReady( api->Check(api->functions().selftest_dependencies(), "ryzenai_corelib_selftest_dependencies"); // Without one corelib still runs a session, on the GPU or the host, which - // is the silent slowness the core backend is meant to rule out. - if (!api->functions().npu_usable()) { - throw std::runtime_error("corelib has no usable NPU"); + // is the silent slowness the core backend is meant to rule out. An AIE2P + // NPU runs corelib's kernels only behind an xclbin, which FastFlowLM never + // hands it, so only AIEM counts. + const ryzenai_corelib_npu npu = api->functions().get_npu(); + if (npu != ryzenai_corelib_npu_aiem) { + const char* name = api->functions().npu_name(npu); + throw std::runtime_error(std::string("corelib has no usable NPU (") + + (name ? name : "unknown") + ")"); } return std::shared_ptr(new CorelibRuntime(std::move(api))); } @@ -62,8 +67,10 @@ std::string CorelibRuntime::NpuArch(const std::filesystem::path& executable_dir) try { std::lock_guard lock(process_mutex); const auto api = process_runtime ? process_runtime->api_ : ProcessApi(executable_dir); - const char* arch = api->functions().npu_arch(); - return arch ? arch : ""; + const ryzenai_corelib_npu npu = api->functions().get_npu(); + if (npu == ryzenai_corelib_npu_none) return ""; + const char* name = api->functions().npu_name(npu); + return name ? name : ""; } catch (...) { return ""; } diff --git a/src/common/models/README.md b/src/common/models/README.md index 811f5a016..2658e0ee0 100644 --- a/src/common/models/README.md +++ b/src/common/models/README.md @@ -25,7 +25,7 @@ away. `get_device()` asks the machine, once per process. A build that links corelib asks corelib what NPU it has (`CorelibRuntime::NpuArch`, corelib's -`ryzenai_corelib_testing_npu_arch` until its public header names one): `AIEM` +`ryzenai_corelib_get_npu` named by `ryzenai_corelib_npu_name`): `AIEM` is `dev04`, and `AIE2P`, no NPU, or a corelib that cannot be loaded is `default_npu_platform()`, `aie2p`. corelib reads the device itself, so FastFlowLM holds no table of devices and names none. A build without corelib @@ -43,7 +43,7 @@ This is a contributor document. For *using* a backend once it exists — `--back |---|---| | **Platform** | Windows and Linux. Both configure and build; only Windows has been run on hardware, and the `src/test/phi4_core` suite is still Windows-only. | | **Hardware** | An dev04 NPU. There is no simulator; a wrong shape shows up as garbage output, not an error. | -| **corelib headers** | The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.60.0`** match on major, minor and patch. [`corelib_api.hpp`](../../include/core/corelib_api.hpp) `#error`s on any other version — deliberately, because the C ABI has changed shape between patch releases. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build gives one `ryzenai_corelib_model_` library per model, carrying its kernels, which must sit beside corelib. A per-op engine enables its model's kernel bundle on the stream it opens; a model corelib runs whole enables its own. | +| **corelib headers** | The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.70.0`** match on major, minor and patch. [`corelib_api.hpp`](../../include/core/corelib_api.hpp) `#error`s on any other version — deliberately, because the C ABI has changed shape between patch releases. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build gives one `ryzenai_corelib_model_` library per model, carrying its kernels, which must sit beside corelib. A per-op engine enables its model's kernel bundle on the stream it opens; a model corelib runs whole enables its own. | | **Weights** | A GGUF corelib's model reads: a Q4_G128 or Q4_G32 one, whose codes it packs as they are, or, for Qwen3 and Llama only, a Q8_0 one, which it requantizes on every load. The catalog pulls what corelib's own runner does: AMD's Q4 file, with no public fallback, for every model but Qwen3-0.6B and Llama, which AMD ships no Q4 of and which stay on a public Q8_0. See §6. | Configure with: diff --git a/src/include/core/corelib_api.hpp b/src/include/core/corelib_api.hpp index 6fadc26f9..c37e10fb7 100644 --- a/src/include/core/corelib_api.hpp +++ b/src/include/core/corelib_api.hpp @@ -2,15 +2,11 @@ #include -#if RYZENAI_CORELIB_VERSION_MAJOR != 0 || RYZENAI_CORELIB_VERSION_MINOR != 60 || \ +#if RYZENAI_CORELIB_VERSION_MAJOR != 0 || RYZENAI_CORELIB_VERSION_MINOR != 70 || \ RYZENAI_CORELIB_VERSION_PATCH != 0 -#error "FastFlowLM requires ryzenai-corelib headers exactly 0.60.0" +#error "FastFlowLM requires ryzenai-corelib headers exactly 0.70.0" #endif -// ryzenai_corelib_testing_npu_arch, until corelib's public header names the -// NPU's architecture. After the pin: a testing hook may change in any release. -#include - #include #include #include @@ -26,8 +22,8 @@ X(status_to_string, ryzenai_corelib_status_to_string) \ X(get_last_error_message, ryzenai_corelib_get_last_error_message) \ X(selftest_dependencies, ryzenai_corelib_selftest_dependencies) \ - X(npu_usable, ryzenai_corelib_npu_usable) \ - X(npu_arch, ryzenai_corelib_testing_npu_arch) \ + X(get_npu, ryzenai_corelib_get_npu) \ + X(npu_name, ryzenai_corelib_npu_name) \ X(object_release, ryzenai_corelib_object_release) \ X(create_stream, ryzenai_corelib_create_stream) \ X(stream_synchronize, ryzenai_corelib_stream_synchronize) \ diff --git a/src/test/phi4_core/fake_corelib.cpp b/src/test/phi4_core/fake_corelib.cpp index f920e0123..66670e3c1 100644 --- a/src/test/phi4_core/fake_corelib.cpp +++ b/src/test/phi4_core/fake_corelib.cpp @@ -174,10 +174,15 @@ struct TypedFake { return current_detail.c_str(); } else if constexpr (std::is_same_v) { return state.selftest_status; - } else if constexpr (std::is_same_v) { - return state.npu_usable; - } else if constexpr (std::is_same_v) { - return state.npu_arch.c_str(); + } else if constexpr (std::is_same_v) { + return state.npu; + } else if constexpr (std::is_same_v) { + switch (std::get<0>(arguments)) { + case ryzenai_corelib_npu_none: return "none"; + case ryzenai_corelib_npu_aiem: return "AIEM"; + case ryzenai_corelib_npu_aie2p: return "AIE2P"; + default: return static_cast(nullptr); + } } else if constexpr (std::is_same_v) { void* object = std::get<0>(arguments); if (object) { @@ -227,7 +232,7 @@ struct TypedFake { constexpr std::int64_t kPage = 4096; const auto wanted = std::max(state.model_max_sequence_length, 1); *out = {sizeof(*out), "fake", 16, state.model_vocab, - 4, (wanted + kPage - 1) / kPage * kPage, -1}; + 4, (wanted + kPage - 1) / kPage * kPage, -1, 0, -1}; } return Status(Tag::name); } else if constexpr (std::is_same_v) { @@ -442,7 +447,8 @@ struct TypedFake { for (const auto n : outputs) { for (const auto m : shipped(n == 1024 ? "matmul-1024" : "matmul-3072")) { const ryzenai_corelib_matmul_bf16_kernel_info info{ - sizeof(info), 0, m, 3072, n, 64, false, 0, 0}; + sizeof(info), 0, m, 3072, n, 64, false, 0, 0, + ryzenai_corelib_matmul_bf16_weights_format_mladf_v4}; callback(ctx, &info); } } @@ -460,7 +466,8 @@ struct TypedFake { if (callback) { if constexpr (std::is_same_v) { const ryzenai_corelib_matmul_bf16_kernel_info info{ - sizeof(info), 0, -1, 0, 0, 0, false, 0, 0}; + sizeof(info), 0, -1, 0, 0, 0, false, 0, 0, + ryzenai_corelib_matmul_bf16_weights_format_mladf_v4}; callback(ctx, &info); } else { const ryzenai_corelib_ssmlp_bf16_kernel_info info{ @@ -666,8 +673,7 @@ void Reset() { RYZENAI_CORELIB_VERSION_PATCH}; state.selftest_status = ryzenai_corelib_status_success; state.default_status = ryzenai_corelib_status_success; - state.npu_usable = true; - state.npu_arch = "AIEM"; + state.npu = ryzenai_corelib_npu_aiem; state.detail.clear(); state.status_text = "success"; state.missing_symbol.clear(); diff --git a/src/test/phi4_core/fake_corelib.hpp b/src/test/phi4_core/fake_corelib.hpp index 458510abe..6ffa7ead3 100644 --- a/src/test/phi4_core/fake_corelib.hpp +++ b/src/test/phi4_core/fake_corelib.hpp @@ -101,10 +101,8 @@ struct State { RYZENAI_CORELIB_VERSION_PATCH}; ryzenai_corelib_status selftest_status{ryzenai_corelib_status_success}; ryzenai_corelib_status default_status{ryzenai_corelib_status_success}; - /// \brief what npu_usable() answers - bool npu_usable{true}; - /// \brief what testing_npu_arch() answers - std::string npu_arch{"AIEM"}; + /// \brief what get_npu() answers; npu_name() names it as corelib does + ryzenai_corelib_npu npu{ryzenai_corelib_npu_aiem}; std::string detail; std::string status_text{"success"}; std::string missing_symbol; diff --git a/src/test/phi4_core/test_corelib_api.cpp b/src/test/phi4_core/test_corelib_api.cpp index 46dda6710..88c056fc7 100644 --- a/src/test/phi4_core/test_corelib_api.cpp +++ b/src/test/phi4_core/test_corelib_api.cpp @@ -222,12 +222,14 @@ void TestRuntimeRunsDependencySelftestAndRequiresAUsableNpu() { }), "selftest_dependencies"); - fake_corelib::Reset(); - fake_corelib::GetState().npu_usable = false; - RequireContains(RequireThrows([] { - CorelibRuntime::CreateForTest(ValidApi()); - }), - "no usable NPU"); + for (const auto npu : {ryzenai_corelib_npu_none, ryzenai_corelib_npu_aie2p}) { + fake_corelib::Reset(); + fake_corelib::GetState().npu = npu; + RequireContains(RequireThrows([] { + CorelibRuntime::CreateForTest(ValidApi()); + }), + "no usable NPU"); + } fake_corelib::Reset(); const auto runtime = CorelibRuntime::CreateForTest(ValidApi()); @@ -239,9 +241,11 @@ void TestNpuArchIsCorelibsAnswerAndEmptyWhenItCannotLoad() { fake_corelib::Reset(); { const auto runtime = CorelibRuntime::CreateForTest(ValidApi()); - fake_corelib::GetState().npu_arch = "AIE2P"; + fake_corelib::GetState().npu = ryzenai_corelib_npu_aie2p; TEST_REQUIRE(CorelibRuntime::NpuArch({}) == "AIE2P"); - fake_corelib::GetState().npu_arch = "AIEM"; + fake_corelib::GetState().npu = ryzenai_corelib_npu_none; + TEST_REQUIRE(CorelibRuntime::NpuArch({}).empty()); + fake_corelib::GetState().npu = ryzenai_corelib_npu_aiem; TEST_REQUIRE(CorelibRuntime::NpuArch({}) == "AIEM"); } CorelibRuntime::ShutdownProcess(); From 6c9926e80c72f458b89373175b08a44bc280667c Mon Sep 17 00:00:00 2001 From: Chi Zhang Date: Sat, 14 Nov 2026 16:18:19 -0800 Subject: [PATCH 2/7] fix(chat): a JSON Schema pattern matches anywhere in the string, and keeps a one-character class's length bounds JSON Schema finds a string pattern anywhere in the string; xgrammar matches the regex against the whole string. A field written as {"type": "string", "minLength": 1, "maxLength": 128, "pattern": "\\S"} ("not blank") therefore allowed exactly one character, and xgrammar also drops minLength/maxLength whenever a pattern is present (it logs "pattern combined with minLength/maxLength is not supported"). Found running an agent benchmark with qwen3.6-moe-core:35b-a3b: every free-text argument of a forced chart-generation tool call came back as one character ("caption":"7","alt_text":"A","labels":["P","C"]). The caller rejected it, the grammar left the model no other call to make, and it repeated until the tool-iteration limit. normalize_schema() now: - pads the side(s) a string pattern leaves unanchored with ".*" ("\S" becomes ".*(\S).*"). A pattern anchored at both ends is left as it is, so "^\d{4}-\d{2}-\d{2}$" keeps its meaning. - lets a lone character class that only has to occur somewhere ("\S", "\d", "[a-z]", ...) lead, and turns the schema's length bounds into the repetition after it: "\S" with maxLength 128 becomes "\S.{0,127}", so the bounds xgrammar would drop are enforced. - keeps minLength/maxLength and drops the pattern instead when that repetition would be longer than 128. xgrammar compiles a longer regex repetition into a rule of its own and cannot build that rule inside a nested object ("cannot build the decoding grammar: Check failed: per_rule_fsm_hashes[...].has_value()"), so the request would fail with HTTP 400; a string past its maxLength is the likelier mistake. response_format schemas now go through normalize_schema() as tool parameters already did, so they get the same reading. New live cases in src/test/openai_api_test: test_tool_calls function_unanchored_pattern and function_nested_long_bounded_pattern, and test_response_format schema_unanchored_pattern. The two unanchored cases fail on main ('P' for "Paris"); the nested one fails with the HTTP 400 above if the bounds are folded at any length. On an AIE4 (Medusa) test machine with qwen3.6-moe-core:35b-a3b: test_tool_calls.py 18/18, test_response_format.py 12/12. Co-authored-by: Cursor --- src/common/chat/chat_grammar.cpp | 98 ++++++++++++++++++- src/test/openai_api_test/README.md | 4 +- .../openai_api_test/test_response_format.py | 11 +++ src/test/openai_api_test/test_tool_calls.py | 44 +++++++++ 4 files changed, 154 insertions(+), 3 deletions(-) diff --git a/src/common/chat/chat_grammar.cpp b/src/common/chat/chat_grammar.cpp index 188908de0..738d8108e 100644 --- a/src/common/chat/chat_grammar.cpp +++ b/src/common/chat/chat_grammar.cpp @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -158,6 +159,93 @@ json qwen_xml_arguments(const json& schema) { // ---------------------------------------------------------------- tools of a request +/// \brief Whether a regex is one character class: ".", "\S", "\d", "[a-z]", ... +bool is_one_character(const std::string& pattern) { + if (pattern == ".") { + return true; + } + if (pattern.size() == 2 && pattern[0] == '\\') { + return pattern[1] != 'b' && pattern[1] != 'B'; // word boundaries match no character + } + if (pattern.size() < 3 || pattern.front() != '[' || pattern.back() != ']') { + return false; + } + for (size_t i = 1; i + 1 < pattern.size(); ++i) { + if (pattern[i] == '\\') { + ++i; + } else if (pattern[i] == ']') { + return false; + } + } + return true; +} + +/// \brief A JSON Schema string pattern as xgrammar's whole-string regex has to spell it +/// \note JSON Schema finds a pattern anywhere in the string, while xgrammar matches +/// the whole string against it, so "\S" (not blank) would allow exactly one +/// character. A side the pattern leaves unanchored gets ".*". +std::string whole_string_pattern(const std::string& pattern) { + const bool starts = !pattern.empty() && pattern.front() == '^'; + size_t escapes = 0; + while (escapes + 1 < pattern.size() && pattern[pattern.size() - 2 - escapes] == '\\') ++escapes; + const bool ends = pattern.size() > (starts ? 1u : 0u) && pattern.back() == '$' && escapes % 2 == 0; + if (starts && ends) { + return pattern; + } + const size_t begin = starts ? 1 : 0; + const std::string core = pattern.substr(begin, pattern.size() - begin - (ends ? 1 : 0)); + if (core.empty()) { + return ".*"; + } + return (starts ? "^(" : ".*(") + core + (ends ? ")$" : ").*"); +} + +bool is_string_schema(const json& schema) { + const json type = schema.value("type", json()); + if (type.is_array()) { + return std::find(type.begin(), type.end(), json("string")) != type.end(); + } + return type == "string"; +} + +/// \brief Longest regex repetition xgrammar builds in place; a longer one becomes a rule +/// of its own, which it cannot build inside a nested object (the check on +/// per_rule_fsm_hashes fails and the request gets no grammar at all). +constexpr int64_t XGRAMMAR_REGEX_REPEAT_LIMIT = 128; + +/// \brief The pattern xgrammar is given for a string schema, which may take over its length bounds +/// \note xgrammar drops minLength/maxLength once a pattern is there. A lone character +/// class that only has to occur somewhere leads instead of being padded, so the +/// bounds become the repetition after it and leave the schema. Bounds too long for +/// that repetition keep the schema's minLength/maxLength and drop the pattern +/// (std::nullopt): a string past its maxLength is the likelier mistake. +std::optional string_pattern(json& schema) { + const std::string pattern = schema["pattern"].get(); + const json min_length = schema.value("minLength", json(0)); + json max_length = schema.value("maxLength", json()); + const bool bounded = schema.contains("minLength") || schema.contains("maxLength"); + if (!bounded || !is_one_character(pattern) || !min_length.is_number_integer() || + !(max_length.is_null() || max_length.is_number_integer())) { + return whole_string_pattern(pattern); + } + if (max_length.is_number_integer() && max_length.get() > std::numeric_limits::max()) { + max_length = json(); + } + const int64_t low = std::max(min_length.get(), 1) - 1; + if (max_length.is_number_integer() && max_length.get() - 1 < low) { + return whole_string_pattern(pattern); // unsatisfiable, which xgrammar reports + } + const bool long_repeat = max_length.is_null() ? low > XGRAMMAR_REGEX_REPEAT_LIMIT + : max_length.get() - 1 > XGRAMMAR_REGEX_REPEAT_LIMIT; + if (long_repeat) { + return std::nullopt; + } + const std::string high = max_length.is_null() ? std::string() : std::to_string(max_length.get() - 1); + schema.erase("minLength"); + schema.erase("maxLength"); + return pattern + ".{" + std::to_string(low) + "," + high + "}"; +} + /// \brief Spell out what a schema leaves implicit where xgrammar needs it /// \note An array without "items" means any items; the qwen_xml style rejects /// every value of such an array unless that is written as "items": {}. @@ -172,6 +260,14 @@ json normalize_schema(json schema) { if (schema.value("type", json()) == "array" && !schema.contains("items") && !schema.contains("prefixItems")) { schema["items"] = json::object(); } + if (is_string_schema(schema) && schema.contains("pattern") && schema["pattern"].is_string()) { + const std::optional pattern = string_pattern(schema); + if (pattern) { + schema["pattern"] = *pattern; + } else { + schema.erase("pattern"); + } + } for (auto& [key, value] : schema.items()) { if (value.is_object() || value.is_array()) { value = normalize_schema(std::move(value)); @@ -445,7 +541,7 @@ json reasoning_prefix(const chat_format_spec& spec, bool allow) { json response_schema(const chat_constraint_request& request) { if (!request.json_schema.is_null()) { - return request.json_schema; + return normalize_schema(request.json_schema); } return json{{"type", "object"}}; } diff --git a/src/test/openai_api_test/README.md b/src/test/openai_api_test/README.md index 1ad630740..6d179eadd 100644 --- a/src/test/openai_api_test/README.md +++ b/src/test/openai_api_test/README.md @@ -34,8 +34,8 @@ Each case prints `PASS`, `FAIL` or `SKIP`. The exit code is non-zero if anything |---|---| | `test_response_shape.py` | Message and chunk shape, one delta kind per chunk, `finish_reason`, `usage` only with `stream_options.include_usage`, stream folds to the non-stream message | | `test_reasoning.py` | `think: true` → `reasoning_content`, `think: false` / `reasoning_effort: "none"` → none, reasoning streamed before content, switch is per request, reasoning in history is accepted | -| `test_tool_calls.py` | `auto` / `required` / forced function / `none`, `parallel_tool_calls: false`, ids, arguments are JSON matching the tool schema (typed: integer, number, boolean, enum, arrays), `strict: false`, streamed call deltas, a tool round trip | -| `test_response_format.py` | `json_object`, `json_schema` (flat, enum, Pydantic `$defs`/`$ref`, `const`/`anyOf`/bounds/`pattern`), streamed, with and without thinking, schema vs. `auto` tools, `text` | +| `test_tool_calls.py` | `auto` / `required` / forced function / `none`, `parallel_tool_calls: false`, ids, arguments are JSON matching the tool schema (typed: integer, number, boolean, enum, arrays, an unanchored string `pattern`, a `\S` pattern with `maxLength`, short or long, nested), `strict: false`, streamed call deltas, a tool round trip | +| `test_response_format.py` | `json_object`, `json_schema` (flat, enum, Pydantic `$defs`/`$ref`, `const`/`anyOf`/bounds/`pattern`, unanchored `pattern`), streamed, with and without thinking, schema vs. `auto` tools, `text` | | `test_logprobs.py` | `logprobs` / `top_logprobs` shape in both modes, greedy token heads its `top_logprobs`, entries spell the content and nothing else (no reasoning, no tool call markup), stream and non-stream report the same tokens, `max_tokens` N writes at most N tokens and `completion_tokens` counts each, malformed values get 400 | | `test_sampling.py` | The same `seed` samples the same text (with an unseeded request in between, streamed or not), `repeat_last_n: 0` turns penalties off, `-1` covers the whole reply, malformed values get 400 | | `test_continue_final_message.py` | `continue_final_message` continues a trailing assistant message instead of answering it: content is only the new text, whitespace as written (a model that reasons first may read the prefix as reasoning, and close it), `logprobs` cover exactly the continued tokens (the first entry scores the next token), stream matches non-stream, a cache-resumed continuation matches a cold one, a normal turn can follow, malformed combinations get 400 | diff --git a/src/test/openai_api_test/test_response_format.py b/src/test/openai_api_test/test_response_format.py index d19885967..fd8c7f13a 100644 --- a/src/test/openai_api_test/test_response_format.py +++ b/src/test/openai_api_test/test_response_format.py @@ -112,6 +112,17 @@ def schema_keywords(api: Api) -> None: nonstream_json(api, body, REVIEW_SCHEMA) +@case +def schema_unanchored_pattern(api: Api) -> None: + """An unanchored pattern only has to match somewhere in the string.""" + schema = {"type": "object", "properties": {"city": {"type": "string", "pattern": "\\S"}}, + "required": ["city"], "additionalProperties": False} + body = api.body(user("Which city is the capital of France? Answer as JSON."), + response_format=json_schema_format("city", schema), max_tokens=64) + _, value = nonstream_json(api, body, schema) + expect(len(value["city"]) > 1, f"the pattern \\S held the city to one character: {value['city']!r}") + + @case def schema_stream(api: Api) -> None: """json_schema, streamed: the folded content validates.""" diff --git a/src/test/openai_api_test/test_tool_calls.py b/src/test/openai_api_test/test_tool_calls.py index dc62c1cbc..a87a2b03c 100644 --- a/src/test/openai_api_test/test_tool_calls.py +++ b/src/test/openai_api_test/test_tool_calls.py @@ -168,6 +168,50 @@ def function_typed_arguments(api: Api) -> None: expect(isinstance(args["refundable"], bool), f"refundable is not a boolean: {args['refundable']!r}") +@case +def function_unanchored_pattern(api: Api) -> None: + """An unanchored string pattern only has to match somewhere in the string, as JSON Schema reads it.""" + tool = json.loads(json.dumps(GET_WEATHER)) + tool["function"]["parameters"]["properties"]["city"] = { + "type": "string", "minLength": 1, "maxLength": 64, "pattern": "\\S"} + choice = api.ok(api.body(user(WEATHER_PROMPT), tools=[tool], tool_choice=function_choice("get_weather"))) + city = json.loads(expect_calls(choice, [tool])[0]["function"]["arguments"])["city"] + expect(len(city) > 1, f"the pattern \\S held the city to one character: {city!r}") + + +NOTE = ("The quarterly review moved to Thursday at three in the afternoon, " + "in the large conference room on the fourth floor.") + + +@case +def function_pattern_keeps_length_bounds(api: Api) -> None: + """A one-character pattern such as \\S still leaves maxLength in force.""" + tool = {"type": "function", "function": { + "name": "save_note", "description": "Save a note", + "parameters": {"type": "object", "required": ["text"], "properties": { + "text": {"type": "string", "minLength": 1, "maxLength": 20, "pattern": "\\S"}}}}} + body = api.body(user(f"Save this note word for word: {NOTE}"), tools=[tool], + tool_choice=function_choice("save_note")) + text = json.loads(expect_calls(api.ok(body), [tool])[0]["function"]["arguments"])["text"] + expect(1 < len(text) <= 20, f"text {text!r} is outside 2..20 characters") + + +@case +def function_nested_long_bounded_pattern(api: Api) -> None: + """A \\S pattern with a long maxLength inside a nested object still gets a grammar.""" + caption = {"type": "string", "minLength": 1, "maxLength": 512, "pattern": "\\S"} + tool = {"type": "function", "function": { + "name": "draw_chart", "description": "Draw a chart", + "parameters": {"type": "object", "required": ["spec"], "additionalProperties": False, + "properties": {"spec": {"type": "object", "required": ["caption"], + "additionalProperties": False, + "properties": {"caption": caption}}}}}} + choice = api.ok(api.body(user("Draw a chart of monthly sales with a short caption."), tools=[tool], + tool_choice=function_choice("draw_chart"))) + spec = json.loads(expect_calls(choice, [tool])[0]["function"]["arguments"])["spec"] + expect(0 < len(spec["caption"]) <= 512, f"caption length {len(spec['caption'])} is outside 1..512") + + @case def function_with_thinking(api: Api) -> None: """A forced call still works when the model thinks first.""" From cab2c2afbe74dac3cc6d437d402fb361dbbe4c2c Mon Sep 17 00:00:00 2001 From: Chi Zhang Date: Sat, 14 Nov 2026 16:18:20 -0800 Subject: [PATCH 3/7] feat(core): ryzenai-corelib 0.87.0, has_feature replaces get_npu and npu_name Moves the core backend to ryzenai-corelib 0.87.0 (corelib main at 5bf19507). What changed in corelib 0.71 to 0.87 that reaches FastFlowLM: - 0.73, NPU query: ryzenai_corelib_get_npu, ryzenai_corelib_npu_name and the ryzenai_corelib_npu enum are gone. ryzenai_corelib_has_feature and ryzenai_corelib_feature_name replace them, over ryzenai_corelib_feature (npu, npu_aiem, npu_aie2p, gpu, npu_direct). - The runtime's readiness check asks has_feature(npu_aiem). AIE2P is still refused, since FastFlowLM never gives corelib an xclbin, and the error names npu_aie2p or none. - NpuArch passes corelib's spelling through (npu_aiem, npu_aie2p, ""), and platform_from_corelib_arch maps npu_aiem to dev04. FastFlowLM still keeps no table of devices. - Everything else FastFlowLM resolves is unchanged: every other entry point in FLM_CORELIB_FUNCTIONS keeps its signature, and model_metadata and session_options keep their layout. 0.87's single PDI list per stream, NPU direct, the .amd.gguf exports (0.80, 0.83) and the new ops do not touch FastFlowLM's calls. - 0.83: corelib's 4-bit GGUF types are renamed (AMD_Q4_G128, AMD_Q4_G32, new AMD_Q4_G64). Files published under the old type codes (ggml 128, 129) are still read as before, so the catalog's AMD files need no change. - Model libraries: ryzenai_corelib_model_d3 is new. The core build copies ryzenai_corelib_model_*.dll by glob, so it lands beside flm.exe without a change; no -core tag uses it yet. - Not taken here: corelib now lets Llama 3.2-1B/3B and 3.1-8B read a Q4_G128 GGUF in place of their Q8_0. Q8_0 still loads, so the catalog's Llama entries stay on Q8_0; switching them is a separate change. The fake corelib answers has_feature from one NPU feature, or none, and feature_name as corelib spells it. The tests refuse none and npu_aie2p, each named in the error, and read "" for none. The Linux setup guide's version check now prints 0, 87, 0. On an AIE4 (Medusa) test machine, with the core flm.exe statically linked against corelib's default build (no --npu-direct): every -core tag except Llama 3B/8B loads, passes the readiness check through has_feature(npu_aiem), reports no fallback and answers correctly, at decode rates matching 0.70's. Gemma 4 E2B/E4B, Qwen3.8 and Qwen3.6 recall a code from the middle of 4k and 12k prompts (Qwen3.6 also at 24k). An flm.exe linked against a corelib built with --npu-direct engages NPU direct for Qwen3.6 and gives the same greedy output, with prefill about 10-15% faster from 12k tokens. The core build stays on corelib's default build: --npu-direct needs CHESS in WSL, so turning it on is a build-only follow-up. Co-authored-by: Cursor --- docs/core-setup-linux.md | 6 ++--- docs/docs/models/phi.md | 4 ++-- docs/docs/models/qwen.md | 2 +- src/common/core/corelib_runtime.cpp | 22 +++++++++++------- src/common/models/README.md | 8 +++---- src/include/core/corelib_api.hpp | 8 +++---- src/include/core/corelib_runtime.hpp | 2 +- src/include/utils/npu_platform.hpp | 4 ++-- .../test_model_list_platform.cpp | 6 ++--- src/test/phi4_core/fake_corelib.cpp | 20 ++++++++++------ src/test/phi4_core/fake_corelib.hpp | 8 +++++-- src/test/phi4_core/test_corelib_api.cpp | 23 +++++++++++-------- 12 files changed, 66 insertions(+), 47 deletions(-) diff --git a/docs/core-setup-linux.md b/docs/core-setup-linux.md index cda2cbfe6..7b4cf9e71 100644 --- a/docs/core-setup-linux.md +++ b/docs/core-setup-linux.md @@ -9,7 +9,7 @@ Written against the working install on this machine. Paths use `$P` for the dependency prefix (here `/home/alfxu/ddprefix`); substitute your own. > **Status:** the core backend has been run on hardware on Windows only. The -> steps below are updated for corelib 0.70.0, but they have not been run end +> steps below are updated for corelib 0.87.0, but they have not been run end > to end on Linux at this revision. The [Known issues](#known-issues) are from > an earlier Linux bring-up. @@ -52,14 +52,14 @@ loaded from, so those libraries must sit beside it; nothing else is needed at run time, no kernel store and no environment variable. **Version must match exactly.** `src/include/core/corelib_api.hpp` has a -`#error` for anything other than corelib **0.70.0**: +`#error` for anything other than corelib **0.87.0**: ```bash grep -E 'CORELIB_VERSION_(MAJOR|MINOR|PATCH)' \ $P/install-corelib/include/ryzenai/corelib/base.h ``` -Must print `0`, `70`, `0`. (`BUILD-NOTES.md` still says 0.4.0 in its opening +Must print `0`, `87`, `0`. (`BUILD-NOTES.md` still says 0.4.0 in its opening line — that text is stale.) ### Verify corelib before going further diff --git a/docs/docs/models/phi.md b/docs/docs/models/phi.md index 7d59c8496..ce7eca239 100644 --- a/docs/docs/models/phi.md +++ b/docs/docs/models/phi.md @@ -75,9 +75,9 @@ The `src/test/phi4_core` suite is still Windows-only and is not configured on Li ### Pointing FastFlowLM at the runtime -A core build (`-DFLM_ENABLE_CORE=ON`) **links corelib in**. corelib keeps its NPU device to itself and opens a hardware context per stream on it; FastFlowLM opens its own device for every other family, and a core model shares no buffer with those. The core backend is offered only where corelib reports an AIEM NPU (`ryzenai_corelib_get_npu`): elsewhere, AIE2P included, corelib would run a session on the GPU or the host. Point the build at the library with `RYZENAI_CORELIB_INCLUDE_DIR` and `RYZENAI_CORELIB_LIBRARY`. `FLM_CORE_CORELIB_PATH` selects a shared library (`.dll` on Windows, `.so` elsewhere) only in the older dynamically loading configuration; in a statically linked core build it is ignored, and `flm` says so if it is set. +A core build (`-DFLM_ENABLE_CORE=ON`) **links corelib in**. corelib keeps its NPU device to itself and opens a hardware context per stream on it; FastFlowLM opens its own device for every other family, and a core model shares no buffer with those. The core backend is offered only where corelib reports an AIEM NPU (`ryzenai_corelib_has_feature(ryzenai_corelib_feature_npu_aiem)`): elsewhere, AIE2P included, corelib would run a session on the GPU or the host. Point the build at the library with `RYZENAI_CORELIB_INCLUDE_DIR` and `RYZENAI_CORELIB_LIBRARY`. `FLM_CORE_CORELIB_PATH` selects a shared library (`.dll` on Windows, `.so` elsewhere) only in the older dynamically loading configuration; in a statically linked core build it is ignored, and `flm` says so if it is set. -The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.70.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build installs `ryzenai_corelib` plus one `ryzenai_corelib_model_` library per model, each carrying that model's kernels; corelib loads them only from its own directory, so they must sit beside it (a Windows core build copies them next to `flm.exe`). Each engine enables its model's kernel bundle on the stream it opens; no kernel store or environment variable is involved. Corelib's own dependency directory must be reachable on `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux). +The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.87.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build installs `ryzenai_corelib` plus one `ryzenai_corelib_model_` library per model, each carrying that model's kernels; corelib loads them only from its own directory, so they must sit beside it (a Windows core build copies them next to `flm.exe`). Each engine enables its model's kernel bundle on the stream it opens; no kernel store or environment variable is involved. Corelib's own dependency directory must be reachable on `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux). ```powershell flm pull phi4-mini-it-core:4b diff --git a/docs/docs/models/qwen.md b/docs/docs/models/qwen.md index b23acf7c0..4b2a00188 100644 --- a/docs/docs/models/qwen.md +++ b/docs/docs/models/qwen.md @@ -111,7 +111,7 @@ flm run qwen3:8b - **Backend:** `core` — one engine for all four sizes. The GGUF's own metadata selects the size and is cross-checked against it. - **Source format:** GGUF, read directly by corelib. With a Hugging Face token that can read AMD's private repositories (`HF_TOKEN`, or `hf auth login`), `flm pull` fetches AMD's 4-bit file, which packs exactly as it is; without one, a tag below on an AMD file cannot be pulled, as corelib's own runner has no public fallback either. Only Qwen3-0.6B and Llama, which AMD ships no 4-bit file of, pull a public `Q8_0`, requantized to 4-bit groups on every load, which is lossy; corelib refuses `Q8_0` for Qwen3.5, Qwen3.6, Gemma 4 and Phi-4-mini. - **Context:** each tag's limit is the most corelib runs that model at; the default is lower where corelib or our own runs found answers going wrong past it. Over-capacity requests are rejected with HTTP 400 before any work reaches the device. -- **Availability:** developer build against corelib 0.70.0 — see [Phi-4 on the core backend](phi.md). +- **Availability:** developer build against corelib 0.87.0 — see [Phi-4 on the core backend](phi.md). The same corelib path runs every `-core` tag: diff --git a/src/common/core/corelib_runtime.cpp b/src/common/core/corelib_runtime.cpp index ea812beb9..c28458b7c 100644 --- a/src/common/core/corelib_runtime.cpp +++ b/src/common/core/corelib_runtime.cpp @@ -47,11 +47,13 @@ std::shared_ptr CorelibRuntime::CreateReady( // is the silent slowness the core backend is meant to rule out. An AIE2P // NPU runs corelib's kernels only behind an xclbin, which FastFlowLM never // hands it, so only AIEM counts. - const ryzenai_corelib_npu npu = api->functions().get_npu(); - if (npu != ryzenai_corelib_npu_aiem) { - const char* name = api->functions().npu_name(npu); + const auto& functions = api->functions(); + if (!functions.has_feature(ryzenai_corelib_feature_npu_aiem)) { + const char* name = functions.has_feature(ryzenai_corelib_feature_npu_aie2p) + ? functions.feature_name(ryzenai_corelib_feature_npu_aie2p) + : nullptr; throw std::runtime_error(std::string("corelib has no usable NPU (") + - (name ? name : "unknown") + ")"); + (name ? name : "none") + ")"); } return std::shared_ptr(new CorelibRuntime(std::move(api))); } @@ -67,10 +69,14 @@ std::string CorelibRuntime::NpuArch(const std::filesystem::path& executable_dir) try { std::lock_guard lock(process_mutex); const auto api = process_runtime ? process_runtime->api_ : ProcessApi(executable_dir); - const ryzenai_corelib_npu npu = api->functions().get_npu(); - if (npu == ryzenai_corelib_npu_none) return ""; - const char* name = api->functions().npu_name(npu); - return name ? name : ""; + const auto& functions = api->functions(); + for (const auto npu : {ryzenai_corelib_feature_npu_aiem, + ryzenai_corelib_feature_npu_aie2p}) { + if (!functions.has_feature(npu)) continue; + const char* name = functions.feature_name(npu); + return name ? name : ""; + } + return ""; } catch (...) { return ""; } diff --git a/src/common/models/README.md b/src/common/models/README.md index 2658e0ee0..5df4d2b6e 100644 --- a/src/common/models/README.md +++ b/src/common/models/README.md @@ -24,9 +24,9 @@ either axis off the other is how adding corelib once took the FastFlowLM models away. `get_device()` asks the machine, once per process. A build that links corelib -asks corelib what NPU it has (`CorelibRuntime::NpuArch`, corelib's -`ryzenai_corelib_get_npu` named by `ryzenai_corelib_npu_name`): `AIEM` -is `dev04`, and `AIE2P`, no NPU, or a corelib that cannot be loaded is +asks corelib what NPU it has (`CorelibRuntime::NpuArch`, the +`ryzenai_corelib_has_feature` NPU named by `ryzenai_corelib_feature_name`): +`npu_aiem` is `dev04`, and `npu_aie2p`, no NPU, or a corelib that cannot be loaded is `default_npu_platform()`, `aie2p`. corelib reads the device itself, so FastFlowLM holds no table of devices and names none. A build without corelib answers `aie2p`: every `dev04` model is a corelib one. Linking corelib still @@ -43,7 +43,7 @@ This is a contributor document. For *using* a backend once it exists — `--back |---|---| | **Platform** | Windows and Linux. Both configure and build; only Windows has been run on hardware, and the `src/test/phi4_core` suite is still Windows-only. | | **Hardware** | An dev04 NPU. There is no simulator; a wrong shape shows up as garbage output, not an error. | -| **corelib headers** | The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.70.0`** match on major, minor and patch. [`corelib_api.hpp`](../../include/core/corelib_api.hpp) `#error`s on any other version — deliberately, because the C ABI has changed shape between patch releases. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build gives one `ryzenai_corelib_model_` library per model, carrying its kernels, which must sit beside corelib. A per-op engine enables its model's kernel bundle on the stream it opens; a model corelib runs whole enables its own. | +| **corelib headers** | The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.87.0`** match on major, minor and patch. [`corelib_api.hpp`](../../include/core/corelib_api.hpp) `#error`s on any other version — deliberately, because the C ABI has changed shape between patch releases. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build gives one `ryzenai_corelib_model_` library per model, carrying its kernels, which must sit beside corelib. A per-op engine enables its model's kernel bundle on the stream it opens; a model corelib runs whole enables its own. | | **Weights** | A GGUF corelib's model reads: a Q4_G128 or Q4_G32 one, whose codes it packs as they are, or, for Qwen3 and Llama only, a Q8_0 one, which it requantizes on every load. The catalog pulls what corelib's own runner does: AMD's Q4 file, with no public fallback, for every model but Qwen3-0.6B and Llama, which AMD ships no Q4 of and which stay on a public Q8_0. See §6. | Configure with: diff --git a/src/include/core/corelib_api.hpp b/src/include/core/corelib_api.hpp index c37e10fb7..ab1a1e536 100644 --- a/src/include/core/corelib_api.hpp +++ b/src/include/core/corelib_api.hpp @@ -2,9 +2,9 @@ #include -#if RYZENAI_CORELIB_VERSION_MAJOR != 0 || RYZENAI_CORELIB_VERSION_MINOR != 70 || \ +#if RYZENAI_CORELIB_VERSION_MAJOR != 0 || RYZENAI_CORELIB_VERSION_MINOR != 87 || \ RYZENAI_CORELIB_VERSION_PATCH != 0 -#error "FastFlowLM requires ryzenai-corelib headers exactly 0.70.0" +#error "FastFlowLM requires ryzenai-corelib headers exactly 0.87.0" #endif #include @@ -22,8 +22,8 @@ X(status_to_string, ryzenai_corelib_status_to_string) \ X(get_last_error_message, ryzenai_corelib_get_last_error_message) \ X(selftest_dependencies, ryzenai_corelib_selftest_dependencies) \ - X(get_npu, ryzenai_corelib_get_npu) \ - X(npu_name, ryzenai_corelib_npu_name) \ + X(has_feature, ryzenai_corelib_has_feature) \ + X(feature_name, ryzenai_corelib_feature_name) \ X(object_release, ryzenai_corelib_object_release) \ X(create_stream, ryzenai_corelib_create_stream) \ X(stream_synchronize, ryzenai_corelib_stream_synchronize) \ diff --git a/src/include/core/corelib_runtime.hpp b/src/include/core/corelib_runtime.hpp index 6e08299bf..bdd642855 100644 --- a/src/include/core/corelib_runtime.hpp +++ b/src/include/core/corelib_runtime.hpp @@ -19,7 +19,7 @@ class CorelibRuntime final { const std::filesystem::path& executable_dir); static std::shared_ptr CreateForTest( std::shared_ptr api); - /// \brief this machine's NPU as corelib names it: "AIEM", "AIE2P", or "" + /// \brief this machine's NPU as corelib names it: "npu_aiem", "npu_aie2p", or "" /// when it knows none or cannot be loaded /// \note Needs no usable NPU and opens no device, so it answers where /// GetOrCreate would refuse; it loads the API GetOrCreate then reuses. diff --git a/src/include/utils/npu_platform.hpp b/src/include/utils/npu_platform.hpp index 96aab6560..b54c469bf 100644 --- a/src/include/utils/npu_platform.hpp +++ b/src/include/utils/npu_platform.hpp @@ -36,9 +36,9 @@ constexpr std::string_view platform_id(npu_platform platform) { constexpr npu_platform default_npu_platform() { return npu_platform::aie2p; } /// \brief the generation corelib's name for this machine's NPU stands for -/// \param arch what corelib reports: "AIEM", "AIE2P", or "" for none it knows +/// \param arch what corelib reports: "npu_aiem", "npu_aie2p", or "" for none it knows constexpr npu_platform platform_from_corelib_arch(std::string_view arch) { - return arch == "AIEM" ? npu_platform::dev04 : default_npu_platform(); + return arch == "npu_aiem" ? npu_platform::dev04 : default_npu_platform(); } /// \brief ask the machine which NPU generation it has diff --git a/src/test/model_list_platform/test_model_list_platform.cpp b/src/test/model_list_platform/test_model_list_platform.cpp index d0a970ad1..bfe7636cc 100644 --- a/src/test/model_list_platform/test_model_list_platform.cpp +++ b/src/test/model_list_platform/test_model_list_platform.cpp @@ -898,10 +898,10 @@ void test_platform_helpers() { utils::npu_platform::dev04); TEST_REQUIRE(utils::default_npu_platform() == utils::npu_platform::aie2p); - // corelib names the NPU; only its AIEM is dev04, and anything else, + // corelib names the NPU; only its npu_aiem is dev04, and anything else, // no NPU included, is the default. - TEST_REQUIRE(utils::platform_from_corelib_arch("AIEM") == utils::npu_platform::dev04); - TEST_REQUIRE(utils::platform_from_corelib_arch("AIE2P") == utils::npu_platform::aie2p); + TEST_REQUIRE(utils::platform_from_corelib_arch("npu_aiem") == utils::npu_platform::dev04); + TEST_REQUIRE(utils::platform_from_corelib_arch("npu_aie2p") == utils::npu_platform::aie2p); TEST_REQUIRE(utils::platform_from_corelib_arch("") == utils::npu_platform::aie2p); // This hardware-free target does not define FLM_ENABLE_CORE, so the diff --git a/src/test/phi4_core/fake_corelib.cpp b/src/test/phi4_core/fake_corelib.cpp index 66670e3c1..872596c80 100644 --- a/src/test/phi4_core/fake_corelib.cpp +++ b/src/test/phi4_core/fake_corelib.cpp @@ -174,13 +174,19 @@ struct TypedFake { return current_detail.c_str(); } else if constexpr (std::is_same_v) { return state.selftest_status; - } else if constexpr (std::is_same_v) { - return state.npu; - } else if constexpr (std::is_same_v) { + } else if constexpr (std::is_same_v) { + const auto feature = std::get<0>(arguments); + if (feature == ryzenai_corelib_feature_npu) return state.npu != fake_corelib::kNoNpu; + return (feature == ryzenai_corelib_feature_npu_aiem || + feature == ryzenai_corelib_feature_npu_aie2p) && + feature == state.npu; + } else if constexpr (std::is_same_v) { switch (std::get<0>(arguments)) { - case ryzenai_corelib_npu_none: return "none"; - case ryzenai_corelib_npu_aiem: return "AIEM"; - case ryzenai_corelib_npu_aie2p: return "AIE2P"; + case ryzenai_corelib_feature_npu: return "npu"; + case ryzenai_corelib_feature_npu_aiem: return "npu_aiem"; + case ryzenai_corelib_feature_npu_aie2p: return "npu_aie2p"; + case ryzenai_corelib_feature_gpu: return "gpu"; + case ryzenai_corelib_feature_npu_direct: return "npu_direct"; default: return static_cast(nullptr); } } else if constexpr (std::is_same_v) { @@ -673,7 +679,7 @@ void Reset() { RYZENAI_CORELIB_VERSION_PATCH}; state.selftest_status = ryzenai_corelib_status_success; state.default_status = ryzenai_corelib_status_success; - state.npu = ryzenai_corelib_npu_aiem; + state.npu = ryzenai_corelib_feature_npu_aiem; state.detail.clear(); state.status_text = "success"; state.missing_symbol.clear(); diff --git a/src/test/phi4_core/fake_corelib.hpp b/src/test/phi4_core/fake_corelib.hpp index 6ffa7ead3..4318116ef 100644 --- a/src/test/phi4_core/fake_corelib.hpp +++ b/src/test/phi4_core/fake_corelib.hpp @@ -86,6 +86,9 @@ struct WeightFromFileRecord { /// \brief the position a forward records when it names none inline constexpr std::size_t kPositionContinue = SIZE_MAX; +/// \brief State::npu for a machine without an NPU corelib knows +inline constexpr auto kNoNpu = static_cast(0); + struct ModelForwardRecord { std::size_t rows{}; std::size_t position{}; @@ -101,8 +104,9 @@ struct State { RYZENAI_CORELIB_VERSION_PATCH}; ryzenai_corelib_status selftest_status{ryzenai_corelib_status_success}; ryzenai_corelib_status default_status{ryzenai_corelib_status_success}; - /// \brief what get_npu() answers; npu_name() names it as corelib does - ryzenai_corelib_npu npu{ryzenai_corelib_npu_aiem}; + /// \brief the NPU has_feature() reports, npu_aiem, npu_aie2p or kNoNpu; + /// feature_name() names features as corelib does + ryzenai_corelib_feature npu{ryzenai_corelib_feature_npu_aiem}; std::string detail; std::string status_text{"success"}; std::string missing_symbol; diff --git a/src/test/phi4_core/test_corelib_api.cpp b/src/test/phi4_core/test_corelib_api.cpp index 88c056fc7..33fad0833 100644 --- a/src/test/phi4_core/test_corelib_api.cpp +++ b/src/test/phi4_core/test_corelib_api.cpp @@ -222,13 +222,16 @@ void TestRuntimeRunsDependencySelftestAndRequiresAUsableNpu() { }), "selftest_dependencies"); - for (const auto npu : {ryzenai_corelib_npu_none, ryzenai_corelib_npu_aie2p}) { + for (const auto& [npu, name] : + {std::pair{fake_corelib::kNoNpu, "(none)"}, + std::pair{ryzenai_corelib_feature_npu_aie2p, "(npu_aie2p)"}}) { fake_corelib::Reset(); fake_corelib::GetState().npu = npu; - RequireContains(RequireThrows([] { - CorelibRuntime::CreateForTest(ValidApi()); - }), - "no usable NPU"); + const std::string error = RequireThrows([] { + CorelibRuntime::CreateForTest(ValidApi()); + }); + RequireContains(error, "no usable NPU"); + RequireContains(error, name); } fake_corelib::Reset(); @@ -241,12 +244,12 @@ void TestNpuArchIsCorelibsAnswerAndEmptyWhenItCannotLoad() { fake_corelib::Reset(); { const auto runtime = CorelibRuntime::CreateForTest(ValidApi()); - fake_corelib::GetState().npu = ryzenai_corelib_npu_aie2p; - TEST_REQUIRE(CorelibRuntime::NpuArch({}) == "AIE2P"); - fake_corelib::GetState().npu = ryzenai_corelib_npu_none; + fake_corelib::GetState().npu = ryzenai_corelib_feature_npu_aie2p; + TEST_REQUIRE(CorelibRuntime::NpuArch({}) == "npu_aie2p"); + fake_corelib::GetState().npu = fake_corelib::kNoNpu; TEST_REQUIRE(CorelibRuntime::NpuArch({}).empty()); - fake_corelib::GetState().npu = ryzenai_corelib_npu_aiem; - TEST_REQUIRE(CorelibRuntime::NpuArch({}) == "AIEM"); + fake_corelib::GetState().npu = ryzenai_corelib_feature_npu_aiem; + TEST_REQUIRE(CorelibRuntime::NpuArch({}) == "npu_aiem"); } CorelibRuntime::ShutdownProcess(); From eb6a9b7930a0a39960b56b7114cef18febc0114a Mon Sep 17 00:00:00 2001 From: Chi Zhang Date: Sat, 14 Nov 2026 16:18:20 -0800 Subject: [PATCH 4/7] test(core): the frontend stubs return a full vocabulary of logits; the DFlash entries hold no licence Two checks added on main left two Windows-only core suites failing on main itself. Only tests change here. - test_phi4_frontend, test_llama_frontend: #874 added a check to AutoModel::_sample that refuses logits shorter than the sampler's in_features, which is vocab_size from config.json. Each test's FakeEngine returned buffer(1) from forward and prefill, so every test that samples failed with "Backend returned fewer logits than the frontend vocabulary". They now return kVocabSize logits, as the core backend does: 200064 for Phi-4 and 128256 for Llama. config.json takes vocab_size from the same constant. - test_model_downloader: #950 dropped LICENSE.OpenMDW-1.1 and LICENSE from the files of nemotron3.5-bolt-dflash:e4b and qwen3.8-dflash2:27b, but the test still expected them. It now expects config.json and dflash_weight.q4nx, as the catalog lists. The check itself is unchanged. The real core backend returns logits as wide as corelib's vocab_size, which equals config.json's for the shipped -core tags, so it does not trip there. Co-authored-by: Cursor --- src/test/llama_core/test_llama_frontend.cpp | 9 ++++++--- src/test/phi4_core/test_model_downloader.cpp | 4 ++-- src/test/phi4_core/test_phi4_frontend.cpp | 9 ++++++--- 3 files changed, 14 insertions(+), 8 deletions(-) diff --git a/src/test/llama_core/test_llama_frontend.cpp b/src/test/llama_core/test_llama_frontend.cpp index be073295c..3babd661e 100644 --- a/src/test/llama_core/test_llama_frontend.cpp +++ b/src/test/llama_core/test_llama_frontend.cpp @@ -35,6 +35,9 @@ const std::vector kPackageEosIds = {128001, 128008, 128009}; /// \brief what llama_core_backend.cpp states const std::vector kCoreEosIds = {128009, 128008, 128001}; constexpr int kBosId = 128000; +/// \brief config.json's vocab_size; a forward returns this many logits, as the +/// frontend refuses fewer +constexpr int kVocabSize = 128256; std::vector g_encoded_tokens; std::vector g_samples; @@ -49,14 +52,14 @@ class FakeEngine final : public causal_lm { forwarded.push_back(token); if (fail_forward) Fail(); ++position; - return buffer(1); + return buffer(kVocabSize); } buffer prefill(std::vector& tokens, void*) override { ++prefill_calls; prefill_sizes.push_back(static_cast(tokens.size())); if (fail_prefill) Fail(); position += static_cast(tokens.size()); - return buffer(1); + return buffer(kVocabSize); } void set_context_length(int value) override { position = value; } void load_weights(Q4NX&) override {} @@ -142,7 +145,7 @@ class TempPackage final { ("flm-llama-frontend-" + std::to_string(++serial)); std::filesystem::create_directories(path_); Write(path_ / "config.json", - {{"model_type", "llama"}, {"vocab_size", 128256}, + {{"model_type", "llama"}, {"vocab_size", kVocabSize}, {"bos_token_id", kBosId}, {"eos_token_id", eos_token_id}}); Write(path_ / "tokenizer_config.json", {{"bos_token", "<|begin_of_text|>"}, {"bos_token_id", kBosId}, diff --git a/src/test/phi4_core/test_model_downloader.cpp b/src/test/phi4_core/test_model_downloader.cpp index 94973194a..5c55aadee 100644 --- a/src/test/phi4_core/test_model_downloader.cpp +++ b/src/test/phi4_core/test_model_downloader.cpp @@ -866,7 +866,7 @@ void TestShippedCatalogDeclaresTheDflashDependant() { TEST_REQUIRE(info.at("details").at("family") == "gemma4e-dflash"); TEST_REQUIRE(info.at("xclbin_folder") == "Nemotron-3.5-Bolt-E4B-DFlash-NPU2"); TEST_REQUIRE(info.at("files").get>() == - (std::vector{"config.json", "dflash_weight.q4nx", "LICENSE.OpenMDW-1.1"})); + (std::vector{"config.json", "dflash_weight.q4nx"})); // the base entry is the shipped plain-decode tag, untouched const auto base = models.get_model_info("nemotron3.5-bolt:e4b").second; TEST_REQUIRE(base.at("name") == "Nemotron-3.5-Bolt-E4B-NPU2"); @@ -899,7 +899,7 @@ void TestShippedCatalogDeclaresTheDflashDependant() { TEST_REQUIRE(q38_info.at("details").at("family") == "qwen3.8-dflash2"); TEST_REQUIRE(q38_info.at("xclbin_folder") == "Qwen3.8-27B-NPU2"); TEST_REQUIRE(q38_info.at("files").get>() == - (std::vector{"config.json", "dflash_weight.q4nx", "LICENSE"})); + (std::vector{"config.json", "dflash_weight.q4nx"})); TEST_REQUIRE(models.get_model_info("qwen3.8-mtp:27b").second.at("name") == "Qwen3.8-27B-NPU2"); } diff --git a/src/test/phi4_core/test_phi4_frontend.cpp b/src/test/phi4_core/test_phi4_frontend.cpp index 7218e54d8..f339a6a8e 100644 --- a/src/test/phi4_core/test_phi4_frontend.cpp +++ b/src/test/phi4_core/test_phi4_frontend.cpp @@ -39,6 +39,9 @@ void register_builtin_backends(BackendRegistry&) {} namespace { constexpr const char* kGguf = "Phi-4-mini-instruct-Q4_G128.gguf"; +/// \brief config.json's vocab_size; a forward returns this many logits, as the +/// frontend refuses fewer +constexpr int kVocabSize = 200064; std::vector g_encoded_tokens; std::vector g_samples; @@ -59,7 +62,7 @@ class FakeEngine final : public causal_lm { throw std::runtime_error("submitted inference failed"); } ++position; - return buffer(1); + return buffer(kVocabSize); } buffer prefill(std::vector& tokens, void*) override { ++prefill_calls; @@ -69,7 +72,7 @@ class FakeEngine final : public causal_lm { throw std::runtime_error("submitted inference failed"); } position += static_cast(tokens.size()); - return buffer(1); + return buffer(kVocabSize); } void set_context_length(int value) override { position = value; } void load_weights(Q4NX&) override {} @@ -183,7 +186,7 @@ nlohmann::json PackageConfig() { return {{"model_type", "phi3"}, {"num_hidden_layers", 32}, {"hidden_size", 3072}, {"intermediate_size", 8192}, {"num_attention_heads", 24}, {"num_key_value_heads", 8}, - {"head_dim", 128}, {"vocab_size", 200064}, + {"head_dim", 128}, {"vocab_size", kVocabSize}, {"rms_norm_eps", 1.0e-5}, {"original_max_position_embeddings", 4096}, {"eos_token_id", 199999}}; } From a7a0134a5e741ff7991b45dbf742fc2d655f326d Mon Sep 17 00:00:00 2001 From: "Zhang, Chi" Date: Mon, 16 Nov 2026 16:02:48 -0800 Subject: [PATCH 5/7] feat(core): ryzenai-corelib 0.103.0 Moves the core backend to ryzenai-corelib 0.103.0 (corelib main at 49a8ed3c). What changed in corelib 0.88 to 0.103 that reaches FastFlowLM: - Every entry point in FLM_CORELIB_FUNCTIONS keeps its signature, and model_metadata keeps its layout, so the only code change is the required version in corelib_api.hpp. - model_session_options: lora is gone (0.97: a session runs LoRA kernels when its model has an adapter set), and low_memory (0.96) and kv_cache_bits (0.98) are appended. FastFlowLM value-initialises the struct and sets only max_sequence_length, so the defaults hold: no LoRA, the embedding table mapped, a BF16 KV cache. - Additive only, as far as FastFlowLM's calls go: uint8 tensors, stream_set_low_memory, PQ2_0 weights, head_gate_bf16, hadamard_bf16, and NPU direct on AIE2P. FastFlowLM still refuses AIE2P. - Model libraries: qwen2, lfm2, lfm3, hyv3, glide, minicpm_ocr and minicpmv47 are new. The core build copies ryzenai_corelib_model_*.dll by glob, so they land beside flm.exe without a change; no -core tag uses them yet. - Not taken here: the 8-bit KV cache (kv_cache_bits 8, AIE4 NPU direct only) and low_memory sessions. Both are opt-in and would be separate changes. The docs' version checks and the required-version notes move to 0.103. On an AIE4 (Medusa) test machine, qwen3-core 0.6b/1.7b/4b/8b, phi4-mini-it-core:4b, qwen3.5-core 4b/9b and llama3.2-core:1b each load, report no fallback and answer correctly, at decode rates matching 0.87's. Co-authored-by: Cursor --- docs/core-setup-linux.md | 6 +++--- docs/docs/models/phi.md | 2 +- docs/docs/models/qwen.md | 2 +- src/common/models/README.md | 2 +- src/include/core/corelib_api.hpp | 4 ++-- 5 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/core-setup-linux.md b/docs/core-setup-linux.md index 7b4cf9e71..7bf6d2ac8 100644 --- a/docs/core-setup-linux.md +++ b/docs/core-setup-linux.md @@ -9,7 +9,7 @@ Written against the working install on this machine. Paths use `$P` for the dependency prefix (here `/home/alfxu/ddprefix`); substitute your own. > **Status:** the core backend has been run on hardware on Windows only. The -> steps below are updated for corelib 0.87.0, but they have not been run end +> steps below are updated for corelib 0.103.0, but they have not been run end > to end on Linux at this revision. The [Known issues](#known-issues) are from > an earlier Linux bring-up. @@ -52,14 +52,14 @@ loaded from, so those libraries must sit beside it; nothing else is needed at run time, no kernel store and no environment variable. **Version must match exactly.** `src/include/core/corelib_api.hpp` has a -`#error` for anything other than corelib **0.87.0**: +`#error` for anything other than corelib **0.103.0**: ```bash grep -E 'CORELIB_VERSION_(MAJOR|MINOR|PATCH)' \ $P/install-corelib/include/ryzenai/corelib/base.h ``` -Must print `0`, `87`, `0`. (`BUILD-NOTES.md` still says 0.4.0 in its opening +Must print `0`, `103`, `0`. (`BUILD-NOTES.md` still says 0.4.0 in its opening line — that text is stale.) ### Verify corelib before going further diff --git a/docs/docs/models/phi.md b/docs/docs/models/phi.md index ce7eca239..d97c5a174 100644 --- a/docs/docs/models/phi.md +++ b/docs/docs/models/phi.md @@ -77,7 +77,7 @@ The `src/test/phi4_core` suite is still Windows-only and is not configured on Li A core build (`-DFLM_ENABLE_CORE=ON`) **links corelib in**. corelib keeps its NPU device to itself and opens a hardware context per stream on it; FastFlowLM opens its own device for every other family, and a core model shares no buffer with those. The core backend is offered only where corelib reports an AIEM NPU (`ryzenai_corelib_has_feature(ryzenai_corelib_feature_npu_aiem)`): elsewhere, AIE2P included, corelib would run a session on the GPU or the host. Point the build at the library with `RYZENAI_CORELIB_INCLUDE_DIR` and `RYZENAI_CORELIB_LIBRARY`. `FLM_CORE_CORELIB_PATH` selects a shared library (`.dll` on Windows, `.so` elsewhere) only in the older dynamically loading configuration; in a statically linked core build it is ignored, and `flm` says so if it is set. -The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.87.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build installs `ryzenai_corelib` plus one `ryzenai_corelib_model_` library per model, each carrying that model's kernels; corelib loads them only from its own directory, so they must sit beside it (a Windows core build copies them next to `flm.exe`). Each engine enables its model's kernel bundle on the stream it opens; no kernel store or environment variable is involved. Corelib's own dependency directory must be reachable on `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux). +The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.103.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build installs `ryzenai_corelib` plus one `ryzenai_corelib_model_` library per model, each carrying that model's kernels; corelib loads them only from its own directory, so they must sit beside it (a Windows core build copies them next to `flm.exe`). Each engine enables its model's kernel bundle on the stream it opens; no kernel store or environment variable is involved. Corelib's own dependency directory must be reachable on `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux). ```powershell flm pull phi4-mini-it-core:4b diff --git a/docs/docs/models/qwen.md b/docs/docs/models/qwen.md index 4b2a00188..84731809e 100644 --- a/docs/docs/models/qwen.md +++ b/docs/docs/models/qwen.md @@ -111,7 +111,7 @@ flm run qwen3:8b - **Backend:** `core` — one engine for all four sizes. The GGUF's own metadata selects the size and is cross-checked against it. - **Source format:** GGUF, read directly by corelib. With a Hugging Face token that can read AMD's private repositories (`HF_TOKEN`, or `hf auth login`), `flm pull` fetches AMD's 4-bit file, which packs exactly as it is; without one, a tag below on an AMD file cannot be pulled, as corelib's own runner has no public fallback either. Only Qwen3-0.6B and Llama, which AMD ships no 4-bit file of, pull a public `Q8_0`, requantized to 4-bit groups on every load, which is lossy; corelib refuses `Q8_0` for Qwen3.5, Qwen3.6, Gemma 4 and Phi-4-mini. - **Context:** each tag's limit is the most corelib runs that model at; the default is lower where corelib or our own runs found answers going wrong past it. Over-capacity requests are rejected with HTTP 400 before any work reaches the device. -- **Availability:** developer build against corelib 0.87.0 — see [Phi-4 on the core backend](phi.md). +- **Availability:** developer build against corelib 0.103.0 — see [Phi-4 on the core backend](phi.md). The same corelib path runs every `-core` tag: diff --git a/src/common/models/README.md b/src/common/models/README.md index 5df4d2b6e..22b025673 100644 --- a/src/common/models/README.md +++ b/src/common/models/README.md @@ -43,7 +43,7 @@ This is a contributor document. For *using* a backend once it exists — `--back |---|---| | **Platform** | Windows and Linux. Both configure and build; only Windows has been run on hardware, and the `src/test/phi4_core` suite is still Windows-only. | | **Hardware** | An dev04 NPU. There is no simulator; a wrong shape shows up as garbage output, not an error. | -| **corelib headers** | The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.87.0`** match on major, minor and patch. [`corelib_api.hpp`](../../include/core/corelib_api.hpp) `#error`s on any other version — deliberately, because the C ABI has changed shape between patch releases. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build gives one `ryzenai_corelib_model_` library per model, carrying its kernels, which must sit beside corelib. A per-op engine enables its model's kernel bundle on the stream it opens; a model corelib runs whole enables its own. | +| **corelib headers** | The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.103.0`** match on major, minor and patch. [`corelib_api.hpp`](../../include/core/corelib_api.hpp) `#error`s on any other version — deliberately, because the C ABI has changed shape between patch releases. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build gives one `ryzenai_corelib_model_` library per model, carrying its kernels, which must sit beside corelib. A per-op engine enables its model's kernel bundle on the stream it opens; a model corelib runs whole enables its own. | | **Weights** | A GGUF corelib's model reads: a Q4_G128 or Q4_G32 one, whose codes it packs as they are, or, for Qwen3 and Llama only, a Q8_0 one, which it requantizes on every load. The catalog pulls what corelib's own runner does: AMD's Q4 file, with no public fallback, for every model but Qwen3-0.6B and Llama, which AMD ships no Q4 of and which stay on a public Q8_0. See §6. | Configure with: diff --git a/src/include/core/corelib_api.hpp b/src/include/core/corelib_api.hpp index ab1a1e536..913b47d0b 100644 --- a/src/include/core/corelib_api.hpp +++ b/src/include/core/corelib_api.hpp @@ -2,9 +2,9 @@ #include -#if RYZENAI_CORELIB_VERSION_MAJOR != 0 || RYZENAI_CORELIB_VERSION_MINOR != 87 || \ +#if RYZENAI_CORELIB_VERSION_MAJOR != 0 || RYZENAI_CORELIB_VERSION_MINOR != 103 || \ RYZENAI_CORELIB_VERSION_PATCH != 0 -#error "FastFlowLM requires ryzenai-corelib headers exactly 0.87.0" +#error "FastFlowLM requires ryzenai-corelib headers exactly 0.103.0" #endif #include From 3d4a048c2177ede55b15d55df0ae49ba1757a1b7 Mon Sep 17 00:00:00 2001 From: tawei-amd Date: Tue, 17 Nov 2026 10:45:51 -0800 Subject: [PATCH 6/7] fix(chat): Gemma4 and LFM2 tool grammars keep a string's minLength/maxLength The argument grammars FastFlowLM writes itself (Gemma4, LFM2) gave every string property the unbounded string rule, so a tool parameter with maxLength 20 took a whole sentence: test_tool_calls function_pattern_keeps_length_bounds failed on gemma4-it:e2b, Linux and Windows. normalize_schema had also folded the bounds into a pattern that only xgrammar's JSON schema reads. These grammars now read the schema as sent and spell bounds up to XGRAMMAR_REGEX_REPEAT_LIMIT: Gemma4 through counted gstr__ rules (a "<" step takes the next character too, so lengths stay exact and "<|" still cannot occur), LFM2 through a {low,high} repetition where an escape is one character. Array items keep their bounds too. Co-Authored-By: Claude Opus 5.5 --- src/common/chat/chat_grammar.cpp | 146 ++++++++++++++++++++---- src/test/chat_tools/test_chat_tools.cpp | 59 +++++++++- 2 files changed, 181 insertions(+), 24 deletions(-) diff --git a/src/common/chat/chat_grammar.cpp b/src/common/chat/chat_grammar.cpp index 738d8108e..f8f4f0a29 100644 --- a/src/common/chat/chat_grammar.cpp +++ b/src/common/chat/chat_grammar.cpp @@ -279,6 +279,9 @@ json normalize_schema(json schema) { struct tool_def { std::string name; json parameters; // a JSON schema, or true for any arguments + /// the schema as the request sent it, for the argument grammars written here: + /// normalize_schema folds length bounds into a pattern only xgrammar reads + json declared; }; std::vector request_tools(const chat_constraint_request& request) { @@ -301,8 +304,9 @@ std::vector request_tools(const chat_constraint_request& request) { } // as xgrammar does: no schema, or a non-strict tool, still gets well formed arguments const bool strict = fn.value("strict", true); - tool.parameters = strict && fn.contains("parameters") && fn["parameters"].is_object() - ? normalize_schema(fn["parameters"]) : json(true); + tool.declared = strict && fn.contains("parameters") && fn["parameters"].is_object() + ? fn["parameters"] : json(true); + tool.parameters = tool.declared.is_object() ? normalize_schema(tool.declared) : tool.declared; tools.push_back(std::move(tool)); } return tools; @@ -343,18 +347,53 @@ std::string value_rule(const json& property, const char* generic) { return generic; } +/// \brief How a dialect writes the values of declared properties +struct value_syntax { + /// spells a string enum value the way the dialect writes strings + std::function quote_string; + /// EBNF between two array items + std::string array_sep; + /// EBNF for a string of low..high characters (high -1: no upper bound) + std::function bounded_string; +}; + +/// \brief A string property's minLength/maxLength as {low, high}, high -1 for none +/// \return std::nullopt when the property is not a plain string with usable bounds: +/// none, unsatisfiable ones (left for the model, as before), or ones past +/// XGRAMMAR_REGEX_REPEAT_LIMIT, which would cost a rule per character +std::optional> string_length_bounds(const json& property) { + if (property.value("type", json()) != "string" || property.contains("enum")) { + return std::nullopt; + } + const json min_length = property.value("minLength", json(0)); + const json max_length = property.value("maxLength", json(-1)); + if (!min_length.is_number_integer() || !max_length.is_number_integer()) { + return std::nullopt; + } + const int64_t low = std::max(min_length.get(), 0); + const int64_t high = max_length.get() < 0 ? -1 : max_length.get(); + if ((low == 0 && high < 0) || (high >= 0 && high < low) || + (high < 0 ? low : high) > XGRAMMAR_REGEX_REPEAT_LIMIT) { + return std::nullopt; + } + return std::make_pair(low, high); +} + /// \brief EBNF for one property's value, from its schema type -/// \param quote_string spells a string enum value the way the dialect writes strings -/// \param array_sep EBNF between two array items -/// \note An array follows its "items" schema, so enums and types inside lists are -/// kept too; objects nested in a value are only held to the dialect's syntax. -std::string property_value(const json& property, const std::function& quote_string, - const std::string& array_sep) { +/// \note An array follows its "items" schema, so enums, types and string lengths +/// inside lists are kept too; objects nested in a value are only held to the +/// dialect's syntax. +std::string property_value(const json& property, const value_syntax& syntax) { std::string value = value_rule(property, "value"); if (value == "array" && property.contains("items") && property["items"].is_object() && !property["items"].empty()) { - const std::string item = property_value(property["items"], quote_string, array_sep); - return "(\"[\" (" + item + " (" + array_sep + " " + item + ")*)? \"]\")"; + const std::string item = property_value(property["items"], syntax); + return "(\"[\" (" + item + " (" + syntax.array_sep + " " + item + ")*)? \"]\")"; + } + if (value == "string") { + if (const auto bounds = string_length_bounds(property)) { + return syntax.bounded_string(bounds->first, bounds->second); + } } if (!value.empty()) { return value; @@ -362,7 +401,7 @@ std::string property_value(const json& property, const std::function()); + value += (first ? "" : " | ") + syntax.quote_string(option.get()); first = false; } return value + ")"; @@ -372,7 +411,7 @@ std::string property_value(const json& property, const std::functionargs; a schema without declared @@ -381,8 +420,7 @@ std::string property_value(const json& property, const std::function is the list /// starting at property i, rest the remainder after something was written. std::string argument_list_rules(const json& schema, const std::string& prefix, const std::string& kv_sep, - const std::string& item_sep, const std::string& array_sep, - const std::function& quote_string, bool sorted) { + const std::string& item_sep, const value_syntax& syntax, bool sorted) { std::ostringstream g; const bool declared = schema.is_object() && schema.contains("properties") && schema["properties"].is_object() && !schema["properties"].empty() && schema.value("additionalProperties", false) == false; @@ -406,7 +444,7 @@ std::string argument_list_rules(const json& schema, const std::string& prefix, c auto rule = [&prefix](const char* kind, size_t i) { return prefix + kind + std::to_string(i); }; for (size_t i = 0; i < n; i++) { g << rule("p", i) << " ::= " << ebnf_literal(names[i]) << " " << kv_sep << " " - << property_value(schema["properties"][names[i]], quote_string, array_sep) << "\n"; + << property_value(schema["properties"][names[i]], syntax) << "\n"; } for (size_t i = 0; i <= n; i++) { if (i == n) { @@ -433,12 +471,73 @@ const char* BOUNDED_NUMBER_RULES = "integer ::= \"-\"? (\"0\" | [1-9] [0-9]{0,15})\n" "number ::= integer (\".\" [0-9]{1,15})? ([eE] [+-]? [0-9]{1,3})?\n"; +/// \brief Rules for Gemma4 string text of a bounded length, named gstr__ +/// \note Gemma4 writes strings raw between <|"|> marks, so the text may hold no "<|". +/// A step takes one character other than "<", or "<" with the character after +/// it (neither "<" nor "|"), and counts what it took: the length comes out exact, +/// which a {low,high} repetition over the two-character step would not give. +class gemma4_bounded_strings { +public: + /// \brief EBNF for a string of low..high characters (high -1: no upper bound) + std::string value(int64_t low, int64_t high) { + wanted_.insert({low, high}); + return "(\"<|\\\"|>\" " + name(low, high) + " \"<|\\\"|>\")"; + } + + /// \brief The rules every value() so far refers to + std::string rules() const { + std::ostringstream g; + std::set> done; + std::vector> todo(wanted_.begin(), wanted_.end()); + while (!todo.empty()) { + const auto [low, high] = todo.back(); + todo.pop_back(); + if (!done.insert({low, high}).second) { + continue; + } + if (low == 0 && high < 0) { + g << name(low, high) << " ::= ([^<] | \"<\" [^|])*\n"; + continue; + } + std::string steps; + for (int64_t width = 1; width <= 2; width++) { + if (high >= 0 && high < width) { + continue; + } + const std::pair next = {std::max(low - width, 0), + high < 0 ? -1 : high - width}; + steps += std::string(steps.empty() ? "" : " | ") + (width == 1 ? "[^<]" : "\"<\" [^<|]") + " " + + name(next.first, next.second); + todo.push_back(next); + } + if (steps.empty()) { + g << name(low, high) << " ::= \"\"\n"; + } else if (low == 0) { + g << name(low, high) << " ::= (" << steps << ")?\n"; + } else { + g << name(low, high) << " ::= " << steps << "\n"; + } + } + return g.str(); + } + +private: + static std::string name(int64_t low, int64_t high) { + return "gstr_" + std::to_string(low) + "_" + (high < 0 ? std::string("x") : std::to_string(high)); + } + + std::set> wanted_; +}; + /// \brief Gemma4 call arguments: {key:<|"|>text<|"|>,n:5,flag:true,list:[...]} std::string gemma4_arguments_grammar(const json& schema) { - auto quote = [](const std::string& s) { return ebnf_literal("<|\"|>" + s + "<|\"|>"); }; + gemma4_bounded_strings strings; + const value_syntax syntax{[](const std::string& s) { return ebnf_literal("<|\"|>" + s + "<|\"|>"); }, "\",\"", + [&strings](int64_t low, int64_t high) { return strings.value(low, high); }}; std::ostringstream g; g << "root ::= \"{\" args \"}\"\n" - << argument_list_rules(schema, "", "\":\"", "\",\"", "\",\"", quote, true) + << argument_list_rules(schema, "", "\":\"", "\",\"", syntax, true) + << strings.rules() << "pair ::= ident \":\" value\n" << "ident ::= [a-zA-Z_] [a-zA-Z0-9_]*\n" << "value ::= string | number | boolean | \"null\" | object | array\n" @@ -452,7 +551,14 @@ std::string gemma4_arguments_grammar(const json& schema) { /// \brief The calls inside one LFM2 block: f(key="text", n=5), g(flag=True) std::string python_calls_grammar(const std::vector& tools, bool parallel) { - auto quote = [](const std::string& s) { return ebnf_literal("\"" + s + "\""); }; + // an escape such as \n is one character of the value + auto bounded = [](int64_t low, int64_t high) { + const std::string count = "{" + std::to_string(low) + "," + (high < 0 ? "" : std::to_string(high)) + "}"; + return "(\"\\\"\" ([^\"\\\\] | \"\\\\\" [^\\n])" + count + " \"\\\"\" | \"'\" ([^'\\\\] | \"\\\\\" [^\\n])" + count + + " \"'\")"; + }; + const value_syntax syntax{[](const std::string& s) { return ebnf_literal("\"" + s + "\""); }, "\",\" \" \"?", + bounded}; std::ostringstream g; g << "root ::= call" << (parallel ? " (\",\" \" \"? call)*" : "") << "\n"; g << "call ::= "; @@ -463,7 +569,7 @@ std::string python_calls_grammar(const std::vector& tools, bool parall for (size_t i = 0; i < tools.size(); i++) { const std::string prefix = "t" + std::to_string(i) + "_"; g << "call_" << i << " ::= " << ebnf_literal(tools[i].name + "(") << " " << prefix << "args \")\"\n" - << argument_list_rules(tools[i].parameters, prefix, "\"=\"", "\",\" \" \"?", "\",\" \" \"?", quote, false); + << argument_list_rules(tools[i].declared, prefix, "\"=\"", "\",\" \" \"?", syntax, false); } g << "pair ::= ident \"=\" value\n" << "ident ::= [a-zA-Z_] [a-zA-Z0-9_]*\n" @@ -513,7 +619,7 @@ tool_syntax tool_tags(const chat_format_spec& spec, const std::vector& case chat_format_kind::GEMMA4: syntax.trigger = "<|tool_call>"; syntax.tags.push_back(tag("<|tool_call>call:" + tool.name, - grammar_content(gemma4_arguments_grammar(tool.parameters)), "")); + grammar_content(gemma4_arguments_grammar(tool.declared)), "")); break; case chat_format_kind::LFM2_PYTHONIC: case chat_format_kind::CONTENT_ONLY: diff --git a/src/test/chat_tools/test_chat_tools.cpp b/src/test/chat_tools/test_chat_tools.cpp index 35d738a05..ed628416f 100644 --- a/src/test/chat_tools/test_chat_tools.cpp +++ b/src/test/chat_tools/test_chat_tools.cpp @@ -152,7 +152,7 @@ void check_prefix_stable(const chat_format_spec& spec, const std::string& text, /// \brief A grammar engine over a vocabulary of single characters, enough to match text class grammar_fixture { public: - grammar_fixture() { + explicit grammar_fixture(const chat_format_spec& spec = QWEN_XML) : spec_(spec) { json vocab = json::object(); for (int c = 1; c < 127; c++) vocab[std::string(1, static_cast(c))] = c; // a plain decoder: every token is the text it stands for @@ -164,7 +164,7 @@ class grammar_fixture { chat_vocab v; v.size = 128; v.tokenizer_json = path_; - engine_ = std::make_unique(v, QWEN_XML); + engine_ = std::make_unique(v, spec_); } ~grammar_fixture() { std::error_code ec; @@ -174,12 +174,12 @@ class grammar_fixture { /// \brief Whether the request's grammar takes the text, without reporting bool takes(const chat_constraint_request& request, const std::string& text) { std::string error; - return engine_->accepts(QWEN_XML, request, "", text, error); + return engine_->accepts(spec_, request, "", text, error); } bool accepts(const chat_constraint_request& request, const std::string& text, const char* label) { std::string error; - if (!engine_->accepts(QWEN_XML, request, "", text, error)) { + if (!engine_->accepts(spec_, request, "", text, error)) { std::printf(" FAIL %s: the grammar rejects the template's call: %s\n", label, error.c_str()); failures++; return false; @@ -188,6 +188,7 @@ class grammar_fixture { } private: + chat_format_spec spec_; std::string path_; std::unique_ptr engine_; }; @@ -552,6 +553,54 @@ void test_gemma4_nullable_types() { CHECK_EQ(tmpl->render(minja::Context::make(minja::Value(json{{"value", out["note"]}}))), "STRING nullable"); } +/// \brief One tool whose string parameter has length bounds, as test_tool_calls.py sends it +const json NOTE_TOOLS = json::array({tool( + "save_note", object_schema({{"text", {{"type", "string"}, {"minLength", 2}, {"maxLength", 6}, {"pattern", "\\S"}}}, + {"tags", {{"type", "array"}, {"items", {{"type", "string"}, {"maxLength", 3}}}}}}, + {"text"}))}); + +chat_constraint_request note_request() { + chat_constraint_request request = tools_request(chat_tool_choice::FUNCTION, "save_note"); + request.tools = &NOTE_TOOLS; + return request; +} + +void test_gemma4_string_length() { + std::printf("gemma4 grammar holds a string to its minLength/maxLength\n"); + grammar_fixture grammar(chat_format_make(chat_format_kind::GEMMA4, true)); + auto note = [](const std::string& text, const std::string& more = "") { + return "<|tool_call>call:save_note{" + more + "text:<|\"|>" + text + "<|\"|>}"; + }; + CHECK(grammar.takes(note_request(), note("ab"))); + CHECK(grammar.takes(note_request(), note("abcdef"))); + CHECK(!grammar.takes(note_request(), note("a"))); + CHECK(!grammar.takes(note_request(), note("abcdefg"))); + // "<" counts as one character, and "<|" still cannot occur in the text + CHECK(grammar.takes(note_request(), note("ax<|\"|>,<|\"|>xyz<|\"|>],"))); + CHECK(!grammar.takes(note_request(), note("ab", "tags:[<|\"|>wxyz<|\"|>],"))); +} + +void test_lfm2_string_length() { + std::printf("lfm2 grammar holds a string to its minLength/maxLength\n"); + grammar_fixture grammar(chat_format_make(chat_format_kind::LFM2_PYTHONIC, true)); + auto note = [](const std::string& text) { + return "<|tool_call_start|>[save_note(text=" + text + ")]<|tool_call_end|>"; + }; + CHECK(grammar.takes(note_request(), note("\"abcdef\""))); + CHECK(grammar.takes(note_request(), note("'ab'"))); + CHECK(!grammar.takes(note_request(), note("\"a\""))); + CHECK(!grammar.takes(note_request(), note("\"abcdefg\""))); + // an escape is one character of the value + CHECK(grammar.takes(note_request(), note("\"abcd\\\"e\""))); + CHECK(!grammar.takes(note_request(), note("\"abcde\\\"f\""))); +} + } // namespace int main() { @@ -565,6 +614,8 @@ int main() { test_continued_call(); test_no_tools_after_tool_turns(); test_gemma4_nullable_types(); + test_gemma4_string_length(); + test_lfm2_string_length(); if (failures) { std::printf("%d check(s) failed\n", failures); return 1; From 8268970e1c79bbe7fd2536f21ae909dc46d996c7 Mon Sep 17 00:00:00 2001 From: Chi Zhang Date: Tue, 17 Nov 2026 14:39:29 -0800 Subject: [PATCH 7/7] fix(chat): pad an unanchored pattern without ending the string; leave flags and inner anchors as sent MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review follow-ups to the pattern rewrite: - The padding is now a class of what a JSON string holds unescaped, not ".*". xgrammar keeps '"', '\' and control characters out of a pattern's matches only when the pattern is printable ASCII; any other pattern compiles without that guard, where ".*" could close the string and add keys ("(é|x)" took x", "k": "y). The folded repetition after a lone class uses the same class. - A pattern opening with an inline flag such as "(?i)" is left as sent: padded, it no longer builds. - A pattern with an anchor inside a group, such as "(?:^abc)", is left as sent: xgrammar ignores an anchor in the middle, so padding would have let "xabc" through. - Each top-level alternative is padded on its own, so "^a|b" reads as "starts with a, or holds b" rather than "^(a|b).*". - normalize_schema no longer enters const, enum, default or examples, which are values, nor "not", whose meaning a rewrite would change. A property named like one of them is still a schema. - The comment on unsatisfiable bounds now says what happens: xgrammar ignores the bounds beside a pattern. The folded form's narrowing is noted. The live tests check for the expected word instead of a length above one, which a valid one-character answer would have failed. New checks in test_chat_tools cover each case through the grammar engine; on the previous code "^a|b" and the non-ASCII injection fail, and the inline-flag and inner-anchor patterns crash the harness. Co-authored-by: Cursor --- src/common/chat/chat_grammar.cpp | 115 +++++++++++++++--- src/test/chat_tools/test_chat_tools.cpp | 78 ++++++++++++ .../openai_api_test/test_response_format.py | 2 +- src/test/openai_api_test/test_tool_calls.py | 4 +- 4 files changed, 180 insertions(+), 19 deletions(-) diff --git a/src/common/chat/chat_grammar.cpp b/src/common/chat/chat_grammar.cpp index f8f4f0a29..a7b2e66da 100644 --- a/src/common/chat/chat_grammar.cpp +++ b/src/common/chat/chat_grammar.cpp @@ -4,6 +4,7 @@ #include "chat/chat_grammar.hpp" #include +#include #include #include #include @@ -180,24 +181,92 @@ bool is_one_character(const std::string& pattern) { return true; } +/// \brief Any character a JSON string holds unescaped, as a regex class +/// \note Not ".": xgrammar keeps '"', '\\' and control characters out of a pattern's +/// matches only when the pattern is printable ASCII. Any other pattern is +/// compiled without that, where "." can close the string and add keys. +const std::string JSON_STRING_CHAR = "[^\"\\\\\\x00-\\x1f]"; + +/// \brief Where a regex splits into alternatives and holds anchors, outside groups and classes +struct regex_shape { + std::vector alternatives; // split at each top-level "|" + bool inner_anchor = false; // a "^" or "$" anywhere but at either end +}; + +regex_shape scan_regex(const std::string& pattern) { + regex_shape shape; + std::string current; + int depth = 0; + bool in_class = false; + for (size_t i = 0; i < pattern.size(); ++i) { + const char c = pattern[i]; + if (c == '\\' && i + 1 < pattern.size()) { + current += pattern.substr(i++, 2); + continue; + } + if (in_class) { + in_class = c != ']'; + } else if (c == '[') { + in_class = true; + if (i + 1 < pattern.size() && pattern[i + 1] == '^') current += pattern[i++]; + if (i + 1 < pattern.size() && pattern[i + 1] == ']') current += pattern[i++]; + } else if (c == '(') { + ++depth; + } else if (c == ')') { + --depth; + } else if (c == '|' && depth == 0) { + shape.alternatives.push_back(std::move(current)); + current.clear(); + continue; + } else if (c == '^' || c == '$') { + const bool at_edge = depth == 0 && (c == '^' ? current.empty() + : i + 1 == pattern.size() || pattern[i + 1] == '|'); + shape.inner_anchor = shape.inner_anchor || !at_edge; + } + current += c; + } + shape.alternatives.push_back(std::move(current)); + return shape; +} + +/// \brief One alternative of a pattern, padded where it leaves a side unanchored +std::string padded_alternative(const std::string& alternative) { + const std::string pad = JSON_STRING_CHAR + "*"; + const bool starts = !alternative.empty() && alternative.front() == '^'; + size_t escapes = 0; + while (escapes + 1 < alternative.size() && alternative[alternative.size() - 2 - escapes] == '\\') ++escapes; + const bool ends = alternative.size() > (starts ? 1u : 0u) && alternative.back() == '$' && escapes % 2 == 0; + const size_t begin = starts ? 1 : 0; + const std::string core = alternative.substr(begin, alternative.size() - begin - (ends ? 1 : 0)); + if (core.empty()) { + return starts && ends ? std::string() : pad; + } + return (starts ? "(" : pad + "(") + core + (ends ? ")" : ")" + pad); +} + /// \brief A JSON Schema string pattern as xgrammar's whole-string regex has to spell it /// \note JSON Schema finds a pattern anywhere in the string, while xgrammar matches /// the whole string against it, so "\S" (not blank) would allow exactly one -/// character. A side the pattern leaves unanchored gets ".*". +/// character. A side an alternative leaves unanchored gets padded. Left as sent: +/// a pattern anchored at both ends, one that opens with an inline flag such as +/// "(?i)", which xgrammar only takes there, and one with an anchor inside a group, +/// which xgrammar ignores rather than reads. std::string whole_string_pattern(const std::string& pattern) { - const bool starts = !pattern.empty() && pattern.front() == '^'; - size_t escapes = 0; - while (escapes + 1 < pattern.size() && pattern[pattern.size() - 2 - escapes] == '\\') ++escapes; - const bool ends = pattern.size() > (starts ? 1u : 0u) && pattern.back() == '$' && escapes % 2 == 0; - if (starts && ends) { + const bool inline_flag = pattern.size() > 2 && pattern.compare(0, 2, "(?") == 0 && + std::isalpha(static_cast(pattern[2])); + const regex_shape shape = scan_regex(pattern); + if (inline_flag || shape.inner_anchor) { return pattern; } - const size_t begin = starts ? 1 : 0; - const std::string core = pattern.substr(begin, pattern.size() - begin - (ends ? 1 : 0)); - if (core.empty()) { - return ".*"; + if (shape.alternatives.size() == 1) { + const std::string padded = padded_alternative(pattern); + return padded.find(JSON_STRING_CHAR) == std::string::npos ? pattern : padded; + } + std::string out; + for (const auto& alternative : shape.alternatives) { + out += (out.empty() ? "(" : "|") + padded_alternative(alternative); } - return (starts ? "^(" : ".*(") + core + (ends ? ")$" : ").*"); + return out + ")"; } bool is_string_schema(const json& schema) { @@ -216,8 +285,10 @@ constexpr int64_t XGRAMMAR_REGEX_REPEAT_LIMIT = 128; /// \brief The pattern xgrammar is given for a string schema, which may take over its length bounds /// \note xgrammar drops minLength/maxLength once a pattern is there. A lone character /// class that only has to occur somewhere leads instead of being padded, so the -/// bounds become the repetition after it and leave the schema. Bounds too long for -/// that repetition keep the schema's minLength/maxLength and drop the pattern +/// bounds become the repetition after it and leave the schema. That is narrower +/// than JSON Schema, which lets the class occur anywhere ("\d" with bounds will not +/// write "Room 5"), but what it writes is always valid. Bounds too long for that +/// repetition keep the schema's minLength/maxLength and drop the pattern /// (std::nullopt): a string past its maxLength is the likelier mistake. std::optional string_pattern(json& schema) { const std::string pattern = schema["pattern"].get(); @@ -233,7 +304,8 @@ std::optional string_pattern(json& schema) { } const int64_t low = std::max(min_length.get(), 1) - 1; if (max_length.is_number_integer() && max_length.get() - 1 < low) { - return whole_string_pattern(pattern); // unsatisfiable, which xgrammar reports + // unsatisfiable; xgrammar ignores the bounds beside a pattern, so this holds to the pattern alone + return whole_string_pattern(pattern); } const bool long_repeat = max_length.is_null() ? low > XGRAMMAR_REGEX_REPEAT_LIMIT : max_length.get() - 1 > XGRAMMAR_REGEX_REPEAT_LIMIT; @@ -243,12 +315,14 @@ std::optional string_pattern(json& schema) { const std::string high = max_length.is_null() ? std::string() : std::to_string(max_length.get() - 1); schema.erase("minLength"); schema.erase("maxLength"); - return pattern + ".{" + std::to_string(low) + "," + high + "}"; + return pattern + JSON_STRING_CHAR + "{" + std::to_string(low) + "," + high + "}"; } /// \brief Spell out what a schema leaves implicit where xgrammar needs it /// \note An array without "items" means any items; the qwen_xml style rejects /// every value of such an array unless that is written as "items": {}. +/// Literal values (const, enum, default, examples) are not schemas, and a +/// pattern under "not" would change meaning if rewritten, so neither is entered. json normalize_schema(json schema) { if (schema.is_array()) { for (auto& item : schema) item = normalize_schema(std::move(item)); @@ -268,8 +342,17 @@ json normalize_schema(json schema) { schema.erase("pattern"); } } + static const std::set NOT_ENTERED = {"const", "enum", "default", "examples", "not"}; + // keyed by name, so a property called "default" is still a schema + static const std::set SCHEMA_MAPS = {"properties", "patternProperties", "$defs", "definitions", + "dependentSchemas"}; for (auto& [key, value] : schema.items()) { - if (value.is_object() || value.is_array()) { + if (NOT_ENTERED.count(key)) { + continue; + } + if (SCHEMA_MAPS.count(key) && value.is_object()) { + for (auto& [name, sub] : value.items()) sub = normalize_schema(std::move(sub)); + } else if (value.is_object() || value.is_array()) { value = normalize_schema(std::move(value)); } } diff --git a/src/test/chat_tools/test_chat_tools.cpp b/src/test/chat_tools/test_chat_tools.cpp index ed628416f..6eb28f1e5 100644 --- a/src/test/chat_tools/test_chat_tools.cpp +++ b/src/test/chat_tools/test_chat_tools.cpp @@ -601,6 +601,81 @@ void test_lfm2_string_length() { CHECK(!grammar.takes(note_request(), note("\"abcde\\\"f\""))); } +chat_constraint_request string_schema_request(const json& property) { + chat_constraint_request request; + request.allow_reasoning = false; + request.json_schema = {{"type", "object"}, {"properties", {{"v", property}}}, {"required", {"v"}}, + {"additionalProperties", false}}; + return request; +} + +void test_string_pattern() { + std::printf("a string pattern matches anywhere, as JSON Schema reads it\n"); + grammar_fixture grammar(HERMES); + auto takes = [&grammar](const json& property, const std::string& value) { + return grammar.takes(string_schema_request(property), "{\"v\": \"" + value + "\"}"); + }; + auto pattern = [](const std::string& p) { return json{{"type", "string"}, {"pattern", p}}; }; + CHECK(takes(pattern("\\S"), "Paris")); + CHECK(takes(pattern("\\S"), "a")); + CHECK(!takes(pattern("\\S"), " ")); + CHECK(takes(pattern("\\d{5}"), "zip 90210")); + // anchored at both ends: as sent + CHECK(takes(pattern("^\\d{4}$"), "2024")); + CHECK(!takes(pattern("^\\d{4}$"), "x2024")); + CHECK(takes(pattern("^ab"), "abc")); + CHECK(!takes(pattern("^ab"), "cab")); + CHECK(takes(pattern("^$"), "")); + CHECK(!takes(pattern("^$"), "a")); + // each alternative keeps its own anchor + CHECK(takes(pattern("^a|b"), "ax")); + CHECK(takes(pattern("^a|b"), "xbx")); + CHECK(!takes(pattern("^a|b"), "xa")); + // an anchor inside a group, which xgrammar would ignore: left as sent + CHECK(takes(pattern("(?:^abc)"), "abc")); + CHECK(!takes(pattern("(?:^abc)"), "xabc")); + // padding never ends the string: a non-ASCII pattern is compiled without xgrammar's + // JSON string guard, where ".*" took "x\", \"k\": \"y" and added a key + CHECK(takes(pattern("(\xC3\xA9|x)"), "ax")); + CHECK(!takes(pattern("(\xC3\xA9|x)"), "x\", \"k\": \"y")); + // a lone class folds its length bounds + const json bounded = {{"type", "string"}, {"minLength", 2}, {"maxLength", 5}, {"pattern", "\\S"}}; + CHECK(takes(bounded, "abcde")); + CHECK(!takes(bounded, "abcdef")); + CHECK(!takes(bounded, "a")); + // past xgrammar's repetition limit the bounds stay and the pattern goes, nested too + const json caption = {{"type", "string"}, {"minLength", 1}, {"maxLength", 512}, {"pattern", "\\S"}}; + chat_constraint_request nested = string_schema_request( + {{"type", "object"}, {"properties", {{"caption", caption}}}, {"required", {"caption"}}, {"additionalProperties", false}}); + CHECK(grammar.takes(nested, "{\"v\": {\"caption\": \"Monthly sales\"}}")); +} + +void test_string_pattern_inline_flag() { + std::printf("a pattern opening with an inline flag is left as sent\n"); + grammar_fixture grammar(HERMES); + // padded, it would read ".*((?i)abc).*", which xgrammar cannot build + CHECK(grammar.takes(string_schema_request({{"type", "string"}, {"pattern", "(?i)abc"}}), "{\"v\": \"abc\"}")); +} + +void test_schema_literals_untouched() { + std::printf("literal values and \"not\" are not rewritten as schemas\n"); + grammar_fixture grammar(HERMES); + // a property named like a keyword is still a schema + chat_constraint_request request; + request.allow_reasoning = false; + request.json_schema = {{"type", "object"}, + {"properties", {{"default", {{"type", "string"}, {"pattern", "\\S"}}}}}, + {"required", {"default"}}, + {"additionalProperties", false}}; + CHECK(grammar.takes(request, "{\"default\": \"Paris\"}")); + // an enum value shaped like a string schema is a value: it is matched as sent + request.json_schema = {{"type", "object"}, + {"properties", {{"v", {{"enum", {{{"type", "string"}, {"pattern", "\\S"}}}}}}}}, + {"required", {"v"}}, + {"additionalProperties", false}}; + CHECK(grammar.takes(request, "{\"v\": {\"type\":\"string\",\"pattern\":\"\\\\S\"}}")); +} + } // namespace int main() { @@ -616,6 +691,9 @@ int main() { test_gemma4_nullable_types(); test_gemma4_string_length(); test_lfm2_string_length(); + test_string_pattern(); + test_string_pattern_inline_flag(); + test_schema_literals_untouched(); if (failures) { std::printf("%d check(s) failed\n", failures); return 1; diff --git a/src/test/openai_api_test/test_response_format.py b/src/test/openai_api_test/test_response_format.py index fd8c7f13a..dd2533fb2 100644 --- a/src/test/openai_api_test/test_response_format.py +++ b/src/test/openai_api_test/test_response_format.py @@ -120,7 +120,7 @@ def schema_unanchored_pattern(api: Api) -> None: body = api.body(user("Which city is the capital of France? Answer as JSON."), response_format=json_schema_format("city", schema), max_tokens=64) _, value = nonstream_json(api, body, schema) - expect(len(value["city"]) > 1, f"the pattern \\S held the city to one character: {value['city']!r}") + expect("paris" in value["city"].lower(), f"the pattern \\S did not leave room for the city: {value['city']!r}") @case diff --git a/src/test/openai_api_test/test_tool_calls.py b/src/test/openai_api_test/test_tool_calls.py index a87a2b03c..258642377 100644 --- a/src/test/openai_api_test/test_tool_calls.py +++ b/src/test/openai_api_test/test_tool_calls.py @@ -176,7 +176,7 @@ def function_unanchored_pattern(api: Api) -> None: "type": "string", "minLength": 1, "maxLength": 64, "pattern": "\\S"} choice = api.ok(api.body(user(WEATHER_PROMPT), tools=[tool], tool_choice=function_choice("get_weather"))) city = json.loads(expect_calls(choice, [tool])[0]["function"]["arguments"])["city"] - expect(len(city) > 1, f"the pattern \\S held the city to one character: {city!r}") + expect("paris" in city.lower(), f"the pattern \\S did not leave room for the city: {city!r}") NOTE = ("The quarterly review moved to Thursday at three in the afternoon, " @@ -193,7 +193,7 @@ def function_pattern_keeps_length_bounds(api: Api) -> None: body = api.body(user(f"Save this note word for word: {NOTE}"), tools=[tool], tool_choice=function_choice("save_note")) text = json.loads(expect_calls(api.ok(body), [tool])[0]["function"]["arguments"])["text"] - expect(1 < len(text) <= 20, f"text {text!r} is outside 2..20 characters") + expect(0 < len(text) <= 20 and not text.isspace(), f"text {text!r} is not 1..20 characters with a non-blank one") @case