Skip to content

feat(core): ryzenai-corelib 0.103.0; a JSON Schema pattern matches anywhere in the string - #1047

Merged
tawei-amd merged 7 commits into
ROCm:mainfrom
chizamd:feat/corelib-0.87
Nov 18, 2026
Merged

tawei-amd merged 7 commits into
ROCm:mainfrom
chizamd:feat/corelib-0.87

Conversation

@chizamd

@chizamd chizamd commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #927. Moves the core backend from ryzenai-corelib 0.60 to 0.103.0 (through 0.70 and 0.87), fixes how a JSON Schema string pattern is turned into a grammar, and fixes two core test suites that #874 and #950 left failing on main. Seven commits on main at fbf2228d (the sixth by @tawei-amd); they merge cleanly with main at e450e495.

Release note: tool-call arguments and response_format strings now read a JSON Schema pattern as the spec does, matching anywhere in the string unless anchored: an unanchored "\d{5}" now means contains five digits, not is five digits. Anchor with ^...$ to match the whole string.

Commits

  1. feat(core): ryzenai-corelib 0.70.0 (corelib 39c30eab). npu_usable and the testing_npu_arch hook go; get_npu (none / aiem / aie2p) and npu_name replace them. The readiness check asks for aiem specifically: get_npu answers aie2p on Strix, where corelib needs an xclbin FastFlowLM never gives it, so "any NPU" would have offered the core backend there. The new model_metadata / session_options fields (draft tokens, audio token) are zero-initialised, so no draft is asked for.
  2. fix(chat): a JSON Schema pattern matches anywhere in the string. JSON Schema finds a pattern anywhere in the string, xgrammar matches it against the whole string, so {"type": "string", "minLength": 1, "pattern": "\\S"} ("not blank") allowed exactly one character. Seen with qwen3.6-moe-core:35b-a3b in an agent benchmark: every free-text argument of a forced tool call came back as one character, and the model looped until the tool-iteration limit. normalize_schema() now pads an unanchored side with .*, folds a lone character class's minLength/maxLength into the regex (xgrammar otherwise drops them), and keeps the length bounds and drops the pattern when that repetition would pass 128, which xgrammar cannot build inside a nested object (HTTP 400). response_format schemas now go through normalize_schema() too. Not core-specific: this applies to every model's tool calls and response_format.
  3. feat(core): ryzenai-corelib 0.87.0 (corelib 5bf19507). get_npu, npu_name and the NPU enum go; has_feature(npu_aiem) is the readiness check and feature_name the spelling NpuArch passes through (npu_aiem maps to dev04). Every other entry point and struct FastFlowLM resolves is unchanged. corelib's renamed 4-bit GGUF types still read the old type codes, so no catalog file changes; the new ryzenai_corelib_model_d3 library is copied by the existing glob and no tag uses it yet. The Linux setup guide's version check now prints 0, 87, 0.
  4. test(core): the frontend stubs return a full vocabulary of logits; the DFlash entries hold no licence. test_phi4_frontend and test_llama_frontend stubbed one logit, which Add stable model extension support #874's vocabulary check now refuses; they return vocab_size logits as the core backend does. test_model_downloader still expected the licence files model list: drop license and repo-doc files from model downloads #950 dropped from the two DFlash entries. Tests only.
  5. feat(core): ryzenai-corelib 0.103.0 (corelib 49a8ed3c). Every entry point FastFlowLM resolves keeps its signature and model_metadata its layout, so the only code change is the required version. In model_session_options, lora is gone and low_memory / kv_cache_bits are appended; FastFlowLM value-initialises it and sets only max_sequence_length, so the defaults hold (no LoRA, a BF16 KV cache). The new model libraries (qwen2, lfm2, lfm3, hyv3, glide, minicpm_ocr, minicpmv47) are copied by the existing glob; no -core tag uses them yet. The 8-bit KV cache and low_memory sessions are opt-in and not taken here.
  6. fix(chat): Gemma4 and LFM2 tool grammars keep a string's minLength/maxLength (@tawei-amd). These two formats write their argument grammar in FastFlowLM rather than through xgrammar's JSON Schema, and gave every string the unbounded rule, so function_pattern_keeps_length_bounds [gemma4-it:e2b] failed on both XRT jobs. They now read the schema as sent (tool_def.declared) and spell bounds up to 128: counted rules for Gemma4, a {low,high} repetition for LFM2.
  7. fix(chat): pad an unanchored pattern without ending the string; leave flags and inner anchors as sent. Review follow-ups to commit 2: the padding is [^"\\\x00-\x1f]* rather than .* (xgrammar drops its JSON string guard for a non-ASCII pattern, where .* could close the string and add keys); a pattern opening with (?i)-style flags or holding an anchor inside a group is left as sent; each top-level alternative is padded on its own (^a|b); normalize_schema no longer enters const / enum / default / examples / not. The live tests check for the expected word rather than a length above one.

Only the core backend (FLM_ENABLE_CORE), the chat grammar and tests change; no FastFlowLM (aie2p) engine or catalog entry is touched.

Testing

Built on Windows with the fake corelib: against the 0.103.0 headers at the head of this branch, core_model 4/4, gemma4_core 3/3, llama_core 1/1, model_backend 1/1, model_list_platform 1/1, phi4_core 5/5; against the 0.87.0 headers at commit 4, the same plus chat_tools 1/1; at the head, chat_tools passes with the new pattern and string-length checks, which on the code before commits 6 and 7 fail or crash the harness (test_real_corelib and test_gemma4_tokenizer skip without an NPU and a GGUF).

On an AIE4 (Medusa) test machine, core flm.exe linked against corelib's default build, flm serve per tag with RYZENAI_CORELIB_REPORT_FALLBACKS=1, multi-turn capped at 2048 tokens. At 0.103.0 and at 0.87.0 every tag loads, passes has_feature(npu_aiem), reports no fallback, and answers 2+2, a one-sentence prompt and a two-turn recall. Decode tok/s (the 2+2 answer / the one-sentence answer):

tag 0.103 0.87 0.70
qwen3-core:0.6b (Q8_0) 140 / 140 137 / 135 118 / 124
qwen3-core:1.7b 100 / 85 85 / 83 84 / 77
qwen3-core:4b 43 / 43 43 / 43 42 / 42
qwen3-core:8b 26 / 26 26 / 26 26 / 26
phi4-mini-it-core:4b 60 / 51 63 / 51 75 / 49
qwen3.5-core:4b 19 / 38 19 / 37 19 / 36
qwen3.5-core:9b 12 / 23 11 / 22 11 / 22
llama3.2-core:1b (Q8_0) 261 / 140 260 / 141 230 / 138

At 0.103 the first Phi-4-mini run answered 2+2 with "2" (at its default temperature 0.8); two reruns answered 4, and the table shows those. At 0.87 Qwen3-0.6B's second turn mentioned the number without saying it was the user's favourite. Phi-4-mini's 2+2 rate is over a 2-token answer.

At 0.87, Gemma 4, Qwen3.8 and Qwen3.6, pulled by this flm.exe, recall a code placed in the middle of a long prompt (not rerun at 0.103: their GGUFs were no longer on the test machine):

tag 4k: prefill tok/s 12k 24k short decode tok/s
gemma4-it-core:e2b 2385 960 not run 78 / 46
gemma4-it-core:e4b 1439 1062 not run 49 / 27
qwen3.8-core:27b 362 340 not run 13 / 8.5
qwen3.6-moe-core:35b-a3b 1300 1264 1033 25 / 48

The chat fix was run live on the same machine with qwen3.6-moe-core:35b-a3b: test_tool_calls.py 18/18 and test_response_format.py 12/12, including the three new cases (function_unanchored_pattern, function_nested_long_bounded_pattern, schema_unanchored_pattern); the two unanchored cases fail on main.

Not run: Gemma 4, Qwen3.8 and Qwen3.6 at 0.103, Llama 3.2-3B / 3.1-8B on hardware, and the AIE2P refusal on a Strix machine.

NPU direct, not turned on

At 0.87, a core flm.exe linked against a corelib built with --npu-direct runs Qwen3.6's attention NPU direct from 12k keys, with the same greedy output and prefill about 10% faster at 12k and 15% at 24k. The core build stays on corelib's default build, since --npu-direct needs CHESS in WSL; turning it on later is a build-only change.

chizamd and others added 4 commits November 14, 2026 16:18
…esting_npu_arch

Moves the core backend to ryzenai-corelib 0.70.0 (corelib main at
39c30eab). What changed in corelib 0.61 to 0.70 that reaches FastFlowLM:

- 0.70, NPU query: ryzenai_corelib_npu_usable and the testing hook
  ryzenai_corelib_testing_npu_arch are gone. ryzenai_corelib_get_npu
  (none / aiem / aie2p) and ryzenai_corelib_npu_name replace them, so
  corelib_api.hpp no longer includes corelib/testing.h.
- The runtime's readiness check now asks for aiem. The old npu_usable was
  false on AIE2P, whose kernels corelib runs only behind an xclbin
  FastFlowLM never gives it; get_npu now answers aie2p there, so checking
  for "any NPU" would have offered the core backend on Strix.
- NpuArch names the enum with npu_name, "" for none, as before, so dev04 /
  aie2p platform detection is unchanged.
- 0.61 / 0.68, model structs: model_metadata gains max_draft_tokens and
  audio_token_id; session_options gains draft_tokens. FastFlowLM
  zero-initialises both at their full sizeof, so no draft is asked for.
- 0.62 / 0.67, matmul: the 4-bit layout is mladf_v4 (was mladf) beside
  AIE2P's mladf_v2, and matmul_bf16_kernel_info gains format. FastFlowLM
  names neither; the fake corelib fills format.
- 0.68: ssmlp_bf16_weights_desc gains ungated; unused here.
- Model libraries and their names are unchanged. The weights loader's
  switch from bad_argument to unsupported for a wrong tensor type does not
  matter here: FastFlowLM does not branch on corelib's status codes.

The fake corelib answers get_npu from one enum and npu_name as corelib
spells it; the tests refuse none and aie2p, and read "" for none.

On an AIE4 (Medusa) test machine, qwen3-core 0.6b/1.7b/4b/8b,
phi4-mini-it-core:4b, qwen3.5-core 4b/9b and llama3.2-core:1b each load,
pass the readiness check through get_npu, report no fallback and answer
correctly, at decode rates within a few percent of corelib 0.59's.

Co-authored-by: Cursor <cursoragent@cursor.com>
…keeps a one-character class's length bounds

JSON Schema finds a string pattern anywhere in the string; xgrammar
matches the regex against the whole string. A field written as

    {"type": "string", "minLength": 1, "maxLength": 128, "pattern": "\\S"}

("not blank") therefore allowed exactly one character, and xgrammar also
drops minLength/maxLength whenever a pattern is present (it logs "pattern
combined with minLength/maxLength is not supported").

Found running an agent benchmark with qwen3.6-moe-core:35b-a3b: every
free-text argument of a forced chart-generation tool call came back as one
character ("caption":"7","alt_text":"A","labels":["P","C"]). The caller
rejected it, the grammar left the model no other call to make, and it
repeated until the tool-iteration limit.

normalize_schema() now:

- pads the side(s) a string pattern leaves unanchored with ".*" ("\S"
  becomes ".*(\S).*"). A pattern anchored at both ends is left as it is,
  so "^\d{4}-\d{2}-\d{2}$" keeps its meaning.
- lets a lone character class that only has to occur somewhere ("\S",
  "\d", "[a-z]", ...) lead, and turns the schema's length bounds into the
  repetition after it: "\S" with maxLength 128 becomes "\S.{0,127}", so
  the bounds xgrammar would drop are enforced.
- keeps minLength/maxLength and drops the pattern instead when that
  repetition would be longer than 128. xgrammar compiles a longer regex
  repetition into a rule of its own and cannot build that rule inside a
  nested object ("cannot build the decoding grammar: Check failed:
  per_rule_fsm_hashes[...].has_value()"), so the request would fail with
  HTTP 400; a string past its maxLength is the likelier mistake.

response_format schemas now go through normalize_schema() as tool
parameters already did, so they get the same reading.

New live cases in src/test/openai_api_test: test_tool_calls
function_unanchored_pattern and function_nested_long_bounded_pattern, and
test_response_format schema_unanchored_pattern. The two unanchored cases
fail on main ('P' for "Paris"); the nested one fails with the HTTP 400
above if the bounds are folded at any length. On an AIE4 (Medusa) test
machine with qwen3.6-moe-core:35b-a3b:
test_tool_calls.py 18/18, test_response_format.py 12/12.

Co-authored-by: Cursor <cursoragent@cursor.com>
…npu_name

Moves the core backend to ryzenai-corelib 0.87.0 (corelib main at
5bf19507). What changed in corelib 0.71 to 0.87 that reaches FastFlowLM:

- 0.73, NPU query: ryzenai_corelib_get_npu, ryzenai_corelib_npu_name and
  the ryzenai_corelib_npu enum are gone. ryzenai_corelib_has_feature and
  ryzenai_corelib_feature_name replace them, over ryzenai_corelib_feature
  (npu, npu_aiem, npu_aie2p, gpu, npu_direct).
- The runtime's readiness check asks has_feature(npu_aiem). AIE2P is
  still refused, since FastFlowLM never gives corelib an xclbin, and the
  error names npu_aie2p or none.
- NpuArch passes corelib's spelling through (npu_aiem, npu_aie2p, ""), and
  platform_from_corelib_arch maps npu_aiem to dev04. FastFlowLM still
  keeps no table of devices.
- Everything else FastFlowLM resolves is unchanged: every other entry
  point in FLM_CORELIB_FUNCTIONS keeps its signature, and model_metadata
  and session_options keep their layout. 0.87's single PDI list per
  stream, NPU direct, the .amd.gguf exports (0.80, 0.83) and the new ops do
  not touch FastFlowLM's calls.
- 0.83: corelib's 4-bit GGUF types are renamed (AMD_Q4_G128, AMD_Q4_G32,
  new AMD_Q4_G64). Files published under the old type codes (ggml 128,
  129) are still read as before, so the catalog's AMD files need no
  change.
- Model libraries: ryzenai_corelib_model_d3 is new. The core build copies
  ryzenai_corelib_model_*.dll by glob, so it lands beside flm.exe without
  a change; no -core tag uses it yet.
- Not taken here: corelib now lets Llama 3.2-1B/3B and 3.1-8B read a
  Q4_G128 GGUF in place of their Q8_0. Q8_0 still loads, so the catalog's
  Llama entries stay on Q8_0; switching them is a separate change.

The fake corelib answers has_feature from one NPU feature, or none, and
feature_name as corelib spells it. The tests refuse none and npu_aie2p,
each named in the error, and read "" for none. The Linux setup guide's
version check now prints 0, 87, 0.

On an AIE4 (Medusa) test machine, with the core flm.exe statically linked
against corelib's default build (no --npu-direct): every -core tag except
Llama 3B/8B loads, passes the readiness check through has_feature(npu_aiem),
reports no fallback and answers correctly, at decode rates matching
0.70's. Gemma 4 E2B/E4B, Qwen3.8 and Qwen3.6 recall a code from the middle
of 4k and 12k prompts (Qwen3.6 also at 24k).

An flm.exe linked against a corelib built with --npu-direct engages NPU
direct for Qwen3.6 and gives the same greedy output, with prefill about
10-15% faster from 12k tokens. The core build stays on corelib's default
build: --npu-direct needs CHESS in WSL, so turning it on is a build-only
follow-up.

Co-authored-by: Cursor <cursoragent@cursor.com>
…e DFlash entries hold no licence

Two checks added on main left two Windows-only core suites failing on
main itself. Only tests change here.

- test_phi4_frontend, test_llama_frontend: ROCm#874 added a check to
  AutoModel::_sample that refuses logits shorter than the sampler's
  in_features, which is vocab_size from config.json. Each test's
  FakeEngine returned buffer<bf16>(1) from forward and prefill, so every
  test that samples failed with "Backend returned fewer logits than the
  frontend vocabulary". They now return kVocabSize logits, as the core
  backend does: 200064 for Phi-4 and 128256 for Llama. config.json takes
  vocab_size from the same constant.
- test_model_downloader: ROCm#950 dropped LICENSE.OpenMDW-1.1 and LICENSE from
  the files of nemotron3.5-bolt-dflash:e4b and qwen3.8-dflash2:27b, but
  the test still expected them. It now expects config.json and
  dflash_weight.q4nx, as the catalog lists.

The check itself is unchanged. The real core backend returns logits as
wide as corelib's vocab_size, which equals config.json's for the shipped
-core tags, so it does not trip there.

Co-authored-by: Cursor <cursoragent@cursor.com>
Moves the core backend to ryzenai-corelib 0.103.0 (corelib main at
49a8ed3c). What changed in corelib 0.88 to 0.103 that reaches FastFlowLM:

- Every entry point in FLM_CORELIB_FUNCTIONS keeps its signature, and
  model_metadata keeps its layout, so the only code change is the
  required version in corelib_api.hpp.
- model_session_options: lora is gone (0.97: a session runs LoRA kernels
  when its model has an adapter set), and low_memory (0.96) and
  kv_cache_bits (0.98) are appended. FastFlowLM value-initialises the
  struct and sets only max_sequence_length, so the defaults hold: no LoRA,
  the embedding table mapped, a BF16 KV cache.
- Additive only, as far as FastFlowLM's calls go: uint8 tensors,
  stream_set_low_memory, PQ2_0 weights, head_gate_bf16, hadamard_bf16, and
  NPU direct on AIE2P. FastFlowLM still refuses AIE2P.
- Model libraries: qwen2, lfm2, lfm3, hyv3, glide, minicpm_ocr and
  minicpmv47 are new. The core build copies ryzenai_corelib_model_*.dll by
  glob, so they land beside flm.exe without a change; no -core tag uses
  them yet.
- Not taken here: the 8-bit KV cache (kv_cache_bits 8, AIE4 NPU direct
  only) and low_memory sessions. Both are opt-in and would be separate
  changes.

The docs' version checks and the required-version notes move to 0.103.

On an AIE4 (Medusa) test machine, qwen3-core 0.6b/1.7b/4b/8b,
phi4-mini-it-core:4b, qwen3.5-core 4b/9b and llama3.2-core:1b each load,
report no fallback and answer correctly, at decode rates matching 0.87's.

Co-authored-by: Cursor <cursoragent@cursor.com>
@chizamd chizamd changed the title feat(core): ryzenai-corelib 0.87.0; a JSON Schema pattern matches anywhere in the string feat(core): ryzenai-corelib 0.103.0; a JSON Schema pattern matches anywhere in the string Nov 17, 2026

@zaneni6 zaneni6 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the grammar change and the CI history. Summary:

CI: the XRT failure is deterministic and comes from the new test, not flakiness. api/test_tool_calls.function_pattern_keeps_length_bounds [gemma4-it:e2b] fails on Test Linux XRT portable and Test Windows XRT in every attempt (run 43657612538 attempts 1–4, run 43985532394 attempt 1). The model copies the whole 113-char note into a field with maxLength: 20. Gemma 4 is the only model in the XRT suite that is shown tools (qwen3:0.6b skips them), and the HRX suites don't run test_tool_calls, so this never showed up in the PR's own testing (qwen3.6-moe-core / qwen3.5-oga-hipep both go through xgrammar).

Root cause: GEMMA4 (and LFM2 pythonic) tool calls don't use xgrammar's JSON Schema converter. tool_tags() → gemma4_arguments_grammar() → property_value() / value_rule() maps every type: string to the hand-written unbounded rule string ::= "<|\"|>" ([^<] | "<" [^|])* "<|\"|>" (chat_grammar.cpp ~L337, ~L445), ignoring pattern, minLength and maxLength, on main as well as here. So for Gemma 4, function_unanchored_pattern and function_nested_long_bounded_pattern also pass only because nothing is constrained. schema_unanchored_pattern does go through xgrammar via response_format, so that pass is real. Options: skip/relax the bounds test for the GEMMA4 / LFM2 formats, or emit a bounded repeat (([^<] | "<" [^|]){min,max}) in that string rule when the schema has length bounds.

The other failures look like infrastructure, not this PR:

  • First-round Windows HRX on xconucstrhl102: ERT state 5 / hrx_buffer_allocate failed. HRX doesn't run the tool tests, and it passed on re-run, though on a different runner.
  • Attempt 2 Windows XRT on xconucstrhl101: flm.exe exit 0xC0000139, a DLL entry point not found.
  • Latest attempt 1 Windows XRT on xsjnuckrck01: the runner lost communication.

XRT / HRX AIE2 impact:

  • Build: none. The corelib changes are under FLM_ENABLE_CORE, and npu_platform.hpp only renames the label the core backend matches (AIEM → npu_aiem).
  • Runtime: only through chat_grammar.cpp, which is shared CPU code. It changes tool calls and response_format with string pattern on every model. Nothing touches the NPU runtime.

The grammar fix itself is correct for the xgrammar paths (Hermes JSON, qwen_xml, response_format). I checked the helpers against the repo's xgrammar in a scratch harness. The both-anchored pass-through, odd/even \ before $, []] / [^]], and the 128 cutoff (it matches kLargeRepeatThreshold, no off-by-one) all behave as intended. Some edge cases are in the inline comments.

Reviewed with the help of Claude Code.

Comment thread src/common/chat/chat_grammar.cpp Outdated
if (core.empty()) {
return ".*";
}
return (starts ? "^(" : ".*(") + core + (ends ? ")$" : ").*");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Edge cases of the .* padding, found in a scratch harness against the bundled xgrammar:

  • Non-ASCII pattern (most serious): xgrammar compiles it through RegexToEBNF without json_string (json_schema_converter.cc ~L2271), where . matches ", \ and newline. So "é" becomes .*(é).*, and the grammar accepts {"city": "café", "x": "y"}, an extra key injected into the JSON. Before this PR, that pattern only allowed one character.
  • qwen_xml: the top-level patterned string is built with force_cfg_expansion=true and no excludes, so a padded value can contain </parameter>. Found in the code, not reproduced.
  • (?i)abc now fails: it becomes .*((?i)abc).*, which xgrammar refuses to build, so the request now gets a 400. It used to work.
  • Anchor inside a group: (?:^abc) or (a$) isn't detected, so the pattern gets padded. xgrammar ignores mid-regex anchors, so xabc is accepted.
  • Top-level alternation with one anchor: ^a|b becomes ^(a|b).*, which is stricter than ECMA (it was also wrong before).

The non-ASCII one probably needs a guard, e.g. skip padding or pad with [^"\\]* when the pattern isn't printable ASCII.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, all confirmed against the bundled xgrammar and addressed in 8268970:

  • Non-ASCII: the padding is now [^"\\\x00-\x1f]* (what a JSON string holds unescaped) instead of .*, and the folded repetition after a lone class uses the same class. On the previous code the new test_string_pattern check reproduces the injection ((é|x) took x", "k": "y); it is rejected now.
  • (?i)abc: a pattern opening with an inline flag is left as sent. In the unit harness the padded form did not just fail to build, it crashed the process.
  • Anchor inside a group: a ^ or $ anywhere but at an edge (outside classes, escapes and groups) leaves the pattern as sent, so (?:^abc) rejects xabc again.
  • Top-level alternation: each alternative is padded on its own, so ^a|b now means "starts with a, or holds b".
  • qwen_xml: left as is. A patterned top-level value would need the model to write </parameter> inside it, and the parser ends the value there as it does for any string; I'd rather not special-case it without a repro.

Each case has a check in src/test/chat_tools.

Comment thread src/common/chat/chat_grammar.cpp Outdated
}
const int64_t low = std::max<int64_t>(min_length.get<int64_t>(), 1) - 1;
if (max_length.is_number_integer() && max_length.get<int64_t>() - 1 < low) {
return whole_string_pattern(pattern); // unsatisfiable, which xgrammar reports

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: for minLength: 0, maxLength: 0 this branch keeps the bounds, but xgrammar ignores bounds once a pattern is present and only errors when min > max. So it doesn't report the schema as unsatisfiable; it generates e.g. "x". The comment is misleading.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, the comment was wrong. Reworded in 8268970: unsatisfiable bounds fall through to the pattern alone, since xgrammar ignores bounds beside a pattern.

return std::nullopt;
}
const std::string high = max_length.is_null() ? std::string() : std::to_string(max_length.get<int64_t>() - 1);
schema.erase("minLength");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest keeping minLength / maxLength here rather than erasing them. xgrammar ignores them once a pattern is present, so the xgrammar paths behave the same. But the hand-written GEMMA4 / LFM2 argument grammars (property_value / value_rule) would then still see the bounds if they learn to apply them. That is the fix the failing function_pattern_keeps_length_bounds [gemma4-it:e2b] needs.

Also worth a comment: the folded form forces the class to the front, so " Paris" or \d + bounds with "Room 5" can no longer be produced. That's narrower than JSON Schema but always valid.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed differently, via @tawei-amd's 3d4a048: the Gemma4/LFM2 grammars now read tool_def.declared, the schema as sent, so they see minLength/maxLength without normalize_schema keeping them (which would also bring back xgrammar's pattern combined with minLength/maxLength warning). The narrowing is now noted in the comment on string_pattern (8268970).

json response_schema(const chat_constraint_request& request) {
if (!request.json_schema.is_null()) {
return request.json_schema;
return normalize_schema(request.json_schema);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this widens the behaviour change: every response_format schema now gets the pattern rewrite, including the edge cases above. Users will also see that an unanchored "\d{5}" changes from exactly five digits to contains five digits. That's spec-correct, but worth a line in the release notes.

normalize_schema also recurses into const / enum / default / examples / not values. Rewriting a pattern inside not changes its meaning. Low risk, but cheap to exclude those keys.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8268970 stops normalize_schema from entering const, enum, default, examples and not; maps keyed by name (properties, patternProperties, $defs, definitions, dependentSchemas) still recurse, so a property called default is still a schema. Both have checks in test_chat_tools. I've added the release-note line about unanchored patterns (\d{5} now means contains five digits) to the PR description.

body = api.body(user(f"Save this note word for word: {NOTE}"), tools=[tool],
tool_choice=function_choice("save_note"))
text = json.loads(expect_calls(api.ok(body), [tool])[0]["function"]["arguments"])["text"]
expect(1 < len(text) <= 20, f"text {text!r} is outside 2..20 characters")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fails deterministically on both XRT jobs for gemma4-it:e2b, because the GEMMA4 tool-argument grammar never applies pattern / minLength / maxLength (see the review summary). It needs a skip for the GEMMA4 / LFM2 formats, or a grammar fix there.

Separately, 1 < len(text) disagrees with the schema's minLength: 1: a valid one-character answer would fail. The same goes for the len(...) > 1 checks in function_unanchored_pattern and schema_unanchored_pattern. Those are trying to catch the old one-character bug, so maybe check against the expected word (e.g. contains "Paris") instead.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Gemma4 failure is fixed by 3d4a048 (the Gemma4/LFM2 grammars apply the bounds). The assertions are relaxed in 8268970: the two unanchored cases check the answer contains Paris, and the bounds case checks 1..20 characters with a non-blank one.

@tawei-amd

Copy link
Copy Markdown
Collaborator

The two NPU test failures here (Test Linux XRT portable, Test Windows XRT) both come from one case: test_tool_calls.function_pattern_keeps_length_bounds [gemma4-it:e2b], where the model saved a 115-character note against maxLength: 20.

Root cause: Gemma4 (and LFM2) tool arguments aren't constrained through xgrammar's JSON-schema path. FLM writes their EBNF itself (gemma4_arguments_grammar / python_calls_grammar in chat_grammar.cpp), and every type: "string" property got the unbounded string rule. So string_pattern()'s \S.{0,19} rewrite never reaches gemma4. normalize_schema also strips minLength/maxLength from the schema those grammars read. qwen3:0.6b skips tool tests, so gemma4 is the only CI model that runs this case.

Fix: branch fix/gemma4-string-bounds, one commit (33c8e225) on top of this PR's head a7a0134a:

  • The EBNF grammars read the tool schema as sent (tool_def.declared) and honour minLength/maxLength up to XGRAMMAR_REGEX_REPEAT_LIMIT.
  • Gemma4: counted gstr_<low>_<high> rules. Lengths are exact even with < in the text, and <| still can't occur inside a value.
  • LFM2: a {low,high} repetition, where an escape counts as one character. Array items keep their bounds too.
  • New gemma4/LFM2 length cases in src/test/chat_tools.

Tested locally (Linux, Strix NPU):

  • test_chat_tools: all checks pass. On this PR's head without the commit, 7 of the new checks fail.
  • gemma4-it:e2b run_all.py: 120 passed, 0 failed, 2 skipped.
  • lfm2.5-tk:1.2b, with vs. without the commit: the only difference is function_pattern_keeps_length_bounds going FAIL → PASS. The 6 other lfm2.5-tk failures (</think> leaks, running out of max_tokens) are the same both ways, and that model isn't in CI.
  • Windows not run locally.

@chizamd could you cherry-pick it onto feat/corelib-0.87?

git fetch https://github.com/ROCm/FastFlowLM.git fix/gemma4-string-bounds
git cherry-pick 33c8e225

Or if you'd rather I push it to your branch (maintainer edits are on), just say so.

🤖 Generated with Claude Code

tawei-amd and others added 2 commits November 17, 2026 14:30
…xLength

The argument grammars FastFlowLM writes itself (Gemma4, LFM2) gave every
string property the unbounded string rule, so a tool parameter with
maxLength 20 took a whole sentence: test_tool_calls
function_pattern_keeps_length_bounds failed on gemma4-it:e2b, Linux and
Windows. normalize_schema had also folded the bounds into a pattern that
only xgrammar's JSON schema reads.

These grammars now read the schema as sent and spell bounds up to
XGRAMMAR_REGEX_REPEAT_LIMIT: Gemma4 through counted gstr_<low>_<high>
rules (a "<" step takes the next character too, so lengths stay exact
and "<|" still cannot occur), LFM2 through a {low,high} repetition where
an escape is one character. Array items keep their bounds too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… flags and inner anchors as sent

Review follow-ups to the pattern rewrite:

- The padding is now a class of what a JSON string holds unescaped, not
  ".*". xgrammar keeps '"', '\' and control characters out of a pattern's
  matches only when the pattern is printable ASCII; any other pattern
  compiles without that guard, where ".*" could close the string and add
  keys ("(é|x)" took x", "k": "y). The folded repetition after a lone
  class uses the same class.
- A pattern opening with an inline flag such as "(?i)" is left as sent:
  padded, it no longer builds.
- A pattern with an anchor inside a group, such as "(?:^abc)", is left as
  sent: xgrammar ignores an anchor in the middle, so padding would have
  let "xabc" through.
- Each top-level alternative is padded on its own, so "^a|b" reads as
  "starts with a, or holds b" rather than "^(a|b).*".
- normalize_schema no longer enters const, enum, default or examples,
  which are values, nor "not", whose meaning a rewrite would change. A
  property named like one of them is still a schema.
- The comment on unsatisfiable bounds now says what happens: xgrammar
  ignores the bounds beside a pattern. The folded form's narrowing is
  noted.

The live tests check for the expected word instead of a length above
one, which a valid one-character answer would have failed.

New checks in test_chat_tools cover each case through the grammar engine;
on the previous code "^a|b" and the non-ASCII injection fail, and the
inline-flag and inner-anchor patterns crash the harness.

Co-authored-by: Cursor <cursoragent@cursor.com>
@chizamd
chizamd deployed to npu-approval November 17, 2026 22:40 — with GitHub Actions Active
@chizamd

chizamd commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

@tawei-amd thanks, cherry-picked as 3d4a048 (authorship kept). I read through it and it looks right to me; one note for the record: a bounded Gemma4 string can no longer hold <<, since a < step takes a character that is neither < nor |. That's stricter than the unbounded rule, which could spell <<|, so I left it.

8268970 on top addresses @zaneni6's inline comments (padding that cannot end the string, inline flags, inner anchors, alternation, literal keywords, test assertions); the replies are on each thread.

@tawei-amd
tawei-amd merged commit 532081f into ROCm:main Nov 18, 2026
73 checks passed

This branch was successfully deployed

1 active deployment
npu-approval — 8268970e Deployed Nov 17, 2026 by chizamd via npu-gate #451
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants