diff --git a/docs/core-setup-linux.md b/docs/core-setup-linux.md index 8d9b64263..7bf6d2ac8 100644 --- a/docs/core-setup-linux.md +++ b/docs/core-setup-linux.md @@ -9,7 +9,7 @@ Written against the working install on this machine. Paths use `$P` for the dependency prefix (here `/home/alfxu/ddprefix`); substitute your own. > **Status:** the core backend has been run on hardware on Windows only. The -> steps below are updated for corelib 0.60.0, but they have not been run end +> steps below are updated for corelib 0.103.0, but they have not been run end > to end on Linux at this revision. The [Known issues](#known-issues) are from > an earlier Linux bring-up. @@ -52,14 +52,14 @@ loaded from, so those libraries must sit beside it; nothing else is needed at run time, no kernel store and no environment variable. **Version must match exactly.** `src/include/core/corelib_api.hpp` has a -`#error` for anything other than corelib **0.60.0**: +`#error` for anything other than corelib **0.103.0**: ```bash grep -E 'CORELIB_VERSION_(MAJOR|MINOR|PATCH)' \ $P/install-corelib/include/ryzenai/corelib/base.h ``` -Must print `0`, `60`, `0`. (`BUILD-NOTES.md` still says 0.4.0 in its opening +Must print `0`, `103`, `0`. (`BUILD-NOTES.md` still says 0.4.0 in its opening line — that text is stale.) ### Verify corelib before going further diff --git a/docs/docs/models/phi.md b/docs/docs/models/phi.md index 9fbba3dcf..d97c5a174 100644 --- a/docs/docs/models/phi.md +++ b/docs/docs/models/phi.md @@ -75,9 +75,9 @@ The `src/test/phi4_core` suite is still Windows-only and is not configured on Li ### Pointing FastFlowLM at the runtime -A core build (`-DFLM_ENABLE_CORE=ON`) **links corelib in**. corelib keeps its NPU device to itself and opens a hardware context per stream on it; FastFlowLM opens its own device for every other family, and a core model shares no buffer with those. The core backend is offered only where corelib reports a usable NPU (`ryzenai_corelib_npu_usable`): elsewhere corelib would run a session on the GPU or the host. Point the build at the library with `RYZENAI_CORELIB_INCLUDE_DIR` and `RYZENAI_CORELIB_LIBRARY`. `FLM_CORE_CORELIB_PATH` selects a shared library (`.dll` on Windows, `.so` elsewhere) only in the older dynamically loading configuration; in a statically linked core build it is ignored, and `flm` says so if it is set. +A core build (`-DFLM_ENABLE_CORE=ON`) **links corelib in**. corelib keeps its NPU device to itself and opens a hardware context per stream on it; FastFlowLM opens its own device for every other family, and a core model shares no buffer with those. The core backend is offered only where corelib reports an AIEM NPU (`ryzenai_corelib_has_feature(ryzenai_corelib_feature_npu_aiem)`): elsewhere, AIE2P included, corelib would run a session on the GPU or the host. Point the build at the library with `RYZENAI_CORELIB_INCLUDE_DIR` and `RYZENAI_CORELIB_LIBRARY`. `FLM_CORE_CORELIB_PATH` selects a shared library (`.dll` on Windows, `.so` elsewhere) only in the older dynamically loading configuration; in a statically linked core build it is ignored, and `flm` says so if it is set. -The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.60.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build installs `ryzenai_corelib` plus one `ryzenai_corelib_model_` library per model, each carrying that model's kernels; corelib loads them only from its own directory, so they must sit beside it (a Windows core build copies them next to `flm.exe`). Each engine enables its model's kernel bundle on the stream it opens; no kernel store or environment variable is involved. Corelib's own dependency directory must be reachable on `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux). +The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.103.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build installs `ryzenai_corelib` plus one `ryzenai_corelib_model_` library per model, each carrying that model's kernels; corelib loads them only from its own directory, so they must sit beside it (a Windows core build copies them next to `flm.exe`). Each engine enables its model's kernel bundle on the stream it opens; no kernel store or environment variable is involved. Corelib's own dependency directory must be reachable on `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux). ```powershell flm pull phi4-mini-it-core:4b diff --git a/docs/docs/models/qwen.md b/docs/docs/models/qwen.md index 22fd08256..84731809e 100644 --- a/docs/docs/models/qwen.md +++ b/docs/docs/models/qwen.md @@ -111,7 +111,7 @@ flm run qwen3:8b - **Backend:** `core` — one engine for all four sizes. The GGUF's own metadata selects the size and is cross-checked against it. - **Source format:** GGUF, read directly by corelib. With a Hugging Face token that can read AMD's private repositories (`HF_TOKEN`, or `hf auth login`), `flm pull` fetches AMD's 4-bit file, which packs exactly as it is; without one, a tag below on an AMD file cannot be pulled, as corelib's own runner has no public fallback either. Only Qwen3-0.6B and Llama, which AMD ships no 4-bit file of, pull a public `Q8_0`, requantized to 4-bit groups on every load, which is lossy; corelib refuses `Q8_0` for Qwen3.5, Qwen3.6, Gemma 4 and Phi-4-mini. - **Context:** each tag's limit is the most corelib runs that model at; the default is lower where corelib or our own runs found answers going wrong past it. Over-capacity requests are rejected with HTTP 400 before any work reaches the device. -- **Availability:** developer build against corelib 0.60.0 — see [Phi-4 on the core backend](phi.md). +- **Availability:** developer build against corelib 0.103.0 — see [Phi-4 on the core backend](phi.md). The same corelib path runs every `-core` tag: diff --git a/src/common/chat/chat_grammar.cpp b/src/common/chat/chat_grammar.cpp index 188908de0..a7b2e66da 100644 --- a/src/common/chat/chat_grammar.cpp +++ b/src/common/chat/chat_grammar.cpp @@ -4,9 +4,11 @@ #include "chat/chat_grammar.hpp" #include +#include #include #include #include +#include #include #include #include @@ -158,9 +160,169 @@ json qwen_xml_arguments(const json& schema) { // ---------------------------------------------------------------- tools of a request +/// \brief Whether a regex is one character class: ".", "\S", "\d", "[a-z]", ... +bool is_one_character(const std::string& pattern) { + if (pattern == ".") { + return true; + } + if (pattern.size() == 2 && pattern[0] == '\\') { + return pattern[1] != 'b' && pattern[1] != 'B'; // word boundaries match no character + } + if (pattern.size() < 3 || pattern.front() != '[' || pattern.back() != ']') { + return false; + } + for (size_t i = 1; i + 1 < pattern.size(); ++i) { + if (pattern[i] == '\\') { + ++i; + } else if (pattern[i] == ']') { + return false; + } + } + return true; +} + +/// \brief Any character a JSON string holds unescaped, as a regex class +/// \note Not ".": xgrammar keeps '"', '\\' and control characters out of a pattern's +/// matches only when the pattern is printable ASCII. Any other pattern is +/// compiled without that, where "." can close the string and add keys. +const std::string JSON_STRING_CHAR = "[^\"\\\\\\x00-\\x1f]"; + +/// \brief Where a regex splits into alternatives and holds anchors, outside groups and classes +struct regex_shape { + std::vector alternatives; // split at each top-level "|" + bool inner_anchor = false; // a "^" or "$" anywhere but at either end +}; + +regex_shape scan_regex(const std::string& pattern) { + regex_shape shape; + std::string current; + int depth = 0; + bool in_class = false; + for (size_t i = 0; i < pattern.size(); ++i) { + const char c = pattern[i]; + if (c == '\\' && i + 1 < pattern.size()) { + current += pattern.substr(i++, 2); + continue; + } + if (in_class) { + in_class = c != ']'; + } else if (c == '[') { + in_class = true; + if (i + 1 < pattern.size() && pattern[i + 1] == '^') current += pattern[i++]; + if (i + 1 < pattern.size() && pattern[i + 1] == ']') current += pattern[i++]; + } else if (c == '(') { + ++depth; + } else if (c == ')') { + --depth; + } else if (c == '|' && depth == 0) { + shape.alternatives.push_back(std::move(current)); + current.clear(); + continue; + } else if (c == '^' || c == '$') { + const bool at_edge = depth == 0 && (c == '^' ? current.empty() + : i + 1 == pattern.size() || pattern[i + 1] == '|'); + shape.inner_anchor = shape.inner_anchor || !at_edge; + } + current += c; + } + shape.alternatives.push_back(std::move(current)); + return shape; +} + +/// \brief One alternative of a pattern, padded where it leaves a side unanchored +std::string padded_alternative(const std::string& alternative) { + const std::string pad = JSON_STRING_CHAR + "*"; + const bool starts = !alternative.empty() && alternative.front() == '^'; + size_t escapes = 0; + while (escapes + 1 < alternative.size() && alternative[alternative.size() - 2 - escapes] == '\\') ++escapes; + const bool ends = alternative.size() > (starts ? 1u : 0u) && alternative.back() == '$' && escapes % 2 == 0; + const size_t begin = starts ? 1 : 0; + const std::string core = alternative.substr(begin, alternative.size() - begin - (ends ? 1 : 0)); + if (core.empty()) { + return starts && ends ? std::string() : pad; + } + return (starts ? "(" : pad + "(") + core + (ends ? ")" : ")" + pad); +} + +/// \brief A JSON Schema string pattern as xgrammar's whole-string regex has to spell it +/// \note JSON Schema finds a pattern anywhere in the string, while xgrammar matches +/// the whole string against it, so "\S" (not blank) would allow exactly one +/// character. A side an alternative leaves unanchored gets padded. Left as sent: +/// a pattern anchored at both ends, one that opens with an inline flag such as +/// "(?i)", which xgrammar only takes there, and one with an anchor inside a group, +/// which xgrammar ignores rather than reads. +std::string whole_string_pattern(const std::string& pattern) { + const bool inline_flag = pattern.size() > 2 && pattern.compare(0, 2, "(?") == 0 && + std::isalpha(static_cast(pattern[2])); + const regex_shape shape = scan_regex(pattern); + if (inline_flag || shape.inner_anchor) { + return pattern; + } + if (shape.alternatives.size() == 1) { + const std::string padded = padded_alternative(pattern); + return padded.find(JSON_STRING_CHAR) == std::string::npos ? pattern : padded; + } + std::string out; + for (const auto& alternative : shape.alternatives) { + out += (out.empty() ? "(" : "|") + padded_alternative(alternative); + } + return out + ")"; +} + +bool is_string_schema(const json& schema) { + const json type = schema.value("type", json()); + if (type.is_array()) { + return std::find(type.begin(), type.end(), json("string")) != type.end(); + } + return type == "string"; +} + +/// \brief Longest regex repetition xgrammar builds in place; a longer one becomes a rule +/// of its own, which it cannot build inside a nested object (the check on +/// per_rule_fsm_hashes fails and the request gets no grammar at all). +constexpr int64_t XGRAMMAR_REGEX_REPEAT_LIMIT = 128; + +/// \brief The pattern xgrammar is given for a string schema, which may take over its length bounds +/// \note xgrammar drops minLength/maxLength once a pattern is there. A lone character +/// class that only has to occur somewhere leads instead of being padded, so the +/// bounds become the repetition after it and leave the schema. That is narrower +/// than JSON Schema, which lets the class occur anywhere ("\d" with bounds will not +/// write "Room 5"), but what it writes is always valid. Bounds too long for that +/// repetition keep the schema's minLength/maxLength and drop the pattern +/// (std::nullopt): a string past its maxLength is the likelier mistake. +std::optional string_pattern(json& schema) { + const std::string pattern = schema["pattern"].get(); + const json min_length = schema.value("minLength", json(0)); + json max_length = schema.value("maxLength", json()); + const bool bounded = schema.contains("minLength") || schema.contains("maxLength"); + if (!bounded || !is_one_character(pattern) || !min_length.is_number_integer() || + !(max_length.is_null() || max_length.is_number_integer())) { + return whole_string_pattern(pattern); + } + if (max_length.is_number_integer() && max_length.get() > std::numeric_limits::max()) { + max_length = json(); + } + const int64_t low = std::max(min_length.get(), 1) - 1; + if (max_length.is_number_integer() && max_length.get() - 1 < low) { + // unsatisfiable; xgrammar ignores the bounds beside a pattern, so this holds to the pattern alone + return whole_string_pattern(pattern); + } + const bool long_repeat = max_length.is_null() ? low > XGRAMMAR_REGEX_REPEAT_LIMIT + : max_length.get() - 1 > XGRAMMAR_REGEX_REPEAT_LIMIT; + if (long_repeat) { + return std::nullopt; + } + const std::string high = max_length.is_null() ? std::string() : std::to_string(max_length.get() - 1); + schema.erase("minLength"); + schema.erase("maxLength"); + return pattern + JSON_STRING_CHAR + "{" + std::to_string(low) + "," + high + "}"; +} + /// \brief Spell out what a schema leaves implicit where xgrammar needs it /// \note An array without "items" means any items; the qwen_xml style rejects /// every value of such an array unless that is written as "items": {}. +/// Literal values (const, enum, default, examples) are not schemas, and a +/// pattern under "not" would change meaning if rewritten, so neither is entered. json normalize_schema(json schema) { if (schema.is_array()) { for (auto& item : schema) item = normalize_schema(std::move(item)); @@ -172,8 +334,25 @@ json normalize_schema(json schema) { if (schema.value("type", json()) == "array" && !schema.contains("items") && !schema.contains("prefixItems")) { schema["items"] = json::object(); } + if (is_string_schema(schema) && schema.contains("pattern") && schema["pattern"].is_string()) { + const std::optional pattern = string_pattern(schema); + if (pattern) { + schema["pattern"] = *pattern; + } else { + schema.erase("pattern"); + } + } + static const std::set NOT_ENTERED = {"const", "enum", "default", "examples", "not"}; + // keyed by name, so a property called "default" is still a schema + static const std::set SCHEMA_MAPS = {"properties", "patternProperties", "$defs", "definitions", + "dependentSchemas"}; for (auto& [key, value] : schema.items()) { - if (value.is_object() || value.is_array()) { + if (NOT_ENTERED.count(key)) { + continue; + } + if (SCHEMA_MAPS.count(key) && value.is_object()) { + for (auto& [name, sub] : value.items()) sub = normalize_schema(std::move(sub)); + } else if (value.is_object() || value.is_array()) { value = normalize_schema(std::move(value)); } } @@ -183,6 +362,9 @@ json normalize_schema(json schema) { struct tool_def { std::string name; json parameters; // a JSON schema, or true for any arguments + /// the schema as the request sent it, for the argument grammars written here: + /// normalize_schema folds length bounds into a pattern only xgrammar reads + json declared; }; std::vector request_tools(const chat_constraint_request& request) { @@ -205,8 +387,9 @@ std::vector request_tools(const chat_constraint_request& request) { } // as xgrammar does: no schema, or a non-strict tool, still gets well formed arguments const bool strict = fn.value("strict", true); - tool.parameters = strict && fn.contains("parameters") && fn["parameters"].is_object() - ? normalize_schema(fn["parameters"]) : json(true); + tool.declared = strict && fn.contains("parameters") && fn["parameters"].is_object() + ? fn["parameters"] : json(true); + tool.parameters = tool.declared.is_object() ? normalize_schema(tool.declared) : tool.declared; tools.push_back(std::move(tool)); } return tools; @@ -247,18 +430,53 @@ std::string value_rule(const json& property, const char* generic) { return generic; } +/// \brief How a dialect writes the values of declared properties +struct value_syntax { + /// spells a string enum value the way the dialect writes strings + std::function quote_string; + /// EBNF between two array items + std::string array_sep; + /// EBNF for a string of low..high characters (high -1: no upper bound) + std::function bounded_string; +}; + +/// \brief A string property's minLength/maxLength as {low, high}, high -1 for none +/// \return std::nullopt when the property is not a plain string with usable bounds: +/// none, unsatisfiable ones (left for the model, as before), or ones past +/// XGRAMMAR_REGEX_REPEAT_LIMIT, which would cost a rule per character +std::optional> string_length_bounds(const json& property) { + if (property.value("type", json()) != "string" || property.contains("enum")) { + return std::nullopt; + } + const json min_length = property.value("minLength", json(0)); + const json max_length = property.value("maxLength", json(-1)); + if (!min_length.is_number_integer() || !max_length.is_number_integer()) { + return std::nullopt; + } + const int64_t low = std::max(min_length.get(), 0); + const int64_t high = max_length.get() < 0 ? -1 : max_length.get(); + if ((low == 0 && high < 0) || (high >= 0 && high < low) || + (high < 0 ? low : high) > XGRAMMAR_REGEX_REPEAT_LIMIT) { + return std::nullopt; + } + return std::make_pair(low, high); +} + /// \brief EBNF for one property's value, from its schema type -/// \param quote_string spells a string enum value the way the dialect writes strings -/// \param array_sep EBNF between two array items -/// \note An array follows its "items" schema, so enums and types inside lists are -/// kept too; objects nested in a value are only held to the dialect's syntax. -std::string property_value(const json& property, const std::function& quote_string, - const std::string& array_sep) { +/// \note An array follows its "items" schema, so enums, types and string lengths +/// inside lists are kept too; objects nested in a value are only held to the +/// dialect's syntax. +std::string property_value(const json& property, const value_syntax& syntax) { std::string value = value_rule(property, "value"); if (value == "array" && property.contains("items") && property["items"].is_object() && !property["items"].empty()) { - const std::string item = property_value(property["items"], quote_string, array_sep); - return "(\"[\" (" + item + " (" + array_sep + " " + item + ")*)? \"]\")"; + const std::string item = property_value(property["items"], syntax); + return "(\"[\" (" + item + " (" + syntax.array_sep + " " + item + ")*)? \"]\")"; + } + if (value == "string") { + if (const auto bounds = string_length_bounds(property)) { + return syntax.bounded_string(bounds->first, bounds->second); + } } if (!value.empty()) { return value; @@ -266,7 +484,7 @@ std::string property_value(const json& property, const std::function()); + value += (first ? "" : " | ") + syntax.quote_string(option.get()); first = false; } return value + ")"; @@ -276,7 +494,7 @@ std::string property_value(const json& property, const std::functionargs; a schema without declared @@ -285,8 +503,7 @@ std::string property_value(const json& property, const std::function is the list /// starting at property i, rest the remainder after something was written. std::string argument_list_rules(const json& schema, const std::string& prefix, const std::string& kv_sep, - const std::string& item_sep, const std::string& array_sep, - const std::function& quote_string, bool sorted) { + const std::string& item_sep, const value_syntax& syntax, bool sorted) { std::ostringstream g; const bool declared = schema.is_object() && schema.contains("properties") && schema["properties"].is_object() && !schema["properties"].empty() && schema.value("additionalProperties", false) == false; @@ -310,7 +527,7 @@ std::string argument_list_rules(const json& schema, const std::string& prefix, c auto rule = [&prefix](const char* kind, size_t i) { return prefix + kind + std::to_string(i); }; for (size_t i = 0; i < n; i++) { g << rule("p", i) << " ::= " << ebnf_literal(names[i]) << " " << kv_sep << " " - << property_value(schema["properties"][names[i]], quote_string, array_sep) << "\n"; + << property_value(schema["properties"][names[i]], syntax) << "\n"; } for (size_t i = 0; i <= n; i++) { if (i == n) { @@ -337,12 +554,73 @@ const char* BOUNDED_NUMBER_RULES = "integer ::= \"-\"? (\"0\" | [1-9] [0-9]{0,15})\n" "number ::= integer (\".\" [0-9]{1,15})? ([eE] [+-]? [0-9]{1,3})?\n"; +/// \brief Rules for Gemma4 string text of a bounded length, named gstr__ +/// \note Gemma4 writes strings raw between <|"|> marks, so the text may hold no "<|". +/// A step takes one character other than "<", or "<" with the character after +/// it (neither "<" nor "|"), and counts what it took: the length comes out exact, +/// which a {low,high} repetition over the two-character step would not give. +class gemma4_bounded_strings { +public: + /// \brief EBNF for a string of low..high characters (high -1: no upper bound) + std::string value(int64_t low, int64_t high) { + wanted_.insert({low, high}); + return "(\"<|\\\"|>\" " + name(low, high) + " \"<|\\\"|>\")"; + } + + /// \brief The rules every value() so far refers to + std::string rules() const { + std::ostringstream g; + std::set> done; + std::vector> todo(wanted_.begin(), wanted_.end()); + while (!todo.empty()) { + const auto [low, high] = todo.back(); + todo.pop_back(); + if (!done.insert({low, high}).second) { + continue; + } + if (low == 0 && high < 0) { + g << name(low, high) << " ::= ([^<] | \"<\" [^|])*\n"; + continue; + } + std::string steps; + for (int64_t width = 1; width <= 2; width++) { + if (high >= 0 && high < width) { + continue; + } + const std::pair next = {std::max(low - width, 0), + high < 0 ? -1 : high - width}; + steps += std::string(steps.empty() ? "" : " | ") + (width == 1 ? "[^<]" : "\"<\" [^<|]") + " " + + name(next.first, next.second); + todo.push_back(next); + } + if (steps.empty()) { + g << name(low, high) << " ::= \"\"\n"; + } else if (low == 0) { + g << name(low, high) << " ::= (" << steps << ")?\n"; + } else { + g << name(low, high) << " ::= " << steps << "\n"; + } + } + return g.str(); + } + +private: + static std::string name(int64_t low, int64_t high) { + return "gstr_" + std::to_string(low) + "_" + (high < 0 ? std::string("x") : std::to_string(high)); + } + + std::set> wanted_; +}; + /// \brief Gemma4 call arguments: {key:<|"|>text<|"|>,n:5,flag:true,list:[...]} std::string gemma4_arguments_grammar(const json& schema) { - auto quote = [](const std::string& s) { return ebnf_literal("<|\"|>" + s + "<|\"|>"); }; + gemma4_bounded_strings strings; + const value_syntax syntax{[](const std::string& s) { return ebnf_literal("<|\"|>" + s + "<|\"|>"); }, "\",\"", + [&strings](int64_t low, int64_t high) { return strings.value(low, high); }}; std::ostringstream g; g << "root ::= \"{\" args \"}\"\n" - << argument_list_rules(schema, "", "\":\"", "\",\"", "\",\"", quote, true) + << argument_list_rules(schema, "", "\":\"", "\",\"", syntax, true) + << strings.rules() << "pair ::= ident \":\" value\n" << "ident ::= [a-zA-Z_] [a-zA-Z0-9_]*\n" << "value ::= string | number | boolean | \"null\" | object | array\n" @@ -356,7 +634,14 @@ std::string gemma4_arguments_grammar(const json& schema) { /// \brief The calls inside one LFM2 block: f(key="text", n=5), g(flag=True) std::string python_calls_grammar(const std::vector& tools, bool parallel) { - auto quote = [](const std::string& s) { return ebnf_literal("\"" + s + "\""); }; + // an escape such as \n is one character of the value + auto bounded = [](int64_t low, int64_t high) { + const std::string count = "{" + std::to_string(low) + "," + (high < 0 ? "" : std::to_string(high)) + "}"; + return "(\"\\\"\" ([^\"\\\\] | \"\\\\\" [^\\n])" + count + " \"\\\"\" | \"'\" ([^'\\\\] | \"\\\\\" [^\\n])" + count + + " \"'\")"; + }; + const value_syntax syntax{[](const std::string& s) { return ebnf_literal("\"" + s + "\""); }, "\",\" \" \"?", + bounded}; std::ostringstream g; g << "root ::= call" << (parallel ? " (\",\" \" \"? call)*" : "") << "\n"; g << "call ::= "; @@ -367,7 +652,7 @@ std::string python_calls_grammar(const std::vector& tools, bool parall for (size_t i = 0; i < tools.size(); i++) { const std::string prefix = "t" + std::to_string(i) + "_"; g << "call_" << i << " ::= " << ebnf_literal(tools[i].name + "(") << " " << prefix << "args \")\"\n" - << argument_list_rules(tools[i].parameters, prefix, "\"=\"", "\",\" \" \"?", "\",\" \" \"?", quote, false); + << argument_list_rules(tools[i].declared, prefix, "\"=\"", "\",\" \" \"?", syntax, false); } g << "pair ::= ident \"=\" value\n" << "ident ::= [a-zA-Z_] [a-zA-Z0-9_]*\n" @@ -417,7 +702,7 @@ tool_syntax tool_tags(const chat_format_spec& spec, const std::vector& case chat_format_kind::GEMMA4: syntax.trigger = "<|tool_call>"; syntax.tags.push_back(tag("<|tool_call>call:" + tool.name, - grammar_content(gemma4_arguments_grammar(tool.parameters)), "")); + grammar_content(gemma4_arguments_grammar(tool.declared)), "")); break; case chat_format_kind::LFM2_PYTHONIC: case chat_format_kind::CONTENT_ONLY: @@ -445,7 +730,7 @@ json reasoning_prefix(const chat_format_spec& spec, bool allow) { json response_schema(const chat_constraint_request& request) { if (!request.json_schema.is_null()) { - return request.json_schema; + return normalize_schema(request.json_schema); } return json{{"type", "object"}}; } diff --git a/src/common/core/corelib_runtime.cpp b/src/common/core/corelib_runtime.cpp index ccf32c220..c28458b7c 100644 --- a/src/common/core/corelib_runtime.cpp +++ b/src/common/core/corelib_runtime.cpp @@ -44,9 +44,16 @@ std::shared_ptr CorelibRuntime::CreateReady( api->Check(api->functions().selftest_dependencies(), "ryzenai_corelib_selftest_dependencies"); // Without one corelib still runs a session, on the GPU or the host, which - // is the silent slowness the core backend is meant to rule out. - if (!api->functions().npu_usable()) { - throw std::runtime_error("corelib has no usable NPU"); + // is the silent slowness the core backend is meant to rule out. An AIE2P + // NPU runs corelib's kernels only behind an xclbin, which FastFlowLM never + // hands it, so only AIEM counts. + const auto& functions = api->functions(); + if (!functions.has_feature(ryzenai_corelib_feature_npu_aiem)) { + const char* name = functions.has_feature(ryzenai_corelib_feature_npu_aie2p) + ? functions.feature_name(ryzenai_corelib_feature_npu_aie2p) + : nullptr; + throw std::runtime_error(std::string("corelib has no usable NPU (") + + (name ? name : "none") + ")"); } return std::shared_ptr(new CorelibRuntime(std::move(api))); } @@ -62,8 +69,14 @@ std::string CorelibRuntime::NpuArch(const std::filesystem::path& executable_dir) try { std::lock_guard lock(process_mutex); const auto api = process_runtime ? process_runtime->api_ : ProcessApi(executable_dir); - const char* arch = api->functions().npu_arch(); - return arch ? arch : ""; + const auto& functions = api->functions(); + for (const auto npu : {ryzenai_corelib_feature_npu_aiem, + ryzenai_corelib_feature_npu_aie2p}) { + if (!functions.has_feature(npu)) continue; + const char* name = functions.feature_name(npu); + return name ? name : ""; + } + return ""; } catch (...) { return ""; } diff --git a/src/common/models/README.md b/src/common/models/README.md index 811f5a016..22b025673 100644 --- a/src/common/models/README.md +++ b/src/common/models/README.md @@ -24,9 +24,9 @@ either axis off the other is how adding corelib once took the FastFlowLM models away. `get_device()` asks the machine, once per process. A build that links corelib -asks corelib what NPU it has (`CorelibRuntime::NpuArch`, corelib's -`ryzenai_corelib_testing_npu_arch` until its public header names one): `AIEM` -is `dev04`, and `AIE2P`, no NPU, or a corelib that cannot be loaded is +asks corelib what NPU it has (`CorelibRuntime::NpuArch`, the +`ryzenai_corelib_has_feature` NPU named by `ryzenai_corelib_feature_name`): +`npu_aiem` is `dev04`, and `npu_aie2p`, no NPU, or a corelib that cannot be loaded is `default_npu_platform()`, `aie2p`. corelib reads the device itself, so FastFlowLM holds no table of devices and names none. A build without corelib answers `aie2p`: every `dev04` model is a corelib one. Linking corelib still @@ -43,7 +43,7 @@ This is a contributor document. For *using* a backend once it exists — `--back |---|---| | **Platform** | Windows and Linux. Both configure and build; only Windows has been run on hardware, and the `src/test/phi4_core` suite is still Windows-only. | | **Hardware** | An dev04 NPU. There is no simulator; a wrong shape shows up as garbage output, not an error. | -| **corelib headers** | The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.60.0`** match on major, minor and patch. [`corelib_api.hpp`](../../include/core/corelib_api.hpp) `#error`s on any other version — deliberately, because the C ABI has changed shape between patch releases. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build gives one `ryzenai_corelib_model_` library per model, carrying its kernels, which must sit beside corelib. A per-op engine enables its model's kernel bundle on the stream it opens; a model corelib runs whole enables its own. | +| **corelib headers** | The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.103.0`** match on major, minor and patch. [`corelib_api.hpp`](../../include/core/corelib_api.hpp) `#error`s on any other version — deliberately, because the C ABI has changed shape between patch releases. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build gives one `ryzenai_corelib_model_` library per model, carrying its kernels, which must sit beside corelib. A per-op engine enables its model's kernel bundle on the stream it opens; a model corelib runs whole enables its own. | | **Weights** | A GGUF corelib's model reads: a Q4_G128 or Q4_G32 one, whose codes it packs as they are, or, for Qwen3 and Llama only, a Q8_0 one, which it requantizes on every load. The catalog pulls what corelib's own runner does: AMD's Q4 file, with no public fallback, for every model but Qwen3-0.6B and Llama, which AMD ships no Q4 of and which stay on a public Q8_0. See §6. | Configure with: diff --git a/src/include/core/corelib_api.hpp b/src/include/core/corelib_api.hpp index 6fadc26f9..913b47d0b 100644 --- a/src/include/core/corelib_api.hpp +++ b/src/include/core/corelib_api.hpp @@ -2,15 +2,11 @@ #include -#if RYZENAI_CORELIB_VERSION_MAJOR != 0 || RYZENAI_CORELIB_VERSION_MINOR != 60 || \ +#if RYZENAI_CORELIB_VERSION_MAJOR != 0 || RYZENAI_CORELIB_VERSION_MINOR != 103 || \ RYZENAI_CORELIB_VERSION_PATCH != 0 -#error "FastFlowLM requires ryzenai-corelib headers exactly 0.60.0" +#error "FastFlowLM requires ryzenai-corelib headers exactly 0.103.0" #endif -// ryzenai_corelib_testing_npu_arch, until corelib's public header names the -// NPU's architecture. After the pin: a testing hook may change in any release. -#include - #include #include #include @@ -26,8 +22,8 @@ X(status_to_string, ryzenai_corelib_status_to_string) \ X(get_last_error_message, ryzenai_corelib_get_last_error_message) \ X(selftest_dependencies, ryzenai_corelib_selftest_dependencies) \ - X(npu_usable, ryzenai_corelib_npu_usable) \ - X(npu_arch, ryzenai_corelib_testing_npu_arch) \ + X(has_feature, ryzenai_corelib_has_feature) \ + X(feature_name, ryzenai_corelib_feature_name) \ X(object_release, ryzenai_corelib_object_release) \ X(create_stream, ryzenai_corelib_create_stream) \ X(stream_synchronize, ryzenai_corelib_stream_synchronize) \ diff --git a/src/include/core/corelib_runtime.hpp b/src/include/core/corelib_runtime.hpp index 6e08299bf..bdd642855 100644 --- a/src/include/core/corelib_runtime.hpp +++ b/src/include/core/corelib_runtime.hpp @@ -19,7 +19,7 @@ class CorelibRuntime final { const std::filesystem::path& executable_dir); static std::shared_ptr CreateForTest( std::shared_ptr api); - /// \brief this machine's NPU as corelib names it: "AIEM", "AIE2P", or "" + /// \brief this machine's NPU as corelib names it: "npu_aiem", "npu_aie2p", or "" /// when it knows none or cannot be loaded /// \note Needs no usable NPU and opens no device, so it answers where /// GetOrCreate would refuse; it loads the API GetOrCreate then reuses. diff --git a/src/include/utils/npu_platform.hpp b/src/include/utils/npu_platform.hpp index 96aab6560..b54c469bf 100644 --- a/src/include/utils/npu_platform.hpp +++ b/src/include/utils/npu_platform.hpp @@ -36,9 +36,9 @@ constexpr std::string_view platform_id(npu_platform platform) { constexpr npu_platform default_npu_platform() { return npu_platform::aie2p; } /// \brief the generation corelib's name for this machine's NPU stands for -/// \param arch what corelib reports: "AIEM", "AIE2P", or "" for none it knows +/// \param arch what corelib reports: "npu_aiem", "npu_aie2p", or "" for none it knows constexpr npu_platform platform_from_corelib_arch(std::string_view arch) { - return arch == "AIEM" ? npu_platform::dev04 : default_npu_platform(); + return arch == "npu_aiem" ? npu_platform::dev04 : default_npu_platform(); } /// \brief ask the machine which NPU generation it has diff --git a/src/test/chat_tools/test_chat_tools.cpp b/src/test/chat_tools/test_chat_tools.cpp index 35d738a05..6eb28f1e5 100644 --- a/src/test/chat_tools/test_chat_tools.cpp +++ b/src/test/chat_tools/test_chat_tools.cpp @@ -152,7 +152,7 @@ void check_prefix_stable(const chat_format_spec& spec, const std::string& text, /// \brief A grammar engine over a vocabulary of single characters, enough to match text class grammar_fixture { public: - grammar_fixture() { + explicit grammar_fixture(const chat_format_spec& spec = QWEN_XML) : spec_(spec) { json vocab = json::object(); for (int c = 1; c < 127; c++) vocab[std::string(1, static_cast(c))] = c; // a plain decoder: every token is the text it stands for @@ -164,7 +164,7 @@ class grammar_fixture { chat_vocab v; v.size = 128; v.tokenizer_json = path_; - engine_ = std::make_unique(v, QWEN_XML); + engine_ = std::make_unique(v, spec_); } ~grammar_fixture() { std::error_code ec; @@ -174,12 +174,12 @@ class grammar_fixture { /// \brief Whether the request's grammar takes the text, without reporting bool takes(const chat_constraint_request& request, const std::string& text) { std::string error; - return engine_->accepts(QWEN_XML, request, "", text, error); + return engine_->accepts(spec_, request, "", text, error); } bool accepts(const chat_constraint_request& request, const std::string& text, const char* label) { std::string error; - if (!engine_->accepts(QWEN_XML, request, "", text, error)) { + if (!engine_->accepts(spec_, request, "", text, error)) { std::printf(" FAIL %s: the grammar rejects the template's call: %s\n", label, error.c_str()); failures++; return false; @@ -188,6 +188,7 @@ class grammar_fixture { } private: + chat_format_spec spec_; std::string path_; std::unique_ptr engine_; }; @@ -552,6 +553,129 @@ void test_gemma4_nullable_types() { CHECK_EQ(tmpl->render(minja::Context::make(minja::Value(json{{"value", out["note"]}}))), "STRING nullable"); } +/// \brief One tool whose string parameter has length bounds, as test_tool_calls.py sends it +const json NOTE_TOOLS = json::array({tool( + "save_note", object_schema({{"text", {{"type", "string"}, {"minLength", 2}, {"maxLength", 6}, {"pattern", "\\S"}}}, + {"tags", {{"type", "array"}, {"items", {{"type", "string"}, {"maxLength", 3}}}}}}, + {"text"}))}); + +chat_constraint_request note_request() { + chat_constraint_request request = tools_request(chat_tool_choice::FUNCTION, "save_note"); + request.tools = &NOTE_TOOLS; + return request; +} + +void test_gemma4_string_length() { + std::printf("gemma4 grammar holds a string to its minLength/maxLength\n"); + grammar_fixture grammar(chat_format_make(chat_format_kind::GEMMA4, true)); + auto note = [](const std::string& text, const std::string& more = "") { + return "<|tool_call>call:save_note{" + more + "text:<|\"|>" + text + "<|\"|>}"; + }; + CHECK(grammar.takes(note_request(), note("ab"))); + CHECK(grammar.takes(note_request(), note("abcdef"))); + CHECK(!grammar.takes(note_request(), note("a"))); + CHECK(!grammar.takes(note_request(), note("abcdefg"))); + // "<" counts as one character, and "<|" still cannot occur in the text + CHECK(grammar.takes(note_request(), note("ax<|\"|>,<|\"|>xyz<|\"|>],"))); + CHECK(!grammar.takes(note_request(), note("ab", "tags:[<|\"|>wxyz<|\"|>],"))); +} + +void test_lfm2_string_length() { + std::printf("lfm2 grammar holds a string to its minLength/maxLength\n"); + grammar_fixture grammar(chat_format_make(chat_format_kind::LFM2_PYTHONIC, true)); + auto note = [](const std::string& text) { + return "<|tool_call_start|>[save_note(text=" + text + ")]<|tool_call_end|>"; + }; + CHECK(grammar.takes(note_request(), note("\"abcdef\""))); + CHECK(grammar.takes(note_request(), note("'ab'"))); + CHECK(!grammar.takes(note_request(), note("\"a\""))); + CHECK(!grammar.takes(note_request(), note("\"abcdefg\""))); + // an escape is one character of the value + CHECK(grammar.takes(note_request(), note("\"abcd\\\"e\""))); + CHECK(!grammar.takes(note_request(), note("\"abcde\\\"f\""))); +} + +chat_constraint_request string_schema_request(const json& property) { + chat_constraint_request request; + request.allow_reasoning = false; + request.json_schema = {{"type", "object"}, {"properties", {{"v", property}}}, {"required", {"v"}}, + {"additionalProperties", false}}; + return request; +} + +void test_string_pattern() { + std::printf("a string pattern matches anywhere, as JSON Schema reads it\n"); + grammar_fixture grammar(HERMES); + auto takes = [&grammar](const json& property, const std::string& value) { + return grammar.takes(string_schema_request(property), "{\"v\": \"" + value + "\"}"); + }; + auto pattern = [](const std::string& p) { return json{{"type", "string"}, {"pattern", p}}; }; + CHECK(takes(pattern("\\S"), "Paris")); + CHECK(takes(pattern("\\S"), "a")); + CHECK(!takes(pattern("\\S"), " ")); + CHECK(takes(pattern("\\d{5}"), "zip 90210")); + // anchored at both ends: as sent + CHECK(takes(pattern("^\\d{4}$"), "2024")); + CHECK(!takes(pattern("^\\d{4}$"), "x2024")); + CHECK(takes(pattern("^ab"), "abc")); + CHECK(!takes(pattern("^ab"), "cab")); + CHECK(takes(pattern("^$"), "")); + CHECK(!takes(pattern("^$"), "a")); + // each alternative keeps its own anchor + CHECK(takes(pattern("^a|b"), "ax")); + CHECK(takes(pattern("^a|b"), "xbx")); + CHECK(!takes(pattern("^a|b"), "xa")); + // an anchor inside a group, which xgrammar would ignore: left as sent + CHECK(takes(pattern("(?:^abc)"), "abc")); + CHECK(!takes(pattern("(?:^abc)"), "xabc")); + // padding never ends the string: a non-ASCII pattern is compiled without xgrammar's + // JSON string guard, where ".*" took "x\", \"k\": \"y" and added a key + CHECK(takes(pattern("(\xC3\xA9|x)"), "ax")); + CHECK(!takes(pattern("(\xC3\xA9|x)"), "x\", \"k\": \"y")); + // a lone class folds its length bounds + const json bounded = {{"type", "string"}, {"minLength", 2}, {"maxLength", 5}, {"pattern", "\\S"}}; + CHECK(takes(bounded, "abcde")); + CHECK(!takes(bounded, "abcdef")); + CHECK(!takes(bounded, "a")); + // past xgrammar's repetition limit the bounds stay and the pattern goes, nested too + const json caption = {{"type", "string"}, {"minLength", 1}, {"maxLength", 512}, {"pattern", "\\S"}}; + chat_constraint_request nested = string_schema_request( + {{"type", "object"}, {"properties", {{"caption", caption}}}, {"required", {"caption"}}, {"additionalProperties", false}}); + CHECK(grammar.takes(nested, "{\"v\": {\"caption\": \"Monthly sales\"}}")); +} + +void test_string_pattern_inline_flag() { + std::printf("a pattern opening with an inline flag is left as sent\n"); + grammar_fixture grammar(HERMES); + // padded, it would read ".*((?i)abc).*", which xgrammar cannot build + CHECK(grammar.takes(string_schema_request({{"type", "string"}, {"pattern", "(?i)abc"}}), "{\"v\": \"abc\"}")); +} + +void test_schema_literals_untouched() { + std::printf("literal values and \"not\" are not rewritten as schemas\n"); + grammar_fixture grammar(HERMES); + // a property named like a keyword is still a schema + chat_constraint_request request; + request.allow_reasoning = false; + request.json_schema = {{"type", "object"}, + {"properties", {{"default", {{"type", "string"}, {"pattern", "\\S"}}}}}, + {"required", {"default"}}, + {"additionalProperties", false}}; + CHECK(grammar.takes(request, "{\"default\": \"Paris\"}")); + // an enum value shaped like a string schema is a value: it is matched as sent + request.json_schema = {{"type", "object"}, + {"properties", {{"v", {{"enum", {{{"type", "string"}, {"pattern", "\\S"}}}}}}}}, + {"required", {"v"}}, + {"additionalProperties", false}}; + CHECK(grammar.takes(request, "{\"v\": {\"type\":\"string\",\"pattern\":\"\\\\S\"}}")); +} + } // namespace int main() { @@ -565,6 +689,11 @@ int main() { test_continued_call(); test_no_tools_after_tool_turns(); test_gemma4_nullable_types(); + test_gemma4_string_length(); + test_lfm2_string_length(); + test_string_pattern(); + test_string_pattern_inline_flag(); + test_schema_literals_untouched(); if (failures) { std::printf("%d check(s) failed\n", failures); return 1; diff --git a/src/test/llama_core/test_llama_frontend.cpp b/src/test/llama_core/test_llama_frontend.cpp index be073295c..3babd661e 100644 --- a/src/test/llama_core/test_llama_frontend.cpp +++ b/src/test/llama_core/test_llama_frontend.cpp @@ -35,6 +35,9 @@ const std::vector kPackageEosIds = {128001, 128008, 128009}; /// \brief what llama_core_backend.cpp states const std::vector kCoreEosIds = {128009, 128008, 128001}; constexpr int kBosId = 128000; +/// \brief config.json's vocab_size; a forward returns this many logits, as the +/// frontend refuses fewer +constexpr int kVocabSize = 128256; std::vector g_encoded_tokens; std::vector g_samples; @@ -49,14 +52,14 @@ class FakeEngine final : public causal_lm { forwarded.push_back(token); if (fail_forward) Fail(); ++position; - return buffer(1); + return buffer(kVocabSize); } buffer prefill(std::vector& tokens, void*) override { ++prefill_calls; prefill_sizes.push_back(static_cast(tokens.size())); if (fail_prefill) Fail(); position += static_cast(tokens.size()); - return buffer(1); + return buffer(kVocabSize); } void set_context_length(int value) override { position = value; } void load_weights(Q4NX&) override {} @@ -142,7 +145,7 @@ class TempPackage final { ("flm-llama-frontend-" + std::to_string(++serial)); std::filesystem::create_directories(path_); Write(path_ / "config.json", - {{"model_type", "llama"}, {"vocab_size", 128256}, + {{"model_type", "llama"}, {"vocab_size", kVocabSize}, {"bos_token_id", kBosId}, {"eos_token_id", eos_token_id}}); Write(path_ / "tokenizer_config.json", {{"bos_token", "<|begin_of_text|>"}, {"bos_token_id", kBosId}, diff --git a/src/test/model_list_platform/test_model_list_platform.cpp b/src/test/model_list_platform/test_model_list_platform.cpp index d0a970ad1..bfe7636cc 100644 --- a/src/test/model_list_platform/test_model_list_platform.cpp +++ b/src/test/model_list_platform/test_model_list_platform.cpp @@ -898,10 +898,10 @@ void test_platform_helpers() { utils::npu_platform::dev04); TEST_REQUIRE(utils::default_npu_platform() == utils::npu_platform::aie2p); - // corelib names the NPU; only its AIEM is dev04, and anything else, + // corelib names the NPU; only its npu_aiem is dev04, and anything else, // no NPU included, is the default. - TEST_REQUIRE(utils::platform_from_corelib_arch("AIEM") == utils::npu_platform::dev04); - TEST_REQUIRE(utils::platform_from_corelib_arch("AIE2P") == utils::npu_platform::aie2p); + TEST_REQUIRE(utils::platform_from_corelib_arch("npu_aiem") == utils::npu_platform::dev04); + TEST_REQUIRE(utils::platform_from_corelib_arch("npu_aie2p") == utils::npu_platform::aie2p); TEST_REQUIRE(utils::platform_from_corelib_arch("") == utils::npu_platform::aie2p); // This hardware-free target does not define FLM_ENABLE_CORE, so the diff --git a/src/test/openai_api_test/README.md b/src/test/openai_api_test/README.md index 1ad630740..6d179eadd 100644 --- a/src/test/openai_api_test/README.md +++ b/src/test/openai_api_test/README.md @@ -34,8 +34,8 @@ Each case prints `PASS`, `FAIL` or `SKIP`. The exit code is non-zero if anything |---|---| | `test_response_shape.py` | Message and chunk shape, one delta kind per chunk, `finish_reason`, `usage` only with `stream_options.include_usage`, stream folds to the non-stream message | | `test_reasoning.py` | `think: true` → `reasoning_content`, `think: false` / `reasoning_effort: "none"` → none, reasoning streamed before content, switch is per request, reasoning in history is accepted | -| `test_tool_calls.py` | `auto` / `required` / forced function / `none`, `parallel_tool_calls: false`, ids, arguments are JSON matching the tool schema (typed: integer, number, boolean, enum, arrays), `strict: false`, streamed call deltas, a tool round trip | -| `test_response_format.py` | `json_object`, `json_schema` (flat, enum, Pydantic `$defs`/`$ref`, `const`/`anyOf`/bounds/`pattern`), streamed, with and without thinking, schema vs. `auto` tools, `text` | +| `test_tool_calls.py` | `auto` / `required` / forced function / `none`, `parallel_tool_calls: false`, ids, arguments are JSON matching the tool schema (typed: integer, number, boolean, enum, arrays, an unanchored string `pattern`, a `\S` pattern with `maxLength`, short or long, nested), `strict: false`, streamed call deltas, a tool round trip | +| `test_response_format.py` | `json_object`, `json_schema` (flat, enum, Pydantic `$defs`/`$ref`, `const`/`anyOf`/bounds/`pattern`, unanchored `pattern`), streamed, with and without thinking, schema vs. `auto` tools, `text` | | `test_logprobs.py` | `logprobs` / `top_logprobs` shape in both modes, greedy token heads its `top_logprobs`, entries spell the content and nothing else (no reasoning, no tool call markup), stream and non-stream report the same tokens, `max_tokens` N writes at most N tokens and `completion_tokens` counts each, malformed values get 400 | | `test_sampling.py` | The same `seed` samples the same text (with an unseeded request in between, streamed or not), `repeat_last_n: 0` turns penalties off, `-1` covers the whole reply, malformed values get 400 | | `test_continue_final_message.py` | `continue_final_message` continues a trailing assistant message instead of answering it: content is only the new text, whitespace as written (a model that reasons first may read the prefix as reasoning, and close it), `logprobs` cover exactly the continued tokens (the first entry scores the next token), stream matches non-stream, a cache-resumed continuation matches a cold one, a normal turn can follow, malformed combinations get 400 | diff --git a/src/test/openai_api_test/test_response_format.py b/src/test/openai_api_test/test_response_format.py index d19885967..dd2533fb2 100644 --- a/src/test/openai_api_test/test_response_format.py +++ b/src/test/openai_api_test/test_response_format.py @@ -112,6 +112,17 @@ def schema_keywords(api: Api) -> None: nonstream_json(api, body, REVIEW_SCHEMA) +@case +def schema_unanchored_pattern(api: Api) -> None: + """An unanchored pattern only has to match somewhere in the string.""" + schema = {"type": "object", "properties": {"city": {"type": "string", "pattern": "\\S"}}, + "required": ["city"], "additionalProperties": False} + body = api.body(user("Which city is the capital of France? Answer as JSON."), + response_format=json_schema_format("city", schema), max_tokens=64) + _, value = nonstream_json(api, body, schema) + expect("paris" in value["city"].lower(), f"the pattern \\S did not leave room for the city: {value['city']!r}") + + @case def schema_stream(api: Api) -> None: """json_schema, streamed: the folded content validates.""" diff --git a/src/test/openai_api_test/test_tool_calls.py b/src/test/openai_api_test/test_tool_calls.py index dc62c1cbc..258642377 100644 --- a/src/test/openai_api_test/test_tool_calls.py +++ b/src/test/openai_api_test/test_tool_calls.py @@ -168,6 +168,50 @@ def function_typed_arguments(api: Api) -> None: expect(isinstance(args["refundable"], bool), f"refundable is not a boolean: {args['refundable']!r}") +@case +def function_unanchored_pattern(api: Api) -> None: + """An unanchored string pattern only has to match somewhere in the string, as JSON Schema reads it.""" + tool = json.loads(json.dumps(GET_WEATHER)) + tool["function"]["parameters"]["properties"]["city"] = { + "type": "string", "minLength": 1, "maxLength": 64, "pattern": "\\S"} + choice = api.ok(api.body(user(WEATHER_PROMPT), tools=[tool], tool_choice=function_choice("get_weather"))) + city = json.loads(expect_calls(choice, [tool])[0]["function"]["arguments"])["city"] + expect("paris" in city.lower(), f"the pattern \\S did not leave room for the city: {city!r}") + + +NOTE = ("The quarterly review moved to Thursday at three in the afternoon, " + "in the large conference room on the fourth floor.") + + +@case +def function_pattern_keeps_length_bounds(api: Api) -> None: + """A one-character pattern such as \\S still leaves maxLength in force.""" + tool = {"type": "function", "function": { + "name": "save_note", "description": "Save a note", + "parameters": {"type": "object", "required": ["text"], "properties": { + "text": {"type": "string", "minLength": 1, "maxLength": 20, "pattern": "\\S"}}}}} + body = api.body(user(f"Save this note word for word: {NOTE}"), tools=[tool], + tool_choice=function_choice("save_note")) + text = json.loads(expect_calls(api.ok(body), [tool])[0]["function"]["arguments"])["text"] + expect(0 < len(text) <= 20 and not text.isspace(), f"text {text!r} is not 1..20 characters with a non-blank one") + + +@case +def function_nested_long_bounded_pattern(api: Api) -> None: + """A \\S pattern with a long maxLength inside a nested object still gets a grammar.""" + caption = {"type": "string", "minLength": 1, "maxLength": 512, "pattern": "\\S"} + tool = {"type": "function", "function": { + "name": "draw_chart", "description": "Draw a chart", + "parameters": {"type": "object", "required": ["spec"], "additionalProperties": False, + "properties": {"spec": {"type": "object", "required": ["caption"], + "additionalProperties": False, + "properties": {"caption": caption}}}}}} + choice = api.ok(api.body(user("Draw a chart of monthly sales with a short caption."), tools=[tool], + tool_choice=function_choice("draw_chart"))) + spec = json.loads(expect_calls(choice, [tool])[0]["function"]["arguments"])["spec"] + expect(0 < len(spec["caption"]) <= 512, f"caption length {len(spec['caption'])} is outside 1..512") + + @case def function_with_thinking(api: Api) -> None: """A forced call still works when the model thinks first.""" diff --git a/src/test/phi4_core/fake_corelib.cpp b/src/test/phi4_core/fake_corelib.cpp index f920e0123..872596c80 100644 --- a/src/test/phi4_core/fake_corelib.cpp +++ b/src/test/phi4_core/fake_corelib.cpp @@ -174,10 +174,21 @@ struct TypedFake { return current_detail.c_str(); } else if constexpr (std::is_same_v) { return state.selftest_status; - } else if constexpr (std::is_same_v) { - return state.npu_usable; - } else if constexpr (std::is_same_v) { - return state.npu_arch.c_str(); + } else if constexpr (std::is_same_v) { + const auto feature = std::get<0>(arguments); + if (feature == ryzenai_corelib_feature_npu) return state.npu != fake_corelib::kNoNpu; + return (feature == ryzenai_corelib_feature_npu_aiem || + feature == ryzenai_corelib_feature_npu_aie2p) && + feature == state.npu; + } else if constexpr (std::is_same_v) { + switch (std::get<0>(arguments)) { + case ryzenai_corelib_feature_npu: return "npu"; + case ryzenai_corelib_feature_npu_aiem: return "npu_aiem"; + case ryzenai_corelib_feature_npu_aie2p: return "npu_aie2p"; + case ryzenai_corelib_feature_gpu: return "gpu"; + case ryzenai_corelib_feature_npu_direct: return "npu_direct"; + default: return static_cast(nullptr); + } } else if constexpr (std::is_same_v) { void* object = std::get<0>(arguments); if (object) { @@ -227,7 +238,7 @@ struct TypedFake { constexpr std::int64_t kPage = 4096; const auto wanted = std::max(state.model_max_sequence_length, 1); *out = {sizeof(*out), "fake", 16, state.model_vocab, - 4, (wanted + kPage - 1) / kPage * kPage, -1}; + 4, (wanted + kPage - 1) / kPage * kPage, -1, 0, -1}; } return Status(Tag::name); } else if constexpr (std::is_same_v) { @@ -442,7 +453,8 @@ struct TypedFake { for (const auto n : outputs) { for (const auto m : shipped(n == 1024 ? "matmul-1024" : "matmul-3072")) { const ryzenai_corelib_matmul_bf16_kernel_info info{ - sizeof(info), 0, m, 3072, n, 64, false, 0, 0}; + sizeof(info), 0, m, 3072, n, 64, false, 0, 0, + ryzenai_corelib_matmul_bf16_weights_format_mladf_v4}; callback(ctx, &info); } } @@ -460,7 +472,8 @@ struct TypedFake { if (callback) { if constexpr (std::is_same_v) { const ryzenai_corelib_matmul_bf16_kernel_info info{ - sizeof(info), 0, -1, 0, 0, 0, false, 0, 0}; + sizeof(info), 0, -1, 0, 0, 0, false, 0, 0, + ryzenai_corelib_matmul_bf16_weights_format_mladf_v4}; callback(ctx, &info); } else { const ryzenai_corelib_ssmlp_bf16_kernel_info info{ @@ -666,8 +679,7 @@ void Reset() { RYZENAI_CORELIB_VERSION_PATCH}; state.selftest_status = ryzenai_corelib_status_success; state.default_status = ryzenai_corelib_status_success; - state.npu_usable = true; - state.npu_arch = "AIEM"; + state.npu = ryzenai_corelib_feature_npu_aiem; state.detail.clear(); state.status_text = "success"; state.missing_symbol.clear(); diff --git a/src/test/phi4_core/fake_corelib.hpp b/src/test/phi4_core/fake_corelib.hpp index 458510abe..4318116ef 100644 --- a/src/test/phi4_core/fake_corelib.hpp +++ b/src/test/phi4_core/fake_corelib.hpp @@ -86,6 +86,9 @@ struct WeightFromFileRecord { /// \brief the position a forward records when it names none inline constexpr std::size_t kPositionContinue = SIZE_MAX; +/// \brief State::npu for a machine without an NPU corelib knows +inline constexpr auto kNoNpu = static_cast(0); + struct ModelForwardRecord { std::size_t rows{}; std::size_t position{}; @@ -101,10 +104,9 @@ struct State { RYZENAI_CORELIB_VERSION_PATCH}; ryzenai_corelib_status selftest_status{ryzenai_corelib_status_success}; ryzenai_corelib_status default_status{ryzenai_corelib_status_success}; - /// \brief what npu_usable() answers - bool npu_usable{true}; - /// \brief what testing_npu_arch() answers - std::string npu_arch{"AIEM"}; + /// \brief the NPU has_feature() reports, npu_aiem, npu_aie2p or kNoNpu; + /// feature_name() names features as corelib does + ryzenai_corelib_feature npu{ryzenai_corelib_feature_npu_aiem}; std::string detail; std::string status_text{"success"}; std::string missing_symbol; diff --git a/src/test/phi4_core/test_corelib_api.cpp b/src/test/phi4_core/test_corelib_api.cpp index 46dda6710..33fad0833 100644 --- a/src/test/phi4_core/test_corelib_api.cpp +++ b/src/test/phi4_core/test_corelib_api.cpp @@ -222,12 +222,17 @@ void TestRuntimeRunsDependencySelftestAndRequiresAUsableNpu() { }), "selftest_dependencies"); - fake_corelib::Reset(); - fake_corelib::GetState().npu_usable = false; - RequireContains(RequireThrows([] { - CorelibRuntime::CreateForTest(ValidApi()); - }), - "no usable NPU"); + for (const auto& [npu, name] : + {std::pair{fake_corelib::kNoNpu, "(none)"}, + std::pair{ryzenai_corelib_feature_npu_aie2p, "(npu_aie2p)"}}) { + fake_corelib::Reset(); + fake_corelib::GetState().npu = npu; + const std::string error = RequireThrows([] { + CorelibRuntime::CreateForTest(ValidApi()); + }); + RequireContains(error, "no usable NPU"); + RequireContains(error, name); + } fake_corelib::Reset(); const auto runtime = CorelibRuntime::CreateForTest(ValidApi()); @@ -239,10 +244,12 @@ void TestNpuArchIsCorelibsAnswerAndEmptyWhenItCannotLoad() { fake_corelib::Reset(); { const auto runtime = CorelibRuntime::CreateForTest(ValidApi()); - fake_corelib::GetState().npu_arch = "AIE2P"; - TEST_REQUIRE(CorelibRuntime::NpuArch({}) == "AIE2P"); - fake_corelib::GetState().npu_arch = "AIEM"; - TEST_REQUIRE(CorelibRuntime::NpuArch({}) == "AIEM"); + fake_corelib::GetState().npu = ryzenai_corelib_feature_npu_aie2p; + TEST_REQUIRE(CorelibRuntime::NpuArch({}) == "npu_aie2p"); + fake_corelib::GetState().npu = fake_corelib::kNoNpu; + TEST_REQUIRE(CorelibRuntime::NpuArch({}).empty()); + fake_corelib::GetState().npu = ryzenai_corelib_feature_npu_aiem; + TEST_REQUIRE(CorelibRuntime::NpuArch({}) == "npu_aiem"); } CorelibRuntime::ShutdownProcess(); diff --git a/src/test/phi4_core/test_model_downloader.cpp b/src/test/phi4_core/test_model_downloader.cpp index 94973194a..5c55aadee 100644 --- a/src/test/phi4_core/test_model_downloader.cpp +++ b/src/test/phi4_core/test_model_downloader.cpp @@ -866,7 +866,7 @@ void TestShippedCatalogDeclaresTheDflashDependant() { TEST_REQUIRE(info.at("details").at("family") == "gemma4e-dflash"); TEST_REQUIRE(info.at("xclbin_folder") == "Nemotron-3.5-Bolt-E4B-DFlash-NPU2"); TEST_REQUIRE(info.at("files").get>() == - (std::vector{"config.json", "dflash_weight.q4nx", "LICENSE.OpenMDW-1.1"})); + (std::vector{"config.json", "dflash_weight.q4nx"})); // the base entry is the shipped plain-decode tag, untouched const auto base = models.get_model_info("nemotron3.5-bolt:e4b").second; TEST_REQUIRE(base.at("name") == "Nemotron-3.5-Bolt-E4B-NPU2"); @@ -899,7 +899,7 @@ void TestShippedCatalogDeclaresTheDflashDependant() { TEST_REQUIRE(q38_info.at("details").at("family") == "qwen3.8-dflash2"); TEST_REQUIRE(q38_info.at("xclbin_folder") == "Qwen3.8-27B-NPU2"); TEST_REQUIRE(q38_info.at("files").get>() == - (std::vector{"config.json", "dflash_weight.q4nx", "LICENSE"})); + (std::vector{"config.json", "dflash_weight.q4nx"})); TEST_REQUIRE(models.get_model_info("qwen3.8-mtp:27b").second.at("name") == "Qwen3.8-27B-NPU2"); } diff --git a/src/test/phi4_core/test_phi4_frontend.cpp b/src/test/phi4_core/test_phi4_frontend.cpp index 7218e54d8..f339a6a8e 100644 --- a/src/test/phi4_core/test_phi4_frontend.cpp +++ b/src/test/phi4_core/test_phi4_frontend.cpp @@ -39,6 +39,9 @@ void register_builtin_backends(BackendRegistry&) {} namespace { constexpr const char* kGguf = "Phi-4-mini-instruct-Q4_G128.gguf"; +/// \brief config.json's vocab_size; a forward returns this many logits, as the +/// frontend refuses fewer +constexpr int kVocabSize = 200064; std::vector g_encoded_tokens; std::vector g_samples; @@ -59,7 +62,7 @@ class FakeEngine final : public causal_lm { throw std::runtime_error("submitted inference failed"); } ++position; - return buffer(1); + return buffer(kVocabSize); } buffer prefill(std::vector& tokens, void*) override { ++prefill_calls; @@ -69,7 +72,7 @@ class FakeEngine final : public causal_lm { throw std::runtime_error("submitted inference failed"); } position += static_cast(tokens.size()); - return buffer(1); + return buffer(kVocabSize); } void set_context_length(int value) override { position = value; } void load_weights(Q4NX&) override {} @@ -183,7 +186,7 @@ nlohmann::json PackageConfig() { return {{"model_type", "phi3"}, {"num_hidden_layers", 32}, {"hidden_size", 3072}, {"intermediate_size", 8192}, {"num_attention_heads", 24}, {"num_key_value_heads", 8}, - {"head_dim", 128}, {"vocab_size", 200064}, + {"head_dim", 128}, {"vocab_size", kVocabSize}, {"rms_norm_eps", 1.0e-5}, {"original_max_position_embeddings", 4096}, {"eos_token_id", 199999}}; }