Repository navigation
Conversation
Assisted-by: OpenAI coding assistant
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Preserve Maple's canonical tokenizer and generation metadata through conversion, tokenization and model-save/load, instead of deriving lexical specialness or all stop behavior from token spelling.
tokenizer.ggml.eog_token_idsUINT32-array extension end to end. Presence makes it authoritative, independently of lexical attributes; absence keeps legacy inference, with USER_DEFINED FIM tokens no longer promoted to control/EOG just because of their spelling.The producer, gguf-py, runtime and saver remain together so the new explicit policy cannot be emitted without its consumer/roundtrip support. This is a proposed optional GGUF extension, not a claim that older runtimes understand it. There are no embedded-LoRA, attention, parser or Unicode-long-s changes in this PR. Unrelated assertion weakening from the research snapshot is deliberately not included; existing GGUF reader validation assertions remain intact.
This standalone branch starts at
prism@6447906. Implementations and regression sources are extracted fromJohnUzoka/llama.cpp@3d1930a, whose original based7b2896is an ancestor ofprism; only the tokenizer-related saver hunks are included.Additional information
Recorded validation, not a new test run
The 2026-11-11 canonical correctness record records:
[151645], and window 513. Tensor conversion was unchanged and was not re-executed.The separate tokenizer-save regression record records loss of explicit empty EOG before the saver correction, then empty/single/multiple EOG, lexical behavior and NFC surviving save/load with the same public probe. Source provenance is preserved.
No build, tests, linters or formatters were run during extraction; this separated branch has not been independently built. Historical counts and results are not fresh CI. No model-quality, throughput or broad tokenizer-family compatibility improvement is claimed.
Requirements