Skip to content

maple: preserve canonical tokenizer and generation metadata - #441

Open
JohnUzoka wants to merge 1 commit into
PrismML-Eng:prismfrom
JohnUzoka:fix/maple-canonical-tokenizer-metadata
Open

JohnUzoka wants to merge 1 commit into
PrismML-Eng:prismfrom
JohnUzoka:fix/maple-canonical-tokenizer-metadata

Conversation

@JohnUzoka

Copy link
Copy Markdown

Overview

Preserve Maple's canonical tokenizer and generation metadata through conversion, tokenization and model-save/load, instead of deriving lexical specialness or all stop behavior from token spelling.

  • Validate the exact NFC/BPE tokenizer family and consistent added-token flags in the Maple producer; preserve lexical FIM/tool/think tokens.
  • Honor the generation configuration's complete EOS policy, including null/empty policies and multiple IDs.
  • Add the optional tokenizer.ggml.eog_token_ids UINT32-array extension end to end. Presence makes it authoritative, independently of lexical attributes; absence keeps legacy inference, with USER_DEFINED FIM tokens no longer promoted to control/EOG just because of their spelling.
  • Preserve typed empty arrays in gguf-py, apply requested BPE NFC normalization, and retain EOG/NFC policy on model save.
  • Normalize the Maple reference sliding window's left span to GGUF's total causal-key count in the producer only (512 past keys plus the current key -> 513).
  • Include the existing synthetic EOG-policy/save-load regressions, GPT-2/Qwen35 NFC vocabulary probes, Python empty-array regressions, and metadata documentation.

The producer, gguf-py, runtime and saver remain together so the new explicit policy cannot be emitted without its consumer/roundtrip support. This is a proposed optional GGUF extension, not a claim that older runtimes understand it. There are no embedded-LoRA, attention, parser or Unicode-long-s changes in this PR. Unrelated assertion weakening from the research snapshot is deliberately not included; existing GGUF reader validation assertions remain intact.

This standalone branch starts at prism@6447906. Implementations and regression sources are extracted from JohnUzoka/llama.cpp@3d1930a, whose original base d7b2896 is an ancestor of prism; only the tokenizer-related saver hunks are included.

Additional information

Recorded validation, not a new test run

The 2026-11-11 canonical correctness record records:

  • The public EOG probe failed against old DLLs and passed against corrected DLLs; NFC probes passed for GPT-2 and Qwen35 vocabularies.
  • Canonical Maple tokenization comparison: 39/52 before, 52/52 after in that recorded cohort.
  • The actual offline vocab-only producer matched ten checked metadata fields and types, with vocabulary size 151936, EOG set [151645], and window 513. Tensor conversion was unchanged and was not re-executed.
  • Python validation totals recorded there are 47 existing conversion tests, 10 canonical-metadata tests and 9 GGUF reader/metadata pytest tests. These totals cover the wider research suites, not this PR's two new Python reader tests alone. Canonical metadata fixtures and tests remain in the public research repository; they also exercise its standalone repair utility and are not represented as native converter unit tests.

The separate tokenizer-save regression record records loss of explicit empty EOG before the saver correction, then empty/single/multiple EOG, lexical behavior and NFC surviving save/load with the same public probe. Source provenance is preserved.

No build, tests, linters or formatters were run during extraction; this separated branch has not been independently built. Historical counts and results are not fresh CI. No model-quality, throughput or broad tokenizer-family compatibility improvement is claimed.

Requirements

  • Contributing guidelines: inspected while preparing this submission. The contributor remains responsible for review and maintenance; no human-review attestation or agreement is made on their behalf.
  • AI usage disclosure: YES. AI-assisted research produced the published source changes. An AI agent inspected the diff and recorded evidence, mechanically separated the producer/runtime/saver dependency group and existing regressions, preserved unrelated upstream assertions, expanded documentation, and prepared this PR. No new runtime validation is claimed.

@github-actions github-actions Bot added documentation Improvements or additions to documentation testing conversion labels Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conversion documentation Improvements or additions to documentation testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant