xwrite

A content engine in Rust: MDX → renderable output, byte-exact TextMate syntax highlighting, LaTeX → MathML, and fast frontmatter extraction.

Extracted from livesession/xyd, where it powers the docs framework. It lives here so it can be used on its own.

Crates

crate	what it does
xwrite_mdx	Rust-first whole-page MDX compiler for prose pages. Drives mdxjs-rs's decomposed pipeline (mdast → hast → swc), applies portable transforms (heading ids, table normalization, directives, @-functions, outputVars, frontmatter + toc), and gates anything it can't handle to a JS fallback sentinel.
xwrite_highlight	A vscode-textmate engine over Oniguruma that reproduces @code-hike/lighter output byte-for-byte. Ships 254 grammars + 27 themes as compressed assets.
xwrite_math	LaTeX ($…$ / $$…$$) → MathML Core via pulldown-latex. Pure Rust — no JS engine. Includes a MathML equivalence metric checked against a real-KaTeX corpus.
xwrite_frontmatter	Batch YAML frontmatter extraction, with a YAML-1.1-vs-1.2 divergence detector that defers ambiguous files rather than guessing.
xwrite_highlight_wasm	Browser build of the highlighter. Reuses the same engine and delegates the regex primitive to vscode-oniguruma's onig.wasm, so client output is identical to the native path. Not a workspace member (see below).

xwrite_mdx ──▶ xwrite_highlight        xwrite_highlight_wasm ──▶ xwrite_highlight
           └─▶ xwrite_math                                       (js-scanner)

third-party/ holds two vendored forks — see third-party/README.md.

Using it

There is no crates.io release yet (all crates are publish = false — see Publishing below). Consume it as a git submodule:

git submodule add https://github.com/livesession/xwrite xwrite

[dependencies]
xwrite_mdx        = { path = "xwrite/crates/xwrite_mdx" }
xwrite_highlight  = { path = "xwrite/crates/xwrite_highlight" }
# Just the forked markdown parser, without pulling in swc:
markdown          = { path = "xwrite/third-party/markdown-fork", features = ["serde"] }

These paths are a stable contract:

path	what it is
crates/xwrite_{frontmatter,highlight,math,mdx}	path-dep targets
crates/xwrite_highlight_wasm	wasm-pack target — never build it for the host
third-party/markdown-fork	the forked markdown crate, usable directly
crates/xwrite_mdx/tests/fixtures/mdx-parity/	the MDX oracle corpus, incl. _harness/{render,normalize}.mjs

Building and testing

cargo test --workspace     # the engine
npm ci                     # only needed for the MDX parity gate (react + react-dom)

Requirements: a C toolchain (onig_sys builds Oniguruma from bundled C) and node for the MDX parity gate.

The MDX parity gate renders compiled output through a committed JS harness and compares it to the oracle. Without node_modules it skips loudly rather than failing with a wall of module-resolution errors. In CI, XWRITE_REQUIRE_PARITY=1 makes a missing prerequisite fatal, so the gate can never silently go dark:

XWRITE_REQUIRE_PARITY=1 cargo test --workspace     # what CI runs
cargo test -p xwrite_mdx --test parity -- --nocapture   # see skip reasons

The two scanner backends

xwrite_highlight needs exactly one (a compile_error! enforces it):

• native-onig (default) — links Oniguruma; the server/native path.
• js-scanner — no C dependency; a host registers a ScannerBackend at runtime. This is what the wasm build uses, wiring it to onig.wasm.

The seam is proven behavior-preserving by a dedicated test that must be asked for explicitly:

cargo test -p xwrite_highlight --no-default-features --features js-scanner --test js_scanner_seam

core-langs trims the embedded grammar set to the top-20 languages (plus their embed closure) for size-sensitive builds.

The wasm build

npm run wasm:build     # wasm-pack build --target nodejs --out-dir pkg-node
npm run wasm:verify    # 27 goldens × 2 themes through the real onig.wasm

Status: the browser path is an unfinished spike. wasm:verify currently matches only json, toml, txt and unknown_lang — the grammars needing little or no Oniguruma regex work. Everything regex-heavy diverges, which points at the UTF-8↔UTF-16 offset conversion in js/onig-binding.js. The Rust seam itself is proven: js_scanner_seam drives the same delegated backend with native Oniguruma and matches all 54 cells byte-for-byte. CI runs wasm:verify non-blocking so the number stays visible; make it a hard gate at 54/54.

xwrite_highlight_wasm is deliberately excluded from the workspace: it is wasm32-only (its regex primitive is a JS import), workspace feature unification would hand xwrite_highlight both backends at once, and it needs its own release profile. CI covers it with explicit --manifest-path steps.

Regenerating oracles

The Rust code never writes goldens. Every oracle is produced by a JS generator and committed; the Rust tests only compare against them.

generator	produces	needs
crates/xwrite_highlight/scripts/vendor-assets.mjs	assets/**	CODE9_DIR + the zstd CLI
crates/xwrite_highlight/scripts/gen-goldens.mjs	tests/goldens/**	CODE9_DIR, HL_BUILD_FIXTURES=1
crates/xwrite_highlight/scripts/gen-codehike-goldens.mjs	tests/goldens-codehike/**	codehike (declared here), HL_BUILD_FIXTURES=1
crates/xwrite_math/reference/gen.mjs	tests/katex-reference.json	standalone npm install in reference/
the mdx-parity corpus	crates/xwrite_mdx/tests/fixtures/mdx-parity/**	generated by xyd's JS pipeline (see below)

CODE9_DIR points at a syntax0/code9 checkout, which is not vendored here — only its output is. The scripts fail with an explicit message when it is unset.

The MDX oracle comes from xyd

The mdx-parity corpus is generated by xyd's live JS pipeline (packages/xyd-content/scripts/gen-mdx-goldens.mjs), because the whole point is to prove the Rust engine matches that pipeline. Regenerating it is a two-repo operation: regenerate into this repo, re-run cargo test -p xwrite_mdx --test parity here, merge here, then bump the submodule pointer in xyd. Landing the JS change first turns this repo's CI red until the goldens follow.

Publishing

Not published, deliberately. Three blockers:

1. xwrite_mdx cannot be published as-is. It path-deps the vendored forks; on crates.io, markdown and mdxjs would resolve to upstream, silently losing the directive and outputVars constructs. Fixing this means publishing renamed forks or upstreaming the patches.
2. The 5.5 MB of grammars and themes in xwrite_highlight/assets/ need a licensing audit first.
3. All crates are 0.0.0; there is no version policy yet.

xwrite_frontmatter and xwrite_math are otherwise publishable — which is why the serde = "=1.0.219" pin is scoped to xwrite_mdx instead of the workspace, where an = requirement would poison them.

Notes on the history

This repo was extracted from xyd with git-filter-repo, so per-file history and git log --follow work across the move. Commits before the rebrand have xwrite_* directory names but still-xyd_* crate names inside, so they do not build in this layout — the usual cost of a history-preserving extraction.

Doc comments referring to packages/xyd-*, @xyd-js/*, or crates/xyd_* are pointers to the xyd repo, where the JS implementations these crates mirror still live. Stage tags like Track C, C-S2, H0–H3, S6+ are xyd's migration milestones, kept because tests/parity.rs documents its capability floors in those terms.

License

MIT — see LICENSE. The vendored forks carry their own upstream MIT licenses.