From 41fd94a424a4adbcd0aaa936d0fd54b91a93b725 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Tue, 15 Sep 2026 09:11:45 -0400 Subject: [PATCH 01/32] Add complete HermitCrab optimization round record (2026-08/09) Full docs-only record of the September 2026 HermitCrab optimization round: 22 ledger attempts (2 earlier shipped wins, 12 closed as measured-does-not-pay, 3 closed before building, 5 reopened this round), the probe design needed to rebuild the measurement instrumentation from scratch, the per-mechanism disposition (9 retained mechanisms with individual records, 4 rejected with individual records), and the branch/worktree inventory. The round's own conclusion, from the disposition doc: do not merge the branch as a package (896 lines across 28 files, not reviewable as one change, and the package speedup could not be attributed to its parts); isolate what's worth isolating separately, and refocus on PR #491 and grammar hygiene where the large returns are. Kept here as a durable "do not retry this" record, not as a merge candidate. Copied from branch perf/hc-optimization-archive onto current master, replacing the stale two-file version of this PR. Minimal fixups only: the ledger, probe-design, disposition, and counts docs were read in full to check for (a) references to branches deleted since the archive was cut and (b) private corpus data. Found and fixed: the disposition doc's several references to `perf/hc-edge-prefilter`, whose own branch and archive tag have both since been deleted; the mechanism itself is not lost, since it also lives in the whole-round archive (branch `origin/perf/hc-optimization-archive`) copied in by this same commit, so the references now name that ref instead of calling the mechanism unrecoverable. The branch-wide parity sentence in the counts doc is qualified to say what `WordAnalysisSignature` actually verifies on the FLEx exports (morph counts, since `Morpheme.Id` is empty), matching the caveat the disposition doc already recorded. The finished execution-plan checklist is dropped now that its outcome is captured in the disposition record instead. No private word forms or corpus data found; the docs already record only word-list indices, counts, and ratios, per the round's own scrubbing note. No findings, numbers, or conclusions were otherwise altered. Docs only, no code changes. Co-Authored-By: Claude Opus 5 Co-Authored-By: Claude Sonnet 5 --- docs/hermitcrab-optimization-ledger.md | 283 ++++++++++++++++++ docs/hermitcrab-perf-2026-09-counts.md | 93 ++++++ docs/hermitcrab-perf-2026-09-disposition.md | 135 +++++++++ docs/hermitcrab-probe-design.md | 238 +++++++++++++++ docs/optimizations/deferred-template-clone.md | 35 +++ docs/optimizations/edge-segment-prefilter.md | 64 ++++ .../feature-struct-frozen-tree-clone.md | 28 ++ .../feature-struct-frozen-tree-equality.md | 29 ++ .../feature-struct-visited-state-reuse.md | 28 ++ docs/optimizations/flat-register-arrays.md | 25 ++ .../shape-copy-single-pass-map.md | 28 ++ docs/optimizations/shape-sharing-on-clone.md | 44 +++ .../syntactic-feature-structure-sharing.md | 32 ++ .../alternatives-memo-and-hoist.md | 15 + .../compounding-split-prefilter.md | 11 + .../final-template-prune.md | 25 ++ .../two-pass-nondeterministic-traversal.md | 14 + 17 files changed, 1127 insertions(+) create mode 100644 docs/hermitcrab-optimization-ledger.md create mode 100644 docs/hermitcrab-perf-2026-09-counts.md create mode 100644 docs/hermitcrab-perf-2026-09-disposition.md create mode 100644 docs/hermitcrab-probe-design.md create mode 100644 docs/optimizations/deferred-template-clone.md create mode 100644 docs/optimizations/edge-segment-prefilter.md create mode 100644 docs/optimizations/feature-struct-frozen-tree-clone.md create mode 100644 docs/optimizations/feature-struct-frozen-tree-equality.md create mode 100644 docs/optimizations/feature-struct-visited-state-reuse.md create mode 100644 docs/optimizations/flat-register-arrays.md create mode 100644 docs/optimizations/shape-copy-single-pass-map.md create mode 100644 docs/optimizations/shape-sharing-on-clone.md create mode 100644 docs/optimizations/syntactic-feature-structure-sharing.md create mode 100644 docs/rejected-optimizations/alternatives-memo-and-hoist.md create mode 100644 docs/rejected-optimizations/compounding-split-prefilter.md create mode 100644 docs/rejected-optimizations/final-template-prune.md create mode 100644 docs/rejected-optimizations/two-pass-nondeterministic-traversal.md diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md new file mode 100644 index 000000000..28443b33f --- /dev/null +++ b/docs/hermitcrab-optimization-ledger.md @@ -0,0 +1,283 @@ +# HermitCrab optimization ledger — tried, closed, do not retry + +**Purpose: stop this work being redone.** One row per optimization attempted, what was expected, +what actually happened, and the number that settled it. If you are about to try something on this +list, read its row first — several of these look irresistible on paper and three of them have +already been independently rediscovered and re-retracted. + +Deliberately in-repo and durable, not evicted into a PR accordion. A closed avenue is only closed +if the next person can find out cheaply that it is closed. + +**This is the only report.** The working docs it was distilled from +(`hermitcrab-packed-forest-research.md`, `hermitcrab-forest-memo-plan.md`, +`hermitcrab-forest-memo-ceiling.md`, `hermitcrab-synthesis-fold-probes.md`) are chronological +process records containing retracted intermediate claims, and are deliberately **not** merged. +They are preserved only as this record. **The evidence branches were deleted after the round +closed**, along with all instrumentation: research scaffolding is not worth maintaining in two +product assemblies, since the probes gate on `volatile bool` reads inside `Matcher.cs`, a hot inner +loop in `SIL.Machine` used by every consumer. + +**`docs/hermitcrab-probe-design.md` is the replacement.** It carries the probe architecture and its +two-assembly gate trap, the eight exclusive wall-time buckets and why `synForward` must be recorded +net of cascade and battery, the per-node decomposition, the census designs, all three key designs +with field-level audits, the classifier's direction trap, and the full A/B methodology with the +three measurement failures that forced each part of it — enough to rebuild any probe in a day. The +expensive part was never the code; it was learning what to measure and which measurements lie. + +**No instrumentation is merged, by design.** The probes gate on `volatile bool` reads inside +`Matcher.cs` — a hot inner loop in `SIL.Machine` used by every consumer — which is an acceptable +cost for a measurement run and not one to carry in product code indefinitely. Rebuilding a probe +from a branch is cheap; maintaining research scaffolding in two product assemblies is not. + +## Final disposition of the September follow-up + +The follow-up branch does deliver product improvements. Its authoritative summary is +`hermitcrab-perf-2026-09-counts.md`, and each retained optimization has one record under +`docs/optimizations/`. The retained set is: deferred template cloning, feature-structure traversal-state +reuse, frozen-flat-tree equality, frozen-flat-tree cloning, single-pass `Shape.CopyTo` mapping, +copy-on-write shape sharing, flat FST register arrays, the analysis edge-segment prefilter, and +copy-on-write syntactic feature structures. + +Four attractive experiments were deliberately excluded from the final product diff and are recorded under +`docs/rejected-optimizations/`: the negative two-pass nondeterministic traversal, the inert final-template +prune, the zero-value alternatives memo/hoist, and the low-ceiling compounding split prefilter. This final +disposition supersedes intermediate “ships” language for those experiments later in this historical ledger. + +The minimized branch passed all 603 runnable tests in the conformance-inclusive suite (one expected +Windows symlink-privilege skip). A fixed three-word sample had exact output parity on Sena, Amharic, +Indonesian, Mbugwe, and Aweti; only Amharic (1.76x) and Mbugwe (1.83x) cleared the recorded spread and timing +reliability rules. The other three ratios are retained as directional evidence only. + +## Earlier ledger round: net speed delivered none + +No optimization in the "closed" section shipped, and none of them made anything faster. The +fastest thing tried was 4% **slower**. The pre-existing wins in this codebase (rows 1–3) predate +this work. What this effort produced is the measurement, the closed family, the located target, +and this ledger. + +--- + +## Earlier shipped results + +| # | Optimization | Expected | Outcome | Number | +| --- | --- | --- | --- | --- | +| 1 | Memoize analysis morphological-rule cascade | Same state reached by many rule orders; cache it | Works, but the cascade was never the cost | 2,555 expansions vs a 2,546-state floor; 1.4 s of a 30.5 s word | +| 2 | Memoize affix-template battery (same key) | Battery was 93% of wall time | **The big win.** Reduced how *often* the battery runs | 38,840 runs -> 2,581; word 30.5 s -> **6.1 s** | + +### Retracted historical prototype + +Row 3 in the original ledger claimed that shape sharing through `CloneShareFrozenShape` had shipped. No +reachable production commit carried that implementation. Its historical prototype measured 4.5–7.6% fewer +allocated bytes and no wall-time change. The September follow-up independently built a different, retained +copy-on-write design through `CloneForEngine`; see `docs/optimizations/shape-sharing-on-clone.md`. + +## Closed — measured, does not pay + +| # | Optimization | Expected | Why it failed | Number | +| --- | --- | --- | --- | --- | +| 4 | **Narrow `AnalysisStateKey`** (drop strictly-shrinking rules' un-application counts) | `MaxApplicationCount` defaults to 1, so the count multiset is "which subset of rules was used" — 2^27 states. Dropping 19 of 27 rules gives 2^8 | Shape + syntactic FS + realizational FS *already* discriminate. The count component was implied by its neighbours. A worst-case bound collapsing is not the realised state count collapsing | Realised state collapse **Sena 1.12x** (state-weighted) vs a 1.3 gate; **4 of 7** heavy probes exactly 1.00, including the 2,556-state probe. The three that moved were 1.13x, 1.16x, and **1.38x**, and all returned zero parses: R tracks failure, not size. **Reopening condition:** a grammar with genuinely high R; the census is cheap and is the right first question to ask of any new grammar | +| 5 | **Fold-step sharing memo** (share deterministic synthesis fold steps by computed value) | Probe measured 3.22x and 8.10x shareable steps; ~50% wall ceiling on two reliable fixtures | A sound key must carry the **ordered remaining trail**. Two candidates then have to agree on their entire future to share a step, which almost never happens | **hits = 0** on the best fixture; **0.96x** (4.5% *slower* — pure key-construction cost); 1.06x on the second, inside 21.9% noise | +| 6 | **Order-insensitive synthesis-input dedupe** | 9,774x duplicate synthesis inputs | HermitCrab's morphological rules are **not order-invariant**: same rule multiset, different order, different output. Merging loses parses | 2 violations on Sena, 12 independently on Indonesian. Sound version: **15–40%** | +| 7 | **Trail-position indexing of synthesis rules** | 11,445,538 of 11,445,538 rejections are "this rule is not the pending trail rule" — 100% of the histogram | Each rejection is one array index plus one reference compare. Count is not cost | 11.4M x 29 ns = 0.33 s of 143 s = **0.2%** | +| 8 | Synthesis-side length bound (Gate A) | Reject candidates that cannot reach the surface length | At the comparison point the candidate is still the bare root; its affix trail applies later inside `_synthesisRule.Apply`. Rejected valid parses | Unit suite 64 -> 34 passing. Reverted | +| 9 | Analysis length ceiling (Gate B) | Same idea, analysis side | Sound and correct. No available corpus contains the pathology it prunes | Byte-identical output, **no measurable speedup** (~4–7% slower, within noise) | +| 10 | Lexical reachability gate (Phase 5) | Prune candidates from which no lexicon root is reachable | Both reference corpora have real compounding in their deepest stratum, which disqualifies the gate everywhere | Proven **no-op** on both corpora | +| 11 | Tandem lexical intersection (T2) | Kill doomed branches early via the lexicon | The oracle only sees lexical unreachability. The expensive words fail on checks that run *after* lexical lookup succeeds | Pooled **23.5%** dead steps vs a 30% gate; one failure word at exactly **0.0%** dead | +| 21 | **Memo-key caching on the frozen `Word`** | Keys are built ~28x per distinct state, which looked like a cheap win. | High count did not imply high cost; caching would add state and invalidation complexity around negligible work. | **CLOSED:** all `AnalysisStateKey` construction and hashing on Amharic measured **3 ms of 81,707 ms = 0.0%**. | +| 12 | Pool small short-lived collections | Fewer allocations, faster | `HashSet/Dictionary.Clear()` is O(capacity); Gen0 already beats pooling at this size | -15–17% bytes but **+8.6% wall**, +12% on the parallel battery. Reverted | + +## Closed before building — motivation refuted by measurement + +| # | Optimization | Expected | Why it died | +| --- | --- | --- | --- | +| 13 | Surface-length pruning inside the synthesis fold (P2) | "The only lever on Amharic's ~160 ms per synthesis run" | Amharic's forward synthesis is **14 ms**. The 160 ms was arithmetic on an unmeasured denominator — dividing 30 s by 186 synthesis inputs and assuming the time was there | +| 14 | Nogood lattice subsumption (P4) | Nogood hits dominate positive replays **434,628 to 25,102** — the engine mostly proves subtrees *empty*, and generalising "empty" over the feature lattice would turn nogoods from points into regions | **CLOSED on direct evidence** (it was previously closed on adjacent evidence, which is why it was reopened). Two heavy Sena probes produced 16,176 expansions -> 16,176 distinct keys -> **16,176 singleton buckets, 0 subsumable (0.0%)**, and 12,595 -> 12,595 -> **12,595 singletons, 0 subsumable**. Not one pair of nogood entries even shares a `(shape, stratum, counts, non-head)` bucket, so there is nothing for a subsumption ordering to act on. Breadth across all 33 conformance fixtures agrees: 7,493 empty expansions, 7,398 distinct keys, **0 subsumable (0.0%)** pooled, every non-empty bucket a singleton. Direction checked explicitly as `earlier.Subsumes(later)` (E subsumes Q, general over specific); the reverse would be unsound. Independently reconfirms row 4 from a different angle: the keys are maximally discriminating | +| 15 | Move constraints into the analysis key (Maxwell & Kaplan's category-splitting) | Their biggest measured win: make the chart prune what the constraint solver otherwise would | Everything synthesis rejects on is root- or realization-dependent, and the root is unknown until `LexicalLookup`. Also a Sena analysis state costs ~0.73 ms while a synthesis input costs at most ~0.12 ms — **states are dearer than synthesis inputs** | + +## Reopened this round, and their outcomes + +| # | Optimization | Expected / observation | Outcome | +| --- | --- | --- | --- | +| 22 | **Analysis-side rule prefilter** | 94.4% of Sena allomorph pattern matches failed, so a cheap necessary-condition check had a large direct-work ceiling. | **RETAINED after the earlier self-time cap was refuted by direct work counts.** The original ~10% ceiling counted matcher self-time but omitted traversal allocation, feature unification, and GC booked elsewhere. The implemented edge-segment prefilter removed **87% of FST traversals**, 54% of arc checks, and 85% of traversal instances on 45 Sena words, with exact parity on those words and all 464 conformance-fixture words. See `docs/optimizations/edge-segment-prefilter.md`. | +| 16 | **`ExpandAlternatives` dedupe** | 20.7% of Sena wall, the largest bucket nobody had instrumented. Apparent census: 395,026 alternatives -> **61** distinct (0.0%), 85% of duplicates same-analysis-word | **CLOSED on the sound census.** With the pending-trail content added to the key, 395,026 -> **351,414 distinct (89.0%)** -- only **11%** are duplicates at all, and all 43,612 of them are same-analysis-word. Sound ceiling = 20.7% x 0.110 = **2.28% of Sena wall**, against a gate of >=10% to build and <5% to close. **The fourth apparent redundancy to evaporate under a complete key** | +| 17 | **Per-node cost in the analysis cascade** | 84% of the Amharic cascade is not linguistic computation; clone is 46.7% and 37.4% was unattributed | **Partly settled, remainder still open.** (a) The remainder split **MISSES its <15% gate by ~2.5x** — still ~38% on Amharic `anCascade`. Memo, Trail and **Equality each measured <0.2%**, so `Word.ValueEquals` behind the cascade's `HashSet` is *not* the missing mass, contrary to the standing hypothesis. **Two caveats the first write-up missed.** (i) The ~38% figure comes from a *contended* run — three timing probes were live — and descheduled wall time lands preferentially in the remainder while deflating every self-time share, so **CPU starvation is itself an unbracketed candidate biasing in exactly the direction observed**. (ii) The claim that GC pause time is unmeasurable is **wrong at the measurement point**: only the library assemblies are netstandard2.0; the probe harness lives in `tests/SIL.Machine.Morphology.HermitCrab.Tests`, which targets **net10.0**, so `GC.GetTotalPauseDuration()` and `GC.GetGCMemoryInfo().PauseTimePercentage` are directly callable. Next probe, gated: re-run serial — remainder <20% means contention was the missing mass and 17(a) recloses on the cheaper-clone target; still >=30% then bracket with `GetTotalPauseDuration()` deltas — ΔPause >=50% of the remainder's absolute ms confirms GC (build target converges with clone/freeze at 46.7%), <=15% refutes it and the next split is enumerator/LINQ orchestration (the 2026-07 alloc audit already flagged `ExecuteCommands` enumerator boxing); a collection census over 24 Amharic words gives gen0 8,217 / gen1 2,729 / gen2 301, an unusually high gen1+gen2 ratio consistent with — not proof of — GC as the remainder. (b) **Clone-discard fraction = 66.2%** (created 2,505, discarded 1,659), a count and therefore contention-immune, against a >=70% gate: **misses by ~4 points, so the build is "make a clone cheaper", not "defer cloning"**. Scoped to `AnalysisAffixTemplateRule`'s eager per-template clone, the one clone-before-verification site; the other four analysis-side sites clone only after a successful match and were excluded rather than blended in | +| 18 | Corpus-scope memoization | `AnalysisScope` dies per word, but `AnalysisStateKey` is word-independent by construction, so entries are reusable across a corpus. Nogood hits dominate positive replays 10-17x, making cross-word nogood reuse the cheapest available form of reuse | **CLOSED, and worse than a no-op.** Indonesian, 121 words, **serial on a clean machine: 1.015x, 1.4% wall-clock reduction** against a >=20% build gate and <5% close gate. Costs **68 MB** added peak managed memory. Cross-word reuse is real but small: 39 memo, 73 nogood, 237 template-nogood hits. Token-vs-type: **121 tokens, 121 distinct**, so no type-level dedupe is available either. **A first, contended run of the same harness reported 2.385x / 58.1%** with a 210.9% spread on the shared arm against 13.7% on the baseline; two testhosts were live. The counts were byte-identical across both runs — only the timing was contaminated, and min-of-N on a high-spread arm picks the luckiest sample. Analysis-set equality held throughout, so the key genuinely is word-independent. **Sena, first 300 words, is the decisive case: the shared scope is 28-32% *slower* than per-word scopes** (two clean serial runs, no contention: 282.7s vs 372.2s and 308.9s vs 396.7s baseline-vs-shared; deterministic counters byte-identical across both runs), and costs **~29.3 GB** added peak managed memory against a 2 GB budget. Root cause: `AnalysisScope.MaxMemoEntries` (100,000) is a *per-scope*, not per-word, budget. Two heavy early words (#74, #76 of 300) alone fill the shared `Memo` table to the cap and `TemplateMemo` to 55,071; every word after that loses the free within-word memoization a fresh per-word scope would have given it for nothing, because `Store` silently no-ops past the cap (see its comment) with no eviction. Cross-word hits (shared minus baseline) are **negative** — memo -77,648, nogood -747,607 — i.e. sharing produced *fewer* total hits than 300 independent fresh-table runs would have on their own; the small `+820` template-hit / `+9` template-nogood gain does not come close to covering it. Amharic, capped at 28 words (word 29 hangs, excluded per standing guidance): **0.9% wall-clock reduction**, table nowhere near the cap (Memo 1,251, TemplateMemo 652 of 100,000), consistent with row 17's finding that Amharic's cost is dominated by clone/allocation per node rather than distinct-state count, so avoiding a handful of cross-word re-derivations barely moves wall time. Token-vs-type on all three corpora, at the runs' own sizes and on each corpus's full word list: **zero duplicate tokens** — whole-word caching is not available as a cheaper alternative anywhere here. No analysis-set divergence on any corpus, at any scope regime: the stop condition that would have put rows 1-3 in question did not fire. **A shared-but-capped memo table is not "the same optimization, smaller ceiling" — past the cap it actively takes away the free per-word memoization the product already has, and retains every live entry for the run's full duration instead of per-word, which is where the 29 GB comes from.** Any future attempt needs real eviction (LRU, generational, or per-word sub-budgets) before this is worth re-measuring, and even then the >=20% gate looks unreachable given Indonesian's ceiling under an uncapped-in-practice regime was already only 1.4% | +| 19 | Contexted constraints / abstract feature-only replay | Pack without merging states by replaying only shape-independent constraints. | **CLOSED.** Cost-weighted P1b die-point histograms put shape-free work at **6.0%** and **0.0%** on two heavy Sena probes and **0.4%** on 28 Amharic words; all fire the <30% close gate. All 33 conformance fixtures were measured for breadth: 24 close, 3 nominally fire the build gate, 3 are undecided, 2 have zero rejections, and 1 has zero words. The three build-gate fixtures are all sub-50 ms and cannot carry timing claims. One reliable fixture confirms that count and cost can diverge, but die-point rejection cost itself is only ~0.1–0.2% of wall on the real corpora. Abstract replay would optimize a phase that costs approximately nothing on the corpora treated as ground truth. Not built. | +| 20 | **Row 5's family on `GenerateWords`** (generation, ~100% synthesis share vs parsing's ~5%/~0.3%) | Row 20 was unmeasured; generation's share is ~100% so row 5's fold ratios would apply at face value if they transferred. `UseSynthesisFoldMemo` had never been wired into `GenerateWords` (only into the parse-then-synthesize path) — wired in under the same untraced/`MaxDegreeOfParallelism==1` restriction to make this measurable at all | Hits are no longer uniformly zero: 3 of 23 lookup-reaching fixtures clear the 5% gate (diacritic-segments 33.3%, suffixing-vowel-harmony 28.6%, suffixing-evidential-adjacency-chain 37.7%; a 4th sits just under it at 4.8%), reopening row 5 narrowly for generation. But the reopened case recloses immediately: the A/B on the best fixture shows the wall-clock "win" sitting inside its own noise floor, and the pooled hit rate is dominated by `deep-optional-affix-nesting` (925 of 1,613 lookups) at hits=0, reproducing row 5's exact 4.5%-slower key-construction cost on the identical fixture. Parity: 0 divergences, 1,236 generations, 32/33 fixtures (33rd throws identically both sides). Best case: 130 hits / 5 reps (37.7% hit rate), **7.8%** wall-clock reduction inside a **22.2%** off-arm spread — not a result. 19/23 lookup-reaching fixtures at exactly **0%**; pooled hit rate **2.67%** (43/1,613), skewed by the one large zero-hit fixture. | + +--- + +## The lesson that generalises + +Three independent measurements, three boundaries, same collapse: + +| measurement | key used | apparent | sound | +| --- | --- | --- | --- | +| synthesis-input dedupe (#6) | order-insensitive | 9,774x | 15–40% | +| fold-entry census | trail position only | 6,476x | not established | +| fold-step sharing (#5) | trail position only | 3.22x / 8.10x | **hits = 0** | + +**The redundancy in HermitCrab's synthesis is apparent, not real. The trail is what makes each +step distinct, and every measurement showing large shareable work is measuring a key that omits +it.** The thing that makes sharing visible is the thing that makes sharing wrong. + +That closes a *family*: packed parse forests **of the merge-by-key kind**, fold-step sharing and +synthesis-input dedupe all require distinct derivations to converge on a genuinely identical +state. In this engine they do not converge -- the same fact as the rules being non-order-invariant, +seen from the other side. This argument did not itself close row 19 because a contexted-constraint forest +packs without merging states; row 19 was later measured independently and closed by its die-point costs. + +## Where the time actually is + +Measured with eight exclusive buckets summing to wall. Amharic is a single run (`unaccounted` +0.1%). **Sena is spliced from two runs -- see the note in its row.** + +| grammar | breakdown | +| --- | --- | +| **Sena** heavy words | battery **51.4%**, cascade 18.4%, all synthesis ~5% -- *pre-`ExpandAlternatives` run (143,303 ms, unaccounted 20.1%)*; `ExpandAlternatives` **20.3%**, unaccounted 1.5% -- *later run (112,713 ms)*. **Two runs: not co-summable.** One re-run would put all Sena shares on one denominator | +| **Amharic** | analysis cascade **~95%** of analysis, which is 99.5% of wall; **all synthesis 0.3%** | + +**Every optimization in this ledger that worked reduced how many nodes are visited. None touched +what a node costs.** Amharic spends **~170 ms per analysis state**. (The *measured* state-count floor -- 2,555 +expansions against 2,546 states -- is a Sena heavy probe. For Amharic, near-floor behaviour is +**inferred** from R ~ 1, which compares the full key against the narrowed key, not against a +semantic floor.) + +### Inside one node (Amharic, 28 words, both probe layers live) + +| bucket | `anCascade` 81,707 ms | `anBattery` 7,368 ms | +| --- | --- | --- | +| clone / freeze / allocation | **46.7%** | 41.9% | +| **remainder (unattributed)** | **37.4%** | 15.4% | +| matcher traversal | 12.5% | **39.3%** | +| FeatureStruct algebra | 3.4% | 3.5% | +| memo-key construction | 0.0% | 0.0% | + +Censuses: 27,385 rule attempts — guards reject 19.5% (application-count 6.3%, unifiability 13.2%), +**80.5% reach matching**. Of 25,178 allomorph pattern matches, **1,938 succeed (7.7%)**, 23,240 +fail (**92.3%**). + +**The headline: only ~16% of the Amharic analysis cascade is *identified* as linguistic +computation** -- matching 12.5% plus feature unification 3.4%. Clone/freeze is **46.7%** and +**37.4% is unattributed**, which is unknown rather than known-non-linguistic. The defensible +statement is that at most a sixth of the cascade has been shown to decide linguistic questions; +of the rest, half is measured (clone) and half is still open. + +**Gate outcomes, written before the run.** "Any bucket >=50% becomes the sole build target" — clone +is 46.7%, does not fire. "Pattern matching dominates *and* >=80% of attempts fail -> build a +prefilter" — 92.3% do fail, but matching is 12.5%, so it does not dominate; does not fire. +**"Remainder >=30% means the decomposition is incomplete and that is the result" — 37.4%, FIRES.** + +So there is no declared build target yet. The next probe splits the 37.4%: `RuleCascade` / +enumerator orchestration, trail dictionary operations, LINQ allocation, memo lookups, GC. + +**A caution on the 46.7% before anyone acts on it.** `Word.Clone` being half the cascade does not +by itself mean cloning is waste — the cascade forks a `Word` per rule application precisely so it +can explore alternatives without mutating shared state. The deciding question is what fraction of +clones are *discarded unused*: cloned, rejected by a guard or a failed match, thrown away. Given +92.3% of pattern matches fail, that fraction has now been **measured at 66.2%** (row 17b), which misses the >=70% gate and points at the cheaper-clone build rather than the defer-clone one. If most +clones are discarded, the lever is **not cloning until a rule commits**. If most survive, the lever +is **making a clone cheaper** (copy-on-write shape, which the RUSTIFY branch already explored). +Different builds; measure first. **Gate: probe the clone-discard fraction; >=70% discarded means +the defer-clone build, below that the cheaper-clone build.** + +## Two durable code constraints + +Neither is expressible in code, and both bite silently. + +1. **`Word.ReplayOnto` does not splice `_mrulesUnapplied`.** It splices `_mruleApps` and + `_nonHeadApps` only. This is safe *solely* because `AnalysisStateKey` includes the per-rule + count multiset, which guarantees a memo hit has identical arrival and stored-arrival counts. + **Anyone narrowing that key breaks correctness with no test failure.** Fix, if the key ever + changes: store arrival counts on `MemoEntry` and compute `stored - storedArrival + query` — a + no-op under today's key, which is also how to test it. This is the one code change from this + work worth making on its own merits, as defensive hardening. +2. **HermitCrab's morphological rules are not order-invariant.** Same pending-rule multiset, + different application order, different synthesis output — 2 cases on Sena, 12 independently on + Indonesian. This bounds every packed-readout scheme. It is the assumption + Maxwell and Kaplan (CL 19(4):571–590) rely on and this engine does not satisfy. + +## Calibration record — predictions vs outcomes + +Written before the measurements, scored after. Kept because the misses are informative. + +| prediction | outcome | +| --- | --- | +| Sena key collapse 2–4x | **1.12x** — wrong, and it was the number that decided the project | +| Indonesian key collapse 1.0–1.3x | 1.17x — right | +| Amharic "most likely to disappoint" | best static classification of the three (31 of 36 rules) — wrong | +| Fold sharing ~50% wall ceiling | **0.96x realised** — wrong; the ceiling was measured with a key that would lose parses | +| Amharic synthesis-bound at ~160 ms/run | forward synthesis is **14 ms** — wrong; arithmetic on an unmeasured denominator | + +## Method rules earned here + +1. **Counting is not timing.** Three published findings in this work were retracted, all arithmetic + on an unmeasured denominator: "synthesis is the bottleneck" (candidate counts), "Amharic is + synthesis-bound" (30 s / 186 inputs), "fixtures only look synthesis-heavy because they are small" + (divided by `synForward`, which excludes the buckets the work runs in). +2. **A ratio measured with an incomplete key is not a ratio.** See the table above, three times. +3. **Warm-up dominates small fixtures.** A first A/B here showed 1.53x purely from JIT; the off-arm + alone varied 42.7% between its own first and second sample. Discard warm-up, take min of N, + interleave arms, and print the off-arm spread as a noise floor beside every speedup. +4. **Sub-50 ms fixtures cannot carry a timing claim.** One 10 ms fixture's share moved 1.3% -> 14.5% + between two runs of identical code. +5. **Three grammars minimum, reported unpooled.** Maxwell & Kaplan measured a 100x swing between two + variants of one grammar. A pooled average across fixtures of different size has had to be + retracted twice in this project. +6. **Min-of-N is only valid when the arm's spread is small relative to the effect.** Row 18's + first run read 2.385x with a 210.9% spread on one arm and 13.7% on the other; serial and clean + it is **1.015x**. Min-of-N picks each arm's luckiest sample independently, so a high-spread arm + wins spuriously. Print the spread beside every speedup and refuse a number whose spread exceeds + it. +7. **Never dispatch timing-sensitive probes concurrently.** Three at once put four testhosts on the + machine and invalidated the bucket percentages in rows 17 and 22 as well as row 18's headline. + Counts survive contention; shares and speedups do not. +8. **A shared bounded cache without eviction is not a smaller version of the optimization** — it + competes with itself for a fixed budget and can land strictly worse than no sharing. Row 18's + shared scope produced *negative* cross-word hits. +9. **Never aggregate the test logger's console output.** It prints every line twice, the second + copy space-prefixed; a `^\s*` grep doubles every total. This produced a 2x error in row 20's + first write-up. +10. **Search completeness is never traded for speed.** HermitCrab is the permanent fallback engine + behind the FST work. +7. **A bucket reporting exactly `0.00` is broken until proven otherwise.** `NodeCostProbe.Enabled` + was declared and read in `Matcher.cs` but set by nothing — two gates in two assemblies, one + wired. Both lower-layer buckets read `0.00ms`, which looks like "measured and negligible" + rather than "never ran", and their time silently inflated the remainder. Wire cross-assembly + probe gates from a single place. +8. **Always carry an explicit remainder column.** It caught the wall-split misattribution and the + dead probe gate. Four of the five retractions in this work would have been caught at the point + of measurement by a remainder that did not sum. + +--- + +## Addendum, September 2026: the round that followed, measured by counts + +Branch `perf/hc-defer-template-clone` (on upstream master, after the merged memoization PR 456) re-ran +this ledger's open questions with deterministic step counters instead of wall time. Its records live +with the code: `docs/hermitcrab-perf-2026-09-counts.md` and one file per change under +`docs/optimizations/`. Corrections to this ledger that fell out of it: + +- **Row 3 ("shape sharing shipped") was not true on master.** No reachable commit carried it; master's + `Word.Clone` deep-copied the Shape and every node feature structure. The follow-up independently implements + a different design: internal `CloneForEngine` copy-on-write with un-share at mutation sites, while public + `Word.Clone` remains a deep mutable clone. A shared shape is frozen, so a missed mutation site throws. +- **Row 22's ~10% cap for an analysis-rule prefilter was wrong by an order of magnitude.** It was + computed from matcher *self-time*. Counted directly, an edge-segment prefilter (patterns are anchored + at both ends and matched by unification, so the first/last literal segment must unify with the + shape's first/last segment) removes **87% of FST traversals** and 54% of arc checks on 45 Sena words, + with identical analyses on every Sena word and all 464 conformance-fixture words. +- **"memo-key construction 0.0%" (row 21) was wrong at the profile level.** Hashing each new word's + syntactic feature structure for `AnalysisStateKey` is ~15% of CPU on the heaviest Sena word; the key + itself is cheap but freezing a fresh clone of a nested feature structure is not. +- **Row 16's complete-key result still stands.** An experimental `ExpandAlternatives` memo/hoist did not + reduce clones and was removed. The final branch has no alternatives-expansion change relative to the + pinned production baseline; the intermediate 75% counter reduction is not a shipped result. +- **The ledger never cited John Maxwell's work**: PR 491 (final-template prune in analysis) and the + 2025 `investigate_linear_traverse` branch. Both were tested. The prune was inert on every local corpus + because each has a partial morpheme, and the traversal measured negative by counts (instances used 2.3x) + because the current traverser already pools instances. Both implementations and their toggles were removed; + their rejected-lead records remain. +- **Wall time on the measurement machine moved 2x between windows with identical binaries** (a session + transition in between). Every verdict in this ledger that rests on a wall-time delta under ~20% should + be read with that in mind; the round's method note is to count steps on the sequential path. diff --git a/docs/hermitcrab-perf-2026-09-counts.md b/docs/hermitcrab-perf-2026-09-counts.md new file mode 100644 index 000000000..16cc86354 --- /dev/null +++ b/docs/hermitcrab-perf-2026-09-counts.md @@ -0,0 +1,93 @@ +# HermitCrab optimization round, September 2026: final disposition + +This is the final record for the count-driven round that followed +`hermitcrab-optimization-ledger.md`. Temporary counters were used to choose targets and were then removed; +the final branch carries only product changes, focused tests, the parity/timing harness, and these records. + +## Measurement scope + +The exploration used sequential parsing, one warm-up followed by repeated samples, and canonical analysis +signature sets as the parity gate. Deterministic work counts came from 45 Sena words (five known-heavy probes +plus the first 40 list entries) and 464 words from 33 conformance fixtures. Timing was never run concurrently. +The final branch-wide sample compares production commit `d7347226` (harness-only baseline tip `4f677735`) +with candidate code tip `7370e2dd`. + +The probe counters are intentionally absent from the final diff. They put volatile gates in hot code across +two assemblies and would create permanent maintenance cost for a completed experiment. The reconstruction +design remains in `hermitcrab-probe-design.md`. + +## Retained optimizations + +Each retained mechanism has one record with its invariant, implementation, evidence, parity coverage, +portability, and limits: + +| Optimization | Principal evidence | Record | +| --- | --- | --- | +| Deferred template clone | 1.054x on the 45-word Sena battery; exact parity | `optimizations/deferred-template-clone.md` | +| Feature-structure traversal-state reuse | equality/freezing allocation left the hot profile; not isolated | `optimizations/feature-struct-visited-state-reuse.md` | +| Frozen flat-tree equality | removes visited sets from the proved tree case; not isolated | `optimizations/feature-struct-frozen-tree-equality.md` | +| Frozen flat-tree clone | removes the copies map from the proved tree case; not isolated | `optimizations/feature-struct-frozen-tree-clone.md` | +| Single-pass `Shape.CopyTo` mapping | removes a second enumeration/dictionary build; not isolated | `optimizations/shape-copy-single-pass-map.md` | +| Copy-on-write shape sharing | clone 36% → 9% of profile; peak working set 11 GB → 6 GB | `optimizations/shape-sharing-on-clone.md` | +| Flat FST register arrays | `CreateInstanceMDArray` 11.5% → 0.2% of profile | `optimizations/flat-register-arrays.md` | +| Analysis edge-segment prefilter | 87% fewer FST traversals and 54% fewer arc checks on 45 Sena words | `optimizations/edge-segment-prefilter.md` | +| Copy-on-write syntactic feature structures | removes repeated clone/freeze/hash work measured at ~15% of a heavy Sena profile | `optimizations/syntactic-feature-structure-sharing.md` | + +Only the deferred clone was timed alone. Profile movement and deterministic work removal justify the other +retained mechanisms; the five-language result below measures the branch as a package and must not be divided +among them. + +## Rejected or removed experiments + +| Experiment | Deciding evidence | Final state | +| --- | --- | --- | +| Two-pass nondeterministic traversal | 2.3x more traversal instances, only 1.6% fewer arc checks, 27% more clones | removed; `rejected-optimizations/two-pass-nondeterministic-traversal.md` | +| Final-template interleaving prune | zero prunes on Sena, Amharic, and Mbugwe because partial morphemes disable it | removed; `rejected-optimizations/final-template-prune.md` | +| `ExpandAlternatives` memo/hoist | word clones unchanged at 4,747,013; no retained difference from production ordering | removed; `rejected-optimizations/alternatives-memo-and-hoist.md` | +| Compounding split prefilter | only ~2% of clones and ~9% of FST traversals were in scope | not built; `rejected-optimizations/compounding-split-prefilter.md` | + +All associated production switches, alternate traversal code, and research counters were deleted. The two +retained internal toggles exist solely for correctness/performance A/B of the edge prefilter and syntactic +feature-structure sharing. + +## Five-language branch-wide sample + +Each row used three fixed word-list indices and two reverse-ordered process rounds (baseline/candidate, then +candidate/baseline). Every process parsed sequentially, discarded one warm-up, and recorded three repetitions, +for six measured samples per arm. Times are the sum of each word's minimum sample across both rounds. Baseline +and candidate used identical grammar names and word keys; `scripts/hc-bench-compare.ps1` compared error +outcomes and rejected missing keys or signature differences. All ten arm comparisons passed the legacy +`WordAnalysisSignature` check, which on these FLEx-exported grammars compares morph counts only (`Morpheme.Id` +is empty; see `hermitcrab-perf-2026-09-disposition.md`), and the saved outputs contained zero parse or load +errors. + +| Language | Word-list indices | Baseline ms | Candidate ms | Ratio | Interpretation | +| --- | --- | ---: | ---: | ---: | --- | +| Sena | 49, 52, 74 | 1,000.7 | 448.3 | 2.23x | directional: 0/3 probes clear both the 50 ms and spread rules | +| Amharic | 8, 9, 267 | 6,410.0 | 3,646.3 | **1.76x** | stable: 3/3 probes clear both rules | +| Indonesian | 0, 9, 44 | 99.0 | 56.0 | 1.77x | exploratory: 0/3 probes clear both rules | +| Mbugwe | 0, 65, 1272 | 8,979.4 | 4,901.6 | **1.83x** | stable: 3/3 probes clear both rules | +| Aweti | 2, 3, 27 | 134.2 | 54.4 | 2.47x | exploratory: 0/3 probes clear both rules | + +The ratios are deliberately unpooled and are not extrapolated to whole corpora. An initially selected heavy +Sena trio did not complete the baseline within five minutes, and an Aweti sample containing index 0 crossed +two minutes; both whole testhosts were terminated and replaced symmetrically before any comparison. This is +sampling evidence, not a claim about those excluded words. + +## Correctness gate + +The minimized code tip was merged into the conformance integration branch and run through the complete +HermitCrab conformance-inclusive project: + +- NUnit discovered and ran 604 non-explicit tests: 603 passed, zero failed. +- One symlink-security test was skipped because the Windows host lacks symlink privilege. +- Fixture manifests, DTD/schema checks, source/graph audits, 452 fixture rows, and 348 coverage items passed. +- The ordinary HermitCrab suite also passed with syntactic feature-structure sharing disabled. + +## What remains + +The edge prefilter leaves roughly 44% of remaining traversals successful, so another edge check has little +headroom. Compounding construction is too small on the measured heavy word. Alternatives and packed-readout +schemes remain bounded by the ordered pending trail. The next credible large change is a persistent/shared +trail representation that can avoid memo-replay and alternative-materialization clones without merging +linguistically distinct derivations; it is a data-structure redesign, not a safe incremental tweak. diff --git a/docs/hermitcrab-perf-2026-09-disposition.md b/docs/hermitcrab-perf-2026-09-disposition.md new file mode 100644 index 000000000..596c4e7cf --- /dev/null +++ b/docs/hermitcrab-perf-2026-09-disposition.md @@ -0,0 +1,135 @@ +# HermitCrab optimization round, September 2026: final disposition of the branch + +**Date:** 2026-09-03. **Branch:** `perf/hc-optimization-pr` @ `179708d2`, renamed `perf/hc-optimization-archive`. +**Decision:** do not merge the branch. Isolate one mechanism onto master as its own small PR; archive the rest with +its records. Re-focus on sillsdev/machine PR #491 and on grammar hygiene, which are where the large returns are. + +## Why the package is not merged + +- 896 source lines across 28 files; about a third inside `SIL.Machine.FeatureModel.FeatureStruct`, the rest in FST + traversal internals, `Word`, `Shape`, and the analysis rules. Not reviewable as one change. +- Only one mechanism (deferred template clone, 1.054x) was ever timed alone. The package figure (1.76x Amharic, + 1.83x Mbugwe) cannot be attributed to its parts (`hermitcrab-perf-2026-09-counts.md`). +- The parity gate that blessed the package compared morph counts, not analyses: `MorpherTests.WordAnalysisSignature` + joins `Morpheme.Id`, which comes from `` and is empty for every FLEx export used. On Sena, Mbugwe + and Amharic the signature therefore degenerates to a morph count per analysis, so every earlier "exact parity" + claim on those grammars verified counts, not analyses. Anything re-measured on them needs a + gloss + allomorph-index + syntactic-feature-structure signature instead. +- Meanwhile PR #491's mechanism, measured with the guard lifted (`AlwaysEnforceFinalTemplates = true`), is + 8–33x on Mbugwe and Sena with identical analysis sets, and 298x on the single heaviest Sena word; Amharic + barely moves because its per-word cost is not in the template interleaving. The branch is worth keeping only + where #491 cannot fire (Amharic-shaped grammars), and only in its most reviewable piece. + +## Disposition per mechanism + +| Mechanism | Lines / files | Evidence on the branch | Disposition | +| --- | --- | --- | --- | +| Analysis edge-segment prefilter | ~120 in 2 rule files + 9 in `Morpher` + tests | 87% fewer FST traversals on 45 Sena words (deterministic counts); one invariant; internal toggle | **Isolated onto master** as `perf/hc-edge-prefilter` (`be7f241d`), see below — **that standalone branch and its own archive tag were later deleted, but the mechanism is preserved in the whole-round archive branch `perf/hc-optimization-archive`** | +| Copy-on-write `Shape` sharing | `Word`, `Shape`, every clone-then-mutate site | clone 36% → 9% of profile; peak working set 11 GB → 6 GB | Archived. Aliasing rules are subtle; re-open only if memory becomes the constraint. Commit `dfd1be94`, `e18ca86d` | +| Copy-on-write syntactic `FeatureStruct` sharing | `Word`, `Morpher`, rules | ~15% of a heavy Sena profile; never isolated | Archived. `4a6a47ed`, `7f9fdef3`, `da35ad4e`, `f08fefa7` | +| `FeatureStruct` equality/clone/freeze fast paths, visited-set pooling | 312 lines in core FeatureModel | "not isolated" | Archived. `325f8419`, `424a3c23`, `b23a7c99`, `00ece2ba`, `2f4dabe0` | +| Flat FST register arrays | FST internals | `CreateInstanceMDArray` 11.5% → 0.2% of profile | Archived. `b1692a7a` | +| Deferred template clone | `AnalysisAffixTemplateRule` | 1.054x alone | Archived. `57703d6b`, `24d75c82` | +| `ExpandAlternatives` memo/hoist | `Morpher` | no retained difference | Already removed on the branch. `d8886b0c`, `f4195be1` | +| Two-pass nondeterministic traversal | FST | negative | Already removed. `c3366fcc` | +| Final-template prune (own version) | analysis rules | zero prunes under the guard | Already removed; superseded by #491 work. `7e6ec59d` | +| PerfBench / compare script / ParseCounters | tests, scripts | harness | Archived. Its signature is degenerate on FLEx exports; any revival must switch to a gloss/allomorph/feature-structure signature first | +| Research records (`docs/optimizations/*`, `docs/rejected-optimizations/*`, ledger, counts, probe design, this file, the #491 re-review) | docs | — | Kept on the archive branch as the record of what was tried and why | + +## The isolated prefilter + +**Status update:** `perf/hc-edge-prefilter` was deleted after this round closed, and its own archive tag was +later deleted too, but the mechanism is not lost: it remains on the whole-round archive +branch `perf/hc-optimization-archive`, which still carries +`Morpher.EdgePrefilterEnabled`, the edge constraints in `AnalysisAffixProcessRule`, and +`AnalysisEdgePrefilterTests`. The description below is kept as a standalone record of the isolated, reviewable +form, in case that smaller diff is preferred over pulling the mechanism out of the full archive again. + +Branch `perf/hc-edge-prefilter`, worktree `.worktrees/prefilter`, one commit `be7f241d` on `origin/master` +(`d7347226`). Content: `AnalysisAffixProcessAllomorphRuleSpec` captures the left/right top-level `Constraint` +feature structures at compile time; `AnalysisAffixProcessRule.Apply` rejects an allomorph before its matcher runs +when the shape's first/last Segment node provably cannot unify with them, using the matcher's own `IsUnifiable` +call and `UseDefaults`; optional edge nodes and non-Constraint edges always pass; tracing emits the same +`MorphologicalRuleNotUnapplied` the matcher would have. `Morpher.EdgePrefilterEnabled` (internal, default true) is +the same-binary A/B switch. The research counters were not carried over. `AnalysisEdgePrefilterTests` (final branch +version, no counter dependency) came with it. HermitCrab suite: 104/104. + +### Standalone measurement (same binary, toggle off = master behavior, toggle on; sequential, memo on) + +Signature: gloss + allomorph index + syntactic feature structure, compared as multisets. Counters: throwaway +increments at the two nondeterministic `Traverse` entries and at `CheckInputMatch` (not committed). + +| Grammar | Words | Wall ms off → on | FST traversals off → on | Arc checks off → on | Parity | +| --- | ---: | ---: | ---: | ---: | --- | +| Sena | 42 (49,52,74,0–39; one invalid line skipped) | 71,296 → 69,900 (−2%) | 8,760,382 → 1,270,122 (−85%) | 12,618,566 → 6,004,919 (−52%) | identical 42/42 | +| Mbugwe | 22 (0,65,1272,1–19) | 110,609 → 96,293 (−13%) | 28,054,872 → 11,419,808 (−59%) | 36,811,510 → 25,960,854 (−29%) | identical 22/22 | +| Amharic | 10 (8,9,267,5,10,12,15,20,25,30) | 111,617 → 103,403 (−7%) | 145,050 → 129,802 (−10%) | 16,705,704 → 15,777,072 (−6%) | identical 10/10 | + +**Reading.** The count reduction reproduces (Sena −85% traversals, matching the branch's −87%), and parity holds +under a real signature. But the traversals removed are the cheap ones: they would have died at their first arc +check, so wall time moves only 2–13%. The branch's package figure (1.76x Amharic, 1.83x Mbugwe) therefore came +mostly from the allocation, clone, and GC work, not from this mechanism. Standalone, the prefilter is correct, +small, and reviewable, but it is **not a large return**. Recommendation at the time: keep `perf/hc-edge-prefilter` +ready as an optional small PR (7–13% on Bantu grammars, where it stacks with #491's prune on the surviving +branches), and do not spend review budget on it ahead of #491. The standalone single-commit branch is deleted +(see status note above); a PR now starts from the mechanism preserved in the whole-round archive +(branch `perf/hc-optimization-archive`) rather than resuming +that branch. + +Raw logs: session scratchpad `pf-sena.log`, `pf-mbugwe.log`, `pf-amharic.log`. + +## Branch and worktree inventory for the round (2026-08-20 to 2026-09-03) + +Every branch below is local unless marked. Worktrees for archived branches were removed on 2026-09-03; the +branches remain. + +| Branch | Tip | What it is | State | +| --- | --- | --- | --- | +| `perf/hc-optimization-archive` (was `perf/hc-optimization-pr`) | `b58b1642`+ | the whole round, code + records | archived, unmerged; worktree `.worktrees/ledger` | +| `perf/hc-defer-template-clone` | `4a6a47ed` | working branch of the round | fully contained in the archive; worktree removed | +| `perf/hc-engine-alloc` | `f22f88a8` | allocation/two-pass side branch | fully contained in the archive; worktree removed | +| `perf/hc-conformance-check` | `0799a977` | archive merged into the conformance integration branch for the 603-test gate | verification only; worktree `.worktrees/hc-conf` can go | +| `perf/hc-edge-prefilter` | `be7f241d` | the one isolated mechanism on `origin/master` | optional small PR at the time; **branch and its own archive tag have both since been deleted, but the mechanism is preserved on branch `perf/hc-optimization-archive`** | +| `pr-491` | `96c97f76` | jtmaxwell3's PR head | worktree `.worktrees/pr491` holds uncommitted throwaway counters + `FinalTemplateUpperBound` harness | +| `docs/hc-optimization-ledger` (remote, PR #490) | `cc806613` | the ledger, now 23 rows | open docs PR | +| `feature/forest-memo`, `feature/synthesis-fold-probes`, `feature/synthesis-fold-sharing`, `feature/per-node-cost` | — | evidence branches of the ledger round | see `hermitcrab-optimization-ledger.md` | +| `investigate-memoization-rule-order` | `4f5aeeb4` | despite the name, conformance-suite commits | belongs to the conformance line, not this round | + +Records in this directory: `hermitcrab-optimization-ledger.md` (23 attempts), `hermitcrab-perf-2026-09-counts.md` +(count round), `hermitcrab-probe-design.md`, `optimizations/*` and `rejected-optimizations/*` (one record per +mechanism), and this file. The round's execution checklist has been folded into this record; it is not kept as +a separate plan document. + +Four further records of the round were deliberately left out of this set, because they concern another author's +open PR and a sibling project rather than this round's own mechanisms. They are described here so that nothing +depends on reading them, and they remain on the `perf/hc-optimization-archive` branch: + +- `pr491-final-template-prune-upper-bound.md` — the measurement record behind the 8–33x figures above: PR #491's + prune with its partiality guard lifted, per-grammar wall time, affix-rule applications and template entries, + plus the analysis-set parity check and the signature caveat quoted earlier. It also proves the guard + over-broad, since partial *lexical entries* can never rescue a pruned branch (only partial rules can, and only + at or below the template's stratum). +- `pr491-review-draft.md` — the review comment drafted from that record: the win with the guard lifted, a + duplicate-parse bug when enforcement is active (an all-slots-skipped clone differing from its input only by + `FinalTemplateState` survives the `ApplyTemplates` equality filter), the rules-only per-stratum guard, and + state transitions that run even while the prune is disabled and cost memo hits. +- `pangloss-final-template-prune-handoff.md` — the same prune handed to PanGloss (the Rust port) for + implementation there. +- `pangloss-hc-rust-speedups-handoff-2026-09-02.md` — speedup leads handed the other way, from PanGloss into + this round. Archived; its section 4 is superseded. + +Privacy: word forms and glosses from the private grammars appear only as word-list indices and rule ids in these +records. The raw A/B logs (which contain word forms) live only in the session scratchpad and are not to be +committed. The main checkout's `.git/info/exclude` now lists the local grammar backups, `Mbugwe.xml`, the Aweti +word list, and the rulestats HTML report so they cannot be staged by accident. + +## What to do next, in order + +1. Fix and merge PR #491 (duplicate parses, memo splitting while disabled, entry-counting guard). Draft comment in the + re-review record. +2. Classify the unclassified affixes in the FLEx projects (Mbugwe 1, Amharic 1, Sena 6); that turns #491's default + prune on. Add the finding to the grammar advisor. +3. ~~Open the prefilter PR from `perf/hc-edge-prefilter`~~ — that branch and its own archive tag are both + deleted; a PR now starts from the mechanism preserved on branch + `perf/hc-optimization-archive` rather than resuming the deleted branch. +4. Port the prune to PanGloss per the handoff (`HANDOFF-pangloss-final-template-prune.md`, session scratchpad). diff --git a/docs/hermitcrab-probe-design.md b/docs/hermitcrab-probe-design.md new file mode 100644 index 000000000..eb6275d85 --- /dev/null +++ b/docs/hermitcrab-probe-design.md @@ -0,0 +1,238 @@ +# HermitCrab probe design — enough to rebuild it + +Companion to `hermitcrab-optimization-ledger.md`. That document records *what was tried and why it +failed*. This one records *how it was measured*, in enough detail to reconstruct the instrumentation +and the keys from scratch. + +**Why this exists:** the seven probe branches that produced the ledger's numbers were deleted after +the round closed. Research scaffolding in two product assemblies is not worth maintaining — the +probes gate on `volatile bool` reads inside `Matcher.cs`, a hot inner loop in `SIL.Machine` used by +every consumer. So the code is gone and this is the design record. Anything here can be rebuilt in a +day; the expensive part was learning what to measure and which measurements lie. + +--- + +## 1. Instrumentation architecture + +Two probe classes, in two assemblies, with **two independent gates**. + +`SynthesisProbe` in `SIL.Machine.Morphology.HermitCrab`, and `NodeCostProbe` in `SIL.Machine`. Each +has its own `internal static volatile bool Enabled`, default `false`. **Both must be set.** Setting +only the HermitCrab one makes every lower-layer bucket read exactly `0.00ms`, which looks like +"measured and negligible" rather than "never ran" — this cost a round (ledger method rule: a bucket +reporting exactly zero is broken until proven otherwise). Wire both from one place in the harness. + +Everything is insertion-only: a `RecordDie(...)` or `Add*Ticks(...)` call placed immediately before +an existing `return` / `continue`, never a change to control flow. Where a site read +`else if (IsTracing) { trace }`, it becomes `else { if (IsTracing) { trace } Record(...) }` — +behaviour-preserving. Counters are `Interlocked`; timers are `Stopwatch.GetTimestamp()` deltas +accumulated with `Interlocked.Add`, with `Reset*` methods so a harness reads per-word deltas. + +### The wall-time split + +**Eight mutually exclusive top-level slices that sum to wall**, measured around +`Morpher.ParseWord`: + +`lookup` (brackets `LexicalLookup`, materialised with `.ToList()` so the bracket covers only the +lookup itself) · `synCascade` · `synBattery` · `synForward` · `synExpand` (brackets +`Word.ExpandAlternatives()` in `SynthesizeSequential`) · `anTotal` (brackets the whole +`_analysisRule.Apply(input)` call) · `unaccounted` (defined as the remainder, never tracked). + +`synForward` is recorded **net of** `synCascade` and `synBattery`: capture both counters before the +`_synthesisRule.Apply` call and subtract their deltas after. This matters enormously and caused a +25x published error — the work a synthesis fold memo would share runs *inside* the cascade and +battery brackets, so the share to divide by is `synCascade + synBattery + synForward`, not +`synForward` alone. Template slot rules compile to the same `SynthesisAffixProcessRule` classes via +`RuleBatch` (`SynthesisAffixTemplateRule` ctor), which is why they land in `synBattery`. + +Nested *inside* `anTotal`, as a breakdown rather than additional slices: `anCascade` (the +`_mrulesRule.Apply` call in `AnalysisStratumRule.ApplyMorphologicalRules`), `anBattery` +(`ApplyTemplateBattery`), `anPhono` (`_prulesRule.Apply`). + +**Always print the remainder as its own column.** It caught the misattributed split, the dead probe +gate, and would have caught four of five retractions at the point of measurement. + +### The per-node decomposition + +Inside `anCascade` and `anBattery`, sub-buckets: `match` · `feature` · `clone` · `key` · `memo` · +`trail` · `equality` · explicit remainder. + +`match` and `feature` come from the `SIL.Machine` layer. To avoid double counting, the HermitCrab +side reads `NodeCostProbe.MatcherTicks` / `FeatureTicks` **before and after** the cascade call and +records the delta — so the lower-layer buckets are slices *inside* the HermitCrab bucket, not +alongside it. Matcher self-time has nested work subtracted at the source (inside `Matcher.cs`); +feature work needs a re-entrancy depth guard or recursive `Unify` calls double-count. + +Measured outcome, so a rebuild can sanity-check itself: on Amharic `anCascade`, clone **46.7%**, +remainder **37.4%**, match 12.5%, feature 3.4%, key **0.0%**, and memo / trail / equality each +**<0.2%**. + +### Censuses + +Rule attempts, split by which guard rejected them (`guardSelector`, +`guardApplicationCount`, `guardUnifiability`, `reachedMatching`); allomorph pattern-match +attempts vs matches; clone created vs discarded vs kept. + +Die points, as an enum with a counter array **and a parallel tick array** — timing each individual +guard check, not the enclosing `Apply()`. Count and cost diverge: `RuleNotApplicableOrPatternMismatch` +was **100.0% of events** and about **0.2% of time**, because each rejection is one list index plus one +reference compare (~29 ns). Never report a die-point histogram by count alone. + +Die points, classified for the abstract-replay question — **shape-free**: synthesis-side +`MaxApplicationCount`, syntactic-feature unification, MPR features, realizational +subsumption/blocking, co-occurrence and obligatory features in `IsWordValid`. **Shape-dependent**: +allomorph pattern match / rule-not-applicable, allomorph environments, final `IsMatch`. +`LexicalLookupMiss` is neither — it precedes any synthesis step. + +--- + +## 2. The keys + +### `AnalysisStateKey` (pre-existing) + +`(Shape, Stratum, SyntacticFeatureStruct, RealizationalFeatureStruct, NonHeadCount, per-rule +un-application count multiset)`. Order-independent by design — no analysis-side rule reads trail +order, which is the redundancy it collapses. The multiset is XOR-combined, not `*31`-rolled, so it +hashes commutatively. + +**Freeze-on-read in the constructor is mandatory.** `AnalysisAffixTemplateRule.Apply` reassigns +`SyntacticFeatureStruct` to a fresh unfrozen clone *after* the owning `Word` is frozen, and that +setter has no `CheckFrozen()` guard. + +### The narrowed variant (row 4) + +Filter the count multiset to rules that are **not** provably shrinking. Two requirements learned the +hard way: build the retain-map **once per grammar**, not per key (key construction is on the hottest +path); and filter during hashing and equality rather than by materialising a filtered dictionary, so +the narrow key allocates no more than the full one. **Absent from the map means retain** — a rule the +classifier walk does not cover must be counted, never silently dropped. + +A `retainedCount` field is needed so equality can compare sizes without enumerating. + +### `SynthesisStateKey` (row 5) — and the trap + +Everything a synthesis step reads: shape+annotations, syntactic FS, realizational FS, MPR set, root +allomorph, disjunctive allomorph indices, applied-rule counts, `IsPartial`, +`IsLastAppliedRuleFinal`, stratum, **and the ordered remaining trail** — +`Word.MorphologicalRuleTrail` sliced to `[0, PendingTrailPosition]`. + +**The trail content is the whole game.** A key with `PendingTrailPosition` (an integer index) and no +trail content is fine for *measuring* and catastrophic for *skipping work*: two candidates at the +same index with different pending sequences compare equal, and merging them discards one +derivation's continuation. Every large apparent redundancy in this engine was measured with such a +key, and each collapsed when the content was added — 9,774x → 15–40%, 61-of-395,026 → 351,414-of-395,026, +3.22x/8.10x → `hits = 0`. + +**`Word.ValueEquals` is not a starting point.** It omits `_syntacticFS`, MPR features and the +disjunctive allomorph indices, all of which synthesis reads. Inheriting from it produces a silently +inflated ratio. + +Two more properties a real memo needs: stored results are **sets**, not values (realizational rules +are trail-exempt — `SynthesisRealizationalAffixProcessRule` has no `IsMorphologicalRuleApplicable` +gate — and several allomorphs of one rule can legitimately match one input before the disjunctive +break); and stored outputs embed the producing candidate's trail, so replay must **re-anchor**, the +mirror of `Word.ReplayOnto`. + +### `RuleLengthClassifier` (row 4's static half) + +Verdicts: `Shrinking` / `NonShrinking` / `Unknown`, where `Unknown` is treated identically to +`NonShrinking` by the key and kept distinct only so a census can tell "this grammar has zero +morphemes" from "this grammar has constructs we cannot analyse". + +An `AffixProcessAllomorph` shrinks iff **every** allomorph (a) inserts at least one *segment* +(`InsertSegments` counting only `HCFeatureSystem.Segment` nodes — a boundary-only insertion is +conservatively NonShrinking, since `InsertSegments.GenerateAnalysisLhs` omits boundaries from the +analysis pattern; `InsertSimpleContext` counts as 1), (b) has **every Lhs part captured** by a Rhs +`CopyFromInput` or `ModifyFromInput`, and (c) copies no part twice. + +The direction trap, which is easy to invert: on the analysis side rules are *un*-applied, so a rule +that inserts on synthesis **removes** on analysis, and a rule that truncates on synthesis +**untruncates** — an Lhs part with no Rhs copy makes the un-applied word **grow**. The authority is +`AnalysisMorphologicalTransform.GenerateShape`, which walks the Lhs parts and either copies the +captured span or calls `Untruncate` to regenerate the part from its pattern. Reduplication (a part +copied twice) and `CompoundingRule` are `Unknown`; conservatism only costs pruning opportunity. + +Measured census, for a rebuild to check against: Sena 19 of 27 rules shrinking, Indonesian 10 of 15, +Amharic 31 of 36. + +--- + +## 3. Harness design + +`[TestFixture]`, `[Explicit]`, environment-driven, writing only `TestContext` lines and no files — +this repo never commits real grammars, and derived corpus data must not leak into a committed path. +**Never print rule, stratum, morpheme or lexical-entry names, or word lists.** Counts, ratios and +timings only. Conformance fixture IDs are committed and safe. + +Env: `HC_MEMO_GRAMMAR`, `HC_MEMO_WORDS`, `HC_MEMO_MAX_WORDS`, `HC_PROBE_WORDS` (explicit +comma-separated list), `HC_PROBE_FIXTURES_ROOT`, `HC_MEMO_TIMEOUT_MS`. + +Breadth comes from `SIL.Machine.Morphology.HermitCrab.Conformance.Fixture.DiscoverAll(root)` over +`conformance/{languages,edge-cases}` — 33 grammars, each `grammar.xml` + `words.yaml` with +hand-derived expectations. Depth comes from the three real corpora. **Amharic: never exceed the first +28 words** — zero-based index 28 is pathological and burns hours. + +Run the memoized path: `new Morpher(traceManager, language, maxDegreeOfParallelism: 1)`. Tracing +disables the memo, so a traced run measures nothing. + +### A/B methodology, all of it earned by being wrong first + +Parity is the gate and speed is only reported. Acceptance is **analysis-set equality on canonical +morpheme-signature sets** (`MorpherTests.WordAnalysisSignature`, sorted) — never object or byte +equality, because a replayed `Word` is legitimately not field-for-field identical. + +Timing: **discard a warm-up rep per arm, then take min of N interleaved samples, and print the +off-arm spread beside every speedup.** Refuse any speedup smaller than its own spread. Three +failures forced each part of this — a first A/B read 1.53x that was entirely JIT; another read +2.385x with a 210.9% spread on one arm that was 1.015x serial; and adding one arm's warm-up while +discarding the other's manufactured a uniform ~2.4x across every fixture including ones with zero +shareable steps. + +**Never dispatch timing-sensitive probes concurrently.** Three at once put four testhosts on the +machine and invalidated three rows' bucket percentages. Check `tasklist | grep -c testhost` before +any timing run and report the count alongside the numbers. Counts survive contention; shares and +speedups do not. + +**Never aggregate the test logger's console output.** It prints every line twice, the second copy +space-prefixed, so a `^\s*` grep doubles every total — this published a 2x error. + +Sub-50 ms fixtures cannot carry a timing claim: one 10 ms fixture's share moved 1.3% → 14.5% +between two runs of identical code. Flag them (`reliable = wallMs >= 50`) rather than filtering them +out silently. + +### One concurrency hazard worth knowing + +A per-word timeout wrapper built on `Task` has no cooperative cancellation, so an abandoned word's +task keeps mutating engine state after the harness moves on. Harmless with a per-word +`AnalysisScope`; under a **shared** scope it collided with the next word on the same +`Memo`/`InProgress` collections and produced `IndexOutOfRangeException` inside +`MemoizedCombinationRuleCascade`. A shared-arm timeout must abort the run, not continue. Also note +`Task.Result` throws `AggregateException`, not the bare exception type. + +### Final reusable harness + +The retained `PerfBench` is narrower than the historical probes above. It accepts `HC_BENCH_GRAMMAR`, +`HC_BENCH_WORD_LIST`, `HC_BENCH_REPS`, `HC_BENCH_OUT`, and `HC_BENCH_LABEL`; it records timings, errors, +resource observations, memo diagnostics, and canonical result signatures. It does not retain the completed +round's hot-loop counters. + +Timeouts are now the caller's responsibility: start one `dotnet test` process for an arm and terminate the +whole process if its budget expires. Do not restore an in-process `Task` timeout—the abandoned parse continues +running, contaminates later samples, and can race shared engine state. `scripts/hc-bench-compare.ps1` compares +grammar and word keys symmetrically before comparing errors, signatures, timing, or spread. + +--- + +## 4. Two durable engine facts, restated here so they survive independently + +**`Word.ReplayOnto` does not splice `_mrulesUnapplied`.** It splices `_mruleApps` and `_nonHeadApps` +only. This is safe *solely* because `AnalysisStateKey` includes the count multiset, which guarantees +a memo hit has identical arrival and stored-arrival counts. **Narrowing that key breaks correctness +with no test failure.** Fix, if the key ever changes: store arrival counts on `MemoEntry` and compute +`stored - storedArrival + query` — a no-op under today's key, which is also how to test it. + +**`AnalysisScope.MaxMemoEntries` is a per-scope cap with no eviction.** Sharing one scope across a +corpus therefore makes words compete for a fixed budget: two heavy Sena words exhausted it at index +74 and 76, and every later word lost the within-word memoization a fresh scope would have given it. +Cross-word hits went *negative* and the corpus ran 28–32% slower. A shared bounded cache without +eviction is not a smaller version of the optimization. diff --git a/docs/optimizations/deferred-template-clone.md b/docs/optimizations/deferred-template-clone.md new file mode 100644 index 000000000..e5c6d5d7f --- /dev/null +++ b/docs/optimizations/deferred-template-clone.md @@ -0,0 +1,35 @@ +# Deferred per-template clone in `AnalysisAffixTemplateRule` + +**Status:** retained. **Portable:** the observation is; the saving is allocation, so the Rust port gains +little unless it also clones eagerly. + +## The observation + +`AnalysisAffixTemplateRule.Apply` made an internal engine clone and froze it once per template before trying any +slot, then ran the slot DFS on the clone. The ledger measured 66% of those clones discarded unused +(2,505 created, 1,659 discarded, Amharic) and closed the "defer" option because it missed a 70% gate by +four points, choosing "make a clone cheaper" instead. + +Reading the code shows the clone had exactly one load-bearing job. The input is already frozen at every +call site; slot rules never mutate their input (`ApplyRhs` clones `match.Input`); intermediate `Source` +pointers are overwritten by the stratum. The only path that emits the input object itself is the +all-slots-skipped terminal, and `Apply` then mutates every emitted word's `SyntacticFeatureStruct`, so +that one path needs a private copy. + +## The change + +Run the DFS on the frozen input directly (untraced path). At the two emission points that can carry the +original input through unmodified (`ApplySlots` terminal, `ParallelApplySlots` push), emit a frozen +clone when the word is reference-equal to the input. The traced path keeps the eager clone because the +trace manager stamps `CurrentTrace` onto the words it is handed. A naive "no clone anywhere" version was +tried first and failed the new tests immediately (`FeatureStruct.Add` threw on a frozen structure). + +## Measured and verified + +Alternating wall-time A/B against the pinned baseline, 45 Sena words, min of two interleaved rounds: +1.054x overall, heavy words 1.02x to 1.08x; parity clean on all words and both rounds. Smaller than the +ledger's framing suggested: the eager clone was not where the heavy words' time went. + +Focused template tests cover required/optional slots, traced execution, and both sequential and parallel +paths. The complete conformance-inclusive HermitCrab run at the final code tip passed 603 tests with one +expected host-privilege skip. diff --git a/docs/optimizations/edge-segment-prefilter.md b/docs/optimizations/edge-segment-prefilter.md new file mode 100644 index 000000000..c1a905480 --- /dev/null +++ b/docs/optimizations/edge-segment-prefilter.md @@ -0,0 +1,64 @@ +# Edge-segment prefilter for analysis affix rules + +**Status:** retained (`AnalysisAffixProcessRule`; internal same-binary toggle +`Morpher.EdgePrefilterEnabled`). **Portable:** yes; it is a property of the pattern semantics, +not of the C# engine. + +## The observation + +Every analysis affix pattern is compiled with `AnchoredToStart = true`, `AnchoredToEnd = true`, +`MatchingMethod = Unification`, and a filter that shows the matcher only Segment-type annotations. The +pattern is built from the allomorph's Lhs/Rhs: copied stem parts become groups with quantifiers, and each +inserted segment becomes a single `Constraint` node carrying that segment's feature structure +(`InsertSegments.GenerateAnalysisLhs`). A `Constraint` compiles to one mandatory arc with no epsilon +alternative (`Constraint.GenerateNfa`). + +So when the pattern's first top-level node is a `Constraint`, the traversal's first step must unify it +with the shape's first segment node, or the traversal fails; likewise the last `Constraint` and the last +segment node. The only way an annotation can be skipped is when it is marked `Optional`. + +On the heaviest Sena word, 3,564,736 pattern-match attempts produced 170,863 matches (95% failed), and +each failed attempt still ran a full nondeterministic traversal: 4,017,574 traversal instances, +9,111,459 arc checks. + +## The change + +At compile time, `AnalysisAffixProcessAllomorphRuleSpec` records `LeftEdgeConstraint` / +`RightEdgeConstraint`: the feature structure of the pattern's first / last child when that child is a +bare `Constraint` (null when it is a group, quantifier or alternation). At run time, before each +allomorph's `MultiplePatternRule.Apply`, `AnalysisAffixProcessRule.Apply` finds the shape's first and +last Segment-type nodes (skipping boundaries exactly as the matcher's filter does) and rejects the +allomorph if `edgeNode.FeatureStruct.IsUnifiable(edgeConstraint)` fails, using the same unification call +and `UseDefaults` setting the matcher itself uses. An `Optional` edge node disables the check on that +side. When tracing, the same `MorphologicalRuleNotUnapplied` event is emitted that the matcher's failure +would have produced, so traces are unchanged. + +## Measured (deterministic counts, sequential path, one parse per word, same binary) + +| 45 Sena words | prefilter off | prefilter on | +| --- | --- | --- | +| FST traversals | 12,376,416 | 1,617,662 | +| FST arc checks | 34,170,777 | 15,583,242 | +| traversal instances used | 15,642,175 | 2,429,643 | +| pattern-match attempts (counted before the check) | 11,979,916 | 11,979,916 | + +Parity: identical analysis signature sets on all 45 Sena words and on all 464 conformance-fixture words +(33 grammars). Fixture traversals fell 21% (31,278 to 24,741); their patterns fail less often. + +`AnalysisEdgePrefilterTests` covers left/right constraints, boundary skipping, optional nodes, and the +disabled path. The broader HermitCrab and conformance suites cover integration behavior; the final +conformance-inclusive run passed 603 tests with one expected host-privilege skip. + +## Why the ledger's estimate was wrong + +The ledger (row 22) capped a prefilter at ~10% using the matcher's *self-time* share. The traversal's cost +is mostly booked elsewhere (instance and register allocation, feature unification, GC), so self-time +understated it by a large factor. By counts, the prefilter removes 87% of traversals outright. + +## Limits + +After the change, ~44% of the remaining traversals succeed, so a second-segment check has little left to +reject. The rule loop still runs ~110 attempts per cascade state; depending on which safe edge constraints +exist and whether the first rejects, an attempt performs zero, one, or two prefilter unifications before any +remaining match. Indexing allomorphs by edge segment could avoid both checks and matching work, but its +headroom was not established and it was not attempted. diff --git a/docs/optimizations/feature-struct-frozen-tree-clone.md b/docs/optimizations/feature-struct-frozen-tree-clone.md new file mode 100644 index 000000000..642c1cde2 --- /dev/null +++ b/docs/optimizations/feature-struct-frozen-tree-clone.md @@ -0,0 +1,28 @@ +# Reuse or skip the feature-structure clone map + +**Status:** retained (`FeatureStruct.Clone`). +**Portable:** yes; the specific pool is .NET-oriented, while the tree proof is general. + +## Invariant + +A frozen structure marked as a flat tree has no aliases or cycles to preserve below its root. Its top-level +clone therefore cannot encounter the same value twice, so the old-to-new copies map has no semantic work to +do. Recursive cloning inside another graph must still use the caller's map even when the child itself looks +tree-shaped, because the parent graph may refer to that child elsewhere. + +## Implementation + +A top-level clone skips the map for a proven frozen flat tree. General clones borrow a per-thread map, clear +it in `finally`, and fall back to an independent map for reentrant calls. The internal recursive constructor +always honors the caller-provided map. + +## Evidence + +This removes dictionary creation and lookup from the common clone case. It travelled with the broader clone +allocation work and was not timed independently, so the record deliberately makes no isolated speed claim. + +## Correctness coverage and limits + +`FeatureStructCloneTests` covers alias preservation, cycles, reentrant cloning, and the distinction between a +top-level tree and a tree embedded in a shared graph. The complete conformance-inclusive run passed 603 tests +with one expected host-privilege skip. diff --git a/docs/optimizations/feature-struct-frozen-tree-equality.md b/docs/optimizations/feature-struct-frozen-tree-equality.md new file mode 100644 index 000000000..5378fb48e --- /dev/null +++ b/docs/optimizations/feature-struct-frozen-tree-equality.md @@ -0,0 +1,29 @@ +# Allocation-free equality for frozen flat feature structures + +**Status:** retained (`FeatureStruct.ValueEquals`). +**Portable:** yes; the proof depends on immutability and tree shape, not the CLR. + +## Invariant + +During `Freeze`, a feature structure is marked as a flat tree only when every reachable value is a +`SimpleFeatureValue`, no value instance repeats, and no nested `FeatureStruct` is present. Once frozen, that +fact cannot change. Comparing two such structures can never trigger the general algorithm's reentrancy or +cycle guard, so a feature-count/key/value comparison is equivalent to the general visited-pair walk. + +## Implementation + +`ValueEquals` uses the direct leaf comparison only when both operands are frozen and carry the tree proof. +Every other shape—including shared leaves, nested structures, and cycles—uses the general algorithm. A +test-only slow-path entry point lets randomized tests compare both implementations on the same inputs. + +## Evidence + +The optimization removes all visited-set work from the dominant frozen, flat case. It was observed in the +same profile that motivated traversal-state reuse, but was not isolated from the other feature-structure +changes; no standalone wall-time ratio is claimed. + +## Correctness coverage and limits + +`FeatureStructTreeFastPathTests` cross-checks randomized flat structures against the slow path and exercises +false comparisons and structures that must not qualify. `FeatureStructValueEqualsFreezeTests` covers shared +and cyclic graphs. The full conformance-inclusive run passed 603 tests with one expected host skip. diff --git a/docs/optimizations/feature-struct-visited-state-reuse.md b/docs/optimizations/feature-struct-visited-state-reuse.md new file mode 100644 index 000000000..d140a2728 --- /dev/null +++ b/docs/optimizations/feature-struct-visited-state-reuse.md @@ -0,0 +1,28 @@ +# Reuse feature-structure traversal state + +**Status:** retained (`FeatureStruct.ValueEquals` and `Freeze`). +**Portable:** the allocation-avoidance principle is portable; the `[ThreadStatic]` pool is .NET-specific. + +## Invariant + +Equality and freezing need visited sets only after a recursive value is reached. A top-level +operation owns its traversal state until it returns. A nested top-level call cannot borrow that same state, +so it falls back to fresh collections. + +## Implementation + +The recursive APIs accept nullable state and allocate it only when traversal requires it. Ordinary top-level +calls borrow pre-sized sets and dictionaries from a per-thread pool, clear them in `finally`, and release the +pool. The pool has an `InUse` guard so reentrant calls retain the old independent-state semantics. + +## Evidence + +The heaviest Sena profile put feature-structure equality/freezing at 13.7% of wall time, with `HashSet` +resizing a major allocation source. After this change, `ValueEqualsImpl` and those resizes disappeared from +the top of the profile. This mechanism was not timed in isolation, so no standalone speedup is claimed. + +## Correctness coverage and limits + +`FeatureStructPooledVisitedSetsTests` covers reuse, clearing after success and failure, and reentrancy. Cyclic +and shared graphs continue down the general path. The complete conformance-inclusive HermitCrab run at the +final code tip passed 603 tests with one expected host-privilege skip. diff --git a/docs/optimizations/flat-register-arrays.md b/docs/optimizations/flat-register-arrays.md new file mode 100644 index 000000000..359abb125 --- /dev/null +++ b/docs/optimizations/flat-register-arrays.md @@ -0,0 +1,25 @@ +# Flat register arrays in the FST traversal + +**Status:** retained (`SIL.Machine.FiniteState`: `Register[]` indexed by `RegisterArray.Idx(r, k)` +replaces `Register[,]` internally; the public `FstResult.Registers` stays two-dimensional and is +materialized once per accepting state). **Portable:** no; a .NET-specific allocation path. + +## The observation + +11.5% of the heaviest Sena word's CPU profile was inside `System.Array.CreateInstanceMDArray`, the slow +generic allocator for multi-dimensional arrays, called from `Matcher.AllMatches` (8.6%) and traversal +setup (2.3%): one `new Register[n, 2]` per match call and per traversal instance. + +## The change + +Mechanical: flat arrays of length `registerCount * 2`, a two-constant index helper, `Array.Copy` on the +flat struct array, and `RegistersEqualityComparer` comparing the flat layout. No behaviour change. + +## Measured + +`CreateInstanceMDArray` 11.5% → 0.2% of the profile. Counts unchanged by construction. Together with the +other allocation changes it contributed to the heaviest word's wall time; not measured in isolation. + +`RegistersEqualityComparerTests` covers equality and hashing of the flattened layout, while the existing FSA +and FST suites cover accepting results and register materialization. The complete conformance-inclusive run +passed 603 tests with one expected host-privilege skip. diff --git a/docs/optimizations/shape-copy-single-pass-map.md b/docs/optimizations/shape-copy-single-pass-map.md new file mode 100644 index 000000000..ed8c15828 --- /dev/null +++ b/docs/optimizations/shape-copy-single-pass-map.md @@ -0,0 +1,28 @@ +# Build `Shape.CopyTo`'s node map in the copy pass + +**Status:** retained (`Shape.CopyTo`). +**Portable:** yes, when a port has the same node-remapping requirement. + +## Invariant + +Annotations refer to nodes in the source shape and must be remapped to the corresponding nodes in the copy. +The source and destination nodes are already visited in matching order while cloning, so that loop can record +the mapping directly. No second `Zip`/`ToDictionary` traversal is required. An empty source range needs no +map; every nonempty range builds one because annotations are queried only after nodes have been copied. + +## Implementation + +`Shape.CopyTo` allocates the map on the first node of a nonempty range and fills it as each node is cloned. +Annotation ranges are then copied through that map exactly as before. + +## Evidence + +The change removes one enumeration and one LINQ-built dictionary from each annotated shape copy. It was not +timed independently from shape copy-on-write and feature-structure clone work, so no standalone ratio is +claimed. + +## Correctness coverage and limits + +`ShapeCopyToTests` covers empty ranges, leaf and spanning annotations, node/range remapping, and independence +of the copy. +The complete conformance-inclusive run passed 603 tests with one expected host-privilege skip. diff --git a/docs/optimizations/shape-sharing-on-clone.md b/docs/optimizations/shape-sharing-on-clone.md new file mode 100644 index 000000000..702de8f35 --- /dev/null +++ b/docs/optimizations/shape-sharing-on-clone.md @@ -0,0 +1,44 @@ +# Copy-on-write Shape sharing in `Word.Clone` + +**Status:** retained (`Word.CloneForEngine`, `EnsureOwnShape`, `ResetShape`, `Shape.CreateEmptyLike`). +**Portable:** partly. The *observation* (three of four clone sites never use the copied shape) is +engine-independent; the mechanism (shared reference to an immutable structure) is how a Rust port would +naturally write it already. + +## The observation + +A CPU profile of the heaviest Sena word (60 s sampled, memoized path) put `Word` cloning at 36% of wall, +of which `Shape.CopyTo` was 28%: every clone deep-copied every `ShapeNode`, every node's feature +structure (`SimpleFeatureValue.CloneImpl` 22%), every annotation, and built a node-mapping dictionary. +The callers: `Word.ReplayOnto` 15% (memo replay; never mutates the shape), `Word.ExpandAlternatives` 9% +(immediately *replaces* `_shape`), `AnalysisAffixProcessAllomorphRuleSpec.ApplyRhs` 5% (`GenerateShape` +immediately `Clear()`s the copy), non-head clones 2.5%, `LexicalLookup` (the `RootAllomorph` setter +replaces `_shape`). + +## The change + +- Internal engine clone: if the source shape is frozen, share the reference and mark `_shapeShared`; otherwise + deep-copy it. Public `Word.Clone()` always returns a deep, independently mutable shape. +- `EnsureOwnShape()`: clone once, on demand, before in-place mutation. Called in `AnalysisStratumRule` and + `SynthesisStratumRule` before phonological rules are applied to a fresh clone (rewrite specs insert + nodes and hold node references obtained by matching, so the un-share must precede the match). +- `ResetShape()`: replace with a new empty shape of the same configuration. Used by the four + `GenerateShape`/rebuild sites (analysis affix, analysis compounding subrule, synthesis affix, synthesis + compounding) instead of clone-then-clear. +- `ExpandAlternatives` marks the adopted shape as shared; the `RootAllomorph` setter and constructors mark + their fresh clones as owned. + +The safety net is that a shared shape is frozen: any mutation through a missed site throws +(`Shape.CheckFrozen`, frozen node feature structures; a missing guard on `Annotation.Range` was added). +During development the first full test run threw at exactly the two stratum sites, which is how they +were found; no other site surfaced across 122 HermitCrab tests, 464 fixture words and 45 Sena words. + +## Measured + +Profile after the change: `Word` clone 36% → 9% of wall, `Shape.CopyTo` 28.6% → 13% (the remainder is +the stratum's `EnsureOwnShape` and part copying inside `GenerateShape`), `ReplayOnto` 21.7% → 6.5%. Peak +working set on the heaviest word 11 GB → 6 GB. Wall-time attribution is unreliable on this machine (see +the round document); counts are unchanged by construction. + +`WordShapeSharingTests` covers public-clone independence, copy-on-write, reset paths, and missed-mutation +guards. The final conformance-inclusive run passed 603 tests with one expected host-privilege skip. diff --git a/docs/optimizations/syntactic-feature-structure-sharing.md b/docs/optimizations/syntactic-feature-structure-sharing.md new file mode 100644 index 000000000..40ff2af22 --- /dev/null +++ b/docs/optimizations/syntactic-feature-structure-sharing.md @@ -0,0 +1,32 @@ +# Copy-on-write syntactic feature structures in engine clones + +**Status:** retained (`Word.CloneForEngine`, `EnsureOwnSyntacticFeatureStruct`). +**Portable:** partly; immutable sharing is general, while the ownership plumbing is specific to this engine. + +## Invariant + +HermitCrab freezes a `Word` before it becomes an input to the analysis or synthesis graph. Internal clones may +therefore share that frozen word's syntactic feature structure until a rule needs to mutate it. The public +`Word.Clone()` contract remains a deep, mutable clone and never exposes copy-on-write sharing to callers. + +## Implementation + +Internal engine sites use `CloneForEngine()`. A per-`Morpher` setting is copied onto new words and controls +whether those clones share a frozen syntactic feature structure. The three in-place analysis mutation sites +call `EnsureOwnSyntacticFeatureStruct()` before modification; assignments of a fresh structure need no guard. +The opt-out is internal and captured per `Morpher`, preventing one parser or test from changing another +existing parser's behavior. + +## Evidence + +The heaviest Sena profile attributed roughly 15% of CPU to freezing and hashing a newly cloned syntactic +feature structure while constructing analysis keys. Sharing removes that repeated deep clone/freeze on the +engine path. The final five-language measurement is branch-wide rather than an isolated toggle result, so no +standalone speedup is assigned here. + +## Correctness coverage and limits + +`WordSyntacticFsSharingTests` covers copy-on-write, nested/shared structures, public-clone mutability, and +isolation between morphers. The broader HermitCrab suite exercises integration behavior and passed both with +sharing enabled and with `HC_SHARE_SYNTACTIC_FS=false`; the full conformance-inclusive run passed 603 tests +with one expected host skip. Sharing is intentionally unavailable for mutable source words. diff --git a/docs/rejected-optimizations/alternatives-memo-and-hoist.md b/docs/rejected-optimizations/alternatives-memo-and-hoist.md new file mode 100644 index 000000000..419cba7ef --- /dev/null +++ b/docs/rejected-optimizations/alternatives-memo-and-hoist.md @@ -0,0 +1,15 @@ +# `ExpandAlternatives` memoization and hoisting + +**Status:** rejected and removed from production. + +The experiment memoized `Source` recursion during `Word.ExpandAlternatives` and computed an analysis word's +expansion once outside the per-root-candidate loop. It was exact, but on the 45-word Sena battery it left word +clones unchanged at 4,747,013: deeper strata did not branch, so it saved only small temporary list work. + +A later change moved work back behind successful lexical lookup, but after the experimental memo/hoist was +removed the final product path is the same ordering as the pinned production baseline: each lexical candidate +expands its synthesis word. There is therefore no retained alternatives optimization in this branch, and the +earlier 75% “alternatives expanded” reduction must not be attributed to the final diff. + +The broader deduplication avenue remains closed by the complete-key census in the main ledger: once pending +trail content is included, only 11% of alternatives duplicate, for a 2.28% wall-time ceiling on Sena. diff --git a/docs/rejected-optimizations/compounding-split-prefilter.md b/docs/rejected-optimizations/compounding-split-prefilter.md new file mode 100644 index 000000000..067fbd07d --- /dev/null +++ b/docs/rejected-optimizations/compounding-split-prefilter.md @@ -0,0 +1,11 @@ +# Compounding split prefilter + +**Status:** rejected before implementation; temporary counters removed. + +The candidate was to query lexical reachability before materializing compounding splits. A census on the +heaviest Sena probe counted 35,176 splits, 5,524 root candidates, and 1,121 outputs. That work represented +about 2% of word clones and 9% of FST traversals—too small a ceiling for an additional indexing and correctness +surface, especially because real compounding prevents the simpler grammar-wide lexical gate. + +No production change was built. Reopen only if a different grammar shows compounding split construction as a +dominant measured bucket and supplies a sound, stratum-aware reachability predicate. diff --git a/docs/rejected-optimizations/final-template-prune.md b/docs/rejected-optimizations/final-template-prune.md new file mode 100644 index 000000000..a6f0002b5 --- /dev/null +++ b/docs/rejected-optimizations/final-template-prune.md @@ -0,0 +1,25 @@ +# Final-template interleaving prune + +**Status:** rejected and removed from production. **Portable:** the proof is portable where its guard holds. + +The experiment mirrored synthesis's rule that a non-template affix process normally cannot follow a final +template. Analysis could skip the inverse interleaving when every relevant template was final and the grammar +contained no partial morphemes. + +The available real grammars defeated the useful guard: Sena has 25 partial lexical entries and six partial +rules; Amharic and Mbugwe each have a partial rule. The prune fired zero times on all three. Synthetic tests +proved that the rule could fire safely, but that establishes correctness, not value. Carrying production +branches, state, toggles, and counters for a zero-effect optimization violated the final minimum-surface goal, +so all of them were removed. + +Reopen only for a corpus with the reported pathological final-template interleavings and no partial +morphemes, or with a proved narrower per-state partiality condition. Measure real prunes before shipping. + +## Addendum 2026-09-03 +Superseded in part. The same prune, measured with the partiality guard lifted, is 8-33x on Mbugwe and Sena with +identical analysis sets (298x on the single heaviest Sena word). The guard, not the mechanism, is what made it +inert: partial *lexical entries* can never rescue a pruned branch, because synthesis applies no template to a +partial root and then fails the candidate with `PartialParse`, so a rules-only, per-stratum guard is strictly +stronger and still sound. Reopen on that basis rather than on the corpus counts above. The full measurement +record is `docs/pr491-final-template-prune-upper-bound.md` on the `perf/hc-optimization-archive` branch, which +is outside this docs set. diff --git a/docs/rejected-optimizations/two-pass-nondeterministic-traversal.md b/docs/rejected-optimizations/two-pass-nondeterministic-traversal.md new file mode 100644 index 000000000..801af1c30 --- /dev/null +++ b/docs/rejected-optimizations/two-pass-nondeterministic-traversal.md @@ -0,0 +1,14 @@ +# Two-pass nondeterministic traversal + +**Status:** measured negative and removed from production. + +This port of the `investigate_linear_traverse` approach first merged traversal instances by state, position, +and variable bindings, then rebuilt captures. HermitCrab's existing traversal already pools instances, and +its patterns do not have the repeated-group shape needed to repay the extra bookkeeping. + +On the 45-word Sena battery the alternative used 2.3 times as many traversal instances, reduced arc checks by +only 1.6%, produced more duplicate matches, and increased word clones by 27%. Conformance fixtures pointed in +the same direction. Toggle hooks, alternate algorithms, and counters were removed; only this result remains. + +Reopen only with a corpus whose automata have substantial equivalent active-state convergence and a census +showing that capture reconstruction is cheaper than the current pooled traversal. From 9914755c31ca1ec695065cd8c4466f3f58793ee5 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 18 Sep 2026 21:00:17 -0400 Subject: [PATCH 02/32] docs: record the state+position traversal dedupe round PR #511 proposes skipping traversal instances whose (State, AnnotationIndex) was already pushed. The motivating pathology is real and reproduces on a 2-state fsa: Advance forks per Optional annotation, so instances are exponential in optional count and independent of the state count. The key is too coarse. A 20,000-case differential fuzz finds 112 cases where Match() changes, 108 from ignored variable bindings and 4 from shortened ranges on the deterministic path, with AllMatches().First() identical in all 20,000 as the control. A narrowed gate -- deterministic method, no capture groups -- measures 0 divergences with the full bound preserved. Two censuses close the extensions: environment matchers are 0.20%/1.18% of traversal instances on Amharic/Mbugwe, and the analysis rules' apparent 93.95% collapse is 99.7% alternate captures, because AnalysisAffixProcessRule and AnalysisCompoundingRule set AllSubmatches and enumerate morph boundaries on purpose. That makes a fourth row for the apparent-vs-sound table. Co-Authored-By: Claude Opus 5 --- docs/hermitcrab-optimization-ledger.md | 4 +- .../state-position-traversal-dedup.md | 95 +++++++++++++++++++ .../two-pass-nondeterministic-traversal.md | 4 + 3 files changed, 102 insertions(+), 1 deletion(-) create mode 100644 docs/rejected-optimizations/state-position-traversal-dedup.md diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index 28443b33f..e6c3acabe 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -85,6 +85,7 @@ copy-on-write design through `CloneForEngine`; see `docs/optimizations/shape-sha | 11 | Tandem lexical intersection (T2) | Kill doomed branches early via the lexicon | The oracle only sees lexical unreachability. The expensive words fail on checks that run *after* lexical lookup succeeds | Pooled **23.5%** dead steps vs a 30% gate; one failure word at exactly **0.0%** dead | | 21 | **Memo-key caching on the frozen `Word`** | Keys are built ~28x per distinct state, which looked like a cheap win. | High count did not imply high cost; caching would add state and invalidation complexity around negligible work. | **CLOSED:** all `AnalysisStateKey` construction and hashing on Amharic measured **3 ms of 81,707 ms = 0.0%**. | | 12 | Pool small short-lived collections | Fewer allocations, faster | `HashSet/Dictionary.Clear()` is O(capacity); Gen0 already beats pooling at this size | -15–17% bytes but **+8.6% wall**, +12% on the parallel battery. Reverted | +| 23 | **`(State, AnnotationIndex)` traversal dedupe** (PR #511) | `Advance` forks per `Optional` annotation, so instances are exponential in optional count and independent of `|States|`; keying on state+position bounds it at `N * |States|` | The key omits what filters the traversal. Variable bindings gate later arcs; registers carry the captures `AllSubmatches` consumers read | **112 of 20,000 differential-fuzz cases change `Match()`** (108 bindings, 4 shortened ranges) while `AllMatches().First()` is identical in all 20,000. Narrowed to the deterministic method with no capture groups: **0 divergences**, full bound preserved. Extension censuses: environment matchers are **0.20%/1.18%** of instances (Amharic/Mbugwe); the analysis rules' apparent **93.95%** collapse is **99.7% alternate captures**. See `rejected-optimizations/state-position-traversal-dedup.md` | ## Closed before building — motivation refuted by measurement @@ -109,13 +110,14 @@ copy-on-write design through `CloneForEngine`; see `docs/optimizations/shape-sha ## The lesson that generalises -Three independent measurements, three boundaries, same collapse: +Four independent measurements, four boundaries, same collapse: | measurement | key used | apparent | sound | | --- | --- | --- | --- | | synthesis-input dedupe (#6) | order-insensitive | 9,774x | 15–40% | | fold-entry census | trail position only | 6,476x | not established | | fold-step sharing (#5) | trail position only | 3.22x / 8.10x | **hits = 0** | +| FSA traversal dedupe (#23) | state + position | 93.95% collapsible | **0.28%** register-identical | **The redundancy in HermitCrab's synthesis is apparent, not real. The trail is what makes each step distinct, and every measurement showing large shareable work is measuring a key that omits diff --git a/docs/rejected-optimizations/state-position-traversal-dedup.md b/docs/rejected-optimizations/state-position-traversal-dedup.md new file mode 100644 index 000000000..40a9fb66a --- /dev/null +++ b/docs/rejected-optimizations/state-position-traversal-dedup.md @@ -0,0 +1,95 @@ +# `(State, AnnotationIndex)` traversal deduplication + +**Status:** rejected in its general form; a narrowed form is sound and was measured effective. + +The idea, proposed independently as PR #511: inside `Traverse`, skip any newly produced traversal +instance whose `(State, AnnotationIndex)` pair was already pushed, on the grounds that the surviving +instance differs only in its registers and "registers only record the match, they do not filter the +traversal." + +## The motivating pathology is real + +The cost driver is not the state count. `TraversalMethodBase.Advance` forks a copy of the instance for +every `Optional` annotation at the next offset, so instance count grows exponentially in the number of +optional annotations along the input and is independent of `|States|`. A synthetic microbenchmark on a +**2-state** FSA measured 2,097,150 instances popped at 20 optional annotations (~2^(k+1)); at k>=24 it +exceeds a 5,000,000 budget. The originally reported "200,000 traversals on a 4-state fsa" is reproduced +and exceeded on fewer states. With the skip in place, the same cells are bounded at `N * |States|` +(measured ratio 0.5-1.0x), turning non-terminating cells into instant ones. + +## Why the general form is unsound + +A differential fuzz — 20,000 random pattern/input pairs, fixed seed, identical generator compiled against +both arms, comparing success, range, every group range and every variable binding — found **112 cases +where `Match()` differs from production**, while `AllMatches().First()` was identical in all 20,000. That +control pins the divergence to exactly the path the skip changes. + +- **108 cases: variable bindings.** Instances carry live `VariableBindings`, which filter later arcs + through `CheckInputMatch`; the key omits them. `Matcher.Compile` determinizes only variable-free + patterns, so every alpha-variable rule runs the nondeterministic method where the key is blindest. + Minimal case: pattern `high=$v0+` anchored both ends over 5 annotations, production matches `[0,5)`, + the skip returns no match at all. +- **4 cases: capture registers, on the deterministic path.** All are alternation plus capture groups and + all shorten the match *range*, e.g. `(g0(back=back+)|g1(high=high+back=back+))` going from `[0,4)` to + `[0,1)`. So restricting to the deterministic method is not by itself a sufficient guard. + +Both committed suites stay green under the unnarrowed change (826/0/3 and 98/0/0), so the existing tests +cannot see any of this. + +## The narrowed form that is sound + +Gate the skip on the deterministic method **and** the absence of capture groups, and leave the +nondeterministic method untouched: + +```csharp +bool dedupeByState = !allMatches && Fst.GroupNames.Count() <= 1; +``` + +Measured: fuzz divergences **112 -> 0**; the group-free microbenchmark sweep cell-for-cell identical to +the unnarrowed change across all 18 cells, so the full bound is preserved; grouped patterns fall back to +production behaviour exactly; both suites unchanged. `GroupNames.Count()` should be hoisted out of +`Traverse` — it is O(1) via the `ICollection` fast path but is evaluated once per call. + +## Where it cannot be extended, and why + +Two censuses attributed every FSA traversal call to its constructing matcher and counted, per category, +how many pushes a `(State, AnnotationIndex)` key would have collapsed — and critically, whether the +collapsed instance's registers were identical to the surviving one's. + +*Environment matchers* (the subject of issue #515) read only `.Success` and `.VariableBindings` from their +matches, never a range or group, so they looked like the ideal target. They are 0.20% of traversal +instances on Amharic and 1.18% on Mbugwe. Removing every redundant environment push would be worth 0.12% +and 0.004% of total pushes respectively. Not built. + +*Analysis-side morphological rules* hold nearly all the volume and all the apparent opportunity: +2,801,478 of Amharic's 2,808,088 pushes, 93.95% collapsible; Mbugwe 14.44%. Labelling the remaining +construction sites showed the entire figure is exactly `AnalysisAffixProcessRule` and +`AnalysisCompoundingRule` — both built with `AllSubmatches = true` and calling `AllMatches` precisely to +enumerate every distinct morph-boundary placement. Register comparison at each would-be collapse: + +| matcher | identical registers | differing registers | +| --- | --- | --- | +| Amharic `analysis-affix` | 7,446 (0.29%) | 2,573,312 (99.71%) | +| Amharic `analysis-compound` | 16 (0.03%) | 50,920 (99.97%) | +| Mbugwe `analysis-affix` | 3,022 (5.49%) | 52,056 (94.51%) | +| Mbugwe `analysis-compound` | 1,886 (2.93%) | 62,396 (97.07%) | + +The genuinely free slice is 0.28% of collapses on Amharic and 4.11% on Mbugwe — at most 0.6% of total +traversal instances, reachable only by adding a per-instance register-equality guard. `RootAllomorphTrie`'s +raw FST collapses 0% despite ~62k pushes, which is correct rather than a missed opportunity: a prefix trie +is already maximally state-shared at construction. + +Adding `VariableBindings` to the key costs nothing on this evidence: `collapsedWithBindings == collapsed` +on every row of both grammars, with a distinct-signature counter confirming the signature function +discriminates. The information that differs lives in the registers, which is exactly what `AllSubmatches` +consumers read. + +## Relationship to `two-pass-nondeterministic-traversal.md` + +That experiment merged on state, position **and bindings** and then rebuilt captures, and was measured +negative largely on the rebuild cost. This one merges on state and position only and keeps the first +instance's captures, which is cheaper and unsound rather than expensive and sound. The two are not +alternatives to the same design; they are the two ways of getting the key wrong in opposite directions. + +**Reopen only** with a grammar where the identical-registers split looks materially different from the +four rows above, or for the narrowed group-free form, which needs no further evidence to be safe. diff --git a/docs/rejected-optimizations/two-pass-nondeterministic-traversal.md b/docs/rejected-optimizations/two-pass-nondeterministic-traversal.md index 801af1c30..344a6bd0e 100644 --- a/docs/rejected-optimizations/two-pass-nondeterministic-traversal.md +++ b/docs/rejected-optimizations/two-pass-nondeterministic-traversal.md @@ -12,3 +12,7 @@ the same direction. Toggle hooks, alternate algorithms, and counters were remove Reopen only with a corpus whose automata have substantial equivalent active-state convergence and a census showing that capture reconstruction is cheaper than the current pooled traversal. + +**See also** `state-position-traversal-dedup.md`, which drops bindings from the key instead of keeping them +and skips capture reconstruction entirely. It is cheaper than this experiment and unsound for the same +reason this one was expensive: the captures are load-bearing for `AllSubmatches` callers. From 4faff4584e20a350f2e1976ac786f69366390e5c Mon Sep 17 00:00:00 2001 From: John Lambert Date: Wed, 21 Oct 2026 10:42:43 -0400 Subject: [PATCH 03/32] docs: research on the asymptotics of the analysis search Co-Authored-By: Codex --- docs/hermitcrab-optimization-ledger.md | 9 + .../hermitcrab-search-asymptotics-research.md | 368 ++++++++++++++++++ 2 files changed, 377 insertions(+) create mode 100644 docs/hermitcrab-search-asymptotics-research.md diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index e6c3acabe..ed8c839e5 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -29,6 +29,15 @@ expensive part was never the code; it was learning what to measure and which mea cost for a measurement run and not one to carry in product code indefinitely. Rebuilding a probe from a branch is cheap; maintaining research scaffolding in two product assemblies is not. +## Unmeasured search-asymptotics research + +[Analysis-search research](hermitcrab-search-asymptotics-research.md) integrates six independent reports, +checks their code claims, and separates new experiments from this ledger's closed families. It ranks +residual lexical viability, a grammar-compiled template battery, recoverable trace-class reduction, and +exact finite-state compilation for a verified subgrammar. It maps them to lexicon-driven pruning during +unapplication and making the affix-template battery cheaper without memoization. **No new measurement, +speedup, or numbered optimization row is claimed.** + ## Final disposition of the September follow-up The follow-up branch does deliver product improvements. Its authoritative summary is diff --git a/docs/hermitcrab-search-asymptotics-research.md b/docs/hermitcrab-search-asymptotics-research.md new file mode 100644 index 000000000..48951c95f --- /dev/null +++ b/docs/hermitcrab-search-asymptotics-research.md @@ -0,0 +1,368 @@ +# HermitCrab analysis search — asymptotic research + +**Disposition: count ceilings before building.** The first two candidates below fit the two open work +items without a search-sized memo. Partial-order reduction has a narrow correctness route and poor prior +coverage. Exact finite-state compilation is a larger, restricted-subgrammar project. None is a measured +optimization, and none establishes a polynomial parser for arbitrary FieldWorks grammars. + +This integrates the six supplied reports: **R1** partial-order reduction and rewriting; **R2** packed forests +and subset DP; **R3** finite-state methods; **R4** search and symbolic sets; **R5** enumeration and graphs; +**R6** abstract interpretation and morphology prior art. The reports are research inputs, not independent +measurements. The [ledger](hermitcrab-optimization-ledger.md), its +[September disposition](hermitcrab-perf-2026-09-disposition.md), and the +[probe design](hermitcrab-probe-design.md) remain the measurement record. + +## The problem in one page + +The motivating case is **a grammar with 16 extra affix processes in one Unordered stratum**. Analysis +starts at the surface, unapplies phonology, then interleaves morphological-rule unapplication with affix +templates. Every candidate can stop for lexical lookup; candidates from intermediate strata also enter +the lookup/synthesis pipeline. A found lexical entry supplies its root allomorphs, each candidate's +alternatives are expanded, and forward synthesis must both pass validity checks and regenerate the surface +form [C1, C4, C5]. + +There are three different sources of growth: + +- **Rule orders.** The combination cascade restarts its rule loop at zero after every successful + unapplication. With `k` enabled, single-use rules, there can be `k!` full orders and + `sum(j=0..k) k!/(k-j)!` ordered prefixes. Multiple application is enabled; ordinary affix/compound + unapplication checks each rule's limit rather than a cascade-wide used-rule mask. C# realizational + unapplication has no such count guard, so a finite count-vector bound needs an eligibility proof for + the rule kinds included [C1, C2]. Allomorph branches and compounding add further choices. The 16 added + processes are not necessarily the total `k`. +- **Optional sites.** Inverse deletion inserts optional nodes into one phonological shape; it does not + immediately allocate every subset shape. Later matching still has consume/skip choices. The Rust root + trie already merges equal active trie positions. General morphological matching also returns captures + and variable bindings, so its branches cannot be merged on position alone [C6, C7]. Optionality can + therefore multiply matcher work even while the shape representation stays compact. +- **Alternatives.** An analysis-state key deliberately omits trail order. The stratum merge keeps + alternatives, and expansion grafts subsequent changes onto their source histories. Forward synthesis + reads the pending rule in order. A shared shape or rule multiset is not a shared synthesis continuation + [C3–C5]. One final analysis does not imply that only one trail needs checking. + +All numbers here are **supplied by the common brief, not reproduced in this job**: + +| Aggregate | Reported value | +| --- | --- | +| Baseline parse time | approximately 15–60 ms per input in both implementations | +| Time after the 16 additions | 17–22 s for one expensive input in Rust | +| Lexical lookups / distinct shapes | 97,575 / 56, from trace mode | +| Distinct canonical states | 10,079; they differ in shape or per-rule counts | +| Template-battery runs / distinct input states | 43,641 / 4,988; approximately 2.9 s | +| Lookup + alternative expansion + synthesis | approximately 6.5 s | +| Alternatives synthesized / distinct final analyses | 367,278 / 1 | +| Compounding bound in the case | `MaxStemCount = 2` | +| Deletion-rule variants | 200–1,600 inputs reach a 1 s per-input timeout | + +These are ceilings to investigate, not interchangeable denominators. Tracing disables the ordinary +analysis merge and C# memoization [C3, C4]; the trace lookup count cannot establish an untraced duplicate +rate. Nor do 56 shapes establish 56 lookup keys across strata. The apparent battery repetition is at most +38,653 calls (88.6%) under the brief's identity, before checking that identity and the cost of avoiding it. +The 367,278-to-one ratio proves neither order invariance nor a synthesis-sharing opportunity. + +## Verified code anchors + +Line numbers below were read directly, not copied from the reports. **C#** means the requested main +checkout, commit `a4b29742b6274a01c7398ca3e800a19fa6d2c9aa`; paths are relative to +`src/SIL.Machine.Morphology.HermitCrab/` unless qualified. The six central files (`AnalysisStratumRule`, +`AnalysisStateKey`, `MemoizedCombinationRuleCascade`, `AnalysisAffixTemplateRule`, `Morpher`, `Word`) are +text-identical in this PR worktree's starting commit `5bbfa018bdbd5228c3f1aa73db951a4f01d167d3`, ignoring +line endings. **Rust** paths are relative to the supplied checkout's `rust/crates/`, with branch tip +`fa3bfb186852baaaf1ed28fcc0ba4602b2c19e96` at inspection. This identifies inspected files; it is not a +claim that either external checkout was clean. + +| Anchor / fact | C# file:line | Rust file:line | +| --- | --- | --- | +| **C1 — cascade and template interleaving** | `AnalysisStratumRule.cs:24–61,128–137,174–244`; base combination loop: `src/SIL.Machine/Rules/CombinationRuleCascade.cs:25–51` | `pg-rules/src/cascade.rs:219–265`; `pg-rules/src/stratum.rs:811–888`; strata/results: `pg-parse/src/morpher.rs:701–765` | +| **C2 — application/compound guards and feature effects** | `MorphologicalRules/AnalysisAffixProcessRule.cs:39–63`; `MorphologicalRules/AnalysisCompoundingRule.cs:39–77,129–133`; realizational exception: `MorphologicalRules/AnalysisRealizationalAffixProcessRule.cs:40–58` | `pg-rules/src/stratum.rs:707–724,792–805` | +| **C3 — existing sharing and actual keys** | `AnalysisStateKey.cs:31–36,60–85,94–104`; merge: `AnalysisStratumRule.cs:130–167`; inner memo: `MemoizedCombinationRuleCascade.cs:39–94`; battery memo: `AnalysisStratumRule.cs:196–218` | key fields: `pg-rules/src/analysis_state_key.rs:22–47`; key construction: `pg-rules/src/stratum.rs:664–694`; ordered output key: `pg-rules/src/word.rs:309–321,410–433`; stratum fold: `pg-rules/src/stratum.rs:1180–1239` | +| **C4 — lexical lookup, expansion, acceptance, fallback** | `Morpher.cs:167–201,346–367,417–445,634–665`; expansion: `Word.cs:445–486` | `pg-parse/src/morpher.rs:459–499,765–826,933–997`; expansion: `pg-rules/src/word.rs:487–548` | +| **C5 — ordered trails are observable** | `Word.cs:202–215,253–259,339–347,371–382,627–646` | `pg-rules/src/word.rs:318–319`; pending-rule test and consumption: `pg-rules/src/stratum.rs:1546–1571,1606–1609` | +| **C6 — optional nodes and root-trie sharing** | `PhonologicalRules/NarrowAnalysisRewriteRuleSpec.cs:40–59`; optional insertion argument: `src/SIL.Machine/Annotations/Shape.cs:160–174` | deletion sites/insertion: `pg-rules/src/rewrite.rs:2536–2581`; root trie consume/skip/dedup: `pg-parse/src/root_trie.rs:227–277` | +| **C7 — general matcher work carries captures** | `MorphologicalRules/AnalysisAffixProcessRule.cs:24–33`; capture-dependent reconstruction: `MorphologicalRules/AnalysisMorphologicalTransform.cs:61–100`; optional forks/registers/bindings: `src/SIL.Machine/FiniteState/TraversalMethodBase.cs:248–308` | template slot results and their semantic deduplication: `pg-rules/src/stratum.rs:1057–1075` | +| **C8 — template admission and slot semantics** | `AnalysisAffixTemplateRule.cs:18–73`; non-disjunctive union: `src/SIL.Machine/Rules/RuleBatch.cs:52–61`, configured at `AnalysisStratumRule.cs:27–30` | `pg-rules/src/stratum.rs:894–962,990–1075` | +| **C9 — proposer is followed by exact-engine confirmation** | exact confirmation target: `Morpher.cs:346–367` | current emitter: `pg-foma/src/emit.rs:2615–2675,2704–2745,2768–2808`; confirmation filters/reparse: `pg-foma-runtime/src/confirm.rs:498–517` | + +In particular, the C# key contains shape, stratum, syntactic and realizational features, non-head count, +and exact per-rule unapplication counts. It contains neither final-template state nor a separate morphology +history. Rust adds both, saturates counts at each rule's maximum, and clears its analysis final-template +state at the stratum fold (`pg-rules/src/stratum.rs:1196–1200`). The same key name does not imply the same +equivalence relation [C3]. Neither key is a synthesis key. This inspection verifies these fields and call +sites; it does not certify a new key as complete for every continuation. + +There is a second relevant implementation difference: Rust's common dispatch applies the count gate and +pushes a pending-rule entry for every morphological rule kind. C# realizational unapplication has no count +gate and records counts without adding a pending-trail entry [C2, C5]. These are verified bookkeeping +differences, not a demonstrated output divergence here. Any proof using bounded counts or trail identity +must address the C# semantics explicitly; copying Rust's control vector is insufficient. + +## Ranked candidates + +Rank is the order in which to spend research effort under the memory constraint, not a promised speedup. +The reference labels point to the bibliography below; **every literature citation is from memory and was +not checked against a publication or online**. + +| Rank / technique | Home field + reference | Mapping in C# / Rust | Exact-parity condition; check and fallback | Expected asymptotic change | Cheapest ceiling experiment | Difference from closed rows | +| --- | --- | --- | --- | --- | --- | --- | +| **1. Grammar-compiled residual viability**; includes lexical goal propagation, abstract interpretation, and residual lexical coaccessibility | Program analysis / demand-driven databases / automata; [L1, L2, L5] | Before recursive expansion in C1/C2, with summaries of templates C8, optional nodes C6, and lexical targets C4; Rust `combination_rec` and `apply_one_mrule` | **Static:** sound transfer summaries for all remaining operations and reachable strata/roots, including compounds. **Runtime:** prune only a proved-empty abstract continuation. Unsupported operation, budget exhaustion, guessing/overlay not covered: unknown, ordinary search | Fewer explored prefixes; a certified precedence restriction can change `k!` orders to `L` viable linear extensions. No universal worst-case improvement | Shadow checks at C1/C2: count provably dead branches and descendant rule/template/lookup work, unpooled; count summary size and query work | Rows 10/11 reopen only with residual transformations **and compounding** covered; root-reachable synthesis failures remain. This is neither row 8's bare-root length bound nor row 19's feature-only synthesis replay | +| **2. Compile the template battery into a shared-prefix control trie** | Grammar factoring / parsing / finite-state compilation; [L3, L5]. A specialization of the reports' compiled-control ideas, not a reported measurement | C8: replace repeated reverse slot walks inside C1's battery. Rust `run_template_batch`, `template_unapply_slots`, `apply_slot_batch` | **Static:** share only identical slot-batch operations and optionality, with per-template admission and terminal labels retained. **Runtime:** identical input to a shared prefix; preserve all branch outcomes and existing dedup semantics. Trace, unsupported effects, or control-size budget: original battery | For `t` templates sharing prefix work `W_p`: `t·W_p` becomes `W_p`, plus divergent branches and terminal bookkeeping. For a deterministic `p`-slot prefix, this is `t·p` to `p`. No reduction of unrelated rule permutations | Compile-only census of distinct reversed slot prefixes; at C8 count attempts/outcomes at common prefixes, admitted templates, skip-only terminals, clones, and maximum DFS depth. Separate per-battery savings from avoiding battery calls | No `(Word,state)` result cache: sharing is in the grammar's control trie. Does not repeat rows 1/2/18 or the alternatives memo/hoist. Deferred cloning and the final-template filter already have separate records | +| **3. Trace-class POR with complete ordering recovery**; critical-pair analysis supplies certificates | Concurrency / term rewriting; [L4] | C1 unordered recursion; class readout at C4/C5. Same boundaries in Rust `combination_rec` and the expansion/synthesis consumer | **Static:** strong independence with enabledness and branch/effect correspondence, not just equal final shapes. **Runtime:** stack-local sleep sets; retain every class's ordered histories and all stopping/template points. Uncertified pair is dependent; unhandled rule kind uses ordinary search | Ordered-prefix exploration becomes trace-class exploration; fully independent single-use, single-outcome rules give `2^k` classes and `O(k·2^k)` transition attempts. Recovery/synthesis can still require `k!` trails | Count certified pairs **where enabled**, classes, and prefix/template work avoided; separately count recovered trails and synthesis. Earlier eligibility was only 0.00–0.05% of attempts in the supplied research context | Recovers the orders discarded by the rejected commuting-order canonicalization. Skipping class members without a full synthesis/output-invariance proof repeats row 6; merging synthesis steps repeats row 5 | +| **4. Exact finite-state compilation of a verified subgrammar** | Computational morphology / transducers; [L5] | Compile C1/C2/C6/C8 and the lexicon/forward checks C4 into a labelled relation; Rust emitter C9 is a starting point for coverage inspection, not an exact replacement | **Static:** exact regular encoding of features, annotations, root/non-head identity, all ordered outputs, copying and rule limits; unsupported construct or compile-size overflow rejects eligibility. **Runtime:** budget checked before allocating input-product storage; ordinary parser on refusal | Control alone: `prod(i)(m_i+1)` states, at most `k` outgoing choices per state; for single use `2^k` states / `O(k·2^k)` arcs. Full relation and output enumeration may still be exponential/factorial | Coverage audit and compiled states/arcs first; then accepted labelled paths, confirmation work actually removed, distinct complete analyses, and bytes. Compare against 367,278 trails, not just one output | Shares a **grammar-compiled** relation without erasing order or adding a per-search chart. A permissive proposer that still reparses has not removed the cascade | + +## 1. Residual viability: prune only a proved-empty continuation + +Let `S` be a concrete HermitCrab internal `Word` plus cascade/stratum control. Let `Reach(S)` include +every possible stopping candidate after the remaining inverse operations, including lookup in intermediate +strata and every permitted compound split. Choose a finite abstract domain, an abstraction `alpha`, and +transfers satisfying `alpha(T(S)) <= T#(alpha(S))` for **sets** of results. Compute a conservative closure +of the remaining transfers. If that closure has empty intersection with an over-approximation of all +compatible lexical targets, no concrete continuation can find a root. Removing that branch therefore +removes no successful synthesis and no reported analysis. This follows by induction over each concrete +continuation; it needs no assumption that rule order commutes. + +The target set must include everything ordinary lookup could supply. Compound non-head lookup is already +inside unapplication [C2]; it cannot be ignored or treated as a single final root. Root guessing must +either have its own sound target abstraction or retain the original candidates for the fallback branch +[C4]. Rust supplied-root overlays also affect targets (`pg-parse/src/morpher.rs:459–499`). A finite list of +ordinary root spellings is insufficient in either situation. + +Start with a grammar-compiled coarse domain: mandatory/optional segment-length intervals, safe segment +classes, and only feature constraints whose transfer is proved. Inverse truncation can **grow** a shape: +`MorphologicalRules/AnalysisMorphologicalTransform.cs:54–80,108–132` reconstructs uncaptured input parts. +Optional nodes require both skip and consume possibilities [C6]. An unsupported transform maps to top, +never to failure. Staged +refinement is useful only if every stage remains an over-approximation; an inconclusive expensive stage +returns unknown. This is the safe part of R6's abstract interpretation, R4's magic sets, R3's lazy product, +and R5's extension-oracle proposals. They are one pruning family, not four independent wins. + +R4's precedence idea is a special case with a stricter obligation. To install `r before s`, prove that +**every successful complete path containing both occurrences** respects that order, including paths with +intervening enabling rules/templates. Failure of the adjacent two-step sequence `s;r` is insufficient. +Rules are optional: `s` without `r` must remain allowed. A runtime implementation can forbid a later `r` +after `s` only under that complete-path certificate; it cannot require `r` before admitting `s` altogether. +Begin with single-use rules, no certified edges for unknown constructs, and no inference from prefix/suffix +direction alone. A cycle says that the affected combination is impossible, not that individual rules +should be deleted. + +This must remain grammar-sized: use finite transfer tables and stack-local query work, not a table keyed +by every visited `S`. A lazy lexical product that needs an unbounded visited-state set fails the memory +constraint; use a statically bounded abstraction or return unknown. Count dead **subtrees and their work** +without double-counting descendants. The ledger's 23.5% pooled dead-step result and zero on one expensive +input are reasons to stop if the stronger oracle still mostly accepts. No lexical oracle can reject a +root-reachable path merely because later synthesis will fail. + +## 2. Make the template battery cheaper by factoring its control + +The code provides a useful, smaller target than a whole-search chart. C# builds a non-disjunctive battery +and non-disjunctive batches for individual slots. Each template walks slots from last to first; an optional +slot permits the unchanged input to continue, while a required slot stops that skip path [C8]. Templates +are therefore finite control programs over **set-valued** morphological transformations. + +Compile a trie of those reversed control programs. Share a prefix only when its slot operation has the +same ordered rule identities, application role, optionality, and relevant compiled settings. Keep terminal +template identities and their entry predicates. At each battery invocation, evaluate each template's +selector/required-feature predicate on the **original input**, as today; a shared edge is eligible only +for the admitted descendant templates. Preserve per-template finality policy in Rust. Do not move entry +predicates to a transformed intermediate input [C8]. + +For an admitted template, the original slot path has an identical path through the trie. Conversely, +every trie terminal has an admitted original template. At a shared edge the inputs and operations are +identical, so applying the operation once yields exactly the result set each original execution would +have produced. Induction over prefix length preserves each template's ordered rule trail and complete +internal `Word` effects. Branches split at differing slots; required slots retain no skip edge, optional +slots retain both routes, and all-skipped terminals remain. The union of terminal results is consequently +the original battery result set. This is control factoring, with no identification of two independently +arrived search states and no commutation assumption. + +Two implementation boundaries matter. First, retain the existing slot/battery deduplication and representative +semantics: C# uses `Word.ValueEquals`, which omits syntactic features; Rust explicitly generalizes syntactic +features on collisions [C5, C8]. A new global semantic-key merge is not justified by the trie proof. The +first eligible fragment should exclude differing duplicate-result representatives unless their equivalence +is certified. Second, keep template-specific effects at terminals; if an effect changes a subsequent slot +operation, it belongs in the prefix identity or prevents sharing. Tracing follows the ordinary battery, +because template-specific trace events observe separate executions. + +The new trie has at most one node per original slot-prefix position: `O(B)` grammar storage for `B` total +slots, plus rule lists and terminal membership. Evaluate it depth-first with only the current internal +`Word`, branch iterator, and admitted-template information on the stack. No completed `(trie node, Word)` +results are retained for later arrivals. Live traversal storage is bounded by grammar depth and the current +rule-result enumeration, not total states visited. Existing result accumulation is still required. If +implementation instead introduces a growing active-state chart, it has left this proposal. + +This can reduce the factor due to duplicated template control. It does **not** turn the 43,641 battery calls +into 4,988 calls; that would need different search structure or the ruled-out memo. No supplied report +measures common slot prefixes, so a compile-only prefix census is the first gate. This specialization +combines R2's structural sharing and the reports' finite-state compilation ideas at a grammar-sized +boundary. It is a derived proposal, not an experiment any report already ran. + +## 3. Partial-order reduction: preserve the trace class, then recover its orders + +R1 supplies the essential correction to canonicalization. A class contains paths related by swapping +adjacent independent **occurrences**, not by sorting a multiset and deleting its other orders. With repeated +applications, occurrence identity includes its count/position; nondeterministic allomorph outcomes must +also have a proved correspondence. Templates, compounding, and unsupported operations initially act as +dependent barriers. + +For every certified pair and reachable eligible state, require (1) either transition preserves the other's +enabledness and limits; (2) both two-step result sets have a branch-by-branch bijection; (3) matched endpoints +agree on everything subsequent unapplication, templates and lookup can read; and (4) occurrence effects can +be replayed in each recovered order with all trail, non-head and feature history preserved. Disjoint-looking +string edits, critical-pair joinability, or local confluence alone prove none of these for the complete +internal `Word`. C2 and C7 show why feature effects and captured reconstruction are part of the obligation. +Mark uncertain pairs dependent; a conservative restricted certificate is preferable to runtime speculation. + +Under these conditions, adjacent swaps neither lose a path nor invent an impossible one. Their transitive +closure partitions executions into trace classes. A sleep-set traversal can explore one representative +of each class, **including classes of every stopping prefix**; the dependency ordering of occurrences +retains its other linearizations. Template opportunities at intermediate prefixes must remain represented, +not just the terminal class. Lazy enumeration of each class's linear extensions, with exact occurrence +effects restored, reconstructs the original ordered synthesis inputs. Running ordinary synthesis on each +restores precisely the original analysis set. This is the completeness argument missing from merely +keeping the representative. + +Store the grammar's conservative pair matrix (`O(k^2)`) and current path/sleep/linearization stacks. The +occurrence bound `D` must come from actual grammar limits; a current class can use `O(D^2)` ordering data. +Replace that class's explicit histories or stream them to the existing consumer; do not add a search-wide +class registry or DPOR backtracking graph. If the certificate or stack budget cannot be satisfied, use +the ordinary traversal. A streaming consumer is an architectural change to C4, not a drop-in set key. + +For fully independent, single-use rules with one outcome per occurrence there is one class per subset, +hence `2^k` classes rather than factorially many ordered prefixes. Allomorph branching can add further +classes. **That bound is on analysis traversal, not end-to-end parsing.** If +forward synthesis or output identity distinguishes class members, all relevant orders still need exact +validation/readout. Skipping their synthesis needs a separate proof of identical complete acceptance and +reported analysis, for every root and allomorph. C5 and ledger rows 5/6 rule out assuming it globally. +R6's stronger output-invariance condition is right for **discarding** orders; R1's recovery construction +is right for avoiding repeated pre-synthesis work while preserving them. The earlier 0.00–0.05% eligibility +is a warning, not evidence that better certificates exist. + +## 4. Finite-state compilation: exact only for a certified fragment + +An exact compiler would encode a relation from surface input to complete accepted analysis labels. It +must represent the inverse/forward rule behavior, lexical alternatives, template paths, stratum control, +features, annotations and non-heads that affect validity, with every observable ordered identity retained. +Compile bounded rule-use counts as a control vector: there are `prod(i)(m_i+1)` vectors and at most `k` +choices out of one vector. The vector alone describes permission to try rules, not their semantic outputs. + +If each primitive's encoding is exact, relational union preserves all branches and composition preserves +all legal sequences. Intersecting with the input and lexical/validity constraints then preserves exactly +the accepting paths. This gives a structural induction proof for the compiled fragment. Output labels +must carry sufficient allomorph/morpheme order, root position, non-head provenance and feature information +to reconstruct the same reported analyses. If compilation is only an over-approximation, it may reject +proved-impossible branches as in candidate 1, but its surviving paths still require ordinary confirmation. + +The compiler must refuse constructs with no established regular encoding. In particular arbitrary copying +of an unbounded captured string is not an ordinary finite-state relation; finite alphabets alone do not +fix that. Bounded copying, finite feature domains and bounded compound/rule counts need explicit encodings, +not assumptions. Unknown annotation or constraint behavior also rejects exact eligibility. Compile-time +state/arc/byte limits bound grammar storage, with fallback if composition expands too far. + +For an accepted finite relation with `Q` states and `E` arcs and an input of length `n`, recognition can +use a grammar/input-bounded product, rather than materializing each complete trail. Epsilon behavior must +be handled exactly, and productive zero-input cycles must be excluded or proved bounded. An `O(nQ)` +reachability/coaccessibility structure is acceptable only with its byte bound checked before allocation; +otherwise use ordinary parsing. Recognition sharing does not bound distinct labelled outputs or the +cost of eliminating duplicate output identities. Both the compiled relation and its readout can still +be exponential; `2^k` is the **control-state** bound, not the number of analyses. + +R3 is right that a compact proposer plus exact reparse does not solve the confirmation cascade. Its +blanket `O(k^2)` description of the present proposer is too broad: the actual emitter has a subset-mask +path for eligible unordered underlying-token zones, a dedicated-level fallback, and a legacy every-rule +per-level path [C9]. The comment in `pg-foma/src/unordered.rs:44–61` describes the latter, not all current +execution. No claim about the proposer's universal completeness or exact parity is established here. + +## Where the six reports disagreed + +| Issue | Resolution after reading code and the ledger | +| --- | --- | +| **Is the cascade already subset DP?** R2/R4/R5 emphasize a new inner chart; R6 says that would repeat memoization | R2 is right about **Rust**: C1 uses recursive enumeration and C3 merges later. R6 is right about **C#**: the sequential inner cascade and battery already replay state-keyed results. A count vector is not a sufficient semantic key. A capped chart still grows with search and is outside the hard constraint | +| **What is in `AnalysisStateKey`?** The brief/R2 describe more than the older probe-design key | R2 matches Rust's morphology-history/finality fields. Applying that description to C# is wrong; C3 lists the actual different keys. Neither warrants synthesis merging | +| **Every rule is bounded by its application count**, assumed in the brief and count-vector proposals | Correct for Rust's shared gate and C# ordinary affix/compound rules; C# realizational unapplication is an exception and does not extend the pending trail [C2, C5]. The general bound is conditional, not established for every rule kind | +| **Can commuting orders be reduced?** R1 permits recoverable classes; R6 requires invariant output | Both conditions have a different scope. Recovery can preserve every path; discarding an order requires R6's stronger condition. This resolves the ledger's parse-signature objection, but does not establish a synthesis saving | +| **Packed forest/ZDD/product graph is the best lead** in R2/R4/R5, versus R6's memory objection | R5 is right that an exact product needs full synthesis state and ordered continuation/provenance; one final analysis supplies no small-state bound. R6 is right under this job's constraint. Generic product charts are not recommended. A ZDD encodes sets; ordered trails need position/occurrence encoding, whose size none of the reports bounds | +| **Optional deletion is already compact**, emphasized by R3/R6 | Correct for shape construction and Rust trie active states [C6]. It does not establish cheap general morphological matching: C7 needs captures/bindings, and ledger row 23 closes position-only deduplication | +| **The proposer is `O(k^2)`**, R3 | Verified only for the legacy branch. C9's executed mode dispatch and mask loop take precedence over the older descriptive comment | +| **Simple static precedence could remove impossible orders**, R4 | Conditional only: adjacent-pair failure does not rule out an intervening enabler, and an optional predecessor cannot become mandatory. Candidate 1 states the stronger complete-path certificate | +| **A 64-key root cache bounds walks by `min(U,64)`**, R5; R4 also proposes bounded reuse | The raw search boundary is reusable under fixed grammar/overlay [C4], but the bound is wrong when `U > 64`: uncached keys are searched on every revisit. A byte cap must also cover shapes and result lists. Under the no-memo instruction this is not recommended | + +## Considered and not recommended + +- **Count-vector charts, generic packed analysis forests, ZDD trail tables, analysis-DAG × synthesis-state + products:** search-growing tables; rows 1/4/5/6/16 already test the necessary convergence, and a fixed cap + with restart does not make these a different allowed technique. +- **Packed synthesis readout keyed only by trail position or current shape:** omits ordered continuation + and root-dependent state; rows 5/6/16 close it. `O(Q+T+A)` only helps after proving `Q` and provenance + readout small, which no report does. +- **State/position-only matcher deduplication or two-pass capture reconstruction:** row 23 and the + [two-pass record](rejected-optimizations/two-pass-nondeterministic-traversal.md) already settle these; + the safe group-free fragment is existing prior work, not a new research result. +- **Fixed-cap exact root-result cache:** useful only for raw lookup, still memoization, and neither + 56 shapes nor 64 entries bounds all `(stratum, shape)` result bytes or repeated misses. +- **Root-only lexical gates, edge prefilters, current-length bounds, feature-only abstract replay, and + nogood subsumption:** rows 8–11/14/19/22 already cover them; candidate 1 survives only with its stronger + residual relation and a new measured dead-subtree ceiling. +- **Commuting-order canonicalization without recovery, symmetry/orbit counts:** distinct rule/morpheme + identities and ordered trails remain observable; no output-fixing grammar automorphism was established. +- **Meet-in-the-middle:** large midpoint frontiers violate the memory constraint; unordered rule splits + do not automatically turn factorial work into two square-root searches. +- **A*, k-best/k-shortest paths, reverse-search enumeration alone:** all valid analyses are required; + rearranging or ranking trail enumeration does not eliminate failed trails without a sound extension oracle. +- **Generic Earley/CKY, scalar semirings, equality saturation/e-graphs:** no proved semantics-preserving + span/equivalence boundary for these transforms; adding full transformation state/provenance restores + the unresolved search size. +- **Bimachines, two-level morphology, compile-replace, or parser names alone:** constrained morphology + formalisms supply candidate 4's tools, not a correctness proof for arbitrary HermitCrab behavior. + +## Implications for the two work items + +**Lexicon-driven pruning during unapplication.** Start with candidate 1 in shadow mode at C1/C2. The +deliverable is a per-input census of proved-dead subtrees, saved descendant work, unknown cases and +abstraction cost. Include remaining morphology, template transitions, intermediate-stratum lexical goals, +optional sites and compound non-head roots. Disable pruning where guessing/overlays are uncovered. Close +it if the result merely reproduces row 10's no-op or row 11's weak root-only ceiling. Exact product +coaccessibility is a possible stronger oracle only after a grammar-sized representation exists; building +a whole search graph to ask whether that graph has a successful leaf is not an allowed shortcut. + +**Making the affix-template battery cheaper without memoization.** Start with candidate 2's static +prefix census and admission/slot work counts. Its target is repeated control **within one invocation**. +Candidate 1 can also avoid an invocation whose entire continuation is proved dead; candidate 3 might +avoid repeated prefix work if independence coverage is real. Neither the 43,641/4,988 ratio nor the +historical battery memo authorizes reinstating a cache. Keep deferred cloning, safe edge dispatch and +the final-template policy distinct from this research: their evidence and limits already have records. +In particular, the [final-template record](rejected-optimizations/final-template-prune.md) has an addendum +reopening a narrower guard; its old zero-prune result is not a new lead to rediscover here. + +For either item, compare complete analysis sets with ordered morpheme/allomorph identities, root position, +features and non-head provenance. Do not use the legacy count-degenerate signature called out in the +September disposition. Tests support a proof; a finite corpus cannot prove parity for every export. +Tracing and stateful selectors use the existing path unless their invocation/event semantics are preserved. +Configured alternative/work limits also use the existing path unless the same limit behavior is reproduced +[C1, C4]; a resource-limited optimized query returns unknown and falls back, never a partial exact result. + +## Evidence limits and references + +**Not verified:** the brief's timings/counters, the earlier POR eligibility percentage, exact applicability +of any proposed certificate/compiler to the motivating grammar, common template-prefix coverage, a small +packed product/ZDD bound, and any end-to-end speedup. No real-grammar benchmark or new optimization was +run. Historical measurements are attributed to the supplied brief or the existing records, not fresh +results. Code inspection verifies the listed control/data facts; it does not establish universal parity +of the current Rust port or the current FST proposer. All future experiments need the stronger output +signature and ordinary-parser fallback. + +All entries below are **[memory; bibliographic details and applicability not independently verified]**. +No network or publication lookup was available. They identify home fields, not proofs of these proposals. + +- **L1.** P. Cousot and R. Cousot (1977), “Abstract Interpretation: A Unified Lattice Model for Static + Analysis of Programs by Construction or Approximation of Fixpoints.” +- **L2.** C. Beeri and R. Ramakrishnan (1991), “On the Power of Magic.” +- **L3.** B. Billot and B. Lang (1989), “The Structure of Shared Forests in Ambiguous Parsing.” Structural + sharing background; the grammar-prefix specialization here does not adopt a runtime parse chart. +- **L4.** A. Mazurkiewicz (1987), “Trace Theory”; P. Godefroid (1996), *Partial-Order Methods for the + Verification of Concurrent Systems*; F. Baader and T. Nipkow (1998), *Term Rewriting and All That*. +- **L5.** R. Kaplan and M. Kay (1994), “Regular Models of Phonological Rule Systems”; M. Mohri (1997), + “Finite-State Transducers in Language and Speech Processing”; K. Beesley and L. Karttunen (2003), + *Finite State Morphology*. + +The reports also invoke Held–Karp sequencing DP, Tomita-style shared forests, Minato's ZDDs, graph +coaccessibility, and PC-KIMMO/AMPLE/XAmple. These are useful vocabulary for the considered designs; +none supplies the missing state-equivalence, memory, or arbitrary-HermitCrab encoding proof. From c300f2e82cd02791b7b5ae24116f4254c51e5d1c Mon Sep 17 00:00:00 2001 From: John Lambert Date: Wed, 21 Oct 2026 20:49:25 -0400 Subject: [PATCH 04/32] Label the C# anchors in the asymptotics research as a pre-#491 snapshot The anchors were read at a4b29742, before #491 added final-template state to AnalysisStateKey, so the key comparison described a master that no longer exists. It is now dated, and says what #491 changed. Co-Authored-By: Claude Opus 5.5 --- .../hermitcrab-search-asymptotics-research.md | 20 ++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/hermitcrab-search-asymptotics-research.md b/docs/hermitcrab-search-asymptotics-research.md index 48951c95f..6c1874377 100644 --- a/docs/hermitcrab-search-asymptotics-research.md +++ b/docs/hermitcrab-search-asymptotics-research.md @@ -67,9 +67,10 @@ checkout, commit `a4b29742b6274a01c7398ca3e800a19fa6d2c9aa`; paths are relative `src/SIL.Machine.Morphology.HermitCrab/` unless qualified. The six central files (`AnalysisStratumRule`, `AnalysisStateKey`, `MemoizedCombinationRuleCascade`, `AnalysisAffixTemplateRule`, `Morpher`, `Word`) are text-identical in this PR worktree's starting commit `5bbfa018bdbd5228c3f1aa73db951a4f01d167d3`, ignoring -line endings. **Rust** paths are relative to the supplied checkout's `rust/crates/`, with branch tip -`fa3bfb186852baaaf1ed28fcc0ba4602b2c19e96` at inspection. This identifies inspected files; it is not a -claim that either external checkout was clean. +line endings. The C# anchors are a snapshot at `a4b29742`, before #491 (`b240b7b5`) merged; they were not +re-read at current master. **Rust** paths are relative to the supplied checkout's `rust/crates/`, with +branch tip `fa3bfb186852baaaf1ed28fcc0ba4602b2c19e96` at inspection. This identifies inspected files; it is +not a claim that either external checkout was clean. | Anchor / fact | C# file:line | Rust file:line | | --- | --- | --- | @@ -83,12 +84,13 @@ claim that either external checkout was clean. | **C8 — template admission and slot semantics** | `AnalysisAffixTemplateRule.cs:18–73`; non-disjunctive union: `src/SIL.Machine/Rules/RuleBatch.cs:52–61`, configured at `AnalysisStratumRule.cs:27–30` | `pg-rules/src/stratum.rs:894–962,990–1075` | | **C9 — proposer is followed by exact-engine confirmation** | exact confirmation target: `Morpher.cs:346–367` | current emitter: `pg-foma/src/emit.rs:2615–2675,2704–2745,2768–2808`; confirmation filters/reparse: `pg-foma-runtime/src/confirm.rs:498–517` | -In particular, the C# key contains shape, stratum, syntactic and realizational features, non-head count, -and exact per-rule unapplication counts. It contains neither final-template state nor a separate morphology -history. Rust adds both, saturates counts at each rule's maximum, and clears its analysis final-template -state at the stratum fold (`pg-rules/src/stratum.rs:1196–1200`). The same key name does not imply the same -equivalence relation [C3]. Neither key is a synthesis key. This inspection verifies these fields and call -sites; it does not certify a new key as complete for every continuation. +In particular, at `a4b29742` the C# key contains shape, stratum, syntactic and realizational features, +non-head count, and exact per-rule unapplication counts. It contains neither final-template state nor a +separate morphology history; #491 later added final-template state to the key, so current master lacks only +the morphology history. Rust adds both, saturates counts at each rule's maximum, and clears its analysis +final-template state at the stratum fold (`pg-rules/src/stratum.rs:1196–1200`). The same key name does not +imply the same equivalence relation [C3]. Neither key is a synthesis key. This inspection verifies these +fields and call sites; it does not certify a new key as complete for every continuation. There is a second relevant implementation difference: Rust's common dispatch applies the count gate and pushes a pending-rule entry for every morphological rule kind. C# realizational unapplication has no count From 85f05af8e3e2a4c3cb6ec1b4ef220bd9dae143fe Mon Sep 17 00:00:00 2001 From: John Lambert Date: Wed, 21 Oct 2026 21:24:25 -0400 Subject: [PATCH 05/32] Say the retained optimizations live on the archive branch, not master "Status: retained" read as shipped, but none of the nine records' symbols exist on master; they are on perf/hc-optimization-archive. Each record and the ledger's disposition now say where they are retained. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-optimization-ledger.md | 12 ++++++------ docs/optimizations/deferred-template-clone.md | 4 ++-- docs/optimizations/edge-segment-prefilter.md | 6 +++--- .../feature-struct-frozen-tree-clone.md | 2 +- .../feature-struct-frozen-tree-equality.md | 2 +- .../feature-struct-visited-state-reuse.md | 5 +++-- docs/optimizations/flat-register-arrays.md | 7 ++++--- docs/optimizations/shape-copy-single-pass-map.md | 2 +- docs/optimizations/shape-sharing-on-clone.md | 8 ++++---- .../syntactic-feature-structure-sharing.md | 5 +++-- 10 files changed, 28 insertions(+), 25 deletions(-) diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index ed8c839e5..cc2b24312 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -40,12 +40,12 @@ speedup, or numbered optimization row is claimed.** ## Final disposition of the September follow-up -The follow-up branch does deliver product improvements. Its authoritative summary is -`hermitcrab-perf-2026-09-counts.md`, and each retained optimization has one record under -`docs/optimizations/`. The retained set is: deferred template cloning, feature-structure traversal-state -reuse, frozen-flat-tree equality, frozen-flat-tree cloning, single-pass `Shape.CopyTo` mapping, -copy-on-write shape sharing, flat FST register arrays, the analysis edge-segment prefilter, and -copy-on-write syntactic feature structures. +The follow-up branch does deliver product improvements, retained on `perf/hc-optimization-archive` and not +merged to master. Its authoritative summary is `hermitcrab-perf-2026-09-counts.md`, and each retained +optimization has one record under `docs/optimizations/`. The retained set is: deferred template cloning, +feature-structure traversal-state reuse, frozen-flat-tree equality, frozen-flat-tree cloning, single-pass +`Shape.CopyTo` mapping, copy-on-write shape sharing, flat FST register arrays, the analysis edge-segment +prefilter, and copy-on-write syntactic feature structures. Four attractive experiments were deliberately excluded from the final product diff and are recorded under `docs/rejected-optimizations/`: the negative two-pass nondeterministic traversal, the inert final-template diff --git a/docs/optimizations/deferred-template-clone.md b/docs/optimizations/deferred-template-clone.md index e5c6d5d7f..59155adf1 100644 --- a/docs/optimizations/deferred-template-clone.md +++ b/docs/optimizations/deferred-template-clone.md @@ -1,7 +1,7 @@ # Deferred per-template clone in `AnalysisAffixTemplateRule` -**Status:** retained. **Portable:** the observation is; the saving is allocation, so the Rust port gains -little unless it also clones eagerly. +**Status:** retained on `perf/hc-optimization-archive`, not merged. **Portable:** the observation is; the +saving is allocation, so the Rust port gains little unless it also clones eagerly. ## The observation diff --git a/docs/optimizations/edge-segment-prefilter.md b/docs/optimizations/edge-segment-prefilter.md index c1a905480..e6215d81b 100644 --- a/docs/optimizations/edge-segment-prefilter.md +++ b/docs/optimizations/edge-segment-prefilter.md @@ -1,8 +1,8 @@ # Edge-segment prefilter for analysis affix rules -**Status:** retained (`AnalysisAffixProcessRule`; internal same-binary toggle -`Morpher.EdgePrefilterEnabled`). **Portable:** yes; it is a property of the pattern semantics, -not of the C# engine. +**Status:** retained on `perf/hc-optimization-archive`, not merged (`AnalysisAffixProcessRule`; internal +same-binary toggle `Morpher.EdgePrefilterEnabled`). **Portable:** yes; it is a property of the pattern +semantics, not of the C# engine. ## The observation diff --git a/docs/optimizations/feature-struct-frozen-tree-clone.md b/docs/optimizations/feature-struct-frozen-tree-clone.md index 642c1cde2..3677c7750 100644 --- a/docs/optimizations/feature-struct-frozen-tree-clone.md +++ b/docs/optimizations/feature-struct-frozen-tree-clone.md @@ -1,6 +1,6 @@ # Reuse or skip the feature-structure clone map -**Status:** retained (`FeatureStruct.Clone`). +**Status:** retained on `perf/hc-optimization-archive`, not merged (`FeatureStruct.Clone`). **Portable:** yes; the specific pool is .NET-oriented, while the tree proof is general. ## Invariant diff --git a/docs/optimizations/feature-struct-frozen-tree-equality.md b/docs/optimizations/feature-struct-frozen-tree-equality.md index 5378fb48e..6f34aa04d 100644 --- a/docs/optimizations/feature-struct-frozen-tree-equality.md +++ b/docs/optimizations/feature-struct-frozen-tree-equality.md @@ -1,6 +1,6 @@ # Allocation-free equality for frozen flat feature structures -**Status:** retained (`FeatureStruct.ValueEquals`). +**Status:** retained on `perf/hc-optimization-archive`, not merged (`FeatureStruct.ValueEquals`). **Portable:** yes; the proof depends on immutability and tree shape, not the CLR. ## Invariant diff --git a/docs/optimizations/feature-struct-visited-state-reuse.md b/docs/optimizations/feature-struct-visited-state-reuse.md index d140a2728..f7c2b63ee 100644 --- a/docs/optimizations/feature-struct-visited-state-reuse.md +++ b/docs/optimizations/feature-struct-visited-state-reuse.md @@ -1,7 +1,8 @@ # Reuse feature-structure traversal state -**Status:** retained (`FeatureStruct.ValueEquals` and `Freeze`). -**Portable:** the allocation-avoidance principle is portable; the `[ThreadStatic]` pool is .NET-specific. +**Status:** retained on `perf/hc-optimization-archive`, not merged (`FeatureStruct.ValueEquals` and +`Freeze`). **Portable:** the allocation-avoidance principle is portable; the `[ThreadStatic]` pool is +.NET-specific. ## Invariant diff --git a/docs/optimizations/flat-register-arrays.md b/docs/optimizations/flat-register-arrays.md index 359abb125..553e7eb99 100644 --- a/docs/optimizations/flat-register-arrays.md +++ b/docs/optimizations/flat-register-arrays.md @@ -1,8 +1,9 @@ # Flat register arrays in the FST traversal -**Status:** retained (`SIL.Machine.FiniteState`: `Register[]` indexed by `RegisterArray.Idx(r, k)` -replaces `Register[,]` internally; the public `FstResult.Registers` stays two-dimensional and is -materialized once per accepting state). **Portable:** no; a .NET-specific allocation path. +**Status:** retained on `perf/hc-optimization-archive`, not merged (`SIL.Machine.FiniteState`: +`Register[]` indexed by `RegisterArray.Idx(r, k)` replaces `Register[,]` internally; the +public `FstResult.Registers` stays two-dimensional and is materialized once per accepting state). +**Portable:** no; a .NET-specific allocation path. ## The observation diff --git a/docs/optimizations/shape-copy-single-pass-map.md b/docs/optimizations/shape-copy-single-pass-map.md index ed8c15828..900515da2 100644 --- a/docs/optimizations/shape-copy-single-pass-map.md +++ b/docs/optimizations/shape-copy-single-pass-map.md @@ -1,6 +1,6 @@ # Build `Shape.CopyTo`'s node map in the copy pass -**Status:** retained (`Shape.CopyTo`). +**Status:** retained on `perf/hc-optimization-archive`, not merged (`Shape.CopyTo`). **Portable:** yes, when a port has the same node-remapping requirement. ## Invariant diff --git a/docs/optimizations/shape-sharing-on-clone.md b/docs/optimizations/shape-sharing-on-clone.md index 702de8f35..a5ffd563f 100644 --- a/docs/optimizations/shape-sharing-on-clone.md +++ b/docs/optimizations/shape-sharing-on-clone.md @@ -1,9 +1,9 @@ # Copy-on-write Shape sharing in `Word.Clone` -**Status:** retained (`Word.CloneForEngine`, `EnsureOwnShape`, `ResetShape`, `Shape.CreateEmptyLike`). -**Portable:** partly. The *observation* (three of four clone sites never use the copied shape) is -engine-independent; the mechanism (shared reference to an immutable structure) is how a Rust port would -naturally write it already. +**Status:** retained on `perf/hc-optimization-archive`, not merged (`Word.CloneForEngine`, `EnsureOwnShape`, +`ResetShape`, `Shape.CreateEmptyLike`). **Portable:** partly. The *observation* (three of four clone sites +never use the copied shape) is engine-independent; the mechanism (shared reference to an immutable +structure) is how a Rust port would naturally write it already. ## The observation diff --git a/docs/optimizations/syntactic-feature-structure-sharing.md b/docs/optimizations/syntactic-feature-structure-sharing.md index 40ff2af22..9f99d4570 100644 --- a/docs/optimizations/syntactic-feature-structure-sharing.md +++ b/docs/optimizations/syntactic-feature-structure-sharing.md @@ -1,7 +1,8 @@ # Copy-on-write syntactic feature structures in engine clones -**Status:** retained (`Word.CloneForEngine`, `EnsureOwnSyntacticFeatureStruct`). -**Portable:** partly; immutable sharing is general, while the ownership plumbing is specific to this engine. +**Status:** retained on `perf/hc-optimization-archive`, not merged (`Word.CloneForEngine`, +`EnsureOwnSyntacticFeatureStruct`). **Portable:** partly; immutable sharing is general, while the ownership +plumbing is specific to this engine. ## Invariant From 54f53a2a5904176c1e1b2456313f9b6eefbc37d0 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Wed, 21 Oct 2026 21:26:11 -0400 Subject: [PATCH 06/32] Qualify the ledger's parity claims as WordAnalysisSignature matches The signature reduces to morph counts on Sena, Mbugwe and Amharic, as the disposition doc says, so "exact output parity" and "identical analyses" overstated what those checks verified. All three places now say what was compared and point to the caveat. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-optimization-ledger.md | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index cc2b24312..4042f16ad 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -52,10 +52,12 @@ Four attractive experiments were deliberately excluded from the final product di prune, the zero-value alternatives memo/hoist, and the low-ceiling compounding split prefilter. This final disposition supersedes intermediate “ships” language for those experiments later in this historical ledger. -The minimized branch passed all 603 runnable tests in the conformance-inclusive suite (one expected -Windows symlink-privilege skip). A fixed three-word sample had exact output parity on Sena, Amharic, -Indonesian, Mbugwe, and Aweti; only Amharic (1.76x) and Mbugwe (1.83x) cleared the recorded spread and timing -reliability rules. The other three ratios are retained as directional evidence only. +The minimized branch passed all 603 runnable tests in the conformance-inclusive suite (one expected Windows +symlink-privilege skip). A fixed three-word sample matched on `WordAnalysisSignature` on Sena, Amharic, +Indonesian, Mbugwe, and Aweti; on Sena, Mbugwe and Amharic that signature compares morph counts, not +analyses (`hermitcrab-perf-2026-09-disposition.md`). Only Amharic (1.76x) and Mbugwe (1.83x) cleared the +recorded spread and timing reliability rules. The other three ratios are retained as directional evidence +only. ## Earlier ledger round: net speed delivered none @@ -108,7 +110,7 @@ copy-on-write design through `CloneForEngine`; see `docs/optimizations/shape-sha | # | Optimization | Expected / observation | Outcome | | --- | --- | --- | --- | -| 22 | **Analysis-side rule prefilter** | 94.4% of Sena allomorph pattern matches failed, so a cheap necessary-condition check had a large direct-work ceiling. | **RETAINED after the earlier self-time cap was refuted by direct work counts.** The original ~10% ceiling counted matcher self-time but omitted traversal allocation, feature unification, and GC booked elsewhere. The implemented edge-segment prefilter removed **87% of FST traversals**, 54% of arc checks, and 85% of traversal instances on 45 Sena words, with exact parity on those words and all 464 conformance-fixture words. See `docs/optimizations/edge-segment-prefilter.md`. | +| 22 | **Analysis-side rule prefilter** | 94.4% of Sena allomorph pattern matches failed, so a cheap necessary-condition check had a large direct-work ceiling. | **RETAINED after the earlier self-time cap was refuted by direct work counts.** The original ~10% ceiling counted matcher self-time but omitted traversal allocation, feature unification, and GC booked elsewhere. The implemented edge-segment prefilter removed **87% of FST traversals**, 54% of arc checks, and 85% of traversal instances on 45 Sena words, with `WordAnalysisSignature` parity on those words (a morph-count comparison on Sena; see `hermitcrab-perf-2026-09-disposition.md`) and all 464 conformance-fixture words. See `docs/optimizations/edge-segment-prefilter.md`. | | 16 | **`ExpandAlternatives` dedupe** | 20.7% of Sena wall, the largest bucket nobody had instrumented. Apparent census: 395,026 alternatives -> **61** distinct (0.0%), 85% of duplicates same-analysis-word | **CLOSED on the sound census.** With the pending-trail content added to the key, 395,026 -> **351,414 distinct (89.0%)** -- only **11%** are duplicates at all, and all 43,612 of them are same-analysis-word. Sound ceiling = 20.7% x 0.110 = **2.28% of Sena wall**, against a gate of >=10% to build and <5% to close. **The fourth apparent redundancy to evaporate under a complete key** | | 17 | **Per-node cost in the analysis cascade** | 84% of the Amharic cascade is not linguistic computation; clone is 46.7% and 37.4% was unattributed | **Partly settled, remainder still open.** (a) The remainder split **MISSES its <15% gate by ~2.5x** — still ~38% on Amharic `anCascade`. Memo, Trail and **Equality each measured <0.2%**, so `Word.ValueEquals` behind the cascade's `HashSet` is *not* the missing mass, contrary to the standing hypothesis. **Two caveats the first write-up missed.** (i) The ~38% figure comes from a *contended* run — three timing probes were live — and descheduled wall time lands preferentially in the remainder while deflating every self-time share, so **CPU starvation is itself an unbracketed candidate biasing in exactly the direction observed**. (ii) The claim that GC pause time is unmeasurable is **wrong at the measurement point**: only the library assemblies are netstandard2.0; the probe harness lives in `tests/SIL.Machine.Morphology.HermitCrab.Tests`, which targets **net10.0**, so `GC.GetTotalPauseDuration()` and `GC.GetGCMemoryInfo().PauseTimePercentage` are directly callable. Next probe, gated: re-run serial — remainder <20% means contention was the missing mass and 17(a) recloses on the cheaper-clone target; still >=30% then bracket with `GetTotalPauseDuration()` deltas — ΔPause >=50% of the remainder's absolute ms confirms GC (build target converges with clone/freeze at 46.7%), <=15% refutes it and the next split is enumerator/LINQ orchestration (the 2026-07 alloc audit already flagged `ExecuteCommands` enumerator boxing); a collection census over 24 Amharic words gives gen0 8,217 / gen1 2,729 / gen2 301, an unusually high gen1+gen2 ratio consistent with — not proof of — GC as the remainder. (b) **Clone-discard fraction = 66.2%** (created 2,505, discarded 1,659), a count and therefore contention-immune, against a >=70% gate: **misses by ~4 points, so the build is "make a clone cheaper", not "defer cloning"**. Scoped to `AnalysisAffixTemplateRule`'s eager per-template clone, the one clone-before-verification site; the other four analysis-side sites clone only after a successful match and were excluded rather than blended in | | 18 | Corpus-scope memoization | `AnalysisScope` dies per word, but `AnalysisStateKey` is word-independent by construction, so entries are reusable across a corpus. Nogood hits dominate positive replays 10-17x, making cross-word nogood reuse the cheapest available form of reuse | **CLOSED, and worse than a no-op.** Indonesian, 121 words, **serial on a clean machine: 1.015x, 1.4% wall-clock reduction** against a >=20% build gate and <5% close gate. Costs **68 MB** added peak managed memory. Cross-word reuse is real but small: 39 memo, 73 nogood, 237 template-nogood hits. Token-vs-type: **121 tokens, 121 distinct**, so no type-level dedupe is available either. **A first, contended run of the same harness reported 2.385x / 58.1%** with a 210.9% spread on the shared arm against 13.7% on the baseline; two testhosts were live. The counts were byte-identical across both runs — only the timing was contaminated, and min-of-N on a high-spread arm picks the luckiest sample. Analysis-set equality held throughout, so the key genuinely is word-independent. **Sena, first 300 words, is the decisive case: the shared scope is 28-32% *slower* than per-word scopes** (two clean serial runs, no contention: 282.7s vs 372.2s and 308.9s vs 396.7s baseline-vs-shared; deterministic counters byte-identical across both runs), and costs **~29.3 GB** added peak managed memory against a 2 GB budget. Root cause: `AnalysisScope.MaxMemoEntries` (100,000) is a *per-scope*, not per-word, budget. Two heavy early words (#74, #76 of 300) alone fill the shared `Memo` table to the cap and `TemplateMemo` to 55,071; every word after that loses the free within-word memoization a fresh per-word scope would have given it for nothing, because `Store` silently no-ops past the cap (see its comment) with no eviction. Cross-word hits (shared minus baseline) are **negative** — memo -77,648, nogood -747,607 — i.e. sharing produced *fewer* total hits than 300 independent fresh-table runs would have on their own; the small `+820` template-hit / `+9` template-nogood gain does not come close to covering it. Amharic, capped at 28 words (word 29 hangs, excluded per standing guidance): **0.9% wall-clock reduction**, table nowhere near the cap (Memo 1,251, TemplateMemo 652 of 100,000), consistent with row 17's finding that Amharic's cost is dominated by clone/allocation per node rather than distinct-state count, so avoiding a handful of cross-word re-derivations barely moves wall time. Token-vs-type on all three corpora, at the runs' own sizes and on each corpus's full word list: **zero duplicate tokens** — whole-word caching is not available as a cheaper alternative anywhere here. No analysis-set divergence on any corpus, at any scope regime: the stop condition that would have put rows 1-3 in question did not fire. **A shared-but-capped memo table is not "the same optimization, smaller ceiling" — past the cap it actively takes away the free per-word memoization the product already has, and retains every live entry for the run's full duration instead of per-word, which is where the 29 GB comes from.** Any future attempt needs real eviction (LRU, generational, or per-word sub-budgets) before this is worth re-measuring, and even then the >=20% gate looks unreachable given Indonesian's ceiling under an uncapped-in-practice regime was already only 1.4% | @@ -273,11 +275,12 @@ with the code: `docs/hermitcrab-perf-2026-09-counts.md` and one file per change `Word.Clone` deep-copied the Shape and every node feature structure. The follow-up independently implements a different design: internal `CloneForEngine` copy-on-write with un-share at mutation sites, while public `Word.Clone` remains a deep mutable clone. A shared shape is frozen, so a missed mutation site throws. -- **Row 22's ~10% cap for an analysis-rule prefilter was wrong by an order of magnitude.** It was - computed from matcher *self-time*. Counted directly, an edge-segment prefilter (patterns are anchored - at both ends and matched by unification, so the first/last literal segment must unify with the - shape's first/last segment) removes **87% of FST traversals** and 54% of arc checks on 45 Sena words, - with identical analyses on every Sena word and all 464 conformance-fixture words. +- **Row 22's ~10% cap for an analysis-rule prefilter was wrong by an order of magnitude.** It was computed + from matcher *self-time*. Counted directly, an edge-segment prefilter (patterns are anchored at both ends + and matched by unification, so the first/last literal segment must unify with the shape's first/last + segment) removes **87% of FST traversals** and 54% of arc checks on 45 Sena words, with identical analysis + signatures on every Sena word (morph counts only on Sena; see the disposition doc) and all 464 + conformance-fixture words. - **"memo-key construction 0.0%" (row 21) was wrong at the profile level.** Hashing each new word's syntactic feature structure for `AnalysisStateKey` is ~15% of CPU on the heaviest Sena word; the key itself is cheap but freezing a fresh clone of a nested feature structure is not. From 06b1b47d02ddf1cc3e57cbc58ccb808067911e04 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Wed, 21 Oct 2026 21:40:44 -0400 Subject: [PATCH 07/32] docs: record the 2026-10-21 search-cost round (rows 24-30) Two port bugs found by diffing HC-Rust against the oracle --- ...hermitcrab-2026-10-21-search-cost-round.md | 377 ++++++++++++++++++ docs/hermitcrab-optimization-ledger.md | 25 +- 2 files changed, 400 insertions(+), 2 deletions(-) create mode 100644 docs/hermitcrab-2026-10-21-search-cost-round.md diff --git a/docs/hermitcrab-2026-10-21-search-cost-round.md b/docs/hermitcrab-2026-10-21-search-cost-round.md new file mode 100644 index 000000000..3522a8a74 --- /dev/null +++ b/docs/hermitcrab-2026-10-21-search-cost-round.md @@ -0,0 +1,377 @@ +# HermitCrab search cost, 2026-10-21 round + +**Disposition: three large reductions in search work stand up. Two of them were bugs in the Rust +port, not algorithmic gains.** HC-Rust ([sillsdev/PanGloss](https://github.com/sillsdev/PanGloss)) +is a port of `SIL.Machine.Morphology.HermitCrab` under test against `hc.dll`. On one private +grammar, HC-Rust did far more search than C# because two kinds of shape node had lost the segment +identity that C# gives them. Fixing both made HC-Rust match C# and removed most of that grammar's +cost. The one algorithmic change that survived, sharing trailing affix-template slot batches, was +built in HC-Rust and then ported to C# as [Machine PR #546](https://github.com/sillsdev/machine/pull/546). +The lead ranked first in the [search-asymptotics research](hermitcrab-search-asymptotics-research.md), +lexically-dead branch pruning, was measured first and rejected on its ceiling. + +This record follows the [optimization ledger](hermitcrab-optimization-ledger.md): one entry per +attempt, the number that settled it, and what differs from earlier attempts. Ledger rows 24-30 +summarise it. + +## Data and units + +**Grammars.** The public reference grammars are Sena, Amharic, Mbugwe, Aweti and Indonesian. The +round's motivating grammar was a **private featureless project (Austronesian, ~30k lexical entries, +3 strata)**. It was measured in three variants: *base*, *v1*, and a *deletion variant* that adds a +rule deleting two stop classes after nasals and has no compounding. Each variant was run on three +word lists: 200 nonwords, a 563-word dev list and a 7,206-word held list. Only aggregate numbers +from it appear here; no word forms, glosses or lexical entries. + +**Units.** The owner rule for this round is that gains are stated as **parser work**: "N% less work +to parse the words". +- **Parser work** is HC-Rust's per-word `work_steps`, summed over a word list. A work unit is one + reservation against HC-Rust's per-word work cap. Every analysis attempt reserves one, and so does + every inner matcher step, segmentation step, root-lookup step and confirmation step. It is + deterministic, so it survives machine contention, unlike wall time. +- **Steps** are analysis attempts (`steps <= work_steps`). They are a secondary figure. +- **Milliseconds** appear only as labelled secondary figures. +- **C# figures** are in C#'s own counters (slot batches, lookups), never in HC-Rust work units. They + are not converted. + +**Parity comparators.** HC-Rust used `rust/tools/parse_compare.py`, which compares complete +analyses per word and reports each row IDENTICAL or not. Entries 090 and 103 ran it with +`--multiset`; entry 095 records only `parse_compare.py`. PR #546 compared ordered morpheme-ID/surface signature lists, +ordinal-sorted, with multiplicity. Neither is the legacy morph-count `WordAnalysisSignature` that the +September disposition calls out. + +**Runs.** Unless noted, HC-Rust runs were `pangloss batch --threads 1` with `HC_STEP_STATS=1`. +Change A's full-list parity runs used `--threads 2` with the default step cap and no wall timeout. + +## What stands up + +| # | Change | Kind | Where | Effect, in parser work | Parses | +|---|---|---|---|---|---| +| A | Analysis-side affix literals keep `StrRep` identity on wide featureless tables | HC-Rust port bug. C# already exact | PanGloss divergence entry 090 (analysis-side `Lhs` follow-up) | Private project, 7 spot words: base **98.5% less** (2,365,602 -> 35,913); worst variant **99.8% less** (50,778,583 -> 83,677). Steps on the full lists fell 46-55x | IDENTICAL on all three private lists and on Amharic, Sena, Indonesian | +| B | A re-inserted segment-class node keeps its members on a featureless grammar | HC-Rust port bug. C# already exact | PanGloss entry 103, [`414ac6b5`](https://github.com/sillsdev/PanGloss/commit/414ac6b5), [record](https://github.com/sillsdev/PanGloss/blob/main/docs/divergences/103-class-deletion-reinsertion-identity.md) | Deletion variant, held list: **94.7% less** (303,966,311 -> 16,220,190). Its 16 words that hit a 1 s limit: **99.9% less** | IDENTICAL on every list. Byte-identical work wherever it cannot fire | +| C | Shared trailing slot trie in the affix-template battery | Algorithmic, no memo | PanGloss entry 095, [record](https://github.com/sillsdev/PanGloss/blob/main/docs/divergences/095-shared-template-slot-batches.md); C# [PR #546](https://github.com/sillsdev/machine/pull/546) | Sena **11.5% less** (all 6,146 words); Mbugwe **18.1% less** (1,364-word partial run); Aweti 0.35% less; zero elsewhere | IDENTICAL in HC-Rust; zero signature differences in C# samples | + +The census instrumentation that settled the rejected lead is +[`d1690c13`](https://github.com/sillsdev/PanGloss/commit/d1690c13). It is gated behind +`HC_BRANCH_CENSUS=1` and inert when unset: parser work and parses are identical to main on six +lists. + +## The lesson: diff the port on the oracle's own export before optimizing the algorithm + +The bugs were found by a comparison, not by a profile. HC-Rust normally loads a FieldWorks project +through its own snapshot reader. Here it was also run on the XML that C# itself exports for the same +project. On the private project's slowest word: + +| Engine and input | Lexical lookups | +|---|---| +| C#, its own export | 49 | +| HC-Rust, C#'s export | 18 | +| HC-Rust, its own snapshot | 1,655 | + +These counts were supplied by the round's brief; this write-up did not reproduce them. A ~90x gap +between two inputs to the same engine is a loading or compilation difference, not a search-strategy +difference. Following it led to both identity bugs: + +- **A (entry 090 follow-up).** HC-Rust's analysis-side affix literals carried identity only through + a single 64-bit id lane. That lane does not exist above 64 character definitions. Provisional + letters pushed the private project to 70, and the grammar has no phonological features, so + nothing else distinguished segments. An enclitic literal therefore matched *any* final segment. + Every word long enough was unapplied against it, and synthesis rejected each bogus candidate. C# + gives a featureless segment a `StrRep` `EqualTo` constraint (`CharacterDefinitionTable.cs:68` at +`a4b29742`, the same snapshot the research doc's anchors use), so + its literal matches only itself. The fix applies HC-Rust's chunked `StrRep` matcher on this path, + only for featureless grammars. Featured wide tables (Amharic) keep the feature-only + over-approximation, which C# shares, because a featured node has no `StrRep`. +- **B (entry 103).** `NarrowAnalysisRewriteRuleSpec.Unapply` re-inserts a deletion rule's LHS + constraint as an optional node. For a `SegmentNaturalClass` that node carries the union of its + members' structures, which on a featureless grammar is "`StrRep` in members", and + `RootAllomorphTrie` unifies against it. HC-Rust gave the node no identity, and its root trie also + ignored a query node's member set. Analysis and root lookup could therefore take any segment at + every re-insertion site. The fix sets the member set on featureless grammars only, and the trie + refuses an edge only when C#'s unification would fail. + +**Neither fix is a C# optimization.** Before the fixes HC-Rust did more work than C#; afterwards it +does what C# does. They are recorded here because they decided where this round's effort went, and +because they explain several figures in the research brief: +- **The deletion variant's 1 s timeouts.** The brief reported 200-1,600 inputs reaching a 1 s + per-input timeout, from an earlier build that had neither fix; that count was not re-measured. + What was measured is narrower: after fix A and before fix B, 16 held-list words still hit a 1 s + limit, and fix B cut their parser work by 99.9%. Both port bugs are implicated, and the two counts + are not comparable. +- **Where the time went.** In a two-word profile of the deletion variant before fix B, analysis was + 0.1-0.3% of parser work, root lookup about 10% and synthesis about 90%. Zero branches were + lexically dead. No analysis-side pruning could have helped. + +One residual is **not explained**. On the slowest word, HC-Rust on C#'s export made 18 lookups where +C# made 49. Parses were identical, but an efficiency nobody can argue for is not a result. It is +listed under open leads rather than claimed. + +**Method rule.** Before measuring an algorithmic lead on a port, run the port on the oracle's own +export of the same grammar and compare a cheap count, such as lookups per word. A large gap between +the port's two inputs is a port bug, and its fix can dwarf any algorithmic change. This round, it +did so twice out of three. + +## C1: prune analysis branches that cannot reach a root (research lead #1) — rejected on its ceiling + +**Expected.** A sound summary of the remaining transformations would prune analysis nodes from +which no lexical lookup can find a root. This is the research doc's rank-1 candidate, residual +viability. It differs from ledger rows 10 and 11 only if compounding and the remaining morphology +are modelled. + +**Measured.** A hindsight census charged each unapplication's steps and work to its input node. +Folded alternatives alias their canonical. A node is *live* when a lookup on it, on an alias or on +a descendant found a root. Dead cost is split three ways: +- **Dead leaves:** a failing rule battery plus one failing lookup. +- **Dead, may compound:** fewer non-heads than `MaxStemCount`, with a compounding rule in this or a + deeper stratum. +- **Dead internal, plain:** the residual. + +Only the plain bucket is what a sound, compound-aware summary could actually remove at a profit. A +dead leaf costs about as much to prove dead as to run. A may-compound node is row 10's +disqualifier. + +| List (words) | Parser work | Analysis / lookup / synthesis | Lexically-dead ceiling | Dead leaves | Dead, may compound | **Dead internal, plain** | Repeat-lookup work | +|---|---|---|---|---|---|---|---| +| Private, base, held (7,206) | 50,345,810 | 52.7 / 11.8 / 35.2% | 22.6% | 6.1% | 14.6% | **1.9%** | 2.4% | +| Private, v1, held (7,206) | 105,768,202 | 70.5 / 6.7 / 22.6% | 38.7% | 19.9% | 6.6% | **12.2%** | 1.8% | +| Private, deletion variant, held (7,206), before fix B | 303,966,311 | 8.5 / 35.7 / 55.8% | 16.0% | 12.7% | 0.0% | **3.4%** | 10.4% | +| Indonesian (70) | 1,906,012 | 85.8 / 2.1 / 12.0% | 41.0% | 15.3% | 23.8% | **1.9%** | 0.7% | +| Amharic (669 completed) | 39,395,506 | 97.2 / 1.4 / 1.4% | 66.1% | 20.3% | 45.8% | **0.0%** | 0.6% | +| Sena (5,134 completed at 30 s) | 271,894,826 | 89.9 / 3.0 / 7.1% | 55.5% | 11.9% | 39.0% | **4.7%** | 2.2% | +| Mbugwe, first 400 (394 completed) | 610,569,774 | 73.6 / 3.3 / 23.0% | 45.5% | 9.7% | 35.8% | **0.0%** | 3.3% | +| Aweti (183 of 206 completed at 30 s) | 488,089,456 | 93.6 / 1.0 / 5.3% | 74.9% | 35.1% | 0.0% | **39.8%** | 1.0% | + +The census builds include fix A but predate change C. Template-heavy shares (Sena, and possibly +Amharic and Mbugwe) are therefore against an older, larger denominator. For scale, Sena was 271.9M +work units on 5,134 completed words then, and is 240.7M on all 6,146 now. + +**Verdict: not built.** +- **Plain bucket.** It is 0-5% of parser work on six of eight lists. The exceptions are v1 at 12.2% + and Aweti at 39.8%, the latter on 183 completed words. +- **Most dead work is unprunable.** It sits in leaves, or in nodes that could still split off a + compound non-head. Amharic, Sena and Mbugwe put 36-46% of parser work there. +- **Any sound summary has a large scope.** It must model reduplication, infixes, truncation, + deletion re-insertion and compounding before it is allowed to prune anything. + +**What is different from prior attempts.** +- **Row 11 ceiling.** Row 11 found 23.5% pooled dead steps. Dead *steps* here are 43-94% of steps, + so dead branches are common. But they are cheap: the ceiling falls once it is weighted by work + and the unprunable buckets are removed. +- **Row 11's other failure mode persists:** the root is reachable but synthesis fails. On the + deletion variant before fix B, 86% of synthesis work went to canonicals whose lookups hit but + never matched the surface. On Mbugwe the share is 98%. No lexical oracle can prune either. + +**C# transfer.** C#'s `Morpher`, `AnalysisStratumRule` and `AnalysisRewriteRule` run the same +search, so the same buckets should apply. No Machine PR unless the shadow experiment under open +leads shows a sound summary deciding a meaningful share of the plain bucket. + +### Per-parse distinct-shape lookup — rejected + +The brief's "97,575 lookups over 56 shapes" came from trace mode, where analysis merging is off. The +research doc warned that a trace-mode count cannot establish an untraced duplicate rate, and the +untraced census confirms it: +- **Calls per key.** The private project makes 1.13-1.49 lookup calls per distinct + (stratum, shape). +- **Ceiling.** Exact repeat-lookup work is 0.5-3.3% of parser work on every list, or 10.4% on the + deletion variant before fix B. +- **Policy.** Avoiding it would be a per-parse memo, which this work excludes. + +C#'s `LexicalLookup` repeats lookups the same way, so there is no Machine PR. Dead-candidate lookups +are 30-75% of lookup work, but lookup is only 1-12% of parser work, so they were not pursued either. + +## C2: share trailing slot batches inside one battery call (research lead #2) — kept, ported to C# + +**Expected.** The battery walks each template's slots last-first. Templates whose trailing slots +use the same rules, in the same order, with the same optionality, run the same slot batch on the +same word. Factoring the templates into a trie runs that batch once. This is the research doc's +rank-2 candidate. It is control factoring, not a cache: nothing is keyed by a visited word, and +nothing outlives one battery call. + +**Battery share of all parser work (HC-Rust, before C):** + +| Grammar | Battery share | +|---|---| +| Mbugwe | 61% (86 step-capped words are about 2/3 of the total) | +| Private project, v1, held | 45% | +| Sena | 23% | +| Amharic | 21% | +| Private project, base | 8% | +| Private project, loaded from C#'s export | 9% | +| Private project, deletion variant | 6.5% | +| Indonesian | 0 (no templates) | + +**Static census.** This counts rule applications on the battery's own input, before and after +sharing: +- **Shared tails:** Sena 111 -> 45, Mbugwe 397 -> 196, Aweti 143 -> 100. +- **No sharing possible:** Amharic 50 -> 50, and the private project 30 -> 30. Each has one template. + +**Dynamic ceiling, Sena.** A trie-exact key (same trailing slots and same full word state) predicts +448,573 repeated slot batches, which is 30.8M work units or 11.3% of all Sena work. The built trie +avoided exactly 448,573. + +### HC-Rust (PanGloss entry 095) + +Run with `--threads 1 --step-cap 2000000` (Aweti 500000) and no wall timeout. + +| Grammar | Words compared | Parses | Parser work | Steps | Fire count | +|---|---|---|---|---|---| +| Sena | 6,146 | IDENTICAL | **11.5% less** | 31.6% fewer | 171,319 shared, 448,573 avoided (= predicted ceiling) | +| Mbugwe | 1,364 (partial run) | IDENTICAL | **18.1% less** | 19.3% fewer | - | +| Aweti | 170 | IDENTICAL | 0.35% less | 0.16% fewer | 1,939 shared, 2,159 avoided | +| Private project (4 grammar variants: base, v1, deletion, C#'s export; x 3 lists) | all | IDENTICAL | 0 | 0 | 0 | +| Amharic, Indonesian | all | IDENTICAL | 0 | 0 | 0 | + +**Tests.** +- **Differential test.** Compares the shared walk with each template's own walk for all words up to + 5 segments. It fails when optionality is dropped from the trie key. +- **Fire-count test.** Fails with sharing disabled. +- **Conformance.** `-Scope all` passed 2043/2043. Fixtures at the PR #480 head showed the same + failures as the baseline. + +### C# (Machine PR #546) + +`AnalysisAffixTemplateRule` has the same shape, so the trie ported directly to sequential +`ApplySlots`. It is restricted to untraced, sequential walks with no alternative limit. Tracing, +`ParallelApplySlots`, finite alternative limits, custom compiled templates, changed slot +optionality, and grammars without shared tails all keep the original battery. +`AnalysisScope.TemplateMemo` (row 2) still wraps the battery unchanged: row 2 reduces how often a +battery runs, and #546 reduces repeats inside one run, so the two compose. + +The figures below are C# slot batches on deterministic 200-row samples, single-threaded and warmed, +with mode order alternated per input: + +| Input | Completed pairs | Slot batches before -> after | Fewer batches | Parse time, secondary (ms under concurrent load) | +|---|---|---|---|---| +| Synthetic shared-tail fixture (10 words x 100) | 1,000 | 2,000 -> 1,000 | 50.0% | 1.27 s -> 1.05 s | +| Sena (200 sampled rows) | 172 | 47,072 -> 32,287 | 31.4% | 20.07 s -> 16.67 s | +| Mbugwe (200 sampled rows; 61 pairs capped at 2 s in either mode, excluded) | 139 | 72,467 -> 58,486 | 19.3% | 58.43 s -> 51.33 s | + +- **Parity in the samples.** Every completed pair had identical signature lists, with zero + differences and zero errors. +- **Differential test.** `SharedWalkMatchesPerTemplateWalkInOrder` compares ordered `Word` equality + and complete `AnalysisStateKey` sequences across four grammar arrangements and 2,904 input/state + combinations. +- **Fire test.** `UntracedBatteryRunsSharedSlotBatchOnce` fails with sharing disabled (`Expected: 1; + But was: 2`). +- **Unit tests.** HermitCrab passed 126/126. +- **Conformance.** The PR #480-head fixtures were run on the branch assembly and on the unchanged + base `aab237b2`. With memoization on, both were 58 passed / 2 failed. With memoization off, the + branch was 58/2 and the base 59/1. Both assemblies fail traced-rule attribution in + `boundary-leftover-in-narrowing-lhs-strrep`. `deep-optional-affix-nesting` crossed its 15 s budget + in three runs; it has no shared slot tails and takes the original walk. + +These two engines measure different things, so do not compare the columns. C#'s "31.4% fewer slot +batches" on Sena and HC-Rust's "31.6% fewer steps" are not the same unit, and neither is HC-Rust's +11.5% less parser work. + +**Conformance pins.** The C# 60/60 self-check below is `hc.dll` at `72dbd4f9`, the PR #480 head. The +58/60 above is PR #546's base `aab237b2`. They are different assemblies, and the two counts are not +in conflict. + +**What is different from prior attempts.** +- **Row 2.** The battery memo reduces battery *invocations*. C reduces repeated slot batches + *within one invocation*, with no `(Word, state)` table. +- **Row 18.** C is not a cross-word cache. +- **The research doc's prediction holds.** It said this would not turn 43,641 battery calls into + 4,988. It does not; it shrinks each call. + +**Rejected alongside.** A rule-level memo per battery call was no bigger than slot-level sharing on +Sena (33.7M vs 32.9M work units of repeats), and it is a memo. + +## Conformance state at the round's close + +The run used the PR #480 head, Machine `72dbd4f9`. + +- **C# self-check.** `hc-conformance --include-pathological` passed **60/60 fixtures**: 609 entries + in the `words.yaml` files, none skipped. Upstream fixtures grew from 44 to 60 since the previous + pin; nothing was removed. +- **HC-Rust.** One fixture failed: `edge-cases/clitic-rule-designation`, 5 words. Machine PR #529 + added `MorphologicalRule@isCliticRule`, which HC-Rust has not yet ported. It is a parity gap, not a + divergence, and the port is under way. Every other new or modified fixture passes in both engines. + No fixture passes in HC-Rust and fails in C#. + +## How today's measurements bear on the six research reports + +The integrated ranking is in the [research doc](hermitcrab-search-asymptotics-research.md). This +section records only what changed. + +| Lead (report) | Status after this round | +|---|---| +| **1. Residual viability**: abstract interpretation with staged refinement (R6), lazy residual-lexicon product (R3), magic sets / goal propagation (R4), extension oracle (R5) | **Ceiling measured, rejected for now** (C1). The plain dead-internal bucket is 0-5% on six of eight lists. Reopens only through the Aweti/v1 shadow experiment below | +| **2. Template battery as a shared control trie** (derived from R2/R3) | **Built, kept, ported** (C; Machine PR #546) | +| **3. Trace-class POR with recovery**: Mazurkiewicz traces (R1), sleep/source-set DPOR (R1), critical-pair certificates (R1); verified POR (R6) | Not pursued. No new measurement. The earlier 0.00-0.05% eligibility still stands as the warning | +| **4. Exact finite-state compilation of a verified subgrammar** (R3, R6, R2 #3) | Not pursued as an exact engine. A narrower **FST-derived precheck** is the best remaining lead for single-template grammars (open leads) | +| Fixed-cap root-result cache (R5 #2; R4 aside) | **Rejected**: repeat-lookup work is 0.5-3.3% of parser work untraced, and the cache is a memo. The 97,575/56 premise was a trace-mode artefact | +| Count-vector chart DP with packed provenance (R2 #1); packed trail forest for synthesis (R2 #2); `AnalysisStateKey` graph with ZDD trails (R4 #1); analysis-DAG x synthesis-state product (R5 #1) | Not pursued. They are search-growing tables, a closed family. The finding that 86-98% of synthesis work is root-reachable failure is consistent with the reports' own warning that one output analysis does not bound product states | +| Proven rule-precedence DAG (R4 #2) | Not pursued. It needs complete-path certificates (research doc section 1) | +| Symmetry reduction (R1 #4) | Not pursued. The reports themselves rated it a probable no-fit | +| Meet-in-the-middle (R3, R4), A*, k-best and k-shortest paths, reverse search (R4, R5), Earley/CKY, scalar semirings (R2), e-graphs (R4), bimachines, two-level morphology, compile-replace (R3) | Still not recommended. Unchanged | + +**Prediction scored.** The research doc said the trace-mode lookup ratio could not establish an +untraced duplicate rate. That was right: 0.5-3.3%. + +**The finding the reports could not have made.** Where it was measured, much of the private +project's cost was two port bugs: 98.5-99.8% less work on the spot words, and 94.7% less on the +deletion variant's held list. No ranked lead addressed that, because each one assumed the port's +search matched the oracle's. It is not the whole story: with fix A in place, v1's held list still +costs 105.8M work units, 45% of it in a one-template battery that C cannot touch (open lead 1). + +## Open leads, not pursued + +Each lead is tagged **[C# analogue]** where the same question applies to `hc.dll`, and +**[HC-Rust]** where it concerns the port only. + +**Work reductions, with a ceiling to measure first:** +1. **FST-derived precheck before slot-rule traversal, for single-template grammars** [C# analogue]. + C cannot help a grammar with one template, and on the private project's v1 variant the battery + is still 45% of parser work. A necessary condition derived from each slot rule's compiled matcher + could reject a rule before traversal begins. This differs from row 22's edge-segment prefilter in + being judged in work units, since a traversal charges per arc, and in being derived from the + whole matcher rather than its first and last segments. Its ceiling, the work spent in + zero-match traversals, has not been measured. +2. **Shadow summary experiment on Aweti and v1** [C# analogue]. These are the only lists whose + plain dead-internal bucket is material (39.8% and 12.2%). Run a sound summary in shadow mode and + count the plain-bucket work it would actually decide, along with the summary's own cost. Close + lead 1 if the decided share is small. +3. **Mbugwe's step-capped words** [C# analogue]. 86 of 1,645 words hit the 2M step cap and account + for about 2/3 of Mbugwe's work. They have not been examined individually. +4. **Root-reachable synthesis failure** [C# analogue]. 86% (deletion variant, before fix B) and 98% + (Mbugwe) of synthesis work goes to candidates whose lookup hits but whose synthesis never matches + the surface. No lexical oracle reaches this. It is the largest unaddressed bucket on Mbugwe. + +**C# port follow-ons to #546** [C#]: + +5. **`ParallelApplySlots`** needs its own design for its layered breadth-first queue. +6. **Share compiled slot rules across templates**, in addition to slot batches. +7. **Measure in a deterministic work counter.** The C# gain is measured in slot batches on samples, + not in a work counter on full lists. + +**Time-only costs (no work units charged, so outside the owner metric):** +8. **`ana_allomorph_matches`** deep-clones segments with `segs.to_vec()` and rebuilds + `format!("p{i}")` part names on every allomorph call [HC-Rust]. About 5.1M segment-vector builds + are avoidable per private-v1 held run: 5.48M slot-rule attempts against 365K slot batches. + Hoisting `segs_of` or borrowing segments in `Transduce::new` removes them. +9. **`root_trie::search_segs_ident`** checks its active set with `Vec::contains`, which is quadratic + [HC-Rust]. A bitset would be a pure time win. + +**Identity over-approximations left in HC-Rust** [HC-Rust]. Synthesis re-validates all of them, so +none can change a parse, but each over-explores relative to C# on some grammar: + +10. An `InsertContext` (natural-class insertion) in analysis-side `Lhs` still carries no identity + above 64 character definitions (entry 090's remainder). +11. Narrowing leftovers from `ana_narrow_general` keep no class identity (entry 103's remainder). No + measured grammar exercises a class LHS there. +12. `union_narrowing_node` checks only `char_def`, so it ignores an LHS class node's member set. +13. RHS class nodes in synthesis (`syn_narrow`, `syn_epenthesis`) lack identity on featureless + grammars. They were left alone because they affect rendering. + +**Unexplained observations:** +14. **The lookup gap** [HC-Rust vs C#]. HC-Rust on C#'s export made 18 lookups on the slowest word + against C#'s 49. Parses were identical; the cause was not investigated. +15. **Sena SKIPPED rows** [HC-Rust]. A census run with a 30 s wall limit reported 1,012 SKIPPED + rows, while no-limit A/B runs completed all 6,146 words in about 50 s. Not investigated. + +## Data handling + +Private-project word lists, per-word logs and exports stay on the measuring machine. This record +carries only aggregate counts. Conformance fixtures are synthetic. diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index 4042f16ad..4582360a2 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -35,8 +35,8 @@ from a branch is cheap; maintaining research scaffolding in two product assembli checks their code claims, and separates new experiments from this ledger's closed families. It ranks residual lexical viability, a grammar-compiled template battery, recoverable trace-class reduction, and exact finite-state compilation for a verified subgrammar. It maps them to lexicon-driven pruning during -unapplication and making the affix-template battery cheaper without memoization. **No new measurement, -speedup, or numbered optimization row is claimed.** +unapplication and making the affix-template battery cheaper without memoization. The research doc itself claims no measurement. Its first two candidates were measured on +2026-10-21 (rows 27 and 26): see [the search-cost round](hermitcrab-2026-10-21-search-cost-round.md). ## Final disposition of the September follow-up @@ -117,6 +117,27 @@ copy-on-write design through `CloneForEngine`; see `docs/optimizations/shape-sha | 19 | Contexted constraints / abstract feature-only replay | Pack without merging states by replaying only shape-independent constraints. | **CLOSED.** Cost-weighted P1b die-point histograms put shape-free work at **6.0%** and **0.0%** on two heavy Sena probes and **0.4%** on 28 Amharic words; all fire the <30% close gate. All 33 conformance fixtures were measured for breadth: 24 close, 3 nominally fire the build gate, 3 are undecided, 2 have zero rejections, and 1 has zero words. The three build-gate fixtures are all sub-50 ms and cannot carry timing claims. One reliable fixture confirms that count and cost can diverge, but die-point rejection cost itself is only ~0.1–0.2% of wall on the real corpora. Abstract replay would optimize a phase that costs approximately nothing on the corpora treated as ground truth. Not built. | | 20 | **Row 5's family on `GenerateWords`** (generation, ~100% synthesis share vs parsing's ~5%/~0.3%) | Row 20 was unmeasured; generation's share is ~100% so row 5's fold ratios would apply at face value if they transferred. `UseSynthesisFoldMemo` had never been wired into `GenerateWords` (only into the parse-then-synthesize path) — wired in under the same untraced/`MaxDegreeOfParallelism==1` restriction to make this measurable at all | Hits are no longer uniformly zero: 3 of 23 lookup-reaching fixtures clear the 5% gate (diacritic-segments 33.3%, suffixing-vowel-harmony 28.6%, suffixing-evidential-adjacency-chain 37.7%; a 4th sits just under it at 4.8%), reopening row 5 narrowly for generation. But the reopened case recloses immediately: the A/B on the best fixture shows the wall-clock "win" sitting inside its own noise floor, and the pooled hit rate is dominated by `deep-optional-affix-nesting` (925 of 1,613 lookups) at hits=0, reproducing row 5's exact 4.5%-slower key-construction cost on the identical fixture. Parity: 0 divergences, 1,236 generations, 32/33 fixtures (33rd throws identically both sides). Best case: 130 hits / 5 reps (37.7% hit rate), **7.8%** wall-clock reduction inside a **22.2%** off-arm spread — not a result. 19/23 lookup-reaching fixtures at exactly **0%**; pooled hit rate **2.67%** (43/1,613), skewed by the one large zero-hit fixture. | +## 2026-10-21 search-cost round + +Full record: [hermitcrab-2026-10-21-search-cost-round.md](hermitcrab-2026-10-21-search-cost-round.md). +The unit is HC-Rust parser work (summed per-word `work_steps`, deterministic), so "N% less work to +parse the words". Steps are secondary, and ms are labelled. C# figures are in C#'s own counters. +Rows 24 and 25 are **HC-Rust port bugs: C# was already exact, and no C# change follows from them.** + +| # | Optimization | Expected | Outcome | Number | +| --- | --- | --- | --- | --- | +| 24 | **Analysis-side affix literals keep `StrRep` identity on wide featureless tables** (HC-Rust only; PanGloss divergence 090 follow-up) | Running HC-Rust on C#'s own export of a private featureless project made 18 lookups on its slowest word, against 1,655 on HC-Rust's own snapshot (C#: 49). That points at a loading difference, not search | **KEPT in HC-Rust; no C# change.** Above 64 character definitions HC-Rust lost the literal's identity, so an enclitic matched any final segment. C# gives featureless segments a `StrRep` `EqualTo` constraint (`CharacterDefinitionTable.cs:68`) | 7 spot words: **98.5% less work** (2,365,602 -> 35,913); worst variant **99.8% less** (50,778,583 -> 83,677). Steps on full lists fell 46-55x. Parses IDENTICAL on three private lists and on Amharic, Sena and Indonesian | +| 25 | **Re-inserted segment-class node keeps its members on featureless grammars** (HC-Rust only; PanGloss entry 103, `414ac6b5`) | A deletion-rule variant of the same project hit 1 s timeouts. A profile showed analysis at 0.1-0.3% of work, synthesis at ~90%, and zero lexically dead branches | **KEPT in HC-Rust; no C# change.** HC-Rust's re-inserted node matched any segment, and its root trie ignored the member set. C# unifies against the class's `StrRep` union | Held list (7,206 words): **94.7% less work** (303,966,311 -> 16,220,190). The 16 words at the 1 s limit: **99.9% less**. IDENTICAL everywhere; byte-identical work where it cannot fire | +| 26 | **Shared trailing slot trie in the template battery** (research candidate 2; PanGloss 095; Machine **PR #546**) | Templates with identical trailing slots (rules, order, optionality) run the same slot batch on the same word. Sena's trie-exact ceiling is 448,573 batches, 11.3% of work | **KEPT and ported.** Control factoring within one battery call, with no cache. In C# it covers sequential, untraced walks without an alternative limit. It composes with row 2 | HC-Rust: Sena **11.5% less work** (31.6% fewer steps, 448,573 batches avoided = ceiling); Mbugwe 18.1% less (1,364-word partial); Aweti 0.35%; 0 on one-template grammars. C# batches: Sena sample 47,072 -> 32,287 (31.4%); Mbugwe sample 72,467 -> 58,486 (19.3%). C# tests: a 2,904-combination differential, 126/126 HermitCrab tests, conformance 58/60 equal to the unchanged base `aab237b2` with memo on | +| 27 | **Prune analysis branches that cannot reach a root** (research candidate 1; census `d1690c13`) | The hindsight lexically-dead ceiling is 16-75% of parser work | **CLOSED on the prunable share.** Most dead work is leaves, which cost as much to prove dead as to run, or nodes that may still compound (row 10's disqualifier, 36-46% on Amharic, Sena and Mbugwe). Root-reachable synthesis failure is untouched: 86-98% of synthesis work where measured | The plain dead-internal bucket, the only one a sound summary could remove at a profit, is **0-5% on 6 of 8 lists**. The exceptions are v1 at 12.2% and Aweti at 39.8%. **Reopening condition:** a shadow summary on Aweti/v1 deciding a meaningful share of that bucket | +| 28 | **Per-parse distinct-shape lookup dedupe / fixed-cap root-result cache** (research reports R4, R5) | "97,575 lookups over 56 shapes" | **CLOSED.** That figure was trace mode, which disables merging. It is a memo regardless | Untraced, 1.13-1.49 calls per distinct (stratum, shape). Repeat-lookup work is **0.5-3.3%** of parser work (10.4% only on row 25's pre-fix bug). Dead-candidate lookups are 30-75% of lookup work, but lookup is 1-12% of parser work | +| 29 | **Rule-level memo per battery call** | Repeated slot-rule applications inside one battery | **CLOSED.** It is no bigger than row 26 and it is a memo | Sena repeats: 33.7M work units (rule-level) vs 32.9M (slot-level) | +| 30 | **Hoist `segs_of` / borrow segments in `Transduce::new`; bitset for the root-trie active set** (HC-Rust) | Fewer allocations and a less quadratic trie walk | **NOT BUILT.** Time-only, with 0 work units by construction, so outside the owner metric | ~5.1M avoidable segment-vector builds per private-v1 held run (5.48M slot-rule attempts vs 365K slot batches). `Vec::contains` on the trie active set is quadratic | + +**Lesson of the round:** two of its three large reductions (rows 24 and 25) were port bugs, found +by running the port on the oracle's own grammar export and comparing lookup counts. Do that before +optimizing the algorithm. + --- ## The lesson that generalises From 1e2134aa79bca3961eea666be440f86af34b6199 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 22 Oct 2026 06:24:10 -0400 Subject: [PATCH 08/32] Correct the Bantu range for the edge prefilter The disposition table gives Sena at -2% and Mbugwe at -13% wall time; the 7% figure is Amharic, which is not Bantu. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-perf-2026-09-disposition.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hermitcrab-perf-2026-09-disposition.md b/docs/hermitcrab-perf-2026-09-disposition.md index 596c4e7cf..7286e5dda 100644 --- a/docs/hermitcrab-perf-2026-09-disposition.md +++ b/docs/hermitcrab-perf-2026-09-disposition.md @@ -70,7 +70,7 @@ under a real signature. But the traversals removed are the cheap ones: they woul check, so wall time moves only 2–13%. The branch's package figure (1.76x Amharic, 1.83x Mbugwe) therefore came mostly from the allocation, clone, and GC work, not from this mechanism. Standalone, the prefilter is correct, small, and reviewable, but it is **not a large return**. Recommendation at the time: keep `perf/hc-edge-prefilter` -ready as an optional small PR (7–13% on Bantu grammars, where it stacks with #491's prune on the surviving +ready as an optional small PR (2–13% on Bantu grammars, where it stacks with #491's prune on the surviving branches), and do not spend review budget on it ahead of #491. The standalone single-commit branch is deleted (see status note above); a PR now starts from the mechanism preserved in the whole-round archive (branch `perf/hc-optimization-archive`) rather than resuming From 5cd2fb7fa4791dbc681c8451b5de2c3506ff8584 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 22 Oct 2026 06:26:26 -0400 Subject: [PATCH 09/32] Record that PR #491 merged and which concerns remain #491 merged on 2026-09-28 as b240b7b5. On master the duplicate parse and the affix-rule state transitions while disabled are fixed; the guard is still grammar-wide and still counts partial lexical entries. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-perf-2026-09-disposition.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/hermitcrab-perf-2026-09-disposition.md b/docs/hermitcrab-perf-2026-09-disposition.md index 7286e5dda..01b1c21bc 100644 --- a/docs/hermitcrab-perf-2026-09-disposition.md +++ b/docs/hermitcrab-perf-2026-09-disposition.md @@ -90,7 +90,7 @@ branches remain. | `perf/hc-engine-alloc` | `f22f88a8` | allocation/two-pass side branch | fully contained in the archive; worktree removed | | `perf/hc-conformance-check` | `0799a977` | archive merged into the conformance integration branch for the 603-test gate | verification only; worktree `.worktrees/hc-conf` can go | | `perf/hc-edge-prefilter` | `be7f241d` | the one isolated mechanism on `origin/master` | optional small PR at the time; **branch and its own archive tag have both since been deleted, but the mechanism is preserved on branch `perf/hc-optimization-archive`** | -| `pr-491` | `96c97f76` | jtmaxwell3's PR head | worktree `.worktrees/pr491` holds uncommitted throwaway counters + `FinalTemplateUpperBound` harness | +| `pr-491` | `96c97f76` | jtmaxwell3's PR head at review time | merged to master as `b240b7b5` on 2026-09-28; worktree `.worktrees/pr491` holds uncommitted throwaway counters + `FinalTemplateUpperBound` harness | | `docs/hc-optimization-ledger` (remote, PR #490) | `cc806613` | the ledger, now 23 rows | open docs PR | | `feature/forest-memo`, `feature/synthesis-fold-probes`, `feature/synthesis-fold-sharing`, `feature/per-node-cost` | — | evidence branches of the ledger round | see `hermitcrab-optimization-ledger.md` | | `investigate-memoization-rule-order` | `4f5aeeb4` | despite the name, conformance-suite commits | belongs to the conformance line, not this round | @@ -125,8 +125,11 @@ word list, and the rulestats HTML report so they cannot be staged by accident. ## What to do next, in order -1. Fix and merge PR #491 (duplicate parses, memo splitting while disabled, entry-counting guard). Draft comment in the - re-review record. +1. ~~Fix and merge PR #491~~ — merged on 2026-09-28 as `b240b7b5`. Two of the three concerns are fixed on + master: a final template that would follow a non-template now returns no output instead of marking its + clone, so the duplicate parse is gone, and the affix-process rules set `FinalTemplateState` only while + the prune is enabled (the compounding rule still sets it unconditionally). The guard is unchanged: it is + still grammar-wide and still counts partial lexical entries (`Morpher.IsPartial`). 2. Classify the unclassified affixes in the FLEx projects (Mbugwe 1, Amharic 1, Sena 6); that turns #491's default prune on. Add the finding to the grammar advisor. 3. ~~Open the prefilter PR from `perf/hc-edge-prefilter`~~ — that branch and its own archive tag are both From fe696cd1494d7b1da99193e7fe89faaa847933f4 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 22 Oct 2026 06:26:34 -0400 Subject: [PATCH 10/32] Correct the wall-time slice count to seven The probe design names seven slices (six measured plus the unaccounted remainder) but called them eight; the ledger repeated the count twice. The probe source is on no branch, so the count follows the named list. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-optimization-ledger.md | 4 ++-- docs/hermitcrab-probe-design.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index 4582360a2..59f2d62a6 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -18,7 +18,7 @@ product assemblies, since the probes gate on `volatile bool` reads inside `Match loop in `SIL.Machine` used by every consumer. **`docs/hermitcrab-probe-design.md` is the replacement.** It carries the probe architecture and its -two-assembly gate trap, the eight exclusive wall-time buckets and why `synForward` must be recorded +two-assembly gate trap, the seven exclusive wall-time buckets and why `synForward` must be recorded net of cascade and battery, the per-node decomposition, the census designs, all three key designs with field-level audits, the classifier's direction trap, and the full A/B methodology with the three measurement failures that forced each part of it — enough to rebuild any probe in a day. The @@ -163,7 +163,7 @@ packs without merging states; row 19 was later measured independently and closed ## Where the time actually is -Measured with eight exclusive buckets summing to wall. Amharic is a single run (`unaccounted` +Measured with seven exclusive buckets summing to wall. Amharic is a single run (`unaccounted` 0.1%). **Sena is spliced from two runs -- see the note in its row.** | grammar | breakdown | diff --git a/docs/hermitcrab-probe-design.md b/docs/hermitcrab-probe-design.md index eb6275d85..88ce6cf49 100644 --- a/docs/hermitcrab-probe-design.md +++ b/docs/hermitcrab-probe-design.md @@ -30,8 +30,8 @@ accumulated with `Interlocked.Add`, with `Reset*` methods so a harness reads per ### The wall-time split -**Eight mutually exclusive top-level slices that sum to wall**, measured around -`Morpher.ParseWord`: +**Seven mutually exclusive top-level slices that sum to wall** (six measured plus the remainder), +measured around `Morpher.ParseWord`: `lookup` (brackets `LexicalLookup`, materialised with `.ToList()` so the bracket covers only the lookup itself) · `synCascade` · `synBattery` · `synForward` · `synExpand` (brackets From c146c78dbc06ef5309955ca1bf72714dd49bc65b Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 22 Oct 2026 06:26:44 -0400 Subject: [PATCH 11/32] Renumber the last two method rules as 11 and 12 They restarted at 7 and 8 after rule 10, so "rule 7" and "rule 8" each named two rules. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-optimization-ledger.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index 59f2d62a6..94e674b2b 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -274,12 +274,12 @@ Written before the measurements, scored after. Kept because the misses are infor first write-up. 10. **Search completeness is never traded for speed.** HermitCrab is the permanent fallback engine behind the FST work. -7. **A bucket reporting exactly `0.00` is broken until proven otherwise.** `NodeCostProbe.Enabled` +11. **A bucket reporting exactly `0.00` is broken until proven otherwise.** `NodeCostProbe.Enabled` was declared and read in `Matcher.cs` but set by nothing — two gates in two assemblies, one wired. Both lower-layer buckets read `0.00ms`, which looks like "measured and negligible" rather than "never ran", and their time silently inflated the remainder. Wire cross-assembly probe gates from a single place. -8. **Always carry an explicit remainder column.** It caught the wall-split misattribution and the +12. **Always carry an explicit remainder column.** It caught the wall-split misattribution and the dead probe gate. Four of the five retractions in this work would have been caught at the point of measurement by a remainder that did not sum. From 3490d744bfe061200b00deeb89828e2cad04dcc6 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 22 Oct 2026 06:26:59 -0400 Subject: [PATCH 12/32] Resolve the clone-discard gate in the ledger The paragraph still listed measuring the clone-discard fraction as an open gate after reporting the 66.2% result. State the outcome, and say why the retained deferred template clone does not contradict it: row 17b counts only that clone site, and deferring it still skips the discarded two thirds there. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-optimization-ledger.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index 94e674b2b..03744c0f6 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -205,15 +205,18 @@ prefilter" — 92.3% do fail, but matching is 12.5%, so it does not dominate; do So there is no declared build target yet. The next probe splits the 37.4%: `RuleCascade` / enumerator orchestration, trail dictionary operations, LINQ allocation, memo lookups, GC. -**A caution on the 46.7% before anyone acts on it.** `Word.Clone` being half the cascade does not -by itself mean cloning is waste — the cascade forks a `Word` per rule application precisely so it -can explore alternatives without mutating shared state. The deciding question is what fraction of -clones are *discarded unused*: cloned, rejected by a guard or a failed match, thrown away. Given -92.3% of pattern matches fail, that fraction has now been **measured at 66.2%** (row 17b), which misses the >=70% gate and points at the cheaper-clone build rather than the defer-clone one. If most -clones are discarded, the lever is **not cloning until a rule commits**. If most survive, the lever -is **making a clone cheaper** (copy-on-write shape, which the RUSTIFY branch already explored). -Different builds; measure first. **Gate: probe the clone-discard fraction; >=70% discarded means -the defer-clone build, below that the cheaper-clone build.** +**A caution on the 46.7% before anyone acts on it.** `Word.Clone` being half the cascade does not by +itself mean cloning is waste — the cascade forks a `Word` per rule application precisely so it can +explore alternatives without mutating shared state. The deciding question was what fraction of +clones are *discarded unused*: cloned, rejected by a guard or a failed match, thrown away. If most +are discarded, the lever is **not cloning until a rule commits**; if most survive, the lever is +**making a clone cheaper** (copy-on-write shape, which the RUSTIFY branch already explored). The +gate was >=70% discarded for the defer-clone build. **Measured at 66.2% (row 17b), it misses the +gate, so the main build target is the cheaper clone.** That does not contradict the retained +deferred template clone: row 17b counts only `AnalysisAffixTemplateRule`'s eager per-template clone, +the same site that change defers, and deferring there still skips the two thirds of those clones +that are discarded. It is a local 1.054x fix, not the main lever; the retained copy-on-write shape +sharing is the cheaper-clone build. ## Two durable code constraints From 25c40214b0b437ad0cb3d88f084120dd44048263 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 22 Oct 2026 06:27:09 -0400 Subject: [PATCH 13/32] Point the disposition at the archive tip and handoff path The archive branch on origin is one squashed commit, 3a0a3316; neither 179708d2 nor b58b1642 is reachable from it. The PanGloss handoff is docs/pangloss-final-template-prune-handoff.md on that branch, not a scratchpad file. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-perf-2026-09-disposition.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/hermitcrab-perf-2026-09-disposition.md b/docs/hermitcrab-perf-2026-09-disposition.md index 01b1c21bc..bcceb2706 100644 --- a/docs/hermitcrab-perf-2026-09-disposition.md +++ b/docs/hermitcrab-perf-2026-09-disposition.md @@ -1,8 +1,9 @@ # HermitCrab optimization round, September 2026: final disposition of the branch **Date:** 2026-09-03. **Branch:** `perf/hc-optimization-pr` @ `179708d2`, renamed `perf/hc-optimization-archive`. -**Decision:** do not merge the branch. Isolate one mechanism onto master as its own small PR; archive the rest with -its records. Re-focus on sillsdev/machine PR #491 and on grammar hygiene, which are where the large returns are. +Origin now holds that branch as one squashed commit, `3a0a3316`; that is the authoritative tip. **Decision:** do not +merge the branch. Isolate one mechanism onto master as its own small PR; archive the rest with its records. Re-focus +on sillsdev/machine PR #491 and on grammar hygiene, which are where the large returns are. ## Why the package is not merged @@ -85,7 +86,7 @@ branches remain. | Branch | Tip | What it is | State | | --- | --- | --- | --- | -| `perf/hc-optimization-archive` (was `perf/hc-optimization-pr`) | `b58b1642`+ | the whole round, code + records | archived, unmerged; worktree `.worktrees/ledger` | +| `perf/hc-optimization-archive` (was `perf/hc-optimization-pr`) | `3a0a3316` (squashed) | the whole round, code + records | archived, unmerged; worktree `.worktrees/ledger` | | `perf/hc-defer-template-clone` | `4a6a47ed` | working branch of the round | fully contained in the archive; worktree removed | | `perf/hc-engine-alloc` | `f22f88a8` | allocation/two-pass side branch | fully contained in the archive; worktree removed | | `perf/hc-conformance-check` | `0799a977` | archive merged into the conformance integration branch for the 603-test gate | verification only; worktree `.worktrees/hc-conf` can go | @@ -135,4 +136,5 @@ word list, and the rulestats HTML report so they cannot be staged by accident. 3. ~~Open the prefilter PR from `perf/hc-edge-prefilter`~~ — that branch and its own archive tag are both deleted; a PR now starts from the mechanism preserved on branch `perf/hc-optimization-archive` rather than resuming the deleted branch. -4. Port the prune to PanGloss per the handoff (`HANDOFF-pangloss-final-template-prune.md`, session scratchpad). +4. Port the prune to PanGloss per the handoff, `docs/pangloss-final-template-prune-handoff.md` on branch + `perf/hc-optimization-archive`. From f7d3d518ce3d472c17d18e59923f0a837d328dce Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 22 Oct 2026 07:01:35 -0400 Subject: [PATCH 14/32] Drop the compounding-state remark from the #491 status The analysis memo is being removed, so FinalTemplateState set unconditionally by the compounding rule no longer splits anything worth recording. Keep the observations about master as they stand. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-perf-2026-09-disposition.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/hermitcrab-perf-2026-09-disposition.md b/docs/hermitcrab-perf-2026-09-disposition.md index bcceb2706..8f7743156 100644 --- a/docs/hermitcrab-perf-2026-09-disposition.md +++ b/docs/hermitcrab-perf-2026-09-disposition.md @@ -126,11 +126,10 @@ word list, and the rulestats HTML report so they cannot be staged by accident. ## What to do next, in order -1. ~~Fix and merge PR #491~~ — merged on 2026-09-28 as `b240b7b5`. Two of the three concerns are fixed on - master: a final template that would follow a non-template now returns no output instead of marking its - clone, so the duplicate parse is gone, and the affix-process rules set `FinalTemplateState` only while - the prune is enabled (the compounding rule still sets it unconditionally). The guard is unchanged: it is - still grammar-wide and still counts partial lexical entries (`Morpher.IsPartial`). +1. ~~Fix and merge PR #491~~ — merged on 2026-09-28 as `b240b7b5`. On master the duplicate parse is fixed (a + final template that would follow a non-template now returns no output instead of marking its clone), and + the affix-process rules set `FinalTemplateState` only while the prune is enabled. The guard, + `Morpher.IsPartial`, is still grammar-wide and still counts partial lexical entries. 2. Classify the unclassified affixes in the FLEx projects (Mbugwe 1, Amharic 1, Sena 6); that turns #491's default prune on. Add the finding to the grammar advisor. 3. ~~Open the prefilter PR from `perf/hc-edge-prefilter`~~ — that branch and its own archive tag are both From e1a212dfb4ad2af7272aee27b9cb7a9efd5583cf Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 22 Oct 2026 07:01:35 -0400 Subject: [PATCH 15/32] State the clone-discard result without an unmeasured reconciliation The previous wording inferred that the deferred template clone's 1.054x came from skipping the discarded clones at that site. That link was never measured, so say only that both concern the same site. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-optimization-ledger.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index 03744c0f6..bd559e674 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -212,11 +212,9 @@ clones are *discarded unused*: cloned, rejected by a guard or a failed match, th are discarded, the lever is **not cloning until a rule commits**; if most survive, the lever is **making a clone cheaper** (copy-on-write shape, which the RUSTIFY branch already explored). The gate was >=70% discarded for the defer-clone build. **Measured at 66.2% (row 17b), it misses the -gate, so the main build target is the cheaper clone.** That does not contradict the retained -deferred template clone: row 17b counts only `AnalysisAffixTemplateRule`'s eager per-template clone, -the same site that change defers, and deferring there still skips the two thirds of those clones -that are discarded. It is a local 1.054x fix, not the main lever; the retained copy-on-write shape -sharing is the cheaper-clone build. +gate, so the main build target is the cheaper clone.** Row 17b counts only +`AnalysisAffixTemplateRule`'s eager per-template clone, the site the retained deferred template +clone changes; how that change's 1.054x relates to the 66.2% was not measured. ## Two durable code constraints From 6c7c7d13a004452eb71085eb052f27261e185c0a Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 22 Oct 2026 15:49:33 -0400 Subject: [PATCH 16/32] Point the shadow experiment at research lead 1 and widen the lookup range "Close lead 1" named the FST precheck in the open-leads list; the shadow experiment closes research lead 1 (C1). The 1-12% lookup share left out the deletion variant's 35.7% before fix B, here and in ledger row 28. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-2026-10-21-search-cost-round.md | 5 +++-- docs/hermitcrab-optimization-ledger.md | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/hermitcrab-2026-10-21-search-cost-round.md b/docs/hermitcrab-2026-10-21-search-cost-round.md index 3522a8a74..92ddde303 100644 --- a/docs/hermitcrab-2026-10-21-search-cost-round.md +++ b/docs/hermitcrab-2026-10-21-search-cost-round.md @@ -177,7 +177,8 @@ untraced census confirms it: - **Policy.** Avoiding it would be a per-parse memo, which this work excludes. C#'s `LexicalLookup` repeats lookups the same way, so there is no Machine PR. Dead-candidate lookups -are 30-75% of lookup work, but lookup is only 1-12% of parser work, so they were not pursued either. +are 30-75% of lookup work, but lookup is only 1-12% of parser work (35.7% on the +deletion variant before fix B), so they were not pursued either. ## C2: share trailing slot batches inside one battery call (research lead #2) — kept, ported to C# @@ -332,7 +333,7 @@ Each lead is tagged **[C# analogue]** where the same question applies to `hc.dll 2. **Shadow summary experiment on Aweti and v1** [C# analogue]. These are the only lists whose plain dead-internal bucket is material (39.8% and 12.2%). Run a sound summary in shadow mode and count the plain-bucket work it would actually decide, along with the summary's own cost. Close - lead 1 if the decided share is small. + research lead 1 (C1) if the decided share is small. 3. **Mbugwe's step-capped words** [C# analogue]. 86 of 1,645 words hit the 2M step cap and account for about 2/3 of Mbugwe's work. They have not been examined individually. 4. **Root-reachable synthesis failure** [C# analogue]. 86% (deletion variant, before fix B) and 98% diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index bd559e674..725afa77f 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -130,7 +130,7 @@ Rows 24 and 25 are **HC-Rust port bugs: C# was already exact, and no C# change f | 25 | **Re-inserted segment-class node keeps its members on featureless grammars** (HC-Rust only; PanGloss entry 103, `414ac6b5`) | A deletion-rule variant of the same project hit 1 s timeouts. A profile showed analysis at 0.1-0.3% of work, synthesis at ~90%, and zero lexically dead branches | **KEPT in HC-Rust; no C# change.** HC-Rust's re-inserted node matched any segment, and its root trie ignored the member set. C# unifies against the class's `StrRep` union | Held list (7,206 words): **94.7% less work** (303,966,311 -> 16,220,190). The 16 words at the 1 s limit: **99.9% less**. IDENTICAL everywhere; byte-identical work where it cannot fire | | 26 | **Shared trailing slot trie in the template battery** (research candidate 2; PanGloss 095; Machine **PR #546**) | Templates with identical trailing slots (rules, order, optionality) run the same slot batch on the same word. Sena's trie-exact ceiling is 448,573 batches, 11.3% of work | **KEPT and ported.** Control factoring within one battery call, with no cache. In C# it covers sequential, untraced walks without an alternative limit. It composes with row 2 | HC-Rust: Sena **11.5% less work** (31.6% fewer steps, 448,573 batches avoided = ceiling); Mbugwe 18.1% less (1,364-word partial); Aweti 0.35%; 0 on one-template grammars. C# batches: Sena sample 47,072 -> 32,287 (31.4%); Mbugwe sample 72,467 -> 58,486 (19.3%). C# tests: a 2,904-combination differential, 126/126 HermitCrab tests, conformance 58/60 equal to the unchanged base `aab237b2` with memo on | | 27 | **Prune analysis branches that cannot reach a root** (research candidate 1; census `d1690c13`) | The hindsight lexically-dead ceiling is 16-75% of parser work | **CLOSED on the prunable share.** Most dead work is leaves, which cost as much to prove dead as to run, or nodes that may still compound (row 10's disqualifier, 36-46% on Amharic, Sena and Mbugwe). Root-reachable synthesis failure is untouched: 86-98% of synthesis work where measured | The plain dead-internal bucket, the only one a sound summary could remove at a profit, is **0-5% on 6 of 8 lists**. The exceptions are v1 at 12.2% and Aweti at 39.8%. **Reopening condition:** a shadow summary on Aweti/v1 deciding a meaningful share of that bucket | -| 28 | **Per-parse distinct-shape lookup dedupe / fixed-cap root-result cache** (research reports R4, R5) | "97,575 lookups over 56 shapes" | **CLOSED.** That figure was trace mode, which disables merging. It is a memo regardless | Untraced, 1.13-1.49 calls per distinct (stratum, shape). Repeat-lookup work is **0.5-3.3%** of parser work (10.4% only on row 25's pre-fix bug). Dead-candidate lookups are 30-75% of lookup work, but lookup is 1-12% of parser work | +| 28 | **Per-parse distinct-shape lookup dedupe / fixed-cap root-result cache** (research reports R4, R5) | "97,575 lookups over 56 shapes" | **CLOSED.** That figure was trace mode, which disables merging. It is a memo regardless | Untraced, 1.13-1.49 calls per distinct (stratum, shape). Repeat-lookup work is **0.5-3.3%** of parser work (10.4% only on row 25's pre-fix bug). Dead-candidate lookups are 30-75% of lookup work, but lookup is 1-12% of parser work (35.7% on the deletion variant before fix B) | | 29 | **Rule-level memo per battery call** | Repeated slot-rule applications inside one battery | **CLOSED.** It is no bigger than row 26 and it is a memo | Sena repeats: 33.7M work units (rule-level) vs 32.9M (slot-level) | | 30 | **Hoist `segs_of` / borrow segments in `Transduce::new`; bitset for the root-trie active set** (HC-Rust) | Fewer allocations and a less quadratic trie walk | **NOT BUILT.** Time-only, with 0 work units by construction, so outside the owner metric | ~5.1M avoidable segment-vector builds per private-v1 held run (5.48M slot-rule attempts vs 365K slot batches). `Vec::contains` on the trie active set is quadratic | From c3100a3a6972ea7edd3b6d94906a70ddf5467d91 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 23 Oct 2026 07:07:46 -0400 Subject: [PATCH 17/32] Qualify the remaining Sena parity claims as morph-count matches The counts table, the deferred-clone record and the edge-prefilter record still said "exact parity" or "identical analysis signature sets" for Sena. On Sena, WordAnalysisSignature compares morph counts only. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-perf-2026-09-counts.md | 2 +- docs/optimizations/deferred-template-clone.md | 3 ++- docs/optimizations/edge-segment-prefilter.md | 3 ++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/hermitcrab-perf-2026-09-counts.md b/docs/hermitcrab-perf-2026-09-counts.md index 16cc86354..941ecaef0 100644 --- a/docs/hermitcrab-perf-2026-09-counts.md +++ b/docs/hermitcrab-perf-2026-09-counts.md @@ -23,7 +23,7 @@ portability, and limits: | Optimization | Principal evidence | Record | | --- | --- | --- | -| Deferred template clone | 1.054x on the 45-word Sena battery; exact parity | `optimizations/deferred-template-clone.md` | +| Deferred template clone | 1.054x on the 45-word Sena battery; `WordAnalysisSignature` parity (morph counts only on Sena) | `optimizations/deferred-template-clone.md` | | Feature-structure traversal-state reuse | equality/freezing allocation left the hot profile; not isolated | `optimizations/feature-struct-visited-state-reuse.md` | | Frozen flat-tree equality | removes visited sets from the proved tree case; not isolated | `optimizations/feature-struct-frozen-tree-equality.md` | | Frozen flat-tree clone | removes the copies map from the proved tree case; not isolated | `optimizations/feature-struct-frozen-tree-clone.md` | diff --git a/docs/optimizations/deferred-template-clone.md b/docs/optimizations/deferred-template-clone.md index 59155adf1..ebd7d24be 100644 --- a/docs/optimizations/deferred-template-clone.md +++ b/docs/optimizations/deferred-template-clone.md @@ -27,7 +27,8 @@ tried first and failed the new tests immediately (`FeatureStruct.Add` threw on a ## Measured and verified Alternating wall-time A/B against the pinned baseline, 45 Sena words, min of two interleaved rounds: -1.054x overall, heavy words 1.02x to 1.08x; parity clean on all words and both rounds. Smaller than the +1.054x overall, heavy words 1.02x to 1.08x; `WordAnalysisSignature` parity on all words and both rounds (on +Sena that signature compares morph counts only; see `hermitcrab-perf-2026-09-disposition.md`). Smaller than the ledger's framing suggested: the eager clone was not where the heavy words' time went. Focused template tests cover required/optional slots, traced execution, and both sequential and parallel diff --git a/docs/optimizations/edge-segment-prefilter.md b/docs/optimizations/edge-segment-prefilter.md index e6215d81b..048bfa802 100644 --- a/docs/optimizations/edge-segment-prefilter.md +++ b/docs/optimizations/edge-segment-prefilter.md @@ -42,7 +42,8 @@ would have produced, so traces are unchanged. | traversal instances used | 15,642,175 | 2,429,643 | | pattern-match attempts (counted before the check) | 11,979,916 | 11,979,916 | -Parity: identical analysis signature sets on all 45 Sena words and on all 464 conformance-fixture words +Parity: identical `WordAnalysisSignature` sets on all 45 Sena words (morph counts only on Sena; see +`hermitcrab-perf-2026-09-disposition.md`) and on all 464 conformance-fixture words (33 grammars). Fixture traversals fell 21% (31,278 to 24,741); their patterns fail less often. `AnalysisEdgePrefilterTests` covers left/right constraints, boundary skipping, optional nodes, and the From 94348274825a9498be0ed1b417247413e0942216 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 23 Oct 2026 07:07:57 -0400 Subject: [PATCH 18/32] Say the probe design's parity gate compares morph counts on FLEx grammars WordAnalysisSignature joins Morpheme.Id, which is empty in the FLEx exports, so a harness rebuilt from this section would pass Sena, Mbugwe or Amharic whenever morph counts match. Point at the gloss + allomorph + feature-structure signature the disposition asks for. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-probe-design.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/hermitcrab-probe-design.md b/docs/hermitcrab-probe-design.md index 88ce6cf49..07f807ab5 100644 --- a/docs/hermitcrab-probe-design.md +++ b/docs/hermitcrab-probe-design.md @@ -179,7 +179,10 @@ disables the memo, so a traced run measures nothing. Parity is the gate and speed is only reported. Acceptance is **analysis-set equality on canonical morpheme-signature sets** (`MorpherTests.WordAnalysisSignature`, sorted) — never object or byte -equality, because a replayed `Word` is legitimately not field-for-field identical. +equality, because a replayed `Word` is legitimately not field-for-field identical. That signature joins +`Morpheme.Id`, which is empty in every FLEx export used, so on Sena, Mbugwe and Amharic it compares morph counts +only. A rebuilt gate for those grammars needs a gloss + allomorph-index + syntactic-feature-structure signature, +compared as multisets (`hermitcrab-perf-2026-09-disposition.md`). Timing: **discard a warm-up rep per arm, then take min of N interleaved samples, and print the off-arm spread beside every speedup.** Refuse any speedup smaller than its own spread. Three From 7b0019998959350efe6993d728326e43a143a393 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 23 Oct 2026 07:08:22 -0400 Subject: [PATCH 19/32] Label the probe design's AnalysisStateKey fields as pre-#491 PR #491 added FinalTemplateState to the key, and #522 then removed AnalysisStateKey with the analysis memo. Name master's AnalysisMergeKey, which carries the listed fields plus FinalTemplateState. Also say that the two engine facts in section 4 concern the removed memo. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-probe-design.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/hermitcrab-probe-design.md b/docs/hermitcrab-probe-design.md index 07f807ab5..6877470f4 100644 --- a/docs/hermitcrab-probe-design.md +++ b/docs/hermitcrab-probe-design.md @@ -88,13 +88,18 @@ allomorph pattern match / rule-not-applicable, allomorph environments, final `Is ## 2. The keys -### `AnalysisStateKey` (pre-existing) +### `AnalysisStateKey` (as it stood before #491) `(Shape, Stratum, SyntacticFeatureStruct, RealizationalFeatureStruct, NonHeadCount, per-rule un-application count multiset)`. Order-independent by design — no analysis-side rule reads trail order, which is the redundancy it collapses. The multiset is XOR-combined, not `*31`-rolled, so it hashes commutatively. +This field list predates #491, which added `FinalTemplateState` to the key; #522 then removed +`AnalysisStateKey` along with the analysis memo. Master's `AnalysisStratumRule.AnalysisMergeKey` carries the +fields above plus `FinalTemplateState`. A rebuilt or narrowed key that omits `FinalTemplateState` merges states +that #491's prune treats differently. + **Freeze-on-read in the constructor is mandatory.** `AnalysisAffixTemplateRule.Apply` reassigns `SyntacticFeatureStruct` to a fresh unfrozen clone *after* the owning `Word` is frozen, and that setter has no `CheckFrozen()` guard. @@ -228,6 +233,9 @@ grammar and word keys symmetrically before comparing errors, signatures, timing, ## 4. Two durable engine facts, restated here so they survive independently +Both facts concern the analysis memo that #522 removed from master. They apply to the archived branches and to +any revival of the memo. + **`Word.ReplayOnto` does not splice `_mrulesUnapplied`.** It splices `_mruleApps` and `_nonHeadApps` only. This is safe *solely* because `AnalysisStateKey` includes the count multiset, which guarantees a memo hit has identical arrival and stored-arrival counts. **Narrowing that key breaks correctness From 87a39b5eb8b9b10e9618a6d3321824bfbb0a777c Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 23 Oct 2026 07:08:40 -0400 Subject: [PATCH 20/32] Say PerfBench and the bench compare script are on the archive branch Neither PerfBench nor scripts/hc-bench-compare.ps1 is on master; both are on perf/hc-optimization-archive. The counts record's "final branch" that carries the harness is the same branch. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-perf-2026-09-counts.md | 3 ++- docs/hermitcrab-probe-design.md | 13 +++++++------ 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/hermitcrab-perf-2026-09-counts.md b/docs/hermitcrab-perf-2026-09-counts.md index 941ecaef0..1063034b7 100644 --- a/docs/hermitcrab-perf-2026-09-counts.md +++ b/docs/hermitcrab-perf-2026-09-counts.md @@ -2,7 +2,8 @@ This is the final record for the count-driven round that followed `hermitcrab-optimization-ledger.md`. Temporary counters were used to choose targets and were then removed; -the final branch carries only product changes, focused tests, the parity/timing harness, and these records. +the final branch (`perf/hc-optimization-archive`, not master) carries only product changes, focused tests, the +parity/timing harness, and these records. ## Measurement scope diff --git a/docs/hermitcrab-probe-design.md b/docs/hermitcrab-probe-design.md index 6877470f4..34baabdee 100644 --- a/docs/hermitcrab-probe-design.md +++ b/docs/hermitcrab-probe-design.md @@ -219,15 +219,16 @@ task keeps mutating engine state after the harness moves on. Harmless with a per ### Final reusable harness -The retained `PerfBench` is narrower than the historical probes above. It accepts `HC_BENCH_GRAMMAR`, -`HC_BENCH_WORD_LIST`, `HC_BENCH_REPS`, `HC_BENCH_OUT`, and `HC_BENCH_LABEL`; it records timings, errors, -resource observations, memo diagnostics, and canonical result signatures. It does not retain the completed -round's hot-loop counters. +The retained `PerfBench` lives on branch `perf/hc-optimization-archive`, not on master, and is narrower than the +historical probes above. It accepts `HC_BENCH_GRAMMAR`, `HC_BENCH_WORD_LIST`, `HC_BENCH_REPS`, `HC_BENCH_OUT`, +and `HC_BENCH_LABEL`; it records timings, errors, resource observations, memo diagnostics, and canonical result +signatures. It does not retain the completed round's hot-loop counters. Timeouts are now the caller's responsibility: start one `dotnet test` process for an arm and terminate the whole process if its budget expires. Do not restore an in-process `Task` timeout—the abandoned parse continues -running, contaminates later samples, and can race shared engine state. `scripts/hc-bench-compare.ps1` compares -grammar and word keys symmetrically before comparing errors, signatures, timing, or spread. +running, contaminates later samples, and can race shared engine state. `scripts/hc-bench-compare.ps1`, also on +that branch only, compares grammar and word keys symmetrically before comparing errors, signatures, timing, or +spread. --- From c0c086f8391e40152e10674d14902c50914e3c3f Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 23 Oct 2026 07:08:49 -0400 Subject: [PATCH 21/32] Point the ledger's inert-prune note at its superseding addendum The note said the #491 prune was inert and removed, without saying that the prune measured 8-33x with the partiality guard lifted and that #491 later merged as b240b7b5. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-optimization-ledger.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index 725afa77f..e89bd8ea5 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -313,7 +313,9 @@ with the code: `docs/hermitcrab-perf-2026-09-counts.md` and one file per change 2025 `investigate_linear_traverse` branch. Both were tested. The prune was inert on every local corpus because each has a partial morpheme, and the traversal measured negative by counts (instances used 2.3x) because the current traverser already pools instances. Both implementations and their toggles were removed; - their rejected-lead records remain. + their rejected-lead records remain. The prune verdict was later superseded: measured with the partiality guard + lifted, the prune is 8-33x on Mbugwe and Sena with identical analysis sets, and PR #491 merged to master as + `b240b7b5` on 2026-09-28 (see the addendum in `rejected-optimizations/final-template-prune.md`). - **Wall time on the measurement machine moved 2x between windows with identical binaries** (a session transition in between). Every verdict in this ledger that rests on a wall-time delta under ~20% should be read with that in mind; the round's method note is to count steps on the sequential path. From 8bf5fd2d4678d84abecc8c89791de0759e5cbe57 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 23 Oct 2026 07:09:02 -0400 Subject: [PATCH 22/32] Update the disposition's ledger row count to rows 1-30 The branch inventory and the records list still said 23 rows; the ledger now runs to row 30. The inventory's tip for this PR's branch was also a stale SHA, so it now names the PR head. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-perf-2026-09-disposition.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hermitcrab-perf-2026-09-disposition.md b/docs/hermitcrab-perf-2026-09-disposition.md index 8f7743156..518ab2632 100644 --- a/docs/hermitcrab-perf-2026-09-disposition.md +++ b/docs/hermitcrab-perf-2026-09-disposition.md @@ -92,11 +92,11 @@ branches remain. | `perf/hc-conformance-check` | `0799a977` | archive merged into the conformance integration branch for the 603-test gate | verification only; worktree `.worktrees/hc-conf` can go | | `perf/hc-edge-prefilter` | `be7f241d` | the one isolated mechanism on `origin/master` | optional small PR at the time; **branch and its own archive tag have both since been deleted, but the mechanism is preserved on branch `perf/hc-optimization-archive`** | | `pr-491` | `96c97f76` | jtmaxwell3's PR head at review time | merged to master as `b240b7b5` on 2026-09-28; worktree `.worktrees/pr491` holds uncommitted throwaway counters + `FinalTemplateUpperBound` harness | -| `docs/hc-optimization-ledger` (remote, PR #490) | `cc806613` | the ledger, now 23 rows | open docs PR | +| `docs/hc-optimization-ledger` (remote, PR #490) | the PR head | the ledger, rows 1-30 | open docs PR | | `feature/forest-memo`, `feature/synthesis-fold-probes`, `feature/synthesis-fold-sharing`, `feature/per-node-cost` | — | evidence branches of the ledger round | see `hermitcrab-optimization-ledger.md` | | `investigate-memoization-rule-order` | `4f5aeeb4` | despite the name, conformance-suite commits | belongs to the conformance line, not this round | -Records in this directory: `hermitcrab-optimization-ledger.md` (23 attempts), `hermitcrab-perf-2026-09-counts.md` +Records in this directory: `hermitcrab-optimization-ledger.md` (rows 1-30), `hermitcrab-perf-2026-09-counts.md` (count round), `hermitcrab-probe-design.md`, `optimizations/*` and `rejected-optimizations/*` (one record per mechanism), and this file. The round's execution checklist has been folded into this record; it is not kept as a separate plan document. From 7e5079fb787521498152060780c993e401aff904 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 23 Oct 2026 09:38:26 -0400 Subject: [PATCH 23/32] Correct step 2 of the HermitCrab disposition: classifying affixes enables the prune only for Mbugwe and Amharic Sena stays partial because of its 25 partial lexical entries, since Morpher.IsPartial counts them. Whether a narrower guard could ignore partial lexical entries is open, tracked in sillsdev/machine#551. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-perf-2026-09-disposition.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/hermitcrab-perf-2026-09-disposition.md b/docs/hermitcrab-perf-2026-09-disposition.md index 518ab2632..b1f922cab 100644 --- a/docs/hermitcrab-perf-2026-09-disposition.md +++ b/docs/hermitcrab-perf-2026-09-disposition.md @@ -131,7 +131,11 @@ word list, and the rulestats HTML report so they cannot be staged by accident. the affix-process rules set `FinalTemplateState` only while the prune is enabled. The guard, `Morpher.IsPartial`, is still grammar-wide and still counts partial lexical entries. 2. Classify the unclassified affixes in the FLEx projects (Mbugwe 1, Amharic 1, Sena 6); that turns #491's default - prune on. Add the finding to the grammar advisor. + prune on for Mbugwe and Amharic. Sena stays partial because of its 25 partial lexical entries, so classifying + its affixes does not enable the prune there. Whether a narrower guard could safely ignore partial lexical + entries is an open question, tracked in sillsdev/machine#551: a partial root makes the word partial, and + synthesis then permits a non-partial rule after a final template, so ignoring lexical entries may not be safe. + Add the finding to the grammar advisor. 3. ~~Open the prefilter PR from `perf/hc-edge-prefilter`~~ — that branch and its own archive tag are both deleted; a PR now starts from the mechanism preserved on branch `perf/hc-optimization-archive` rather than resuming the deleted branch. From 08d32446a9222249392f2d847694c141b00be3e1 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 23 Oct 2026 09:38:33 -0400 Subject: [PATCH 24/32] Drop the ReplayOnto hardening constraint from the optimization ledger The analysis memo that MemoEntry belonged to was removed in #522 and will not return, so its arrival-count constraint no longer applies. The remaining order-invariance constraint becomes the section's only item. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-optimization-ledger.md | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-) diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index e89bd8ea5..c7a2f322a 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -216,18 +216,11 @@ gate, so the main build target is the cheaper clone.** Row 17b counts only `AnalysisAffixTemplateRule`'s eager per-template clone, the site the retained deferred template clone changes; how that change's 1.054x relates to the 66.2% was not measured. -## Two durable code constraints - -Neither is expressible in code, and both bite silently. - -1. **`Word.ReplayOnto` does not splice `_mrulesUnapplied`.** It splices `_mruleApps` and - `_nonHeadApps` only. This is safe *solely* because `AnalysisStateKey` includes the per-rule - count multiset, which guarantees a memo hit has identical arrival and stored-arrival counts. - **Anyone narrowing that key breaks correctness with no test failure.** Fix, if the key ever - changes: store arrival counts on `MemoEntry` and compute `stored - storedArrival + query` — a - no-op under today's key, which is also how to test it. This is the one code change from this - work worth making on its own merits, as defensive hardening. -2. **HermitCrab's morphological rules are not order-invariant.** Same pending-rule multiset, +## One durable code constraint + +It is not expressible in code, and it bites silently. + +**HermitCrab's morphological rules are not order-invariant.** Same pending-rule multiset, different application order, different synthesis output — 2 cases on Sena, 12 independently on Indonesian. This bounds every packed-readout scheme. It is the assumption Maxwell and Kaplan (CL 19(4):571–590) rely on and this engine does not satisfy. From 0fa30529b8e31585dac66b1655248ef7418bf59e Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 23 Oct 2026 09:38:39 -0400 Subject: [PATCH 25/32] Remove the stale TemplateMemo composition claim from the search-cost round AnalysisScope.TemplateMemo no longer wraps the battery on master, so the statement that #546 composes with it is removed. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-2026-10-21-search-cost-round.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/hermitcrab-2026-10-21-search-cost-round.md b/docs/hermitcrab-2026-10-21-search-cost-round.md index 92ddde303..9c235a161 100644 --- a/docs/hermitcrab-2026-10-21-search-cost-round.md +++ b/docs/hermitcrab-2026-10-21-search-cost-round.md @@ -234,9 +234,8 @@ Run with `--threads 1 --step-cap 2000000` (Aweti 500000) and no wall timeout. `AnalysisAffixTemplateRule` has the same shape, so the trie ported directly to sequential `ApplySlots`. It is restricted to untraced, sequential walks with no alternative limit. Tracing, `ParallelApplySlots`, finite alternative limits, custom compiled templates, changed slot -optionality, and grammars without shared tails all keep the original battery. -`AnalysisScope.TemplateMemo` (row 2) still wraps the battery unchanged: row 2 reduces how often a -battery runs, and #546 reduces repeats inside one run, so the two compose. +optionality, and grammars without shared tails all keep the original battery. Row 2's battery memo is no +longer on master, so there is no memo for this change to compose with. The figures below are C# slot batches on deterministic 200-row samples, single-threaded and warmed, with mode order alternated per input: From 6ef517ed6c3bae29871f0c12771584c0617a01a4 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Fri, 23 Oct 2026 09:39:27 -0400 Subject: [PATCH 26/32] Drop the memo-revival facts from the probe design doc Section 4 kept two engine facts for any revival of the analysis memo that #522 removed. The memo is not coming back, so both facts now describe code that does not exist. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-probe-design.md | 19 ------------------- 1 file changed, 19 deletions(-) diff --git a/docs/hermitcrab-probe-design.md b/docs/hermitcrab-probe-design.md index 34baabdee..9d4129084 100644 --- a/docs/hermitcrab-probe-design.md +++ b/docs/hermitcrab-probe-design.md @@ -229,22 +229,3 @@ whole process if its budget expires. Do not restore an in-process `Task` timeout running, contaminates later samples, and can race shared engine state. `scripts/hc-bench-compare.ps1`, also on that branch only, compares grammar and word keys symmetrically before comparing errors, signatures, timing, or spread. - ---- - -## 4. Two durable engine facts, restated here so they survive independently - -Both facts concern the analysis memo that #522 removed from master. They apply to the archived branches and to -any revival of the memo. - -**`Word.ReplayOnto` does not splice `_mrulesUnapplied`.** It splices `_mruleApps` and `_nonHeadApps` -only. This is safe *solely* because `AnalysisStateKey` includes the count multiset, which guarantees -a memo hit has identical arrival and stored-arrival counts. **Narrowing that key breaks correctness -with no test failure.** Fix, if the key ever changes: store arrival counts on `MemoEntry` and compute -`stored - storedArrival + query` — a no-op under today's key, which is also how to test it. - -**`AnalysisScope.MaxMemoEntries` is a per-scope cap with no eviction.** Sharing one scope across a -corpus therefore makes words compete for a fixed budget: two heavy Sena words exhausted it at index -74 and 76, and every later word lost the within-word memoization a fresh scope would have given it. -Cross-word hits went *negative* and the corpus ran 28–32% slower. A shared bounded cache without -eviction is not a smaller version of the optimization. From af0508ecfbf2e03d8904d01445c9d37a4b59937b Mon Sep 17 00:00:00 2001 From: John Lambert Date: Mon, 26 Oct 2026 07:49:48 -0400 Subject: [PATCH 27/32] Update ledger next step for partial lexical entries PR #553 shows ignoring partial lexical entries in the final-template prune guard is safe, so Sena gets the prune once it merges. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-perf-2026-09-disposition.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/hermitcrab-perf-2026-09-disposition.md b/docs/hermitcrab-perf-2026-09-disposition.md index b1f922cab..c9a805d81 100644 --- a/docs/hermitcrab-perf-2026-09-disposition.md +++ b/docs/hermitcrab-perf-2026-09-disposition.md @@ -132,9 +132,9 @@ word list, and the rulestats HTML report so they cannot be staged by accident. `Morpher.IsPartial`, is still grammar-wide and still counts partial lexical entries. 2. Classify the unclassified affixes in the FLEx projects (Mbugwe 1, Amharic 1, Sena 6); that turns #491's default prune on for Mbugwe and Amharic. Sena stays partial because of its 25 partial lexical entries, so classifying - its affixes does not enable the prune there. Whether a narrower guard could safely ignore partial lexical - entries is an open question, tracked in sillsdev/machine#551: a partial root makes the word partial, and - synthesis then permits a non-partial rule after a final template, so ignoring lexical entries may not be safe. + its affixes alone does not enable the prune there. PR sillsdev/machine#553 shows that ignoring partial lexical + entries is safe and makes the guard ignore them, so once #553 merges, Sena gets the prune without reclassifying + its entries (tracked in sillsdev/machine#551). Add the finding to the grammar advisor. 3. ~~Open the prefilter PR from `perf/hc-edge-prefilter`~~ — that branch and its own archive tag are both deleted; a PR now starts from the mechanism preserved on branch From a8d2cf09eb3ff33b0eb9a3c0158cd9689e3ed37c Mon Sep 17 00:00:00 2001 From: John Lambert Date: Tue, 27 Oct 2026 08:58:00 -0400 Subject: [PATCH 28/32] Record the synthesis-to-analysis pruning round Answer John Maxwell's question about which synthesis tests can move into analysis. Add a failure-reason census from research/failure-reason-stats @ af97348b with its caveats, a direct answer for each of his four candidates, the literature that bears on it, and six ranked experiments with soundness arguments, hook points, stress fixtures and kill criteria. Point ledger rows 31-37 at the new record. Co-Authored-By: Claude Opus 5.5 --- ...-2026-10-27-synthesis-to-analysis-round.md | 474 ++++++++++++++++++ docs/hermitcrab-optimization-ledger.md | 19 + 2 files changed, 493 insertions(+) create mode 100644 docs/hermitcrab-2026-10-27-synthesis-to-analysis-round.md diff --git a/docs/hermitcrab-2026-10-27-synthesis-to-analysis-round.md b/docs/hermitcrab-2026-10-27-synthesis-to-analysis-round.md new file mode 100644 index 000000000..d89abad31 --- /dev/null +++ b/docs/hermitcrab-2026-10-27-synthesis-to-analysis-round.md @@ -0,0 +1,474 @@ +# HermitCrab synthesis-to-analysis pruning, 2026-10-27 round + +**Disposition: planned, not built.** This round answers John Maxwell's question about which +synthesis tests could move into analysis. It records a failure-reason census, gives a direct answer +for each of his four candidates, and ranks six experiments. No product code has changed. Every +expected saving below is either a ceiling taken from the census or a worst-case argument. None is a +measured result. + +This record follows the [optimization ledger](hermitcrab-optimization-ledger.md). Ledger rows 31-37 +point here. + +## The question + +HermitCrab is generate-and-test. Analysis unapplies rules to generate candidate analyses, and +synthesis re-applies them to test each candidate. A test that runs in analysis prunes a candidate +before it is expanded, looked up and synthesized. [PR #519](https://github.com/sillsdev/machine/pull/519) +did this for reduplication copy agreement. Maxwell asked whether other tests could move. His +candidates were: + +- excluded `MorphemeCoOccurrenceRules` (monotone: once a morpheme is present it stays); +- `DisjunctiveAllomorph`; +- `Environments`: check a required environment against the input word in linear time, finding the + left context and then the right context after it. An excluded environment would need its left and + right contexts to be present everywhere; +- `ExcludedMprFeatures`. + +His principle is that an excluded property can be checked early if nothing later can remove it, and +a required one usually cannot. He also asked whether there are statistics on failure reasons. There +were none: `FailureReason` does not appear in any document in PR #490. This round adds them. + +## Scope: the target is the space of grammars, not the five samples + +The owner set the scope during the round. The five sampled reference grammars (Sena, Amharic, +Aweti, Mbugwe, Indonesian) rarely use co-occurrence rules, environments, disjunctive allomorphs or +MPR features. A census of their XML finds 0-2 co-occurrence rules each, required MPR features on 0-9 +allomorphs, and no `Overwrite` MPR groups. The target is every grammar HermitCrab can represent that +some language family would plausibly use. Each candidate is therefore ranked by soundness and by its +benefit across that space. The measured samples are one data point, and they do not decide the +ranking. For every candidate this record names: + +- (a) the language types that lean on the feature; +- (b) the worst-case blow-up when the check runs late compared with early, in steps; +- (c) a stress fixture that models such a language at realistic density. + +Every experiment's measurement must include its stress fixture as well as the conformance suite and +the five real grammars. + +## Failure-reason census + +**Reproduce.** Branch `research/failure-reason-stats` @ `af97348b` is master `c2454760` with PR #480 +merged. Its harness is test-only: +`tests/SIL.Machine.Morphology.HermitCrab.Tests/Research/{FailureReasonStatsHarness.cs, +FailureStatsTraceManager.cs, run_failure_stats.sh, make_failure_stats_report.py}`. Run +`bash tests/SIL.Machine.Morphology.HermitCrab.Tests/Research/run_failure_stats.sh OUT_DIR` from the +repository root. It writes one TSV per corpus and a `tables.md`. No product code changed. + +**Definitions.** +- A *step* is one rule-level synthesis trace event: a rule applied, or one subrule attempt that did + not apply. Steps are counted, never timed. +- A *terminal outcome* is a synthesis branch that ends in `Failed(reason)`, `Successful` or `Blocked`. +- *Steps wasted per reason* attaches each step to its word through `Word.CurrentTrace` lineage, then + splits it equally among that word's terminal descendants. +- *Rule-attempt reasons* (`RequiredMprFeatures`, `Pattern`, `AllomorphNotApplicable`, ...) mean one + subrule did not fire. The word goes on, so these are never terminal by themselves. + +**Corpora.** The real grammars are samples, not whole corpora: Indonesian all 70 words, Aweti every +2nd (103), Amharic the first 28, Mbugwe every 40th (42), Sena every 100th (62). The conformance suite +has 150 fixture runs and 1,492 words. + +### Headline numbers + +Real grammars, pooled (27,912 failed and 422 successful terminal outcomes; 279,872 synthesis steps): + +| Terminal reason | % of failed branches | % of synthesis steps | +|---|---:|---:| +| PartialParse / ApplicableTemplatesNotApplied | 71.8% | 43.5% | +| SurfaceFormMismatch | 4.9% | 25.6% | +| (no terminal event) | - | 22.1% | +| (success) | - | 3.8% | +| PartialParse / MorphologicalRulesPending | 14.8% | 2.1% | +| PartialParse / NonFinalTemplateAppliedLast | 5.2% | 1.6% | +| PartialParse / StratumRulesRemaining | 3.3% | 1.2% | +| RequiredStemName, Environments/Require, BoundRoot, DisjunctiveAllomorph | 0.1% together | 0.1% together | + +Rule-attempt events, real grammars: `RequiredMprFeatures` 41.4%, `Applied` 21.7% (morphological) +plus 1.7% (phonological), phonological `Pattern` 17.0%, `AllomorphNotApplicable` 11.6%, morphological +`Pattern` 5.2%, and `ExcludedMprFeatures` 2 events (0.0%). + +Per grammar, the step shares differ widely. Unpooled: + +| Grammar | Largest wasted-step buckets | +|---|---| +| Mbugwe | ApplicableTemplatesNotApplied 46.3%, SurfaceFormMismatch 27.2%, no terminal 22.8% | +| Aweti | MorphologicalRulesPending 30.4%, NonFinalTemplateAppliedLast 28.5%, success 16.0% | +| Amharic | MorphologicalRulesPending 60.0%, SurfaceFormMismatch 16.9% | +| Indonesian | success 51.9%, SurfaceFormMismatch 37.9% | +| Sena | no terminal 56.3%, MorphologicalRulesPending 13.3%, ApplicableTemplatesNotApplied 6.8% | + +The conformance suite is dominated by one pathological fixture. `deep-optional-affix-nesting` +contributes 5,556 of 6,789 failed branches. Pooled, StratumRulesRemaining has 40.7% of the suite's +steps, success 44.1% and SurfaceFormMismatch 6.9%. Maxwell's four reasons together have 0.9% of +conformance steps. + +### Caveats on the method + +These are read before any number above is used as a ceiling. + +1. **Tracing changes the search.** `Failed(...)` is raised only while tracing. Tracing turns off + equivalent-analysis merging (`AnalysisStratumRule.cs:142`), so traced parses explore more + analyses than production parses. The counts describe a traced parse. +2. **A step is a count, not a cost.** A `RequiredMprFeatures` event is one set lookup + (`SynthesisAffixProcessRule.cs:144-160`). An `Applied` or `AllomorphNotApplicable` event is a full + allomorph pattern match plus an output shape (`:179`). Both count as one step. Ledger row 7 is the + precedent: 100% of a rejection histogram cost 0.2% of wall time. +3. **Analysis work is outside the step total.** On Mbugwe the sampled words had 4,598,689 + analysis-side morphological rule events (unapplied plus not unapplied) against 260,070 synthesis steps. A + synthesis-only saving of N% is not N% of the work to parse the words. Every brief reports + analysis events and synthesis steps separately. +4. **Most `RequiredMprFeatures` events are allomorph selection, not failure.** Mbugwe has + `requiredMPRFeatures` on only 9 allomorphs, yet it logged 113,419 such events against 56,405 + morphological applications. The allomorph loop logs one event for each gated allomorph it skips + on the way to the one that fires. These events show where the loop spends time. They are not why + candidates die. +5. **Branch outcomes are not candidate outcomes.** `ApplicableTemplatesNotApplied` fires on one path + through a stratum. `SynthesisStratumRule.Apply` (`:58`, `:104-140`) runs the templates both on the + input and after each morphological-rule output, so a candidate can lose one path and succeed on + another. A prune that drops whole candidates needs a candidate-level census: the steps spent in + candidates with zero successes. That census was not run. The 2026-10-21 round measured the + HC-Rust analogue at 86-98% of synthesis work (ledger row 27, "root-reachable synthesis failure"). +6. **The 22.1% "no terminal event" share is unattributed.** It is steps on branches that ended with + no terminal callback. Template slot paths that die inside `SynthesisAffixTemplateRule.ApplySlots` + while a sibling path survives are one source. Silently dropped words are another: the realizational + gate at `SynthesisAffixTemplatesRule.cs:27-28` and `HashSet` duplicates. This split is an + inference and was not measured. Treat the share as mostly template-path failure. +7. **Samples and budgets.** A 30 s per-word budget dropped 2 Mbugwe words, so the heaviest words are + under-represented. `run_failure_stats.sh` runs the five grammars concurrently, so which words hit + the budget can vary between runs. A/B runs must be sequential and must drop a word that aborted + in either arm. Amharic stops at word 28 (word 29 hangs). + +## Direct answers to Maxwell's four candidates + +All four are sound to move in at least a restricted form, and the census predicts little benefit +from any of them on the samples. On the samples the work goes to template-path failures and surface +mismatches. + +| Candidate | Sound early? | Where | Terminal share on the samples | Who needs it | +|---|---|---|---|---| +| Excluded `MorphemeCoOccurrenceRules` / `AllomorphCoOccurrenceRules` | Yes for `Anywhere`, with a zero-width caveat; not for `Somewhere*` or `Adjacent*` without linear order | Analysis, per rule (H1) | Real 0; conformance 6 branches (0.1% of steps) | Polysynthetic and Bantu morphotactics; FLEx "ad hoc co-prohibitions" written to kill spurious parses | +| `DisjunctiveAllomorph` | Mostly already early; the rest needs the surface | Synthesis loop | Real 13 (0.0%); conformance 13 (0.2%) | Romance and Semitic suppletion, IE stem allomorphy | +| Required `Environments` (input-word prefilter) | Yes, under two conditions (below) | Analysis, per allomorph (H1) | Real 8 (0.0%); conformance 15 (0.4%) | Turkic and Uralic harmony, Bantu, Austronesian nasal substitution | +| `ExcludedMprFeatures` | Yes, by trail replay after lexical lookup | After lookup (H4) | Real 0 terminal, 2 rule events; conformance 11 rule events | IE declension and conjugation classes, Bantu noun class | + +Hook points (origin/master `c2454760`): **H1**, per analysis rule, after `MorphologicalRuleUnapplied` +at `MorphologicalRules/AnalysisAffixProcessRule.cs:92`. **H3**, end of analysis, +`Morpher.cs:179-193`. **H4**, after lexical lookup and `ExpandAlternatives`, once per alternative, just +before `_synthesisRule.Apply` at `Morpher.cs:353-358` (sequential) and `:386-395` (parallel). The filter +must see each alternative, not the canonical word. With merging on, which is the production setting, +`ExpandAlternatives` (`Word.cs:440-482`) rebuilds merged analyses with their own rules and +allomorphs, so dropping the canonical would drop analyses with different trails. Traced runs hide +this trap, because tracing turns merging off. + +**One principle covers all four.** A successful candidate applies every entry of its trail +(`Word._mruleApps`) in a fixed order, with the allomorph analysis recorded for it +(`Word.IsAllomorphApplicable`, `Word.cs:254-261`, added by PR #548). Any quantity that depends only +on the root, the trail and the recorded allomorphs therefore has one value on every successful path. +A predicate that is false for that value on every successful path can drop the candidate without +losing a parse. Maxwell's "cannot be removed later" is the special case where analysis can compute +that value before the root is known. + +### Excluded co-occurrence + +`MorphCoOccurrenceRule.IsWordValid` (`MorphCoOccurrenceRule.cs:82-87`) rejects an `Exclude` rule +when every listed other co-occurs with the key (`:92-170`). Under `Anywhere` that is set membership, +and adding morphs never undoes it, so it is anti-monotone in the Apriori sense. There is one trap the +monotonicity argument misses. The "others" are read from `Word.AllomorphsInMorphOrder` (`Word.cs:127`), +which is built from morph annotations. `MarkMorph` (`Word.cs:274-296`) creates no annotation for a +zero-width allomorph. A null affix on the trail is therefore invisible to the rule in synthesis. An +early check may count an "other" only when its recorded allomorph always leaves a morph annotation. +`SomewhereToLeft/Right` need surface linear order, which the trail gives only for simple prefixes +and suffixes, so they stay late. `Require` and `Adjacent*` are not monotone and stay late. The key is +always checked, because `Morpher.IsWordValid` iterates `word.Allomorphs` (`Morpher.cs:661`), which +includes zero-width allomorphs. + +Benefit: zero on the samples. In a grammar that uses co-prohibitions it is the only candidate here +that prunes analysis work as well as synthesis. Experiment E4. + +### DisjunctiveAllomorph + +Most of this check already runs early. An environment-free earlier allomorph with no required +syntactic FS blocks later ones at rule-application time, through the `break` at +`SynthesisAffixProcessRule.cs:234-241`. The test left in `Allomorph.IsWordValid` (`Allomorph.cs:127-151`) +covers earlier allomorphs whose environment holds on the final surface. That needs the morph's +surface position, which analysis does not have. Where the disjunction is conditioned by MPR features +(Romance conjugation classes, for example), the MPR replay in E2 covers it. No separate experiment. +E1 shortens the same loop. + +### Environments + +Environments are checked on the **final** synthesized word, after every stratum's phonology +(`Morpher.cs:358` runs `IsWordValid` on `_synthesisRule.Apply(...)`, and `Allomorph.cs:110-125` checks +the morph's neighbours there). For any success, `IsMatch` (`CharacterDefinitionTable.cs:274-282`) +has accepted that shape against the input word. A1's worry, that later rules could change the +context, therefore does not apply here, because no rule runs after the check. Maxwell's prefilter is +sound only under two conditions that the code imposes: + +1. **Boundaries.** The environment matchers accept `Boundary` and `Anchor` annotations + (`AllomorphEnvironment.cs:38-41`, `:55-58`), which the input shape lacks. A boundary node in a + context must match the empty string in the prefilter. +2. **Underspecified surface segments.** `IsMatch` compiles the synthesized shape to a regex, so an + underspecified node (an archiphoneme such as N that no rule fills in) can match an input character + whose features conflict with the context. For example, a [+cons] node unifies with a [+voice] + context and also matches input `t`. Run against the input, the prefilter would then reject a valid + parse. The prefilter is sound only when every surface node is fully specified. Featureless + grammars meet that by `StrRep` identity. Grammars without underspecified segments in lexical + entries, inserted affix material or rule outputs also meet it. Otherwise the prefilter must stay + off. + +Excluded environments cannot be prefiltered this way, as Maxwell said. Benefit: on the samples, 23 +terminal branches across all corpora. The real case is a zero allomorph with an environment: analysis +can unapply it anywhere in any word, and today only `IsWordValid` disposes of it. Experiment E5. + +### ExcludedMprFeatures + +On a successful path, MPR features evolve deterministically along the trail. They start from the +root's set (`Word.cs:152-153`), and each applied allomorph calls `AddOutput` (`MprFeatureSet.cs:29-44`; +`SynthesisAffixProcessAllomorphRuleSpec.cs:209`). This is exact except where something outside the +trail touches the set: realizational rules (not on the trail, `Word.cs:340`), compounding +(`SynthesisCompoundingRule.cs:178-179`), `ChooseInflectionalStem` swapping in a relative's entry +(`SynthesisAffixTemplatesRule.cs:83-115`), and `Overwrite` groups (which remove features, so a +feature can be lost). Excluded MPR features can therefore be checked at H4, but not before lookup, +because the root contributes MPR features. Required MPR features can be checked at the same point, +which matters more: required MPR features are how FLEx inflection classes gate affix allomorphs. +Benefit on the samples: no terminal failure is attributed to either reason. Required failures may be +hidden inside the PartialParse buckets, and E2's shadow census measures that. Experiment E2. + +## Literature that bears on this + +The full survey is in this round's working notes, and only the parts that change a decision are kept +here. Unless marked, the sources were checked at the abstract level only. + +- **Anti-monotone pruning** (Agrawal & Srikant 1994, Apriori). A set that contains a forbidden subset + is forbidden. This justifies E4 and the excluded half of E2. It does not apply to required + properties, which are monotone in the other direction. +- **Nogoods and forward checking in CSP search** (Dechter; Schiex & Verfaillie 1993, cited + second-hand). A failed partial assignment is a nogood. Ledger row 14 already found 0 subsumable + nogood pairs, so nogood *learning* is closed. Forward checking ("can this still be satisfied?") is + the model for E2 and E3. Both ask whether a trail can still succeed once the root is known. +- **Constraint automata intersected with the input** (Koskenniemi 1990; Bird & Ellison 1994; Kaplan & + Kay 1994). This is the principled form of Maxwell's environment check, and E5 is its cheapest + instance: a greedy left-then-right scan, linear in word length for each context. Baeza-Yates (1991) + subsequence automata pay off only when one word is tested against many contexts. +- **Maxwell (1994), "Parsing Using Linearly Ordered Phonological Rules"** (arXiv cmp-lg/9411015, full + text not read). It is HermitCrab's own ancestor: generate-and-test with ambiguity encoded into the + form being parsed. Reading where it rejects candidates is open. +- **Billot & Lang (1989)** warn that look-ahead added to chart parsing can cost more than it saves. + This is why every experiment here has a kill criterion in steps, including the cost of the check. + +No published source was found on tracking morph spans through rule inversion. That is what a +position-aware environment check would need. + +## Ranked experiments + +Ranked by soundness plus benefit across the grammar space, with cost as the tiebreak. "Ceiling" is +the census share that the change could at most remove. It is not a prediction. Every experiment is +measured in steps: synthesis steps, analysis events and candidates reaching synthesis, each reported +as "N% less work" per corpus, unpooled. Every experiment's parses must be identical, per word, on the +conformance suite, the five real grammars and its stress fixture. + +### E1. Stop the synthesis allomorph loop after the recorded allomorph + +- **Hypothesis.** `SynthesisAffixProcessRule.Apply` keeps applying allomorphs after the one analysis + recorded. Each of those applications is a full pattern match and output shape, and its result is + thrown away at `:199` (`AllomorphNotApplicable`). Breaking the loop once `i` passes the recorded + index removes them, along with their MPR checks. +- **Soundness.** An output from allomorph `i > r` fails `IsAllomorphApplicable` and is never added. + `appliedAllomorphIndices` (`:138`, `:196-197`) is copied into the outputs at the moment each is + built (`Word.MorphologicalRuleApplied`, `Word.cs:374-375`, `UnionWith`). Indices added after `r` + therefore never reach `r`'s output. When the recorded allomorph is null, nothing changes. Earlier + allomorphs (`i < r`) are untouched, because their application decides the disjunctive `break` and + the recorded indices. +- **Hook.** `MorphologicalRules/SynthesisAffixProcessRule.cs:140-247`. It needs a `Word` accessor for + the recorded allomorph at the current trail position (`Word.cs:254-261`). +- **Language types, and blow-up.** Any affix with an ordered allomorph list. Examples are Turkic + -DI/-lAr sets written as 4-8 environment-conditioned allomorphs (environment-bearing allomorphs + never `break`, so today every one is applied), Bantu class prefixes, and IE case endings by + declension class. Late: each rule application costs `A` allomorph applications. Early: `r + 1`. + The waste is linear, up to a factor of `A`, for every synthesis of every candidate, including the + successful ones. +- **Stress fixture.** New synthetic fixture `stress-allomorph-list-density`. It has a harmony-style + suffix grammar with four suffix slots, each with 8 environment-conditioned allomorphs, and with the + recorded allomorph spread across positions in the lists. `languages/suffixing-vowel-harmony` and + `edge-cases/disjunctive-recheck` are the soundness controls. +- **Expected savings.** The ceiling is the `AllomorphNotApplicable` events with `i > r` plus the + `i > r` part of `RequiredMprFeatures`. `AllomorphNotApplicable` is 11.6% of real-grammar synthesis + steps (Mbugwe 12.3%, Sena 11.4%, Aweti 2.6%) and 1.0% of conformance steps. The census did not split + by `i > r`. On the stress fixture the ceiling approaches (A-1)/A of allomorph applications. +- **Cost.** Very low: about 10 lines plus a `Word` accessor. +- **Kill.** Less than 2% fewer synthesis steps on every real grammar *and* less than 20% on the + stress fixture, or any parse difference. + +### E2. Replay MPR features and stem names along the trail at lexical lookup + +- **Hypothesis.** Once the root is known (H4), the MPR set that each trail entry will see on a + successful path can be computed without shapes. A candidate whose recorded allomorph fails its + required MPR features (or matches its excluded ones), or whose rule's `RequiredStemName` differs + from the root's, can be dropped before synthesis. +- **Soundness.** Replay in synthesis order (trail from last to first). Start from the root's + `MprFeatures` and add each recorded allomorph's `OutMprFeatures`. On a successful path this gives + the set the synthesis check sees (`SynthesisAffixProcessRule.cs:144-177`), except for contributions + the replay can only bound. The required test runs against an upper bound: that set plus every + realizational and compounding output, with `Overwrite` removals ignored. The excluded test runs + against a lower bound, and only when the grammar has no `Overwrite` group and the trail has no + compounding entry. If the root's entry has a `Family`, the prune stays off, because + `ChooseInflectionalStem` replaces the root. With a null recorded allomorph, the candidate is dropped only if every allomorph + of the rule fails. Stem names: `SynthesisAffixProcessRule.cs:107` compares only with + `input.RootAllomorph.StemName`, which is known exactly at H4. +- **Hook.** H4, per alternative (`Morpher.cs:353-358` and `:386-395`), after the `MaxAlternatives` + count so that the exception behaviour does not change. Both paths call it. +- **Language types, and blow-up.** IE declension and conjugation classes (Latin and Russian + declensions; Romance -ar/-er/-ir conjugations), Bantu noun class written as MPR features, and + Australian conjugation classes. With `c` inflection classes and endings that are homophonous + across classes, each ending's analysis meets roots of every class. Late, each wrong-class root + costs a synthesis up to the gated rule, plus every template path. Early, it costs a set replay of + length |trail|. With agreement prefixes gated in `k` slots, late synthesis grows as `c^k` + candidates. The early check is still `c^k` replays, but each one is trivially cheap. +- **Stress fixture.** New synthetic `stress-mpr-inflection-classes`: five declension classes as MPR + features on 20 roots, case and number endings gated by required MPR features, deliberate homophony + across classes, one excluded-MPR exception and one `Overwrite` group (which must disable the + prune). Soundness controls: `edge-cases/mpr-overwrite-order-dependence`, + `edge-cases/mpr-group-overwrite-without-realizational`, `edge-cases/mpr-gated-exception`, and + `languages/fusional-realizational-morphology`. +- **Expected savings.** On the samples the floor is near 0: no terminal failure has an MPR reason. + The ceiling is the MPR-caused part of PartialParse (48.4% of real-grammar steps across all its + causes). That part is unknown, and phase 1 (a shadow census) measures it. Mbugwe and Aweti are the + only samples with gated allomorphs (9 and 6 required, 41 and 26 excluded). On the stress fixture, + expect most of the wrong-class syntheses, roughly (c-1)/c of candidates. +- **Cost.** Low to medium: one replay function and a `Morpher` switch defaulting on, following PR + #519's `PruneDisagreeingCopies`. +- **Kill.** The shadow census decides less than 3% of synthesis steps on every real grammar *and* + less than 30% on the stress fixture. A shadow run in which any successful candidate fails the + predicate is a soundness bug: stop and fix it, do not tune it. + +### E3. Replay the template and final-state outcome at lexical lookup (after a dead-candidate census) + +- **Hypothesis.** Most synthesis steps on the template-heavy samples go to candidates that die on + shape-free conditions. The conditions are an applicable template whose required slot no pending + rule fills (`SynthesisAffixTemplatesRule.cs:57-63`), a non-final template applied last + (`SynthesisStratumRule.cs:70-74`), rules left in a stratum (`:75-87`), and trail rules never + consumed (`Morpher.cs:639`). An abstract replay at H4 can follow the trail pointer, template + finality, `IsPartial` and the syntactic FS while assuming every pattern match succeeds. That + replay over-approximates success, so it can prove a candidate dead. +- **Soundness.** Abstract interpretation of `SynthesisStratumRule.Apply` and + `SynthesisAffixTemplatesRule.Apply`. Assuming every allomorph pattern matches only adds successful + paths, and so does treating each realizational rule as both applied and not applied. If the + abstract replay finds no path, the concrete synthesis finds none either. Template applicability + (`:37`) needs the syntactic FS at that point. That FS is exact on successful paths: the root entry's + FS (`Word.cs:151`) is unified with each `Required` and then `PriorityUnion`-ed with each `Out` + (`SynthesisAffixProcessRule.cs:123`, `:182-183`). It is not exact when realizational FS output or + `ChooseInflectionalStem` intervene, and the replay must bail out there. +- **Phase 0 (must come first).** A candidate-level census: synthesis steps in candidates (root + allomorph lookups) with zero successes, split by the first shape-free reason the replay would + give. This settles caveats 5 and 6 above. +- **Hook.** H4, as in E2. E3 extends E2's replay. +- **Language types, and blow-up.** Every grammar with affix templates and required slots. FLEx + templates are the normal way to write Bantu verbs, Athabaskan and Algonquian position classes, + Mayan and Quechuan inflection. Late, each dead candidate pays lookup, `ExpandAlternatives` and + synthesis of every stratum below the failing one, plus each template path. Early, it pays a replay + of length |trail| times the number of templates per stratum. On Mbugwe, 19,734 candidates reached + synthesis and 149 branches succeeded. +- **Stress fixture.** New synthetic `stress-required-template-slot`. It models a Bantu-like verb: + one stratum, a template with a required TAM slot and optional subject and object slots, noun and + verb roots with overlapping shapes, and words whose bare-stem and partial analyses find roots. + `languages/prefixal-discontinuous-slot-dependency` and `edge-cases/final-template-partial-*` are + the soundness controls. `edge-cases/deep-optional-affix-nesting` is the pathological control. +- **Expected savings.** The branch-attributed ceiling is PartialParse at 48.4% of real-grammar + synthesis steps plus part of the 22.1% unattributed share. Mbugwe has 46.3% plus 22.8%, Aweti 58.9% + (MorphologicalRulesPending plus NonFinalTemplateAppliedLast) and Amharic 60.0%. Ledger row 19 is + the counterweight: shape-free die points were 0.0-6.0% of cost on heavy Sena probes and 0.4% on 28 + Amharic words. Sena and Amharic are therefore expected low and Mbugwe and Aweti decide. +- **Cost.** Medium to high. It mirrors two synthesis classes' control flow for Linear and Unordered + strata. It is the riskiest experiment for soundness, and the one with the largest measured target. +- **Kill.** Phase 0 shows that shape-free reasons decide less than 10% of dead-candidate synthesis + steps on both Mbugwe and Aweti *and* less than 30% on the stress fixture. Or any successful + candidate fails the abstract replay. + +### E4. Excluded co-occurrence during analysis (Maxwell) + +- **Hypothesis.** An `Exclude` + `Anywhere` morpheme (or allomorph) co-occurrence rule whose key and + others are all on an analysis word's trail rejects that word and everything analysis derives from + it. Checking it at H1 prunes the analysis subtree as well as the lookups and syntheses. +- **Soundness.** As argued above (anti-monotone under `Anywhere`), with three restrictions. An + "other" counts only if its recorded allomorph always leaves a morph annotation. A rule whose key or + others include a root morpheme is checked at H4, not H1. Allomorph rules count only entries whose + allomorph is recorded (not null). `Somewhere*`, `Adjacent*` and `Require` stay late. +- **Hook.** `MorphologicalRules/AnalysisAffixProcessRule.cs:92` (after `MorphologicalRuleUnapplied`, + before `output.Add` at `:96`). Root-involving rules use H4. +- **Language types, and blow-up.** Polysynthetic prefix incompatibilities (Athabaskan, Iroquoian), + Bantu negation against tense, Quechua and Aymara suffix incompatibilities, and any FLEx project + that uses ad hoc co-prohibitions to suppress spurious parses. Take `k` optional slots with `m` + fillers each. Late, every analysis that holds a forbidden pair is expanded below that point, looked + up and synthesized. Early, the subtree is cut where the pair completes. The worst case is + `m^(k-d)` analysis nodes saved per cut at depth `d`, so it is exponential in the slots that remain. +- **Stress fixture.** New synthetic `stress-excluded-cooccurrence`: six optional prefix slots with + four fillers each, about ten `Exclude`/`Anywhere` rules, and a zero-width affix as an "other" in one + rule. That word must still parse. `languages/suffixing-evidential-adjacency-chain`, + `languages/templatic-root-modification` and `edge-cases/morphotactic-attribute-breadth` are the + soundness controls (they hold 3, 1 and 8 exclude rules). +- **Expected savings.** Samples: about 0. Conformance: 2 `MorphemeCoOccurrenceRules/Exclude` and 4 + `AllomorphCoOccurrenceRules/Exclude` branches, 14 steps (0.2%). The stress fixture should show a + reduction in analysis events, which no other experiment here can produce. +- **Cost.** Low. Rule lists can be indexed per morpheme at `Morpher` construction. +- **Kill.** Less than 25% fewer analysis events plus synthesis steps on the stress fixture, or any + parse difference. A zero effect on the samples is expected and is not a kill. + +### E5. Required-environment prefilter on the input word (Maxwell) + +- **Hypothesis.** For each allomorph whose environments are all `Require`, compute once per input + word whether any of them can match anywhere: the left context ending at some position `p`, and the + right context starting at some `q >= p`. Allomorphs that cannot match are skipped during analysis. +- **Soundness.** Environments are checked on the final shape, and `IsMatch` ties that shape to the + input, as argued above. Three restrictions apply. Boundary nodes match empty. The prefilter is + enabled only when the grammar cannot produce underspecified surface segments (a static scan of + lexical entries, inserted affix material and rule outputs; featureless grammars qualify). An + allomorph with any `Exclude` environment is never skipped. Skipping analysis allomorph `i` is sound + only because PR #548 restricts synthesis to the recorded allomorph. +- **Hook.** Compute the set per word in `Morpher.ParseWord` (`Morpher.cs:167-179`). Consult it in the + allomorph loop of `AnalysisAffixProcessRule.Apply` (`:61-102`), skipping `_rules[i]`. +- **Language types, and blow-up.** Environment-conditioned allomorphy is everywhere: Turkic and + Uralic harmony, Bantu, Korean, Austronesian nasal substitution. It matters most for **zero + allomorphs with environments**, which analysis can unapply in any word. With `z` optional + zero-allomorph affixes, late checking multiplies the analysis tree by up to `2^z` before + `IsWordValid` disposes of the extras. The prefilter removes each one whose context is absent from + the whole word. It is weak when contexts are common, because it does not know positions. +- **Stress fixture.** New synthetic `stress-env-zero-allomorphs`: three affixes, each with an overt + allomorph and a zero allomorph conditioned by an environment (for example ∅ / _ [+nasal]), with + words that mostly lack the conditioning contexts. Controls: `edge-cases/discontinuous-morph-environment`, + `edge-cases/diacritic-segments` and `languages/suffixing-vowel-harmony`. Sena (72 required-environment + sets) is the real grammar most likely to move. +- **Expected savings.** Samples: `Environments/Require` has 98 real steps (0.0%) and 32 conformance + steps (0.4%). Pruning at H1 also removes analysis events, which the terminal counts do not show. + The stress fixture is the real test. +- **Cost.** Low to medium. The left matcher runs right to left from each position, with the existing + compiled patterns reused on the input shape. +- **Kill.** Less than 20% fewer analysis events plus synthesis steps on the stress fixture, or the + grammar condition disables it on four of the five real grammars and on most conformance languages. + Any parse difference is a soundness bug. + +### E6. Intermediate-shape check for SurfaceFormMismatch (research only, no brief) + +`SurfaceFormMismatch` is the largest terminal reason that no experiment above touches: 25.6% of +real-grammar steps, 37.9% on Indonesian and 27.2% on Mbugwe. It fails last, after a full synthesis. A +check at each stratum boundary that the synthesized shape still matches the analysis shape the +candidate came from could stop a candidate early. There is no soundness argument yet. Analysis +shapes are over-approximations with optional and underspecified nodes, and ledger row 8 (a +synthesis-side length bound) lost valid parses. Run a census only: at each boundary, record whether +the check would fire, and whether any successful candidate ever fails it. **Kill** if any success +fails it on any corpus. + +### Not an experiment + +- **Cheaper `RequiredMprFeatures` rejection** as such: each rejection is one set lookup (caveat 2), + and ledger row 7 is the precedent. E1 removes the ones past the recorded allomorph as a side effect. +- **Nogood learning over candidates**: ledger row 14 found 0 subsumable pairs. +- **Moving constraints into the analysis merge key**: ledger row 15. Every synthesis rejection depends + on the root, so H4, not the analysis key, is the place. + +## Open questions + +- A candidate-level dead-step census (E3 phase 0) is the most useful single measurement left. It + turns this round's branch-level ceilings into candidate-level ones. +- Whether synthesis phonological deletion removes morph annotations. E4's zero-width restriction may + need to cover deleted morphs too. The E4 brief requires a test. +- Whether `ChooseInflectionalStem` families occur in FLEx exports at all. None of the five samples has + one. E2 and E3 bail out on them. diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index c7a2f322a..22e2789cd 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -138,6 +138,25 @@ Rows 24 and 25 are **HC-Rust port bugs: C# was already exact, and no C# change f by running the port on the oracle's own grammar export and comparing lookup counts. Do that before optimizing the algorithm. +## 2026-10-27 synthesis-to-analysis round (planned) + +Full record: [hermitcrab-2026-10-27-synthesis-to-analysis-round.md](hermitcrab-2026-10-27-synthesis-to-analysis-round.md). +It answers John Maxwell's question about which synthesis tests can move into analysis. The unit is +C# synthesis steps (rule-level trace events) and analysis events, counted by the test-only harness +on `research/failure-reason-stats` @ `af97348b`, never ms. Rows 32-37 are **planned and unmeasured**. +Ranking covers every grammar type HermitCrab can represent, not only the five samples, and each +experiment carries a synthetic stress fixture. + +| # | Optimization | Expected | Outcome | Number | +| --- | --- | --- | --- | --- | +| 31 | **Failure-reason census** (measurement, not a change) | Maxwell's four candidates (excluded co-occurrence, disjunctive allomorph, environments, excluded MPR) would be where synthesis work dies | **MEASURED; the samples refute it.** On the five samples they are almost never terminal. The work dies on template paths and surface mismatches | Real grammars: ApplicableTemplatesNotApplied **43.5%** of synthesis steps, SurfaceFormMismatch **25.6%**, unattributed 22.1%, Maxwell's four **<0.1%**. `RequiredMprFeatures` is 41.4% of rule events but is mostly allomorph selection. Traced, so merging is off; analysis work is outside the step total | +| 32 | **E1: stop the synthesis allomorph loop after the recorded allomorph** | Allomorphs past the one analysis recorded are applied and then discarded | PLANNED | Ceiling: `AllomorphNotApplicable` with `i > r`, at most 11.6% of real synthesis steps | +| 33 | **E2: replay MPR features and stem names along the trail at lookup** (covers excluded MPR) | Inflection-class gates decided before synthesis | PLANNED; shadow census first | Samples near 0 terminal; target is IE and Bantu inflection classes | +| 34 | **E3: replay the template and final-state outcome at lookup** | Shape-free template-path deaths decided before synthesis | PLANNED; candidate-level dead-step census first | Branch ceiling 48.4% of real steps (Mbugwe 46.3% plus unattributed). Row 19's 0-6% on Sena and Amharic is the counterweight | +| 35 | **E4: excluded co-occurrence during analysis** (Maxwell) | Anti-monotone; prunes analysis subtrees | PLANNED | Samples about 0; conformance 0.2%. Zero-width "others" are invisible to the synthesis check and must not count | +| 36 | **E5: required-environment prefilter on the input word** (Maxwell) | Linear left-then-right scan per allomorph | PLANNED | Samples 0.0%. Sound only without underspecified surface segments, and with boundaries matching empty | +| 37 | **E6: intermediate-shape check against SurfaceFormMismatch** | Stop a candidate at the first stratum boundary where it diverges | RESEARCH ONLY; no soundness argument yet | SurfaceFormMismatch is 25.6% of real steps. Census only, killed by any success that fails it | + --- ## The lesson that generalises From 253b1bfea8f80de6a23f17291866391d6923ff13 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Tue, 27 Oct 2026 13:02:35 -0400 Subject: [PATCH 29/32] Record the results of the HermitCrab synthesis-to-analysis round State the outcome of experiments E1-E5 in steps: E1 and E2 kept, E3 kept pending memoisation, E4 parked, E5 a dead end. Add an answer to John Maxwell's questions, the share of total work each experiment saves, and the baseline problems found. Update ledger rows 31-37. Co-Authored-By: Claude Sonnet 5.5 --- ...-2026-10-27-synthesis-to-analysis-round.md | 259 +++++++++++++++++- docs/hermitcrab-optimization-ledger.md | 18 +- 2 files changed, 261 insertions(+), 16 deletions(-) diff --git a/docs/hermitcrab-2026-10-27-synthesis-to-analysis-round.md b/docs/hermitcrab-2026-10-27-synthesis-to-analysis-round.md index d89abad31..eabe3be01 100644 --- a/docs/hermitcrab-2026-10-27-synthesis-to-analysis-round.md +++ b/docs/hermitcrab-2026-10-27-synthesis-to-analysis-round.md @@ -1,14 +1,61 @@ # HermitCrab synthesis-to-analysis pruning, 2026-10-27 round -**Disposition: planned, not built.** This round answers John Maxwell's question about which -synthesis tests could move into analysis. It records a failure-reason census, gives a direct answer -for each of his four candidates, and ranks six experiments. No product code has changed. Every -expected saving below is either a ceiling taken from the census or a worst-case argument. None is a -measured result. +**Disposition: round finished; no product code is proposed for this PR.** This round answers John +Maxwell's question about which synthesis tests could move into analysis. It records a failure-reason +census, gives a direct answer for each of his four candidates, and ranks six experiments. Five were +built and measured on throwaway branches (none pushed); E6 stays research-only. The results are in +[Results of the round](#results-of-the-round). Measurements are in steps, never time. This record follows the [optimization ledger](hermitcrab-optimization-ledger.md). Ledger rows 31-37 point here. +## Answer to John Maxwell's questions + +**Failure-reason statistics.** They exist now (census below, harness `research/failure-reason-stats` +@ `af97348b`). On the five real grammars, pooled, 71.8% of failed synthesis branches and 43.5% of +synthesis steps end in `PartialParse / ApplicableTemplatesNotApplied`, and 25.6% of steps end in +`SurfaceFormMismatch`. Your four reasons together are 0.1% of failed branches and 0.1% of steps. +Caveats from the method section that bind every number: the counts come from traced parses (tracing +turns merging off, so more analyses are explored); a step is a count, and `RequiredMprFeatures` +events are mostly allomorph selection, not failure; branch outcomes are not candidate outcomes; 22.1% +of steps have no terminal event and are unattributed; the samples are strides of five grammars, with +heavy words dropped by the per-word budget; and the census counts synthesis steps only, so it does +not see analysis work. + +**Excluded co-occurrence (E4): sound, cheap, effective only where exclusions are dense.** The check +is sound: parses are identical on every corpus, and the zero-width "other" trap you might expect does +not occur for affix-process allomorphs. It is the only experiment of the round that removes analysis +events. On the five real grammars it removes 0% (none has an affix-only exclude rule), on the +conformance suite 0.23% of total work. On the brief's own stress fixture it gives 3.2%, and on two +deliberately dense synthetic grammars 31.3% (Athabaskan-style prefix complex) and 19.8% +(Bantu-style extension chain). Verdict PARK: land it, if at all, as a cheap safety net for grammars +with many ad hoc co-prohibitions. + +**DisjunctiveAllomorph: already mostly early; no experiment.** The environment-free `break` in the +synthesis allomorph loop already stops later allomorphs, and what remains needs the surface form. +Disjunctions conditioned by MPR features are covered by E2, and the loop itself is shortened by E1. +Real-grammar terminal share is 0.0%. + +**Environments (E5, the input-word prefilter): dead end as specified.** The soundness scan has to +disable the prefilter on 5 of 5 real grammars (four have ambiguous digraph segmentation, one has +non-uniform feature sets) and on 145 of 150 conformance runs. Forced on against that scan it is net +negative on two of three stress fixtures once matcher invocations are counted, and about neutral on +Aweti (net -0.4%). See E5 below for what would have to change. + +**ExcludedMprFeatures (covered by E2, the MPR and stem-name trail replay): sound; worth keeping, +but small in total work.** Excluded MPR features are replayed with the required ones at lexical +lookup. The prune is sound (a shadow census found no successful candidate that it rejects), and it +saves 56.9% of synthesis steps on Aweti and 11.4% on Mbugwe. Those are shares of synthesis, which is +a small part of the whole parse (next paragraph). + +**What the data says matters instead.** Synthesis steps are a small fraction of total work, which is +analysis events plus synthesis steps. On the Aweti sample synthesis is 1.0% of the total (15,940 of +1,560,180). On the Mbugwe sample it is 9.0% (130,647 of 1,449,301). E1, E2 and E3 save synthesis +steps only, so their "% less work" is a share of synthesis. In total work they save at most 1.1% on a +real grammar (table below). The dominant cost is analysis. The next round should target analysis: for +example, extending E4-style monotone checks earlier, and the `ApplicableTemplatesNotApplied` analysis +paths that the census ranks first. + ## The question HermitCrab is generate-and-test. Analysis unapplies rules to generate candidate analyses, and @@ -464,10 +511,208 @@ fails it on any corpus. - **Moving constraints into the analysis merge key**: ledger row 15. Every synthesis rejection depends on the root, so H4, not the analysis key, is the place. +## Results of the round + +Five experiments were built, each on its own unpushed branch from `origin/master` `c2454760`, with the +harness commit `af97348b`. Every result is steps only. "Synth" is synthesis steps, "total" is +analysis events plus synthesis steps for the same words in the same arm pair, and "% less work" is +the relative drop. Parses were identical per word in every arm pair on every corpus, except that the +four `truncate-morphotactic*` conformance fixtures are nondeterministic on master (see Open problems). + +### Decisions + +| Experiment | Branch @ SHA | Decision | +|---|---|---| +| E1 allomorph-loop cutoff | `perf/hc-allomorph-loop-cutoff` @ `d21af37e` | **KEEP.** Candidate for its own PR | +| E2 MPR and stem-name trail replay | `perf/hc-trail-mpr-replay` @ `4fbbfb14` (product `84f76cdf`) | **KEEP.** Candidate PR, to combine with E3: both use the same post-lookup hook | +| E3 template replay | `perf/hc-template-replay` @ `6edc4375` | **KEEP pending memoisation.** See E3 | +| E4 excluded co-occurrence | `perf/hc-excluded-cooccurrence` @ `02873608` (product `b07fc4d6`) | **PARK** | +| E5 required-environment prefilter | `perf/hc-env-input-prefilter` @ `9d55dade` (product `1af79e20`) | **DEAD END** as specified | +| E6 intermediate-shape check | none | Not run (research only) | + +None of the branches is pushed. A branch name and SHA is the citation. + +### Synthesis share of total work + +Synthesis steps and analysis events for the identical words, switch off: + +| Corpus | Synth | Analysis events | Synth share of total | +|---|---:|---:|---:| +| Aweti stride 2 | 15,940 | 1,544,240 | 1.0% | +| Mbugwe stride 40 (E2-E4 word set) | 130,647 | 1,318,654 | 9.0% | +| Sena stride 100 | 1,908 | 103,277 | 1.8% | +| Amharic limit 28 | 1,277 | 7,403 | 14.7% | +| Indonesian | 677 | 6,552 | 9.4% | +| Conformance | 8,246 | 27,765 | 22.9% | + +The E1 run used a Mbugwe word set with 3,451,016 analysis events (a different drop set), which is +why its Mbugwe row differs below. Analysis events never changed in E1, E2, E3 and E5-off arms. + +### E1. Allomorph-loop cutoff: KEEP + +`SynthesisAffixProcessRule.Apply` now stops after the recorded allomorph (`Morpher.CutAllomorphLoopAtRecorded`, +default on). Kill criterion not met. The soundness argument held: no parse changed. + +| Corpus | Synth off to on | % less synth | % less total work | +|---|---|---:|---:| +| Mbugwe stride 40 | 151,505 to 138,737 | 8.43% | 0.35% | +| Sena stride 100 | 1,908 to 1,772 | 7.13% | 0.13% | +| Aweti stride 2 | 15,940 to 15,831 | 0.68% | 0.007% | +| Amharic limit 28 | 1,277 to 1,271 | 0.47% | 0.07% | +| Conformance (minus 4 flaky) | 8,150 to 8,109 | 0.50% | 0.12% | +| Indonesian | 677 to 677 | 0% | 0% | +| `stress-allomorph-list-density` (Turkic-style) | 566 to 337 | 40.46% | 8.45% | +| `stress-prefix-class-density` (Bantu-style) | 341 to 190 | 44.28% | 9.49% | + +Cost not in the counters: one `IndexOf` over the rule's allomorphs per rule application. Unaffected: +analysis events and candidates (equal in both arms). + +### E2. MPR and stem-name trail replay: KEEP + +`TrailMprReplay` and `Morpher.PruneByTrailReplay` (default on) drop a candidate at lexical lookup +when its trail cannot satisfy required or excluded MPR features or the stem name. Two refinements to +the brief's predicate: the excluded test is disabled only when an `Overwrite` group is reachable from +a rule output, and the stem-name check stops after a compounding entry. Family roots are never +pruned. The shadow census agreed exactly with the on arm and found no successful terminal under a +rejected candidate anywhere. Kill criterion not met. + +| Corpus | Synth off to on | % less synth | % less total work | Replay checks | +|---|---|---:|---:|---:| +| Aweti stride 2 | 15,940 to 6,873 | 56.9% | 0.58% | 14,064 | +| Mbugwe stride 40 | 130,647 to 115,747 | 11.4% | 1.03% | 26,299 | +| Conformance | 8,246 to 8,218 | 0.3% | 0.08% | 251 | +| Sena, Amharic, Indonesian | unchanged | 0% | 0% | 0 | +| `stress-mpr-inflection-classes` (IE declensions) | 494 to 278 | 43.7% | 6.45% | 180 | +| `stress-mpr-bantu-concord` (three gated slots) | 3,376 to 382 | 88.7% | 31.85% | 652 | + +Replay checks are set operations, a different unit from synthesis steps (a step is a pattern match), +so they are reported beside the savings and not netted. The Aweti and Mbugwe gain comes from required +and excluded gates (Mbugwe 41 excluded, Aweti 26 excluded). + +### E3. Template replay: KEEP pending memoisation + +`TemplateReplay` and `Morpher.PruneByTemplateReplay` (default on) walk the stratum, template and rule +control flow over abstract states (trail pointer, partial, final), assuming every pattern match +succeeds. It bails out on family roots, non-affix-process rules on the trail, any realizational rule +in the grammar and a null stratum. The shadow census found no successful terminal under a rejected +candidate. Phase 0 census (steps in candidates with no success and shape-free): Mbugwe 40.0%, Aweti +73.4%, so the kill rule for the census was not met. + +| Corpus | Synth off to on | % less synth | % less total work | Replay states | +|---|---|---:|---:|---:| +| Aweti stride 2 | 15,940 to 5,112 | 67.9% | 0.69% | 22,308 | +| Mbugwe stride 40 | 130,647 to 114,833 | 12.1% | 1.09% | 133,606 | +| Amharic limit 28 | 1,277 to 532 | 58.3% | 8.58% | 718 | +| Sena stride 100 | 1,908 to 1,790 | 6.2% | 0.11% | 4,166 | +| Conformance | 8,246 to 8,181 | 0.8% | 0.18% | 10,622 | +| Indonesian | 677 to 667 | 1.5% | 0.14% | 389 | +| `stress-derivation-chain-homophone-roots` (Turkic-like) | 300 to 176 | 41.3% | unmeasured | 948 | +| `stress-bantu-extension-required-slot` | 279 to 225 | 19.4% | unmeasured | 763 | +| `stress-required-template-slot` (control) | 155 to 155 | 0% | 0% | 329 | + +The three stress rows do not give analysis event counts in the result files, so their total-work +share is unmeasured. The control has no gain because dead candidates there cost 0 steps. + +The brief's blow-up kill rule says to kill if walk states exceed steps saved, and it fired on every +corpus (Mbugwe 133,606 states against 15,814 steps saved; Aweti 22,308 against 10,828). The result +file therefore said PARK. The owner overrode the literal rule: a replay state is an integer set +operation and a saved step is a pattern match, so the two are different units and cannot be compared +by count. **Decision: KEEP pending memoisation.** Next step: memoise verdicts by root entry and trail, +then re-measure the replay cost in units comparable to a step. The result file also suggests skipping +the walk when the grammar has no required slots. + +### E4. Excluded co-occurrence during analysis: PARK + +`ExcludedCoOccurrencePruner` indexes active `Exclude` + `Anywhere` morpheme and allomorph +co-occurrence rules at `Morpher` construction. Rules that name only affixes are checked in analysis, +right after unapplying (H1). Rules that name a root are checked per alternative after lookup (H4). +It is sound and it is the only experiment that cuts analysis events. + +| Corpus | Analysis events off to on | Synth off to on | % less total work | +|---|---|---|---:| +| Aweti, Mbugwe, Sena, Amharic, Indonesian | unchanged | unchanged | 0% | +| Conformance (1,492 words) | 27,765 to 27,705 | 8,246 to 8,223 | 0.23% | +| `stress-excluded-cooccurrence` (brief's fixture) | 4,690 to 4,605 | 326 to 250 | 3.2% | +| `stress-athabaskan-prefix-complex` | 8,937 to 6,347 | 891 to 404 | 31.3% | +| `stress-bantu-extension-chain` | 10,552 to 8,548 | 515 to 332 | 19.8% | + +The brief's kill criterion (under 25% on its own fixture) fires on 3.2%. The two dense fixtures were +added at the owner's request. **Decision: PARK.** Land it only as a cheap safety net for +co-occurrence-heavy grammars. Findings that affect any landing: the "zero-width other is invisible" +trap does not occur for affix-process allomorphs (synthesis marks the morph on the last output node); +merged analyses must be filtered per alternative, not on the canonical word; with pruning on, +`MaxAlternatives` counts fewer candidates, so its exception can stop firing; and the rule index is +read at construction, so a rule added later is not pruned early. + +### E5. Required-environment input prefilter: DEAD END as specified + +The code is recoverable at `1af79e20`. The sound static scan disables the prefilter on 5 of 5 real +grammars (four with ambiguous digraph segmentation, one with non-uniform feature sets) and on 145 of +150 conformance runs, so it gives 0% on every real grammar. Forced on, with the scan bypassed (an +unsound licence), Sena saves 4.0% of synthesis steps and 7.3% of analysis events (net 5.7% after +counting matcher invocations) and Aweti is net -0.4%. + +| Fixture | Synth | Analysis | Gross % less total | Net % less work (matcher calls counted) | +|---|---|---|---:|---:| +| `stress-env-zero-allomorphs` | 87.4% less | 74.1% less | not given | 65.2% | +| `stress-env-bantu-concord` | 1,344 to 732 | 2,036 to 1,207 | 42.6% | -3.8% | +| `stress-env-austronesian-nasal-prefix` | 936 to 883 | 2,206 to 1,471 | 25.1% | -37.4% | + +Forced on, it is net negative on two of the three fixtures. Soundness findings: the brief's +counterexample direction (an archiphoneme `[+cons]` against a `[+voice]` context) does not hold under +subsumption. The real unsound case is a synthesized node more specific than the input character +definition (test word "atpa": `D` against `t`); forcing the prefilter on loses that parse, and the +static scan is what prevents it. + +**What would have to change to revive it.** (1) A scan that stops rejecting digraph grammars, which +needs a segmentation of the input that is unambiguous per grammar, or a prefilter that tests all +segmentations. (2) A treatment of non-uniform feature sets and of synthesized nodes more specific +than the input segment definitions. (3) A prefilter cost small enough that matcher invocations do not +cancel the saving, since two of three fixtures go net negative. (4) Optionally, root allomorph +environments (67 in Sena) prefiltered at lookup, which was not built. Absent (1) and (2) the +eligible set on real grammars is empty (indonesian 0 of 13, aweti 47 of 394, amharic 0 of 93, mbugwe +0 of 415, sena 5 of 117). + +### E6. Intermediate-shape check: not run + +No soundness argument yet; `SurfaceFormMismatch` (25.6% of real synthesis steps) is still untouched. + +### Conclusion + +Each of E1-E3 is sound on every corpus and cuts synthesis steps by a large amount on grammars that +lean on the feature (inflection classes, ordered allomorph lists, template slots). But synthesis is +1.0% of total work on Aweti and 9.0% on the Mbugwe sample, so on real grammars each removes about 1% or less of +total work (E2: 0.58% Aweti, 1.03% Mbugwe; E3: 0.69% and 1.09%; E1: 0.007% and 0.35%). E2 and E3 overlap +and were not measured together. **The dominant cost is analysis-side.** The next round should +target analysis, for example by extending E4-style monotone checks earlier, and the +`ApplicableTemplatesNotApplied` analysis paths. E4 shows an analysis check can cut work, but only +where the grammar has the constraint; the question is which constraints real grammars have. + +### Open problems found at baseline + +1. The four `truncate-morphotactic*` conformance fixtures (`truncate-morphotactic`, `-copy-u`, + `-identity`, `-subsumption`) are nondeterministic on master: their output varies run to run with no + change. They fail in both arms of every experiment, and `EveryConformanceFixturePassesSelfCheck` + fails on them. +2. `FieldworksProducibilityLedgerTests.CoversExactlyEveryFailureReasonAndEveryInterfaceAttributeInThisRepo` + fails on master because PR #548 added `AllomorphNotApplicable` without updating that ledger. +3. The stress fixtures live under `tests/SIL.Machine.Morphology.HermitCrab.Tests/Research/stress-fixtures/`, + not under `conformance/`, because registering a fixture needs ledger and coverage regeneration + (a full test cycle takes 34 to 53 minutes under load). E5's branch registered three fixtures and + broke 8 pinned ledger tests (row totals, fixture count, layer counts, interaction and inventory + ledgers, real-corpus pair counts); E2's registration attempt broke 7. Moving the fixtures is a + ledger-regeneration job. +4. Full-suite evidence: E3's run finished at 215 of 217 passing; the two failures are the truncate + flake and a SIGTERM under load. E4's run was 723 of 725, with the truncate flake and the ledger + failure of item 2. `local_check.sh --agent-strict` could not finish on these machines because the + SentencePiece step needs `cmake`; comment hygiene and formatting passed. + ## Open questions -- A candidate-level dead-step census (E3 phase 0) is the most useful single measurement left. It - turns this round's branch-level ceilings into candidate-level ones. +- The candidate-level dead-step census (E3 phase 0) was run: shape-free dead steps are 73.4% on Aweti + and 40.0% on Mbugwe. What is left is the cost of the replay in comparable units (E3 memoisation). +- Which analysis-side constraints real grammars carry, and whether they can be checked before a + candidate is expanded. This is the next round's target. - Whether synthesis phonological deletion removes morph annotations. E4's zero-width restriction may need to cover deleted morphs too. The E4 brief requires a test. - Whether `ChooseInflectionalStem` families occur in FLEx exports at all. None of the five samples has diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index 22e2789cd..fc0589d55 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -138,24 +138,24 @@ Rows 24 and 25 are **HC-Rust port bugs: C# was already exact, and no C# change f by running the port on the oracle's own grammar export and comparing lookup counts. Do that before optimizing the algorithm. -## 2026-10-27 synthesis-to-analysis round (planned) +## 2026-10-27 synthesis-to-analysis round (finished) Full record: [hermitcrab-2026-10-27-synthesis-to-analysis-round.md](hermitcrab-2026-10-27-synthesis-to-analysis-round.md). It answers John Maxwell's question about which synthesis tests can move into analysis. The unit is C# synthesis steps (rule-level trace events) and analysis events, counted by the test-only harness -on `research/failure-reason-stats` @ `af97348b`, never ms. Rows 32-37 are **planned and unmeasured**. +on `research/failure-reason-stats` @ `af97348b`, never ms. Rows 32-36 were built and measured on unpushed branches; row 37 was not run. Ranking covers every grammar type HermitCrab can represent, not only the five samples, and each experiment carries a synthetic stress fixture. | # | Optimization | Expected | Outcome | Number | | --- | --- | --- | --- | --- | -| 31 | **Failure-reason census** (measurement, not a change) | Maxwell's four candidates (excluded co-occurrence, disjunctive allomorph, environments, excluded MPR) would be where synthesis work dies | **MEASURED; the samples refute it.** On the five samples they are almost never terminal. The work dies on template paths and surface mismatches | Real grammars: ApplicableTemplatesNotApplied **43.5%** of synthesis steps, SurfaceFormMismatch **25.6%**, unattributed 22.1%, Maxwell's four **<0.1%**. `RequiredMprFeatures` is 41.4% of rule events but is mostly allomorph selection. Traced, so merging is off; analysis work is outside the step total | -| 32 | **E1: stop the synthesis allomorph loop after the recorded allomorph** | Allomorphs past the one analysis recorded are applied and then discarded | PLANNED | Ceiling: `AllomorphNotApplicable` with `i > r`, at most 11.6% of real synthesis steps | -| 33 | **E2: replay MPR features and stem names along the trail at lookup** (covers excluded MPR) | Inflection-class gates decided before synthesis | PLANNED; shadow census first | Samples near 0 terminal; target is IE and Bantu inflection classes | -| 34 | **E3: replay the template and final-state outcome at lookup** | Shape-free template-path deaths decided before synthesis | PLANNED; candidate-level dead-step census first | Branch ceiling 48.4% of real steps (Mbugwe 46.3% plus unattributed). Row 19's 0-6% on Sena and Amharic is the counterweight | -| 35 | **E4: excluded co-occurrence during analysis** (Maxwell) | Anti-monotone; prunes analysis subtrees | PLANNED | Samples about 0; conformance 0.2%. Zero-width "others" are invisible to the synthesis check and must not count | -| 36 | **E5: required-environment prefilter on the input word** (Maxwell) | Linear left-then-right scan per allomorph | PLANNED | Samples 0.0%. Sound only without underspecified surface segments, and with boundaries matching empty | -| 37 | **E6: intermediate-shape check against SurfaceFormMismatch** | Stop a candidate at the first stratum boundary where it diverges | RESEARCH ONLY; no soundness argument yet | SurfaceFormMismatch is 25.6% of real steps. Census only, killed by any success that fails it | +| 31 | **Failure-reason census** (measurement, not a change) | Maxwell's four candidates (excluded co-occurrence, disjunctive allomorph, environments, excluded MPR) would be where synthesis work dies | **MEASURED; the samples refute it.** On the five samples they are almost never terminal. The work dies on template paths and surface mismatches. Synthesis is only 1.0% (Aweti) to 9.0% (Mbugwe sample) of total work, so the dominant cost is analysis-side | Real grammars: ApplicableTemplatesNotApplied **43.5%** of synthesis steps, SurfaceFormMismatch **25.6%**, unattributed 22.1%, Maxwell's four **<0.1%**. Traced, so merging is off. Harness `research/failure-reason-stats` @ `af97348b`. See the [round record](hermitcrab-2026-10-27-synthesis-to-analysis-round.md#results-of-the-round) | +| 32 | **E1: stop the synthesis allomorph loop after the recorded allomorph** | Allomorphs past the one analysis recorded are applied and then discarded | **KEEP; candidate for its own PR.** Sound, parses identical. Unpushed branch `perf/hc-allomorph-loop-cutoff` @ `d21af37e` | Synthesis steps: Mbugwe **-8.43%**, Sena **-7.13%**, stress fixtures **-40.5% / -44.3%**. Share of total work (analysis plus synthesis): Mbugwe 0.35%, Sena 0.13%, Aweti 0.007%, stress 8.5% / 9.5% | +| 33 | **E2: replay MPR features and stem names along the trail at lookup** (covers excluded MPR) | Inflection-class gates decided before synthesis | **KEEP; candidate PR, combine with E3** (same post-lookup hook). Shadow census: no success rejected. Unpushed branch `perf/hc-trail-mpr-replay` @ `4fbbfb14` (product `84f76cdf`) | Synthesis steps: Aweti **-56.9%**, Mbugwe **-11.4%**, stress **-43.7% / -88.7%**. Total work: Aweti 0.58%, Mbugwe 1.03%, stress 6.45% / 31.85%. Replay checks (set operations) are reported beside, not netted | +| 34 | **E3: replay the template and final-state outcome at lookup** | Shape-free template-path deaths decided before synthesis | **KEEP pending memoisation.** The brief's kill rule (replay states above steps saved) fired on every corpus but compares int operations with pattern matches, so the owner overrode the literal PARK. Next: memoise verdicts by root entry and trail, re-measure in comparable units. Unpushed branch `perf/hc-template-replay` @ `6edc4375` | Synthesis steps: Aweti **-67.9%**, Amharic **-58.3%**, Mbugwe **-12.1%**. Total work: Aweti 0.69%, Mbugwe 1.09%, Amharic 8.58%; stress fixtures unmeasured (no analysis counts). Replay states: Mbugwe 133,606 against 15,814 steps saved | +| 35 | **E4: excluded co-occurrence during analysis** (Maxwell) | Anti-monotone; prunes analysis subtrees | **PARK.** Sound, and the only experiment that cuts analysis work, but 0% on the five real grammars. Land only as a cheap safety net. Unpushed branch `perf/hc-excluded-cooccurrence` @ `02873608` (product `b07fc4d6`) | Total work: real grammars **0%**, conformance 0.23%, brief's fixture 3.2% (kill under 25% fires), dense fixtures **31.3% / 19.8%**. Zero-width "others" are not invisible for affix-process allomorphs | +| 36 | **E5: required-environment input prefilter** (Maxwell) | Linear left-then-right scan per allomorph | **DEAD END as specified.** The soundness scan disables it on 5 of 5 real grammars and 145 of 150 conformance runs. Forced on it is net negative on two of three fixtures. Revival needs digraph-safe segmentation, handling of non-uniform feature sets and over-specific synthesized nodes, and a cheaper prefilter. Unpushed branch `perf/hc-env-input-prefilter` @ `9d55dade` (product at `1af79e20`) | Real grammars **0%**. Forced on: Sena -5.7% net, Aweti -0.4% net. Stress fixtures net 65.2% / -3.8% / -37.4% | +| 37 | **E6: intermediate-shape check against SurfaceFormMismatch** | Stop a candidate at the first stratum boundary where it diverges | RESEARCH ONLY; not run; no soundness argument yet | SurfaceFormMismatch is 25.6% of real synthesis steps. Census only, killed by any success that fails it | --- From 85956adfa325b0f51cd87f241a596f0072dc8950 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Tue, 27 Oct 2026 13:05:30 -0400 Subject: [PATCH 30/32] Drop memoisation from the E3 template-replay next step The added memory must stay within a few MB, so the replay walk is to be cut directly instead of cached. Co-Authored-By: Claude Opus 5.5 --- ...tcrab-2026-10-27-synthesis-to-analysis-round.md | 14 ++++++++------ docs/hermitcrab-optimization-ledger.md | 2 +- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/hermitcrab-2026-10-27-synthesis-to-analysis-round.md b/docs/hermitcrab-2026-10-27-synthesis-to-analysis-round.md index eabe3be01..53562033d 100644 --- a/docs/hermitcrab-2026-10-27-synthesis-to-analysis-round.md +++ b/docs/hermitcrab-2026-10-27-synthesis-to-analysis-round.md @@ -525,7 +525,7 @@ four `truncate-morphotactic*` conformance fixtures are nondeterministic on maste |---|---|---| | E1 allomorph-loop cutoff | `perf/hc-allomorph-loop-cutoff` @ `d21af37e` | **KEEP.** Candidate for its own PR | | E2 MPR and stem-name trail replay | `perf/hc-trail-mpr-replay` @ `4fbbfb14` (product `84f76cdf`) | **KEEP.** Candidate PR, to combine with E3: both use the same post-lookup hook | -| E3 template replay | `perf/hc-template-replay` @ `6edc4375` | **KEEP pending memoisation.** See E3 | +| E3 template replay | `perf/hc-template-replay` @ `6edc4375` | **KEEP, without memoisation.** See E3 | | E4 excluded co-occurrence | `perf/hc-excluded-cooccurrence` @ `02873608` (product `b07fc4d6`) | **PARK** | | E5 required-environment prefilter | `perf/hc-env-input-prefilter` @ `9d55dade` (product `1af79e20`) | **DEAD END** as specified | | E6 intermediate-shape check | none | Not run (research only) | @@ -589,7 +589,7 @@ Replay checks are set operations, a different unit from synthesis steps (a step so they are reported beside the savings and not netted. The Aweti and Mbugwe gain comes from required and excluded gates (Mbugwe 41 excluded, Aweti 26 excluded). -### E3. Template replay: KEEP pending memoisation +### E3. Template replay: KEEP, without memoisation `TemplateReplay` and `Morpher.PruneByTemplateReplay` (default on) walk the stratum, template and rule control flow over abstract states (trail pointer, partial, final), assuming every pattern match @@ -617,9 +617,11 @@ The brief's blow-up kill rule says to kill if walk states exceed steps saved, an corpus (Mbugwe 133,606 states against 15,814 steps saved; Aweti 22,308 against 10,828). The result file therefore said PARK. The owner overrode the literal rule: a replay state is an integer set operation and a saved step is a pattern match, so the two are different units and cannot be compared -by count. **Decision: KEEP pending memoisation.** Next step: memoise verdicts by root entry and trail, -then re-measure the replay cost in units comparable to a step. The result file also suggests skipping -the walk when the grammar has no required slots. +by count. **Decision: KEEP, without memoisation.** Memoising verdicts by root entry and trail is +ruled out: the owner will not accept a memory footprint above a few MB, and earlier memo rows (18, 28, +29) show how such tables grow. Next step: cost the replay walk in units comparable to a step, then cut +the walk itself without a cache, for example by skipping it when the grammar has no required slots or +by stopping at the first violated constraint. ### E4. Excluded co-occurrence during analysis: PARK @@ -710,7 +712,7 @@ where the grammar has the constraint; the question is which constraints real gra ## Open questions - The candidate-level dead-step census (E3 phase 0) was run: shape-free dead steps are 73.4% on Aweti - and 40.0% on Mbugwe. What is left is the cost of the replay in comparable units (E3 memoisation). + and 40.0% on Mbugwe. What is left is the cost of the replay in comparable units, cut without a memo (E3). - Which analysis-side constraints real grammars carry, and whether they can be checked before a candidate is expanded. This is the next round's target. - Whether synthesis phonological deletion removes morph annotations. E4's zero-width restriction may diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index fc0589d55..bebe5a3ac 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -152,7 +152,7 @@ experiment carries a synthetic stress fixture. | 31 | **Failure-reason census** (measurement, not a change) | Maxwell's four candidates (excluded co-occurrence, disjunctive allomorph, environments, excluded MPR) would be where synthesis work dies | **MEASURED; the samples refute it.** On the five samples they are almost never terminal. The work dies on template paths and surface mismatches. Synthesis is only 1.0% (Aweti) to 9.0% (Mbugwe sample) of total work, so the dominant cost is analysis-side | Real grammars: ApplicableTemplatesNotApplied **43.5%** of synthesis steps, SurfaceFormMismatch **25.6%**, unattributed 22.1%, Maxwell's four **<0.1%**. Traced, so merging is off. Harness `research/failure-reason-stats` @ `af97348b`. See the [round record](hermitcrab-2026-10-27-synthesis-to-analysis-round.md#results-of-the-round) | | 32 | **E1: stop the synthesis allomorph loop after the recorded allomorph** | Allomorphs past the one analysis recorded are applied and then discarded | **KEEP; candidate for its own PR.** Sound, parses identical. Unpushed branch `perf/hc-allomorph-loop-cutoff` @ `d21af37e` | Synthesis steps: Mbugwe **-8.43%**, Sena **-7.13%**, stress fixtures **-40.5% / -44.3%**. Share of total work (analysis plus synthesis): Mbugwe 0.35%, Sena 0.13%, Aweti 0.007%, stress 8.5% / 9.5% | | 33 | **E2: replay MPR features and stem names along the trail at lookup** (covers excluded MPR) | Inflection-class gates decided before synthesis | **KEEP; candidate PR, combine with E3** (same post-lookup hook). Shadow census: no success rejected. Unpushed branch `perf/hc-trail-mpr-replay` @ `4fbbfb14` (product `84f76cdf`) | Synthesis steps: Aweti **-56.9%**, Mbugwe **-11.4%**, stress **-43.7% / -88.7%**. Total work: Aweti 0.58%, Mbugwe 1.03%, stress 6.45% / 31.85%. Replay checks (set operations) are reported beside, not netted | -| 34 | **E3: replay the template and final-state outcome at lookup** | Shape-free template-path deaths decided before synthesis | **KEEP pending memoisation.** The brief's kill rule (replay states above steps saved) fired on every corpus but compares int operations with pattern matches, so the owner overrode the literal PARK. Next: memoise verdicts by root entry and trail, re-measure in comparable units. Unpushed branch `perf/hc-template-replay` @ `6edc4375` | Synthesis steps: Aweti **-67.9%**, Amharic **-58.3%**, Mbugwe **-12.1%**. Total work: Aweti 0.69%, Mbugwe 1.09%, Amharic 8.58%; stress fixtures unmeasured (no analysis counts). Replay states: Mbugwe 133,606 against 15,814 steps saved | +| 34 | **E3: replay the template and final-state outcome at lookup** | Shape-free template-path deaths decided before synthesis | **KEEP, without memoisation.** The brief's kill rule (replay states above steps saved) fired on every corpus but compares int operations with pattern matches, so the owner overrode the literal PARK. No memo: the owner caps added memory at a few MB. Next: cost the replay in comparable units and cut the walk itself (skip it when no slot is required, stop at the first violation). Unpushed branch `perf/hc-template-replay` @ `6edc4375` | Synthesis steps: Aweti **-67.9%**, Amharic **-58.3%**, Mbugwe **-12.1%**. Total work: Aweti 0.69%, Mbugwe 1.09%, Amharic 8.58%; stress fixtures unmeasured (no analysis counts). Replay states: Mbugwe 133,606 against 15,814 steps saved | | 35 | **E4: excluded co-occurrence during analysis** (Maxwell) | Anti-monotone; prunes analysis subtrees | **PARK.** Sound, and the only experiment that cuts analysis work, but 0% on the five real grammars. Land only as a cheap safety net. Unpushed branch `perf/hc-excluded-cooccurrence` @ `02873608` (product `b07fc4d6`) | Total work: real grammars **0%**, conformance 0.23%, brief's fixture 3.2% (kill under 25% fires), dense fixtures **31.3% / 19.8%**. Zero-width "others" are not invisible for affix-process allomorphs | | 36 | **E5: required-environment input prefilter** (Maxwell) | Linear left-then-right scan per allomorph | **DEAD END as specified.** The soundness scan disables it on 5 of 5 real grammars and 145 of 150 conformance runs. Forced on it is net negative on two of three fixtures. Revival needs digraph-safe segmentation, handling of non-uniform feature sets and over-specific synthesized nodes, and a cheaper prefilter. Unpushed branch `perf/hc-env-input-prefilter` @ `9d55dade` (product at `1af79e20`) | Real grammars **0%**. Forced on: Sena -5.7% net, Aweti -0.4% net. Stress fixtures net 65.2% / -3.8% / -37.4% | | 37 | **E6: intermediate-shape check against SurfaceFormMismatch** | Stop a candidate at the first stratum boundary where it diverges | RESEARCH ONLY; not run; no soundness argument yet | SurfaceFormMismatch is 25.6% of real synthesis steps. Census only, killed by any success that fails it | From 51a8bcede8428b3071e4e2e34c9729ffeb1f416c Mon Sep 17 00:00:00 2001 From: John Lambert Date: Tue, 27 Oct 2026 21:12:43 -0400 Subject: [PATCH 31/32] Record the HermitCrab analysis-side optimization round R1, R2, R5 and T1 are kept with their net-of-own-cost numbers in steps, R4 is a recorded dead end, R6 is unbuilt, R3 and R1 on #546 are in progress. Adds ledger rows 38-46. Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-2026-10-27-analysis-round.md | 377 ++++++++++++++++++ ...-2026-10-27-synthesis-to-analysis-round.md | 3 +- docs/hermitcrab-optimization-ledger.md | 38 +- 3 files changed, 416 insertions(+), 2 deletions(-) create mode 100644 docs/hermitcrab-2026-10-27-analysis-round.md diff --git a/docs/hermitcrab-2026-10-27-analysis-round.md b/docs/hermitcrab-2026-10-27-analysis-round.md new file mode 100644 index 000000000..fbcb2706f --- /dev/null +++ b/docs/hermitcrab-2026-10-27-analysis-round.md @@ -0,0 +1,377 @@ +# HermitCrab analysis-side optimization, 2026-10-27 round + +**Disposition: four candidates are PRs (R1 #575, R2 #572, T1 #571, R5 #574), one is a dead end (R4), one +is not built (R6), and two are in progress (R3, R1 on #546).** This round follows the +[synthesis-to-analysis round](hermitcrab-2026-10-27-synthesis-to-analysis-round.md), whose conclusion was that +synthesis is 1.0% (Aweti) to 9.0% (Mbugwe sample) of total work and the dominant cost is analysis. Here the +target is the analysis side: the template battery, rule outputs that cannot reach a root, matcher work per +attempt, and trace memory. Measurements are in steps, never ms. + +This record follows the [optimization ledger](hermitcrab-optimization-ledger.md). Ledger rows 38-46 point here. + +## Unit and method + +- **Work** is analysis events (the census harness's `ITraceManager` callbacks) plus synthesis steps for the same + words. "% less work" is `(off - on) / off`, per grammar, never pooled. Where a check does work of its own, a + net figure is given in one unit and the unit is named. +- Harness: `research/analysis-cost-census` @ `38e9622e`, based on master `c2454760` (includes PR #480's + conformance suite). Each candidate is an internal Off / Shadow / On mode on `Morpher`, product default On. + The same binary runs both arms; census mode is traced and unmerged, `censusmerge` uses the merged walk. +- Grammars: Indonesian (all 70 words), Aweti (stride 2), Amharic (limit 28), Mbugwe (stride 40, 150 s per + word), Sena (stride 100), the conformance suite (150 runs), and one synthetic stress fixture per candidate. + Words that abort in either arm are dropped from both and listed in the result files. +- Soundness: a shadow mode counts violations (a parse below a fired prune). Any violation stops the candidate. + Parses are compared per word by a signature of root, rule trail and allomorphs. +- Gates for every PR: the HermitCrab test project on the branch and on master (every branch failure also fails on + master), conformance per-fixture verdicts identical, `dotnet csharpier check .`, comment hygiene. The full + `local_check.sh --agent-strict` stops at the SentencePiece build, which needs `cmake`; CI is the first full run. + +## Decisions + +| Candidate | Issue | PR | Decision | +|---|---|---|---| +| R1 required-slot look-ahead | #561 | #575 | **KEEP.** Kill rule waived, see R1 | +| R2 root-substring prune | #562 | #572 | **KEEP**, with a probe skip | +| T1 lean trace | #563 | #571 | **KEEP.** Lands after #534 | +| R5 edge-segment prefilter | #564 | #574 | **KEEP.** The plan's score of 0 was wrong | +| R4 subsequence fallback | #568 | none | **DEAD END** | +| R6 merging while tracing | #565 | none | Not built. Owner product decision | +| R3 ordered look-ahead | #567 | | In progress | +| R1 on #546 | #566 | | In progress | + +Earlier-round PRs: **#560** (E1, allomorph-loop cutoff) and **#570** (MPR and stem-name trail replay). E3 (template +replay) was dropped on cost; its code stays at `perf/hc-lookup-replay-templates` @ `9efd059c`. + +## What the census said to aim at + +From the [analysis-cost census](hermitcrab-optimization-ledger.md) the plan took these facts: + +- The template battery is 69-98% of analysis events on Aweti, Mbugwe and Sena. Mbugwe has 63.8% of its events in + template partials that never reach lookup (Sena 22.1%, Aweti 11.7%). +- Not-unapplied attempts are 70-91% of events. Skipping one with a cheaper test removes no step; it makes a step + cheaper. The plan therefore scored prefilters 0 in steps (R5). That scoring turned out to be the lesson of R5. +- The root-substring predicate at rule outputs had 13.6-38.5% ceilings with 0 violations on the five real + grammars, but 29 success-path nodes on conformance, so it needed static soundness flags (R2). +- PR #546 shares trailing slot batches only in untraced, sequential walks, so it is inert under tracing. + +Eight issues were filed in sillsdev/machine (#561-#568), each checked against open and closed issues and PRs for +duplicates. Linked as related, not duplicates: #485, #512, #551, PRs #546, #534, #490, #480. + +## R1. Required-slot look-ahead: KEEP (issue #561, PR #575) + +**Claim.** A template partial that reaches a required slot where no rule can unapply dies there, after paying for +every optional slot in between. A necessary condition for a required slot to unapply later is that, for some rule +and allomorph in it, every inserted affix segment unifies with some Segment node of the current shape. The check +runs after the final-template guard, before recursing in `ApplySlots`, and before pushing work in +`ParallelApplySlots`. It is used for slot j only when every slot walked after j is pure (copy and insert only, no +part that can match zero segments). Optional nodes count as present. Memory is grammar-sized tables only: no +per-word state, no cache. + +**Soundness.** Shadow mode, 0 violations and 0 completions under a fired check: + +| grammar | fired | violated | +|---|---:|---:| +| Mbugwe | 803,228 | 0 | +| Sena | 6,402 | 0 | +| Aweti | 5,948 | 0 | +| Amharic | 584 | 0 | +| Indonesian | 0 | 0 | +| conformance (150 runs) | 298 | 0 | +| stress-bantu-tam | 689 | 0 | + +Parses were equal per word, traced and merged. Mutation checks: removing the purity gate fails the impurity +tests, and disabling the mode fails the prune tests. + +**Event-based result** (analysis events, fewer is better; synthesis steps unchanged): + +| grammar | analysis events off to on | % less events | % less work (events plus synthesis) | +|---|---|---:|---:| +| Mbugwe (40 words) | 5,289,909 to 4,940,905 | 6.60% | 6.29% | +| Sena (51) | 110,986 to 76,809 | 30.79% | 30.27% | +| Aweti (99) | 1,596,848 to 1,475,588 | 7.59% | 7.52% | +| Amharic (24) | 8,560 to 7,800 | 8.88% | 7.73% | +| conformance | 38,393 to 37,796 | 1.55% | 1.28% | +| Indonesian (70) | 7,468 to 7,468 | 0% | 0% | +| stress-bantu-tam (60) | 45,342 to 11,497 | 74.64% | 74.35% | + +**Unification-work result.** The unit is top-level `FeatureStruct` unification calls (`IsUnifiable` plus `Unify`) +over the whole parse, look-ahead checks included, counted by census-only counters. Positive is less work: + +| grammar | calls off to on | % less unification work | direction | +|---|---|---:|---| +| Mbugwe | 21,515,185 to 21,444,795 | +0.33 | break-even, slightly less | +| Sena | 1,308,998 to 1,288,517 | +1.56 | less | +| Aweti | 17,699,599 to 16,902,830 | +4.50 | less | +| Amharic | 411,073 to 413,674 | **-0.63** | **more work** | +| Indonesian | 662,201 to 662,201 | 0.00 | none | +| conformance | 303,738 to 303,313 | +0.14 | break-even | +| stress-bantu-tam | 162,015 to 100,593 | +37.91 | less | + +Mbugwe's large event saving is mostly walk and trace work that costs few unifications, so little of it shows in +this unit. The check cost 1,112,761 unifications on Mbugwe against 349,004 events saved. Mbugwe reaches only +10.3% of its no-lookup ceiling, because its required slots mostly insert one segment (`-a`, `a-`), which is nearly +always present somewhere in the shape. R3 is the follow-up. + +**Waived kill rule.** The plan said to close R1 if check unifications exceeded 3x the events saved on the grammar +with the largest saving. That fired on Mbugwe at 3.19x. It was **waived**, because it compared unification calls +with walk and trace events, which are unlike units. A same-unit comparison replaced it (table above). The earlier +ratios (1.5x to 12.9x) are withdrawn. The review asked for this to be stated plainly in the PR, with a direction +word on every figure, and it is. + +**Tried and rejected.** Skipping slots whose every allomorph inserts a single segment: it did not cut Mbugwe's +check cost (1,119,073 against 1,112,761 check unifications) and halved Sena's saving (17.75% against 30.79% of +events; unification work -0.70% instead of +1.56%). The product keeps the uncut test. + +**Gates.** HermitCrab tests 168 passed, 0 failed on the branch (master 151 passed). Conformance 146 passed and 4 +failed on both, the four being the nondeterministic `truncate-morphotactic` family. csharpier clean, comment +hygiene clean. Product branch `perf/hc-required-slot-lookahead` @ `0bf9e646` at the time of the result (changed by +review); measurement branch `research/hc-rsl-census` @ `6ad11702`; the unification counters live only there. + +**Risks.** Amharic is net more work in unification calls, on 24 words. The stress fixture is synthetic. PR #534 +edits the same file; the diff is three hooks and one helper. Aborted words (the heaviest) are dropped from both +arms, so the event saving is a floor for them but unproven in unification calls. + +## R2. Root-substring prune: KEEP (issue #562, PR #572) + +**Claim.** When a node's shape contains no root allomorph as a contiguous substring, and nothing that can still be +unapplied below it can restore, insert or rejoin stem material, no lookup below it can hit. The probe runs at +the three analysis rule outputs. Three static per-stratum conditions turn it off: a rule that drops a part that +can hold segments, copies a part more than once, copies out of order or modifies a part; deeper phonological +rules; and lexical-pattern roots. A rule type it does not know also turns it off. + +**Why it is not row 27.** Row 27 closed root reachability summaries (the prunable bucket was 0-5% on 6 of 8 lists, +and nodes may still compound). A substring test passes a compound, because the compound contains its head root; +it runs at every rule output, including template-slot outputs; and the flags turn it off where stem material can +be restored. Row 27's reopening condition, a shadow summary deciding a meaningful share, is met on Sena (17.7%) and +the stress fixture (67.7%), with 0 violations. + +**Where the flag is on** (first failing condition per stratum): Sena all three strata; the stress fixture's main +stratum; conformance the main stratum in 126 of 150 runs. Off: Indonesian (a rule copies a part more than once; +deeper phonology), Mbugwe (a rule drops a part that can hold segments), Amharic (a rule drops a part), Aweti (a +rule drops a part). Census ceilings for those four, from the plan, were Mbugwe 38.5%, Indonesian 29.6%, Aweti 14.8% +and Amharic 13.6%, which are not reachable soundly (see R4). + +**Net of probes.** After review, each probe is charged as one event: + +| list | gross % less work | net % less work | probes | note | +|---|---:|---:|---:|---| +| Sena (51 words) | 17.7 | **15.3** | 2,698 | parse sets equal | +| stress, early-lexicon agglutinative (60) | 67.7 | **66.5** | 245 | 13,584 events saved | +| conformance | 0.2 | **-15.3** (more work) | 7,237 | buys 92 events; 2 words save more than they probe | +| Indonesian, Mbugwe, Amharic, Aweti | 0 | **0** | 0 | flag off | + +The conformance result is the honest cost: it is a suite of tiny words and small lexicons, so the probes exceed the +events saved, and there is no adaptive off switch. Only Sena among real grammars gains. + +**Probe skip.** Added after review: the probe is skipped when the output keeps at least as many segments as its +input within the same stratum. It is sound (a skip only keeps a word). It removed only 18 of 7,255 probes. + +**Rejected: a root-count gate.** Turning the prune off for strata with fewer than 16 roots would have hidden the +conformance loss. The coordinator rejected it: it is tuned to the toy lexicons in the conformance suite, and no +real grammar would trigger it. Its census was cancelled, so it has no measured result. + +**Gates.** HermitCrab tests 161 passed, 0 failed (master 151). 10 new tests in `RootSubstringPruneTests`; +removing each flag condition turns exactly its test red. Conformance 146 passed and 4 failed on both arms, per +fixture identical. Branch `perf/hc-root-substring-prune` @ `d58a8e88`. + +**Risks.** Pruned analyses no longer count toward the analysis-side alternative limit, so a parse that threw +`MaxAlternativesExceeded` may now succeed, never the reverse. Try-a-Word traces lose the subtree below a pruned +output. Soundness depends on `AnalysisMorphologicalTransform.GenerateShape` semantics; a future analysis step that +changes shape outside rules and stratum entry would break it silently. + +## T1. Lean trace: KEEP, lands after #534 (issue #563, PR #571) + +**Claim.** The default `TraceManager` pins about 1 GB on Mbugwe yɔɔna (1,031.95 MiB retained for 21,200 analysis +events and 41,130 synthesis steps, from the trace-memory study) because every node pins a `Word`, either live or +as one of six deep clones. An opt-in `TraceManager.RetainWords = false` keeps shape text (`Trace.InputText`, +`Trace.OutputText`) and no `Word`. Steps are unchanged by construction. The default stays `true`: `Trace.Input` +and `Trace.Output` are public and would be null in lean mode, so flipping is an owner decision. The CLI opts in. + +Retained trace memory, same word, same run, product arm against lean arm: + +| grammar | words | mean full MiB | mean lean MiB | % less (sum) | max full to lean MiB | +|---|---:|---:|---:|---:|---| +| Mbugwe (stride 40) | 40 | 407.2 | 25.10 | 93.8% | 5,485.6 to 389.3 | +| Aweti (stride 2) | 99 | 35.6 | 2.28 | 93.6% | 656.2 to 29.4 | +| Sena (stride 100) | 51 | 4.35 | 0.35 | 92.0% | 27.5 to 2.0 | +| Amharic (limit 28) | 24 | 6.86 | 0.08 | 98.8% | 17.5 to 0.2 | +| Indonesian | 70 | 2.24 | 0.03 | 98.7% | 18.1 to 0.3 | +| conformance | 1,476 | 0.45 | 0.01 | 97.4% | 329.0 to 7.5 | +| stress deep-derivation | 10 | 200.36 | 4.99 | 97.5% | 343.0 to 7.8 | + +yɔɔna: 1,035.0 to 22.6 MiB (97.8% less); peak falls from 1,038 to 89 MiB. The kill criterion (under 50% on +yɔɔna) is not met. What still holds the lean memory is the node objects and two strings per node, about 190 bytes +a node on yɔɔna (an estimate from totals, not a heap count). Allocation is flat: lean adds text formatting and saves +clones. + +**Steps are identical.** Parse counts and node counts equal for every compared word; the census rows for Indonesian +(70 words) and yɔɔna are byte-equal (md5) between base and product; conformance per-fixture verdicts are identical; +CLI output is byte-identical to master with one processor apart from the "Parse time" line. With default +parallelism master itself orders some synthesis nodes differently run to run. + +**Landing order.** PR #534 rewrites nearly every `TraceManager` callback, `Trace.cs` and `ParseCommand.cs`, so a +textual conflict is certain. Land #534 first, then rebase this onto it, projecting #534's fields in lean mode: +copy strings or ids at the callback (summarise `FailureObject`, keep the allomorph id, the blocking entry name and +the slot rule name) instead of references. + +**Gates.** HermitCrab tests 156 passed, 0 failed (master 151). 5 new `LeanTraceTests`; forcing lean mode to store +words fails 2 of 5. csharpier and hygiene clean. Product commit `9f7828e8` on `perf/hc-lean-trace`. + +**Risks.** Lean text of an unfrozen word is taken at callback time; full mode formats at read time. An unfrozen +word mutated after the callback would differ. An instrumented run found none: unfrozen words reach callbacks only +as private pre-rule clones, compounding temporaries and phonological outputs the manager clones. Default mode still +pins about 1 GB on yɔɔna. + +## R5. Edge-segment prefilter: KEEP (issue #564, PR #574) + +**Claim.** Port ledger row 22's analysis affix prefilter to master. A pattern anchored at both ends must match its +first and last segment by unification, so a mandatory edge segment that cannot unify rejects the attempt before +the matcher runs. It skips boundaries, lets optional edge segments through, and emits the existing failed +unapplication trace callback. An internal Off/On switch defaults On. + +**The plan scored it 0, and that was wrong.** Census steps (analysis events and synthesis steps) are **unchanged** +on all 1,796 completed pairs per mode, so by the owner's step metric the candidate removes nothing. It makes each +failed attempt cheaper. The first measurement reported gross matcher work and left out the prefilter's own +unifications; review required the net. The headline is **net matcher work**: traversals plus arc checks, with each +prefilter unification charged as one arc check. This is the lesson: a step count is blind to a change that +cheapens each step. + +Census mode (unmerged), pairs completed in both arms: + +| list | traversals off to on | arc checks off to on | prefilter unifications | gross less | net less | +|---|---|---|---:|---:|---:| +| Aweti | 1,539,228 to 444,856 | 18,959,730 to 9,102,289 | 1,221,061 | 53.43% | **47.47%** | +| Mbugwe | 4,894,285 to 2,464,790 | 18,994,380 to 13,973,576 | 2,763,590 | 31.19% | **19.62%** | +| Sena | 110,368 to 30,893 | 1,297,930 to 1,014,492 | 87,560 | 25.77% | **19.55%** | +| Amharic | 10,143 to 6,931 | 345,463 to 300,000 | 3,305 | 13.69% | **12.76%** | +| Indonesian | 21,107 to 18,767 | 654,511 to 610,183 | 2,682 | 6.91% | **6.51%** | +| stress | 20,284 to 3,589 | 343,290 to 58,511 | 19,154 | 82.92% | 77.65% | +| conformance | 43,682 to 35,667 | 252,488 to 175,788 | 16,248 | 28.60% | 23.12% | + +`censusmerge` gives the same ordering (Aweti 49.24%, Mbugwe 21.24%, Sena 19.83%, Amharic 12.94%, Indonesian 6.60%). +Counts are unweighted, not a timing claim. The user-specified kill criterion (net under 5% on every real grammar) +is not met: the smallest net is Indonesian at 6.51%. + +**Parity.** All 1,796 completed pairs per mode have identical parse counts and signature sets. The +`edge-cases/truncate-morphotactic` family is excluded (repeated runs differ on master itself). Exclusions are +listed in `edge-prefilter-evidence/census-net-summary.txt` of the working set. + +**Memory.** Mbugwe has 874 compiled analysis affix-rule instances, 1,620 allomorph specs and 3,648 list-capacity +slots, giving a conservative 108.4 KiB of retained metadata (below 159 KiB with abandoned list backing arrays). +Counts measured, byte figures are 64-bit layout estimates, not a heap profile. Nothing per word, no cache. + +**Gates.** HermitCrab tests 169 passed (master 151); tests cover the exact failure trace on matcher and prefilter +rejection, alpha-variable constraints, a group with an optional leading constraint, and a multi-segment suffix +ending in the `ts` digraph. Conformance verdicts identical (146 passed, 4 failed on both). Both `local_check.sh` +runs fail at the SentencePiece `cmake` step. Branch `perf/hc-edge-segment-prefilter` @ `bb525683`. + +## R4. Subsequence fallback and insertion classes: DEAD END (issue #568) + +**Claim tested.** R2 is off on four of five real grammars because their rules restore, reorder or duplicate stem +material. R4 asked whether a weaker sound test still prunes: "some root is a subsequence of the shape" for +order-preserving removal, with the segment classes that a dropped part can restore treated as wildcard gaps. +Ceilings from the census, minus any R1 overlap: Mbugwe 38.5%, Indonesian 29.6%, Aweti 14.8%, Amharic 13.6%. + +**Built.** `RootPruneRuleShape` classifies each rule as substring-safe, subsequence-safe or order-free-safe, with +the segment classes its dropped parts restore. `RootSubsequenceTrie` holds the roots of the stratum and deeper +strata; root segments that unify with a gap class are skipped, since a restorable segment may be re-inserted +anywhere; the earliest-match search runs depth first; order-free mode ignores positions. Each stratum gets the +tier Substring, else Subsequence, else Order-free, and turns off for rules that modify a part, unknown rules, +deeper phonology, lexical-pattern roots, or when every root is made of restorable segments. Memory is one small +trie per stratum (Mbugwe 165 roots). Product branch `perf/hc-root-subsequence-prune`, stacked on R2's `d58a8e88`, +left unsquashed with the order-free code at **`71cba028`**. No PR. + +**Cost accounting.** One probe is 1 step and each trie arc examined is 1 step (`RootSubsequenceSteps`, a hook +that is null in production). + +Sound subsequence tier with gaps, net: + +| list | status | analysis events off to on | net % less work | +|---|---|---|---:| +| Indonesian | off (REDUP-meN copies parts out of LHS order) | 7,468 to 7,468 | 0.0 | +| Amharic | off (a rule modifies a part) | 8,560 to 8,560 | 0.0 | +| Aweti | off (a rule copies parts out of LHS order) | 1,596,848 to 1,596,848 | 0.0 | +| Mbugwe | off (every root is made of restorable segments) | 3,500,941 to 3,500,941 | 0.0 | +| Sena | on, R2's tier unchanged | 110,986 to 91,010 | 15.3 | +| stress | on | 19,959 to 6,375 | 66.5 | +| conformance | on in many fixtures | 38,393 to 38,225 | -19.1 | + +With the order-free tier added (rules that also reorder parts): + +| list | status | events off to on | gross % | net % | probes | arcs | shadow violations | +|---|---|---|---:|---:|---:|---:|---:| +| Indonesian | on (Clitics, Surface off: deeper phonology) | 7,468 to 4,956 | 30.8 | **-182.9** (more work) | 631 | 16,779 | 0 | +| Aweti | off (every root is restorable) | unchanged | 0.0 | 0.0 | 0 | 0 | n/a | +| Mbugwe, Amharic | off, as above | unchanged | 0.0 | 0.0 | | | | +| stress | on | 19,959 to 6,375 | 67.7 | 66.5 | 245 | 0 | 0 | +| conformance | on | 38,393 to 38,225 | 0.4 | -19.1 | 7,432 | 1,657 | 0 | + +**Kill criterion.** Under 5% net on every one of Mbugwe, Indonesian, Aweti, Amharic: met (0.0, -182.9, 0.0, 0.0). +No parse differences outside the known truncate-morphotactic family. Verdict DEAD-END. + +**Why.** + +- **The ceilings are unsound.** The flag-ignoring census (fires of the plain subsequence predicate, share of events + below the topmost pruned nodes) is Indonesian 55.2%, Mbugwe 36.1%, Aweti 21.7%, Amharic 50.9%, Sena 16.8%, + stress 50.6%. Most of it in the three restoring grammars is unsound: **violation events 334,508 (Mbugwe), + 50,187 (Aweti)**, 2,264 (Amharic) and 1,105 (Indonesian). A raw subsequence prune would lose real parses. +- **The insertion-class (wildcard gap) variant never fires** on Mbugwe, Aweti or Amharic: their dropped parts + describe classes that cover every root segment, so no word can be ruled out. Position-specific gaps were not + tried. +- **Amharic is blocked** by a rule that modifies a part, for which there is no sound test. +- **Indonesian is the one grammar where the sound tier works** (30.8% fewer events, 0 shadow violations), and it + costs 26.6 trie arcs per probe against about 4 events saved per probe: a net loss at one step per arc. Every + cheaper structure considered (a set of present token classes, a bitmask per root) costs about the same 25-40 steps + per probe as there are token classes. + +**Revival would need** a test that costs well under one step per saved event on Indonesian, position-specific gaps +for the restoring grammars, and a sound treatment of rules that modify a part. None is in sight. + +**Known nondeterminism.** Conformance parse signatures differ for the same words between off-arm census and +censusmerge runs (`truncate-morphotactic-identity#default` word 2), and the on arms differ only on words in the +`truncate-morphotactic` family. Parse counts are equal. + +## R6, R3 and R1 on #546 + +- **R6, merging while tracing (#565): not built.** Census merging changes analysis events by only 0.2-2.4%, but + under tracing lookups are 1.67x (Mbugwe) to 2.19x (Aweti) more frequent. Whether Try-a-Word shows folded + duplicates is an owner product decision, and it is low in steps. It interacts with #534 and #563. +- **R3, ordered and edge-anchored look-ahead (#567): in progress.** It targets the 90% of Mbugwe's no-lookup + ceiling that R1 leaves, where required slots insert a single segment. Result not recorded here. +- **R1 on #546 (#566): in progress.** It measures R1 on the shared-slot walk and reports what #546 leaves. The rule + set in advance: if R1 saves under 3% of slot events on the #546 walk where it saved over 5% on master, the two + overlap and only one should land. Further sharing of non-trailing slots needs a key on the shape, which is a memo, + so any residual is reported as a ceiling and not proposed. + +## Cross-cutting lessons + +1. **Count a check's own cost in the same unit as what it saves.** R1, R2 and R5 all first reported gross numbers. + Net, R1 broke even on Mbugwe and lost on Amharic; R2 lost 15.3% on conformance; R5 fell from 6.91% to 6.51% on + Indonesian and from 31.19% to 19.62% on Mbugwe. R1's 3x kill rule compared unification calls with walk and trace + events and was waived for that reason, with a same-unit comparison in its place. R4 was killed by the same + accounting: 30.8% gross became 182.9% more work. +2. **Truncate-morphotactic conformance fixtures are nondeterministic on master and must be excluded from parity.** + `truncate-morphotactic`, `-copy-u`, `-identity` and `-subsumption` give different output run to run with no + change. Their mismatch detail text differs between two base runs, so only per-fixture PASS and FAIL is + comparable, and the same four fail on both arms of every experiment. PanGloss waives the same family. +3. **Census steps miss work that cheapens each step.** R5 scores 0 in steps and saves 6.5% to 47.5% of matcher + work. Choose the unit the change acts on, report steps beside it, and do not rank by steps alone. +4. **No memoization (owner rule).** It is why E3's replay was dropped instead of cached, why R1 and R5 keep + grammar-sized tables only, and why shared work beyond #546 is a reported ceiling and not a proposal. +5. **A port can expose a divergence in the other implementation.** The PanGloss port of machine #548 found and + fixed a real Rust divergence in alpha-environment selection (PanGloss PR #13). Porting a change to the second + engine is a cheap second oracle. +6. **A ceiling measured with the flags ignored can be mostly unsound.** R4's census ceilings (up to 55.2%) were + mostly restoration violations; the sound share on the four target grammars was zero. +7. **A gate tuned to the test lexicons is overfitting.** R2's root-count gate would have removed the conformance + loss and no real-grammar gain, so it was rejected. + +## Open questions + +- R3 and R1-on-#546 results (issues #567, #566). +- Whether a sound subsequence test can be made cheap enough for Indonesian-style grammars, where the sound tier + prunes 30.8% of events but costs 26.6 arcs per probe. +- R6: whether Try-a-Word should show folded duplicates (owner decision). +- Landing order of #571 and #534, and whether the default of `RetainWords` should ever flip (a major-version + decision). +- R2 on conformance-like workloads: the prune has no adaptive off switch, and loses 15.3% there. diff --git a/docs/hermitcrab-2026-10-27-synthesis-to-analysis-round.md b/docs/hermitcrab-2026-10-27-synthesis-to-analysis-round.md index 53562033d..f49172e83 100644 --- a/docs/hermitcrab-2026-10-27-synthesis-to-analysis-round.md +++ b/docs/hermitcrab-2026-10-27-synthesis-to-analysis-round.md @@ -7,7 +7,8 @@ built and measured on throwaway branches (none pushed); E6 stays research-only. [Results of the round](#results-of-the-round). Measurements are in steps, never time. This record follows the [optimization ledger](hermitcrab-optimization-ledger.md). Ledger rows 31-37 -point here. +point here. The analysis-side round that followed is +[hermitcrab-2026-10-27-analysis-round.md](hermitcrab-2026-10-27-analysis-round.md); it records that E3 was dropped on cost. ## Answer to John Maxwell's questions diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index bebe5a3ac..2a2511ff5 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -152,11 +152,47 @@ experiment carries a synthetic stress fixture. | 31 | **Failure-reason census** (measurement, not a change) | Maxwell's four candidates (excluded co-occurrence, disjunctive allomorph, environments, excluded MPR) would be where synthesis work dies | **MEASURED; the samples refute it.** On the five samples they are almost never terminal. The work dies on template paths and surface mismatches. Synthesis is only 1.0% (Aweti) to 9.0% (Mbugwe sample) of total work, so the dominant cost is analysis-side | Real grammars: ApplicableTemplatesNotApplied **43.5%** of synthesis steps, SurfaceFormMismatch **25.6%**, unattributed 22.1%, Maxwell's four **<0.1%**. Traced, so merging is off. Harness `research/failure-reason-stats` @ `af97348b`. See the [round record](hermitcrab-2026-10-27-synthesis-to-analysis-round.md#results-of-the-round) | | 32 | **E1: stop the synthesis allomorph loop after the recorded allomorph** | Allomorphs past the one analysis recorded are applied and then discarded | **KEEP; candidate for its own PR.** Sound, parses identical. Unpushed branch `perf/hc-allomorph-loop-cutoff` @ `d21af37e` | Synthesis steps: Mbugwe **-8.43%**, Sena **-7.13%**, stress fixtures **-40.5% / -44.3%**. Share of total work (analysis plus synthesis): Mbugwe 0.35%, Sena 0.13%, Aweti 0.007%, stress 8.5% / 9.5% | | 33 | **E2: replay MPR features and stem names along the trail at lookup** (covers excluded MPR) | Inflection-class gates decided before synthesis | **KEEP; candidate PR, combine with E3** (same post-lookup hook). Shadow census: no success rejected. Unpushed branch `perf/hc-trail-mpr-replay` @ `4fbbfb14` (product `84f76cdf`) | Synthesis steps: Aweti **-56.9%**, Mbugwe **-11.4%**, stress **-43.7% / -88.7%**. Total work: Aweti 0.58%, Mbugwe 1.03%, stress 6.45% / 31.85%. Replay checks (set operations) are reported beside, not netted | -| 34 | **E3: replay the template and final-state outcome at lookup** | Shape-free template-path deaths decided before synthesis | **KEEP, without memoisation.** The brief's kill rule (replay states above steps saved) fired on every corpus but compares int operations with pattern matches, so the owner overrode the literal PARK. No memo: the owner caps added memory at a few MB. Next: cost the replay in comparable units and cut the walk itself (skip it when no slot is required, stop at the first violation). Unpushed branch `perf/hc-template-replay` @ `6edc4375` | Synthesis steps: Aweti **-67.9%**, Amharic **-58.3%**, Mbugwe **-12.1%**. Total work: Aweti 0.69%, Mbugwe 1.09%, Amharic 8.58%; stress fixtures unmeasured (no analysis counts). Replay states: Mbugwe 133,606 against 15,814 steps saved | +| 34 | **E3: replay the template and final-state outcome at lookup** | Shape-free template-path deaths decided before synthesis | **KEEP, without memoisation.** The brief's kill rule (replay states above steps saved) fired on every corpus but compares int operations with pattern matches, so the owner overrode the literal PARK. No memo: the owner caps added memory at a few MB. Next: cost the replay in comparable units and cut the walk itself (skip it when no slot is required, stop at the first violation). Unpushed branch `perf/hc-template-replay` @ `6edc4375` | Synthesis steps: Aweti **-67.9%**, Amharic **-58.3%**, Mbugwe **-12.1%**. Total work: Aweti 0.69%, Mbugwe 1.09%, Amharic 8.58%; stress fixtures unmeasured (no analysis counts). Replay states: Mbugwe 133,606 against 15,814 steps saved **Superseded: E3 was later dropped on cost, see row 38.** | | 35 | **E4: excluded co-occurrence during analysis** (Maxwell) | Anti-monotone; prunes analysis subtrees | **PARK.** Sound, and the only experiment that cuts analysis work, but 0% on the five real grammars. Land only as a cheap safety net. Unpushed branch `perf/hc-excluded-cooccurrence` @ `02873608` (product `b07fc4d6`) | Total work: real grammars **0%**, conformance 0.23%, brief's fixture 3.2% (kill under 25% fires), dense fixtures **31.3% / 19.8%**. Zero-width "others" are not invisible for affix-process allomorphs | | 36 | **E5: required-environment input prefilter** (Maxwell) | Linear left-then-right scan per allomorph | **DEAD END as specified.** The soundness scan disables it on 5 of 5 real grammars and 145 of 150 conformance runs. Forced on it is net negative on two of three fixtures. Revival needs digraph-safe segmentation, handling of non-uniform feature sets and over-specific synthesized nodes, and a cheaper prefilter. Unpushed branch `perf/hc-env-input-prefilter` @ `9d55dade` (product at `1af79e20`) | Real grammars **0%**. Forced on: Sena -5.7% net, Aweti -0.4% net. Stress fixtures net 65.2% / -3.8% / -37.4% | | 37 | **E6: intermediate-shape check against SurfaceFormMismatch** | Stop a candidate at the first stratum boundary where it diverges | RESEARCH ONLY; not run; no soundness argument yet | SurfaceFormMismatch is 25.6% of real synthesis steps. Census only, killed by any success that fails it | +## 2026-10-27 analysis-side round + +Full record: [hermitcrab-2026-10-27-analysis-round.md](hermitcrab-2026-10-27-analysis-round.md). +It follows the synthesis-to-analysis round, whose conclusion was that the dominant cost is analysis. The unit is +analysis events plus synthesis steps from the census harness (`research/analysis-cost-census` @ `38e9622e`, master +base `c2454760`), never ms. Where a check does work of its own, the table gives the figure **net** of that work in +one unit, and says which unit. Issues #561-#568 and PRs #571, #572, #574, #575 are in sillsdev/machine. + +| # | Optimization | Expected | Outcome | Number | +| --- | --- | --- | --- | --- | +| 38 | **Earlier-round PRs: E1 (PR #560), MPR and stem-name replay (PR #570); E3 template replay dropped** (rows 32-34) | E1 and E2 were KEEP candidates; E3 was KEEP without memoisation, pending a replay cost in comparable units | **E1 and E2 became PRs #560 and #570. E3 was dropped on cost** (the owner rule forbids the memo that would have cut the replay walk). The code is retained at `perf/hc-lookup-replay-templates` @ `9efd059c` | Row 34's "KEEP" is superseded by this row. Synthesis is 1.0-9.0% of total work on real grammars, so none of the three could save more than about 1% of it | +| 39 | **R1: required-slot look-ahead in the template walk** (issue #561; **PR #575**) | Mbugwe 15-35% of events (est.), Sena 3-10%, stress fixture at least 50% | **KEEP.** Sound: 0 shadow violations on every grammar, parses equal per word. The plan's kill rule (check unifications more than 3x events saved) fired on Mbugwe at 3.19x and was **waived**: it compared unification calls with walk and trace events, which are unlike units. Replaced by net unification work in one unit. Overlaps PR #546, see row 43 | Analysis events, less work: Mbugwe **6.6%**, Sena **30.8%**, Aweti **7.6%**, Amharic **8.9%**, conformance 1.6%, Indonesian 0%, stress **74.6%**. Net in unification calls over the whole parse, check included: Sena **1.56% less**, Aweti **4.50% less**, stress **37.91% less**, Mbugwe 0.33% less and conformance 0.14% less (both break-even), Indonesian 0, **Amharic 0.63% MORE** (411,073 to 413,674 calls, 24 words) | +| 40 | **R2: root-substring prune at rule outputs, gated by static restoration flags** (issue #562; **PR #572**; reopens row 27) | Sena 8-17%, the other four real grammars 0 (flags off), stress fixture at least 40% | **KEEP, with a probe skip added.** Row 27's reopening condition (a shadow summary deciding a meaningful share) is met on Sena and the stress fixture, 0 violations. The prune is off on four of five real grammars (dropped part that can hold segments, reduplication, deeper phonology). A static gate on root count was **rejected as overfitting toy grammars** | Net of probes (1 probe = 1 event), less work: Sena **15.3%** (gross 17.7%), stress **66.5%** (gross 67.7%), Aweti, Amharic, Indonesian, Mbugwe **0 (flag off)**, conformance **15.3% MORE** (7,237 probes buy 92 events). The probe skip (keep when the output has at least as many segments as its input) removed 18 of 7,255 probes | +| 41 | **T1: lean trace, nodes keep shape text and no `Word`** (issue #563; **PR #571**; scored on memory) | Retained trace memory down 90-97% on Mbugwe yɔɔna (1,032 MiB) | **KEEP; lands after PR #534** (it rewrites the same callbacks). Opt-in `TraceManager.RetainWords = false`; default behaviour unchanged. Steps identical by construction and verified | Retained trace memory, lean vs full, sum over words: Mbugwe **93.8% less**, Aweti 93.6%, Sena 92.0%, Amharic 98.8%, Indonesian 98.7%, conformance 97.4%, stress 97.5%. yɔɔna 1,035.0 to 22.6 MiB. Parse counts, node counts, census rows and conformance verdicts identical | +| 42 | **R5: edge-segment prefilter port** (row 22's archive; issue #564; **PR #574**) | Scored 0 by the plan: a prefilter removes no step, it makes each step cheaper | **KEEP.** The plan's 0 was the wrong score: census steps do not move, and the work per step falls. The first write-up was gross and left out the prefilter's own unifications; the net table charges each one as one arc check | **Net matcher work** (traversals plus arc checks plus prefilter unifications), census, less work: Aweti **47.5%**, Mbugwe **19.6%**, Sena **19.6%**, Amharic **12.8%**, Indonesian **6.5%** (gross 6.9%), stress 77.7%, conformance 23.1%. Analysis events and synthesis steps **unchanged** on all 1,796 completed pairs per mode | +| 43 | **R1 on PR #546's shared-slot walk** (issue #566) | R1 and #546 compose; the residual after both is a ceiling only | **IN PROGRESS.** Not measured when this row was written | none yet. Reopen condition for dropping one of the two: R1 saves under 3% of slot events on the #546 walk where it saved over 5% on master | +| 44 | **R3: ordered and edge-anchored required-slot look-ahead (R1 v2)** (issue #567) | Closes the part of R1's ceiling that single-segment affixes (`-a`, `a-`) leave: Mbugwe reaches only 10% of its no-lookup ceiling | **IN PROGRESS.** Not measured when this row was written | none yet | +| 45 | **R4: subsequence fallback and insertion classes for R2** (issue #568) | Ceilings: Mbugwe 38.5%, Indonesian 29.6%, Aweti 14.8%, Amharic 13.6% | **DEAD END.** Sound, but the sound version fires on none of the four target grammars except Indonesian, where it is a net loss. No PR. Code at `perf/hc-root-subsequence-prune`, order-free tier at `71cba028` | Net (probe = 1 step, each trie arc = 1 step): Mbugwe **0**, Aweti **0**, Amharic **0**; Indonesian **182.9% MORE work** with the order-free tier (gross 30.8% less, 631 probes, 16,779 arcs); conformance 19.1% more. The ceilings are unsound: the plain subsequence test violates on **334,508** events (Mbugwe), **50,187** (Aweti), 2,264 (Amharic), 1,105 (Indonesian) | +| 46 | **R6: keep equivalent-analysis merging on while tracing** (issue #565) | Events down 0.2-2.4%; lookups under tracing down by 1.67x (Mbugwe), 2.19x (Aweti) | **NOT BUILT.** It changes how Try-a-Word shows folded duplicates, which is an owner product decision, and it is low in steps | none | + +**Lessons of the round** (detail in the round doc): + +1. **Count a check's own cost in the same unit as what it saves.** R1, R2 and R5 each first reported gross + numbers, and each changed sign or size once its own work was counted. R1's 3x kill rule was waived because it + compared unlike units. +2. **Census steps miss work that cheapens each step.** R5 scored 0 in steps and saves 6.5-47.5% of matcher + work. Pick the unit that the change acts on, and report steps beside it. +3. **The `truncate-morphotactic*` conformance fixtures are nondeterministic on master.** Their output differs + run to run with no change. Exclude them from every parity comparison, and say so. +4. **No memoization** (owner rule). It is why E3's replay was dropped rather than cached, and why shared work + beyond #546 is reported as a ceiling only. +5. **Porting a change can expose a divergence in the other implementation.** The PanGloss port of machine #548 + found and fixed a real Rust divergence in alpha-environment selection (PanGloss PR #13). +6. **A ceiling measured with the flags ignored can be mostly unsound.** R4's census ceilings were mostly + restoration violations; the sound share was zero. + --- ## The lesson that generalises From 9a796d2a74b62276f0364a6b2ac5903c658ba845 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Tue, 27 Oct 2026 22:34:37 -0400 Subject: [PATCH 32/32] Record R3 and R1-on-#546 results in the HermitCrab ledger Rows 43 and 44 move from in progress to done. R1 on #546 is draft PR #577 (commit bbb81a00, shadow 0 violations); R3 ordered look-ahead is draft PR #576 (shadow 0 violations). Adds the round-doc section for both and the PanGloss port issues (sillsdev/PanGloss #65, #66). Co-Authored-By: Claude Opus 5.5 --- docs/hermitcrab-2026-10-27-analysis-round.md | 32 +++++++++++++++----- docs/hermitcrab-optimization-ledger.md | 4 +-- 2 files changed, 27 insertions(+), 9 deletions(-) diff --git a/docs/hermitcrab-2026-10-27-analysis-round.md b/docs/hermitcrab-2026-10-27-analysis-round.md index fbcb2706f..780d36bcd 100644 --- a/docs/hermitcrab-2026-10-27-analysis-round.md +++ b/docs/hermitcrab-2026-10-27-analysis-round.md @@ -1,7 +1,7 @@ # HermitCrab analysis-side optimization, 2026-10-27 round -**Disposition: four candidates are PRs (R1 #575, R2 #572, T1 #571, R5 #574), one is a dead end (R4), one -is not built (R6), and two are in progress (R3, R1 on #546).** This round follows the +**Disposition: six candidates are PRs (R1 #575, R2 #572, T1 #571, R5 #574, R3 #576, R1 on #546 #577), one is a +dead end (R4), and one is not built (R6).** This round follows the [synthesis-to-analysis round](hermitcrab-2026-10-27-synthesis-to-analysis-round.md), whose conclusion was that synthesis is 1.0% (Aweti) to 9.0% (Mbugwe sample) of total work and the dominant cost is analysis. Here the target is the analysis side: the template battery, rule outputs that cannot reach a root, matcher work per @@ -36,8 +36,8 @@ This record follows the [optimization ledger](hermitcrab-optimization-ledger.md) | R5 edge-segment prefilter | #564 | #574 | **KEEP.** The plan's score of 0 was wrong | | R4 subsequence fallback | #568 | none | **DEAD END** | | R6 merging while tracing | #565 | none | Not built. Owner product decision | -| R3 ordered look-ahead | #567 | | In progress | -| R1 on #546 | #566 | | In progress | +| R3 ordered look-ahead | #567 | #576 | **KEEP.** Stacks on #575 | +| R1 on #546 | #566 | #577 | **KEEP.** Stacks on #546 and #575 | Earlier-round PRs: **#560** (E1, allomorph-loop cutoff) and **#570** (MPR and stem-name trail replay). E3 (template replay) was dropped on cost; its code stays at `perf/hc-lookup-replay-templates` @ `9efd059c`. @@ -336,13 +336,31 @@ censusmerge runs (`truncate-morphotactic-identity#default` word 2), and the on a - **R6, merging while tracing (#565): not built.** Census merging changes analysis events by only 0.2-2.4%, but under tracing lookups are 1.67x (Mbugwe) to 2.19x (Aweti) more frequent. Whether Try-a-Word shows folded duplicates is an owner product decision, and it is low in steps. It interacts with #534 and #563. -- **R3, ordered and edge-anchored look-ahead (#567): in progress.** It targets the 90% of Mbugwe's no-lookup - ceiling that R1 leaves, where required slots insert a single segment. Result not recorded here. -- **R1 on #546 (#566): in progress.** It measures R1 on the shared-slot walk and reports what #546 leaves. The rule +- **R3, ordered and edge-anchored look-ahead (#567): KEEP, PR #576.** It targets the 90% of Mbugwe's no-lookup + ceiling that R1 leaves, where required slots insert a single segment. Results are in the section above. +- **R1 on #546 (#566): KEEP, PR #577; results in the section above.** It measures R1 on the shared-slot walk and reports what #546 leaves. The rule set in advance: if R1 saves under 3% of slot events on the #546 walk where it saved over 5% on master, the two overlap and only one should land. Further sharing of non-trailing slots needs a key on the shape, which is a memo, so any residual is reported as a ceiling and not proposed. +## R3 and R1 on #546: KEEP, and the two stack (issues #567, #566; PRs #576, #577) + +- **R3, ordered and edge-anchored look-ahead (#567; PR #576, stacked on #575).** Within a rule, the required inserted + segments must embed in order; the check is anchored at the edge it can reach; and a passing required slot narrows the + window for the slots behind it. Shadow: 0 violations on every grammar, and parses are equal per word. Events plus + synthesis steps, less work than master: Sena **40.9% less**, Mbugwe **7.3% less**, Aweti **11.0% less**, Amharic + **11.9% less**. Unification calls, less than master: Amharic **2.75% less** (R1 alone was 0.63% MORE), Sena + **5.22% less**, Aweti **9.27% less**, Mbugwe **1.83% less**. Mbugwe gains little over R1 because most of its required + slots are not testable; the next step is an R1 purity extension with its own soundness argument. PanGloss port: + sillsdev/PanGloss #65. +- **R1 on #546 (#566; PR #577, stacked on #546 and #575, commit `bbb81a00`).** The look-ahead runs on the shared trie + walk, so a template whose required slot cannot unapply is dropped for its whole subtree. Reviewed LAND. Shadow: 0 + violations. Events plus synthesis steps, less work than master: Sena **55.64% less**, Mbugwe **32.15% less**, Aweti + **20.06% less**, Amharic **9.10% less**. Unification calls, less than master: Sena **4.81% less**, Mbugwe **15.19% + less**, Aweti **12.18% less**; Amharic **0.63% MORE**. Against #546 alone, Mbugwe **1.07% MORE** and Sena **1.40% + MORE** unification calls. The two stack in events, but on Mbugwe and Sena the check's own unifications cost more than + the walk work they save. PanGloss port: sillsdev/PanGloss #66 (for #546 and #566). + ## Cross-cutting lessons 1. **Count a check's own cost in the same unit as what it saves.** R1, R2 and R5 all first reported gross numbers. diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md index 2a2511ff5..eaddf40f9 100644 --- a/docs/hermitcrab-optimization-ledger.md +++ b/docs/hermitcrab-optimization-ledger.md @@ -172,8 +172,8 @@ one unit, and says which unit. Issues #561-#568 and PRs #571, #572, #574, #575 a | 40 | **R2: root-substring prune at rule outputs, gated by static restoration flags** (issue #562; **PR #572**; reopens row 27) | Sena 8-17%, the other four real grammars 0 (flags off), stress fixture at least 40% | **KEEP, with a probe skip added.** Row 27's reopening condition (a shadow summary deciding a meaningful share) is met on Sena and the stress fixture, 0 violations. The prune is off on four of five real grammars (dropped part that can hold segments, reduplication, deeper phonology). A static gate on root count was **rejected as overfitting toy grammars** | Net of probes (1 probe = 1 event), less work: Sena **15.3%** (gross 17.7%), stress **66.5%** (gross 67.7%), Aweti, Amharic, Indonesian, Mbugwe **0 (flag off)**, conformance **15.3% MORE** (7,237 probes buy 92 events). The probe skip (keep when the output has at least as many segments as its input) removed 18 of 7,255 probes | | 41 | **T1: lean trace, nodes keep shape text and no `Word`** (issue #563; **PR #571**; scored on memory) | Retained trace memory down 90-97% on Mbugwe yɔɔna (1,032 MiB) | **KEEP; lands after PR #534** (it rewrites the same callbacks). Opt-in `TraceManager.RetainWords = false`; default behaviour unchanged. Steps identical by construction and verified | Retained trace memory, lean vs full, sum over words: Mbugwe **93.8% less**, Aweti 93.6%, Sena 92.0%, Amharic 98.8%, Indonesian 98.7%, conformance 97.4%, stress 97.5%. yɔɔna 1,035.0 to 22.6 MiB. Parse counts, node counts, census rows and conformance verdicts identical | | 42 | **R5: edge-segment prefilter port** (row 22's archive; issue #564; **PR #574**) | Scored 0 by the plan: a prefilter removes no step, it makes each step cheaper | **KEEP.** The plan's 0 was the wrong score: census steps do not move, and the work per step falls. The first write-up was gross and left out the prefilter's own unifications; the net table charges each one as one arc check | **Net matcher work** (traversals plus arc checks plus prefilter unifications), census, less work: Aweti **47.5%**, Mbugwe **19.6%**, Sena **19.6%**, Amharic **12.8%**, Indonesian **6.5%** (gross 6.9%), stress 77.7%, conformance 23.1%. Analysis events and synthesis steps **unchanged** on all 1,796 completed pairs per mode | -| 43 | **R1 on PR #546's shared-slot walk** (issue #566) | R1 and #546 compose; the residual after both is a ceiling only | **IN PROGRESS.** Not measured when this row was written | none yet. Reopen condition for dropping one of the two: R1 saves under 3% of slot events on the #546 walk where it saved over 5% on master | -| 44 | **R3: ordered and edge-anchored required-slot look-ahead (R1 v2)** (issue #567) | Closes the part of R1's ceiling that single-segment affixes (`-a`, `a-`) leave: Mbugwe reaches only 10% of its no-lookup ceiling | **IN PROGRESS.** Not measured when this row was written | none yet | +| 43 | **R1 on PR #546's shared-slot walk** (issue #566) | R1 and #546 compose; the residual after both is a ceiling only | **KEEP, stacked on #546 and #575.** Draft PR #577 (commit `bbb81a00`, Closes #566). Reviewed LAND. Shadow: 0 violations. The two compose in events, but on Mbugwe and Sena the check's own unifications cost more than the walk they save, so #546 alone keeps fewer unification calls there. | Events plus synthesis steps vs master, less work: Mbugwe **32.15%**, Sena **55.64%**, Aweti **20.06%**, Amharic **9.10%**. Unification calls vs master: Mbugwe **15.19% less**, Sena **4.81% less**, Aweti **12.18% less**, Amharic **0.63% MORE**. Against #546 alone: Mbugwe **1.07% MORE** and Sena **1.40% MORE** unification calls. Reopen condition for dropping one of the two (unchanged): R1 saves under 3% of slot events on the #546 walk where it saved over 5% on master. PanGloss port: sillsdev/PanGloss #66 (for #546 and #566) | +| 44 | **R3: ordered and edge-anchored required-slot look-ahead (R1 v2)** (issue #567) | Closes the part of R1's ceiling that single-segment affixes (`-a`, `a-`) leave: Mbugwe reaches only 10% of its no-lookup ceiling | **KEEP. Draft PR #576, stacked on #575.** Shadow: 0 violations on every grammar, parses equal per word. Mbugwe gains little over R1 in unification calls, because most of its required slots are not testable; the next step is an R1 purity extension with its own soundness argument, not more ordering. PanGloss port: sillsdev/PanGloss #65 (for #576) | Events plus synthesis steps vs master, less work: Sena **40.9%**, Mbugwe **7.3%**, Aweti **11.0%**, Amharic **11.9%**. Unification calls vs master, less: Amharic **2.75% less** (R1 alone was 0.63% MORE), Sena **5.22% less**, Aweti **9.27% less**, Mbugwe **1.83% less** | | 45 | **R4: subsequence fallback and insertion classes for R2** (issue #568) | Ceilings: Mbugwe 38.5%, Indonesian 29.6%, Aweti 14.8%, Amharic 13.6% | **DEAD END.** Sound, but the sound version fires on none of the four target grammars except Indonesian, where it is a net loss. No PR. Code at `perf/hc-root-subsequence-prune`, order-free tier at `71cba028` | Net (probe = 1 step, each trie arc = 1 step): Mbugwe **0**, Aweti **0**, Amharic **0**; Indonesian **182.9% MORE work** with the order-free tier (gross 30.8% less, 631 probes, 16,779 arcs); conformance 19.1% more. The ceilings are unsound: the plain subsequence test violates on **334,508** events (Mbugwe), **50,187** (Aweti), 2,264 (Amharic), 1,105 (Indonesian) | | 46 | **R6: keep equivalent-analysis merging on while tracing** (issue #565) | Events down 0.2-2.4%; lookups under tracing down by 1.67x (Mbugwe), 2.19x (Aweti) | **NOT BUILT.** It changes how Try-a-Word shows folded duplicates, which is an owner product decision, and it is low in steps | none |