Skip to content

Add allMatches to Traverse function to improve performance - #511

Draft
jtmaxwell3 wants to merge 25 commits into
fix-ResultComparefrom
add-allMatches-to-Traverse
Draft

jtmaxwell3 wants to merge 25 commits into
fix-ResultComparefrom
add-allMatches-to-Traverse

Conversation

@jtmaxwell3

@jtmaxwell3 jtmaxwell3 commented Sep 17, 2026 •

Copy link
Copy Markdown
Collaborator

I discovered when parsing wemulujaʼjawype in the Aweti project that a single call to Traverse in DeterministicFsaTraversalMethod could take 4 seconds. It was processing more than 200,000 traversals, even though the input only had 47 characters and the fsa only had 4 states. This was because the DeterministicFsaTraversalInstance data structure included the registers, which encoded the match to that point. Processing 200,000 traversals was especially annoying since the caller only wanted one match.

The registers only record the match, they don't filter the traversal in any way. So it should be possible to traverse in two passes, where the first pass finds traversals that are acceptable without recording the registers and the second pass extracts the registers for each of the successful traversals. But it turns out that the code path that was causing the performance issue only takes the first match. So I optimized for the case when the caller only wanted one match. This can be done in one pass.

To fix the performance issue, I added an allMatches parameter to Traverse, and changed the code to ignore traversals that arrived at a previously processed <State, AnnotationIndex> tuple. Since the only difference between the new traversal and the previously processed traversal is in the registers, we can just ignore the new traversal. This means that in the worst case it only takes O(N * |States|) to find a traversal. I tried returning immediately when I found the first match, but that produces the shortest match and many of the MatcherTests failed. Letting it run to the end finds matches of all lengths. Otherwise, the choice of match doesn't seem to matter.

With this change in place, parsing wemulujaʼjawype went from 83 seconds to 3 seconds and parsing 185 Aweti words went from 4 minutes to 1 minute.


This change is Reviewable

@codecov-commenter

codecov-commenter commented Sep 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 63.63636% with 84 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.30%. Comparing base (b9e7db4) to head (72d3c02).
⚠️ Report is 6 commits behind head on fix-ResultCompare.

Files with missing lines Patch % Lines
src/SIL.Machine/FiniteState/TraversalMethodBase.cs 75.28% 34 Missing and 10 partials ⚠️
src/SIL.Machine/FeatureModel/VariableBindings.cs 0.00% 25 Missing ⚠️
src/SIL.Machine/FiniteState/Fst.cs 37.50% 9 Missing and 1 partial ⚠️
...ine/FiniteState/DeterministicFsaTraversalMethod.cs 58.33% 0 Missing and 5 partials ⚠️
Additional details and impacted files
@@                  Coverage Diff                  @@
##           fix-ResultCompare     #511      +/-   ##
=====================================================
- Coverage              74.35%   74.30%   -0.06%     
=====================================================
  Files                    456      456              
  Lines                  38283    38512     +229     
  Branches                5245     5297      +52     
=====================================================
+ Hits                   28467    28615     +148     
- Misses                  8666     8732      +66     
- Partials                1150     1165      +15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@johnml1135

Copy link
Copy Markdown
Collaborator

The optimization is real and worth having, but the (State, AnnotationIndex) key drops traversal state that filters later arcs, so Match() returns different answers than master — including outright lost matches.

The mechanism you found reproduces, and is worse than you reported. It isn't the state count: TraversalMethodBase.Advance (src/SIL.Machine/FiniteState/TraversalMethodBase.cs:275-292) forks an instance for every Optional annotation at the next offset, so instances grow exponentially in the number of optional annotations and independently of |States|. On a synthetic 2-state FSA with 20 optional annotations I measured 2,097,150 instances popped (~2^(k+1)); at k≥24 it blows past a 5,000,000 budget. Your fix bounds it at ≤ N·|States| — measured ratio 0.5–1.0×, turning never-returns cases into instant. That part holds up completely.

The problem. A differential fuzz — 20,000 random pattern/input pairs, fixed seed, identical generator compiled against both master and this branch, comparing success, range, every group range and every variable binding — found:

cases=20000  divergences=112  allMatchesFirstDiff=0

AllMatches().First() is identical in all 20,000 cases, which pins the divergence to exactly the allMatches=false path this PR changes. Two classes:

1. Variable bindings (108 cases). TraversalInstance.VariableBindings is not just a record of the match — CheckInputMatch → Input.Matches unifies against it, so which branch survives decides which later arcs can match. Matcher.Compile (src/SIL.Machine/Matching/Matcher.cs:89) only determinizes when a pattern has no variables, so every variable-bearing pattern — i.e. every alpha-variable phonological rule — runs NondeterministicFsaTraversalMethod, where the new key is blindest. Minimal case:

pattern: high=$v0+      (anchored both ends, 5 annotations)
master : success=True;  range=[0,5); v0=-
this PR: success=False; range=<null>; v0=<unbound>

2. Capture registers (4 cases), on the deterministic path. All four are variable-free, verified to compile with IsDeterministic=True, and all have alternation plus capture groups. They don't merely change captures — they shorten the match range:

pattern: (g0(back=back+)|g1(high=high+back=back+))
master : range=[0,4); g1=[0,4)
this PR: range=[0,1); g0=[0,1)

So "deterministic method only" would not be a sufficient guard; group-bearing patterns diverge there too.

b348f48f does not affect this: re-running the same fuzz against da352c7a produced a byte-identical candidate output file (same SHA-256), same 112 divergences, same gate hit counts. That commit fixes instance-release bookkeeping after the skip decision, not the key itself. Both committed suites stay green on this branch (826/0/3 and 98/0/0), so the existing tests cannot see this.

A narrowing that keeps the win. I prototyped and tested one:

// DeterministicFsaTraversalMethod
bool dedupeByState = !allMatches && Fst.GroupNames.Count() <= 1;

with NondeterministicFsaTraversalMethod reverted to master (keeping the allMatches parameter only to satisfy ITraversalMethod). Results: fuzz divergences 112 → 0; the group-free microbenchmark sweep is cell-for-cell identical to this PR across all 18 cells, so the speedup is fully preserved where patterns have no capture groups; grouped patterns fall back to master's behaviour exactly; suites unchanged. (GroupNames.Count() should be hoisted out of Traverse — it is O(1) via the ICollection fast path but is evaluated once per call.)

Where this could go further. The rewrite-rule environment matchers look like the best target. RewriteRuleSpec (src/SIL.Machine.Morphology.HermitCrab/PhonologicalRules/RewriteRuleSpec.cs:83-107) reads only leftEnvMatch.Success and leftEnvMatch.VariableBindings from those matches — never the range, never a group. So for environment matching the registers are pure overhead, exactly as your description says, and the sound dedup key there is (State, AnnotationIndex, VariableBindings), with bindings droppable entirely when IgnoreVariables is set. That would extend the bound to the variable-bearing patterns this narrowing has to give up, without any capture-reconstruction pass.

One caveat on the measurements. I could not reproduce the 83s → 3s end to end. On the Aweti grammar I have (aweti-hc.xml re-exported 2026-09-16 from Jun-26 data), that word does not complete on either build — both grow to a 40 GB heap cap in ~5 minutes. Instrumenting it showed zero DeterministicFsaTraversalMethod calls out of 7.3M, i.e. the run dies during analysis and never reaches synthesis where that method is used, so we are clearly not parsing the same grammar you were. Across five grammars (Sena, Amharic, Indonesian, Mbugwe, Aweti; sampled words, parity clean) I measured 0.90×–1.01×. If you can share the exact grammar export you measured, I can re-run the comparison on it.

Happy to open a PR with the narrowed gate, the differential fuzz harness, and a regression test for the lost-match case if that is useful. Separately, the fuzz also found 19 cases where Match() throws NullReferenceException on master (e.g. back=$v0*, anchored to start) — pre-existing and unrelated to this PR; I will file that on its own.

🤖 Analysis performed with Claude Code

johnml1135 added a commit that referenced this pull request Sep 19, 2026
PR #511 proposes skipping traversal instances whose (State, AnnotationIndex)
was already pushed. The motivating pathology is real and reproduces on a
2-state fsa: Advance forks per Optional annotation, so instances are
exponential in optional count and independent of the state count.

The key is too coarse. A 20,000-case differential fuzz finds 112 cases where
Match() changes, 108 from ignored variable bindings and 4 from shortened
ranges on the deterministic path, with AllMatches().First() identical in all
20,000 as the control. A narrowed gate -- deterministic method, no capture
groups -- measures 0 divergences with the full bound preserved.

Two censuses close the extensions: environment matchers are 0.20%/1.18% of
traversal instances on Amharic/Mbugwe, and the analysis rules' apparent 93.95%
collapse is 99.7% alternate captures, because AnalysisAffixProcessRule and
AnalysisCompoundingRule set AllSubmatches and enumerate morph boundaries on
purpose. That makes a fourth row for the apparent-vs-sound table.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@johnml1135

Copy link
Copy Markdown
Collaborator

Rebuilt on the stack you named — master + filter-final-templates-in-analysis + change-add-to-priority-union — and the Aweti word still does not complete here. But the attempt turned up something you'll want to know about #491 independently.

All five of our grammars are partial, so #491's prune is inert by default on every one of them. Morpher.IsPartial is true for each, so AlwaysEnforceFinalTemplates is the only way the prune engages:

grammar partial morphemes
Aweti 3 (2 stems, 1 affix)
Sena 27
Mbugwe 1
Amharic 1
Indonesian 4

That is presumably not true of whatever you measured on, and it would explain why the stack did nothing for us.

Forcing the guard on does change the behaviour, just not enough. With AlwaysEnforceFinalTemplates = true, the failure mode flips from memory-bound to time-bound: without it the word hits a 40 GB heap cap in ~250 s; with it both arms plateau around 16.7 GB and are still running at 480 s. Real, reproducible, and in the direction you would want — but the word does not finish either way, so there is still at least one more factor.

Measurements, all on Aweti source-list index 182, maxDegreeOfParallelism: 1, 40 GB cap:

arm guard result
master + #491 + #494 off killed, 248 s, 40.17 GB
+ #511 (da352c7) off killed, 248 s, 40.16 GB
+ narrowed #511 off killed, 238 s, 40.08 GB
master + #491 + #494 on killed at 480 s, 16.64 GB
+ #511 on killed at 480 s, 16.73 GB

None produced a single parse, so there is nothing to compare for either speed or parity, and I can't yet tell you whether the narrowed gate would cost you your win.

Three things would let me reproduce it rather than keep guessing:

  1. The exact aweti-hc.xml you measured on. Mine was re-exported 2026-09-16 from the Jun-26 .fwdata; if yours came out of FLEx earlier or with different export settings it may be a materially different grammar. This is my main suspicion.
  2. Whether AlwaysEnforceFinalTemplates was set, given the table above.
  3. How you invoked the parse — through FLEx's parser, or Morpher.ParseWord directly, and at what parallelism.

Worth noting for its own sake: when I instrumented the word on plain master, DeterministicFsaTraversalMethod was never invoked at all — 7.3M traversal calls, all nondeterministic. The run dies in analysis and never reaches synthesis, which is where the deterministic method gets used. So on my export I am not even reaching the code path your PR optimizes, which is consistent with everything above.

🤖 Analysis performed with Claude Code

johnml1135 and others added 8 commits September 21, 2026 08:05
…516)

Two hand-built cases distilled from a 20,000-case differential fuzz
(TraversalDedupDifferentialFuzzTests) comparing this branch against
master. Both fail here and pass on master:

- NondeterministicTraversal_DedupOnVariableBindingLosesMatch: an
  anchored high=$v0+ match disappears entirely because two instances
  reach the same (State, AnnotationIndex) with different
  VariableBindings, and the surviving one can never complete.
- DeterministicTraversal_DedupOnRegistersShortensMatch: an alternation
  match is shortened because two lineages converge on the same
  (State, AnnotationIndex) with different open-group registers, and
  the surviving lineage completes earlier than the correct one.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
johnml1135 added a commit that referenced this pull request Sep 25, 2026
PR #511 proposes skipping traversal instances whose (State, AnnotationIndex)
was already pushed. The motivating pathology is real and reproduces on a
2-state fsa: Advance forks per Optional annotation, so instances are
exponential in optional count and independent of the state count.

The key is too coarse. A 20,000-case differential fuzz finds 112 cases where
Match() changes, 108 from ignored variable bindings and 4 from shortened
ranges on the deterministic path, with AllMatches().First() identical in all
20,000 as the control. A narrowed gate -- deterministic method, no capture
groups -- measures 0 divergences with the full bound preserved.

Two censuses close the extensions: environment matchers are 0.20%/1.18% of
traversal instances on Amharic/Mbugwe, and the analysis rules' apparent 93.95%
collapse is 99.7% alternate captures, because AnalysisAffixProcessRule and
AnalysisCompoundingRule set AllSubmatches and enumerate morph boundaries on
purpose. That makes a fourth row for the apparent-vs-sound table.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
}
foreach (LatticeArc latticeArc in incoming)
{
foreach (TInst source in ExpandArcInstances(latticeArc, lattice, allMatches))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F6. Stack depth during extraction grows with match length. ExpandInstances and ExpandArcInstances recurse once per lattice edge, where master used an explicit stack. A long enough input to the public Matcher could throw StackOverflowException, which cannot be caught. Unverified; HermitCrab words are short.


namespace SIL.Machine.Matching;

// Minimal, hand-built reproductions of two cases found by a differential fuzz

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F7. The comments here break the code-comments standard that --agent-strict enforces. This header is provenance and points to TraversalDedupDifferentialFuzzTests, which is not in the tree. Line 36 still says the key is (State, AnnotationIndex) alone. The second test's name says the dedup "shortens" the match, yet the test asserts that shorter match as correct.

);
}
}
if (!allMatches && instances.Count > 1)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F8. The allMatches parameter is dead here. ExtractResults, ExpandArcInstances and ExpandInstances run only on the !allMatches path, so the parameter is always false and this check always passes. Keeping one element does not need a full sort: a single MinBy scan, as Fst.cs already uses, does the same.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

F8 still applies at e11e65e, in part. allMatches now reaches ExtractResults, so that half is addressed. Keeping one element still sorts the whole list, now with an O(n²) insertion sort. A scan for the first minimum is just as stable and runs in linear time.

@claude

This comment has been minimized.

return x.State.Equals(y.State) && x.AnnotationIndex.Equals(y.AnnotationIndex);
return x.State.Equals(y.State)
&& x.AnnotationIndex.Equals(y.AnnotationIndex)
&& x.VariableBindings.ValueEquals(y.VariableBindings);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F10. Equality and hash disagree here. Line 720 still hashes VariableBindings by reference, and every key holds a fresh Clone(), so value-equal keys land in different buckets and ValueEquals never sees them. This is unreachable through Matcher, because determinized patterns set IgnoreVariables and the bindings are null. The new public VariableBindings.ValueEquals therefore has no effective caller.

newInst.Registers
);
if (!traversed.Contains(key))
bool skip = false;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F11. bool skip = false; if (!skip) is left over from the revert. The block always runs, so this file differs from master only by the extra nesting and the unused allMatches parameter.

@claude

This comment has been minimized.

@jtmaxwell3
jtmaxwell3 changed the base branch from master to fix-ResultCompare October 20, 2026 15:34
);

var curResults = new List<FstResult<TData, TOffset>>();
var lattice = !HasAcceptable(Fst) ? CreateFstLattice() : null;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major: F12. DeterministicTraversal_DedupOnRegistersShortensMatch now fails. dotnet test tests/SIL.Machine.Tests -c Release gets range [0,4) where the test asserts [0,1). PatternBuilder.cs:14 defaults Acceptable to match => true, so HasAcceptable is true for every Pattern.New() pattern and this test never reaches the lattice. I ran this on the merge commit b4965c3; master has changed nothing under FiniteState or Matching since.

_cachedInstances.Enqueue(inst);
}

protected bool HasAcceptable(Fst<TData, TOffset> fst)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: F13. This gate also turns the dedup off for every pattern built with Pattern.New(), because of the always-true default at PatternBuilder.cs:14. HermitCrab's XML loader builds environments with the constructor, so those keep the lattice, but RewriteSubrule's default environments and any pattern FieldWorks builds fluently do not. Does the Aweti speedup survive this commit? Unverified: Aweti not run.

while (instStack.Count != 0)
{
DeterministicFsaTraversalInstance<TData, TOffset> inst = instStack.Pop();
DeterministicFsaTraversalInstance<TData, TOffset> origInst = CopyInstanceAndBindings(inst);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F14. This copy is still made on every pop when the lattice is off, and it is never released. Only RecordedInstance and RecordFinalArc read origInst, and CopyInstance takes from _cachedInstances, so each pop drains the pool and later Advance calls allocate. Patterns with an Acceptable now do more work than on master. Copying only when lattice != null fixes it.


protected bool HasAcceptable(Fst<TData, TOffset> fst)
{
foreach (State<TData, TOffset> state in fst.States)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: F15. This scans every state's accept infos on each Traverse call, and Fst.Transduce calls Traverse once per start annotation. The answer cannot change once the Fst is frozen, so computing it once in the constructor is enough.

@claude

claude Bot commented Oct 20, 2026

Copy link
Copy Markdown
Contributor
  1. Verdict: request changes. 25a7e50 fixes F1 and F9 by skipping the lattice for patterns with an Acceptable. But that gate also catches every Pattern.New() pattern, and the PR's own test now fails.
  2. Most important: F12 at src/SIL.Machine/FiniteState/DeterministicFsaTraversalMethod.cs:37. PatternBuilder.cs:14 gives fluent patterns an always-true Acceptable, so DeterministicTraversal_DedupOnRegistersShortensMatch skips the lattice and fails.
  3. Counts (open): Critical 1 (F12), Important 2 (F3, F13), Low 7 (F6, F7, F8, F10, F11, F14, F15).
  4. What I ran: dotnet test Machine.sln -c Release on the merge commit b4965c3. Machine had 888 passed, 1 failed, 3 skipped; the failure was DeterministicTraversal_DedupOnRegistersShortensMatch, and it failed again on a rerun. HermitCrab 108/0, Thot 93/0, SentencePiece 4/0. This environment blocked ./local_check.sh, dotnet csharpier check, and checking out other commits. CI on 25a7e50 was still running when I posted this.
  5. Not verified: the failure at 25a7e50 itself rather than the merge commit, though master has changed nothing under FiniteState or Matching since. Whether the Aweti speedup survives the gate (F13). F3 for a constructor-built pattern.

API and compatibility: unchanged since the last round. VariableBindings (public) adds IValueEquatable<VariableBindings> and ValueEquals. Matcher.Match/AllMatches behavior still changes for determinized, constructor-built patterns without an Acceptable (F3). No target framework or package changes. machine.py parity: none verified.

Findings: F1 addressed; F2 addressed; F3 open; F4 addressed; F5 addressed; F6 open, unverified; F7 open; F8 open; F9 addressed; F10 open; F11 open; F12 new; F13 new, unverified; F14 new; F15 new.

Reviewed at 25a7e50

johnml1135 added a commit that referenced this pull request Oct 22, 2026
PR #511 proposes skipping traversal instances whose (State, AnnotationIndex)
was already pushed. The motivating pathology is real and reproduces on a
2-state fsa: Advance forks per Optional annotation, so instances are
exponential in optional count and independent of the state count.

The key is too coarse. A 20,000-case differential fuzz finds 112 cases where
Match() changes, 108 from ignored variable bindings and 4 from shortened
ranges on the deterministic path, with AllMatches().First() identical in all
20,000 as the control. A narrowed gate -- deterministic method, no capture
groups -- measures 0 divergences with the full bound preserved.

Two censuses close the extensions: environment matchers are 0.20%/1.18% of
traversal instances on Amharic/Mbugwe, and the analysis rules' apparent 93.95%
collapse is 99.7% alternate captures, because AnalysisAffixProcessRule and
AnalysisCompoundingRule set AllSubmatches and enumerate morph boundaries on
purpose. That makes a fourth row for the apparent-vs-sound table.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
johnml1135 added a commit that referenced this pull request Oct 23, 2026
PR #511 proposes skipping traversal instances whose (State, AnnotationIndex)
was already pushed. The motivating pathology is real and reproduces on a
2-state fsa: Advance forks per Optional annotation, so instances are
exponential in optional count and independent of the state count.

The key is too coarse. A 20,000-case differential fuzz finds 112 cases where
Match() changes, 108 from ignored variable bindings and 4 from shortened
ranges on the deterministic path, with AllMatches().First() identical in all
20,000 as the control. A narrowed gate -- deterministic method, no capture
groups -- measures 0 divergences with the full bound preserved.

Two censuses close the extensions: environment matchers are 0.20%/1.18% of
traversal instances on Amharic/Mbugwe, and the analysis rules' apparent 93.95%
collapse is 99.7% alternate captures, because AnalysisAffixProcessRule and
AnalysisCompoundingRule set AllSubmatches and enumerate morph boundaries on
purpose. That makes a fourth row for the apparent-vs-sound table.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
johnml1135 added a commit that referenced this pull request Oct 26, 2026
PR #511 proposes skipping traversal instances whose (State, AnnotationIndex)
was already pushed. The motivating pathology is real and reproduces on a
2-state fsa: Advance forks per Optional annotation, so instances are
exponential in optional count and independent of the state count.

The key is too coarse. A 20,000-case differential fuzz finds 112 cases where
Match() changes, 108 from ignored variable bindings and 4 from shortened
ranges on the deterministic path, with AllMatches().First() identical in all
20,000 as the control. A narrowed gate -- deterministic method, no capture
groups -- measures 0 divergences with the full bound preserved.

Two censuses close the extensions: environment matchers are 0.20%/1.18% of
traversal instances on Amharic/Mbugwe, and the analysis rules' apparent 93.95%
collapse is 99.7% alternate captures, because AnalysisAffixProcessRule and
AnalysisCompoundingRule set AllSubmatches and enumerate morph boundaries on
purpose. That makes a fourth row for the apparent-vs-sound table.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants