Repository navigation
Add allMatches to Traverse function to improve performance - #511
jtmaxwell3 wants to merge 25 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## fix-ResultCompare #511 +/- ##
=====================================================
- Coverage 74.35% 74.30% -0.06%
=====================================================
Files 456 456
Lines 38283 38512 +229
Branches 5245 5297 +52
=====================================================
+ Hits 28467 28615 +148
- Misses 8666 8732 +66
- Partials 1150 1165 +15 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
The optimization is real and worth having, but the The mechanism you found reproduces, and is worse than you reported. It isn't the state count: The problem. A differential fuzz — 20,000 random pattern/input pairs, fixed seed, identical generator compiled against both
1. Variable bindings (108 cases). 2. Capture registers (4 cases), on the deterministic path. All four are variable-free, verified to compile with So "deterministic method only" would not be a sufficient guard; group-bearing patterns diverge there too.
A narrowing that keeps the win. I prototyped and tested one: // DeterministicFsaTraversalMethod
bool dedupeByState = !allMatches && Fst.GroupNames.Count() <= 1;with Where this could go further. The rewrite-rule environment matchers look like the best target. One caveat on the measurements. I could not reproduce the 83s → 3s end to end. On the Aweti grammar I have ( Happy to open a PR with the narrowed gate, the differential fuzz harness, and a regression test for the lost-match case if that is useful. Separately, the fuzz also found 19 cases where 🤖 Analysis performed with Claude Code |
PR #511 proposes skipping traversal instances whose (State, AnnotationIndex) was already pushed. The motivating pathology is real and reproduces on a 2-state fsa: Advance forks per Optional annotation, so instances are exponential in optional count and independent of the state count. The key is too coarse. A 20,000-case differential fuzz finds 112 cases where Match() changes, 108 from ignored variable bindings and 4 from shortened ranges on the deterministic path, with AllMatches().First() identical in all 20,000 as the control. A narrowed gate -- deterministic method, no capture groups -- measures 0 divergences with the full bound preserved. Two censuses close the extensions: environment matchers are 0.20%/1.18% of traversal instances on Amharic/Mbugwe, and the analysis rules' apparent 93.95% collapse is 99.7% alternate captures, because AnalysisAffixProcessRule and AnalysisCompoundingRule set AllSubmatches and enumerate morph boundaries on purpose. That makes a fourth row for the apparent-vs-sound table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Rebuilt on the stack you named — master + All five of our grammars are partial, so #491's prune is inert by default on every one of them.
That is presumably not true of whatever you measured on, and it would explain why the stack did nothing for us. Forcing the guard on does change the behaviour, just not enough. With Measurements, all on Aweti source-list index 182,
None produced a single parse, so there is nothing to compare for either speed or parity, and I can't yet tell you whether the narrowed gate would cost you your win. Three things would let me reproduce it rather than keep guessing:
Worth noting for its own sake: when I instrumented the word on plain master, 🤖 Analysis performed with Claude Code |
…516) Two hand-built cases distilled from a 20,000-case differential fuzz (TraversalDedupDifferentialFuzzTests) comparing this branch against master. Both fail here and pass on master: - NondeterministicTraversal_DedupOnVariableBindingLosesMatch: an anchored high=$v0+ match disappears entirely because two instances reach the same (State, AnnotationIndex) with different VariableBindings, and the surviving one can never complete. - DeterministicTraversal_DedupOnRegistersShortensMatch: an alternation match is shortened because two lineages converge on the same (State, AnnotationIndex) with different open-group registers, and the surviving lineage completes earlier than the correct one. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
PR #511 proposes skipping traversal instances whose (State, AnnotationIndex) was already pushed. The motivating pathology is real and reproduces on a 2-state fsa: Advance forks per Optional annotation, so instances are exponential in optional count and independent of the state count. The key is too coarse. A 20,000-case differential fuzz finds 112 cases where Match() changes, 108 from ignored variable bindings and 4 from shortened ranges on the deterministic path, with AllMatches().First() identical in all 20,000 as the control. A narrowed gate -- deterministic method, no capture groups -- measures 0 divergences with the full bound preserved. Two censuses close the extensions: environment matchers are 0.20%/1.18% of traversal instances on Amharic/Mbugwe, and the analysis rules' apparent 93.95% collapse is 99.7% alternate captures, because AnalysisAffixProcessRule and AnalysisCompoundingRule set AllSubmatches and enumerate morph boundaries on purpose. That makes a fourth row for the apparent-vs-sound table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
| } | ||
| foreach (LatticeArc latticeArc in incoming) | ||
| { | ||
| foreach (TInst source in ExpandArcInstances(latticeArc, lattice, allMatches)) |
There was a problem hiding this comment.
FYI: F6. Stack depth during extraction grows with match length. ExpandInstances and ExpandArcInstances recurse once per lattice edge, where master used an explicit stack. A long enough input to the public Matcher could throw StackOverflowException, which cannot be caught. Unverified; HermitCrab words are short.
|
|
||
| namespace SIL.Machine.Matching; | ||
|
|
||
| // Minimal, hand-built reproductions of two cases found by a differential fuzz |
There was a problem hiding this comment.
FYI: F7. The comments here break the code-comments standard that --agent-strict enforces. This header is provenance and points to TraversalDedupDifferentialFuzzTests, which is not in the tree. Line 36 still says the key is (State, AnnotationIndex) alone. The second test's name says the dedup "shortens" the match, yet the test asserts that shorter match as correct.
| ); | ||
| } | ||
| } | ||
| if (!allMatches && instances.Count > 1) |
There was a problem hiding this comment.
FYI: F8. The allMatches parameter is dead here. ExtractResults, ExpandArcInstances and ExpandInstances run only on the !allMatches path, so the parameter is always false and this check always passes. Keeping one element does not need a full sort: a single MinBy scan, as Fst.cs already uses, does the same.
There was a problem hiding this comment.
F8 still applies at e11e65e, in part. allMatches now reaches ExtractResults, so that half is addressed. Keeping one element still sorts the whole list, now with an O(n²) insertion sort. A scan for the first minimum is just as stable and runs in linear time.
This comment has been minimized.
This comment has been minimized.
| return x.State.Equals(y.State) && x.AnnotationIndex.Equals(y.AnnotationIndex); | ||
| return x.State.Equals(y.State) | ||
| && x.AnnotationIndex.Equals(y.AnnotationIndex) | ||
| && x.VariableBindings.ValueEquals(y.VariableBindings); |
There was a problem hiding this comment.
FYI: F10. Equality and hash disagree here. Line 720 still hashes VariableBindings by reference, and every key holds a fresh Clone(), so value-equal keys land in different buckets and ValueEquals never sees them. This is unreachable through Matcher, because determinized patterns set IgnoreVariables and the bindings are null. The new public VariableBindings.ValueEquals therefore has no effective caller.
| newInst.Registers | ||
| ); | ||
| if (!traversed.Contains(key)) | ||
| bool skip = false; |
There was a problem hiding this comment.
FYI: F11. bool skip = false; if (!skip) is left over from the revert. The block always runs, so this file differs from master only by the extra nesting and the unused allMatches parameter.
This comment has been minimized.
This comment has been minimized.
| ); | ||
|
|
||
| var curResults = new List<FstResult<TData, TOffset>>(); | ||
| var lattice = !HasAcceptable(Fst) ? CreateFstLattice() : null; |
There was a problem hiding this comment.
Major: F12. DeterministicTraversal_DedupOnRegistersShortensMatch now fails. dotnet test tests/SIL.Machine.Tests -c Release gets range [0,4) where the test asserts [0,1). PatternBuilder.cs:14 defaults Acceptable to match => true, so HasAcceptable is true for every Pattern.New() pattern and this test never reaches the lattice. I ran this on the merge commit b4965c3; master has changed nothing under FiniteState or Matching since.
| _cachedInstances.Enqueue(inst); | ||
| } | ||
|
|
||
| protected bool HasAcceptable(Fst<TData, TOffset> fst) |
There was a problem hiding this comment.
Minor: F13. This gate also turns the dedup off for every pattern built with Pattern.New(), because of the always-true default at PatternBuilder.cs:14. HermitCrab's XML loader builds environments with the constructor, so those keep the lattice, but RewriteSubrule's default environments and any pattern FieldWorks builds fluently do not. Does the Aweti speedup survive this commit? Unverified: Aweti not run.
| while (instStack.Count != 0) | ||
| { | ||
| DeterministicFsaTraversalInstance<TData, TOffset> inst = instStack.Pop(); | ||
| DeterministicFsaTraversalInstance<TData, TOffset> origInst = CopyInstanceAndBindings(inst); |
There was a problem hiding this comment.
FYI: F14. This copy is still made on every pop when the lattice is off, and it is never released. Only RecordedInstance and RecordFinalArc read origInst, and CopyInstance takes from _cachedInstances, so each pop drains the pool and later Advance calls allocate. Patterns with an Acceptable now do more work than on master. Copying only when lattice != null fixes it.
|
|
||
| protected bool HasAcceptable(Fst<TData, TOffset> fst) | ||
| { | ||
| foreach (State<TData, TOffset> state in fst.States) |
There was a problem hiding this comment.
FYI: F15. This scans every state's accept infos on each Traverse call, and Fst.Transduce calls Traverse once per start annotation. The answer cannot change once the Fst is frozen, so computing it once in the constructor is enough.
API and compatibility: unchanged since the last round. Findings: F1 addressed; F2 addressed; F3 open; F4 addressed; F5 addressed; F6 open, unverified; F7 open; F8 open; F9 addressed; F10 open; F11 open; F12 new; F13 new, unverified; F14 new; F15 new. Reviewed at 25a7e50 |
PR #511 proposes skipping traversal instances whose (State, AnnotationIndex) was already pushed. The motivating pathology is real and reproduces on a 2-state fsa: Advance forks per Optional annotation, so instances are exponential in optional count and independent of the state count. The key is too coarse. A 20,000-case differential fuzz finds 112 cases where Match() changes, 108 from ignored variable bindings and 4 from shortened ranges on the deterministic path, with AllMatches().First() identical in all 20,000 as the control. A narrowed gate -- deterministic method, no capture groups -- measures 0 divergences with the full bound preserved. Two censuses close the extensions: environment matchers are 0.20%/1.18% of traversal instances on Amharic/Mbugwe, and the analysis rules' apparent 93.95% collapse is 99.7% alternate captures, because AnalysisAffixProcessRule and AnalysisCompoundingRule set AllSubmatches and enumerate morph boundaries on purpose. That makes a fourth row for the apparent-vs-sound table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PR #511 proposes skipping traversal instances whose (State, AnnotationIndex) was already pushed. The motivating pathology is real and reproduces on a 2-state fsa: Advance forks per Optional annotation, so instances are exponential in optional count and independent of the state count. The key is too coarse. A 20,000-case differential fuzz finds 112 cases where Match() changes, 108 from ignored variable bindings and 4 from shortened ranges on the deterministic path, with AllMatches().First() identical in all 20,000 as the control. A narrowed gate -- deterministic method, no capture groups -- measures 0 divergences with the full bound preserved. Two censuses close the extensions: environment matchers are 0.20%/1.18% of traversal instances on Amharic/Mbugwe, and the analysis rules' apparent 93.95% collapse is 99.7% alternate captures, because AnalysisAffixProcessRule and AnalysisCompoundingRule set AllSubmatches and enumerate morph boundaries on purpose. That makes a fourth row for the apparent-vs-sound table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PR #511 proposes skipping traversal instances whose (State, AnnotationIndex) was already pushed. The motivating pathology is real and reproduces on a 2-state fsa: Advance forks per Optional annotation, so instances are exponential in optional count and independent of the state count. The key is too coarse. A 20,000-case differential fuzz finds 112 cases where Match() changes, 108 from ignored variable bindings and 4 from shortened ranges on the deterministic path, with AllMatches().First() identical in all 20,000 as the control. A narrowed gate -- deterministic method, no capture groups -- measures 0 divergences with the full bound preserved. Two censuses close the extensions: environment matchers are 0.20%/1.18% of traversal instances on Amharic/Mbugwe, and the analysis rules' apparent 93.95% collapse is 99.7% alternate captures, because AnalysisAffixProcessRule and AnalysisCompoundingRule set AllSubmatches and enumerate morph boundaries on purpose. That makes a fourth row for the apparent-vs-sound table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
I discovered when parsing wemulujaʼjawype in the Aweti project that a single call to Traverse in DeterministicFsaTraversalMethod could take 4 seconds. It was processing more than 200,000 traversals, even though the input only had 47 characters and the fsa only had 4 states. This was because the DeterministicFsaTraversalInstance data structure included the registers, which encoded the match to that point. Processing 200,000 traversals was especially annoying since the caller only wanted one match.
The registers only record the match, they don't filter the traversal in any way. So it should be possible to traverse in two passes, where the first pass finds traversals that are acceptable without recording the registers and the second pass extracts the registers for each of the successful traversals. But it turns out that the code path that was causing the performance issue only takes the first match. So I optimized for the case when the caller only wanted one match. This can be done in one pass.
To fix the performance issue, I added an allMatches parameter to Traverse, and changed the code to ignore traversals that arrived at a previously processed <State, AnnotationIndex> tuple. Since the only difference between the new traversal and the previously processed traversal is in the registers, we can just ignore the new traversal. This means that in the worst case it only takes O(N * |States|) to find a traversal. I tried returning immediately when I found the first match, but that produces the shortest match and many of the MatcherTests failed. Letting it run to the end finds matches of all lengths. Otherwise, the choice of match doesn't seem to matter.
With this change in place, parsing wemulujaʼjawype went from 83 seconds to 3 seconds and parsing 185 Aweti words went from 4 minutes to 1 minute.
This change is