Repository navigation
Restore analysis of literal rewrites without phonological features - #538
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #538 +/- ##
==========================================
+ Coverage 74.38% 74.43% +0.04%
==========================================
Files 456 456
Lines 38343 38360 +17
Branches 5264 5267 +3
==========================================
+ Hits 28522 28552 +30
+ Misses 8665 8643 -22
- Partials 1156 1165 +9 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This comment has been minimized.
This comment has been minimized.
Ledger 071, its index row and the staged fixture now point at sillsdev/machine#538. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a57d923 to
960a710
Compare
…ut pruning Depends on #538's literal-rewrite unapplication (fix/hc-strrep-unapplication). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
…nstead of sharing one instance Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ous application modes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rewrittenForms covering a {t,k} -> d StrRep lhs in both application modes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
960a710 to
ebd3a38
Compare
…ut pruning Depends on #538's literal-rewrite unapplication (fix/hc-strrep-unapplication). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Verdict: approve. F3 is fixed. Since the last round the PR was rebased onto
Public API: adds History was rewritten since Findings: F1 addressed (68cd30e), F2 addressed (960a710), F3 addressed (ebd3a38). Reviewed at ebd3a38 |
* Restore analysis of literal rewrites without phonological features
* Shorten a pre-existing comment rejected by the hygiene gate
* Pin analysis of kad and rejection of kat after a literal rewrite
* Give each analysis rewrite its own copy of the literal StrRep value instead of sharing one instance
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Run the literal-rewrite analysis test in both Iterative and Simultaneous application modes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Add MultiSpellingLiteralRewriteAnalyzesEachSourceSpellingAndRejectsUnrewrittenForms covering a {t,k} -> d StrRep lhs in both application modes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Prune reduplication matches whose copies cannot agree
When unapplying an affix process rule that copies a part more than once,
Morpher.PruneDisagreeingCopies (off by default) skips matches whose copies
cannot unify segment by segment. Synthesis writes every copy from the same
input, so such a match never survives synthesis. Copies containing an
optional node or modified by the rule are always kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Pin copy pruning against a cap-limited deletion in one copy
A self-feeding deletion strips two segments from the second copy only;
with DeletionReapplications 0 analysis restores one. Forward generation
defines the expected parse, and pruning must return the unpruned result.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Turn copy-agreement pruning on by default
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Compute HasRepeatedParts once per copy-agreement rule
Rules that copy no part twice no longer rescan their captured parts on every
unapplication. Mirrors PanGloss 77aca87f, which measured 0.62-0.64x of the
prior time on Aweti with identical analyses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Address review of copy pruning
- Prune compounding rules as well as affix-process rules.
- Compare every pair of copies, since unifiability is not transitive.
- Skip pruning while tracing, as MergeEquivalentAnalyses does, so traces
still show each doomed split failing in synthesis.
- Collapse the SkippedOptionalNodes forwarder into a protected internal
static GetSkippedOptionalNodes; split HasDisagreeingCopies into helpers.
- Rename to DisagreeingCopiesPruningRule / PruneDisagreeingCopiesTests.
- Add tests: realizational and compounding paths, three copies,
parallelism and memo settings, partial morphemes with and without
AlwaysEnforceFinalTemplates, DeletionReapplications = 2, tracing, and a
counterbleeding opacity case pinned against forward generation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Prune disagreeing copies through Pattern.Acceptable, also while tracing
The copy check is now the analysis rule spec's Pattern.Acceptable, so the
three analysis rules use MultiplePatternRule unchanged and
DisagreeingCopiesPruningRule is deleted. Affix-process specs set the
predicate only when a part is copied, so other rules pay nothing; the
compounding spec combines it with its existing head-capture check.
Tracing no longer turns pruning off: a pruned match is a pattern that did
not match, which a trace already leaves out.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Line up copies across unapplied deletions instead of keeping them
A copy containing an optional segment was always kept. Now copies are
compared by alignment: an optional segment may stand for any number of
segments of the other copy it unifies with, including none, and the copies
disagree only if no alignment exists. A single optional node can stand for
several segments because DeletionReapplications (0 by default) lets
analysis restore fewer segments than a deletion removed; matching it to at
most one segment pruned the valid parse in
CapLimitedDeletionInOneCopyParsesTheSameWithAndWithoutPruning.
Copies without optional nodes keep the linear comparison. Optional nodes
that are not segments still make a copy undecidable.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Restore the public and protected signatures of the analysis rule specs and wire the morpher through an internal property
The pruning change had altered the protected constructor of AnalysisMorphologicalTransformRuleSpec, added an optional morpher parameter to the public AnalysisAffixProcessAllomorphRuleSpec and AnalysisCompoundingSubruleRuleSpec constructors, and turned GetSkippedOptionalNodes into a protected internal static. Those are back to their master form; the rules now set an internal PruningMorpher property and the pruning code calls an internal static CollectSkippedOptionalNodes.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Pin reduplication pruning for optional non-segments, word-initial optional segments and empty copies with exact expected analyses
Each new test asserts the exact analysis forms produced with pruning off and on. WordInitialOptionalNonSegmentIsNeverPruned fails if the skipped word-initial nodes are dropped, OptionalNonSegmentInACopyIsNeverPruned fails if the optional non-segment bailout is removed, and EmptyCopyIsNeverPruned covers the unsuccessful-capture bailout; with the existing tests, TryGetSegmentCopies is now fully line and branch covered.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Decide once in the transform base whether a rule repeats a part and install the copy-agreement predicate only then
The base now records HasRepeatedParts once at construction and exposes CopyAgreementCheck, which is null when nothing repeats. The affix spec and the compounding spec both consume it, so the per-spec LINQ re-evaluation and the third check inside CopiesMayAgree are gone; behaviour is unchanged. This also drops a stray byte-order mark the earlier commit had added to AnalysisCompoundingSubruleRuleSpec.cs.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Pin that a literal rewrite of one copy parses the same with and without pruning
Depends on #538's literal-rewrite unapplication (fix/hc-strrep-unapplication).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Reword the StrRep-only rewrite test comment to say analysis must unapply the rewrite for pruning to keep the parse
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Skip the copy-agreement predicate for a part that is only repeated because the rule also modifies it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Build the list of copy-checked parts once in the transform constructor
HasRepeatedParts and HasDisagreeingCopies each restated which parts are
compared (at least two copies, not modified). Both now read one list,
so the predicate's installation and its loop cannot disagree again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Quick summary
A literal phonological rule such as
m -> p / x_now unapplies during analysis when the phonemes carry no phonological features, soxpuma(the synthesis ofxmuma) parses again instead of returning nothing.The cause was the nonvacuous-unapplication guard in
FeatureAnalysisRewriteRuleSpec: it looked only at symbolic features, and in a featureless table the only thing a literal rule changes isStrRep, a string feature.Feature-defined segments keep their existing path; synthesis is untouched.
Where to look
FeatureAnalysisRewriteRuleSpec: the inverse now restores the inputStrRep(it previously kept the output spelling), and a changed string feature counts as nonvacuous. Pinned byStrRepRewriteRuleTests.LiteralRewriteWithoutFeaturesAnalyzesItsSynthesisAndRejectsVacuousUnapplication(xpuma->xmuma;muma,xumakept;xmuma,pumarejected; a second unapplication on the same word is vacuous).t -> d / V_withV={a}synthesizeskataskad.LiteralRewriteAfterSegmentClassAnalyzesKadAndRejectsKatpins thatkadanalyzes tokatand thatkatand five neighbours are rejected.StringFeatureValue.IsSupersetOf(StringFeatureValue): a public wrapper over the existing protected superset check, mirroringSymbolicFeatureValue.Deliberately not included
./local_check.sh --agent-strictstopped at SentencePiece's native build after hygiene, restore and formatting passed. CI covers it.721c6043shortens one pre-existing comment inScrVersExtensions.csthat the comment-hygiene gate rejects onmaster. It is a separate commit with no behaviour change; drop it if it is unwanted here.Validation
dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests-- 108 passed, 0 failed.dotnet test ... --filter FullyQualifiedName~StrRepRewriteRuleTestswith the production fix reverted -- both tests fail; restored, both pass.dotnet build src/SIL.Machine.Morphology.HermitCrab -c Releaseandsrc/SIL.Machine.Morphology.HermitCrab.Tool -c Release-- 0 warnings, 0 errors.dotnet csharpier check .-- clean.pwsh -NoProfile -File scripts/comment-hygiene.ps1-- clean../local_check.sh --agent-strict-- not completed: CMake missing (see above).Issue / porting context
Found while measuring FieldWorks projects with an underdefined phonology through HCLoader and HermitCrab (context in #537). In a FieldWorks project whose phonemes have no features, any literal-phoneme rewrite rule loses every analysis that needs it to unapply. PanGloss (the Rust port) carries the same fix and pins it with an oracle-diffed fixture.
How it was found
A synthetic FieldWorks project defined
m -> p / x_with featureless phonemes. Exported through HCLoader 9.3.10 and run on Machine 3.8.2,xpumahad no analysis althoughxmumasynthesizes to it. A probe that replaced only the target pattern'sAcceptablepredicate withtruemade the target match and the existingx_environment accept, which isolated the guard. Live HCLoader and the exported XML agreed on every word, so the loader is not involved.🤖 Generated with Claude Code
This change is