Skip to content

Restore analysis of literal rewrites without phonological features - #538

Merged
johnml1135 merged 6 commits into
masterfrom
fix/hc-strrep-unapplication
Oct 22, 2026
Merged

johnml1135 merged 6 commits into
masterfrom
fix/hc-strrep-unapplication

Conversation

@johnml1135

@johnml1135 johnml1135 commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Quick summary

A literal phonological rule such as m -> p / x_ now unapplies during analysis when the phonemes carry no phonological features, so xpuma (the synthesis of xmuma) parses again instead of returning nothing.
The cause was the nonvacuous-unapplication guard in FeatureAnalysisRewriteRuleSpec: it looked only at symbolic features, and in a featureless table the only thing a literal rule changes is StrRep, a string feature.
Feature-defined segments keep their existing path; synthesis is untouched.

Where to look

  • FeatureAnalysisRewriteRuleSpec: the inverse now restores the input StrRep (it previously kept the output spelling), and a changed string feature counts as nonvacuous. Pinned by StrRepRewriteRuleTests.LiteralRewriteWithoutFeaturesAnalyzesItsSynthesisAndRejectsVacuousUnapplication (xpuma -> xmuma; muma, xuma kept; xmuma, puma rejected; a second unapplication on the same word is vacuous).
  • A rule after a segment class: t -> d / V_ with V={a} synthesizes kat as kad. LiteralRewriteAfterSegmentClassAnalyzesKadAndRejectsKat pins that kad analyzes to kat and that kat and five neighbours are rejected.
  • StringFeatureValue.IsSupersetOf(StringFeatureValue): a public wrapper over the existing protected superset check, mirroring SymbolicFeatureValue.
  • Both tests fail with the production change removed.

Deliberately not included

  • A full local solution build: CMake is not installed on this machine, so ./local_check.sh --agent-strict stopped at SentencePiece's native build after hygiene, restore and formatting passed. CI covers it.
  • 721c6043 shortens one pre-existing comment in ScrVersExtensions.cs that the comment-hygiene gate rejects on master. It is a separate commit with no behaviour change; drop it if it is unwanted here.

Validation

  • dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests -- 108 passed, 0 failed.
  • dotnet test ... --filter FullyQualifiedName~StrRepRewriteRuleTests with the production fix reverted -- both tests fail; restored, both pass.
  • dotnet build src/SIL.Machine.Morphology.HermitCrab -c Release and src/SIL.Machine.Morphology.HermitCrab.Tool -c Release -- 0 warnings, 0 errors.
  • dotnet csharpier check . -- clean.
  • pwsh -NoProfile -File scripts/comment-hygiene.ps1 -- clean.
  • ./local_check.sh --agent-strict -- not completed: CMake missing (see above).

Issue / porting context

Found while measuring FieldWorks projects with an underdefined phonology through HCLoader and HermitCrab (context in #537). In a FieldWorks project whose phonemes have no features, any literal-phoneme rewrite rule loses every analysis that needs it to unapply. PanGloss (the Rust port) carries the same fix and pins it with an oracle-diffed fixture.


How it was found

A synthetic FieldWorks project defined m -> p / x_ with featureless phonemes. Exported through HCLoader 9.3.10 and run on Machine 3.8.2, xpuma had no analysis although xmuma synthesizes to it. A probe that replaced only the target pattern's Acceptable predicate with true made the target match and the existing x_ environment accept, which isolated the guard. Live HCLoader and the exported XML agreed on every word, so the loader is not involved.

🤖 Generated with Claude Code


This change is Reviewable

@codecov-commenter

codecov-commenter commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.11765% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 74.43%. Comparing base (8983141) to head (960a710).

Files with missing lines Patch % Lines
...honologicalRules/FeatureAnalysisRewriteRuleSpec.cs 92.85% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #538      +/-   ##
==========================================
+ Coverage   74.38%   74.43%   +0.04%     
==========================================
  Files         456      456              
  Lines       38343    38360      +17     
  Branches     5264     5267       +3     
==========================================
+ Hits        28522    28552      +30     
+ Misses       8665     8643      -22     
- Partials     1156     1165       +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@claude

This comment has been minimized.

johnml1135 added a commit to sillsdev/PanGloss that referenced this pull request Oct 9, 2026
Ledger 071, its index row and the staged fixture now point at
sillsdev/machine#538.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@johnml1135
johnml1135 force-pushed the fix/hc-strrep-unapplication branch from a57d923 to 960a710 Compare October 20, 2026 18:17
johnml1135 added a commit that referenced this pull request Oct 20, 2026
…ut pruning

Depends on #538's literal-rewrite unapplication (fix/hc-strrep-unapplication).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude

This comment has been minimized.

johnml1135 and others added 6 commits October 21, 2026 09:57
…nstead of sharing one instance

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ous application modes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rewrittenForms covering a {t,k} -> d StrRep lhs in both application modes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@johnml1135
johnml1135 force-pushed the fix/hc-strrep-unapplication branch from 960a710 to ebd3a38 Compare October 21, 2026 14:11
johnml1135 added a commit that referenced this pull request Oct 21, 2026
…ut pruning

Depends on #538's literal-rewrite unapplication (fix/hc-strrep-unapplication).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@johnml1135
johnml1135 requested a review from jtmaxwell3 October 21, 2026 14:14
@claude

claude Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Verdict: approve. F3 is fixed. Since the last round the PR was rebased onto master and gained one new test, and I found no new defect. The production change still applies only to segments defined without features.

  • Most important: nothing open. The last finding, F3 (FeatureAnalysisRewriteRuleSpec.cs:62), is now covered by StrRepRewriteRuleTests.cs:106.
  • Counts: 0 Critical, 0 Important, 0 Low new.
  • What I ran: dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests --filter FullyQualifiedName~StrRepRewriteRuleTests gave 5 passed, 0 failed. I also ran /code-review high. Its candidates repeat ones dropped in earlier rounds: the rhsAntiFSs index skew, string-feature variables, LINQ cost, the ScrVersExtensions.cs comment, and AnalysisMorphologicalTransform. For CI, gh pr checks 538 shows Build on ubuntu-22.04 passing and Build on windows-latest still pending. The advisory Comment hygiene check passes, but it does not stand in for --agent-strict, which I left to CI.
  • Not verified: whether ModifyFromInput affix-process analysis has the same featureless StrRep defect (AnalysisMorphologicalTransform.cs:33). If it does, that is a follow-up, not part of this PR.

Public API: adds StringFeatureValue.IsSupersetOf(StringFeatureValue, bool) to SIL.Machine. No target framework, package, or machine.py parity contract changes.

History was rewritten since 960a710 (the compare reports diverged), so I reviewed the whole PR again.

Findings: F1 addressed (68cd30e), F2 addressed (960a710), F3 addressed (ebd3a38).

Reviewed at ebd3a38

@johnml1135
johnml1135 merged commit 36ec7a8 into master Oct 22, 2026
12 checks passed
@johnml1135
johnml1135 deleted the fix/hc-strrep-unapplication branch October 22, 2026 17:49
johnml1135 added a commit that referenced this pull request Oct 22, 2026
* Restore analysis of literal rewrites without phonological features

* Shorten a pre-existing comment rejected by the hygiene gate

* Pin analysis of kad and rejection of kat after a literal rewrite

* Give each analysis rewrite its own copy of the literal StrRep value instead of sharing one instance

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Run the literal-rewrite analysis test in both Iterative and Simultaneous application modes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add MultiSpellingLiteralRewriteAnalyzesEachSourceSpellingAndRejectsUnrewrittenForms covering a {t,k} -> d StrRep lhs in both application modes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Prune reduplication matches whose copies cannot agree

When unapplying an affix process rule that copies a part more than once,
Morpher.PruneDisagreeingCopies (off by default) skips matches whose copies
cannot unify segment by segment. Synthesis writes every copy from the same
input, so such a match never survives synthesis. Copies containing an
optional node or modified by the rule are always kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pin copy pruning against a cap-limited deletion in one copy

A self-feeding deletion strips two segments from the second copy only;
with DeletionReapplications 0 analysis restores one. Forward generation
defines the expected parse, and pruning must return the unpruned result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Turn copy-agreement pruning on by default

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Compute HasRepeatedParts once per copy-agreement rule

Rules that copy no part twice no longer rescan their captured parts on every
unapplication. Mirrors PanGloss 77aca87f, which measured 0.62-0.64x of the
prior time on Aweti with identical analyses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address review of copy pruning

- Prune compounding rules as well as affix-process rules.
- Compare every pair of copies, since unifiability is not transitive.
- Skip pruning while tracing, as MergeEquivalentAnalyses does, so traces
  still show each doomed split failing in synthesis.
- Collapse the SkippedOptionalNodes forwarder into a protected internal
  static GetSkippedOptionalNodes; split HasDisagreeingCopies into helpers.
- Rename to DisagreeingCopiesPruningRule / PruneDisagreeingCopiesTests.
- Add tests: realizational and compounding paths, three copies,
  parallelism and memo settings, partial morphemes with and without
  AlwaysEnforceFinalTemplates, DeletionReapplications = 2, tracing, and a
  counterbleeding opacity case pinned against forward generation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Prune disagreeing copies through Pattern.Acceptable, also while tracing

The copy check is now the analysis rule spec's Pattern.Acceptable, so the
three analysis rules use MultiplePatternRule unchanged and
DisagreeingCopiesPruningRule is deleted. Affix-process specs set the
predicate only when a part is copied, so other rules pay nothing; the
compounding spec combines it with its existing head-capture check.

Tracing no longer turns pruning off: a pruned match is a pattern that did
not match, which a trace already leaves out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Line up copies across unapplied deletions instead of keeping them

A copy containing an optional segment was always kept. Now copies are
compared by alignment: an optional segment may stand for any number of
segments of the other copy it unifies with, including none, and the copies
disagree only if no alignment exists. A single optional node can stand for
several segments because DeletionReapplications (0 by default) lets
analysis restore fewer segments than a deletion removed; matching it to at
most one segment pruned the valid parse in
CapLimitedDeletionInOneCopyParsesTheSameWithAndWithoutPruning.

Copies without optional nodes keep the linear comparison. Optional nodes
that are not segments still make a copy undecidable.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Restore the public and protected signatures of the analysis rule specs and wire the morpher through an internal property

The pruning change had altered the protected constructor of AnalysisMorphologicalTransformRuleSpec, added an optional morpher parameter to the public AnalysisAffixProcessAllomorphRuleSpec and AnalysisCompoundingSubruleRuleSpec constructors, and turned GetSkippedOptionalNodes into a protected internal static. Those are back to their master form; the rules now set an internal PruningMorpher property and the pruning code calls an internal static CollectSkippedOptionalNodes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Pin reduplication pruning for optional non-segments, word-initial optional segments and empty copies with exact expected analyses

Each new test asserts the exact analysis forms produced with pruning off and on. WordInitialOptionalNonSegmentIsNeverPruned fails if the skipped word-initial nodes are dropped, OptionalNonSegmentInACopyIsNeverPruned fails if the optional non-segment bailout is removed, and EmptyCopyIsNeverPruned covers the unsuccessful-capture bailout; with the existing tests, TryGetSegmentCopies is now fully line and branch covered.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Decide once in the transform base whether a rule repeats a part and install the copy-agreement predicate only then

The base now records HasRepeatedParts once at construction and exposes CopyAgreementCheck, which is null when nothing repeats. The affix spec and the compounding spec both consume it, so the per-spec LINQ re-evaluation and the third check inside CopiesMayAgree are gone; behaviour is unchanged. This also drops a stray byte-order mark the earlier commit had added to AnalysisCompoundingSubruleRuleSpec.cs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Pin that a literal rewrite of one copy parses the same with and without pruning

Depends on #538's literal-rewrite unapplication (fix/hc-strrep-unapplication).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Reword the StrRep-only rewrite test comment to say analysis must unapply the rewrite for pruning to keep the parse

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Skip the copy-agreement predicate for a part that is only repeated because the rule also modifies it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Build the list of copy-checked parts once in the transform constructor

HasRepeatedParts and HasDisagreeingCopies each restated which parts are
compared (at least two copies, not modified). Both now read one list,
so the predicate's installation and its loop cannot disagree again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
johnml1135 added a commit that referenced this pull request Oct 22, 2026
kan+mo has no n+b sequence for prPostNasal to rewrite, and since #538/#540
the engine no longer traces the rule on that parse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
johnml1135 added a commit that referenced this pull request Oct 26, 2026
kan+mo has no n+b sequence for prPostNasal to rewrite, and since #538/#540
the engine no longer traces the rule on that parse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants