Repository navigation
Match LIFT list values ignoring surrounding whitespace - #1200
Draft
imnasnainaec wants to merge 4 commits into
Draft
imnasnainaec wants to merge 4 commits into
imnasnainaec wants to merge 4 commits into
Conversation
LIFT import matched list items, such as semantic domains, only on exact text. Semantic domain lists from FieldWorks templates before 2017 have trailing spaces in some English names (e.g., "Necessary "), so a LIFT value of "9.4.2.3 Necessary" from another program created a duplicate domain instead of linking to the existing one. Also key possibility lookups on names and abbreviations trimmed of surrounding whitespace, and fall back to the trimmed value when looking up a trait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1200 +/- ##
==========================================
+ Coverage 39.88% 39.90% +0.01%
==========================================
Files 1542 1542
Lines 355361 355378 +17
Branches 41320 41326 +6
==========================================
+ Hits 141753 141807 +54
+ Misses 184253 184222 -31
+ Partials 29355 29349 -6
🚀 New features to boost your workflow:
|
Storing trimmed aliases in the possibility maps let an alias claim a key before an item whose name matches exactly, and items created during import got no alias, so whitespace variants within one file still created duplicates. Keep the maps as built from the project, and only when the exact and lowercase lookups miss, scan for a key that matches once trimmed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolving the old entry-type trait looked up complex form and variant types by exact or lowercase text only, so an existing variant type "Foo " was missed and a new complex form type "Foo" was created. Use the same lookup as the other LIFT trait values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolving an old entry-type value checked complex form types first, including a trimmed match, so a complex form type "Foo " beat a variant type that matched "Foo" exactly. Try exact matches on both lists before trimmed matches on either; complex form types still win at equal strength. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Semantic domain lists from FieldWorks templates before 2017 have trailing spaces in 16 English names (
"Necessary ","Imperative ","Animism ", most of 9.6.x, ...). Projects created then still have them. A LIFT file from another program (here, The Combine; see sillsdev/TheCombine#4410) writes the trimmed9.4.2.3 Necessary. That doesn't match the key9.4.2.3 Necessarythat FLEx builds from the project, so import adds a duplicate domain9.4.2.3 Necessary - 9.4.2.3 Necessary.The possibility maps are unchanged, so an exact match always wins over a trimmed one. The scan also finds items created earlier in the same import, so whitespace variants within one file don't create duplicates. The scan runs only on a miss, which happens about once per distinct new value, since the created item is then in the map.
Evidence
New tests in
LiftMergerTests:Before:
ImportSemanticDomainTrait_MatchesExistingDomainWithUntrimmedNameon main:Should not create a new semantic domain. Expected: property Count equal to 1 But was: 2ImportOldEntryTypeTrait_MatchesExistingVariantTypeWithUntrimmedNamewithout trimmed entry-type lookup:Expected: 0 But was: 1(complex form reference instead of variant)ImportOldEntryTypeTrait_PrefersExactVariantTypeOverTrimmedComplexFormTypewith trimmed lookup but complex form types checked first:Expected: 0 But was: 1After: all of
LiftMergerTests: 69 passed, 0 failed, 2 skipped.Merge Danger
Door: two-way
Import-time matching only. No model or data migration.
Blast Radius: LIFT import
The trimmed fallback applies to every list whose LIFT traits go through
TryGetPossibilityMatchingTrait: semantic domains, parts of speech, complex form and variant types (including the oldentry-typetrait), anthropology codes, dialects, status, domain, sense, usage and translation types, locations, languages and publication types. A value that differs from an existing item only by surrounding whitespace, and has no exact match, now links to that item instead of creating a new one.Out of scope: reversal parts of speech and lexical relation types use their own lookups and still match without trimming.
Considered and rejected
ProcessSemanticDomainmatches range elements only by guid. A range element with an unknown guid creates a new domain whatever its label says, trimmed or not. That's a separate, pre-existing behavior and isn't changed here.🤖 Generated with Claude Code
This change is