Skip to content

fix(general_text): keep cleaned text and count each modified row once in ReferenceRemoverRefiner - #561

Closed
cyberspace-cs wants to merge 1 commit into
OpenDCAI:mainfrom
cyberspace-cs:fix/reference-remover-refiner-discarded-result
Closed

cyberspace-cs wants to merge 1 commit into
OpenDCAI:mainfrom
cyberspace-cs:fix/reference-remover-refiner-discarded-result

Conversation

@cyberspace-cs

Copy link
Copy Markdown
Contributor

Summary

Fixes #554.

ReferenceRemoverRefiner built the cleaned text with ref_pattern.subn() and cite_pattern.subn(), but then appended the original item instead of the cleaned one:

refined_text, ref_count = ref_pattern.subn('', refined_text)
refined_text, cite_count = cite_pattern.subn('', refined_text)
...
refined_data.append(item)   # item is still the original text

So the operator never removed anything — <ref>...</ref> tags and {{cite ...}} templates stayed in the output column.

In addition, the modified-row counter was incremented twice per changed row (once inside the pattern check and again via a redundant modified flag), so the logged "Total modified items" was double the real count.

Fix

  • Append refined_text (the cleaned text) instead of the original item.
  • Remove the redundant second increment so each modified row is counted exactly once.

Behavior

For the reproducer in #554:

Input Before After
Paris is the capital of France.<ref>Smith 2020</ref> unchanged Paris is the capital of France.
`Water boils at 100 C.{{cite web url=...}}` unchanged

Summary log for two modified rows: Total modified items: 4 → Total modified items: 2.

Tests

Added test/cpu_only/test_reference_remover_refiner.py (cpu-only):

  • removes closed <ref> tags and {{cite}} templates
  • removes unclosed forms (matched by the existing patterns)
  • leaves rows without markers untouched
  • asserts each modified row is counted exactly once via the summary log

pytest -m cpu test/cpu_only/test_reference_remover_refiner.py → 4 passed.

… in ReferenceRemoverRefiner (OpenDCAI#554)

The operator built the cleaned text with ref_pattern.subn() and
cite_pattern.subn() but appended the original item instead of the
refined one, so <ref>/<ref> tags and {{cite}} templates were never
removed from the output column. Append the cleaned text instead.

The modified-row counter was also incremented twice per changed row
(once inside the pattern check and again via a redundant 'modified'
flag), doubling the reported 'Total modified items'; remove the
duplicate count.

Adds cpu-only regression tests covering closed/unclosed markers,
untouched rows and the non-doubled summary count.
@cyberspace-cs

Copy link
Copy Markdown
Contributor Author

Closing this as a duplicate — I missed #555 when I opened it. Same root cause, same fix (reassign item before appending, count each modified row once), and #555 was opened five days earlier, so it should be the one that lands.

My branch additionally covers the double-counted "Total modified items" log line and a no-substitution case; I can send those as a small follow-up to #555 if you want them. Sorry for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ReferenceRemoverRefiner leaves the text unchanged (refined result is discarded)

1 participant