[mevichitra]

Summary

Real commit histories rarely edit the flagged line and introduce few new findings, so the replay harness cannot say which kinds of change break which identity schemes (RQ2). This PR adds a controlled experiment. research/finding_identity/transforms.py rewrites one Python file around one finding and returns the new text plus an exact old-line → new-line map, which is the ground truth by construction. A new transform subcommand samples files with findings, applies every transformation, scans before and after, and writes an invariance matrix (transformation × scheme, false-new and hidden rates) as CSV and Markdown, with an optional matplotlib heatmap. Research code only; nothing in the codeguard package changes.

Type of change

• Bug fix (false positive / false negative correction)
• New rule (attach rule ID from the tracker)
• Engine / runner change
• CLI change
• Docs only
• Tooling / CI (research tooling under research/)

Checklist

• git commit -s — DCO sign-off on all commits
• pytest passes locally (652 passed)
• ruff check src/ tests/ clean; ruff check research/ and ruff format --check research/ clean
• mypy src/ clean; MYPYPATH=src mypy --strict research/finding_identity clean
• If adding a rule: n/a
• News fragment: n/a (changes under research/ need none)

Transformations

Transformation	What it does	Behaviour
reformat	black at line length 79 (ruff format if black is missing); the line map pairs the syntax trees before and after, so a finding is followed into a split line	kept
insert_top	a comment and an unused import below the docstring and __future__ imports	kept
insert_below	a comment line directly below the flagged statement (targets GitHub's hash, which reads 100 non-whitespace characters forward)	kept
rename_local	renames a non-parameter local used on the flagged line, across its function (libcst scope analysis)	kept
rename_function	renames the enclosing function or method and its references in the file	kept in the file
move_function	swaps the enclosing function with an adjacent function in the same body	kept
change_literal	_v2 appended to a string literal on the line, or a number incremented	changed on purpose
extract_helper	moves the flagged statement into a new module-level helper; locals it reads become parameters, locals it assigns are returned	kept
duplicate_above / duplicate_below	copies the flagged statement directly above or below itself (a genuinely new finding)	changed on purpose
ai_refactor	optional: asks a language model to refactor the enclosing function; model and key come from --ai-model/FINDING_IDENTITY_AI_MODEL and --ai-api-key/ANTHROPIC_API_KEY; replies cached in .cache/ai/; ground truth is the linemap.py diff and every case is flagged for manual review	checked by hand

Invariance matrix

Command: python -m research.finding_identity transform <repo> --samples 200 --heatmap, seed 0, all rule categories, CodeGuard 2.2.0.

• Each cell is the false-new % of the target finding (persisting finding reported as new).
• For the duplicate rows (and the one extract_helper case in DefectDojo that introduced a finding), the cell reads false-new % / hidden %. Hidden % is the share of introduced findings matched to an old finding.
• n is the number of scored cases, i.e. targets still detected after the change.

DefectDojo (dev @ 559a677, 200 files with findings)

transform	n	line	line_text	github	codeguard	cg_no_scope	cg_no_mask	cg_raw
reformat	199	76.4	7.0	14.6	7.0	7.0	7.0	7.0
insert_top	200	97.0	0.0	0.0	0.0	0.0	0.0	0.0
insert_below	200	0.0	0.0	88.5	0.0	0.0	0.0	0.0
rename_local	76	0.0	100.0	100.0	100.0	100.0	100.0	100.0
rename_function	186	0.0	67.7	67.7	100.0	67.7	100.0	100.0
move_function	169	97.6	0.0	4.1	0.0	0.0	0.0	0.0
change_literal	71	0.0	100.0	98.6	0.0	0.0	88.7	100.0
extract_helper	15	100.0 / 0.0	86.7 / 0.0	86.7 / 0.0	100.0 / 0.0	0.0 / 0.0	100.0 / 0.0	100.0 / 0.0
duplicate_above	75	94.7 / 100.0	96.0 / 100.0	8.0 / 7.9	84.0 / 100.0	82.7 / 100.0	94.7 / 100.0	97.3 / 100.0
duplicate_below	75	0.0 / 5.3	0.0 / 3.9	92.0 / 92.1	0.0 / 15.8	0.0 / 17.1	0.0 / 5.3	0.0 / 2.6

Flask (main @ d086db8, all 12 files with findings)

transform	n	line	line_text	github	codeguard	cg_no_scope	cg_no_mask	cg_raw
reformat	12	83.3	0.0	0.0	0.0	0.0	0.0	0.0
insert_top	12	100.0	0.0	0.0	0.0	0.0	0.0	0.0
insert_below	12	0.0	0.0	66.7	0.0	0.0	0.0	0.0
rename_local	3	0.0	100.0	100.0	100.0	100.0	100.0	100.0
rename_function	10	0.0	40.0	40.0	100.0	40.0	100.0	100.0
move_function	12	91.7	0.0	8.3	0.0	0.0	0.0	0.0
change_literal	7	0.0	100.0	100.0	0.0	0.0	100.0	100.0
extract_helper	7	100.0	42.9	100.0	100.0	0.0	100.0	100.0
duplicate_above	7	100.0 / 100.0	100.0 / 100.0	0.0 / 0.0	85.7 / 100.0	85.7 / 100.0	100.0 / 100.0	100.0 / 100.0
duplicate_below	7	0.0 / 0.0	0.0 / 0.0	100.0 / 100.0	0.0 / 11.1	0.0 / 11.1	0.0 / 0.0	0.0 / 0.0

What the matrix says

• Each scheme has its own blind spot. line breaks on every move (insertions above, moved functions, reformatting). github breaks on edits after the flagged line (insert_below: 88.5%). codeguard breaks on renames of the enclosing function or of a local on the line, and on extraction. Only cg_no_scope survives rename_function and extract_helper.
• Literal masking does its job. codeguard and cg_no_scope keep every change_literal case, while line_text, github and cg_raw lose almost all of them.
• Hidden findings depend on where the copy lands. Schemes pair equal keys in line order, so a copy above the finding is matched to the old finding by every scheme except github (100% hidden), and the original is reported as new. GitHub's forward-reading hash fails the other way: a copy below takes over the original's hash (92% hidden). codeguard's 15.8% hidden below comes from keys that already collide with another finding in the file.
• rename_function hits text schemes only through the def line. The 67.7% for line_text/github comes from findings reported on the def line itself (CG-QUAL-003, CG-QUAL-005).
• Extraction often removes the finding. extract_helper lost the target in 145 attempts: CG-PERF-002 118, CG-PERF-005 25, plus one each of CG-SEC-001 and CG-SEC-014. Outside its loop the statement is no longer a finding. Lost targets are counted, not scored, and the next finding in the file is tried, which leaves only 15 scored cases for that row.

Test evidence

$ python -m pytest research/finding_identity/tests -q
63 passed, 1 skipped
$ python -m pytest -q
652 passed
$ ruff check research/ && ruff format --check research/
All checks passed!
21 files already formatted
$ MYPYPATH=src mypy --strict research/finding_identity
Success: no issues found in 19 source files

tests/test_transforms.py checks, for each deterministic transformation:

• the output parses and the line map is exact (moved lines keep their text; rewritten lines are the expected ones);
• behaviour is unchanged (the sample module is executed before and after) for every behaviour-preserving transformation;
• "not applicable" cases are rejected;
• ai_refactor works with a fake model: caching, offline replay, rejection of a reply that renames the function.

tests/test_experiment.py runs the whole experiment end to end on a small fixture directory (expected matrix cells), the CLI subcommand, ground-truth resolution for split lines, and the lost-target path.

Notes for reviewer

• cli.py changes by 6 additive lines, to stay mergeable with parallel work: the subcommand lives in transform_cli.py, which imports the experiment lazily, so replay and summarize don't need libcst.
• Extra dependencies (libcst, black, optional matplotlib and anthropic) are in research/finding_identity/requirements.txt, not in pyproject.toml.
• The AI refactor was not run. No API key was available in this environment, so it was not part of either matrix. It is covered by the tests with a fake model. --transforms all includes it only when a model is configured. No model identifier is written in the repository.
• Applicability is uneven. Many Flask and DefectDojo findings sit on def or except lines (complexity, parameters, swallowed exceptions). The statement-level transformations do not fit those findings, so their n is smaller. Reasons for every "n/a" are recorded in summary.json.
• Flask is a small sample. Only 12 of its files have CodeGuard findings, so its matrix uses all of them.
• reformat rarely touches the target in Flask. Flask is already formatted at 88 columns, so black at 79 columns changes few lines and seldom touches the target.
• No results or caches are committed. They stay in git-ignored folders (results/, .repos/, .cache/).