[mevichitra]

Summary

Adds the finding-identity methods that real tools ship to the replay harness, and lets the replay take its findings from Bandit or Semgrep as well as CodeGuard. Each port is pinned to a tool version, cites the file and function it ports, and is checked against the original tool:

• sonar: SonarQube 26.10.0.133078 issue tracking (sonar-core issue/tracking/Tracker.java, AbstractTracker.java, BlockRecognizer.java, BlockHashSequence.java, hash/SourceLineHashesComputer.java). It is a six-pass matcher with block-hash code-move detection. Its tie-breaking depends on java.util.HashMap iteration order, which is ported too, including treeified buckets and removals.
• bandit: bandit -b baseline.json (Bandit 1.9.4, core/manager.py _compare_baseline_results, core/issue.py Issue.__eq__). Equality ignores location (it compares text, severity, CWE, confidence, file and test), and the baseline check is set membership, so once one issue is baselined any number of equal new ones stay hidden.
• semgrep: Semgrep 1.180.0 match_based_id (rule_match.py get_match_based_key / get_match_based_id / RuleMatches.add, rule.py formula_string), as emitted in JSON extra.fingerprint and SARIF fingerprints["matchBasedId/v1"]. Applies to Semgrep findings only.
• replay --analyzer codeguard|bandit|semgrep: Bandit runs in process. Semgrep runs offline with the local rules in semgrep_rules.yml; every proxy points at a closed port, and a placeholder token un-redacts the IDs. Research-only dependencies are in research/finding_identity/requirements.txt.

Multi-pass matchers plug into matching.scheme_pairs through trackers.MATCHERS. Observation gains the analyzer message and the file lines the Sonar tracker needs. replay.py and summarize.py are unchanged.

Type of change

• Bug fix (false positive / false negative correction)
• New rule (attach rule ID from the tracker)
• Engine / runner change
• CLI change
• Docs only
• Tooling / CI (research code under research/, not part of the package)

Checklist

• git commit -s on all commits
• pytest passes locally (652 passed), and python -m pytest research/finding_identity/tests (74 passed, 1 skipped)
• ruff check src/ tests/ and ruff check research/, ruff format --check research/ clean
• mypy src/ and MYPYPATH=src python -m mypy --strict research/finding_identity clean (the latter needs main's optional matplotlib installed)
• News fragment: not needed for research/

Fidelity evidence

• SonarQube: fidelity/sonar/build.sh compiles the unmodified tracker sources at the pinned tag with the dependency jars that tag declares (checked by SHA-256). SonarTrackerHarness.java runs Tracker.trackNonClosed on generated cases. Results:

  • 20,000 cases / ~253,000 issues, 0 mismatches between port and Java;
  • 12,000 HashMap key orders, 0 mismatches against Guava's ArrayListMultimap;
  • breaking the HashMap order emulation, or dropping the code-move pass, is detected.

  400 cases and 400 orders are recorded in tests/fixtures/sonar_tracker.json.

• Semgrep: the port reproduces all 361 IDs Semgrep emitted on whole-repository scans of Flask and DefectDojo (156 with a non-zero index); these are recorded in tests/fixtures/semgrep_match_based_id.json. During the replays it also reproduced all 716 emitted IDs (217 Flask + 499 DefectDojo). A live test checks that SARIF carries the same IDs as JSON.

• Bandit: a live test runs bandit, then bandit -b on the next version. The port flags exactly the issues Bandit reports as new, including hiding a newly added duplicate of a baselined B602. The in-process analyzer reports what the command line reports.

Test evidence

$ python -m pytest research/finding_identity/tests -q
74 passed, 1 skipped
$ python -m research.finding_identity.fidelity.sonar_cases --count 5000 --seed 1
5000 cases, 62799 raw issues (51369 matched): 0 mismatches
400 HashMap orders: 0 mismatches

Results per analyzer

python -m research.finding_identity replay https://github.com/pallets/flask --commits 100 and ... https://github.com/DefectDojo/django-DefectDojo --ref dev --commits 200, with each --analyzer. All runs had 0 parse failures. These tables were produced on c1c37a3 (this branch before merging main). Main's later CodeGuard change (source lines split on LF/CR only) can shift the codeguard-family columns slightly; a re-run on the merged head will follow in a comment.

flask: 100 commit pairs, 75 file changes with findings, 275 persisting findings (274 moved, 1 edited), 24 introduced

scheme       false-new %  (moved)  (edited)  hidden %  mispair %  collide %
---------------------------------------------------------------------------
line               52.00    51.82    100.00      0.00       0.36       7.04
line_text           0.36     0.00    100.00      0.00       0.00      38.12
github              0.73     0.36    100.00      0.00       0.00       7.04
codeguard           0.00     0.00      0.00      0.00       0.00      29.33
cg_no_scope         4.36     4.38      0.00     50.00       0.00      59.82
cg_no_mask          0.36     0.00    100.00      0.00       0.00      11.73
cg_raw              0.36     0.00    100.00      0.00       0.00      11.73
sonar               0.36     0.00    100.00      0.00       0.00      45.16
bandit              0.00     0.00      0.00     83.33       5.45      57.77

flask-bandit: 100 commit pairs, 107 file changes with findings, 5624 persisting findings (5567 moved, 57 edited), 126 introduced

scheme       false-new %  (moved)  (edited)  hidden %  mispair %  collide %
---------------------------------------------------------------------------
line               41.64    41.31     73.68     15.87      13.30       0.00
line_text           1.03     0.04     98.25      0.79       0.57      39.19
github              1.94     1.10     84.21      0.00       0.09       0.00
codeguard           0.53     0.09     43.86      3.97       0.07      42.44
cg_no_scope         0.91     0.52     38.60     26.19       2.04      72.33
cg_no_mask          0.55     0.05     49.12      1.59       0.00      16.78
cg_raw              1.07     0.05    100.00      0.00       0.00      15.76
sonar               0.36     0.00     35.09      4.76       0.57      43.47
bandit              0.02     0.00      1.75     88.89      25.18      98.32

flask-semgrep: 100 commit pairs, 78 file changes with findings, 188 persisting findings (186 moved, 2 edited), 4 introduced

scheme       false-new %  (moved)  (edited)  hidden %  mispair %  collide %
---------------------------------------------------------------------------
line               50.00    50.00     50.00     25.00       0.00       0.00
line_text           1.06     0.00    100.00      0.00       0.53      12.90
github              3.19     2.15    100.00      0.00       0.00       0.00
codeguard           0.00     0.00      0.00      0.00       0.00       0.00
cg_no_scope         0.00     0.00      0.00      0.00       0.00      40.55
cg_no_mask          0.00     0.00      0.00      0.00       0.00       0.00
cg_raw              1.06     0.00    100.00      0.00       0.00       0.00
sonar               0.53     0.00     50.00     25.00       0.00      22.12
bandit              0.00     0.00      0.00     25.00       0.00      40.55
semgrep             0.00     0.00      0.00     25.00       0.00       0.00

django-DefectDojo: 200 commit pairs, 104 file changes with findings, 420 persisting findings (416 moved, 4 edited), 21 introduced

scheme       false-new %  (moved)  (edited)  hidden %  mispair %  collide %
---------------------------------------------------------------------------
line               49.52    49.04    100.00      0.00       2.38       0.00
line_text           0.95     0.00    100.00      0.00       0.00       8.29
github              1.43     0.48    100.00      0.00       0.00       0.00
codeguard           0.95     0.00    100.00      0.00       0.00       8.00
cg_no_scope         0.95     0.00    100.00      0.00       0.00      19.43
cg_no_mask          0.95     0.00    100.00      0.00       0.00       3.43
cg_raw              0.95     0.00    100.00      0.00       0.00       3.43
sonar               0.95     0.00    100.00      0.00       0.00       8.86
bandit              4.29     3.85     50.00      0.00       0.00      42.29

django-DefectDojo-bandit: 200 commit pairs, 47 file changes with findings, 208 persisting findings (206 moved, 2 edited), 26 introduced

scheme       false-new %  (moved)  (edited)  hidden %  mispair %  collide %
---------------------------------------------------------------------------
line               60.10    60.19     50.00      0.00       4.81       0.00
line_text           0.96     0.00    100.00      0.00       0.00      39.85
github              1.44     0.49    100.00      0.00       0.00       0.00
codeguard           1.44     0.49    100.00      0.00       0.00      20.05
cg_no_scope         1.44     0.49    100.00      3.85       0.00      61.18
cg_no_mask          1.44     0.49    100.00      0.00       0.00      10.80
cg_raw              1.44     0.49    100.00      0.00       0.00      10.80
sonar               0.48     0.00     50.00      0.00       0.00      39.85
bandit              0.00     0.00      0.00     15.38       6.73      77.89

django-DefectDojo-semgrep: 200 commit pairs, 78 file changes with findings, 325 persisting findings (325 moved, 0 edited), 12 introduced

scheme       false-new %  (moved)  (edited)  hidden %  mispair %  collide %
---------------------------------------------------------------------------
line               61.85    61.85       n/a      0.00       2.46       0.00
line_text           0.00     0.00       n/a      0.00       0.00      19.64
github              0.00     0.00       n/a      0.00       0.00       0.00
codeguard           0.31     0.31       n/a      0.00       0.00      46.49
cg_no_scope         0.00     0.00       n/a      0.00       0.00      78.56
cg_no_mask          0.31     0.31       n/a      0.00       0.00       5.61
cg_raw              0.31     0.31       n/a      0.00       0.00       3.61
sonar               0.00     0.00       n/a      0.00       0.00      23.25
bandit              0.00     0.00       n/a      0.00       1.85      85.77
semgrep             0.31     0.31       n/a      0.00       0.00       0.00

Notes for reviewer

• Bandit's baseline hides most genuinely new findings on Flask (83–89%), and 0–15% on DefectDojo. Equality ignores location and the baseline check ignores multiplicity, so a new instance of an already-baselined issue is never reported. Its 98% collision rate on Bandit findings shows the cause.
• SonarQube's tracker is low on both kinds of error. Its false-new rate stays between 0.00% and 0.95% on every run. Its hidden rate is 0% except 4.76% on Flask/Bandit and 1 in 4 on Flask/Semgrep.
• Message-based identity breaks when messages embed locations. CodeGuard's taint and SQL-injection messages contain source line numbers ("built on line N"), so bandit (which compares the message) reports 3.85% of unchanged DefectDojo findings as new.
• Semgrep's ID has no location. Its ordinal suffix carries the identity of repeated matches, e.g. py-sql-format matches on different lines share a key.
• Small counts. Flask has few Semgrep findings under the local rules (4 introduced), so its hidden rates are 1 in 4. Both CodeGuard runs also have few introduced findings (24 and 21).
• Snyk Code (closed source) and reviewdog (diff filtering, not identity) are not included; the README says why.

[mevichitra]

Re-ran all six replays on the merged head (7f3924e, which includes main's line-splitting change). Flask was at 559a677 and DefectDojo dev at d086db8, the same as the first run. The six rate tables are identical to the ones in the description, digit for digit. All runs again had 0 parse failures, and the Semgrep port again reproduced every emitted match_based_id (217 on Flask and 499 on DefectDojo, 0 mismatches).

---

Generated by Claude Code