[mevichitra]

Summary

Adds the manual check of the harness's ground truth (the line diff of linemap.py) that the paper's threats section needs, and the tooling behind it. A seeded, stratified sample of 100 CodeGuard findings was labelled by a human rater and, independently, by an LLM agent rater. Neither saw the harness's answer or the other's labels. The disagreements were then adjudicated. The ground truth holds for 59 of 60 persisting findings but only 30 of 40 introduced ones. Its errors sit where a scheme matched an "introduced" finding, so the paper's hidden rates are upper bounds: SonarQube's corrects from 1.47% to about 0.24%.

Type of change

• Bug fix (false positive / false negative correction)
• New rule (attach rule ID from the tracker)
• Engine / runner change
• CLI change
• Docs only
• Tooling / CI (research code under research/, not part of the released package)

Checklist

• git commit -s — DCO sign-off on all commits
• pytest passes locally
• ruff check research/ and ruff format --check research/ clean
• MYPYPATH=src mypy --strict research/finding_identity clean
• News fragment: not needed for research/

What changed

Tooling

• gt_sample.py also writes items.json: the sampled findings shuffled under neutral ids, without the harness's answer, the stratum or any scheme's outcome. sample.csv gains the item key column. Introduced items carry the old code at the same place, found with the line map (corresponding_old_line).
• gt_agreement.py (new) scores raters against the key, per item kind and per stratum, with Wilson intervals, and computes agreement between two raters. --adjudicated scores the final verdicts. --results estimates each content scheme's hidden rate with the ground-truth errors removed. --human-kind limits the person's labels to one kind, and --agent can be repeated so a later round overrides an earlier one.

Data and write-up (paper/ground_truth_check/)

• The labels of both rounds, the human rater's adjudication, adjudicated.json (final verdict and source per item), agreement.md (scores), and a README.md describing the procedure.
• The procedure includes a flaw in round 1: introduced items showed no code from before the commit, so the human rater's round-1 labels on them are not used.
• claims.md adds G1–G6, marks the hidden rates as upper bounds, and notes that Bandit's own findings were not checked; outline.md's threats section says the same.

Results

	ground truth holds	Wilson 95% CI
persisting findings (pairs)	59 / 60 (98.3%)	91.1–99.7
introduced findings	30 / 40 (75.0%)	59.8–85.8
new_but_matched	10 / 20 (50.0%)	29.9–70.1
agree_new	20 / 20	83.9–100

The errors are findings that persisted while the line diff lost them: the line was edited past the similarity threshold, or moved out of a removed block. Corrected hidden rates on CodeGuard findings: sonar 1.47 → 0.24 (0.07–0.66), github 0.81 → 0.41, line_text 1.17 → 0.87, codeguard 1.87 → 1.31.

Three sampled complexity and nesting findings (F036, F065, F076) sit on an unchanged def line and both raters called them "existed". Re-running the rule on both versions shows each function was not reported before the commit pushed it over the limit, so they are new. They are recorded in adjudicated.json with the re-run as the source.

Test evidence

$ python -m pytest research/finding_identity/tests -q
135 passed, 1 skipped          # the skip is a live Semgrep test
$ ruff check research/ && ruff format --check research/
All checks passed!
$ MYPYPATH=src python -m mypy --strict research/finding_identity
Success: no issues found in 40 source files

New tests cover Cohen's kappa against hand-computed values, the verdict mapping for both item kinds, loading labels from a file and from the labelling page's exported documents, per-stratum scoring, the corresponding-line lookup, and the hidden-rate correction.

Notes for reviewer

• The LLM rater read only its blinded copy of the items, by instruction; that is not enforced by the system.
• The 27 introduced findings the LLM rater confirmed as new had no human check, and Bandit's own findings were not sampled. Both are stated in the README and claims.md.
• Cohen's kappa between the raters is 0.00 because the human rater confirmed every pair, so chance agreement is total. Raw agreement (59 of 60) is the meaningful figure.

---

Generated by Claude Code