[mevichitra]

Summary

Scales the finding-identity study (stacked on #51) from a two-repository pilot to a 35-repository, security-focused, pinned corpus with proper statistics:

• Corpus (research/finding_identity/corpus.csv): 35 open-source web applications (25 Python: Django, Flask, FastAPI, Pyramid, web.py, Frappe; 10 JavaScript/TypeScript: Express, Next.js, NestJS), each with pinned SHA, branch, SPDX licence, framework and a rationale naming its security-relevant code. Chosen from 60 candidates by written criteria; the 25 rejections and their reasons are in the README.
• corpus pin / corpus check: pins rows to their branch head and re-measures the objective criteria at the pinned commit from git alone: ≥200 first-parent commits in the 730 days before it, pinned commit ≤90 days old, one OSI licence for the whole repository (no ee/ carve-outs, no BSL/SUL/Commons Clause), and the back-end language with ≥100 analysable files and more files than the other language.
• batch: replays every repository at its pinned SHA, in parallel (--jobs), one replay process per repository. Resumable: it skips a repository whose summary.json matches the pinned SHA and commit count. Failures go to batch.jsonl and per-repository logs without stopping the run.
• stats (stats.py): pooled (micro) rates with Wilson 95% CIs; a cluster bootstrap over repositories (10,000 resamples, fixed seed, paired across tables); macro rates with bootstrap CIs; per-repository median and IQR. Every table also comes restricted to security findings. Outputs are CSV, Markdown and an optional box plot (matplotlib in requirements.txt).
• Per-category totals in the harness (additive): summary.json gains by_category. Keys include the rule ID and the ground truth pairs only same-rule findings, so the categories sum exactly to the totals (tested).

schemes.py is untouched. scan.py, replay.py and cli.py get only additive hooks (new attributes and functions, an optional parameter, and a three-line subcommand registration) to keep conflicts with parallel work small.

Type of change

• Bug fix (false positive / false negative correction)
• New rule (attach rule ID from the tracker)
• Engine / runner change
• CLI change
• Docs only
• Tooling / CI (research tooling under research/, not part of the released package)

Checklist

• git commit -s — DCO sign-off on all commits
• pytest passes locally (629 passed; research/finding_identity/tests: 40 passed)
• ruff check research/ and ruff format --check research/ clean
• MYPYPATH=src mypy --strict research/finding_identity clean
• News fragment: not needed for research/

Results: full corpus, 200 commit pairs per repository

Runtime: 14 min 44 s wall clock on 4 cores with --jobs 4 (2,781 s summed over repositories, fetching included). That covers 7,000 commit pairs and 34,275 file versions scanned, with 36,232 findings (8,703 security). There were no failures.

All findings

35 repositories, 35 of them with persisting findings: 23237 persisting findings, 1974 introduced. Rates in %.

scheme	false-new (pooled)	Wilson 95% CI	cluster bootstrap 95% CI	macro (bootstrap CI)	median [IQR]
line	52.57	[51.92, 53.21]	[48.51, 57.62]	54.49 [50.96, 57.93]	54.43 [46.63, 63.08]
line_text	1.89	[1.73, 2.08]	[0.80, 3.51]	1.87 [0.93, 3.23]	0.57 [0.30, 1.77]
github	3.14	[2.92, 3.37]	[1.96, 4.86]	3.31 [2.21, 4.81]	1.85 [1.35, 3.68]
codeguard	1.79	[1.63, 1.97]	[0.72, 3.39]	1.71 [0.77, 3.07]	0.48 [0.15, 1.38]
cg_no_scope	1.82	[1.66, 2.00]	[0.75, 3.40]	1.72 [0.78, 3.07]	0.49 [0.20, 1.35]
cg_no_mask	1.82	[1.66, 2.00]	[0.74, 3.42]	1.75 [0.81, 3.10]	0.52 [0.20, 1.56]
cg_raw	1.91	[1.74, 2.09]	[0.81, 3.52]	1.88 [0.93, 3.23]	0.57 [0.29, 1.77]

scheme	false-new, moved	false-new, edited	hidden	mispair	collision
line	52.46 [48.44, 57.49]	69.72 [59.46, 79.49]	1.22 [0.63, 2.16]	2.82 [1.57, 3.93]	2.09 [1.08, 3.51]
line_text	1.29 [0.24, 2.83]	100.00 [100.00, 100.00]	1.17 [0.52, 2.21]	0.19 [0.05, 0.39]	17.44 [13.99, 21.28]
github	2.62 [1.49, 4.29]	87.32 [80.00, 93.97]	0.81 [0.22, 1.87]	0.09 [0.00, 0.22]	2.09 [1.08, 3.51]
codeguard	1.35 [0.30, 2.88]	74.65 [65.10, 83.33]	1.87 [0.92, 3.50]	0.34 [0.10, 0.68]	18.56 [12.91, 24.85]
cg_no_scope	1.37 [0.33, 2.89]	74.65 [65.10, 83.33]	2.48 [1.47, 3.98]	0.41 [0.16, 0.76]	23.64 [18.96, 28.98]
cg_no_mask	1.32 [0.27, 2.86]	83.80 [74.66, 92.42]	1.62 [0.64, 3.30]	0.24 [0.05, 0.51]	15.59 [10.91, 20.58]
cg_raw	1.30 [0.26, 2.84]	100.00 [100.00, 100.00]	1.01 [0.40, 1.97]	0.14 [0.02, 0.32]	13.10 [8.93, 17.66]

Security findings

35 repositories, 35 of them with persisting findings: 5794 persisting findings, 648 introduced. Rates in %.

scheme	false-new (pooled)	Wilson 95% CI	cluster bootstrap 95% CI	macro (bootstrap CI)	median [IQR]
line	50.71	[49.42, 51.99]	[43.08, 62.85]	55.56 [50.34, 60.97]	53.27 [43.41, 69.27]
line_text	1.86	[1.55, 2.25]	[0.46, 4.24]	1.58 [0.66, 2.91]	0.18 [0.00, 1.40]
github	3.02	[2.61, 3.49]	[1.46, 5.63]	3.21 [1.82, 4.93]	1.39 [0.17, 3.72]
codeguard	1.90	[1.58, 2.28]	[0.44, 4.30]	1.44 [0.55, 2.72]	0.00 [0.00, 1.33]
cg_no_scope	1.97	[1.64, 2.36]	[0.58, 4.25]	1.42 [0.56, 2.71]	0.00 [0.00, 1.33]
cg_no_mask	1.90	[1.58, 2.28]	[0.44, 4.29]	1.48 [0.55, 2.79]	0.00 [0.00, 1.13]
cg_raw	1.92	[1.59, 2.30]	[0.49, 4.30]	1.61 [0.67, 2.94]	0.18 [0.00, 1.21]

scheme	false-new, moved	false-new, edited	hidden	mispair	collision
line	50.60 [43.00, 62.72]	76.00 [52.94, 95.83]	0.62 [0.00, 1.65]	3.40 [0.95, 5.38]	2.99 [1.19, 5.51]
line_text	1.44 [0.13, 3.61]	100.00 [100.00, 100.00]	2.01 [0.31, 4.91]	0.67 [0.11, 1.54]	31.18 [21.75, 43.31]
github	2.67 [1.19, 5.15]	84.00 [66.67, 100.00]	1.70 [0.14, 4.47]	0.36 [0.00, 0.95]	2.99 [1.19, 5.51]
codeguard	1.63 [0.24, 3.90]	64.00 [44.44, 80.95]	3.40 [0.48, 8.07]	1.09 [0.14, 2.62]	29.55 [12.93, 49.28]
cg_no_scope	1.70 [0.37, 3.84]	64.00 [44.44, 80.95]	4.78 [1.48, 8.93]	1.28 [0.30, 2.78]	41.81 [28.43, 57.66]
cg_no_mask	1.56 [0.18, 3.81]	80.00 [60.71, 95.83]	2.78 [0.16, 7.31]	0.76 [0.00, 1.92]	22.45 [10.00, 38.44]
cg_raw	1.49 [0.16, 3.67]	100.00 [100.00, 100.00]	1.70 [0.16, 4.28]	0.52 [0.00, 1.33]	20.18 [8.65, 35.04]

Pooled = total numerator over total denominator. Brackets after a pooled rate are the cluster bootstrap 95% CI over repositories; macro = mean of per-repository rates; IQR over per-repository rates.

Per-repository spread

Box plot of per-repository false-new rates (paper/false_new_by_repo.png, symlog axis):

False-new rate per repository (%), all and security findings (all schemes in paper/per_repo.md)

repository	known (all)	line	github	codeguard	known (security)	line	github	codeguard
defectdojo	420	49.52	1.43	0.95	19	73.68	0.00	0.00
saleor	610	68.03	0.49	0.00	72	41.67	1.39	0.00
netbox	367	71.39	1.36	0.00	40	77.50	0.00	0.00
zulip	785	53.25	0.64	0.13	120	39.17	0.83	0.00
healthchecks	414	29.95	2.42	0.48	91	58.24	1.10	0.00
inventree	911	62.13	0.88	0.11	37	67.57	2.70	0.00
addons-server	554	42.96	1.62	0.36	41	56.10	0.00	0.00
kitsune	910	46.59	2.09	0.33	104	31.73	0.00	0.00
readthedocs	302	48.34	0.99	0.33	204	46.08	0.98	0.49
weblate	241	68.05	0.83	0.00	42	45.24	2.38	0.00
mealie	118	57.63	0.85	0.00	26	34.62	3.85	0.00
lnbits	2230	46.73	1.44	0.18	1708	40.57	1.41	0.12
jumpserver	355	58.59	2.25	0.56	89	71.91	1.12	1.12
warehouse	167	61.08	2.99	0.60	66	62.12	3.03	1.52
openlibrary	663	55.81	4.07	1.96	195	55.90	3.59	3.59
ckan	742	43.26	4.58	1.35	30	36.67	0.00	0.00
indico	284	50.35	2.11	1.41	10	50.00	0.00	0.00
securedrop	225	46.67	1.33	0.00	94	37.23	0.00	0.00
kiwi	56	50.00	1.79	0.00	31	45.16	3.23	0.00
pgadmin	920	64.02	1.85	1.20	88	61.36	2.27	1.14
erpnext	1355	40.15	1.55	0.22	1	100.00	0.00	0.00
edx-platform	599	58.76	1.00	0.17	194	52.58	1.03	0.00
intelowl	117	64.96	7.69	0.85	43	51.16	16.28	2.33
arches	2635	43.76	1.71	0.57	94	45.74	6.38	1.06
geonode	922	43.06	1.52	0.22	107	53.27	0.93	0.00
ghost	687	66.81	22.27	19.94	192	65.10	21.88	18.75
nodebb	964	37.24	0.93	0.52	591	30.80	0.34	0.51
calcom	1183	74.47	7.69	6.00	655	81.37	7.48	6.72
documenso	643	54.43	3.11	2.18	155	70.97	2.58	2.58
hoppscotch	1246	62.04	7.06	5.38	235	56.60	0.00	0.00
immich	445	44.27	2.02	0.22	178	30.90	0.56	0.00
etherpad	91	68.13	3.30	0.00	33	72.73	9.09	0.00
uptime-kuma	792	64.77	4.80	3.66	95	73.68	8.42	5.26
habitica	236	55.93	6.78	3.81	78	74.36	3.85	2.56
hedgedoc	48	54.17	8.33	6.25	36	52.78	5.56	2.78

Observations

• line loses about half of all persisting findings (pooled 52.6%, bootstrap CI 48.5–57.6), and as many security findings (50.7%, CI 43.1–62.9).
• The text-based schemes lose about 2% pooled. Per repository the median is about 0.5%, and 0% for codeguard on security findings. The pooled rate is pulled up by a few JS/TS repositories (Ghost about 20%, cal.com, Hoppscotch and hedgedoc about 6%). The cause is renamed files: every key includes the path, so a moved or .js→.ts file loses all its findings. For example, 130 of Ghost's 137 codeguard false-new findings are on unchanged lines in 54 renamed file changes. This is now listed in the README's limitations.
• The schemes trade false-new against collisions. codeguard collides on 18.6% of findings (29.6% of security findings), github on 2.1%, and cg_no_scope on 23.6%. Its hidden rate is 1.9% (CI 0.9–3.5) against 0.8% for github.
• Wilson intervals are much narrower than the cluster-bootstrap intervals (for example line_text 1.73–2.08 against 0.80–3.51). Findings within a repository are strongly correlated, so the bootstrap intervals are the ones to quote.
• Security findings are 25% of persisting findings (5,794 of 23,237), and every repository has some.

Test evidence

$ python -m pytest research/finding_identity/tests -q
40 passed
$ python -m pytest -q
629 passed
$ ruff check research/ && ruff format --check research/
All checks passed! / 26 files already formatted
$ MYPYPATH=src mypy --strict research/finding_identity
Success: no issues found in 22 source files

New tests:

• test_stats.py checks against hand-computed values: Wilson intervals (k=n/2, 0, n; n=0), type-7 quantiles, pooled vs macro (1/4 and 3/6 give 0.4 vs 0.375), the cluster bootstrap with fixed draws (percentile interpolation worked by hand), collapse on identical repositories, undefined resamples dropped, security-scope extraction, a missing scope skipped, and CSV/Markdown outputs.
• test_batch.py covers resume (is_done needs a matching SHA and enough pairs and survives corrupt JSON), skipped/stale/new/failed handling, the journal and logs, retrying only failures, --force, and a real end-to-end fetch and replay of a local repository pinned behind its branch head. It also covers the licence classifier (GPLv3 mentioning the AGPL, v. 3.0, carve-outs, BSL) and the corpus round trip.
• test_categories.py checks that per-category totals add up exactly to the overall totals.

Notes for reviewer

• CTFd was excluded during the run. It commits Vite build output (CTFd/themes/*/static/assets/*-<hash>.js) that the harness's vendored-file filter does not skip, so its replay compared 12,944 findings in its first 75 pairs. No other repository exceeded about 4,000 over 200 pairs. This adds a hand-judged "no committed build output" criterion, disclosed in the README. A content-based minified-file filter in scan.py could bring it back; I left that alone because scan.py is being edited in parallel. Its replay finished afterwards (200 pairs, 28 min; the full 36-repository batch took 31 min 14 s wall time, with 0 failures). As a sensitivity check, here are the results with CTFd included:

  scope	scheme	metric	35 repos: pooled [bootstrap CI]	median	36 with CTFd: pooled [bootstrap CI]	median
  all	line	false-new	52.57 [48.51, 57.62]	54.43	75.29 [49.32, 89.13]	55.12
  all	github	false-new	3.14 [1.96, 4.86]	1.85	49.39 [2.14, 77.16]	1.94
  all	codeguard	false-new	1.79 [0.72, 3.39]	0.48	48.68 [0.88, 76.88]	0.50
  all	codeguard	collision	18.56 [12.91, 24.85]	11.92	50.31 [14.07, 74.32]	12.14
  security	codeguard	false-new	1.90 [0.44, 4.30]	0.00	4.57 [0.63, 12.38]	0.00

  CTFd alone has 22,122 persisting findings, as many as the other 35 repositories together (23,237). It swamps the pooled rates, while the per-repository medians hardly move. That is the case for excluding it, and for reporting medians and cluster-bootstrap intervals next to pooled rates.

• DefectDojo and IntelOwl are pinned to their development branches (dev, develop): their default branches only get releases (184 and 14 first-parent commits in two years).

• Committed: corpus.csv and paper/ (pooled.csv 12 KB, per_repo.csv 146 KB, repos.csv, tables.md, per_repo.md, the plot). Clones, results/ and tables/ are git-ignored.

• The PR base is claude/nice-allen-ptg63r (Add finding-identity research harness for studying scheme effectiveness #51), so the diff shows only this work.