Skip to content

Add finding-identity research harness for studying scheme effectiveness - #51

Merged
mevichitra merged 1 commit into
mainfrom
claude/nice-allen-ptg63r
Oct 14, 2026
Merged

mevichitra merged 1 commit into
mainfrom
claude/nice-allen-ptg63r

Conversation

@mevichitra

@mevichitra mevichitra commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Summary

This PR adds a research harness for studying how well different identity schemes keep static-analysis findings stable across code changes. The harness replays git histories, scans files before and after each commit with CodeGuard, and scores seven identity schemes (line number, line text, GitHub's primaryLocationLineHash, CodeGuard's fingerprint, and three ablations) against ground truth built from line diffs. This is research code not part of the released package.

Type of change

  • Bug fix (false positive / false negative correction)
  • New rule (attach rule ID from the tracker)
  • Engine / runner change
  • CLI change
  • Docs only
  • Tooling / CI

Checklist

  • pytest passes locally
  • ruff check src/ tests/ clean
  • mypy src/ clean
  • Research code is self-contained in research/ directory
  • Comprehensive unit tests added for core modules
  • README with usage instructions included

Test evidence

$ pytest research/finding_identity/tests/ -v
test_schemes.py::test_github_hash_matches_reference PASSED
test_schemes.py::test_github_hash_counts_duplicates PASSED
test_schemes.py::test_codeguard_key_equals_runner_fingerprint PASSED
test_schemes.py::test_codeguard_key_ignores_literals_and_spacing PASSED
test_matching.py::test_map_lines_insertion_and_edit PASSED
test_matching.py::test_map_lines_unrelated_replacement_is_unmapped PASSED
test_matching.py::test_line_scheme_hides_a_new_finding_and_loses_the_old_one PASSED
test_replay.py::test_replay_scores_schemes PASSED

Notes for reviewer

Research code only: This is not part of the released CodeGuard package. It lives in research/finding_identity/ and is invoked via python -m research.finding_identity.

Key components:

  • schemes.py: Seven identity schemes including a faithful port of GitHub's primaryLocationLineHash algorithm (verified against the original fingerprints.ts)
  • matching.py: Ground truth computation via line mapping and scheme evaluation against it
  • replay.py: Main harness that walks git history, scans changed files, and aggregates results
  • gitrepo.py: Minimal git interface for history and blob access
  • linemap.py: Line mapping with similarity-based alignment for edited blocks
  • scan.py: CodeGuard integration with caching and collision detection
  • summarize.py: Rate table generation for paper reporting
  • cli.py: Command-line interface for replay and summarize subcommands

Testing: Includes end-to-end test that builds a tiny git repository, replays it, and verifies scheme behavior on line moves and scope changes. GitHub hash implementation verified against reference values from the original codeql-action.

Research tooling for a study of how well static-analysis findings keep
their identity as code changes. The harness replays a repository's
first-parent history, scans each changed file before and after every
commit with CodeGuard, and scores identity schemes against a ground
truth built from a line diff.

Schemes: line number, exact line text, GitHub code scanning's
primaryLocationLineHash (a port of codeql-action's fingerprints.ts,
verified against the original), CodeGuard's codeguard/v1 fingerprint,
and three ablations of it. Each new finding is classified as correctly
matched, mispaired, falsely new, or falsely known (hidden), and
same-file key collisions are counted.

Lives under research/ and is not part of the released package. Results
and clones are git-ignored.

Signed-off-by: Vichitra Attri <mevichitra@gmail.com>
@mevichitra
mevichitra merged commit 749cd79 into main Oct 14, 2026
16 checks passed
mevichitra added a commit that referenced this pull request Oct 17, 2026
PRs #53 (long-span drift mode) and #55 (corpus, batch runner,
statistics and the paper/ tables) were merged into the branch of
#51 after #51 itself had been merged, so they never reached main.

Conflicts, all resolved by keeping both sides:
- cli.py: register the transform subcommand (main) next to drift,
  drift-summary and the corpus commands; replay keeps main's
  --analyzer and per-analyzer schemes.
- scan.py: the Scanner keeps main's pluggable analyzer and gains the
  per-category totals.
- requirements.txt: main's pinned research dependencies; matplotlib
  now also serves `stats --plot`.
- .gitignore: ignore tables/ as well as fidelity/.build/ and .cache/.
- README.md: keep both the Fidelity/RQ2 sections and the
  Corpus/Batch/Statistics sections; note that the committed paper/
  tables predate the sonar and bandit schemes.

Signed-off-by: Vichitra Attri <mevichitra@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant