Repository navigation
Add finding-identity research harness for studying scheme effectiveness - #51
Merged
Merged
Conversation
Research tooling for a study of how well static-analysis findings keep their identity as code changes. The harness replays a repository's first-parent history, scans each changed file before and after every commit with CodeGuard, and scores identity schemes against a ground truth built from a line diff. Schemes: line number, exact line text, GitHub code scanning's primaryLocationLineHash (a port of codeql-action's fingerprints.ts, verified against the original), CodeGuard's codeguard/v1 fingerprint, and three ablations of it. Each new finding is classified as correctly matched, mispaired, falsely new, or falsely known (hidden), and same-file key collisions are counted. Lives under research/ and is not part of the released package. Results and clones are git-ignored. Signed-off-by: Vichitra Attri <mevichitra@gmail.com>
mevichitra
added a commit
that referenced
this pull request
Oct 17, 2026
PRs #53 (long-span drift mode) and #55 (corpus, batch runner, statistics and the paper/ tables) were merged into the branch of #51 after #51 itself had been merged, so they never reached main. Conflicts, all resolved by keeping both sides: - cli.py: register the transform subcommand (main) next to drift, drift-summary and the corpus commands; replay keeps main's --analyzer and per-analyzer schemes. - scan.py: the Scanner keeps main's pluggable analyzer and gains the per-category totals. - requirements.txt: main's pinned research dependencies; matplotlib now also serves `stats --plot`. - .gitignore: ignore tables/ as well as fidelity/.build/ and .cache/. - README.md: keep both the Fidelity/RQ2 sections and the Corpus/Batch/Statistics sections; note that the committed paper/ tables predate the sonar and bandit schemes. Signed-off-by: Vichitra Attri <mevichitra@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a research harness for studying how well different identity schemes keep static-analysis findings stable across code changes. The harness replays git histories, scans files before and after each commit with CodeGuard, and scores seven identity schemes (line number, line text, GitHub's primaryLocationLineHash, CodeGuard's fingerprint, and three ablations) against ground truth built from line diffs. This is research code not part of the released package.
Type of change
Checklist
pytestpasses locallyruff check src/ tests/cleanmypy src/cleanresearch/directoryTest evidence
Notes for reviewer
Research code only: This is not part of the released CodeGuard package. It lives in
research/finding_identity/and is invoked viapython -m research.finding_identity.Key components:
schemes.py: Seven identity schemes including a faithful port of GitHub'sprimaryLocationLineHashalgorithm (verified against the originalfingerprints.ts)matching.py: Ground truth computation via line mapping and scheme evaluation against itreplay.py: Main harness that walks git history, scans changed files, and aggregates resultsgitrepo.py: Minimal git interface for history and blob accesslinemap.py: Line mapping with similarity-based alignment for edited blocksscan.py: CodeGuard integration with caching and collision detectionsummarize.py: Rate table generation for paper reportingcli.py: Command-line interface forreplayandsummarizesubcommandsTesting: Includes end-to-end test that builds a tiny git repository, replays it, and verifies scheme behavior on line moves and scope changes. GitHub hash implementation verified against reference values from the original codeql-action.