From c1c37a3e3cff8b4b34ef23afb8bc7ed7c026746c Mon Sep 17 00:00:00 2001 From: Vichitra Attri Date: Wed, 14 Oct 2026 13:03:22 +0000 Subject: [PATCH 1/2] research: compare the identity methods of SonarQube, Bandit and Semgrep Line-number identity alone is a straw man, and every finding came from CodeGuard. Add the matching methods real tools ship, ported from their source at pinned versions and checked against the original tools, and let the replay take its findings from Bandit or Semgrep as well. Schemes: - sonar: SonarQube 26.10 issue tracking, a six-pass matcher with block-hash code-move detection (Tracker, AbstractTracker, BlockRecognizer, BlockHashSequence), including a port of java.util.HashMap's iteration order, which breaks ties. - bandit: `bandit -b baseline.json` (Bandit 1.9.4): Issue.__eq__ fields and set membership, so duplicates of a baselined issue stay hidden. - semgrep: Semgrep 1.180.0 match_based_id, as emitted in JSON and SARIF. Multi-pass matchers plug into matching.scheme_pairs through trackers.MATCHERS; Observation gains the message and file lines they need. Fidelity: fidelity/sonar compiles the unmodified SonarQube tracker with a small Java harness; the port agrees with it on generated cases and on HashMap key orders (recorded in tests/fixtures). The Semgrep port reproduces every ID Semgrep emitted on Flask and DefectDojo (recorded). Live tests run Bandit's baseline mode and Semgrep's JSON and SARIF output when the tools are installed. Analyzers: `replay --analyzer codeguard|bandit|semgrep`. Bandit runs in process; Semgrep runs offline with local rules in semgrep_rules.yml. Research-only dependencies go in research/finding_identity/requirements.txt. Signed-off-by: Vichitra Attri --- research/finding_identity/.gitignore | 1 + research/finding_identity/README.md | 126 +- research/finding_identity/analyzers.py | 316 ++ research/finding_identity/cli.py | 29 +- .../finding_identity/fidelity/__init__.py | 0 .../fidelity/semgrep_cases.py | 154 + .../fidelity/sonar/SonarTrackerHarness.java | 143 + .../finding_identity/fidelity/sonar/build.sh | 55 + .../finding_identity/fidelity/sonar_cases.py | 315 ++ research/finding_identity/matching.py | 14 +- research/finding_identity/requirements.txt | 5 + research/finding_identity/scan.py | 58 +- research/finding_identity/schemes.py | 172 +- research/finding_identity/semgrep_rules.yml | 388 ++ .../fixtures/semgrep_match_based_id.json | 3692 +++++++++++++++++ .../tests/fixtures/sonar_tracker.json | 1 + research/finding_identity/tests/test_tools.py | 273 ++ research/finding_identity/trackers.py | 860 ++++ 18 files changed, 6561 insertions(+), 41 deletions(-) create mode 100644 research/finding_identity/analyzers.py create mode 100644 research/finding_identity/fidelity/__init__.py create mode 100644 research/finding_identity/fidelity/semgrep_cases.py create mode 100644 research/finding_identity/fidelity/sonar/SonarTrackerHarness.java create mode 100755 research/finding_identity/fidelity/sonar/build.sh create mode 100644 research/finding_identity/fidelity/sonar_cases.py create mode 100644 research/finding_identity/requirements.txt create mode 100644 research/finding_identity/semgrep_rules.yml create mode 100644 research/finding_identity/tests/fixtures/semgrep_match_based_id.json create mode 100644 research/finding_identity/tests/fixtures/sonar_tracker.json create mode 100644 research/finding_identity/tests/test_tools.py create mode 100644 research/finding_identity/trackers.py diff --git a/research/finding_identity/.gitignore b/research/finding_identity/.gitignore index 351a656..a1d2958 100644 --- a/research/finding_identity/.gitignore +++ b/research/finding_identity/.gitignore @@ -1,2 +1,3 @@ .repos/ results/ +fidelity/.build/ diff --git a/research/finding_identity/README.md b/research/finding_identity/README.md index 6c915c5..eee36a7 100644 --- a/research/finding_identity/README.md +++ b/research/finding_identity/README.md @@ -6,22 +6,76 @@ code changes?* It is not part of the released `codeguard` package. Baselines, suppressions, "fail only on new findings" CI checks and alert de-duplication all rely on recognising that a finding in version *n+1* of a file is the same finding as one in version *n*. This harness replays real git histories, scans every changed file before and -after each commit with CodeGuard, and scores several **identity schemes** against a -**ground truth** built from the diff. +after each commit with an analyzer (CodeGuard, Bandit or Semgrep), and scores several +**identity schemes** against a **ground truth** built from the diff. ## Schemes -Every key includes the rule ID and the file path. +Every scheme includes the rule ID and the file path. The schemes of shipped tools are +ports of their source code at a pinned version, checked against the original tool (see +[Fidelity](#fidelity)); the docstrings cite the files and functions they port. + +| Scheme | Identity is built from | Ported from | +|---|---|---| +| `line` | the line number | | +| `line_text` | the exact text of the flagged line | | +| `github` | GitHub code scanning's `primaryLocationLineHash`: a rolling hash over the 100 non-blank characters from the start of the line, plus an occurrence count | github/codeql-action `src/fingerprints.ts` | +| `sonar` | SonarQube's issue tracker, a **multi-pass matcher**: (1) same rule, line, line hash and message; (2) same rule, line and line hash; (3) code moves, by unique 11-line block hashes and then by the longest identical block; (4) same rule, line hash and message; (5) same rule, line and message; (6) same rule and line hash. The line hash is the MD5 of the line without spaces and tabs | SonarQube 26.10.0.133078, `sonar-core` `issue/tracking/Tracker.java`, `AbstractTracker.java`, `BlockRecognizer.java`, `BlockHashSequence.java`, `hash/SourceLineHashesComputer.java` | +| `bandit` | `bandit -b baseline.json`: a result is known when an issue with the same text, severity, CWE, confidence, file, test name and test ID is in the baseline. No location at all, and **set membership**: once one such issue is baselined, any number of equal new ones are hidden | Bandit 1.9.4 `bandit/core/manager.py` (`_compare_baseline_results`), `bandit/core/issue.py` (`Issue.__eq__`) | +| `semgrep` | Semgrep's `match_based_id` (JSON `extra.fingerprint`, SARIF `fingerprints["matchBasedId/v1"]`): BLAKE2b of the rule's patterns with metavariables replaced by the code they bound, the path and the rule ID, plus an ordinal among equal keys in the file. No line, column or matched code beyond the bindings. **Semgrep findings only** | Semgrep 1.180.0 `semgrep/rule_match.py` (`get_match_based_key`, `get_match_based_id`, `RuleMatches.add`), `semgrep/rule.py` (`formula_string`) | +| `codeguard` | CodeGuard's `codeguard/v1` fingerprint: enclosing function/class plus the normalized statement (literals masked, comments and whitespace removed) | `src/codeguard/engine/fingerprint.py` | +| `cg_no_scope` | ablation: `codeguard` without the enclosing scope | | +| `cg_no_mask` | ablation: `codeguard` without masking literals | | +| `cg_raw` | ablation: `codeguard` without any normalization (scope plus raw line) | | + +`sonar` and `bandit` match with an algorithm rather than a key (`trackers.py`, plugged in +through `trackers.MATCHERS`); their keys hold what that algorithm compares and are used +only for the collision rate (`sonar`: rule and line hash). Bandit pairs nothing, so to +score mispairs the `bandit` matcher pairs equal issues in line order and sends surplus +new issues to the last equal old one. + +Where a port fixes what the tool leaves to its environment: + +- **SonarQube.** Every issue of the previous analysis gets the same update date, so a pass + with several candidates takes the first in report order (the Java sort is stable). + Issues reach the tracker in the analyzer's report order. Line hashes cover whole lines, + as SonarQube computes them for external issues on files without "significant code" + ranges; a SonarQube language plugin that reports such ranges hashes only those. Only + the tracking of open issues runs (there are no closed issues to reopen between two + consecutive versions). The tie-breaking order of `BlockRecognizer`, which iterates + `java.util.HashMap` key sets, is reproduced exactly by a port of `HashMap`. +- **Bandit.** Findings from other analyzers fill Bandit's fields with their closest + equivalents: CodeGuard's description, severity, confidence, CWE and rule title; + Semgrep's message, severity and the rule's `confidence` and `cwe` metadata. +- **Semgrep.** Needs the rule formula and metavariable bindings, so it applies only to + `--analyzer semgrep`. + +Not included: Snyk Code's matching is closed source; reviewdog filters findings by the +diff's added lines rather than carrying identity across versions; the SARIF SDK's +multi-tool result matcher is not used by any of the analyzers here. + +## Analyzers + +`--analyzer` chooses what produces the findings; every analyzer is scored with all the +schemes it supports (all of the above, plus `semgrep` for Semgrep). + +| Analyzer | Languages | Runs | +|---|---|---| +| `codeguard` (default) | Python, JavaScript, TypeScript | CodeGuard's rules in the `--categories` | +| `bandit` | Python | Bandit 1.9.4 with its default plugins, in process (the code path of the command line) | +| `semgrep` | Python, JavaScript, TypeScript | Semgrep 1.180.0 with the local rules in `semgrep_rules.yml` (or `--semgrep-rules`), one file at a time, offline | + +Bandit and Semgrep are research-only dependencies: -| Scheme | Identity is built from | -|---|---| -| `line` | the line number | -| `line_text` | the exact text of the flagged line | -| `github` | GitHub code scanning's `primaryLocationLineHash` (exact port of `fingerprints.ts` from github/codeql-action, verified against the original) | -| `codeguard` | CodeGuard's `codeguard/v1` fingerprint: enclosing function/class plus the normalized statement (literals masked, comments and whitespace removed) | -| `cg_no_scope` | ablation: `codeguard` without the enclosing scope | -| `cg_no_mask` | ablation: `codeguard` without masking literals | -| `cg_raw` | ablation: `codeguard` without any normalization (scope plus raw line) | +```bash +pip install -r research/finding_identity/requirements.txt +``` + +Semgrep redacts `match_based_id` and metavariable bindings unless an app token is set. +The harness passes a placeholder token and points every proxy variable at a closed local +port, so the scan cannot reach the network. The `semgrep` key is the ID Semgrep itself +emitted; every file also checks the port against it, and `analyzer.json` in the results +records the count (`port_checked`, `port_mismatches`). ## Ground truth @@ -66,14 +120,20 @@ python -m research.finding_identity replay https://github.com/pallets/flask --co python -m research.finding_identity replay https://github.com/DefectDojo/django-DefectDojo \ --ref dev --commits 200 --label defectdojo --categories security,performance,quality --details +# The same history with Bandit or Semgrep findings (results go to results/flask-bandit, ...) +python -m research.finding_identity replay https://github.com/pallets/flask --commits 200 --analyzer bandit +python -m research.finding_identity replay https://github.com/pallets/flask --commits 200 --analyzer semgrep + # Rate tables for one or more runs; pools all runs into an "ALL" table python -m research.finding_identity summarize research/finding_identity/results/* --csv rates.csv ``` -Clones go to `.repos/` and results to `results/