Repository navigation
Conversation
Results are stored under the unqualified function name. That is unambiguous for the C corpus, but C++ overloads and same-named methods in different scopes share a short name while being different functions, so they collapsed into one entry. Fixing a single backend was not enough: relabeling, metrics, source lookup, result artifacts and re-evaluation could each lose the distinction again. Identity now derives from the complete DWARF target set rather than from whatever subset a backend recovered. A name that is unique in the binary keeps its historical plain key; only an ambiguous name becomes name@0x<canonical-low-pc>. Backend addresses are normalized to the matched DWARF low_pc, so every consumer keys off the same primitive. The map used to decide ambiguity is the binary-wide target set, kept separate from the per-run selection. Selecting one member of an overload group therefore does not rename the others, which makes storage identity invariant under sample-set slicing. Where an exact identity is unavailable the code abstains instead of guessing: an ambiguous CFG is not scored, a collision-qualified function does not borrow name-keyed ground truth, and it contributes no prototype to another overload's compile context. Likewise a DWARF range lookup given an address matches that address or fails, rather than falling back to the first DIE sharing the short name. DWARF low_pc == 0 sentinels left by linker ICF are rejected only when the ELF section table proves address zero is not executable, so genuine bare-metal code at zero is preserved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Covers the collisions the identity change exists for: DWARF extraction that must not collapse overloads, slice invariance of canonical keys, metric and CFG abstention for ambiguous names, address-keyed source recovery, linker-ICF zero-address sentinels, ARM Thumb address normalization, and end-to-end backend behaviour. Tests needing a cross-compiler or linker (arm-none-eabi, ld.lld, mingw) skip when it is absent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review found three places where the new storage-key format was written but not read back, so the distinction this change exists to preserve was lost again at the boundary. pipeline/materialized.py rebuilt results with its own `(\S+)` marker regex and put the marker's first field straight into FunctionDecompilation.name. For a collision that made the name `same@0x1000`, and ByteMatch feeds that field to function_bytes(), the compile fixup and object symbol lookup as a real C identifier -- so re-evaluating a materialized C++ artifact fed it a string that is not a function name. The same regex could not see a marker whose DWARF name contains a space (`operator new`) at all. It now reads through results_tree.split_functions() and recovers the semantic name with parse_function_storage_key(), keeping the storage key only as the map key. TypeMatch's address-keyed ground-truth walker visited only the compile unit's direct children, while identity extraction walks the whole DIE tree. A C++ subprogram inside a namespace or class scope was therefore named `foo@0x...` correctly and then had no ground truth at that address, so it dropped out of the metric silently. It now walks CU.iter_DIEs(). Three further copies of the marker pattern are folded into the shared one: repair_decompiled_flags compares against FunctionData keys and so wants the storage key, while reeval_ged and audit_historical_ged_iso compare against Joern body names and historical score keys and so want the semantic half. Tests cover same@0x1000 / same@0x2000 / `operator new` round-tripping, and namespace-scoped overloads reaching the address-keyed ground truth. The latter needs real ELF: `g++` exists on macOS but links Mach-O, which made this module raise ELFError instead of skipping, so the guard now probes the linker output. That also turns the pre-existing sibling test from a failure into a skip off Linux. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
A few follow-up concerns I found while reviewing this draft more closely. I don't plan to address all of these in the current revision immediately, but I want to document them explicitly before further review:
None of these change the intended status of this PR: I also still consider remote CI approval/run necessary before treating the focused validation lane as green; the current Actions run is |
|
One additional related PR worth calling out is #36. I don't think #36 and this PR have a large direct file-level conflict today: #36 is currently limited to the experimental MSVC/Wine compile container, smoke script, and There is, however, substantial design and follow-up overlap, so I think the two paths should be reconciled rather than developed independently. Relation to #36 — MSVC/PDB integration#36 established that real MSVC builds are feasible on the benchmark host and, importantly, identified PDB/CodeView as the native ground-truth path for Windows targets. Its follow-up plan includes:
PR #103 implements or changes several of those same boundaries:
So although #36 itself should still be independently useful as the compiler/container side of native MSVC support, its documented Python integration plan is now partially superseded or at least needs to be updated against the implementation in this draft. There are also a few concrete places where the contracts should be reconciled:
My current view is therefore:
If #36 lands independently, I think its follow-up integration work should use or adapt the native-debug contracts from this branch rather than introducing a second PDB/type pipeline. Conversely, this PR should not be interpreted as replacing the useful compiler/container work from #36: the native MSVC build path is still required if those Windows targets are eventually brought into the real benchmark corpus. |
This branch has not been deployed
1 participant
[AUTOMATED]
Qualification / dependency experiment follow-up (proposed)
This follow-up strengthens evidence reuse, specifies experimental qualification
boundaries and adds reproducible compiler controls. It does not adopt official
C++ semantics, update the Rust-Joern pin, or qualify corpus C++ byte matching.
Native/source provenance
modification, removal or appearance, including changes during preparation.
Preserve UUID/GUID/Age matching and embedded-DWARF priority; track caller-path
aliases. This is a filesystem-version guard, not cryptographic attestation.
physical unit and compilation directory even when file/line evidence is missing.
No source definition is guessed or newly admitted by this production patch.
unconnected identities, including 27 missing Windows TUs and 19 missing locations.
A new build does not provide ground truth for the old image/address plan.
Counterfactual Rust-Joern replay
Using identical prepared bytes, four frozen producer outputs and 45 fixed cells,
compare the pinned
9fd97f5with Rust-Joern PR Noelo-Lab/rust-joern#3 (52a8899),with producer-output defects rejected before graph admission:
These are dependency/profile diagnostics, not producer quality or leaderboard
improvements. The two new source rows are optimized variants of the same Snappy
definition, not independent programs. Type eligibility is not expanded.
inputs change directed CFG topology/entry-exit roles, none is lost.
references, isolated from the replacement shim. Three initial
.iireferenceattempts were rejected by the original generator; identical-byte
.cppretriescomplete all captures, with initial infrastructure errors retained separately.
candidate matches original references on 32/87;
joern_loop_conditions=Truematches 82/87, including all 55 previously parsed changed inputs and 27 new
inputs. Four new inputs have no original exported function; one newly parsed
Ghidra pseudocode input remains non-isomorphic (194/289 versus 199/297 nodes/edges).
prior graphs; those 10 now match original Joern. Full CPG/type validity/execution
parity and the historical 8,808-case audit are not established by this replay.
Keep the production pin and parser options unchanged. Loop policy, the one
remaining non-isomorphism and unavailable original exports require review before
adopting a new diagnostic profile/cache generation. No function/address exception
is added to implementation.
Semantic policy and compiler controls
docs/cpp-evaluation-qualification.mdproposes separate packaging, exact sourceidentity, syntactic-CFG and instantiated/effect qualification gates. Generic
syntax does not supply concrete template arguments, construction/destruction or
exception cleanup. The proposal is not an official maintainer-adopted metric.
The optional development script
scripts/audit_cpp_toolchains.pypreservescommands and source/preprocessing/AST/IR/object evidence for a header-free fixture.
Apple Clang 17.0.0 and Homebrew Clang 22.1.8 each pass ELF/AArch64, COFF/x86-64
MSVC ABI and Mach-O/arm64 at O0/O2/O2-noinline: 18 controls. They verify target
and data-model assertions, concrete template arguments -2/2, construction/throw
AST, exceptional cleanup IR and identical code sections after preprocessing.
The script rejects wrong object targets and does not link or execute the fixture.
Zero corpus C++ recompiles are qualified. Clang targeting PE is not native
MSVC, and Clang targeting ELF is not original GCC 13.3. Standard-library/runtime
ABI, exact original commands, link/LTO/ICF and recovered-code recompilation still
need qualification. No new runtime dependency is added; legacy byte arithmetic
and cache generation remain unchanged.
Final production-patch regression evidence
results, populations, exclusions, native addresses and producer hashes match.
Source CFG eligibility remains 54/250; source/ABI type eligibility remains 26/250.
Two complete fresh replays have identical semantic reports for 45/45 cells after
removing only computation-time fields. Frozen producer outputs are reused.
aliases, embedded debug priority and compiler/ABI controls.
selected owners pass. Full baseline: 980 passed; candidate: 997 passed. Both
retain identical 19 failures /24 errors /28 skips (43 failure/error identities).
Full tests are not green. A separately expanded binfmt/PDB mypy audit has 27
baseline versus 26 candidate errors, no new error message; it is not green.
cannot validate it. Fix C++ function identity collisions in the benchmark pipeline #91 remains open and this remains a dependent draft.
Complete proposed patch, source hashes, original references, comparisons, compiler
controls and JUnit evidence are retained locally in Fission
benchmark/artifacts/cpp-pipeline-qualification/2026-10-19/.Datasets remain qualification,
frozen pilot
and earlier raw corpus.
No corpus membership, Hub files, published results, default metrics or Union
are updated by this follow-up.
Earlier implementation and evidence
Producer-output inspection follow-up
This revision proposes a bounded, oracle-free producer-output contract for the experimental C++ evaluators. It does not adopt an official C++ policy or update the Rust-Joern dependency pin.
metadata.output_contract; the Ghidra/Fission/angr/Kuna adapters declare C-like pseudocode with partial declarations. Legacy records stay unknown. Code, variables and native address/line evidence are preserved.cpp-ged-origin-v2-diagnostic.Follow-up measured evidence
Same 45 cells / 250 fixed native requests per tool, same saved producer code hashes and exact source evidence, fresh metric computation, no corpus/recovered-code execution:
9fd97f50885202596b125ebd2504e64910874bcb.Point.columnaccess crashed locally. Byte-offset coordinate calculation preserves diagnostics and then passes 10,000 fresh inspections with identical evidence. This is an integration workaround, not a claim to fix the native dependency; see py-tree-sitter issue 487 anddocs/cpp-producer-contract.md.qualified=false/scoring_ready=falseand incomplete C++ feature/provenance gates remain explicit.Documentation:
docs/cpp-producer-contract.md. Complete local patch, evaluator deltas, original fault logs, JUnit reports and dataset audit are retained in Fissionbenchmark/artifacts/cpp-output-contract/2026-10-19/.Qualification corpus · Frozen pilot manifest · Earlier raw corpus
Remote CI has not run for this proposed follow-up. The PR remains a dependent draft; #91 sequencing and maintainer review still apply.
Earlier implementation and qualification evidence
Scope
Add an opt-in C++ source/ABI type graph proposal, native debug transport and
scoped Fission definition-line evidence for the qualification corpus.
enums and virtual-base locators are compared structurally after type-blind
variable correspondence. Name/total size alone never earns object credit.
ambiguity, layout, cycle and incomplete-evidence abstention.
definitions for the bounded LP64/LLP64 natural-layout subset.
line_mappings_nirwithdefined_variablesscope. Directvariable PCs stay primary; RHS occurrences do not inherit LHS definition PCs.
Legacy whole-line mappings preserve their serialization/ownership contract.
including namespace syntax that the C grammar can misinterpret.
with real DWARF and CodeView/PDB compiler fixtures.
CppTypeMatchMetricremains experimental and outside default metrics/Union.Maintainers must choose official C++ policy and qualification before adoption.
No leaderboard, website, snapshot or CHANGELOG update is included. Legacy native
provenance/parser changes bump
type_matchcache generation to 25.Prerequisites
Depends on #91 (currently open, head 8eeceb8).
This branch includes its C++ identity prerequisites and the subsequent native
PDB/Mach-O/source transport work. The base main is
14e14b8. Please treat this as a dependent draft;
the prerequisite diff can be removed after #91 lands.
The companion Fission patch adds conservative producer line origins. Old Fission
CLIs remain supported using direct variable addresses and empty line maps.
Datasets
Measured evidence
Frozen qualification-v2 dataset revision 07dd95dbe6214b8a78fc1e4a55961ab515986489:
250 address-guided stripped functions across 45 cells (Linux 108 / Windows 97 /
macOS 45). Source/debug information stays exclusively with the evaluator.
zero changed NIR texts or direct variable lists.
all 9,482 direct variable/PC pairs also survive native validation.
21 with object graphs; v1 admitted 5. Perfect source/ABI: Fission 0/26,
Kuna 1.808 1/26. Kuna outputs are reused from the earlier frozen run.
not a decompiler quality/ranking improvement or a full C++ score.
perfect, Kuna 32 / 0 perfect (97 requested Windows functions).
See docs/validation/cpp-type-policy.md, the compact summary and frozen population
for hashes, profiles, exclusions and reproduction commands. Full artifacts are
retained locally in Fission benchmark/artifacts/cpp-type-policy/2026-10-13/.
Validation
CLI, workspace fmt and owner-boundary scan passed.
baseline/final counts and identities. No new failure identity remains.
The existing Rust-Joern CFG dependency is pinned to
9fd97f50885202596b125ebd2504e64910874bcb for this lane.
Bounds
Maps represent single result definitions, not full expression/effect lineage.
Computed/split debug locations, formal/storage ambiguity, anonymous/forward
objects, function/member pointers, inline identity and unlocated native vptrs
still abstain. Virtual locators retain their debug format; cross-format
normalization is not guessed. Instance storage/inheritance is measured, not a
complete class API, RTTI, exception behavior or execution parity.
Follow-up review fixes (2026-10-18)
preparation (
UnsupportedType/PdbFormatError). Unrelated evaluator/parserValueErrornow fails evaluation rather than shrinking eligibility; recoveryscoring runs outside the oracle-abstention boundary.
shared-matcher boundaries. Ambiguous names receive no scoped PCs. Direct
variable PCs remain primary; legacy whole-line maps retain their contract.
tree-sitter-cppdependency wording and explicitly retainnative virtual-base locator equality as an unresolved official-policy limit.
The latest correctness follow-up below advances them to 25 and 4.
Formal C++ comparison policy remains
cpp-types-v2-proposal, outside Union.Ten new regression cases fail before and pass after these fixes. Focused local
lane: 166 passed /5 off-host skips, including real DWARF/dSYM and CodeView/PDB
compiler fixtures. Four policy mypy owners and scoped Black/Ruff pass. Shared
Ruff still has one unchanged baseline SIM105; whole-repository lint is not green.
Full local pytest: baseline 863 passed /20 failed /28 skipped /24 errors;
candidate 873 passed /20 failed /28 skipped /24 errors. All 44 failure/error
identities agree; no new identity.
Fresh evaluator replay of the same frozen 250 Fission and 250 Kuna outputs across
45 cells yields identical baseline/candidate reports after excluding only legacy
PDB diagnostic
computation_time_seconds: eligibility 26/250,Fission source/ABI perfect 0/26, Kuna 1.808 perfect 1/26. Producers are not rerun;
these are not new current-tool scores or a ranking update. All 572 input/debug/
producer artifact hashes are verified. Full follow-up evidence is in
docs/validation/cpp-review-fixes.mdand its compact JSON.Remote CI for this follow-up must still run on the exact submitted head. The
previous upstream run remains
action_required; local checks do not substitutefor maintainer approval/remote CI. This remains a dependent draft on #91.
Experimental C++ GED connection diagnostics (follow-up)
Add an explicitly opt-in
cpp-ged-origin-v1-diagnosticevaluator and CLI, outsidedefault GED/Union. Keep every parser definition, bind native entries to exact
qualified source definitions and producer-record storage keys, support GCC/MSVC
line origins and native compilation-directory / recorded saved-temp TU transport.
Ambiguous identities and conflicting compile commands abstain; unexpected
evaluator/infrastructure failures do not silently reduce the source population.
Frozen same-250 / 45-cell replay: source CFG population 48/250 for both tools;
producer CFGs connect 249/250 Fission and 246/250 Kuna 1.808. Actual diagnostic GED
is measurable for 48 and 47 rows, with 18 and 20 distance-zero results. This is
evaluator coverage evidence, not a decompiler quality/ranking/Union improvement.
The 202 shared oracle/policy exclusions and per-output parser failures are
explicit. Two fresh CFG passes are byte-identical for 45/45 cell reports.
Definition-only parsing, template specialization, implicit lifetime/exception
edges, full-TU context and independent build/link qualification remain bounded.
Compilation metadata is asserted evaluator input. See
docs/validation/cpp-ged-identity.mdfor populations, exclusions and limits.New regressions: 33 passed. Updated focused workflow locally: 202 passed / 5
platform skips. Full local baseline 874 passed, candidate 907 passed; both have
the same 19 failures / 24 errors / 28 skips, with no changed failure identity.
New owners pass mypy and scoped Black/Ruff. Remote CI for this follow-up is
pending submission/maintainer approval and is not claimed green.
Follow-up: exact native-origin retention and recompiler target checks
This follow-up is based on ea2b461 and remains experimental. The
qualification/raw corpus links above and the frozen 250 requests are unchanged.
It does not implement full C++ semantics or a default C++ metric.
rather than discarding it through system-header filtering. Record retained
and filtered origins, primary TU, exact definition digest and native identity.
No basename/leaf-name fallback, resampling or generic template substitution.
coverage from 48 to 54/250. Fission paired/zero: 54/18; Kuna: 53/21. Every
previously measured function result is identical, and no source row is lost.
This is evaluator coverage, not a Fission quality delta or leaderboard score.
native index is shared within each cell and bound to immutable inputs.
Six new identities have independent original-source checks; one requires
direct namespace inspection because of a raw-parser scope limitation.
Native ELF gcc selection is now Linux-only. Compiled relocatable objects must
match the expected format family/architecture/bitness before byte extraction;
unproven/incompatible targets abstain and stay out of the binary aggregate.
Byte-match cache generation becomes 9. Existing Linux selection and compatible
byte-comparison arithmetic are retained; no published C results are re-scored.
lacks the Linux cross-compiler; native MSVC and Mach-O C++ byte matching remain
unqualified. No C++ byte score was produced.
The current source exclusions include 84 template-specialization cases, 27
Windows TU misses without retained preprocessing (Detours/WinSparkle), and 19
Windows entries without usable native file/line facts. Implicit object lifetime,
exception cleanup and full-TU callee context are unqualified. Split/computed
locations, ambiguous PDB formal status, incomplete objects and vptr placement
still abstain; the prior 26/250 type-policy population was not re-scored here.
Local validation: focused workflow 246 passed / 5 platform skips, scoped Black
and Ruff plus existing C++ owner mypy clean. Full baseline 907 passed; candidate
927 passed. Both have the same 19 failures / 24 errors / 28 skips, with all 43
failure/error identities unchanged. The updated remote CI has not run before
submission; the prior head's green fork run does not validate this new delta.
See docs/validation/cpp-oracle-qualification.md. Full local evidence is retained
in Fission benchmark/artifacts/cpp-oracle-qualification/2026-10-18/. No official
C++ policy, Union, site/snapshot, result overlay or Hugging Face upload is included.
Correctness review follow-up (base 9f50ebf)
preparation/scoring. Validate after plan filtering, preserving empty recovery
for missing eligible output and excluding extras outside the plan.
functions with reference/rvalue-reference parameters; valid C keeps C grammar.
&&,&,*.Object&&becomesrref(Object), ABIpointer(Object). Invalid nested references abstain; boundedLP64/LLP64 layout and ILP32 explicit-evidence requirements are retained.
remains
cpp-types-v2-proposal, outside default metrics/Union.Before fixes, new regressions: 14 failed /6 passed. Current focused workflow
locally: 268 passed /5 platform skips; scoped Black/Ruff and seven existing
C++ mypy owners pass. Full baseline: 927 passed /19 failed /28 skipped /24 errors;
candidate: 949 passed /19 failed /28 skipped /24 errors. All 43 existing
failure/error identities agree; no new/removed/changed identity. Full tests are
not green. Real compiler-generated DWARF and clang-cl/PDB fixtures compare
nominal rvalue-reference graphs in source/ABI views without execution; they do
not qualify the complete MSVC pipeline.
Fresh type-policy replay of all frozen 250 requests /45 cells keeps eligibility
26/250 for both tools/views, Fission perfect 0/26 and Kuna 1.808 perfect 1/26.
All function results and exclusions are identical. Frozen producer outputs are
re-ingested, not regenerated; this is evaluator regression evidence, not a new
GED result, decompiler quality/ranking/Union update or full C++ score.
The local harness freshly prepares immutable native readers, source indexes and
oracle rows once per image process, independently for baseline and candidate;
metric caches are disabled and all metric results computed fresh. An initial
unshared large-image attempt timed out at 300 seconds and is retained as partial
evidence. Nineteen completed cell executions match unshared controls after
excluding timing/preparation metadata. The final 45-cell replay completes with
zero failed/timed-out cells. Production performance code is unchanged.
See
docs/validation/cpp-type-policy.md. Complete local patch, logs, commands,hashes and reports: Fission
benchmark/artifacts/cpp-review-correctness/2026-10-18/.#91 remains open; wait for it to land before rebase/prerequisite removal. The new
submitted head still needs exact-SHA remote CI/maintainer approval; previous-head
fork success does not validate this delta. Virtual-base cross-format equivalence
and future malformed-PDB fuzz/property gates remain separate qualification work.