Skip to content

Publish linear benchmark charts and complete software provenance - #114

Merged
konard merged 3 commits into
mainfrom
issue-113-5c8587ee6cc2
Oct 6, 2026
Merged

konard merged 3 commits into
mainfrom
issue-113-5c8587ee6cc2

Conversation

@konard

@konard konard commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

The generated READMEs showed only logarithmic charts and omitted Doublets/toolchain versions, while stale historical sections claimed results that the current measurements do not support. This change publishes both chart scales, records measured software provenance, and generates conclusions from the same comparisons as the tables.

Fixes #113.

Changes

  • Add eight linear charts alongside the existing eight log charts in both READMEs. Linear bars have a disclosed visibility floor of 0.5% of each panel's maximum; log bars and table values retain the measured times. Move legends below the plots and use distinct colors for the 13 C# object variants.
  • Remove the stale sections after the generated markers in English and Russian. Generate per-language/category counts of faster, slower, and approximately equal Doublets operations using the existing noise rule and SQLite baseline of matching durability.
  • Snapshot doublets from Cargo.lock, Platform.Data.Doublets from the C# project, and the actual Rust compiler/.NET SDK in every future benchmark job. Recorded versions take precedence over current manifests; old reports without metadata explicitly identify what was not recorded.
  • Recover provenance for all 20 existing reports from their measured commit 4e269b4 and original benchmark run 37381004025: doublets 0.5.0 / Rust 1.99.0 and Platform.Data.Doublets 0.18.1 / .NET SDK 10.0.401. The reusable backfill experiment checks the log's source SHA before reading those historical manifests. Every original measurement and metadata field is preserved.
  • Keep PostgreSQL, Npgsql, and EF Core versions inside the same italic provenance line.
  • Remove the unimplemented C++ directory and its active documentation, ignore rules, and CI references.

Reproduction and regression coverage

Before the fix, the new tests demonstrate a missing linear chart, missing recorded library/compiler versions, missing generated conclusions, and PostgreSQL metadata outside the closing italic delimiter.

test_linear_charts_keep_small_bars_visible_and_log_charts_keep_actual_values inspects real Matplotlib axes and bars: a 1 ns measurement stays 1 ns on the log chart and displays as 10 ns on the linear chart with a 2,000 ns panel maximum. The input remains unchanged. Provenance tests cover historical version precedence, manifest parsing, snapshots, malformed metadata, and both languages' italic lines. A README regression verifies both committed documents exactly match regeneration and have no stale tail.

python -m unittest discover -s scripts -v
python scripts/benchmark_report.py docs/benchmarks/results \
  --readme README.md --readme README.ru.md --charts docs/benchmarks

Historical provenance can be recovered again with:

mkdir -p ci-logs
gh run view 37381004025 --repo linksplatform/Comparisons.SQLiteVSDoublets \
  --log > ci-logs/benchmarks-37381004025.log
python experiments/backfill_benchmark_provenance.py docs/benchmarks/results \
  ci-logs/benchmarks-37381004025.log

Local validation passes: 68 Python report/CI tests, 6 evidence tests, 9 Rust tests, and all 43 C# tests with PostgreSQL configured (none skipped). Ruff, formatting, mypy, Bandit, Rust formatting/Clippy, C# formatting/build with warnings as errors, Markdown/local links, actionlint/ShellCheck, both zizmor profiles, secretlint, and the preserved evidence integrity check pass.

Hosted validation passes on commit d76ac6c: CI quality, all eight benchmark jobs and report generation, Codacy, and CodeFactor. The eight downloaded smoke-test artifacts validate successfully, contain measured library/toolchain snapshots, and match the latest PR merge commit and branch tree.

Chart previews

C# objects, 32-bit ids, showing the absolute differences with the disclosed minimum bar height:

Linear benchmark chart

The accompanying log chart shows the same measurements and retains small values:

Logarithmic benchmark chart

Changes

  • 51 file(s) modified
  • 607 line(s) added
  • 361 line(s) removed
  • Files:
    • .github/workflows/benchmarks.yml
    • .github/workflows/quality.yml
    • .gitignore
    • CONTRIBUTING.md
    • README.md
    • README.ru.md
    • cpp/README.md
    • cpp/conanfile.txt
    • docs/benchmarks/links-csharp-32-linear.png
    • docs/benchmarks/links-csharp-32.png
    • docs/benchmarks/links-csharp-64-linear.png
    • docs/benchmarks/links-csharp-64.png
    • docs/benchmarks/links-rust-32-linear.png
    • docs/benchmarks/links-rust-32.png
    • docs/benchmarks/links-rust-64-linear.png
    • docs/benchmarks/links-rust-64.png
    • docs/benchmarks/objects-csharp-32-linear.png
    • docs/benchmarks/objects-csharp-32.png
    • docs/benchmarks/objects-csharp-64-linear.png
    • docs/benchmarks/objects-csharp-64.png
    • docs/benchmarks/objects-rust-32-linear.png
    • docs/benchmarks/objects-rust-32.png
    • docs/benchmarks/objects-rust-64-linear.png
    • docs/benchmarks/objects-rust-64.png
    • docs/benchmarks/results/links-csharp-32-100000.json
    • docs/benchmarks/results/links-csharp-32-1000000.json
    • docs/benchmarks/results/links-csharp-32-10000000.json
    • docs/benchmarks/results/links-csharp-64-100000.json
    • docs/benchmarks/results/links-csharp-64-1000000.json
    • docs/benchmarks/results/links-csharp-64-10000000.json
    • docs/benchmarks/results/links-rust-32-100000.json
    • docs/benchmarks/results/links-rust-32-1000000.json
    • docs/benchmarks/results/links-rust-32-10000000.json
    • docs/benchmarks/results/links-rust-64-100000.json
    • docs/benchmarks/results/links-rust-64-1000000.json
    • docs/benchmarks/results/links-rust-64-10000000.json
    • docs/benchmarks/results/objects-csharp-32-100000.json
    • docs/benchmarks/results/objects-csharp-32-1000000.json
    • docs/benchmarks/results/objects-csharp-64-100000.json
    • docs/benchmarks/results/objects-csharp-64-1000000.json
    • docs/benchmarks/results/objects-rust-32-100000.json
    • docs/benchmarks/results/objects-rust-32-1000000.json
    • docs/benchmarks/results/objects-rust-64-100000.json
    • docs/benchmarks/results/objects-rust-64-1000000.json
    • experiments/backfill_benchmark_provenance.py
    • scripts/benchmark_provenance.py
    • scripts/benchmark_report.py
    • scripts/ci_checks.py
    • scripts/test_benchmark_provenance.py
    • scripts/test_benchmark_report.py
    • …and 1 more

Adding .gitkeep for PR creation (default mode).
This file will be removed when the task is complete.

Issue: #113
@konard konard self-assigned this Oct 6, 2026
@konard konard changed the title [WIP] Sync with the other comparisons: linear charts, stale text below the results, full provenance Publish linear benchmark charts and complete software provenance Oct 6, 2026
@konard
konard marked this pull request as ready for review October 6, 2026 23:25
@konard

konard commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

Working session summary

Implemented all five requirements in PR #114: linear charts, generated conclusions, complete software provenance, corrected PostgreSQL formatting, and removal of the C++ placeholder.

Local tests and hosted CI pass. Original measurements are preserved. Commit d76ac6c is pushed, the working tree is clean, and the PR is ready for review.


This summary was automatically extracted from the AI working session output.

@konard

konard commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost estimation:

  • Model: GPT-6.1 Sol
  • Provider: OpenAI
  • Public pricing estimate: $2.024174

📊 Context and tokens usage:

  • 209.3K input tokens across requests (cumulative, larger than the 200K context window, so not one request's context), 32.5K / 128K (25%) output tokens

Total: (209.3K + 3.5M cached) input tokens, 32.5K output tokens, $2.024174 cost

🤖 Models used:

  • Tool: OpenAI Codex
  • Requested: gpt-6.1-sol
  • Thinking level: xhigh (~31999 tokens)
  • Model: GPT-6.1 Sol (gpt-6.1-sol)

📎 Log file uploaded as Gist (3022KB)


Now working session is ended, feel free to review and add any feedback on the solution draft.

@konard
konard merged commit 7e6d533 into main Oct 6, 2026
18 checks passed
@konard

konard commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

🎉 Auto-merged

This pull request has been automatically merged by hive-mind.

  • All CI checks have passed

Auto-merged by hive-mind with --auto-merge flag

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sync with the other comparisons: linear charts, stale text below the results, full provenance

1 participant