Skip to content

Bump the build benchmark's moq pin past poll-first - #646

Merged
kixelated merged 5 commits into
mainfrom
quest/m4/bench-moq-pin
Nov 7, 2026
Merged

kixelated merged 5 commits into
mainfrom
quest/m4/bench-moq-pin

Conversation

@kixelated

@kixelated kixelated commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m4/bench-moq-pin.md, planned in moq-dev/rust2#631 from moq-dev/rust2#615's report.

Problem

The moq corpus pinned f36c848, from before poll-first (moq-dev/moq#7607). The benchmark was measuring a moq its developers no longer edit: the hub edit rebuilt 8 crates instead of main's 10, and the async body edit hit a pre-poll-first async fn.

Approach

  • Pin moq-dev/moq@142b03b, moq main on 2026-11-07. The leaf, hub, sync, and async targets still match verbatim. Session::closed is now kio::wait(|waiter| self.poll_closed(waiter)).await, and the comment says so.
  • Drop moq-pure-crypto and mark moq as no-c. Since the pin, moq main has moved moq-relay's crypto onto rind, its own pure-Rust fork of graviola, through rind-rustls, rind-rcgen, and rind-jsonwebtoken. At 142b03b, moq-relay's build graph has no aws-lc-sys, ring, cc, cmake, or links key. So moq itself is now the pure-crypto build, and the comparison the spike corpus existed for (aws-lc-rs against pure Rust) no longer has two sides at main. The no-c guard moves to moq, and the workflow's moq pure-crypto job goes with the corpus.
  • Rebuild the test list from the workspace crates moq-relay builds at the pin. moq-clock and moq-flate are gone. kio-macros, moq-congestion, and moq-crypto are new. kio no longer needs its version. moq-pattern joins moq-auth as unnameable: moq-auth's dev-dependency on its published self brings a published moq-pattern at the same version, so cargo pkgid -p moq-pattern@0.1.2 is ambiguous. All 12 remaining specs resolve to workspace crates (cargo tree -p ... at the pin).
  • Keep the cranelift variant building moq. rind's naked_asm! takes sym operands, which cg_clif cannot compile yet (rust-lang/rustc_codegen_cranelift#1620, open). The first bench run at the new pin failed there. A variant gains a config key of cargo --config values, and cranelift picks its backend by profile, with LLVM for rind alone. A RUSTFLAGS backend can't do this: cargo passes RUSTFLAGS after the profile's per-package flag, so it would win. Each cargo command takes the values last: cargo drops a --config given before an external subcommand, so a leading one would have left nextest building with LLVM. The Cranelift retrial quest reads these moq rows, so dropping moq from the variant would have lost its signal.
  • quest/m3/cranelift-unwind.md: the failure count at the new pin (58, was 42), and the rind constraint a Cranelift build of moq now has.
  • README: the corpus notes, the edit-async note, the Cranelift note, and the ra-jit notes, taken from the run at the new pin. ra-jit serves every edit-leaf, edit-hub, and edit-sync run. It hands every edit-async run to cargo on usage-argv's parsed_values_given, at 50.6 s and 196 CPU-s against cargo's 10.5 s and 14.5 CPU-s. The aws-lc-sys fallback note goes with aws-lc. The cargo ra-jit check note is dated to f36c848, since nothing reruns it.

Impact

  • No public API. just bench --corpus moq-pure-crypto is now an unknown corpus.
  • Bench numbers on either side of this PR do not compare, as corpus.toml asks. The bump lands on its own.

Alternatives

  • The quest's recommendation: merge moq main into a new spike/pure-crypto branch in moq-dev/moq and pin that. I tried it. Merging main into the spike conflicts in 60 files across 1478 commits, because main replaced the same crypto code the spike patched (graviola became rind, built-in ACME was removed in moq-dev/moq#7803). Even resolved, it would reproduce what main already ships. I pushed nothing to moq-dev/moq.
  • Keep an aws-lc-rs variant of moq at main for a C-vs-Rust comparison. That would mean a new spike reverting moq's rind move. moq has already made that call, so the comparison has no decision left to inform.

Choices made without asking

  • Took the alternative above over the quest's recommended spike branch. The quest's premise, that the bump leaves moq-pure-crypto comparing a different tree, no longer holds, since moq is that tree.
  • Kept moq on the cranelift variant with rind on LLVM, rather than dropping moq from it until rust-lang/rustc_codegen_cranelift#1620 lands. The moq Cranelift numbers now include one crate built with LLVM, which is how a Cranelift user would build moq today.
  • Dropped moq-pattern's tests from test-* rather than naming it by a path+file:// spec, which would depend on the checkout path.

Validation

  • nix develop --command just check: passed.
  • nix develop --command just ci: only the two known path-dependent TypeId snapshots fail (lang/dyns ids.ts, lang/panics/hook lib.ts). They fail the same way on bare origin/main, and moq-dev/rust2#633 fixes them.
  • The bench at 010a935 (run): moq stable, moq ra-jit, and pure passed, so the no-c guard and the body edits hold at the new pin. moq nightly failed in the cranelift warm-up on rind, which led to the fix above.
  • Locally, the new config path on regex: just bench --corpus regex --variant nightly --variant cranelift --runs 1 passed, and cargo -v with the same values shows -Z codegen-backend=cranelift on every regex crate, with no warning for the unmatched rind. On moq's rind alone, rind gets llvm and its dependencies and build scripts get cranelift.
  • The bench at 5015a6e (run): all six jobs passed. At the new pin, cranelift builds moq 15.8% cheaper in CPU than nightly, and its test builds 18.6% cheaper. Test runs cost 26.9% more, and 58 of 6379 tests fail, from the same two causes as before: 27 catch a panic cg_clif builds as an abort, and 31 hit llvm.x86.sse2.pmulhu.w. Under LLVM, 1 test fails.

Follow-ups

None.

Models

Size: S
Implementer: Claude Code, claude-opus-5-5, high effort

(Written by Claude Opus 5.5 with high effort)

🤖 Generated with Claude Code

Review

Rewritten by quest review after each round.

  • Reviewer: Codex, gpt-6.1-sol, medium effort
  • Session: 01a1b524-ea63-75c2-8a20-9cad88c366a6, round 3
  • Head: 59c3a2a
  • Base: origin/main at efa6300
  • Score: 10/10, The new documentation matches the recorded benchmark results, the configuration-forwarding fix remains intact, and no actionable defects were found in the full branch diff.
  • Findings: none

Quest-Claim: 1b78c0bd989c3ff1
@pullfrog

pullfrog Bot commented Nov 7, 2026

Copy link
Copy Markdown

Pullfrog  | Review this ➔

kixelated and others added 4 commits November 6, 2026 23:31
…rypto

moq main moved moq-relay's crypto onto rind, so the moq corpus is now the
pure-crypto build: it takes moq-pure-crypto's no-c guard and that corpus goes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rind's naked_asm! takes sym operands, which cg_clif cannot compile yet
(rust-lang/rustc_codegen_cranelift#1620). A variant's new `config` key passes
cargo --config values, so the cranelift variant picks its backend by profile
and overrides it for rind; RUSTFLAGS would come after any per-package flag.
The README's ra-jit notes now come from the run at the new pin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cargo drops a --config given before an external subcommand, so nextest built
the cranelift variant's tests with LLVM. Every cargo command now takes the
config last, and `sub` holds only the subcommand again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 7, 2026 12:34

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed head: 59c3a2a
Base: efa6300

No actionable correctness findings.

Direction and coverage:

  • The new moq pin puts the async edit on Session::closed's poll_closed wrapper, matching the benchmark's updated description. Retiring the separate pure-crypto corpus and moving its no-C guard onto moq keeps the comparison aligned with the new corpus rather than retaining a different historical tree.
  • Variant config is rebuilt for each variant and passed as individually quoted trailing arguments in the warm-up, edit preparation, and timed cargo commands. This also reaches the external nextest subcommand. The Cranelift profile override leaves rind on LLVM instead of allowing a global backend RUSTFLAGS value to override it.
  • Documentation identifies the changed corpus and mixed-backend caveat, and keeps the Cranelift failures tracked rather than treating a successful benchmark workflow as passing tests.

Verification limits: GitHub-only static inspection of the diff, complete benchmark runner, and pinned moq session/profile configuration. Workflow run 42357544525 is recorded as successful at 5015a6e, but I did not execute builds, benchmarks, graph checks or tests, or independently reproduce its timing and failure-count measurements. COMMENT only, not approval.

(Written by review (OpenAI))

@coderabbitai

coderabbitai Bot commented Nov 7, 2026

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The build benchmark updates the moq corpus, removes the separate moq-pure-crypto corpus and its matrix entry, and configures Cranelift through Cargo profile settings. The runner now passes variant Cargo configuration to timed, warm-up, and edit-scenario commands. The documentation and quest notes record updated benchmark results and Cranelift limitations. The m4 quest no longer includes the moq-pin task.

Merge Risk: Minimal · up to 59c3a

The benchmark pin, corpus selection, and Cranelift configuration updates have no identified merge-blocking issue. Merge after normal checks pass.

Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title describes the moq benchmark pin update.
Description check Passed The description covers the pin update and related corpus, configuration, and documentation changes.
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 3 functions across 1 files. (3 skipped: 3 unsupported.)

Finishing Touches

Simplify code

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: ade0b724-6dec-4a82-abb4-e430284bd2d4
Commits

Reviewing files that changed from the base of the PR and between efa6300 and 59c3a2a.

Files selected for processing: 7

  • .github/workflows/bench.yml
  • bench/build/README.md
  • bench/build/corpus.toml
  • bench/build/run.sh
  • quest/m3/cranelift-unwind.md
  • quest/m4/README.md
  • quest/m4/bench-moq-pin.md
Files with no reviewable changes: 1

  • quest/m4/bench-moq-pin.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@kixelated
kixelated added this pull request to the merge queue Nov 7, 2026
Merged via the queue into main with commit c0f5344 Nov 7, 2026
13 checks passed
@kixelated
kixelated deleted the quest/m4/bench-moq-pin branch November 7, 2026 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant