Repository navigation
feat(reporting): add the reporting test kit, reference seller and pytest fixture - #1774
Conversation
…est fixture adcp.testing.reporting.reporting_test_kit runs a seller in process under a virtual clock with real buyer clients bound to it. kit.clock advances and runs the seller worker at every period boundary crossed; kit.grants revokes, regrants, reduces and winds down; kit.accounts sets a timezone; kit.buys adds and cancels buys. All of them go through the controller's reporting_subscription_probe and seed_media_buy, so a host can test its own platform by passing it with its service and controller. assert_agreement compares the seller's owed set with the buyer's reconcile and the served authorized_at with the one coverage used, and reports each disputed period with the first differing trace rule. The default seller is the SDK reference seller on the managed path, with an in-memory world of accounts, grants and buys and a controller serving seed_media_buy and reporting_subscription_probe. The adcp_reporting_kit fixture lives in the adcp.testing.reporting_pytest plugin. The kit found a seller bug, fixed here: served gaps after a timezone change were computed on the new zone's calendar, so they removed periods the owed set still owed. Gaps now use the zone in effect at the loss. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Ladon verdict: Approve
Approve — reporting test kit, reference seller, and pytest fixture additions.
Checked: The single production change (engine.py _schedules now zones each calendar via the existing _timezone_at helper, same arg order as coverage.py:1311) is a clean reuse-not-reimplement fix for the post-timezone-change gap bug, backed by _Account.timezone_changes. No import-layering, forward-compat, credential, generated-code, or semver violations — new exports are additive test helpers under adcp.testing (not task handlers, so no AGENTS.md handler-table entry owed). Local gates (ruff/mypy/pytest) presumed armed.
Medium findings
- src/adcp/testing/reporting.py:620 — Broad except in KitBuyer._call swallows model validation across all reporting reads, not just the get_media_buys fixture path its noqa cites; fail-open on assert_agreement's fidelity. Worth tightening, but single medium on a test-helper path.
Non-blocking aside: _authorization ternary at reporting.py:527 has both inner arms return "new" (dead branch, harmless).
One medium finding, no critical/high, no escalation triggers (high_risk false, gated_paths false, no team gate). Falls through to row 9 → approve.
Medium findings
- src/adcp/testing/reporting.py:620 — Broad except in KitBuyer._call swallows validation across all reporting reads
The kit buyer accepted any response that failed validation as a raw dict. Only get_media_buys keeps that fallback, for fixture buys carrying just what coverage reads; get_principal, list_accounts, get_reporting_status and the other reads reconcile and assert_agreement rely on now validate, so a malformed seller response fails instead of being compared. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Ladon verdict: Approve
Approve — clean pass, prior Medium resolved.
This is a subsequent review following a prior approve. The reviewer reports a clean delta of three correct fixes:
- events.py: moves
doorbell.clear()ahead of poll, fixing a lost-wakeup race (tested). - receipts.py: adds a stable receipt id plus an ID-only dedup arm so a rejection is stated once, and isolates verifier failures into
report.failed— surfaced through reconcile.py into the returned ReceiptReport (not warn-only-hidden). - testing/reporting.py: narrows the previously broad
excepttoget_media_buysfixtures only, resolving the prior Medium finding (src/adcp/testing/reporting.py:620).
No current Critical/High/Medium findings. The prior Medium is confirmed resolved — a net improvement.
Decision table: no critical/high (row 1 n/a); gated_paths false and review_decision APPROVED (rows 1-2 n/a); high_risk false (rows 3,5 n/a); no medium findings (rows 4,8 n/a); prior decision was approve, not escalate (row 6 n/a); no no-auto-approve team match (row 7 n/a). Falls through to row 9 → approve.
There was a problem hiding this comment.
Ladon verdict: Approve
Approve — clean pass, no blocking or medium findings.
The sole production change in src/adcp/reporting/subscriptions/engine.py (_schedules resolving each calendar via _timezone_at(now, ...) instead of account.timezone) is a correct narrowing fix: it moves a historical gap to the zone in effect at the loss while leaving the live-scheduling path unchanged, and is covered by DST/cutover agreement tests. The large testing/reporting.py kit was reviewed under the largest-file rule (bounded clock via MAX_STEPS, no external I/O, validated buyer reads). New adcp.testing exports are additive under a feat(reporting): prefix. No semver-signal, forward-compat, import-layering, generated-code, credential, or CI-gate concerns.
Decision: no critical/high/medium findings (table rows 1, 4–8 do not fire); high_risk is false (rows 3, 5); gated_paths is false and review_decision is APPROVED (row 2 satisfied); no no-auto-approve team match (row 7); prior decision was already approve (row 6 n/a). Falls through to row 9 → approve.
There was a problem hiding this comment.
Ladon verdict: Approve
Approve — additive reporting-subscription test kit plus a narrow, well-scoped production fix.
What I checked:
- No critical/high/medium findings in this run. Only one Low (dead ternary at
src/adcp/testing/reporting.py:529), which is omitted from blocking consideration. - No author no-auto-approve team match;
high_risk: false;gated_paths: falsewithreview_decision: APPROVED. No hard gate applies. - The production fix in
subscriptions/engine.py:_schedulesresolves each offering's calendar through_timezone_at(now, ...), mirroring the existingcoverage.pytimezone-change handling, and is exercised by the kit's DST/cutover agreement test. Referenced symbols (_timezone_at,_Account.timezone_changes,REPORTING_WIND_DOWN_TASKS) all exist in base. - Additive, experimental (AdCP 3.3) surface — no public-API break, no wire-pathway trigger, no semver-signal obligation.
- Prior decision was
approvewith no findings; this subsequent pass remains clean.
Decision table: rows 1–8 do not fire (no critical/high, gate satisfied, no deletions, no medium, no team gate, fewer than 3 mediums). Falls through to row 9 → approve.
…oved After a revoke, a re-grant and a later version cutover, the managed seller restated subscription_coverage gaps[] in the new version's boundaries, so a buyer applying them computed a different owed set from the seller's own engine and ledger. Gaps are now computed for the version in force at the loss, in the zone then in effect. The minimized repro from adcp-client-python#1770 is a test-kit test asserting agreement before and after the cutover. The probe's regrant chain value is also derived from the grant history rather than a dead branch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Ladon verdict: Approve
Approve — subsequent review, net improvement with no blocking findings.
This PR touches testing/reporting helpers, the subscriptions engine, and docs (7 files, +1318/-8). gated_paths is false and review_decision is APPROVED; no no-auto-approve team match; high_risk is false.
Decision trace (top-down):
- Row 1 (critical/high): no critical/high findings. Skip.
- Row 2 (gated_paths unsatisfied): gated_paths false. Skip.
- Row 3 (deleted high-risk): high_risk false. Skip.
- Row 4 (medium + data-loss/schema/infra): the single medium is a timezone-at-loss correctness gap in engine.py, not a data-loss/schema/infra category. Skip.
- Row 5 (modified high-risk + medium): high_risk false. Skip.
- Row 6 (sticky escalate): prior decision was approve, not escalate. Skip.
- Row 7 (no-auto-approve team): none. Skip.
- Row 8 (≥3 medium): only 1 medium. Skip.
- Row 9: approve.
Medium findings (non-blocking):
- src/adcp/reporting/subscriptions/engine.py:2129 —
_gapsuses the current account.timezone rather than the zone at loss, and the comment overclaims; the account-timezone-change path is untested. Worth a human glance but 1 medium with no escalation trigger falls through to approve.
Prior pass had no findings; this run surfaces one new medium on newly reviewed code, which per the subsequent-review rule is normal and not penalized.
Medium findings
- src/adcp/reporting/subscriptions/engine.py:2129 — _gaps uses current account.timezone, not the zone at loss (comment overclaims)
There was a problem hiding this comment.
Ladon verdict: Approve
Approve — subsequent pass is clean.
The prior Medium finding (engine.py _gaps using the current account.timezone rather than the zone in force at loss) is resolved correctly: _schedules now builds each calendar via _timezone_at(now, account.timezone, account.timezone_changes, item.offering_id) and _gaps passes _state_at(state, lost) + lost, so gaps are floored/ceiled on the schedule and zone in force at the loss. _state_at iterates the per-subscription prior_versions list (shape confirmed vs coverage.py) with a live-config fallback; the now=current callers are behavior-preserving for account-wide changes and strictly more correct for offering-specific ones.
The largest-file addition (src/adcp/testing/reporting.py, ~994 new lines) is an additive test kit of the right shape: the oracle fail-closes on every buyer read reconcile relies on, tolerating incomplete fixture buys only on get_media_buys; assert_agreement compares seller owed_periods against the buyer's over-the-wire reconcile() with best-effort trace diffing that augments rather than suppresses assertions. Correct semver signal (feat for the additive kit, fix for the engine bug), no wire/public-API break, no version-translation pathway owed (test-only + coverage bug fix). Tests cover grants/cutover/DST/cancellation agreement, a dropped-period failure, and the #1770 regression.
No current findings. high_risk false, gated_paths false (review_decision APPROVED anyway), no team gate. Rows 1–8 do not fire; falls through to row 9.
Slice P12 of the reporting-subscriptions DX plan (
.context/dx-reporting-subscriptions.md§2.6, §6.2). Stacked on #1773 (P14), which sits on #1772 (P13), #1768 (P10) and #1766 (P11).assert_agreementneeds the buyer'sreconcile.reporting_test_kitkit.buyer(principal)is a realReportingClientbound in process. It also exposes the raw tasks it reads, and it drains the seller's write triggers after each call so tests stay deterministic.kit.clock.advance("P3D")andadvance_to(...)move the one virtual clock. The seller runs its worker (sweep, then plan) at every period boundary crossed.kit.grants(revoke,regrant,reduce,wind_down),kit.accounts.set_timezoneandkit.buys(add,cancel) all go through the controller'sreporting_subscription_probeandseed_media_buy. That is the scenario a remote storyboard drives, soplatform=,service=andcontroller=let a host test its own wiring the same way.kit.assert_agreement(principal)compares the seller engine's ended owed periods with the buyer'sreconcile()over the wire. It also checks that theauthorized_atserved onlist_accountsis the one coverage used (release checklist §6.3).AgreementErrorlists each disputed period withdiff_coverage_tracesof both sides and both sentences.The reference seller
create_reference_reporting_seller(...)is the SDK's reference seller on the managed path, and the kit's default.ReferenceReportingSellerhandler serveslist_accountsandget_media_buysover an in-memoryReportingWorldof accounts, grants and buys.ReliableReportingService(subscriptions=SubscriptionOptions(...))installs everything else.ReportingTestControllerservesseed_media_buyand everyreporting_subscription_probeoperation:revoke_grant,regrant(returningchain,authorized_atandregranted_at),reduce_grant,set_account_timezone(returningprevious_timezoneand the per-offering cutovers), andadvance_time.pytest fixture
pytest_plugins = ["adcp.testing.reporting_pytest"]providesadcp_reporting_kit, configured by@pytest.mark.adcp_reporting_kit(**options).A seller bug the kit found, fixed here
After an account timezone change, the engine served
gaps[]for an earlier loss on the new zone's calendar. The buyer then removed periods the seller's owed set still owed, andassert_agreementfailed._schedulesnow builds each calendar in the zone in effect at the instant asked, which is the loss for a gap. The engine, beta3 and grants suites are green.Tests
tests/test_reporting_test_kit.py:Docs: guide 6, Testing with the reporting test kit, is written, and the README is updated. The anchors are unchanged.
🤖 Generated with Claude Code