Skip to content

feat(reporting): add the reporting test kit, reference seller and pytest fixture - #1774

Merged
bokelley merged 8 commits into
nextfrom
claude/issue-1695-p12-test-kit
Nov 1, 2026
Merged

bokelley merged 8 commits into
nextfrom
claude/issue-1695-p12-test-kit

Conversation

@bokelley

@bokelley bokelley commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Slice P12 of the reporting-subscriptions DX plan (.context/dx-reporting-subscriptions.md §2.6, §6.2). Stacked on #1773 (P14), which sits on #1772 (P13), #1768 (P10) and #1766 (P11). assert_agreement needs the buyer's reconcile.

reporting_test_kit

async with reporting_test_kit(start=..., offerings=[offerings.daily_analytics(alignment="account_timezone")],
                              accounts=[...], buys=[...]) as kit:
    buyer = kit.buyer("agency-x")
    await buyer.reporting.subscribe(analytics_daily)
    await kit.clock.advance("P3D")
    await kit.grants.revoke("agency-x", "acc_a")
    await kit.assert_agreement("agency-x")
  • kit.buyer(principal) is a real ReportingClient bound in process. It also exposes the raw tasks it reads, and it drains the seller's write triggers after each call so tests stay deterministic.
  • kit.clock.advance("P3D") and advance_to(...) move the one virtual clock. The seller runs its worker (sweep, then plan) at every period boundary crossed.
  • kit.grants (revoke, regrant, reduce, wind_down), kit.accounts.set_timezone and kit.buys (add, cancel) all go through the controller's reporting_subscription_probe and seed_media_buy. That is the scenario a remote storyboard drives, so platform=, service= and controller= let a host test its own wiring the same way.
  • kit.assert_agreement(principal) compares the seller engine's ended owed periods with the buyer's reconcile() over the wire. It also checks that the authorized_at served on list_accounts is the one coverage used (release checklist §6.3). AgreementError lists each disputed period with diff_coverage_traces of both sides and both sentences.

The reference seller

create_reference_reporting_seller(...) is the SDK's reference seller on the managed path, and the kit's default.

  • A ~100-line ReferenceReportingSeller handler serves list_accounts and get_media_buys over an in-memory ReportingWorld of accounts, grants and buys. ReliableReportingService(subscriptions=SubscriptionOptions(...)) installs everything else.
  • ReportingTestController serves seed_media_buy and every reporting_subscription_probe operation: revoke_grant, regrant (returning chain, authorized_at and regranted_at), reduce_grant, set_account_timezone (returning previous_timezone and the per-offering cutovers), and advance_time.
  • PR 6 (next in the stack) serves it standalone and runs the TypeScript storyboard against it.

pytest fixture

pytest_plugins = ["adcp.testing.reporting_pytest"] provides adcp_reporting_kit, configured by @pytest.mark.adcp_reporting_kit(**options).

A seller bug the kit found, fixed here

After an account timezone change, the engine served gaps[] for an earlier loss on the new zone's calendar. The buyer then removed periods the seller's owed set still owed, and assert_agreement failed. _schedules now builds each calendar in the zone in effect at the instant asked, which is the loss for a gap. The engine, beta3 and grants suites are green.

Tests

tests/test_reporting_test_kit.py:

  • Agreement across one scenario: revoke and regrant (continued chain), a timezone cutover, a cancellation, and a delivered status.
  • Wind-down keeps coverage.
  • DST: the 2026-11-01 New York day is owed as a 25-hour period.
  • A seller that drops an owed period fails agreement with traces.
  • Custom-platform validation.
  • The pytest fixture with a marker.

Docs: guide 6, Testing with the reporting test kit, is written, and the README is updated. The anchors are unchanged.

🤖 Generated with Claude Code

…est fixture

adcp.testing.reporting.reporting_test_kit runs a seller in process under
a virtual clock with real buyer clients bound to it. kit.clock advances
and runs the seller worker at every period boundary crossed; kit.grants
revokes, regrants, reduces and winds down; kit.accounts sets a timezone;
kit.buys adds and cancels buys. All of them go through the controller's
reporting_subscription_probe and seed_media_buy, so a host can test its own
platform by passing it with its service and controller. assert_agreement
compares the seller's owed set with the buyer's reconcile and the served
authorized_at with the one coverage used, and reports each disputed period
with the first differing trace rule.

The default seller is the SDK reference seller on the managed path, with
an in-memory world of accounts, grants and buys and a controller serving
seed_media_buy and reporting_subscription_probe. The adcp_reporting_kit
fixture lives in the adcp.testing.reporting_pytest plugin.

The kit found a seller bug, fixed here: served gaps after a timezone change
were computed on the new zone's calendar, so they removed periods the owed
set still owed. Gaps now use the zone in effect at the loss.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Comment thread src/adcp/testing/reporting.py Resolved

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — reporting test kit, reference seller, and pytest fixture additions.

Checked: The single production change (engine.py _schedules now zones each calendar via the existing _timezone_at helper, same arg order as coverage.py:1311) is a clean reuse-not-reimplement fix for the post-timezone-change gap bug, backed by _Account.timezone_changes. No import-layering, forward-compat, credential, generated-code, or semver violations — new exports are additive test helpers under adcp.testing (not task handlers, so no AGENTS.md handler-table entry owed). Local gates (ruff/mypy/pytest) presumed armed.

Medium findings

  • src/adcp/testing/reporting.py:620 — Broad except in KitBuyer._call swallows model validation across all reporting reads, not just the get_media_buys fixture path its noqa cites; fail-open on assert_agreement's fidelity. Worth tightening, but single medium on a test-helper path.

Non-blocking aside: _authorization ternary at reporting.py:527 has both inner arms return "new" (dead branch, harmless).

One medium finding, no critical/high, no escalation triggers (high_risk false, gated_paths false, no team gate). Falls through to row 9 → approve.

Medium findings

  • src/adcp/testing/reporting.py:620 — Broad except in KitBuyer._call swallows validation across all reporting reads

bokelley and others added 2 commits November 1, 2026 18:14
The kit buyer accepted any response that failed validation as a raw dict.
Only get_media_buys keeps that fallback, for fixture buys carrying just what
coverage reads; get_principal, list_accounts, get_reporting_status and the
other reads reconcile and assert_agreement rely on now validate, so a
malformed seller response fails instead of being compared.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — clean pass, prior Medium resolved.

This is a subsequent review following a prior approve. The reviewer reports a clean delta of three correct fixes:

  • events.py: moves doorbell.clear() ahead of poll, fixing a lost-wakeup race (tested).
  • receipts.py: adds a stable receipt id plus an ID-only dedup arm so a rejection is stated once, and isolates verifier failures into report.failed — surfaced through reconcile.py into the returned ReceiptReport (not warn-only-hidden).
  • testing/reporting.py: narrows the previously broad except to get_media_buys fixtures only, resolving the prior Medium finding (src/adcp/testing/reporting.py:620).

No current Critical/High/Medium findings. The prior Medium is confirmed resolved — a net improvement.

Decision table: no critical/high (row 1 n/a); gated_paths false and review_decision APPROVED (rows 1-2 n/a); high_risk false (rows 3,5 n/a); no medium findings (rows 4,8 n/a); prior decision was approve, not escalate (row 6 n/a); no no-auto-approve team match (row 7 n/a). Falls through to row 9 → approve.

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — clean pass, no blocking or medium findings.

The sole production change in src/adcp/reporting/subscriptions/engine.py (_schedules resolving each calendar via _timezone_at(now, ...) instead of account.timezone) is a correct narrowing fix: it moves a historical gap to the zone in effect at the loss while leaving the live-scheduling path unchanged, and is covered by DST/cutover agreement tests. The large testing/reporting.py kit was reviewed under the largest-file rule (bounded clock via MAX_STEPS, no external I/O, validated buyer reads). New adcp.testing exports are additive under a feat(reporting): prefix. No semver-signal, forward-compat, import-layering, generated-code, credential, or CI-gate concerns.

Decision: no critical/high/medium findings (table rows 1, 4–8 do not fire); high_risk is false (rows 3, 5); gated_paths is false and review_decision is APPROVED (row 2 satisfied); no no-auto-approve team match (row 7); prior decision was already approve (row 6 n/a). Falls through to row 9 → approve.

Comment thread src/adcp/testing/reporting.py Resolved

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — additive reporting-subscription test kit plus a narrow, well-scoped production fix.

What I checked:

  • No critical/high/medium findings in this run. Only one Low (dead ternary at src/adcp/testing/reporting.py:529), which is omitted from blocking consideration.
  • No author no-auto-approve team match; high_risk: false; gated_paths: false with review_decision: APPROVED. No hard gate applies.
  • The production fix in subscriptions/engine.py:_schedules resolves each offering's calendar through _timezone_at(now, ...), mirroring the existing coverage.py timezone-change handling, and is exercised by the kit's DST/cutover agreement test. Referenced symbols (_timezone_at, _Account.timezone_changes, REPORTING_WIND_DOWN_TASKS) all exist in base.
  • Additive, experimental (AdCP 3.3) surface — no public-API break, no wire-pathway trigger, no semver-signal obligation.
  • Prior decision was approve with no findings; this subsequent pass remains clean.

Decision table: rows 1–8 do not fire (no critical/high, gate satisfied, no deletions, no medium, no team gate, fewer than 3 mediums). Falls through to row 9 → approve.

bokelley and others added 2 commits November 1, 2026 18:52
…oved

After a revoke, a re-grant and a later version cutover, the managed seller
restated subscription_coverage gaps[] in the new version's boundaries, so
a buyer applying them computed a different owed set from the seller's own
engine and ledger. Gaps are now computed for the version in force at the
loss, in the zone then in effect. The minimized repro from
adcp-client-python#1770 is a test-kit test asserting agreement before and
after the cutover. The probe's regrant chain value is also derived from the
grant history rather than a dead branch.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — subsequent review, net improvement with no blocking findings.

This PR touches testing/reporting helpers, the subscriptions engine, and docs (7 files, +1318/-8). gated_paths is false and review_decision is APPROVED; no no-auto-approve team match; high_risk is false.

Decision trace (top-down):

  • Row 1 (critical/high): no critical/high findings. Skip.
  • Row 2 (gated_paths unsatisfied): gated_paths false. Skip.
  • Row 3 (deleted high-risk): high_risk false. Skip.
  • Row 4 (medium + data-loss/schema/infra): the single medium is a timezone-at-loss correctness gap in engine.py, not a data-loss/schema/infra category. Skip.
  • Row 5 (modified high-risk + medium): high_risk false. Skip.
  • Row 6 (sticky escalate): prior decision was approve, not escalate. Skip.
  • Row 7 (no-auto-approve team): none. Skip.
  • Row 8 (≥3 medium): only 1 medium. Skip.
  • Row 9: approve.

Medium findings (non-blocking):

  • src/adcp/reporting/subscriptions/engine.py:2129 — _gaps uses the current account.timezone rather than the zone at loss, and the comment overclaims; the account-timezone-change path is untested. Worth a human glance but 1 medium with no escalation trigger falls through to approve.

Prior pass had no findings; this run surfaces one new medium on newly reviewed code, which per the subsequent-review rule is normal and not penalized.

Medium findings

  • src/adcp/reporting/subscriptions/engine.py:2129 — _gaps uses current account.timezone, not the zone at loss (comment overclaims)

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — subsequent pass is clean.

The prior Medium finding (engine.py _gaps using the current account.timezone rather than the zone in force at loss) is resolved correctly: _schedules now builds each calendar via _timezone_at(now, account.timezone, account.timezone_changes, item.offering_id) and _gaps passes _state_at(state, lost) + lost, so gaps are floored/ceiled on the schedule and zone in force at the loss. _state_at iterates the per-subscription prior_versions list (shape confirmed vs coverage.py) with a live-config fallback; the now=current callers are behavior-preserving for account-wide changes and strictly more correct for offering-specific ones.

The largest-file addition (src/adcp/testing/reporting.py, ~994 new lines) is an additive test kit of the right shape: the oracle fail-closes on every buyer read reconcile relies on, tolerating incomplete fixture buys only on get_media_buys; assert_agreement compares seller owed_periods against the buyer's over-the-wire reconcile() with best-effort trace diffing that augments rather than suppresses assertions. Correct semver signal (feat for the additive kit, fix for the engine bug), no wire/public-API break, no version-translation pathway owed (test-only + coverage bug fix). Tests cover grants/cutover/DST/cancellation agreement, a dropped-period failure, and the #1770 regression.

No current findings. high_risk false, gated_paths false (review_decision APPROVED anyway), no team gate. Rows 1–8 do not fire; falls through to row 9.

Base automatically changed from claude/issue-1695-p14-receipts to next November 1, 2026 22:14
@bokelley
bokelley merged commit 0dac622 into next Nov 1, 2026
41 checks passed
@bokelley
bokelley deleted the claude/issue-1695-p12-test-kit branch November 1, 2026 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant