Skip to content

fix: real, storage-backed catalog for synced ad servers (AI-6) - #117

Open
Sirajmx wants to merge 5 commits into
mainfrom
feat/AI-6-real-catalog-for-synced-ad-servers
Open

Sirajmx wants to merge 5 commits into
mainfrom
feat/AI-6-real-catalog-for-synced-ad-servers

Conversation

@Sirajmx

@Sirajmx Sirajmx commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Problem

GET /products (and everything reading the catalog) served the static 13-product default catalog even after a real ad-server sync (GAM/FreeWheel/S3) had run. ProductSetupFlow.state.products is per-flow-instance and never outlives the request, and the catalog service's in-memory cache is per-worker-process (infra/docker/Dockerfile runs uvicorn with --workers 2), so at most one of two worker processes ever saw synced data. Docs claimed GAM/FreeWheel support that the code didn't deliver.

Fix

Three layers, all required together for a real GAM deployment to actually work:

  1. Catalog-serving layer — synced products now persist under a dedicated synced_product:{id} storage key (deliberately separate from the generic product:{id} key negotiation_service/media_kit_service already write, which is a narrower proposal-time snapshot) and are served fresh from storage on every catalog read when a real ad server is configured, matching how synced packages already worked. Stale products no longer re-seeded by a sync run are pruned instead of lingering as phantom inventory. CSV-mode and no-ad-server-configured behavior is unchanged.
  2. GAM client layer — GAMRestClient.connect() called googleapiclient.discovery.build("admanager", "v1", ...) with no discoveryServiceUrl. GAM self-hosts its own discovery document rather than registering centrally, so every real connection raised UnknownApiNameOrVersion and silently degraded to mocks. Fixed by pointing build() at admanager.googleapis.com/$discovery/rest?version=v1 directly.
  3. Deployment layer — infra/docker/Dockerfile only installed the redis extra; the gam extra (googleads/google-api-python-client/google-auth) was never in the shipped production image, so even with (1) and (2) fixed, the container would ImportError before ever reaching GAM. Fixed by installing ".[redis,gam]".

All three verified together end to end against a real GAM network with real service-account credentials, rebuilding the real Dockerfile from scratch with zero manual installs: list_inventory() returned 100 real ad units, POST /packages/sync completed with zero warnings, GET /products served 50 real synced products.

Not yet verified: FreeWheel's real-data path — no real FreeWheel credentials available. Its list_inventory() looks architecturally sound (real OAuth 2.1 MCP tool call) but unproven live.

Reconciliation with main

This branch had drifted behind main. Merged origin/main in, resolving conflicts in CHANGELOG.md, interfaces/api/deps.py, interfaces/api/main.py, interfaces/cli/main.py, and services/catalog_service.py — all were additive overlaps (new unrelated routes/fixes landing alongside this branch's own catalog work), not competing logic changes.

Reconciling against main's newer default_price_floor_cpm setting (AI-8) surfaced one real gap: tests/unit/test_synced_product_catalog.py's hand-rolled settings fixture predated that field, so the second sync in the prune test crashed before pruning ever ran — masking as a "pruning doesn't work" failure when the actual cause was an unrelated settings-fixture gap. Fixed by adding the field to the fixture.

Testing

  • Full suite green post-merge: 1726 unit+integration+smoke tests passed, 28 skipped (env-gated), 0 failed.
  • Booted the real server and confirmed /health and /products respond correctly post-merge.
  • No merge-conflict markers remain; diffed the merge result against an independent resolution to confirm consistency.

…g-for-synced-ad-servers

# Conflicts:
#	CHANGELOG.md
#	src/ad_seller/interfaces/api/deps.py
#	src/ad_seller/interfaces/api/main.py
#	src/ad_seller/interfaces/cli/main.py
#	src/ad_seller/services/catalog_service.py
@Sirajmx
Sirajmx marked this pull request as ready for review October 14, 2026 13:20
Kishore-2572
Kishore-2572 previously approved these changes Oct 14, 2026

@Kishore-2572 Kishore-2572 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Live-tested the safety-critical part given this codebase's history with #100: simulated an ad-server call failing mid-fetch with a real product already persisted — confirmed the real product survives, the failure path never reaches the pruning logic.

Closed the cross-repo gap live: drove the real unmodified persistence/serving functions with realistic fake GAM inventory into a real DB, started a real server against it, confirmed GET /products returns real synced products not the static catalog, then got a correct quote through real buyer-agent code end to end.

GAM discovery fix and Dockerfile gam extra both confirmed in code. Full suite: 1726 passed, 0 failed, matches the PR's numbers. Approving on substance — no blocking issues found.

@Kishore-2572
Kishore-2572 requested a review from atc964 October 14, 2026 13:24
@aleksUIX

Copy link
Copy Markdown

A successful sync that returns no inventory deletes the stored products, and the next catalog read serves the static defaults again. sync_from_ad_server still calls _finish_sync on that path, the prune removes every synced_product key, and an empty key set is treated as never synced. I ran a synthetic GAM sync: one real product was served, then an empty success made GET /products return the 13 demo products. A failed sync keeps the real product. Can we record a completed sync, including an empty one, and return an empty catalog?

…talog, not the static demo products (AI-6)

A successful sync that finds zero inventory items left zero
synced_product:* keys, indistinguishable from "never synced" --
build_synced_product_catalog() fell back to the 13-product static
demo catalog, silently serving fake products as if real.

Add a synced_catalog:completed marker, set only on the real
ad-server success path (never the mock path). The catalog is now
treated as real/synced if EITHER a synced_product:* key exists
(covers products persisted mid-sync, before the marker is set) OR
the completion marker is present (covers a completed sync that
genuinely found nothing) -- only "neither" still falls back to the
static catalog.

Found by aleksUIX in PR #117 review. Verified with a new unit
regression test plus a full e2e pass (real FastAPI app, real sqlite
storage, real sync method, only the GAM network call faked): sync
with items -> empty sync -> simulated transient failure -> recovery
sync, confirmed correct at each step, including from a second
independent server process reading the same storage.
…report it as failed (AI-6)

persist_synced_product/_upsert_synced_package write straight to
storage one item at a time inside the loop. An exception partway
through left every item already written by THIS run permanently in
storage, even though the sync is reported as failed -- contradicting
the existing claim that a failure never silently changes the synced
layer.

Capture each key's prior value before writing it, and on any
exception, roll back precisely: restore the prior value if the key
already existed (a re-synced item can share an id with one an
earlier successful sync persisted), or delete it if this run
introduced it fresh. Same fix applied to both the product loop and
the package loop (identical structural risk).

Reproduced live: synced A+B cleanly, then a sync that persists C and
crashes before D -- A and B survived as expected, but C leaked in
anyway. Fixed; added as two permanent regression tests (the leak
case, and the re-synced-existing-id-restores-correctly case).

Full suite: 1729 passed, 0 failed.
@Sirajmx

Sirajmx commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Good catch, thanks for catching this! Fixed — a sync completing with zero items now serves an honest empty catalog instead of falling back to the demo one, via a synced_catalog:completed marker tracking whether a real sync has ever run.

Also found and fixed a related issue while verifying: a sync that crashes partway through persisting items was leaving the already-persisted new items in storage despite being reported as failed. Now rolled back precisely on failure.

Both covered by new tests, verified end-to-end. Commits: 4751ceb, dec2b49.

@aleksUIX

Copy link
Copy Markdown

An empty successful sync now stays empty. A finished real sync sets synced_catalog:completed, including when the ad server returns no items, and the catalog read returns that empty catalog instead of the static defaults. The new test covers a one-product sync followed by an empty success.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants