[camd]

Route read-only endpoints to a PostgreSQL read replica

Summary

Adds an opt-in mechanism to serve a defined set of read-only GET endpoints from a PostgreSQL read replica instead of the primary, to relieve the primary of read load that competes with the data-ingestion write workload. Ships off by default and is controlled entirely by environment variables — no code change needed to turn it on or off.

How it works

• ReadReplicaRouter (treeherder/config/db_routing.py) — a Django DB router that sends reads to a read_replica alias when a thread-local flag is set, but only for models whose app label is in an allow-list ({"perf", "model"}). Everything else stays on the primary.
• ReadReplicaMixin — a viewset/APIView mixin that flips the flag for safe HTTP methods only (GET/HEAD/OPTIONS). Unsafe methods (POST/PUT/PATCH/DELETE) always use the primary, so a viewset can mix routed reads and primary writes safely.
• Auto-fallback — on a replica OperationalError/InterfaceError, the request is retried once against the primary and a db_routing_fallback WARNING is logged. A degraded replica means "served from primary," not errors.
• Kill switch — gated on READ_REPLICA_ENABLED; when off (or the replica URL is unset), the alias isn't registered and the router isn't installed (complete no-op). Celery, ETL, and management commands are untouched and always use the primary.

Endpoints routed

• Jobs view polling (the busiest read traffic): /api/jobs/, /api/project/{repo}/push/, /api/project/{repo}/jobs/{id}/ (detail + similar/text-log actions).
• Group summary: /api/groupsummary/ — a heavy day-wide multi-join aggregation, one of the most expensive reads in the app.
• Perfherder reads: (not as expensive, low hanging fruit): framework, signatures, platforms, job-data, data, bug-template, issue-tracker, validity-dashboard, summary, alertsummary-tasks, perfcompare results.

Perfherder mutating endpoints (alert summaries, alerts, tags) intentionally stay on the primary so a user's write is immediately visible on re-read.

Consistency / safety

The routed jobs/push/group/Perfherder-read paths have no synchronous read-after-write: job actions go through Taskcluster + ingestion (not a direct DB write the user re-reads), and job classification updates failure_classification_id and last_modified in the same row, so a lagging replica returns either the whole old row or the whole new row — no torn reads. Polling tolerates replica lag via the existing cursor safety buffer; worst case a new job/push appears one poll cycle later.

Enabling (for deploy/app-config owners)

Set both (both required to activate):

Variable	Value
READ_REPLICA_DATABASE_URL	replica DSN, same format as DATABASE_URL
READ_REPLICA_ENABLED	true

Verify from inside the app with ./manage.py dbshell --database=read_replica. Recommended rollout: enable in stage → soak several days → enable in prod. Kill switch is READ_REPLICA_ENABLED=false (no deploy). During soak, watch: primary DB CPU/connections (should drop), replica CPU (should rise — currently near-zero, ample headroom), and db_routing_fallback log volume (should stay ~0).

Testing

• Unit tests for the router and mixin (tests/config/test_db_routing.py) — allow-list routing, safe-vs-unsafe methods, state cleanup, one-shot fallback.
• End-to-end routing tests (tests/webapp/api/test_read_replica_routing.py) asserting per-alias query capture for the jobs, push, job-detail, group-summary, and Perfherder endpoints, plus a control proving a non-opted-in viewset stays on primary.
• Existing job/push/group/Perfherder API suites updated to run with both aliases (transaction=True, databases=["default", "read_replica"]) since routed reads use a separate connection that only sees committed data.

Follow-ups (not in this PR)

Once proven in prod, the next routing candidates are the Intermittent Failures views (/api/failures/, /api/failuresbybug/, /api/failurecount/) — heavy date-range aggregations, GET-only, lag-tolerant, all in the already-allow-listed model app.

Next candidates

• Tier 1 (do next): the Intermittent Failures views — Failures, FailuresByBug, FailureCount (intermittents_view.py). Heavy date-range Count/TruncDate aggregations, GET-only, highly lag-tolerant, all in the model app → no allow-list change. This is the standout.
• Tier 2: InfraCompareView, NoteViewSet, BugJobMapViewSet, HashViewSet.tocommit, FilesBugzillaMapViewSet — with a flagged read-after-write nuance on notes/bug-job-map (classify→view) to decide on first.
• Tier 3: small refdata endpoints — harmless but negligible.
• Not routable without an allow-list change: ChangelogViewSet (changelog app), UserViewSet (auth app).
• Key reassurance for the comment: every routable candidate is in {model, perf}, so they're all "add mixin + test," no router change.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 59.77860% with 109 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.27%. Comparing base (5acee2d) to head (79651ce).

Files with missing lines	Patch %	Lines
tests/config/test_db_routing.py	26.98%	92 Missing ⚠️
treeherder/config/db_routing.py	71.15%	15 Missing ⚠️
treeherder/config/settings.py	60.00%	2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (5acee2d) and HEAD (79651ce). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (5acee2d)	HEAD (79651ce)
	5	4

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #9528       +/-   ##
===========================================
- Coverage   82.98%   69.27%   -13.71%     
===========================================
  Files         670      669        -1     
  Lines       40875    41030      +155     
  Branches     3918     3933       +15     
===========================================
- Hits        33920    28425     -5495     
- Misses       6479    12129     +5650     
  Partials      476      476               

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[netlify]

✅ Deploy Preview for treeherder ready!

Name	Link
🔨 Latest commit	79651ce
🔍 Latest deploy log	https://app.netlify.com/projects/treeherder/deploys/6ad79c1b0935e1465c4edba3
😎 Deploy Preview	https://deploy-preview-9528--treeherder.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.