[muhammad-ali-e]

Requested by ali · project thread

What

• New platform_metrics backend app serving GET /metrics in Prometheus text format (text/plain; version=0.0.4).
• Metrics, all unstract_-prefixed, no per-org labels:
  • unstract_queue_messages{queue,state} with state = ready | claimed | scheduled, and unstract_queue_oldest_message_age_seconds{queue} (from pg_queue_message)
  • unstract_workflow_executions{status} for pending / executing, and unstract_workflow_executions_recent{status} for completed / error / stopped that finished within a window (default 1h)
  • unstract_batches_in_progress, unstract_batches_stuck (from pg_barrier_state)
  • unstract_metrics_last_successful_refresh_timestamp{group}, unstract_metrics_refresh_failures_total{group} (one cluster-wide total kept in Redis), unstract_metrics_cache_up (0 while this process can't reach Redis)
• Known queues and statuses are always present and read 0 when empty. A queue outside the known list still shows up when it holds messages.

Why

• LLMWhisperer exposes queue metrics on its backend /metrics; Unstract had nothing equivalent. The PG reaper's /metrics doesn't split waiting from in-progress, has no execution counts, drops empty queues, and is empty on a standby reaper.

How

• Groups (groups.py): each MetricGroup has query() (DB, returns a JSON-safe snapshot) and families() (snapshot to Prometheus families). They're registered in METRIC_GROUPS, so a new group needs no change to the endpoint or the caching.
• Cache (cache.py): each group's snapshot lives in Redis for PLATFORM_METRICS_CACHE_TTL_SECONDS (15s), behind a non-blocking redis.lock. One pod refreshes a stale group and the rest serve the previous snapshot, so DB cost is about one refresh per group per 15s however many replicas get scraped. The cache is re-checked after the lock is taken, and the lock's expiry is sized from each group's statement count times the query timeout, so it can't expire mid-refresh.
• Resilience: a scrape always returns 200. If a refresh fails, the group serves its last snapshot and the failure counter goes up; the other groups aren't affected.
  • If Redis is down, or a Redis read or write fails, each pod serves its own newest copy and refreshes it at most once per TTL, so DB load stays bounded. The failure is still counted, and unstract_metrics_cache_up drops to 0.
  • Refresh failures are summed in Redis (HINCRBY + HGETALL in one transaction), so every pod and gunicorn worker reports the same monotonic total instead of a per-process count that would look like counter resets. While Redis is down the counter is left out of the scrape, and the buffered increments are flushed on the next successful one.
  • Each refresh runs under SET LOCAL statement_timeout (PLATFORM_METRICS_QUERY_TIMEOUT_MS, 5s), and the view is non_atomic_requests so that timeout never outlives the refresh.
• Consistency with the reaper: queue rows and oldest age use the same population as refresh_queue_gauges (all rows, now() - min(enqueued_at)), so the per-state counts add up to pg_queue_depth. batches_stuck is the reaper's stranded predicate, read from the same WORKER_PG_BATCH_STUCK_TIMEOUT_SECONDS. Queue metrics cover the PG transport only; work dispatched through Celery sits in RabbitMQ.
• Query cost (checked with EXPLAIN ANALYZE on 400k executions):
  • Active executions go through the partial we_active_by_workflow_idx.
  • Recent terminal executions are a we_created_at_idx range scan bounded by PLATFORM_METRICS_RECENT_LOOKBACK_SECONDS (24h). They are then filtered on the finish time, created_at + execution_time, which both status writers stamp on the terminal transition. modified_at is only the fallback, for rows terminalised without execution_time (the undispatched-execution sweep), because later writes such as result acknowledgement bump it.
  • The queue and barrier tables hold only in-flight rows, and the reaper already aggregates them the same way.
• Access:
  • The endpoint is mounted at the root next to /health. The public ingress only routes prefixed paths (/api/v1, /deployment, /public, …) to the backend, so /metrics is reachable in-cluster on the backend service only.
  • It is exempt from session auth (WHITELISTED_PATHS).
  • Setting PLATFORM_METRICS_TOKEN additionally requires Authorization: Bearer <token>, compared in constant time.

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

• No. It is a new route at a path no existing URL uses; the customer dashboard stays at /api/v1/unstract/<org>/metrics/. No execution path, model or migration changes. The only shared state is new Redis keys under platform_metrics:v1:*.

Database Migrations

• None.

Env Config

All optional:

• PLATFORM_METRICS_TOKEN (default empty, meaning no token check)
• PLATFORM_METRICS_CACHE_TTL_SECONDS (15)
• PLATFORM_METRICS_QUERY_TIMEOUT_MS (5000)
• PLATFORM_METRICS_RECENT_WINDOW_SECONDS (3600)
• PLATFORM_METRICS_RECENT_LOOKBACK_SECONDS (86400)
• PLATFORM_METRICS_QUEUES: comma-separated list of always-reported queues. It defaults to the queues the PG consumers in docker-compose.yaml consume. Cloud should override it if it runs extra PG queues.
• The existing WORKER_PG_BATCH_STUCK_TIMEOUT_SECONDS is now also read by the backend.

Relevant Docs

• Jira UN-4251. LLMWhisperer backend/app/controller/metrics.py was the reference pattern.

Related Issues or PRs

• UN-3672 (UN-3672 [FEAT] PG queue — application-level /metrics exporter (queue depth, message age, barriers, reaper outcomes) #2143), the reaper's /metrics, which this PR leaves unchanged. Endpoints for autoscaling are a separate ticket. PodMonitoring/scrape config in the cloud chart is out of scope here.

Dependencies Versions

• prometheus-client>=0.17.0,<1.0.0 added to the backend (same range the workers already pin; locks to 0.26.0).

Notes on Testing

• backend/platform_metrics/tests/, 43 tests:
  • 38 unit tests, with no DB or Redis. They cover:
    • output format and metric names
    • zero-filling
    • per-group failure isolation
    • cache/lock behaviour: fresh, stale, lock held elsewhere, re-check under the lock, Redis down, a failed re-read under the lock, a failed Redis write, an unreadable entry, lock sizing
    • the shared failure counter across processes, including a Redis outage and an unreadable hash value
    • access control
  • 5 DB tests (auto-marked integration) run against Postgres 16. They check each group's counts and edges:
    • the oldest age across states
    • window and lookback bounds, including a completion touched after it finished and swept rows
    • the stuck predicate, including remaining=0 lingerers
    • the statement timeout cancelling a slow query
• Smoke test with runserver against Postgres + Redis:
  • Requests without the token get 401.
  • Two scrapes in a row are served from the same Redis snapshot.
  • With Redis shut down, a scrape still returns 200 in about 10ms and the failure counter goes up.
• On a PodMonitoring setup, aggregate with max(), not sum(): every pod serves the same shared snapshot, and the failure counter is the same cluster-wide total on every pod.

Screenshots

Checklist

I have read and understood the Contribution Guidelines.

🤖 Generated with Claude Code

https://claude.ai/code/session_016Hov58G5waUqpMHz9P6FmL

[greptile-apps]

via Greptile

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge, with a non-blocking issue in how Redis retries count refresh failures.

Findings

1.  Refresh failures counted twice ▶

Fix with agent prompt

### Issue 1
backend/platform_metrics/cache.py:309-311
If Redis applies the transaction but the connection drops before its reply arrives, `pipe.execute()` raises and `_restore(pending)` queues the same increments again. The next successful scrape counts those failures twice, inflating `unstract_metrics_refresh_failures_total` and potentially triggering unnecessary alerts.

Make retries safe to repeat, for example by storing each process's cumulative count and adding only its unseen increase.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds a backend /metrics endpoint for queue, execution, and batch counts, with shared Redis snapshots and an optional bearer token.

• The metrics show queue, workflow, and batch activity.
• Prometheus can read platform metrics from the backend.
• Metric scrapes reuse cached snapshots and report refresh health.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["GET /metrics"] --> B{"Token accepted?"}
    B -->|No| C["401"]
    B -->|Yes| D["Resolve each metric group"]
    D --> E{"Fresh snapshot?"}
    E -->|Yes| H["Render metrics"]
    E -->|No| F["Take Redis refresh lock"]
    F --> G["Query database or serve saved snapshot"]
    G --> H
    H --> I["Flush and read shared failure totals"]
    I --> J["Return Prometheus text"]

Loading

_{Reviews (3) · Last reviewed commit: "UN-4251 Share refresh-failure totals acr..." · Reviewed by Greptile}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[greptile-apps]

Refresh failures counted twice

If Redis applies the transaction but the connection drops before its reply arrives, pipe.execute() raises and _restore(pending) queues the same increments again. The next successful scrape counts those failures twice, inflating unstract_metrics_refresh_failures_total and potentially triggering unnecessary alerts.

Make retries safe to repeat, for example by storing each process's cumulative count and adding only its unseen increase.

Prompt To Fix With AI

This is a comment left during a code review.
Path: backend/platform_metrics/cache.py
Line: 309-311

Comment:
**Refresh failures counted twice**

If Redis applies the transaction but the connection drops before its reply arrives, `pipe.execute()` raises and `_restore(pending)` queues the same increments again. The next successful scrape counts those failures twice, inflating `unstract_metrics_refresh_failures_total` and potentially triggering unnecessary alerts.

Make retries safe to repeat, for example by storing each process's cumulative count and adding only its unseen increase.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[muhammad-ali-e]

Acknowledged, but I'm not changing this. Full reasoning is in #2378 (comment).

In short: the double count needs EXEC to succeed and its reply to be lost. When that happens, the counter over-reports by at most one scrape's increments for one process, and the error only ever goes up. Tracking a per-process cumulative count would leave an orphaned hash field behind every time a gunicorn worker is recycled. For a diagnostic counter that alerts on rate() > 0 / cache_up == 0, that bookkeeping costs more than the rare over-count it prevents.

---

Generated by Claude Code

[muhammad-ali-e]

Re Greptile P2 "Refresh failures counted twice" (cache.py:309-311): acknowledged, not changing this.

You're right that it can happen. Redis has to apply the MULTI/EXEC and then lose the reply on the way back, and in that case those increments are counted again on the next scrape. That window is narrow: it needs a connection drop between EXEC and its reply. The effect is also bounded and only goes up. The failure count over-reports by at most one scrape's worth of increments for this process.

The suggested fix is to store a cumulative count per process and add only the unseen delta. That needs a stable key for each process, and gunicorn recycles workers, so every recycle would leave behind a hash field that is never cleaned up. Summing those fields would also need the cleanup logic we'd have to build alongside it. For a diagnostic counter, where alerts should key on rate() > 0 or on unstract_metrics_cache_up == 0 rather than on exact totals, an occasional small over-count is the better trade than that bookkeeping.

---

Generated by Claude Code

[github-actions]

Unstract test results

Per-group results

Status	Group	Tier	Passed	Failed	Errors	Skipped	Duration (s)
⚪	e2e-agent-kv	e2e	0	0	0	1	0.1
✅	e2e-api-deployment	e2e	3	0	0	0	12.3
✅	e2e-coowners	e2e	1	0	0	0	1.3
✅	e2e-etl	e2e	1	0	0	0	8.7
✅	e2e-login	e2e	2	0	0	0	1.1
✅	e2e-prompt-studio	e2e	1	0	0	0	11.8
✅	e2e-smoke	e2e	2	0	0	0	1.0
✅	e2e-workflow	e2e	1	0	0	0	20.3
✅	frontend	unit	629	0	0	0	18.3
✅	integration-backend	integration	1425	0	0	26	65.7
✅	integration-connectors	integration	1	0	0	7	8.0
✅	integration-workers	integration	181	0	0	1	45.6
❌	ui	e2e	0	1	0	0	0.0
✅	unit-agent-kv-schema	unit	143	0	0	0	1.7
✅	unit-backend	unit	2071	0	0	1	58.1
✅	unit-connectors	unit	82	0	0	0	11.3
✅	unit-core	unit	281	0	0	0	3.2
✅	unit-filesystem	unit	1	0	0	0	4.4
✅	unit-platform-service	unit	18	0	0	0	2.7
✅	unit-rig	unit	120	0	0	0	4.4
✅	unit-runner	unit	12	0	0	0	3.0
✅	unit-sdk1	unit	878	0	0	0	41.4
✅	unit-tool-sidecar	unit	8	0	0	0	1.8
✅	unit-workers	unit	1534	0	0	2	130.1
	TOTAL		7395	1	0	38	456.2

Critical paths

⚠️ Critical paths not yet covered

• workflow-execution-fan-out — Multi-file workflow execution fans out to file-processing workers and rejoins. (declared coverage: no groups declared)

✅ Covered critical paths

• auth-login — covered by e2e-login
• adapter-register-llm — covered by integration-backend
• workflow-author — covered by integration-backend
• co-owner-manage — covered by integration-backend, e2e-coowners
• workflow-create-execute — covered by e2e-workflow
• api-deployment-provision — covered by integration-backend
• api-deployment-auth — covered by integration-backend
• api-deployment-run — covered by e2e-api-deployment
• mcp-server-auth — covered by integration-backend
• mcp-platform-auth — covered by integration-backend
• platform-key-whoami — covered by integration-backend
• prompt-studio-author — covered by integration-backend
• prompt-studio-fetch-response — covered by e2e-prompt-studio
• connector-register-test — covered by integration-backend
• pipeline-etl-execute — covered by e2e-etl
• usage-aggregate-read — covered by integration-backend
• usage-token-tracking — covered by e2e-api-deployment
• callback-result-delivery — covered by e2e-api-deployment