Skip to content

fix(asap-tools): score SQL query results in calculate_fidelity - #896

Merged
milindsrivastava1997 merged 1 commit into
mainfrom
885-asap-tools-calculate_fidelity-never-scores-sql-queries
Oct 15, 2026
Merged

milindsrivastava1997 merged 1 commit into
mainfrom
885-asap-tools-calculate_fidelity-never-scores-sql-queries

Conversation

@milindsrivastava1997

Copy link
Copy Markdown
Contributor

Closes #885.

Problem: SQL results are stored as TabSeparated text in raw_text_result, and get_all_timeseries only read result, so calculate_fidelity never scored a SQL query.

Fix:

  • parse_tabseparated_result(query, raw_text) turns TSV rows into the same {frozenset(label → value): float} shape result uses.
  • Label names come from the query's outermost SELECT list. Bare columns are labels, and the single other item is the value.
  • QueryResult.keyed_result() returns result, or the parsed TSV; get_all_timeseries uses it. Parsing happens at scoring time, so loaded JSONL results are covered too.
  • SELECT-list names (not positions) mean a ClickHouse TSV exact result also matches an estimate in Prometheus JSON, e.g. from the rqe route (feat(asap-tools): rqe query protocol and frontend switch for e2e runs #890).
  • Malformed results raise instead of being skipped: wrong column count, duplicate label set, non-numeric value, or not exactly one aggregate.

Known limitation: ASAPQuery's ClickHouse adapter writes label columns in alphabetical order (KeyByLabelNames::new sorts them), while ClickHouse uses SELECT order. If a query's GROUP BY columns aren't alphabetical in its SELECT list, an ASAPQuery TSV estimate's series won't match the exact series and get skipped; in rare cases values could pair with the wrong labels. Answers through the rqe route are unaffected, since they carry label names. A follow-up could make the adapter emit SELECT order.

Also not covered:

  • the parquet serializer drops raw_text_result;
  • recommended_sketch_configs.py and kll_rank_error.py still read .result;
  • ClickHouse's escaping of tabs and backslashes inside values isn't undone.

Tests: asap-tools/experiments/tests/test_calculate_fidelity.py (13).

🤖 Generated with Claude Code

SQL results arrive as ClickHouse TabSeparated text in
QueryResult.raw_text_result with `result` left None, so
get_all_timeseries found no series and SQL queries were never scored.

Parse the TSV into the same {frozenset(label -> value): float} form as
`result`. TSV has no header, so label names come from the query's
outermost SELECT list (bare columns are labels, the single other item is
the value), read in SELECT order as ClickHouse returns them. Parsing
happens in get_all_timeseries via QueryResult.keyed_result(), so both
freshly-run and JSONL-loaded results are covered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gKTCAEz3sJ1Hzde16xxU1
@milindsrivastava1997 milindsrivastava1997 linked an issue Oct 15, 2026 that may be closed by this pull request
@milindsrivastava1997
milindsrivastava1997 merged commit 3ff12e2 into main Oct 15, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

asap-tools: calculate_fidelity never scores SQL queries

1 participant