[KyPython]

Wraps Datadog's ARFBench (Anomaly Reasoning Framework Benchmark, Apache 2.0) as an inspect_ai task so a real vendor benchmark runs through our own eval harness.

• 750 multiple-choice QA pairs from 63 real Datadog production incidents (8 categories, 3 difficulty tiers)
• v2 adds compact per-question time-series context (statistical summaries, not raw dumps) with -T series_context=true/false
• Validated with mockllm; needs a real model API key for scored runs
• Dataset: https://huggingface.co/datasets/Datadog/ARFBench (public); eval code: https://github.com/DataDog/ARFBench

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-21T23:48:27.905930Z	3664150	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: c2afc63c-8e83-4c5c-afdb-463fac889165

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f0420fc3d2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Commit actual gzip bytes instead of base64 text

The committed blob begins with H4sI..., which is base64 text representing a gzip stream rather than a gzip file. Consequently, every invocation reaches _load_rows() and gzip.open() raises BadGzipFile: Not a gzipped file (b'H4'), including runs with series_context=false; decode the artifact before committing it.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Declare dependencies used by the default mode

On a clean checkout installed from the root requirements.txt, the default invocation always fails here because neither pandas nor pyarrow is declared; the README's referenced sibling virtualenv also does not exist in this repository. Since series_context defaults to true, add the required packages to the reproducible environment or make the dependency-free mode the default.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Add regression coverage for the new task

This new task adds dataset loading, filtering, context generation, and scoring behavior without adding any automated test in the commit. At minimum, construct the task from the committed dataset in both context modes so packaging errors such as the unreadable gzip artifact are caught before merge.

AGENTS.md reference: AGENTS.md:L18-L18

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Memoize unavailable time-series groups

When a group has no remote parquet file, this path leaves no cached negative result, so every later sample using the same group retries all three URLs. The dataset contains heavily repeated groups and the README states that most groups lack time-series data, so a full default run performs thousands of redundant network requests before model evaluation; cache unavailable groups for the task lifetime.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Pin the remote dataset revision

Resolving every parquet from the mutable main ref makes benchmark inputs depend on cache state and run date: a clean machine fetches the latest bytes while an existing cache retains older bytes, so identical commits and model settings can produce different scores. The upstream dataset has already used replacement uploads, as shown by its re-upload commit; pin a dataset commit and verify downloaded checksums.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Include the Apache license with redistributed artifacts

The commit redistributes Datadog's Apache-2.0-licensed QA and time-series artifacts, but the repository contains no LICENSE file; merely naming the license here does not provide recipients a copy. Apache License 2.0 section 4(a) explicitly requires redistributions to include the license, so add its full text and retain any upstream attribution notices that apply. Apache License 2.0

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Preserve temporal precision for timestamp questions

The loader never considers the available 10- or 60-second files and starts at five-minute bars, even though the committed start/end questions contain materially closer choices—for example, row 3 asks the model to distinguish 18:19:10 from 18:19:43. Those alternatives collapse into the same five-minute observation, and the later summary further renders the peak only as HH:MM, so these samples cannot be answered from the supplied context; select a resolution based on the timestamp choices and retain full timestamp precision.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Supply the benchmark's task instructions

The task invokes only Inspect's generic multiple-choice solver, so models never receive ARFBench's definitions of anomaly magnitude, the six anomaly categories, correlation, or leading/lagging indicators. Datadog's upstream benchmark prompt supplies these definitions on every evaluation; omitting them changes what the 104 categorization and 76 magnitude samples ask the model to infer and makes scores incomparable even after accounting for the compact time-series representation.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2d99b902a8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Fail on transient time-series download errors

When Hugging Face is temporarily unreachable or returns a timeout, DNS/TLS error, or 5xx response, this broad handler treats the failure exactly like an absent dataset file. _summarize_ts() then returns no context and the default evaluation silently continues with description-only prompts, so an outage can produce a complete but invalid benchmark score. Only expected not-found responses should trigger the fallback; transient and infrastructure failures should be retried or surfaced.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Keep downloaded cache files out of the worktree

Whenever the default mode successfully fetches a previously uncached group, it writes the parquet into evals/arfbench/data/ts, but that generated path is not covered by the repository's .gitignore. A normal benchmark run therefore leaves many large untracked artifacts in the checkout and makes them easy to commit accidentally; use a cache directory outside the repository or ignore the generated parquet files while explicitly retaining any intended fixture.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Keep each sample's shuffle stable across tier filters

Because tier filtering occurs before this call, shuffle_choices(seed=42) advances its single RNG over a different sequence of samples in a tier-only run than in the full run. The same Tier 2 or Tier 3 question therefore receives a different option permutation when evaluated alone versus as part of all 750 samples, so residual model position bias can change per-tier scores solely because of the invocation mode. Derive the permutation from the sample ID or otherwise shuffle each sample independently so its ordering remains stable across filters.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Reject invalid series_context values

For string-valued task parameters, every value other than true, 1, or yes is silently interpreted as false. A typo such as -T series_context=ture therefore disables the time-series context and produces a successful description-only evaluation instead of reporting the invalid configuration, which can yield misleading benchmark results. Recognize an explicit set of true and false spellings and raise for anything else.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 14a0812d39

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Decode the committed parquet cache artifact

The committed file contains base64 text (UEFSM...) rather than a parquet stream (PAR1). When the default run reaches query group 35928_0, pd.read_parquet() rejects the tracked artifact and the exception handler deletes it; an offline run consequently loses the intended context, while an online run replaces the tracked file with downloaded binary data and dirties the worktree. Decode this artifact before committing it, as _load_rows() already does for the similarly encoded CSV.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Reject nonpositive limit values

When a caller supplies a negative task limit, Python slicing interprets it relative to the end, so -T limit=-1 evaluates 749 samples instead of rejecting the invalid maximum. Because real runs are explicitly described as incurring API cost, a mistaken negative value can unexpectedly launch almost the entire benchmark; validate that limit is positive before slicing.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f6196b9869

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Validate the parquet engine before enabling context

When pandas is installed but neither pyarrow nor another parquet engine is available, this dependency guard succeeds, but every downstream pd.read_parquet() raises an ImportError; _fetch_ts() catches that exception as if the cache were corrupt, deletes each downloaded file, and ultimately runs the benchmark without time-series context. Check for the required parquet engine here so this environment fails clearly instead of silently producing description-only scores.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Require both series before emitting paired context

For a correlation or leading/lagging sample where only one comma-separated query group has usable parquet data, this loop silently drops the missing group and returns the surviving block. _to_sample() then marks the sample as having series context and omits the missing-data note, even though these questions require comparing anomalies in both series; the resulting one-sided prompt can produce a successful but invalid benchmark score. Treat paired context as all-or-none, or explicitly surface the missing side.

Useful? React with 👍 / 👎.

[KyPython]

Codex review remediation — all 12 threads addressed

Each finding below is fixed in 1bf6d9f9 / f6196b98 (CI 7/7 green, new regression suite evals/arfbench/test_arfbench.py passes):

1. Base64 gzip blob → _load_rows() now accepts real gzip or base64 text (the push path can't carry binary). Verified: committed artifact loads 750 rows.
2. Undeclared pandas/pyarrow → added to root requirements.txt (pandas==3.0.6, pyarrow==26.0.0); README's sibling-venv reference replaced.
3. No regression tests → evals/arfbench/test_arfbench.py (7 tests: data loads, task constructs in both context modes with ts downloads stubbed, shuffle stability, position-hack invariant, param validation, fine-resolution detection); wired into tests/test_eval_suites.py.
4. Memoize unavailable groups → _TS_UNAVAILABLE negative cache for the task lifetime.
5. Pin dataset revision → HF_REVISION = 1cc8ae5... (== main as of 2026-10-19, so existing caches stay valid).
6. Apache-2.0 license text → evals/arfbench/LICENSE with attribution to Datadog/ARFBench.
7. Timestamp precision → questions naming HH:MM:SS timestamps now try 10/60s data first and render peaks with second precision; coarser questions unchanged.
8. Missing task instructions → TASK_INSTRUCTIONS (distilled from Datadog's upstream benchmark prompt) prepended to every prompt.
9. Silent fallback on transient errors → only HTTP 404 falls back; timeouts/DNS/5xx now raise instead of producing a silently invalid score.
10. ts cache not gitignored → evals/arfbench/data/ts/ added to .gitignore.
11. Shuffle stability → per-sample deterministic shuffle (seed from sample id); identical permutations across tier slices and full runs (0 mismatches verified).
12. series_context validation → unrecognized string values now raise ValueError instead of silently disabling context.

Hack-pass note: the position hack from the 2026-10-19 report is dead (always-A: 61.7% → ~23%); residual no-reasoning baseline ~33% documented in code and README.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5e2304cdea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Select resolution from the choices' actual granularity

This pattern treats every full timestamp as requiring 10-second data, including timestamps ending in :00. In the committed dataset, 74 of the 88 timestamp samples have only minute-aligned choices, yet each starts by downloading the 10-second parquet; oversized files are then deleted and can be downloaded again for another sample using the same group. Infer the required interval from the choice spacing or nonzero seconds so ordinary minute-level questions begin at 60- or 300-second data.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Update the README to match the implemented shuffle

These lines still state that choices are unshuffled and instruct users to add shuffling, but _to_sample() already applies a per-sample SHA-derived permutation. Following this guidance changes the option ordering again, while the later verification section separately claims the task uses dataset.shuffle_choices(seed=42), so neither the documented reproducibility scheme nor its reported position baseline describes the task users actually run. Update both sections to describe the implemented per-sample shuffle.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Use one resolution for both sides of paired questions

Each side of a paired correlation or indicator sample independently selects its first resolution below the row limit, so groups of different sizes can be summarized at different granularities such as 5 minutes and 1 hour. Their reported peak timestamps are then not temporally comparable and can change the inferred leading/lagging order, directly affecting the benchmark's 163 indicator samples. Negotiate a common usable resolution across all parts before producing either summary.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 366415011c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Land the required ARFBench spec before this code

This commit introduces executable evaluation code, but a repository-wide search finds no ARFBench requirements under specs/ or .kiro/specs/. The governance verdict requires code to implement a spec that already exists on the base branch, so this change cannot pass the mandatory gate; have Kiro or Ky land the spec separately, reference it from the PR, and cite its criterion IDs in these tests.

AGENTS.md reference: AGENTS.md:L19-L22

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Preserve fine timing for indicator questions

For all 163 Anomaly Indicator samples, the question and choices describe leading/lagging relationships without containing literal timestamps, so this check always selects the 300-second-or-coarser path. The committed data demonstrates why that is insufficient: indicator row 35 pairs groups 35950_1 and 35950_6, whose associated recovery samples distinguish 18:19:00 from 18:19:10; five-minute bars can erase precisely the ordering being scored. Select resolution using the task category and the paired groups' required timing granularity, not only timestamp text in the current choices.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Include option-named channels in identification summaries

For Anomaly Identification samples, the answer choices ask about specific channel labels, but this unconditional top-eight truncation can omit those channels whenever a group has more than eight variates. Selection by rolling-z rank is not equivalent to selection by the benchmark's candidate labels—level shifts, trend changes, and normal channels needed to establish “No anomaly” can rank below noisy unrelated series—so the model receives no observations for choices it must compare. Preserve every option-named channel in the summary before filling any remaining slots by deviation.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Stream parquet downloads instead of buffering them

On an uncached group, r.read() materializes the entire parquet response in memory before the row-count limit is checked. The benchmark documents groups with thousands of variates and also attempts 10-second files, so a default run can allocate a very large response only to discover afterward that it exceeds TS_MAX_ROWS, potentially exhausting memory before evaluation begins. Stream into a temporary file and inspect parquet metadata or enforce a byte limit before promoting it into the cache.

Useful? React with 👍 / 👎.