All pull requests

Pull requests

• Add HAE-RAE Bench (five original subsets)

  #1485·jhwanseok opened Nov 6, 2026·

  ·
  1

• fix(metrics): stop counting missing samples as failures in pass@k

  #1484·aidless opened Nov 6, 2026·

  ·
  1

• fix: compute multiclass F1 averages over all classes

  #1482·aidless opened Nov 6, 2026·

  ·

• Add a Count column with the evaluated sample count to the markdown results summary

  #1480·amkkoussay opened Nov 5, 2026·

  ·
  1

• Read numbers grouped with no-break, narrow no-break or thin spaces

  #1478·ChrystianSchutz opened Nov 5, 2026·

  ·
  1

• Do not let an extracted expression end with an operator (bold answers)

  #1477·ChrystianSchutz opened Nov 5, 2026·

  ·
  1

• fix(cache): persist regenerated responses after invalid cache entries

  #1474·Cookie0103 opened Nov 3, 2026·

  ·

• Fill in the translation literals for Latvian, Lithuanian, Maltese and Slovenian

  #1473·inchang-ing opened Nov 3, 2026·

  ·
  1

• fix(litellm): log and back off on non-filter BadRequestError

  #1472·pcbeingused333 opened Nov 3, 2026·

  ·

• fix(hle): scale confidence to 0-1 before computing calibration error

  #1470·damienen opened Nov 2, 2026·

  ·
  1

• Combine subtask stderrs in quadrature for the _average and all rows

  #1468·Andreas-Hum opened Nov 1, 2026·

  ·
  1

• Fix double BOS when using chat templates (vllm, transformers, sglang backends)

  #1467·EvanZhuang opened Nov 1, 2026·

  ·

• fix: give the LCB sandbox's fake stdout a binary buffer

  #1465·abhishekcpp007 opened Nov 1, 2026·

  ·
  1

• fix(cache): stabilize solver task hashes

  #1462·jovial-liu opened Nov 1, 2026·

  ·

• Fix Probability metric to aggregate over gold choices, not all choices

  #1461·Sivadheeraj5356 opened Oct 31, 2026·

  ·

• Limit the memory a generated LiveCodeBench solution can add

  #1460·williamcaban opened Oct 29, 2026·

  ·
  1

• Fix typos in docs, docstrings, comments and log messages

  #1458·sarankumar1325 opened Oct 29, 2026·

  ·

• Bump doc-builder to get the docs machine-translated

  #1457·mishig25 opened Oct 29, 2026·

  ·
  1

• Strip reasoning section when only the closing </think> tag is generated

  #1454·selimfedakar opened Oct 28, 2026·

  ·
  1

• Fix aimo_progress_prize_1 exact match on integer gold answers

  #1452·mottopanikeiku opened Oct 27, 2026·

  ·

• fix(metrics): compare mathematical equivalence in Inspect math_scorer (#1430)

  #1451·pfolex opened Oct 24, 2026·

  ·
  1

• Keep Recall@k tie-breaking consistent with Recall@1

  #1450·yunyv opened Oct 24, 2026·

  ·

• Fix source offsets in extracted fragment strings

  #1449·yunyv opened Oct 24, 2026·

  ·

• Skip batch size search when model parallel has several processes

  #1447·chetana1410 opened Oct 23, 2026·

  ·
  1

• Restore TriviaQA prediction normalization

  #1446·mottopanikeiku opened Oct 22, 2026·

  ·
  1