[janosg]

Problem

optimagic's benchmark sets (More-Wild, Cartis-Roberts) are least-squares problems. There is no set to benchmark optimizers for maximum likelihood estimation (e.g. the planned rewrite of bhhh) on realistic likelihoods, including misspecified models, bad start values and bounds.

Changes

• Four new sets in get_benchmark_problems:

  set	instances	problems	params
  likelihood_small	33	66	3–25
  likelihood_large	24	48	25–150
  likelihood_misspecified_small	36	72	3–25
  likelihood_misspecified_large	23	46	25–150

• 18 econometric models on simulated data (optimagic.benchmarking.likelihood.models): logit, probit, Poisson, fixed effects Poisson, negative binomial, zero inflated Poisson, heteroskedastic linear regression, Tobit, gamma regression, stochastic frontier, ordered probit, multinomial logit, Weibull duration, multivariate normal, GARCH(1, 1), random effects probit, mixed logit (simulated likelihood), normal mixture and Heckman selection. Models are reused with different numbers of params.

• Each instance has a good start (what a practitioner would choose) and a bad start (far from the solution, finite criterion).

• All problems minimize the negative log-likelihood and provide closed form jacobians of the contributions, passed as jac=[gradient, jacobian] so that scalar and likelihood optimizers both get exact derivatives. Some problems have natural bounds; some solutions lie on a bound. With noise, the closed form derivatives are dropped.

• The misspecified sets range from tiny to severe misspecification: over- and underdispersed counts, t errors, misclassified outcomes, IIA violations, neglected frailty, multivariate t data, ignored heteroskedasticity and more. info holds the severity, the eigenvalue range of inv(H) @ OPG at the solution (one in the population for correctly specified models) and whether the reference computation found other local optima.

• Data are simulated from uniform draws via inverse CDFs, so they do not depend on NumPy's distribution algorithms; a test pins the random stream.

• Reference solutions are stored in solutions.py and created by .tools/create_likelihood_solutions.py (L-BFGS-B from several starts, then projected Newton steps). Tests check that the stored values are reproduced and that the solutions satisfy the optimality conditions.

• run_benchmark summed squared likelihood contributions as if they were least-squares residuals; it now sums them.

• A paragraph on the new sets in the benchmarking how-to.

Checks

• For correctly specified instances, a Wald test of the estimates against the true params passes everywhere except the large mixed logit, where 50 simulation draws bias the simulated likelihood (it passes with 400 draws; documented in the model docstring).
• One L-BFGS-B run per set (1000 evaluations, convergence disabled, one core) takes 1–14 s.

🤖 Generated with Claude Code

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines	Coverage Δ
...c/optimagic/benchmarking/get_benchmark_problems.py	94.79% <100.00%> (+0.87%)	⬆️
src/optimagic/benchmarking/likelihood/models.py	100.00% <100.00%> (ø)
src/optimagic/benchmarking/likelihood/problems.py	100.00% <100.00%> (ø)
src/optimagic/benchmarking/likelihood/sampling.py	100.00% <100.00%> (ø)
src/optimagic/benchmarking/likelihood/solutions.py	100.00% <ø> (ø)
src/optimagic/benchmarking/run_benchmark.py	95.83% <100.00%> (+3.61%)	⬆️
...s/optimagic/benchmarking/likelihood/test_models.py	100.00% <100.00%> (ø)
...optimagic/benchmarking/likelihood/test_problems.py	100.00% <100.00%> (ø)
...optimagic/benchmarking/likelihood/test_sampling.py	100.00% <100.00%> (ø)
...imagic/benchmarking/test_get_benchmark_problems.py	100.00% <100.00%> (ø)
... and 1 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.