Model conformance tests

Tests for conformance between the two PWN model implementations, pwn (Go) and PWNModel.jl (Julia).

Both implementations run the same headless config, and their CSV output must be identical. That config is generated from the documented model parameters (model-spec/parameters.yaml), merged with configs/base.yaml, which holds what is not a model parameter (seed, reporters). As both models reject unknown and missing parameters, running them with it also checks that the documented parameters are exactly those the models take.

Further, the documented parameters themselves are checked, as well as that the default configs shipped with both models match them.

Running locally

With model-spec/, pwn/ and PWNModel.jl/ checked out next to this repository:

python scripts/check_params.py      # checks parameters.yaml, and the models' config.yaml files against it
python scripts/params_to_config.py  # generates the config from parameters.yaml  -> out/default.yaml
scripts/run_go.sh                   # runs the Go model with out/default.yaml     -> out/go/
scripts/run_julia.sh                # runs the Julia model with out/default.yaml  -> out/julia/
python scripts/compare.py           # compares the outputs as numbers, non-zero exit on mismatch

The run scripts optionally take another config file as argument. The Python scripts require PyYAML (except compare.py).

CI

The workflow conformance.yml runs nightly and on pull requests, with all repositories at their default branch. It can also be triggered manually (Actions → Conformance → Run workflow), with the refs of both model repositories and of model-spec as inputs. The generated config and the model outputs are uploaded as a workflow artifact.