Skip to content

ops: replay the Sentinel enforcement arithmetic over recorded commitments (G5) - #191

Merged
minuszka merged 2 commits into
mainfrom
ops/dsl-enforcement-replay
Sep 20, 2026
Merged

minuszka merged 2 commits into
mainfrom
ops/dsl-enforcement-replay

Conversation

@minuszka

Copy link
Copy Markdown
Owner

What

ops/dsl-enforcement-replay.py: replays the Sentinel layer's enforcement arithmetic over the commitments already on the chain, so the question that decides when punishment may be switched on (H2) -- what would enforcement have done? -- is answered from recorded data, with no chain change. This is the G5 gate of v23-prep/sentinel-acceptance-criteria.md, which until now had no tool.

  • A line-for-line port of CDeterministicMNList::ApplyServiceCommitment (Core evo/deterministicmns.cpp, v22.1.x at 0c758792ca): absent epoch freezes, mass outage changes nothing, unobserved changes nothing, online resets (and when enforcing clears suspension and ban), missed increments (and when enforcing suspends at 4, bans at 5).
  • Inputs: the explorer's /api/v1/dsl/epochs rows and the deterministic list at each epoch base from ops/dsl-baselists-dump.sh (paced, read-only protx diff 1 <base> on the VPS). The bit order is that list sorted in the node's internal byte order; the stored proTxHash lists are cross-checked against the index resolution.
  • Reports per-window events, final counters, a G5 verdict against an expected stopped set, a cross-check against the node's live counters, and branch coverage -- it says so when the unobserved-hold branch guarded no streak, which is true of every recorded devnet epoch so far.

Validation on real data (2026-09-20)

window expected result
shadow outage 2026-09-03 (5 stopped, epochs 300-312) exactly the five suspended and banned PASS
E1b 2026-09-05 (the chain enforced) suspended @8496, banned @8520, recovered @8568, the five host-1 masternodes PASS, byte for byte
E2 2026-09-06 (22 then 23 of 152) guard opens at 23, nobody suspended PASS
baseline 2026-09-15 (epochs 581-680) nobody PASS
whole history vs the node's missedServiceEpochs equal 152 of 152
control: guard raised to 16% on E2 must now suspend 21 suspended (FAIL as required)

The first real-data run had the right heights and four wrong names: the tool sorted display hex, the node sorts internal bytes. The built-in cross-check flagged every epoch; the fix is canonical_order, and the fixture pins it.

Tests

ops/tests/test_dsl_enforcement_replay.py (14 cases, whole ops suite 49/49): a synthetic fourteen-masternode chain whose display order is the reverse of the internal order, every branch exercised -- including the unobserved-hold on a live streak that real data never produced -- and four negative controls that each flip one rule and must change the answer. Fourteen members and not six, because with six one miss is already 16.7% and the 15% guard swallows every epoch (the first draft did exactly that).

Not in this PR

The explorer's /masternodes/events?type= enum does not include the service_* types the poller writes, so the chain's own suspension/ban events cannot be queried by type over the API; the E1b ground truth above comes from the run record. Separate small fix.

🤖 Generated with Claude Code

…ents (G5)

The acceptance criteria for switching Sentinel punishment on (H2) end in one measurement nothing else can substitute for: replay the enforcement arithmetic over the commitments already on the chain and require that the suspended and banned sets are exactly the masternodes an operator stopped for long enough, and nobody else. Until now that gate had no tool.

ops/dsl-enforcement-replay.py is a line-for-line port of CDeterministicMNList::ApplyServiceCommitment (evo/deterministicmns.cpp, v22.1.x at 0c758792ca): an absent epoch freezes every counter, a mass outage (missed*100 >= size*15) changes nothing, an unobserved member changes nothing, an online member resets and -- when enforcing -- clears the suspension and the DSL ban, a missed member increments and -- when enforcing -- is suspended at 4 and banned at 5. It reads the explorer's epoch rows and the deterministic list at each epoch base (ops/dsl-baselists-dump.sh, a paced read-only protx diff per epoch), because the bit order is that list sorted by proTxHash in the node's INTERNAL byte order. The stored proTxHash lists are cross-checked against the index resolution: sorting the display hex instead names the wrong masternodes, which is exactly what the first real-data run did (right heights, wrong names) and what the cross-check caught.

Validated on 2026-09-20 against every window with a known answer: the 2026-09-03 shadow outage (five stopped -> exactly the five suspended and banned), E1b of 2026-09-05 where the chain itself enforced (suspended at 8496, banned at 8520, recovered at 8568, the five host-1 masternodes -- reproduced byte for byte), E2 of 2026-09-06 (the guard opens at 23 of 152, nobody suspended), the 100-epoch baseline (nobody), and the whole history against the node's live missedServiceEpochs (152 of 152 equal). Raising the guard to 16% suspends 21 in E2: the control that the guard branch is live. The tool reports branch coverage, and says so when the unobserved-hold branch guarded no streak -- which is true of every recorded devnet epoch so far.

The unit test uses a synthetic fourteen-masternode chain whose display order is the reverse of the internal order, and exercises every branch including the unobserved-hold on a live streak that real data has never produced, plus four negative controls that each flip one rule and must change the answer. Fourteen and not six: with six members one miss is already 16.7% and the guard swallows every epoch, which is what the first draft of the fixture did.
… name the limits

Follow-up to the independent review of #191 (four notes, none blocking):

- A member of the epoch-base list that is gone from the list at the boundary
  block is skipped, as deterministicmns.cpp:423 skips it. The boundary list is
  the next epoch's base list (base(k+1) == boundary(k)); without it every base
  member is counted and the run says so. On the recorded history the count is
  0 in every window -- no masternode has ever left this devnet's list.
- --enforce-from-height applies the node's own rule (boundary height >=
  nDSLEnforcementHeight, :419); the epoch form stays for counterfactual
  windows. E1b replayed from height 8304 gives the same events as from epoch
  346 -- on this data, because nobody had a streak at 345.
- The counter cross-check reports how many node counters are non-zero and says
  when the comparison is empty: at the 16627 tip all 152 were 0, so "152/152
  equal" carried no weight. The docstring says so instead of implying otherwise.
- The docstring narrows the claim to a first-order counterfactual: a real DSL
  ban sets isValid=false and changes later commitments, so the replay answers
  the G5 gate and is not a proof that enforcement is safe. --expect is named
  as window-only with a clean-start assumption; the stored-vs-index cross-check
  as two ports of one ordering rule, the node's counter as the only oracle.

Tests: 19 (was 14) -- the removed member with its negative control (no
boundary list -> counted, warned; list at the wrong height -> not used), the
height rule at the boundary block and one block later. Ops suite 54/54. The
real-data matrix re-run with this version is unchanged (shadow 5/5/5, E1b
@8496/@8520/@8568 by height and by epoch, E2 nobody, baseline nobody, control
A 21/21 as required).
@minuszka

Copy link
Copy Markdown
Owner Author

Review follow-up — 3429323

The independent review approved with four non-blocking notes; all four are in 3429323.

  1. The "152/152 counters equal" check was empty at that tip. Every missedServiceEpochs was 0, so an all-zero replay would pass too. The tool now prints how many node counters are non-zero and says when the comparison cannot discriminate; the docstring no longer presents it as evidence. It earns weight only on a snapshot taken while streaks are live (the next outage run).
  2. First-order counterfactual. Under real enforcement a DSL ban sets isValid=false, which changes the sentinel assignment and who can report, so later commitments would differ. The docstring now says the replay answers the G5 gate ("would anyone have been punished in this window?") and is not a proof that enforcement is safe.
  3. Port gap closed. A base-list member that is gone from the list at the boundary block is skipped, as deterministicmns.cpp:423 skips it. The boundary list is the next epoch's base list (base(k+1) == boundary(k)); without it every member is counted and the run warns. Real data: 0 such member-epochs in every window.
  4. --enforce-from-height applies the node's rule (boundary height >= nDSLEnforcementHeight, :419). E1b from height 8304 reproduces the same events as the epoch form, on this data only: nobody had a streak at epoch 345, and 8304 is epoch 345's boundary, not 346's, which is exactly the one-epoch slip the note warned about.

Nit, also in: --expect is documented as window-only with a clean-start assumption; the stored-vs-index cross-check as two ports of the same ordering rule, not an oracle.

Tests 19 (was 14): the removed member with two negative controls, the height rule at the boundary block and one block later. Ops suite 54/54. The real-data matrix re-run with this version is unchanged (shadow 5/5/5; E1b @8496/@8520/@8568; E2 nobody; baseline nobody; control A 21/21 as required).

@minuszka
minuszka merged commit 13f2ee4 into main Sep 20, 2026
6 checks passed
@minuszka
minuszka deleted the ops/dsl-enforcement-replay branch September 20, 2026 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant