Repository navigation
build(core)!: point the prod build at audius-mainnet-beta — *DO NOT MERGE* - #553
Open
rickyrombo wants to merge 2 commits into
Open
rickyrombo wants to merge 2 commits into
rickyrombo wants to merge 2 commits into
Conversation
rickyrombo
added a commit
that referenced
this pull request
Aug 28, 2026
Every step that cannot be executed without a code change now names it, so the dependency is visible where the work happens rather than only in a list at the top: #553 for the binary (step 3), api#1029 for play routing (5 and 13), #551 before the fleet state syncs (10), api#1018 before flushing (11), api#1028 for the indexer bounds (12). Also records two things found while deriving the bootstrap's identity for #553: the node key is the comet key, so a node's P2P id is its validator address and the bootstrap's is derivable before it runs; and the bootstrap must not inherit node_key.json from the writer output, or its id will not match the peer list.
rickyrombo
added a commit
that referenced
this pull request
Sep 8, 2026
…cutover plan (#550) * docs(genesis-writer): seal the artifact DB before starting any service A verify node started against the pristine chain DB with a binary that did not embed the artifact's genesis took its "generate new genesis" path and ran the core migrations down, destroying the writer output. OPENAUDIO_ENV=dev loads the embedded dev.json, so the binary looks normal right up until it drops the tables. Document sealing the artifact with ALLOW_CONNECTIONS false and serving from a TEMPLATE copy, so no connection string can reach the pristine write, and note that artifact verification belongs after services start, not before. * docs(genesis-writer): correct the height check and record the writer's real prerequisites Five fixes from a read-through: - The runbook told the operator to confirm block 5,819. Both verification artifacts end at 5,839, so the check inverted: a healthy chain reads as the dead-chain failure mode. Every prod run ends at a different height, so the check now points at the writer's reported height instead of any literal. - The writer's prerequisites were undocumented. rewards is its FIRST step and needs --core-dsn to the old core chain, the launchpad secrets from env, and the mints file; the destination database must exist and be empty in both Postgres and its CometBFT directory. Anything missing kills a ten-hour run in the first second. - Step 3 had no pass criterion, and parity legitimately reports divergences -- including several where the migration is more correct than the source. Lists the expected set so a reviewer does not flag the migration for being right. - The ~10h writer estimate is unsourced; verification runs took 3h09m and 3h36m. Says to re-derive rather than substituting another machine's number. - Step 13 fell outside the reversibility framing, but it retires the two old-chain state-sync servers and so ends the cheap rollback path. * docs(genesis-writer): replace the runbook with the sequenced cutover plan The previous runbook ordered the work but left the two hardest transitions underspecified: how plays avoid being split across chains while the fleet is migrating, and how the indexer moves chains without duplicating or dropping writes. Sixteen steps now, with the indexer switch pinned to a height L chosen in the future so it does not race the config rollout, and the flusher filtered below L so the boundary is exact rather than inferred. Plays are routed through two old-network nodes for the duration, which keeps them on the chain the indexer is actually reading. Records what each irreversible step costs: 14 is the point of no return, 15 retires the rollback anchors, and jailing stops entirely once the old network is down to ~30 active validators -- after which a halt cannot be recovered by waiting. * docs(genesis-writer): reconcile the reference with the new runbook and with the code Six places where the reference either contradicted the runbook or stated something the code does not do: - The facts table gave prod's validator count as 9. That is the genesis list; the live set is 67 of 72 registered nodes, and the runbook's quorum and jailing math is over the live set. - Section 7 said to leave BlockInterval at its default, which the runbook now overrides with 20,000. Records why the default is wrong here, that it is a producer-side setting, and that retention is Keep x BlockInterval so it cannot be lowered freely. - The state-sync truncation table listed blobs but not uploads, audio_previews or qm_audio_analyses, which are also wiped. Adds them with how each recovers, and notes #551 makes the row moot. - Section 3 asserted the bootstrap validator holds power 100. Its own registration rewrites that to ValidatorVotingPower, 10 on mainnet, because the writer never seeds core_registered_nodes. - The cursor discussion now points at api#1018, which is open and must land before flushing is first enabled. - Section 11 omitted the chain-aware fallback to core_indexed_blocks, which is only reached on ErrNoRows and so never runs on a database that has indexed the old chain. That is why the failure is a silent stall. * docs(genesis-writer): drop the genesis-vs-key startup refusal, it would block the rollout The reference suggested refusing to start when the validator key disagrees with genesis. New-chain genesis lists one validator and every other node joins by registration, so that check would stop the entire fleet from ever reaching the new chain -- it is the rollout mechanism, not a fault. It also named ensureGenesisFiles, which does not exist. The genesis-file handling is setup.go:114-126 and compares nothing, correctly. The check worth having is already there: ensurePrivValidator compares the key file against the key derived from the delegate key and refuses only when prior signing history makes a mismatch a double-sign risk. The T7 node that motivated the suggestion was behaving correctly -- a node whose key is not in genesis block-syncs and does not propose. It read as a dead chain only because it was the sole node. Replaces the recommendation with a diagnostic: log when the key is absent from the genesis set and say the node will not propose until registered. * docs(genesis-writer): reconcile the persistent-peers guidance with the new binary Reference SS5 said the bootstrap node would dial old-network hosts and should suppress them with OPENAUDIO_PERSISTENT_PEERS, no release needed. That assumed the node runs the existing binary with only genesis swapped. Runbook step 3 builds a binary whose ProdPersistentPeers is already the new bootstrap nodes, so there are no old-network hosts in the list and nothing to suppress. Outbound dialing of the old network does not happen on the step 3 binary. Inbound does, and no env var here changes it: old nodes hold this host in their address books and keep dialing until those entries age out. Those connections are rejected at the handshake on the network mismatch. The env override at step 4 is still worth setting, but for the narrower reason now stated there: the second bootstrap node does not exist until step 7, so the baked list would otherwise churn against a host that is not up. * docs(genesis-writer): clear the indexer bounds after the cutover Neither bound is consumed by the ETL; both are re-read on every startup. Left set, a start height makes each restart re-index from it and duplicate plays, and an end height stalls the indexer at L permanently. * docs(genesis-writer): add a TL;DR table of contents Sixteen steps across three phases is more than fits in someone's head, and the irreversible ones are buried at the end of a long document. One table, the three failure modes that actually bite, and the PRs that must merge first. * docs(genesis-writer): cite the PRs each step depends on Every step that cannot be executed without a code change now names it, so the dependency is visible where the work happens rather than only in a list at the top: #553 for the binary (step 3), api#1029 for play routing (5 and 13), #551 before the fleet state syncs (10), api#1018 before flushing (11), api#1028 for the indexer bounds (12). Also records two things found while deriving the bootstrap's identity for #553: the node key is the comet key, so a node's P2P id is its validator address and the bootstrap's is derivable before it runs; and the bootstrap must not inherit node_key.json from the writer output, or its id will not match the peer list. * docs(genesis-writer): record what the runbook alone did not carry Someone reading only this document would have hit each of these cold: - how to actually run the replay and parity, including that --db must be an ETL database and the replay must point at the serve copy, not the sealed original - calibration figures from the 2026-08-25 run, framed as sanity checks rather than expected values, since a fresh snapshot changes all of them - the row-level parity mismatches that are not in the divergence table: genre re-casing, musical_key dropped by the allowlist, and track_downloads disagreeing on parent_track_id for deleted tracks. All pre-existing ETL behaviour; every missing row should reconcile to a listed cause - why plays need routing at all: logTrackListen runs when a node serves audio, and plays never enter the relay queue, so a migrated node writes them to a chain nobody is indexing - the arithmetic behind ten-at-a-time: eligibility is ~4.5h of proposing nothing, not warden throughput, and attestation quorum never binds before the chain has already halted - that the routine DP backup is not restorable as taken, and that user_balance_history is ~85 GB the migration never reads * docs(genesis-writer): restructure into overview, steps, appendix The document had grown to where the instructions and the reasoning were the same text, and an operator working through it had to read an argument to find an action. Three parts now. The overview says what the migration is and why it is awkward, with a linked table of contents. The steps are imperative -- 22 to 110 words each, pointing at the appendix rather than explaining inline. The appendix holds the reasoning: nine sections keyed to the steps that need them, then the existing reference material. No content dropped. Every explanation that was inline is now an appendix section with an anchor, and all 26 cross-links resolve. * docs(genesis-writer): lead with why the migration exists The overview described the mechanism without saying what it is for. audius-mainnet-alpha-beta began part-way through the protocol's life, so most of the network's history -- the majority of users, tracks, playlists, follows, saves, reposts and plays -- is not on it and lives only in the application database. Reconstructing state from that chain alone yields a fraction of it. That is the reason the new chain is written rather than synced: syncing would carry the same gap forward. It is also why verification dominates step 1, since the artifact is the history and there is no second source once it ships. Also drops 'Discovery Provider' for 'production snapshot', per the terminology note in CLAUDE.md. * docs(genesis-writer): make the PR references clickable Thirteen references across the document were plain text, so following one meant knowing which repo it belonged to. api#N now resolves to AudiusProject/api and bare #N to OpenAudio/go-openaudio, which is also the disambiguation the plain text was silently relying on.
rickyrombo
added a commit
that referenced
this pull request
Sep 16, 2026
…and let mediorum follow core's rules (#572) * feat(core): schedule auth and content-auth enforcement on audius-mainnet-beta The new chain has no entry in the upgrade schedule, so #553's build would run it with no signer enforcement and no track-cid authorization, and mediorum would never attest. Schedule both at height 1: the genesis replay projects auth state and a cid claim for every migrated track, so nothing that exists is made unwritable, and no live block is ever produced under the relaxed rules. Mediorum's attest/require-userId decision moves from the environment name to the chain named in the embedded genesis. The prod build serves audius-mainnet-alpha-beta until the rollover and audius-mainnet-beta after, and attestations are refused before a chain's gate, so keying on "prod" would fail every audio upload on whichever chain lacked the gate. Keying on the genesis makes the switch flip with the genesis swap and nothing else. It also stops stage attesting on audius-testnet-alpha, which has no gate scheduled. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(core): only resolve a genesis for named environments in ContentAuthScheduled genesis.Read defaults unknown names to devnet, which turned content auth on for the mediorum test network (Env "test") and rejected its unattributed uploads. Unnamed environments get no gate, as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(core): pin the content-auth schedule invariant and mediorum's literal answers The genesis-following test computed its expectation with the same two calls the function makes, so it could only fail if the function stopped being its own implementation. Replace it with literal per-environment answers, so the rollover build that swaps prod.json has to flip prod to true here on purpose. The schedule test asserted both heights were 1 on three chains but not the relationship that matters: content auth without signer auth at or before it leaves the track-cid check dormant. Assert that over every schedule entry. Trim the rationale that was repeated on the mediorum config field; it lives on ContentAuthScheduled. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(storage): accept uploads while core is starting; only the transcoder waits The create-time 503 from #581 assumed the client's node selection would retry elsewhere. It does not: the SDK's tus upload retries the same node for about 38 seconds and then fails, so any boot window longer than that would fail every attributed audio upload against a restarting node. Upload creation never touches core, and the transcoder already waits for core before pulling work, so accepting the upload and letting the job wait is both safe and invisible to the client, whose processing poll allows an hour. Drop the gate from tus, multipart, and gRPC; keep the transcoder wait and the sender's readiness check. The wait no longer depends on content auth being on: it costs nothing on a chain without it, and the next commit makes that answer come from core, which cannot be asked during the window the wait guards. The mediorum test harness passed a bare CoreService that never registers, which the unconditional wait would treat as a core forever starting. It now passes nil, the shape the code already reads as "no core wired", and the two in-process core calls that lacked a nil guard get one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(storage): consult core's next-block rules through one blocking accessor Mediorum decided whether to attest from a boolean set at boot by reading the embedded genesis for the environment and asking the schedule whether content auth was scheduled at all. That was correct only because every scheduled activation is at height 1, it duplicated the environment switch from genesis.Read to dodge its devnet fallback, and the next rule gated on a height that touches mediorum would need the same workaround. Ask core instead, and never default. CoreService.NextBlockRules resolves the rules for the next block from the live schedule and cached height, which is exactly what the mempool applies to a transaction submitted now, and errors until core registers: before that there is no chain to ask, and a fabricated answer is indistinguishable from a real one to the caller. On the mediorum side every rule consult goes through chainRules, which waits for core rather than returning anything it does not have. Background workers pass the lifecycle context and wait the boot window out. Request handlers pass the request context, so a client that gives up cancels the wait; a slow response beats a refusal, because the SDK's upload retries stay on one node and give up within a minute. Attribution, attestation, and preview claims all take the rules from the same call, so nothing can fail open during boot, and the transcoder's separate wait goes away since attestation now waits at the point of decision. The async tus post-create hook, whose context expires shortly after the request, parses the id without consulting the rules; the pre-create hook already enforced them. ContentAuthScheduled, its allowlist, and its test go away. The config field remains as the override for tests with no core wired. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * ci: dump devnet container logs when integration tests fail The go test output on a devnet readiness timeout says only which node never answered. The reason is in that node's log, and the harness leaves the containers running on failure, so print them before shutdown. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
rickyrombo
force-pushed
the
mjp-new-network-binary
branch
from
September 16, 2026 22:33
06f11e8 to
346a662
Compare
rickyrombo
force-pushed
the
mjp-new-network-binary
branch
2 times, most recently
from
October 9, 2026 05:37
1824438 to
cdf2d89
Compare
rickyrombo
force-pushed
the
mjp-new-network-binary
branch
from
October 9, 2026 08:28
cdf2d89 to
6011082
Compare
rickyrombo
force-pushed
the
mjp-new-network-binary
branch
3 times, most recently
from
October 23, 2026 21:32
4a082e4 to
5b46734
Compare
DO NOT MERGE. Genesis is compiled in, so merging this and shipping it as stable IS the mass migration -- every node that pulls it leaves the current chain. The point is the image CI builds from the branch, which the bootstrap nodes run during Runbook steps 4-10. prod.json is the verified 2026-10-21 artifact (genesis sha256 438b7f12..., source snapshot discProvProduction-2026-10-18): chain audius-mainnet-beta, one validator at power 100, migration end height 5899, so the first live block is 5900. ProdPersistentPeers is the two bootstrap nodes. The node key is the comet key (setup.go:105), so a node's P2P id is its validator address -- the bootstrap's id is its genesis validator address, lowercased. That also means the bootstrap must not inherit a node_key.json from the writer output, or its id will not match; the one in the artifact belongs to the verification node. v.audius.rickyrombo.com (5c082268...a20de) is behind the same router as the bootstrap and is reached on external port 26666, the address it advertises. ProdStateSyncRpcs names both bootstrap hosts. CometBFT refuses to state sync from fewer than two, so both must serve before any node joins. The two prod peer-list guards assert properties of a mature network -- twenty peers, and a per-node selection that avoids a gossip hub. Neither can hold for a chain with one node. They now step aside only when the embedded genesis has a single validator, and were confirmed to still fail against the nine-validator genesis.
The new chain is a major break from audius-mainnet-alpha-beta, so the build for it reports a new major version. This branch is never merged, so release-please never sees the bump; main keeps its own version line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rickyrombo
force-pushed
the
mjp-new-network-binary
branch
from
October 25, 2026 09:23
5b46734 to
79be379
Compare
rickyrombo
changed the title
rickyrombo
added a commit
that referenced
this pull request
Oct 25, 2026
Nodes run openaudio/go-openaudio:mainnet-beta, retagged by hand from the merge-ref image CI builds for #553, never :edge or :stable. Step 15 says how #553 merges so release-please releases 2.0.0 rather than 1.x or 3.0.0. The mirror's skip rule lists taken handles alongside reused track ids, and the relay reference links the API cleanup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rickyrombo
added a commit
that referenced
this pull request
Oct 29, 2026
…out runbook (#611) * docs(genesis-writer): replace the flusher with the mirror in the rollout runbook The API's new-chain queue and flusher are retired. The mirror (#605, #608, #610) replays the old chain's ManageEntity and play transactions onto the new chain from blocks, which covers every write source and carries plays. Step 11 now starts the mirror: flags, what /status reports, how each kind of refusal is handled, and throughput. Step 12 is the zero-downtime indexer switch: stop the old indexer at X, then derive H from the new block holding the last mirrored transaction at or below X, which block-by-block mirroring makes exact. Steps 13-15, appendices G and I, and the relay, rollback and open-question reference sections follow from that. The manifest field keeps its name; its comment now says it is the mirror's start height. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(genesis-writer): run :mainnet-beta, and merge #553 as 2.0.0 Nodes run openaudio/go-openaudio:mainnet-beta, retagged by hand from the merge-ref image CI builds for #553, never :edge or :stable. Step 15 says how #553 merges so release-please releases 2.0.0 rather than 1.x or 3.0.0. The mirror's skip rule lists taken handles alongside reused track ids, and the relay reference links the API cleanup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
⛔ DO NOT MERGE
Genesis is compiled into the binary (
//go:embed prod.json). Merging this and shipping it asstableis the mass migration — every node that pulls it leaves the current chain. This branch exists for the image CI builds from it, which the bootstrap nodes run during Runbook steps 4–10.*DO NOT MERGE*sits at the end of the title becausepr-title-lintrequires a conventional-commit prefix; it would reject the marker up front.Why a new chain
To put the network's history on chain.
audius-mainnet-alpha-betabegan part-way through the protocol's life. Everything before it — most users, tracks, playlists, follows, saves, reposts and plays — exists only in the application database. The chain holds only what happened since, so anyone rebuilding state from the chain alone gets a fraction of it.The new chain can't be synced from the old one, because syncing would carry the same gap forward. Instead
genesis-writerreads a production snapshot and writes that history directly into CometBFT blocks as migration transactions. The result is a chain that contains the full history before it has ever run consensus. That needs a new genesis, and so a new chain ID.What's different from
audius-mainnet-alpha-betaaudius-mainnet-alpha-betaaudius-mainnet-betaManageEntitymust be signed by the user's wallet or an app the user granted1.x2.0.0(this build)Nothing changes for clients: the same APIs, entity types and IDs. Writes made on the old chain after the snapshot are carried over block by block by the mirror (#605) until the fleet and the indexer have moved. See
cmd/genesis-writer/ROLLOUT.mdfor the rollout.What changed
pkg/core/config/genesis/prod.json— the verified 2026-10-21 artifact (sha256438b7f12…ea737a, built fromdiscProvProduction-2026-10-18):ProdPersistentPeers— the two bootstrap nodes:The node key is the comet key (
p2p.NodeKey{PrivKey: envConfig.CometKey},setup.go:105), so a node's P2P id is its validator address. The bootstrap's id is therefore its genesis validator address, lowercased — derivable without the node running.Two consequences worth stating plainly:
node_key.jsonfrom the writer output. The one sitting in the artifact belongs to the verification node, which generated it on first start. On the 10-21 artifact that node also rewrotepriv_validator_key.json,addrbook.jsonandpriv_validator_state.json, so none of the four should be seeded. Seeding it would give the bootstrap a P2P identity unrelated to its delegate key and unrelated to this entry. Delete it and let the node derive its own.v.audius.rickyrombo.comis reached on 26666. It sits behind the same router as the bootstrap, which forwards external 26666 to its 26656, and it advertises that address. Its id comes from its delegate key, so restarting the chain from a new artifact does not change it.ProdStateSyncRpcs— both bootstrap hosts. CometBFT refuses to state sync from fewer than two servers, so both must be serving before any node tries to join.The test changes are the interesting part
Two existing guards failed, correctly:
Both assert properties of a mature network — many peers, and a per-node selection that keeps gossip off a single hub. Neither can hold for a chain with one node in it, which is what a chain being born is.
Rather than weaken them, they now step aside only when the embedded genesis has a single validator, and stay in force otherwise. Verified by swapping the nine-validator genesis back in: both fail again exactly as before. So the guard still protects the established network, and re-tightens on its own once validators register and the list is repopulated.
Version
This build reports
2.0.0(pkg/version/.version.json). The title carries!so that on merge release-please also goes1.13.0 → 2.0.0. It reads the current version from.release-please-manifest.json, not from.version.json, so don't hand-edit the manifest here, or the!would land on 3.0.0.Not included
ProdPersistentPeersdeliberately does not carry the old-network hosts. That means a node on this build does not dial the old chain at all — see Reference §5 in the runbook, which used to say otherwise and was corrected in #550.