add SWIP-39: balanced neighbourhood registry aka smart neighbourhood management

[zelig]

UPDATED AUGUST 2026

First: the PR was heavily restructured on 2026-07-27 (two-transaction join, no
reservation held, identity-keyed registrations, concurrent departures, ICBT as a
separate container contract) — several of the points below are answered in the text
now, so please re-read the current version before diving into the old one.
Point-by-point:

1. Why do we need on-chain topology?

Because assignment must be litigable. The redistribution game checks, at claim
time, that a node plays in the neighbourhood it was assigned — that check has to run
in the contract, against state the contract trusts. Off-chain assignment can be
neither enforced (nothing stops a node self-selecting) nor verified (the contract
has no source of truth to check an overlay against). The same goes for the two
enforcement events: forfeiting an expired registration and slashing a defaulted
donor — both need the assignment state on-chain to be adjudicable. And grind-proof
randomness (stake locked before entropy known, seed from a committed block height)
only means anything if the commit itself is on-chain. What stays off-chain is
everything that can: target-prefix computation is a read-only call, mining is local,
and the join costs exactly two transactions.

2. Sorted ring + sparse buckets — formalize and compare.

Happy to add a comparison subsection to the SWIP; here is the summary. The layout
(doubly-linked list in overlay order + mapping(prefix at staking depth => [first node, count])) is good at what rings are good at: O(1)-write
insertion/removal once the position is known, O(1) neighbourhood membership query,
cheap ordered iteration. It is weak exactly where this SWIP lives:

• Uniform random selection. The core operation is "pick uniformly among the
  $2^{d+1}-N$ free neighbourhoods" (and among $N-2^d$ donor pairs). A ring+buckets
  layout either scans buckets — O(2^d) — or maintains hierarchical per-subtree
  counts to support O(log N) rank selection. The moment you add the counts
  hierarchy (your "buckets hierarchy?" bullet concedes this), you have rebuilt the
  ICBT: the trie is the counting structure, with the ring's information implicit
  in it.
• Depth transitions. Buckets keyed by "prefix at staking depth" must be re-keyed
  when $d$ changes — at $N=2^{d+1}$ every key changes, an O(N) migration or a
  lazy-migration scheme with its own bookkeeping. The ICBT never re-keys: $d$ is
  derived, indexes are stable, a depth transition is zero writes.
• Write cost is a wash. ICBT: O(log N) counter updates per activation/departure
  (path to root, ~30 slots at a million nodes). Ring: O(1) link writes + O(log N)
  anyway for whatever counting structure supports selection. We are comparing
  log-vs-log; the constant matters less than the re-keying cliff above.

Where the ring genuinely wins: iterating a neighbourhood's members in overlay order
(we never need this — one node per leaf by construction) and finding the successor
of an arbitrary address (the ICBT does it in O(log N) via nodeFor, good enough for
a view function). So: formal comparison in the SWIP yes, layout change no.

3. High-level operations.

Now in the SWIP: join = register + activate (§Join protocol), departure =
deregister (+ donor redraw path) (§Departure and rebalancing), neighbourhood
queries = getPrefix / nodeFor (read-only). Redistribution eligibility is one
prefix check against the assignment record — the staking contract calls
getPrefix(identity) and compares against the claimed neighbourhood. If a specific
operation list is wanted verbatim in the issue's terms, point me at it.

4. Random/balanced assignment without A and R.

Reading "A and R" as the ordinal mapping and the reservation from the old draft:
the restructured version already dropped both. There is no reservation
(target prefix is a read-only computation, nothing locked, activation revalidates
against current state) and no request IDs / ordinal indirection (registrations are
keyed by staking identity; the seed is H(domain ‖ identity ‖ blockhash)). If A and R
meant something else, tell me what and I'll answer that instead.

5. Comparison with compacted binary trie.

A compacted (path-compressed) trie saves storage when keys are sparse and clustered
— but our key population is dense by construction: the invariant forces every
prefix at depth $d$ or $d+1$ to be occupied, i.e. the trie is always complete to
within one level. There is nothing to compact — path compression on a complete tree
adds skip-pointers that must be maintained on every split/collapse and saves zero
levels. The implicit heap layout additionally removes all pointer storage: parent,
children, sibling are arithmetic on the index, so a "node" is just its counter
slots. Compaction pays off for arbitrary key sets; balanced assignment is precisely
the regime where it cannot. Will add this as a paragraph to the comparison
subsection.

6. Depth transitions.

Walked through in the SWIP (§Balance invariant: preservation under
insertion/removal; §Counting: the $N=2^D$ boundary cases for splitCount/donorCount).
The short version: transitions are emergent, not an event — no stored $d$, no
migration, the counters at the root already equal $2^{d+1}-N$ and $N-2^d$ and both
hit the boundary values exactly at powers of two. If a worked $N=2^D$ example would
help, I can add one next to the existing worked examples.

7. Complexity / gas.

§Gas and performance analysis: selection O(d) reads, activation/departure O(d)
writes, storage O(N), registration/expiry O(1) amortized (monotonic queue head,
bounded per call — no unbounded iteration anywhere). Mining is the real cost and it
is off-chain: expected $2^\ell$ hashes, $\ell \approx \log_2 N + 1$. Benchmarks are
listed as a reference-implementation deliverable; concrete numbers per depth once
there is a contract to measure.

8. The earlier notes (shrink, queue, gas of updates).

• Shrink/withdrawals: the departure path is now first-class — direct collapse
  completes immediately; the donor path holds the departing node active until a
  donor lands, with forfeiture + redraw on default. Withdrawal of stake itself is
  the staking contract's business (strict separation in the SWIP).
• Queue: yes — two commit queues ($C_R$, $C_D$), append-only with monotonic
  block heights, head-advancing expire bounded per call.
• Gas of data-structure updates: O(log N) slot writes per structural change; see
  point 2 for why the proposed alternative doesn't beat it once selection is
  accounted for.

this comment was added here #74 (comment)

[Copilot]

Pull Request Overview

This PR introduces SWIP-39, a smart neighbourhood management system for decentralized service networks. The proposal aims to solve the "one operator, one node in a neighbourhood" problem through a balanced assignment mechanism that ensures fair load distribution and prevents sybil attacks.

Key changes include:

• A comprehensive specification for balanced neighbourhood registry with random assignment
• Smart contract implementation for managing node registration and neighbourhood assignments
• Mathematical formulations for neighbourhood depth calculation and overlay address validation

[0xCardiE]

Thanks for the well-thought-out SWIP — the design is elegant and clearly addresses Sybil resistance and balanced assignment. I had a few questions and points I’d like to discuss for further clarity and robustness:

• Overlay mining and gaming – how is the off-chain mining step protected against precomputation attacks or delay tactics to gain strategic placement?
• Expired stake handling – have you considered slashing (partial or full) instead of just locking the stake on expiry?

[0xCardiE]

Could _upgradeDepth() become too expensive to execute as the number of assigned nodes grows?

The _upgradeDepth() function doubles the assignment and remaining lists, copies all existing nodes to their new positions using bitwise logic, and clears/rebuilds state — all in a single call. If the number of nodes reaches high volumes (e.g. 1,000+ or 10,000+), this could approach or exceed the block gas limit, making the function fail or stall the system.

Proposed solutions:
• Consider breaking _upgradeDepth() into incremental steps, e.g., by:
• Using checkpointing to process part of the upgrade each time it is triggered.
• Spreading the reassignment of existing entries across multiple assign() calls.
• Alternatively, lazy-initialize entries in the assign() function, only when a neighbourhood is actually being filled.

[0xCardiE]

Can the committers array become inefficient with a large number of registrants (e.g. 20k nodes)?

The committers[] array is iterated over in _expire(), _findEntryFor(), and _removeCommitter() using for loops. If a large number of nodes register, or if expired entries are not promptly cleared, the gas cost of these operations can grow linearly and become prohibitively expensive.

Proposed solutions:
• Maintain a mapping from address to index to allow constant-time lookup and removal.
• Use a circular buffer or a “head” pointer to avoid shifting array entries when removing expired ones.
• Alternatively, mark entries as expired/inactive with a boolean flag instead of removing them from the array immediately.

[zelig]

Can the committers array become inefficient with a large number of registrants (e.g. 20k nodes)?

I did not consider it realistic, since each registrant entry expires in max 256 blocks, that is in a matter of <4 game rounds and they lose their deposit if they refuse to pay, so likely all the potential players may organically wait out.

The committers[] array is iterated over in _expire(), _findEntryFor(), and _removeCommitter() using for loops. If a large number of nodes register, or if expired entries are not promptly cleared, the gas cost of these operations can grow linearly and become prohibitively expensive.

Proposed solutions: • Maintain a mapping from address to index to allow constant-time lookup and removal. • Use a circular buffer or a “head” pointer to avoid shifting array entries when removing expired onesr . • Alternatively, mark entries as expired/inactive with a boolean flag instead of removing them from the array immediately.

but they need to be removed at some point.... and I am not sure how a mapping that needs to be reindexed after every entry removed, will solve this.

[zelig]

Can the committers array become inefficient with a large number of registrants (e.g. 20k nodes)?

well, maybe. To be honest, there is also another way. We do not need to allow, just any length of the committer list. The length represents the queue, and the length of valid entries are the ones in the queue you can skip. This effectively quantifies the tries that you got (effectively mining) but also the realistic probability that that someone will come in and change the neighbourhood you (thought you were) assigned to. If this probability is high (there is a lot of nodes that can submit mined overlays), then it can easily happen, that whenever an assigned neighbourhood is read off, nodes will frontrun. So it would just make sense to limit this skip queue to a fix constant number. But this means that the committers list should effectively have a limited length. Now if we siply reject registrations beyond this limit, then the shorter this length, the harder it is for the same amount of currently aspiring nodes to commit. Now in order to avoid that the registration tx needs to be continuously retried (due to it most likely be frontrun by competing resistrants), we should introduce another proper FIFO queue (that is unlimited but does not need iteration). In this case the validity period starts when you enter the limited queue.

Proposed solutions: • Maintain a mapping from address to index to allow constant-time lookup and removal. • Use a circular buffer or a “head” pointer to avoid shifting array entries when removing expired ones. • Alternatively, mark entries as expired/inactive with a boolean flag instead of removing them from the array immediately.

Not sure I get how these structures would be useful: index needs reindexing or keeps inactive entries, head pointer just delays the problem and so does the inactive flag.

[significance]

Thanks for the well-thought-out SWIP — the design is elegant and clearly addresses Sybil resistance and balanced assignment. I had a few questions and points I’d like to discuss for further clarity and robustness:

• Overlay mining and gaming – how is the off-chain mining step protected against precomputation attacks or delay tactics to gain strategic placement?
• Expired stake handling – have you considered slashing (partial or full) instead of just locking the stake on expiry?

i. the mining step is just offloading computation rather than POW, strategic placement is prevented by random allocation, economic disincentives to be quantified forwith
ii. in this model the stake as discussed if always burned here could otherwise be seen as a fee which perhaps should be considered as recirculating to rewards and/or burned

[significance]

Could _upgradeDepth() become too expensive to execute as the number of assigned nodes grows?

The _upgradeDepth() function doubles the assignment and remaining lists, copies all existing nodes to their new positions using bitwise logic, and clears/rebuilds state — all in a single call. If the number of nodes reaches high volumes (e.g. 1,000+ or 10,000+), this could approach or exceed the block gas limit, making the function fail or stall the system.

Proposed solutions: • Consider breaking _upgradeDepth() into incremental steps, e.g., by: • Using checkpointing to process part of the upgrade each time it is triggered. • Spreading the reassignment of existing entries across multiple assign() calls. • Alternatively, lazy-initialize entries in the assign() function, only when a neighbourhood is actually being filled.

agree with this, some discussion around implementing binary trie or similar datastructure which will ensure uniform gas usage while providing for the necessary functionality

[significance]

very good swip, a few thoughts for discussion and expansion in the doc:

1. it is important to ensure currency of nodes, methods must be added that record recent activity and penalise nodes not taking part in this by at least removal from the allocated nodes pool, thus preventing squat attacks 🏋️
2. a thorough quantisation and parameterisation of economic disincentives should be performed, including costs arising from capital illiquidity
3. a withdrawal queue or delay should be considered to improve network integrity promises
4. likewise an node onboarding processes which perhaps could include proof of well behaved protocol adherance prior to admission into rewards pool
5. it would improve ui/security to provide facility to decouple keys: network key, withdrawal address similar to eth, maybe also a nominated admin address for web ui too
6. a rigorous approach to nomenclature at this point would be prudent, given that the tree depths discussed here are distinct from those in the storage network itself

[zelig]

First: the PR was heavily restructured on 2026-07-27 (two-transaction join, no
reservation held, identity-keyed registrations, concurrent departures, ICBT as a
separate container contract) — several of the points below are answered in the text
now, so please re-read the current version before diving into the old one.
Point-by-point:

1. Why do we need on-chain topology?

Because assignment must be litigable. The redistribution game checks, at claim
time, that a node plays in the neighbourhood it was assigned — that check has to run
in the contract, against state the contract trusts. Off-chain assignment can be
neither enforced (nothing stops a node self-selecting) nor verified (the contract
has no source of truth to check an overlay against). The same goes for the two
enforcement events: forfeiting an expired registration and slashing a defaulted
donor — both need the assignment state on-chain to be adjudicable. And grind-proof
randomness (stake locked before entropy known, seed from a committed block height)
only means anything if the commit itself is on-chain. What stays off-chain is
everything that can: target-prefix computation is a read-only call, mining is local,
and the join costs exactly two transactions.

2. Sorted ring + sparse buckets — formalize and compare.

Happy to add a comparison subsection to the SWIP; here is the summary. The layout
(doubly-linked list in overlay order + mapping(prefix at staking depth => [first node, count])) is good at what rings are good at: O(1)-write
insertion/removal once the position is known, O(1) neighbourhood membership query,
cheap ordered iteration. It is weak exactly where this SWIP lives:

• Uniform random selection. The core operation is "pick uniformly among the
  $2^{d+1}-N$ free neighbourhoods" (and among $N-2^d$ donor pairs). A ring+buckets
  layout either scans buckets — O(2^d) — or maintains hierarchical per-subtree
  counts to support O(log N) rank selection. The moment you add the counts
  hierarchy (your "buckets hierarchy?" bullet concedes this), you have rebuilt the
  ICBT: the trie is the counting structure, with the ring's information implicit
  in it.
• Depth transitions. Buckets keyed by "prefix at staking depth" must be re-keyed
  when $d$ changes — at $N=2^{d+1}$ every key changes, an O(N) migration or a
  lazy-migration scheme with its own bookkeeping. The ICBT never re-keys: $d$ is
  derived, indexes are stable, a depth transition is zero writes.
• Write cost is a wash. ICBT: O(log N) counter updates per activation/departure
  (path to root, ~30 slots at a million nodes). Ring: O(1) link writes + O(log N)
  anyway for whatever counting structure supports selection. We are comparing
  log-vs-log; the constant matters less than the re-keying cliff above.

Where the ring genuinely wins: iterating a neighbourhood's members in overlay order
(we never need this — one node per leaf by construction) and finding the successor
of an arbitrary address (the ICBT does it in O(log N) via nodeFor, good enough for
a view function). So: formal comparison in the SWIP yes, layout change no.

3. High-level operations.

Now in the SWIP: join = register + activate (§Join protocol), departure =
deregister (+ donor redraw path) (§Departure and rebalancing), neighbourhood
queries = getPrefix / nodeFor (read-only). Redistribution eligibility is one
prefix check against the assignment record — the staking contract calls
getPrefix(identity) and compares against the claimed neighbourhood. If a specific
operation list is wanted verbatim in the issue's terms, point me at it.

4. Random/balanced assignment without A and R.

Reading "A and R" as the ordinal mapping and the reservation from the old draft:
the restructured version already dropped both. There is no reservation
(target prefix is a read-only computation, nothing locked, activation revalidates
against current state) and no request IDs / ordinal indirection (registrations are
keyed by staking identity; the seed is H(domain ‖ identity ‖ blockhash)). If A and R
meant something else, tell me what and I'll answer that instead.

5. Comparison with compacted binary trie.

A compacted (path-compressed) trie saves storage when keys are sparse and clustered
— but our key population is dense by construction: the invariant forces every
prefix at depth $d$ or $d+1$ to be occupied, i.e. the trie is always complete to
within one level. There is nothing to compact — path compression on a complete tree
adds skip-pointers that must be maintained on every split/collapse and saves zero
levels. The implicit heap layout additionally removes all pointer storage: parent,
children, sibling are arithmetic on the index, so a "node" is just its counter
slots. Compaction pays off for arbitrary key sets; balanced assignment is precisely
the regime where it cannot. Will add this as a paragraph to the comparison
subsection.

6. Depth transitions.

Walked through in the SWIP (§Balance invariant: preservation under
insertion/removal; §Counting: the $N=2^D$ boundary cases for splitCount/donorCount).
The short version: transitions are emergent, not an event — no stored $d$, no
migration, the counters at the root already equal $2^{d+1}-N$ and $N-2^d$ and both
hit the boundary values exactly at powers of two. If a worked $N=2^D$ example would
help, I can add one next to the existing worked examples.

7. Complexity / gas.

§Gas and performance analysis: selection O(d) reads, activation/departure O(d)
writes, storage O(N), registration/expiry O(1) amortized (monotonic queue head,
bounded per call — no unbounded iteration anywhere). Mining is the real cost and it
is off-chain: expected $2^\ell$ hashes, $\ell \approx \log_2 N + 1$. Benchmarks are
listed as a reference-implementation deliverable; concrete numbers per depth once
there is a contract to measure.

8. The earlier notes (shrink, queue, gas of updates).

• Shrink/withdrawals: the departure path is now first-class — direct collapse
  completes immediately; the donor path holds the departing node active until a
  donor lands, with forfeiture + redraw on default. Withdrawal of stake itself is
  the staking contract's business (strict separation in the SWIP).
• Queue: yes — two commit queues ($C_R$, $C_D$), append-only with monotonic
  block heights, head-advancing expire bounded per call.
• Gas of data-structure updates: O(log N) slot writes per structural change; see
  point 2 for why the proposed alternative doesn't beat it once selection is
  accounted for.

[zelig]

@significance Dan — your Aug/Sep comments all predate the 2026-07-27 restructure. Several of them are now addressed in the text: the binary trie you endorsed is in as the ICBT (implicit complete binary trie, with a layout-comparison subsection), de/registration runs through commit queues with bounded expiry, and there is now a proper Terminology section separating these tree depths from storage depth. Still open from your list: activity/liveness tracking (squat-attack eviction), quantified economic disincentives (explicitly deferred to deployment/staking spec), key decoupling, and onboarding adherence proofs.

Please re-read the current version and leave a proper GitHub review (approve / request changes with the open points) rather than comments — it would help move this toward Accepted.

[Copilot]

🟡 Changes recommended

The SWIP text still contains an unresolved (?) ambiguity and it contradicts the PR description about appended/generated Solidity material.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 2
• Review effort level: Lite

[significance]

it was discussed that the scheme is at direct risk of a secondary market arising if there is some value to acquiring contiguous operating addresses which in the current swarm architecture there indeed is

it was discussed that the possibility of a secondary market becoming established could be to some extent mitigated by forcing an EOA account which cannot be changed and is allowed to withdraw stake in a way that cannot be prevented

it is further emphasised that neutering the ability should be explicitly avoided (and a comment retained in the relevant parts of the code) despite this approach necessitating undesirable ux as it requires forcing unstaking/restaking and reallocation of overlay address location if this withdrawal address needed to be changed eg. in the event of compromise

further note: the withdrawal address must be proven owned to satisfy this requirement - i.e. cannot be an obviously fake address eg. the zero address

[significance]

relevant....

Forcing EOA-only calls in Solidity

The check

require(msg.sender == tx.origin && msg.sender.code.length == 0, "EOA only");

• msg.sender == tx.origin — rejects contracts, including ones calling from their constructor.
• code.length == 0 — rejects EIP-7702 delegated accounts, which pass the first check while running code.

Either clause alone is insufficient: code.length alone is bypassed by the constructor trick (runtime code isn't stored until construction finishes, so a contract calling out mid-constructor reports zero); tx.origin alone is bypassed by 7702.

Risks

Risk	Detail
Excludes real users	Breaks ERC-4337 smart accounts, Safes, and anyone with a 7702 delegation set — a growing share of wallets.
Not a real boundary	Attacker clears the delegation, splits across transactions, or colludes with a builder.
No composability	Nothing can integrate with you — no routers, aggregators, vaults, or batchers, ever.
Opcode risk	tx.origin semantics have been repeatedly proposed for change; a future fork could alter the guarantee.
Wrong target	It gates account shape, not the behaviour you actually care about.

[significance]

@zelig leaving this here so it can be considered while i continue the review - the extreme manifestation of this is when viewed from the perspective of the last unmarried sister in the depth - for them it is a no brainer to stake the contiguous slot presuming there is economic benefit from doing so as there is in eg. the redistribution game. the obvious solution leads to a trade off of eroded efficiency in depth saturation, which can be mitigated by redundancy/shared responsibility, but this now becomes an essential constituent and hence should form part of this specification. perhaps there is a better way?

The problem (at it's most extreme) is that when a depth $d+1$ neighbourhood is nearly full, the remaining free addresses inside that neighbourhood can collapse the selection entropy, in the limit to a single address: if $2^(d+1)-1$ of its $2^(d+1)$ addresses are already occupied, the final address is predictable from the existing state, so treating that depth $d+1$ neighbourhood as a fixed candidate gives the adversary a deterministic target. The fix is to expand the candidate neighbourhood across the boundary until it contains more than one free address and select the new assignment from this new expanded set. In other words, preserve the local geometry as far as possible, but when the boundary becomes saturated, let the candidate region grow just enough that the choice is again genuinely non-deterministic.

[awmacpherson]

I'm writing my first batch of feedback here rather than commenting on specific paragraphs in code review. These comments are all about high-level motivation rather than implementation.

The only concrete motivation given is to make it more costly to execute what I will call a deduplication attack, i.e. running two node identities on the same underlying storage resources, masquerading it as replicated storage. To accept this as motivation, we need an assessment of the likelihood and harm of this type of attack. (The motivation in terms of DSNs is not concrete because it doesn't name a specific issue that Swarm faces today; see below.)

A second possible motivation that we discussed on our 1-1 call is to make it harder to Sybil the consensus over the reserve. However, since it didn't seem like that was your primary focus, and you haven't mentioned it in the SWIP document, I won't address that point in this review.

Harm. How harmful is the attack? Basically, a node that represents itself as multiple nodes does two things:

1. Misrepresent risk metrics. Misrepresents the number of replicas of a particular shard that the network is hosting. If replica count is considered as the primary proxy of data durability, this ultimately means the risk of data loss is misrepresented. If, on the other hand, we are primarily relying on client-side erasure coding for fault tolerance, replica count may not actually be that important (see Long-term roadmap).
2. Weaken the supply side by pricing out competitors. An operator who saves on compute/storage resources by masquerading it as multiple nodes in the same neighbourhood has better unit economics than "honest" nodes, and can therefore eventually unfairly outcompete them and take over the network. Nodes should not be able to save money by claiming to provide more storage capacity than they actually do.

Status quo. The current system defends against the deduplication attack via the elective (self-selected) stake system: if you want to scale your revenue in a single neighbourhood, it is cheaper and simpler to do so by scaling stake alone instead of deploying a full Sybil identity. This shrinks the population of entities who might attempt this attack, i.e. it makes it less likely.

Given the elective stake system in place, it is not clear to me that there is any urgent need for further defense against the deduplication attack; especially when the measure involves such a dramatic overhaul of the current system.

Under centrally quoted stake. If we take it as given that we want to do away with the elective stake system, replacing it with a mechanism where the system tells you how much you must stake — and I suspect this is among your assumptions, even though it is not stated explicitly in the SWIP — then we may need a new mitigation.

Long term roadmap. I think we both believe that in the long term, Swarm ought to move towards a ~1 average replication rate and rely on erasure coding for fault tolerance. In that régime, it's not clear to me that a deduplication "attack" is actually harmful:

1. Risk metrics. No one is basing their durability assumptions on the existence of full replicas at the raw chunk level.
2. Undercut honest nodes. Most nodes are alone in their neighbourhood anyway. There's no "per-neighbourhood market" left to take over, so no incentive to pretend to host replicas.

In summary, I find the motivation plausible in the medium term (before Swarm does away with replication as the primary means to underwrite durability), conditional on getting rid of elective stake. This should probably be clarified in the Scope section.

Recommendations

1. Make it explicit that the motivation for this change is predicated on the assumption that the elective stake system will be replaced by a centrally quoted stake per node. We will also need to adequately motivate the centrally quoted system and settle on the details of how it works before rolling this out.

2. You also have some abstract theory of DSNs that you're invoking in the motivation section. I don't believe this theory will be useful in motivating these specific changes until it can be linked to a specific problem we have with Swarm today, e.g. if you can argue that the current Swarm network node population is not "as balanced" as it would be under the counterfactual proposal.

   The main questions I have about that are:

   • Is the current Swarm node population really not "balanced enough?" How balanced would be sufficient?
   • Why does the property you want from "balanced" (i.e. uniform distribution of work among nodes) necessitate the specific design philosophy of this SWIP (random assignment by a central authority)?

[awmacpherson]

The use of "areas of responsibility" is a bit confusing since the proposal doesn't specify anything about the responsibility of the nodes. Is the intention that nodes continue determine their AoR as they do today, i.e. using the depth that gives them between 2^21 and 2^22 chunks? If so, what does this proposal have to do with how those AoRs cover the address space?

OTOH if this is a statement about the set of balls defined by leaf prefixes (which are not generally the AoRs of the assigned nodes unless the method of determining AoR has changed), you are asking that these balls partition the address space. Why is that a goal?

[awmacpherson]

Why is this a goal?

[awmacpherson]

This was hard to understand on a first reading because you are using jargon from the construction. It reads as though the author defined the construction first and repackaged part of the construction as a goal, which it tautologically achieves.

The condition can be expressed in higher-level language, e.g. "Newly joining nodes are assigned prefixes uniformly at random from a set of available prefixes," which at least aids comprehensibility (though it is still unclear why this is a goal).

[awmacpherson]

If we accept that the main objective of this proposal is to centrally assign prefixes, it's easy to see why you want this.

This proposal does not actually achieve this requirement in all cases: if there are exactly $2^d-1$ nodes, the next node to join knows exactly which prefix it will receiv

[awmacpherson]

Again, jargon. Suggest "a bounded procedure for handling nodes that fail to take up their assigned prefix."

[awmacpherson]

What exactly is being validated here and by whom? That a given prefix is assigned to a node? This sounds like a very weak condition, why is this being mentioned?

[awmacpherson]

In this section, you appear to be defining a property of a function $L:S \rightarrow Prefixes$. I think it would be clearer if you made it explicit that it's a definition and use function notation.

[awmacpherson]

Here you are defining a procedure to construct a balanced $L:S\rightarrow Prefixes$ given balanced $L:S\setminus\{s\}\rightarrow Prefixes$.

[awmacpherson]

Suggest using a more compact notatation for the complement of a byte, e.g. $\bar{b}$ which is standard.

[awmacpherson]

Defining balanced $L:S\setminus\{s\}\rightarrow Prefixes$ given balanced $L:S\rightarrow Prefixes$

[awmacpherson]

Suggest moving this section to the top, after Abstract.

[awmacpherson]

I write this review under the assumption that the high level goal is to ensure that the number of nodes serving each point in the address space is "as close to constant as possible." The proposal doesn't specify whether or how nodes' responsibilities would change, so I'll assume they continue to reckon their AoR as they do today based on the overlay address they mine and the number of chunks they can find in proximity to their address. I will refer to the view of depth stored by each node based on this count as the reserve depth, and the $d$ used in the proposal as registry depth.

The healthy situation is that reserve depth is less than registry depth. In this case, the proposal guarantees that the number of prefixes assigned in two distinct neighbourhoods (at the reserve depth) differs by at most a factor of two. (Of course, it doesn't guarantee that assigned prefixes correspond to currently active nodes, but we can likely get fairly high confidence of that by tuning incentives.) I don't know whether the distribution of node count in each neighbourhood at a given $N$ and depths has a standard name, but we can easily simulate it. If my characterisation of the high level goal is correct, I suggest someone run these simulations and attach plots of the distributions to the proposal. We can then compare those plots to the allocation of nodes today to better understand whether SWIP-39 would improve things.

[awmacpherson]

I'm sure this has already been raised, but a major concern is churn triggered by node exits that require a node to be transplanted from another prefix. The cost of exiting would have to be tuned to reflect the costs to the donor node (who would likely stop earning while they sync the new neighbourhood) and the externality on the network. This would likely make it very expensive to exit large numbers of nodes, which in turn makes it less attractive to operator fleets at scale.

Exiting large numbers of nodes would cause more harm to the network under SWIP-39 than it does today, because as well as the loss of replicas you are also likely triggering large numbers of overlay changes at the same time.