Skip to content

fix(rest): unbreak CSI clone-from-volume and make snapshot restore idempotent - #190

Open
Andrei Kvapil (kvaps) wants to merge 4 commits into
mainfrom
fix/csi-clone-and-restore-idempotency
Open

Andrei Kvapil (kvaps) wants to merge 4 commits into
mainfrom
fix/csi-clone-and-restore-idempotency

Conversation

@kvaps

@kvaps Andrei Kvapil (kvaps) commented Sep 2, 2026 •

Copy link
Copy Markdown
Member

Two CSI-facing defects reported from a Cozystack stand running the blockstor backend.

The internal clone-snapshot reap that used to ride here moved to #200, stacked on this one, and the test-harness fixes to #199. This one is the two fixes and what they strictly need: a refactor that splits store-error and body-decode answers from their envelope, and delete_namespaces on resource-definition and group modify, which the clone path uses.

Clone rejected every request golinstor sends

The endpoint declared five fields and decoded with DisallowUnknownFields, so layer_list, which linstor-csi always sends, was a 400 before any of the handler ran, and every clone-from-volume failed. resource_group was the next 400 behind it. Both are now honoured on both clone paths, validated the way rg modify validates them, the stack stored in canonical case, and the group is checked to exist. external_name and volume_passphrases are refused rather than accepted and dropped.

A layer_list on a source with volumes must match the source's stack as a set: the clone restores the source's bytes and then brings the stack up over them, so an added LUKS layer formats the restored data and a dropped one leaves the target reading bytes it cannot decode. Without layer_list, the target is stamped with the source's data-plane stack, so the target group's stack cannot change what the control plane believes about it. The volume-less shell clone gets the same stamp.

Refusals reach linstor-csi as the []ApiCallRc golinstor decodes; python-linstor keeps the CloneStarted object it decodes. The shape is picked by User-Agent. golinstor turns any 404 into a bare NotFoundError, so a 404's cause never reaches the PVC's events.

Restore through linstor-csi was refused on its first call

The actual cause of #186. linstor-csi's VolFromSnap creates the resource definition itself, restores the volume definitions, then calls snapshot-restore-resource. The definition it created carried no restore marker, so the restore took it for somebody else's and answered 409 on the first call, every time. That is word for word the error in the report.

A prepared target is now accepted when its volumes are exactly the snapshot's, it has no live replica, its group still exists, and its layer set matches the source's. The marker is stamped only after all of that, so a refusal leaves an operator's definition untouched. The CLI follows the same LINSTOR sequence (rd c, s vd restore, s rsc restore). The decisive reads go past the informer cache, since linstor-csi's calls can land on different apiserver replicas.

Retries resume instead of failing for good

CSI requires CreateVolume to be idempotent. A leftover carrying this operation's marker is resumed, judged by one function both doors and the CLI share: tear-down first, then the parent group, then an abandoned-rollback mark, then the volumes and a live replica. A finished leftover is answered as a replay and is left alone. Only a replica with a disk counts: a diskless or tie-breaker one never makes a leftover finished. An operator's diskless replica on a node the request places onto is refused; the controller's tie-breaker witness there is promoted the way autoplace promotes one. A finished restore still answers as a replay after its snapshot is deleted, judged against the record of the volumes the snapshot held.

Two requests on one target are handled explicitly. A request that resumes a leftover marks it adopted before writing anything, and the creator's rollback re-reads that mark and yields instead of deleting what the other request answered 201 for. Both sides fail closed: a rollback that cannot hold or read the handshake deletes nothing and says to retry. A failed placement that already left the definition finished, with every volume and one replica holding data, is kept rather than rolled back on both doors, since a replay may have answered for it; the error names the requested replicas that never landed.

The props blockstor writes on a definition (the restore marker, the record of the snapshot's volumes, the adoption and rollback marks) are refused in clone prop edits, rd modify and the CLI property verbs.

Testing

Every behavioural change is pinned by a test checked by reverting the change and confirming the named test goes red, and a mutation sweep over every new error return leaves none uncovered.

docs/cli-parity-known-deltas.md rows 86 and 87 describe the clone refusals and the retry semantics, including one accepted window: a replica an operator adds by hand between the prepared-target check and the marker counts as part of the restore.

The placer flake is #198. Locally, pkg/rest has a server did not stop within 2s flake in the test harness; the harness PR fixes the related port race but not this one.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants