Backup fails when a Restic lock cannot be removed #8239

A module backup can fail because Restic cannot remove its own lock from the repository, when the backup destination returns a temporary error at that moment. The lock stays in the repository and the retention policy of the same backup run fails.

Steps to reproduce

The problem is timing-dependent and was observed on a production node, with a Backblaze B2 destination:

• Run a scheduled backup of an application to a cloud destination (B2/S3)
• After the backup, the restic stats step creates its lock, then removes it when it finishes
• If the destination returns a temporary error when the lock is removed, Restic receives 404 Not Found instead of an error, and does not retry

Remove(<lock/e8dd84ad74>) failed: <lock/e8dd84ad74> does not exist
error while unlocking: <lock/e8dd84ad74> does not exist
repo already locked, waiting up to 0s for the lock
unable to create lock in backend: repository is already locked by PID 1 on <node> by root (UID 0, GID 0)
lock was created at 2026-10-15 23:01:50 (27.67484682s ago)
[ERROR] retention policy failed for <destination>/mattermost/<uuid>: Command '[...'forget', '--prune', '--keep-last=5']' returned non-zero exit status 11.
mattermost1: backup 1 failed (exit code 1)

The lock was still in the repository the next morning, readable and listed.

Expected behavior

A temporary error of the destination is retried, and the lock is removed.

Actual behavior

The backup data is saved, but the backup is reported as failed because the retention policy (forget --prune) finds the leftover lock. The next run removes it, as it is older than 30 minutes (#8229).

When rclone serves the Restic REST protocol and the lookup of a file to delete fails for any reason, it replies 404 Not Found, while the same failure on a read request gets 500 Internal Server Error. Restic considers the 404 final: it does not retry and leaves the lock in place. The real error is logged by rclone only at debug level.

The rclone-gateway configures its backends with LowLevelRetries set to 1, so rclone does not retry a failed request to the destination either: a single temporary error is enough to trigger the problem. Backblaze B2 returns temporary 500 errors from time to time.

Proposed fix: raise the rclone low level retries of the rclone-gateway backends, so that temporary errors of the destination are retried by rclone before replying to Restic.

Components

NethServer 8 core 3.23.1

See also

• Backup fails because of a stale Restic lock #8229