Repository navigation
Backblaze B2 transient REST 500 leaves stale restic locks and breaks subsequent backups #8164
Description
Steps to reproduce
- Configure scheduled NS8 application backups with a Backblaze B2 destination.
- Configure a retention policy (
forget --prune --keep-last=7). - Run the scheduled backup.
- The issue is intermittent and cannot currently be reproduced on demand.
During a scheduled run, the following sanitized sequence was recorded:
2026-07-16T00:05:16+02:00 Remove(<lock/c4fc23a454>) failed:
<lock/c4fc23a454> does not exist
2026-07-16T00:05:22+02:00 Load(<lock/c4fc23a454>, 0, 0)
returned error: unexpected HTTP response (500): 500 Internal Server Error
2026-07-16T00:05:25+02:00 retention policy failed:
restic forget --prune --keep-last=7 returned exit status 11
2026-07-16T00:05:30+02:00 Load(<key/afd159cbff>, 0, 0)
returned error: unexpected HTTP response (500): 500 Internal Server Error
The error affected different application repositories in the same time window, so it was not specific to Traefik application data.
Expected behavior
A transient backend error should not leave the repository permanently unusable. A later scheduled run should either safely remove a stale lock when no Restic process is active, or report a clear recovery action.
Repository access failures should not cause the agent to attempt restic init on an already initialized repository.
Actual behavior
The lock created during the failed operation remained in the B2 repository. Retention failed on every subsequent run with exit code 11:
repository is already locked by PID 1 on ns1 by root
lock was created at 2026-07-15 22:05:08
The same lock was still reported about 216 hours later and again about 240 hours later. Eventually scheduled backups failed until a manual restic unlock was performed.
A later run also tried to initialize the repository after access failed because of the lock, producing the misleading secondary error:
Fatal: create repository failed: config file already exists
Before unlocking, no Restic or backup process was running. After the manual unlock:
restic checksucceeded on the affected repositories;- snapshots were readable;
- a complete scheduled backup finished successfully.
Components
- NS8 single-node installation
- Backup destination: Backblaze B2
- Image at first occurrence:
ghcr.io/nethserver/restic:3.20.0 - Current image:
ghcr.io/nethserver/restic:3.21.1 - Affected repositories included Traefik and Loki
- The simultaneous WebTop key-load failure indicates a shared destination/gateway failure rather than an application-specific issue
No credentials, bucket names, customer identifiers or repository UUIDs are included in this report.
See also
- crash on unlock restic/restic#3541
- restic trying to get lock file that no longer exists restic/restic#4515
Reported from a customer installation; all logs have been sanitized.
Activity
DavidePrincipi commentedon Oct 16, 2026
Analysis
We reproduced this problem and found its causes. There are three separate parts.
1. Restic leaves its lock after a temporary B2 error (
Remove(<lock/...>) failed: ... does not exist)Restic does not access B2 directly: it goes through the rclone-gateway, which runs
rclone serve restic. When rclone receives aDELETErequest and the lookup of the file fails for any reason, it replies404 Not Found, while aGETwith the same error gets500. Restic considers the 404 final: it does not retry, and the lock stays in the repository. The rclone-gateway setsLowLevelRetriesto 1, so rclone does not retry the failed B2 request either. A single temporary B2 error is enough.We observed the same sequence on another installation with a B2 destination, and reproduced the rclone behavior deterministically with plain rclone and restic, without NethServer:
2. The lock is never removed, and backups keep failing until a manual
restic unlockNothing removed stale locks. Fixed by NethServer/ns8-core#1351 (#8229), released in core 3.23.1: before each application backup,
restic unlockremoves the locks older than 30 minutes. A leftover lock now makes at most one backup run fail, and the next run removes it.3. Misleading
restic initattempt (create repository failed: config file already exists)The backup ran
restic initwheneverrestic snapshotsfailed, also when the repository exists but cannot be accessed, for example because of a leftover exclusive lock. NethServer/ns8-core#1366 runsrestic initonly when Restic reports that the repository does not exist (exit code 10), and otherwise reports the real failure.Issue #8239 was a duplicate of this one and has been closed.
DavidePrincipi commentedon Oct 19, 2026
In Testing core 3.23.2-dev.1
DavidePrincipi commentedon Oct 19, 2026
VERIFIED
DavidePrincipi commentedon Oct 19, 2026
Released https://github.com/NethServer/ns8-core/releases/tag/3.23.2