Keep unreferenced blobs newer than prune_grace_period (closes #387)
check / check (push) Failing after 5m50s

A backup uploads its blobs before the manifest that lists them, and the
process lock covers one user on one machine, so a prune from another host
or user sharing the destination deleted a running backup's blobs.

Prune now deletes an unreferenced blob only once its listed time is older
than the new config key prune_grace_period (default 24h; a negative value
is refused). A newer blob, or one listed with no time, is kept and counted
(blobs_too_new). A backup uploads a blob already on the destination again
instead of skipping it, so it gets a fresh time. The rclone backend takes
each object's time from the listing (use_server_modtime).

Cost: after database delete, the next backup uploads all its data again.
Side effect: a rebuilt blob replaces the stored copy; if compression_level
or the recipients changed since, older snapshots listing it fail verify.

Model: opus-5-5
This commit is contained in:
2026-10-14 08:57:35 +00:00
parent 2590f49583
commit 63c824220c
25 changed files with 639 additions and 114 deletions
+4 -3
View File
@@ -90,7 +90,7 @@ Stores information about packed, compressed, and encrypted blob files.
- `created_ts` (INTEGER NOT NULL) - Creation timestamp
- `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress)
- `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression
- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and encryption. Once uploaded, the size of the object in storage: when the blob was already stored, under another compression level or other recipients, this can differ from the size of the local copy
- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and encryption. Once uploaded, the size of the object in storage
- `uploaded_ts` (INTEGER) - Upload completion timestamp (NULL if not uploaded)
**Purpose:** Tracks blob lifecycle from creation through upload. The UUID primary key allows immediate association of chunks with blobs.
@@ -185,7 +185,7 @@ Tracks blob upload metrics.
- `UPDATE blobs SET blob_hash = ?, finished_ts = ?` - Finalize blob after packing
4. **Upload**
- `UPDATE blobs SET uploaded_ts = ?, compressed_size = ?` - Mark blob as uploaded, with the size of the object in storage (the upload is skipped when the blob is already stored)
- `UPDATE blobs SET uploaded_ts = ?, compressed_size = ?` - Mark blob as uploaded, with the size of the object in storage (a blob already stored is uploaded again, so that a prune does not take it for an unreferenced blob older than `prune_grace_period`)
- `INSERT INTO uploads` - Record upload metrics with snapshot_id
- `INSERT INTO snapshot_blobs` - Associate blob with snapshot
@@ -228,7 +228,8 @@ The restore process doesn't use the local database. Instead:
1. **Identify Unreferenced Blobs**
- Query blobs not referenced by any remaining snapshot
- Delete from S3 and local database
- Delete from S3, once older than `prune_grace_period`, and from the local
database
### 6. Incomplete Snapshot Cleanup