Keep unreferenced blobs newer than prune_grace_period (closes #387)
check / check (push) Failing after 5m50s
check / check (push) Failing after 5m50s
A backup uploads its blobs before the manifest that lists them, and the process lock covers one user on one machine, so a prune from another host or user sharing the destination deleted a running backup's blobs. Prune now deletes an unreferenced blob only once its listed time is older than the new config key prune_grace_period (default 24h; a negative value is refused). A newer blob, or one listed with no time, is kept and counted (blobs_too_new). A backup uploads a blob already on the destination again instead of skipping it, so it gets a fresh time. The rclone backend takes each object's time from the listing (use_server_modtime). Cost: after database delete, the next backup uploads all its data again. Side effect: a rebuilt blob replaces the stored copy; if compression_level or the recipients changed since, older snapshots listing it fail verify. Model: opus-5-5
This commit is contained in:
+4
-3
@@ -90,7 +90,7 @@ Stores information about packed, compressed, and encrypted blob files.
|
||||
- `created_ts` (INTEGER NOT NULL) - Creation timestamp
|
||||
- `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress)
|
||||
- `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression
|
||||
- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and encryption. Once uploaded, the size of the object in storage: when the blob was already stored, under another compression level or other recipients, this can differ from the size of the local copy
|
||||
- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and encryption. Once uploaded, the size of the object in storage
|
||||
- `uploaded_ts` (INTEGER) - Upload completion timestamp (NULL if not uploaded)
|
||||
|
||||
**Purpose:** Tracks blob lifecycle from creation through upload. The UUID primary key allows immediate association of chunks with blobs.
|
||||
@@ -185,7 +185,7 @@ Tracks blob upload metrics.
|
||||
- `UPDATE blobs SET blob_hash = ?, finished_ts = ?` - Finalize blob after packing
|
||||
|
||||
4. **Upload**
|
||||
- `UPDATE blobs SET uploaded_ts = ?, compressed_size = ?` - Mark blob as uploaded, with the size of the object in storage (the upload is skipped when the blob is already stored)
|
||||
- `UPDATE blobs SET uploaded_ts = ?, compressed_size = ?` - Mark blob as uploaded, with the size of the object in storage (a blob already stored is uploaded again, so that a prune does not take it for an unreferenced blob older than `prune_grace_period`)
|
||||
- `INSERT INTO uploads` - Record upload metrics with snapshot_id
|
||||
- `INSERT INTO snapshot_blobs` - Associate blob with snapshot
|
||||
|
||||
@@ -228,7 +228,8 @@ The restore process doesn't use the local database. Instead:
|
||||
|
||||
1. **Identify Unreferenced Blobs**
|
||||
- Query blobs not referenced by any remaining snapshot
|
||||
- Delete from S3 and local database
|
||||
- Delete from S3, once older than `prune_grace_period`, and from the local
|
||||
database
|
||||
|
||||
### 6. Incomplete Snapshot Cleanup
|
||||
|
||||
|
||||
Reference in New Issue
Block a user