Keep unreferenced blobs newer than prune_grace_period (closes #387)

A backup uploads its blobs before the manifest that lists them, and the
process lock covers one user on one machine, so a prune from another host
or user sharing the destination deleted a running backup's blobs.

Prune now deletes an unreferenced blob only once its listed time is older
than the new config key prune_grace_period (default 24h; a negative value
is refused). A newer blob, or one listed with no time, is kept and counted
(blobs_too_new). The rclone backend takes each object's time from the
listing (use_server_modtime).

Decision: a blob a backup reuses from an earlier run that stopped before
its manifest keeps its old time and is not protected; the docs and config
comments say so.

Model: opus-5-5

README.md

@@ -683,7 +683,7 @@ complete annotated example also lives in
 | `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
 | `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
 | `compression_level` | `3` | zstd compression level (1-19) |
-| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it. A backup uploads its blobs before the manifest that lists them, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). A duration such as `36h` or `90m`; `0s` deletes such blobs at once |
+| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it. A backup uploads its blobs before the manifest that lists them, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs at once |
 | `hostname` | system hostname | Hostname used in snapshot IDs |
 | `index_path` | platform data dir | Local SQLite index path |
 

config.example.yml

@@ -333,6 +333,9 @@ compression_level: 5
 # A backup uploads its blobs before the manifest that lists them, so on a
 # destination shared between hosts or users this must be longer than the
 # longest backup run
+# It does not protect a blob a backup reuses from an earlier run that stopped
+# before its manifest: that blob keeps the time of its first upload, so a
+# prune can delete it
 # Supports: 36h, 90m, etc.; 0s deletes them at once
 # Default: 24h
 #prune_grace_period: 24h

internal/cli/config.go

@@ -242,7 +242,10 @@ storage_url: ""
 # How old a blob that no snapshot lists must be before prune deletes it.
 # A backup uploads its blobs before the manifest that lists them, so on a
 # destination shared between hosts or users this must be longer than the
-# longest backup run. Accepts: 36h, 90m, etc. 0s deletes them at once.
+# longest backup run. It does not protect a blob a backup reuses from an
+# earlier run that stopped before its manifest: that blob keeps the time of
+# its first upload, so a prune can delete it.
+# Accepts: 36h, 90m, etc. 0s deletes them at once.
 # Default: 24h
 # prune_grace_period: 24h