Keep unreferenced blobs newer than prune_grace_period (closes #387)

A backup uploads its blobs before the manifest that lists them, and the
process lock covers one user on one machine, so a prune from another host
or user sharing the destination deleted a running backup's blobs.

Prune now deletes an unreferenced blob only once its listed time is older
than the new config key prune_grace_period (default 24h; a negative value
is refused). A newer blob, or one listed with no time, is kept and counted
(blobs_too_new). The rclone backend takes each object's time from the
listing (use_server_modtime).

Decision: a blob a backup reuses from an earlier run that stopped before its manifest keeps its old time and is not protected; the docs say so.

Model: opus-5-5

README.md

@@ -96,6 +96,14 @@ vaultik snapshot restore <snapshot-id> ~/vaultik-restore
 # 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w
 ```
 
+On a destination shared between hosts or users, the prune in that cron line
+relies on `prune_grace_period` (default `24h`) to leave the blobs of a backup
+another one is still running, so it must be longer than the longest backup
+run. It does not protect a blob that backup reuses from an earlier run that
+stopped before its manifest: the prune can delete such a blob, and the new
+snapshot then cannot be restored. To not depend on the grace period, give each
+host its own prefix or directory. See [locking](#locking).
+
 ## restoring on another machine
 
 Restoring on a host that never ran the backup — a replacement machine
@@ -192,7 +200,22 @@ process-wide lock at `$XDG_DATA_HOME/vaultik/vaultik.pid`
 (`~/.local/share/vaultik/vaultik.pid` on Linux) for the whole run. Only
 one of them runs at a time: a second one exits immediately with an
 "already running" error rather than waiting, so two writers can never
-corrupt the local index or the destination store.
+corrupt the local index.
+
+The lock covers one user on one machine. It does not stop another user,
+or another host, from writing to the same destination store at the same
+time. A backup uploads its blobs before the manifest that lists them, so a
+prune from elsewhere at that moment finds blobs no snapshot lists. It
+deletes such a blob only once it is older than `prune_grace_period`
+(default `24h`), which therefore must be longer than the longest backup run
+of any host or user sharing the destination. The grace period does not
+protect a blob a backup reuses from an earlier run that stopped before its
+manifest: a backup skips the upload of a blob already on the destination, so
+that blob keeps the time of its first upload, and a prune from elsewhere
+during the backup can delete it; the new snapshot then cannot be restored.
+`remote nuke` deletes everything regardless. To not depend on the grace
+period, give each host or user its own prefix (or directory) on the
+destination.
 
 Read-only commands — `info`, `snapshot list`, `snapshot verify`, and
 `remote info` — do not take the lock and are never blocked, so they run
@@ -296,7 +319,8 @@ value where a list belongs or a word where a number belongs), is refused.
 * `--cron`: Silent on total success; warnings and errors are still printed
   (for crontab)
 * `--prune`: After backup, drop older snapshots of each backed-up name and
-  remove orphaned blobs from remote storage. By default keeps only the latest
+  remove orphaned blobs older than `prune_grace_period` from remote storage,
+  as `prune` does. By default keeps only the latest
   snapshot per name; use `--keep-newer-than` for a rolling window. When
   `--skip-errors` skipped a file or directory it could not read or list, the
   prune does not run and every older snapshot is kept, since one of them may
@@ -423,9 +447,15 @@ local snapshot whose remote metadata is missing is dropped from the
 local index; (2) delete orphaned local rows (files, chunks, blobs no
 longer referenced by any completed snapshot); (3) list every remote
 manifest on the destination store to compute the still-referenced blob
-set and delete any blob not in that set. Step (3) reads all remote
-manifests — network cost scales with the number of snapshots. `snapshot
-create --prune` runs the same cleanup automatically; this is the
+set and delete any blob not in that set that is older than
+`prune_grace_period` (default `24h`). A newer one is kept and counted: it
+may belong to a backup still running, on this host or another sharing the
+destination store, that has not yet uploaded its manifest. The grace period
+does not protect a blob such a backup reuses from an earlier run that stopped
+before its manifest: that blob keeps the time of its first upload, so it can
+be deleted. Step (3) reads all remote manifests — network cost scales with
+the number of snapshots.
+`snapshot create --prune` runs the same cleanup automatically; this is the
 manual entry point for the same work. If the destination store cannot
 be listed, or the local index holds snapshots and the destination has
 no `vaultik-destination.txt` (an unplugged volume's empty mount point),
@@ -436,6 +466,9 @@ exits non-zero.
 * `--json`: Output stats as JSON. Also skips the confirmation prompt, as
   `--force` does. A blob that cannot be deleted is counted in
   `blobs_failed`, and the document is printed before the non-zero exit.
+  A blob kept for being newer than `prune_grace_period`, or because the
+  listing of the destination gives no time for it, is counted in
+  `blobs_too_new` and not in `blobs_found`.
 
 **`info`**: Display system configuration, storage settings, encryption
 recipients, and local database statistics.
@@ -510,6 +543,14 @@ being written, a killed upload can leave a truncated object under its name,
 which a later backup takes for complete. A leftover `.partial` file is ignored
 and can be deleted.
 
+A destination shared between hosts or users, each with its own local index,
+relies on `prune_grace_period` to keep one's prune from deleting the blobs of
+another's backup still running, so it must be longer than the longest backup
+run. It does not protect a blob that backup reuses from an earlier run that
+stopped before its manifest, which the prune can delete. To not depend on the
+grace period, give each host its own prefix (or directory). See
+[locking](#locking).
+
 Legacy S3 configuration via `s3.*` fields (endpoint, bucket, prefix, etc.) is
 still supported for backward compatibility. `storage_url` takes precedence if
 both are set.
@@ -596,7 +637,8 @@ format does and does not protect.
 1. List all snapshot manifests
 2. Build set of all referenced blob hashes
 3. List all blobs in storage
-4. Delete any blob not in the referenced set
+4. Delete any blob not in the referenced set that is older than
+   `prune_grace_period`
 
 ### chunking and deduplication
 
@@ -641,6 +683,7 @@ complete annotated example also lives in
 | `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
 | `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
 | `compression_level` | `3` | zstd compression level (1-19) |
+| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it. A backup uploads its blobs before the manifest that lists them, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). A duration such as `36h` or `90m`; `0s` deletes such blobs at once |
 | `hostname` | system hostname | Hostname used in snapshot IDs |
 | `index_path` | platform data dir | Local SQLite index path |