Go on with a backup when the destination marker cannot be checked (closes #389)

A source host whose credentials allow only PUT, which the documents
say is enough to back up, failed every backup after its first: the
marker check looks the marker up with an S3 HEAD, which those
credentials are refused, and any error other than not found was
fatal.

Such an error now means the check could not be made: it is logged at
info level, so --cron stays silent, and the command goes on. Not found
still stops it. The documents now say so, and that such a host does
not notice an emptied destination, so vaultik database delete must be
run on it after the destination is emptied. The fault-injecting test
store gains a Stat hook for the test.

Model: opus-5-5

ARCHITECTURE.md

@@ -382,15 +382,19 @@ bucket/
         └── manifest.json.zst        # Blob list (for pruning/verification)
 ```
 
-A destination without `vaultik-destination.txt` is treated as not present
-while the local index holds snapshots stored there (`checkDestinationMarker`
-in `internal/vaultik/destination_marker.go`): `snapshot create` refuses to
-write to it, `prune`, `snapshot purge` and `remote nuke` fail, and
-`snapshot list` and `snapshot remove` behave as for a destination that cannot
-be listed. This keeps those commands from taking an unplugged volume's empty
-mount point for a destination whose snapshots are all gone. The `file://`
-backend never creates its destination directory, so a write to a missing one
-fails.
+A destination that answers a look-up of `vaultik-destination.txt` with not
+found is treated as not present while the local index holds snapshots stored
+there (`checkDestinationMarker` in `internal/vaultik/destination_marker.go`):
+`snapshot create` refuses to write to it, `prune`, `snapshot purge` and
+`remote nuke` fail, and `snapshot list` and `snapshot remove` behave as for a
+destination that cannot be listed. This keeps those commands from taking an
+unplugged volume's empty mount point for a destination whose snapshots are
+all gone. A destination that refuses the look-up, as S3 does to credentials
+that allow only PUT, is not checked, so such a host does not notice an
+emptied destination; see
+[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#marker-file-vaultik-destinationtxt).
+The `file://` backend never creates its destination directory, so a write to a
+missing one fails.
 
 The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
 snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never

README.md

@@ -332,7 +332,9 @@ value where a list belongs or a word where a number belongs), is refused.
   command fails before the backup starts.
 * Fails before it writes anything when a `file://` destination directory
   does not exist, or when the local index holds snapshots and the
-  destination has no `vaultik-destination.txt` (see
+  destination answers that it has no `vaultik-destination.txt`. A
+  destination that refuses that look-up, as S3 does to credentials that
+  allow only PUT, is not checked (see
   [remote storage layout](#remote-storage-layout))
 
 **`snapshot list`**: Show every snapshot known to this host — the union
@@ -577,15 +579,22 @@ For an implementation-level view of the internals — the data model, the
 ```
 
 * `vaultik-destination.txt` holds one fixed line of text, with no host
-  name, path or time. A destination without it is treated as not
-  present while the local index holds snapshots stored there:
-  `snapshot create` refuses to write to it, `prune`, `snapshot purge`
-  and `remote nuke` fail, and `snapshot list` and `snapshot remove`
-  behave as for a destination that cannot be listed. This keeps an
-  unplugged volume's empty mount point from being taken for a
-  destination whose snapshots are all gone. Mount
+  name, path or time. While the local index holds snapshots stored
+  there, a destination that answers a look-up of it with not found is
+  treated as not present: `snapshot create` refuses to write to it,
+  `prune`, `snapshot purge` and `remote nuke` fail, and `snapshot list`
+  and `snapshot remove` behave as for a destination that cannot be
+  listed. This keeps an unplugged volume's empty mount point from being
+  taken for a destination whose snapshots are all gone. Mount
   the volume if it is removable; if the destination was emptied on
   purpose, run `vaultik database delete` and back up again.
+  When the store refuses the look-up, the check is skipped and the
+  command goes on. S3 refuses it on every run to credentials that
+  allow only PUT, so a host with such credentials does not notice a
+  destination emptied since its last backup, and its next backup would
+  leave out the data its local index records as already stored. After
+  the destination is emptied, run `vaultik database delete` on such a
+  host before it backs up again.
 * Blobs are two-level directory sharded using the first 4 hex chars of the blob hash
 * `db.zst.age` is a binary SQLite database (zstd compressed, age encrypted)
   containing all file metadata, chunk mappings, and relationships for the snapshot

docs/REPOSTRUCTURE.md

@@ -27,7 +27,9 @@ The metadata subdirectory is named with the **remote key**, a one-way hash of th
 
 ## Marker File (`vaultik-destination.txt`)
 
-Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. A destination without it is treated as not present while the local index holds snapshots stored there: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove`, `snapshot list` and `remote nuke` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last.
+Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. While the local index holds snapshots stored there, a destination that answers a look-up of the marker with not found is treated as not present: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove`, `snapshot list` and `remote nuke` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last.
+
+When the store refuses the look-up of the marker, the check is skipped and the command goes on. S3 refuses it on every run to credentials that allow only PUT, so a host with such credentials does not notice a destination emptied since its last backup, and its next backup would leave out the data its local index records as already stored. After the destination is emptied, run `vaultik database delete` on such a host before it backs up again.
 
 ## Blobs Directory (`blobs/`)
 

internal/vaultik/destination_marker.go

@@ -11,9 +11,10 @@ import (
 
 // destinationMarkerKey is the object every backup writes at the root of
 // the backup destination store. While the local index holds a snapshot
-// stored there, a destination without it is treated as not present: an
-// unplugged volume leaves an empty directory at its mount point, which
-// would otherwise read as a store whose snapshots are all gone.
+// stored there, a destination that answers a look-up of it with not
+// found is treated as not present: an unplugged volume leaves an empty
+// directory at its mount point, which would otherwise read as a store
+// whose snapshots are all gone.
 const destinationMarkerKey = "vaultik-destination.txt"
 
 // destinationMarkerText is the whole content of the marker. It names no
@@ -21,8 +22,8 @@ const destinationMarkerKey = "vaultik-destination.txt"
 const destinationMarkerText = "This is a vaultik backup destination store.\n"
 
 // errDestinationNotPresent is returned when the local index holds a
-// snapshot stored on the backup destination store and the store has no
-// marker.
+// snapshot stored on the backup destination store and the store answers
+// that it has no marker.
 var errDestinationNotPresent = errors.New("backup destination store is not present")
 
 // checkDestinationMarker returns errDestinationNotPresent when the local

internal/vaultik/prune.go

@@ -58,8 +58,8 @@ func (v *Vaultik) NukeRemote(force bool) error {
 
 	// Checked before anything is deleted: running nuke again would fail
 	// the same way, so the advice below must not follow these errors.
-	// The listing also fails on a destination without the marker while
-	// the local index holds snapshots.
+	// The listing also fails on a destination that answers that it has
+	// no marker while the local index holds snapshots.
 	err := v.EnsureStorageBinding()
 	if err != nil {
 		return err