Commit Graph
144 Commits
Author SHA1 Message Date
sneak 52f44af4e2 Refuse a backup destination an unplugged volume left empty (closes #239)
check / check (push) Waiting to run
Every backup writes vaultik-destination.txt at the destination root.
While the local index holds snapshots, a destination without it is
not present: snapshot create refuses before writing, prune and
snapshot purge fail, and snapshot list and snapshot remove act as for
a destination that cannot be listed. snapshot create checks before it
drops incomplete snapshots, snapshot remove before it looks on the
destination. remote nuke deletes the marker last. The file:// backend
never creates its destination directory; that protects a directory on
a removable volume, not a mount point.

Tests back up into directories that exist; fixtures for a written
destination carry the marker.

Judgement call: the marker is rewritten on every backup.
Deviation: for the complexity limit, two blocks moved, unchanged, into
snapshotNamesToCreate and confirmRemoveSnapshot.

Model: opus-5-5
2026-10-14 02:22:47 +00:00
clawbot d156a02c49 Name each prefix once in restore, verify and purge errors (closes #327)
check / check (push) Waiting to run
A missing blob during restore, a failed shallow verify and an invalid
purge duration each printed their prefix twice. FetchBlob now returns
the storage error as is, since restore and deep verify name the blob
themselves; purge returns the duration parser's error unchanged; and a
failed shallow verify wraps "problems found in storage" instead of
"verification failed", which the command already prints. Deep verify
words a storage error other than not found as a blob it could not
check, and its log line no longer says S3.

Already fixed for #314: the
deep verify size message naming S3 and other errors called missing;
tests now check both.
Not tested: the log line.

Model: opus-5-5
2026-10-14 04:04:14 +02:00
clawbot 9fd88a34c7 Fail snapshot remove when a destination-only snapshot's metadata cannot be deleted (closes #340)
check / check (push) Waiting to run
When a snapshot was only on the destination store and deleting its
metadata there failed, snapshot remove warned and exited 0 having
removed nothing. That failure is now an error. A snapshot in the local
index keeps its behaviour: it is removed from the local index and the
failed delete on the destination store is a warning.

The README and the help text said the `vaultik prune` follow-up is
printed on every success; they now say it is printed only when metadata
was removed from the destination store, and never under --json. The
comments on RemoveSnapshot and removeSnapshotRemote describe both
cases; removeSnapshotRemote's also says it returns true only when the
delete finished without an error and removed at least one object.

Model: opus-5-5
2026-10-14 02:20:01 +02:00
clawbot 5db2e920e5 Label the index cleanup totals as totals (closes #306)
check / check (push) Waiting to run
The summary printed by the local index cleanup before each backup gave,
in brackets after each removed count, the number of snapshots, files,
chunks or blobs left in the local index, worded "(N remain)", which
read as incomplete or orphaned rows left over: removing the only
incomplete snapshot printed "1 removed (1 remain)". Each line now
labels that figure as the total of its kind left in the local index.

Model: opus-5-5
2026-10-14 02:03:51 +02:00
clawbot 66ea7f5f45 Free a cached blob once no file still to restore needs it (closes #324)
check / check (push) Waiting to run
A restore of selected paths, or one where files failed under
--skip-errors, kept the blobs those files share in the temporary cache
until it ended: the sweeper freed a blob only once every file in the
snapshot database that uses it had been restored, and files outside the
selection or that failed never are.

The sweeper now asks the restore plan, through pendingFileUsesBlob,
whether any pending file still uses a cached blob. The plan holds only
this run's files and drops each one once it is restored or has failed.

The new test backs up 16 KiB files so it stays quick; the locality
backup helper takes the file size for that.

Model: opus-5-5
2026-10-14 01:47:48 +02:00
clawbot 30285c7a8c Silence prune and snapshot removal status under --cron and --quiet (closes #294)
check / check (push) Waiting to run
PruneBlobs, CleanupLocalSnapshots, RemoveSnapshot, RemoveAllSnapshots
and snapshot purge wrote their status lines straight to stdout, so
`snapshot create --cron --prune` printed on every successful run and
--quiet did not silence `prune`, `snapshot purge`, `snapshot remove` or
`remote nuke`. Those lines now go through internal/ui, which drops them
under --cron and --quiet; warnings, errors and --json documents still
print.

The confirmation prompts still write straight to stdout, as the
`database delete` prompt does, together with what they ask to delete:
the snapshot list before the purge prompt, and the blob count and size
before the prune prompt. With --force those lines are status lines.

The two purge test fixtures gained the UI writer production always sets.

Model: opus-5-5
2026-10-14 01:10:43 +02:00
clawbot d8f78a1f45 Keep remote metadata when purge cannot remove a snapshot locally (closes #337)
check / check (push) Waiting to run
snapshot purge, and the purge step of snapshot create --prune, logged a
failure to delete a snapshot from the local index, deleted that
snapshot's metadata from the destination store anyway, and exited 0. The
purge now stops at that snapshot with an error before touching its
remote metadata, so the command exits non-zero. Stopping matches what
the same loop already does when the remote delete fails; snapshots
purged before the failure stay purged.

The test drives snapshot purge only: the trigger that makes the local
index refuse the delete also breaks the backup's metadata export, so
snapshot create --prune cannot be tested that way.

Model: opus-5-5
2026-10-13 23:36:22 +02:00
clawbot 2c087be577 Fail restore for a requested path that matches nothing (closes #309)
check / check (push) Waiting to run
snapshot restore said nothing about a requested path that matched no
entry in the snapshot (a typo, a relative path, or a path through a
symlinked snapshot path, whose entries are stored under the resolved
directory) and exited 0, even when nothing matched. Each such path is
now named in a warning as soon as the file list is built, so it shows
even when the restore then stops on an error, and restore returns an
error after restoring the paths that do match. The restore summary
moved into finishRestore to keep Restore within the length limit.

Making a path through a symlink match is not part of this change.
Judgement call: when the restore itself fails, that error is returned.

Model: opus-5-5
2026-10-13 23:14:22 +02:00
clawbot 37d525c322 Make remote nuke delete every object under metadata/ and blobs/ (closes #291)
check / check (push) Waiting to run
remote nuke deleted only the snapshot metadata and blobs the snapshot
and blob listings recognise, so a snapshot's database whose manifest
was never uploaded, and any name under blobs/ that is not a blob hash,
survived while nuke reported the store empty.

After removing every snapshot, nuke now deletes every object listed
under metadata/ and blobs/, whatever its name, in place of the blob
prune step. A delete that fails is logged and the others still run. A
listing afterwards decides the result: nuke reports the store empty
only when it finds nothing, and otherwise names each object left and
fails.

Judgement call: nuke no longer prints the blob prune summary.

Model: opus-5-5
2026-10-13 22:50:09 +02:00
clawbot 5d8c515192 List a file:// destination through a symlinked directory (closes #296)
check / check (push) Waiting to run
List and ListStream walked the destination with afero.Walk, which uses
Lstat and so saw a symlinked metadata/ or blobs/ as one file, while Put,
Get and Stat follow the link. The store then listed as empty, and prune
dropped the local snapshot records and deleted every blob. The listings,
and DeletePartialUploads (used by remote nuke), now walk a view without
Lstat, so afero falls back to Stat and enters the link. A prefix lists
as empty only when truly missing; a missing symlink target on it or
above it, such as an unmounted volume, fails the listing.

Judgement call: a symlink loop is not detected; it is followed until the
system refuses the path, and the listing fails.

Model: opus-5-5
2026-10-13 21:36:23 +02:00
clawbot 42eb50b151 Report blob counts from snapshot verify --deep --json (closes #314)
check / check (push) Waiting to run
Deep verify set verified only on success and never set missing, so a
failed --deep --json run reported zero for both. It now counts each blob
whose contents pass, and its existence check covers every blob, counting
not-found ones as missing and wrong-size ones as mismatched, as shallow
verify does; it fails if it found any. Any other storage error stops the
check and fails with that error, so a blob that cannot be checked is
never reported as missing.

Deep verify still stops at the first damaged blob and reads no contents
while any blob is missing or the wrong size, so verified is 0 there.

Model: opus-5-5
2026-10-13 21:26:55 +02:00
clawbot 3361e9ebd9 Record each snapshot's uncompressed size when its backup runs (closes #360)
check / check (push) Waiting to run
snapshot list summed the chunks a snapshot reaches through its file rows
in the local index. A file's row is shared by every snapshot that lists
it and rewritten when the file changes, so backing up a changed file
changed every older snapshot's UNCOMPRESSED SIZE.

The backup now takes that total when it records the snapshot's stats,
into a new snapshots column, uncompressed_chunk_size, and snapshot list
reads the column, as it already does for new_chunk_size.

Model: opus-5-5
2026-10-13 20:43:19 +02:00
clawbot fb8abcd049 Stop restore --verify re-reporting files that failed to restore (closes #323)
check / check (push) Waiting to run
With --skip-errors, a file that failed to restore was verified anyway,
failed again because it was never written, and was added to the same
failure list as the restore failure. restore --verify then listed it
twice and called every entry a checksum mismatch.

Verification now skips the files that failed to restore and keeps its
own failures in a separate list, so failed restores and failed
verifications are reported apart, each path once. The restore still
exits non-zero. The tests cover a restore with skipped files where
nothing fails verification, and one where another restored file also
fails it.

Judgement call: the second test has its own trailing-byte file-system
wrapper; the one in restore_perms_test.go is in the internal test package.

Model: opus-5-5
2026-10-13 20:12:48 +02:00
clawbot 6aceaaf086 Record the stored size of a blob a backup finds already stored (closes #300)
check / check (push) Waiting to run
When a finished blob already existed at the destination, the upload was
skipped but the blob row, and so the manifest, kept the size of the
local copy. That copy can differ from the stored object when
compression_level or the recipients changed since it was written, and
then shallow and deep verify failed on a good snapshot. Marking a blob
uploaded now records the size of the object in storage: for an existing
blob, the size Stat reports.

Judgement call: the test adds a recipient rather than changing the
compression level, because an extra age recipient always makes the
local copy larger, while two zstd levels need not compress the test
data differently.

Model: opus-5-5
2026-10-13 20:01:42 +02:00
clawbot dfa3bbc9e1 Stop a stalled snapshot database read when restore or deep verify is interrupted (closes #356)
check / check (push) Waiting to run
Restore and deep verify read the snapshot database straight from
storage rather than through FetchBlob, so the earlier fix for stalled
blob reads did not cover it. A database read stalled on a slow USB
stick or network share kept the operation from returning after Ctrl-C;
the process then exited at the shutdown timeout and left the decrypted
database in the temp directory.

Both reads now wrap the storage reader in the close-on-cancel reader
FetchBlob uses, so a cancel ends the stalled read and the temp
directory is removed. The interrupt test now stalls either a blob or
the snapshot database, for both commands.

Limitation: a read stuck in the kernel that closing cannot interrupt,
such as one on an unresponsive hard NFS mount, still cannot be stopped.

Model: opus-5-5
2026-10-13 19:36:43 +02:00
clawbot 740f474c59 Make prune and remote nuke fail when a delete fails (closes #290)
check / check (push) Waiting to run
prune, and the prune step of snapshot create --prune, counted a blob
it could not delete and still exited 0. It now prints its summary or
--json document and then returns an error, which replaces the
"Failed to delete N blob(s)" line.

remote nuke only warned when a snapshot's metadata could not be
deleted, advised vaultik prune, which never removes metadata, and then
reported the store empty. It now stops with an error, does not report
the store empty, and says to run vaultik remote nuke --force again. It
checks that the local index belongs to the configured destination
before deleting anything, and that error carries no such advice. A
successful nuke no longer advises prune.

Model: opus-5-5
2026-10-13 19:29:52 +02:00
clawbot 4193ced1f6 Report blobs shallow verify could not check apart from missing ones (closes #318)
check / check (push) Waiting to run
Shallow snapshot verify counted every error from checking a blob or the
encrypted database as "missing" and printed the error nowhere, so a
permission error or an S3 403 read as data loss. Now only a not-found
answer from storage counts as missing. Any other error is reported as
could not be checked, with the error storage returned, in the text
report and under --json (the new unchecked list and
database_check_error field), and the run fails. Under --quiet, which
drops the report, each of those errors is logged as a warning on
stderr.

The database check moved into its own small function because the verify
function was at the linter's length limit.

Model: opus-5-5
2026-10-13 18:50:48 +02:00
clawbot e48bafccef Restore the other files when a chunk is in no blob under --skip-errors (closes #326)
check / check (push) Waiting to run
Restore planning stopped at the first file needing a chunk that no blob
holds, so --skip-errors never saw that file and nothing was restored.
The planner now leaves such a file out of the plan and returns it with
the reason, and each one goes through handleRestoreFileError like any
other failed file: skipped and counted under --skip-errors, fatal
without it. The fatal error now suggests --skip-errors, as other restore
failures do.

The skip-errors tests now share one backup helper; the new case removes
a blob from the snapshot metadata and exports it again.

Model: opus-5-5
2026-10-13 16:53:05 +02:00
clawbot bffd4578fc Record each snapshot's new chunk size when its backup runs (closes #311)
check / check (push) Waiting to run
snapshot list worked out NEW CHUNK SIZE from the snapshot's file rows. A
file's row is shared by every snapshot that lists it and rewritten when
the file changes, so a snapshot whose change was a modified file showed
0 B, and older snapshots' figures moved after later backups.

The scanner now totals the size of the chunks it stores that were not
stored before (the chunks chunk_count counts), the backup writes that
total to a new snapshots column, new_chunk_size, and snapshot list reads
it.

Judgement call: the figure is fixed when the backup runs, so removing an
earlier snapshot no longer raises a later snapshot's figure.

Model: opus-5-5
2026-10-13 16:44:14 +02:00
clawbot 005927a7fc Print snapshot list and purge sizes like every other command (closes #312)
check / check (push) Waiting to run
`snapshot list`, the `snapshot purge` listing and the `bytes` value in
terminal log lines divided by 1024 while labelling the result KB or MB,
so 6,329,198 bytes read 6.0 MB there and 6.3 MB in every other command.
All three now use humanize.Bytes, as the rest of the program does, and
the 1024-based helper in internal/vaultik is removed.

The size shared by the list tests is now 5,000,000 bytes, which the two
ways of counting print differently.

Judgement call: the log handler's copy of the formatter is changed too,
since the issue asks that every command print a byte count the same way.

Model: opus-5-5
2026-10-13 16:22:03 +02:00
clawbot 09a0c01c43 Stop a stalled blob read when restore or deep verify is interrupted (closes #325)
check / check (push) Waiting to run
Restore and deep verify looked for a cancel only between blobs, and the
file backend's reader ignores the context. A blob read stalled on a
slow USB stick or network share outlasted the 30-second shutdown
timeout, so the process exited before removing the decrypted snapshot
database and blob cache from the temp directory.

FetchBlob, through which both read every blob, now closes the storage
reader as soon as the context is cancelled, ending a blocked read on
any backend; Read then returns the context's error, not the closed
reader's.

Limitation: a read stuck in the kernel that closing cannot interrupt,
such as one on an unresponsive hard NFS mount, still cannot be stopped.

Model: opus-5-5
2026-10-13 15:48:00 +02:00
clawbot 35eed79b1a Check --keep-newer-than before the backup starts (closes #307)
check / check (push) Successful in 13m18s
snapshot create read --keep-newer-than only in the post-backup prune,
so a value the duration parser rejects failed the run after the
snapshot was uploaded: a nightly cron line added a snapshot each night,
reported it as failed, and never deleted any. Without --prune the flag
was ignored. The command now checks the flag before it starts and fails
on an invalid duration or a missing --prune.

A failed post-backup prune is now wrapped in ErrPostBackupPruneFailed
and reported as "Backup completed, but post-backup prune failed: ..."
rather than as a failed snapshot creation. The exit status stays 1.

Model: opus-5-5
2026-10-13 09:44:44 +02:00
clawbot 94412d9ef3 Stop remote info counting a non-blob file under blobs/ (closes #295)
check / check (push) Canceled after 0s
remote info counted every key under blobs/ with four or more path parts
as a blob, while prune skips a name that is not a blob hash. A stray
file such as .DS_Store was reported as an orphaned blob, with advice to
run prune, which then deleted nothing. remote info now takes its blob
list from prune's listAllRemoteBlobs, so its blob and orphan figures,
in the table and in --json, cover exactly what prune would consider.

blobKeyParts moves to prune.go, its only remaining user.

Model: opus-5-5
2026-10-13 08:33:27 +02:00
clawbot 167c0a0746 Make snapshot remove fail for an unknown snapshot ID (closes #293)
check / check (push) Successful in 12m4s
snapshot remove never checked that the ID existed: it deleted nothing,
reported the snapshot removed from the local database and from the
destination store, and exited 0, under --json, --dry-run and
--local-only too. It now looks the ID up in the local index and, only
when that has no row and --local-only is not set, on the destination
store; finding neither is an error. The command reports a removal from
the local database only when the local index had the row, and
remote_removed is true only when metadata was actually deleted.

Judgement call: under --local-only the destination store is not
consulted, so the snapshot must be in the local index.

Model: opus-5-5
2026-10-13 05:55:22 +02:00
clawbot 07c957f84b Bind the local index to the s3.* destination too (closes #298)
check / check (push) Successful in 15m35s
The local index was bound to its destination only through storage_url.
With the s3.* form nothing was checked, so after s3.bucket or s3.prefix
changed the next backup treated every chunk as already stored, uploaded
nothing, and completed a snapshot that could not be restored.

Without storage_url, the binding now records the destination the s3.*
fields name, written as an s3:// URL from s3.endpoint, s3.bucket and
s3.prefix, and a change is refused with the same error and advice. The
advice line now names the s3.* fields as well as storage_url.

Judgement call: the fields are compared as written, so an equivalent
spelling (a trailing slash on the prefix, a scheme on the endpoint) is
also refused, as it already is for storage_url.

Model: opus-5-5
2026-10-13 03:48:36 +02:00
clawbot c9a519c233 Reject a retention duration longer than about 292 years (closes #289)
check / check (push) Failing after 3m36s
parseDuration multiplied each number by the length of its unit with no
limit, so from 293y (or 15251w, 3559mo, 106752d, or a sum such as
200y200y) the result wrapped to a negative duration. The purge cutoff
then lay in the future and every snapshot of the name was selected, for
--older-than and --keep-newer-than alike. Each part is now checked
before it is added, and a total past the largest time.Duration is
rejected as an invalid duration, so nothing is deleted. Go units were
already safe: time.ParseDuration rejects an overflow.

With --prune the backup still runs before the duration is read; only
the prune step fails.

Model: opus-5-5
2026-10-13 01:51:24 +02:00
clawbot 34f61fb408 Make remote nuke delete leftover .partial uploads (closes #281)
check / check (push) Canceled after 0s
The file and rclone listings skip an object whose name ends in
`.partial`, the temporary name a `file://` or rclone upload writes
before moving the object into place. `remote nuke` deletes only what
the listings return, so it left the `.partial` objects killed uploads
leave behind and still reported the destination store empty.

Storer gains DeletePartialUploads. The file and rclone backends remove
every `.partial` object under the prefix; S3 has none to remove, since
it shows an object only once its upload completes. `remote nuke` calls
it for `metadata/` and `blobs/` as its last step.

Empty directories under a `file://` destination are still left behind.

Model: opus-5-5
2026-10-08 14:17:20 +02:00
clawbot 05bf73c48a Never skip a local index error for a directory or symlink (closes #284)
check / check (push) Canceled after 0s
Under --skip-errors, an error recording a directory or symlink in the
local index was skipped like an unreadable file. The snapshot completed
without the entry, and a restore did not recreate it.

processFileWithErrorHandling now records a directory or symlink itself
and returns any error, which stops the backup with or without the flag.
The skip sees only a regular file's errors, so countFailedFile no longer
checks for a directory.

The test injects the error with a SQLite trigger that refuses the
directory's files row.

Model: opus-5-5
2026-10-08 12:59:26 +02:00
clawbot 79a73fa122 Count a file a backup could not store as failed (closes #280)
check / check (push) Canceled after 0s
A file that phase 1 of a backup counted and phase 2 could not open,
because it was unreadable under --skip-errors or removed in between,
was added to the unchanged count while its size stayed in
BytesScanned. The summary showed it as unchanged with its bytes backed
up, and the snapshots row's file_count and total_size included it.

The scanner now counts such a file in FilesFailed and takes its size
out of BytesScanned. The summary's files line adds "N failed", and
file_count leaves the file out.

A directory phase 2 cannot record is not counted as failed, since
phase 1 counts no directories; that case has no test.

Model: opus-5-5
2026-10-08 10:12:10 +02:00
clawbot 0d0368df81 Report unknown blob figures for a snapshot whose manifest cannot be read (closes #272)
check / check (push) Canceled after 0s
When remote info could not read a snapshot's manifest, the orphan
figures were unknown but the snapshot's row still gave 0 blobs and 0 B,
in the table and in --json. The row's blob count and blob size are now
unknown, and null in --json.

A directory with no manifest, as an interrupted backup leaves, still
shows 0: the orphan figures count its blobs as orphaned, so it
references none.

The constant holding the "unknown" text is renamed from countUnknown to
unknownText, since it now also stands for a size.

Model: opus-5-5
2026-10-08 08:46:11 +02:00
clawbot 70f008a21a Document that --json skips the confirmation prompt (closes #268)
check / check (push) Canceled after 0s
snapshot remove and prune delete without asking under --json, because a
prompt on stdout would break the JSON document. Their --json help and
README entries named only --force as skipping the prompt. Both now say
--json skips it too. One test checks the --json help of both commands;
another runs prune with --json, no --force and empty stdin, and checks
that the unreferenced blob is deleted and stdout holds only the JSON
document.

Model: opus-5-5
2026-10-08 04:59:30 +02:00
clawbot e161343eac Speed up the vaultik and database package tests (closes #235)
check / check (push) Canceled after 0s
In the Dockerfile test phase, internal/vaultik spent 23.6s of 31.9s in
36 tests run one at a time. 24 of them were serial only because they
call log.Initialize; they now call it before t.Parallel(), so the
logger is replaced before any parallel test runs. The 12 still serial
change the umask, TMPDIR, os.Stderr or time.Local.

TestLargeDatasets took 8.4s of the 9.8s internal/database run by
committing each of its 1,500 inserts on its own; it now makes them in
one transaction. TestDedupOnlySnapshotRestores gives its second backup
its own snapshot name instead of sleeping 1.1s for a new snapshot ID.

Model: opus-5-5
2026-10-07 17:12:07 +02:00
clawbot d53202eb86 Fix two misleading messages (closes #240)
check / check (push) Canceled after 0s
The warning for a config file that others can read always said the
file contained S3 credentials, so a file:// config with none got a
false claim. When s3.access_key_id or s3.secret_access_key is set it
now says the file may contain them, because Load sees the values only
after smartconfig has replaced any ${...} reference, so a set
credential need not be in the file. Otherwise it says the file is
readable by others.

snapshot purge wrapped the listing error, which already starts with
"listing remote snapshots:", in that prefix a second time.
syncWithRemote now returns it unwrapped, as CleanupLocalSnapshots
does.

Model: opus-5-5
2026-10-07 15:29:07 +02:00
clawbot 3fc8a8f2f4 Read the snapshot name using the stored hostname (closes #230)
check / check (push) Canceled after 0s
A snapshot ID is hostname_name_timestamp, and purge took the name to be
everything between the first and the last underscore. With a hostname
such as my_host the name home came out as host_home, so
`snapshot purge --keep-latest --snapshot home` found nothing to delete
and `snapshot create --prune` purged nothing without a message. The name
is now read by removing the hostname stored with the snapshot, in the
short form the ID uses, so both may contain underscores. This was chosen
over rejecting underscores in `hostname` when the config loads, which
would also stop restores on such a host.

The purge consistency test stored a hostname that did not match its
snapshot IDs; it now matches, as it always does in production.

Model: opus-5-5
2026-10-07 12:12:08 +02:00
clawbot 7696f83258 Leave remote info orphan figures unknown when a manifest is unreadable (closes #228)
check / check (push) Canceled after 0s
When a manifest could not be read, remote info skipped it, counted that
snapshot's blobs as orphaned and advised running prune. The orphan
figures are now unknown in that case, with no prune advice; --json gives
them as null and lists the unreadable remote keys in
unreadable_manifests. Only a listed manifest.json.zst is read, so a
directory without one, as an interrupted backup leaves, keeps the
figures known.

Names under metadata/ were used unchecked and printed raw. A name that
is not a remote key is now skipped with a warning. A manifest under it
is not read either, so it also leaves the figures unknown; --json counts
such manifests in skipped_manifest_count.

Model: opus-5-5
2026-10-07 10:59:26 +02:00
clawbot b57ce2277d Store and compare file mtimes to the nanosecond (closes #226)
check / check (push) Canceled after 0s
The files table held mtime in whole seconds and the scanner compared
whole seconds. A file rewritten with its size unchanged and a new mtime
in the same second as the indexed one was treated as unchanged, and
every later snapshot restored the old content. A new mtime_nsec column
now holds the nanoseconds within the second that mtime holds, and the
scanner compares the full mtime.

A local index created before this change lacks the column and is
rebuilt with `vaultik database delete` and a full backup. A snapshot
made before it cannot be restored by this version.

Model: opus-5-5
2026-10-07 07:12:12 +02:00
clawbot cdc60c4dfa Write only the document to stdout from snapshot remove --json (closes #251)
check / check (push) Canceled after 0s
When the destination store cannot be reached, `snapshot remove` still
removes the snapshot from the local index and warns. The warning went
through the UI, which writes to stdout, so under `--json` it landed
ahead of the document and `| jq` failed on a command that exited 0.
Under `--json` the UI warning is now skipped; the logger's warning on
stderr carries the follow-up, with the snapshot ID as a field.

The follow-up, in the warning, the README and the command's help, said
`vaultik prune` would finish the cleanup, but `prune` never removes
snapshot metadata. They now say to run `vaultik snapshot remove` for the
snapshot again once the destination store is reachable.

Model: opus-5-5
2026-10-07 04:46:13 +02:00
clawbot 49eed7a3e5 Count each file, byte and upload once in backup statistics (closes #225)
check / check (push) Canceled after 0s
The scanner added a changed file's bytes again for each new chunk and
counted a file as unchanged for each chunk already stored. It now
counts files and bytes once, in the scan phase, and counts its own
uploads, so a --cron run, which has no progress reporter, records
them. The blob count no longer adds earlier paths' blobs again.

The snapshots row now stores the size of all files in total_size and
the referenced blobs' sizes in blob_size, blob_uncompressed_size and
compression_ratio, as docs/DATAMODEL.md says. Those sizes come from
one query, and a failed query fails the snapshot. DATAMODEL.md now
says chunk_count and blob_count count what the run added.

Removed UpdateSnapshotStats and GetCountBySnapshot, which nothing
calls any more.

Model: opus-5-5
2026-10-07 02:59:27 +02:00
clawbot 85d4ef118d Keep command output to the README's stdout and stderr rules (closes #224)
check / check (push) Successful in 16m49s
The startup banner moves from stdout to stderr, so a `completion`
script, a `config get` value and the hidden `__complete` command print
only their own output. `--quiet`, `--cron` and `--json` still suppress
it.

A failing `remote info`, `prune` or `snapshot remove` under `--json`
now reports its error on stderr. Their reporters returned early under
`--json`, so the failure reached neither stream.

`snapshot verify --quiet` writes no report. A failure is still
returned and printed on stderr, with the same exit status.

Judgement call: the banner's stream, posted on the issue for the owner.

Model: opus-5-5
2026-10-07 01:12:15 +02:00
clawbot b06f992152 Start the progress reporter once per snapshot, not once per path (closes #253)
check / check (push) Successful in 19m54s
A backup without --cron of a snapshot with two or more paths panicked
with "close of closed channel". Scan runs once per path, and it started
the progress reporter and deferred its Stop each time; Stop closes the
reporter's signal channel, so the second path's Stop panicked.
scanAllDirectories now starts the reporter before the first path and
stops it after the last, and Scan no longer starts or stops it. A new
test backs up a two-path snapshot with the reporter on and restores
both paths.

Model: opus-5-5
2026-10-07 00:12:10 +02:00
clawbot 5d685f03ce List the files under a path by path, not by string prefix (closes #223)
check / check (push) Successful in 12m16s
FileRepository.ListByPrefix matched with SQL LIKE: a plain string
prefix that ignores ASCII case and treats _ and % as wildcards.
Restoring /home/u/doc also restored doc2, DOC and doc.txt.bak, and a
backup counted the files of a longer sibling path as deleted. It is
now ListUnderPath, which returns the file at the path and every file
whose path starts with the path plus a slash, compared exactly. A
trailing slash is ignored, so "/" still lists every file.

Three tests relied on string-prefix matching and now name a full path
or call ListAll.

ListIDsWithChunksNotInUploadedBlobs keeps its LIKE: it only adds file
IDs the scan never looks up.

Model: opus-5-5
2026-10-06 22:12:16 +02:00
clawbot f59086c0e5 Return an error, not a panic, on a malformed snapshot database (closes #231)
check / check (push) Successful in 13m39s
Restore cut chunk hashes from the snapshot database to 16 characters
for its error messages, so a shorter hash panicked. Those messages now
use shortHash. Under --verify, a file_chunks row with no chunks row was
dereferenced, and the chunk size from the database was allocated in
one piece, so a negative or huge size panicked. A missing row is now an
error, a negative size is rejected, and each chunk is hashed by
streaming it from the restored file.

A restored file shorter than its chunks now fails verify as a short
read instead of an unexpected EOF.

Model: opus-5-5
2026-10-06 21:16:28 +02:00
clawbot 5aa5ba5544 Apply restored owners, modes and times in an order that keeps them (closes #219)
check / check (push) Successful in 11m53s
A directory got its stored mode and mtime before its contents were
written, so a read-only directory came back without its files and a
non-empty one carried the time of the restore. Directories are now
created owner-only (0700) and get their stored owner, mode and mtime
after the restore loop, each before its parent, skipping any whose
place a symlink has since taken. A file's mode is now applied after its
chown, which on Linux clears setuid and setgid. A symlink gets its
stored owner (as root) and mtime on the link itself, through
golang.org/x/sys/unix, now a direct dependency.

An interrupted restore leaves its directories at 0700.

Model: opus-5-5
2026-10-06 18:29:09 +02:00
clawbot 14fc4c9893 Load a config that has no age recipient (closes #221)
check / check (push) Successful in 13m47s
The README's steps for restoring on another machine failed at the first
command: `config init` wrote a placeholder recipient, and `config.Load`
rejects any recipient that does not parse. `config init` now writes an
empty `age_recipients` list, `config.Load` accepts an empty list, and
`snapshot create` refuses to start without a recipient. A malformed
recipient is still rejected at load.

The recovery-host test now builds its config with `config init` and
`config set` and reads it through `config.Load`, so it imports
`internal/cli`. On a fresh file, `config set age_recipients.0` writes the
list in flow style (`[age1...]`).

Model: opus-5-5
2026-10-06 15:46:13 +02:00
clawbot 66c80a70e3 Skip files of an unreadable blob under restore --skip-errors (closes #218)
check / check (push) Successful in 12m34s
A blob that failed to download ended `snapshot restore` even with
--skip-errors, after restoring whichever files came first. The download
error now goes through the same per-file handling as any other restore
error, once for every pending file that references the blob. With
--skip-errors those files are reported as failed, the rest are
restored, and the command still exits non-zero. Without the flag the
restore still aborts; the error now also names one affected file and
suggests --skip-errors. A cancelled restore still ends at once. The
--skip-errors help text and README now limit the packing and storage
caveat to snapshot creation.

Model: opus-5-5
2026-10-06 14:46:05 +02:00
clawbot ea72697992 List a missing file:// destination directory as an error (closes #220)
check / check (push) Successful in 5m53s
check / check (pull_request) Successful in 4m53s
The file backend listed a destination directory that does not exist as
an empty store. With the volume unplugged, snapshot list reported every
local snapshot as missing from the store, snapshot remove said it had
removed metadata it never reached, and prune dropped every local
snapshot record. List and ListStream now fail when the destination
directory is missing, so those commands take their existing path for a
store that cannot be listed. A missing prefix under an existing
directory is still an empty listing, and a first backup still creates
the directory.

Three tests listed a file:// destination nothing had created; they now
create it.

Model: opus-5-5
2026-10-06 08:46:17 +02:00
clawbot 713be502bd Reject a duration with characters outside its parts (closes #215)
check / check (push) Successful in 7m28s
check / check (pull_request) Successful in 5m39s
parseDuration fell back to an unanchored search for number-and-unit
pieces when time.ParseDuration failed, and skipped everything in
between. 1.0y became 0, so `snapshot create --prune --keep-newer-than
1.0y` deleted every snapshot of the backed-up names, the new one
included. 2.1w became one week and 1,5y five years. The fallback now
requires the whole input to be whole-number-and-unit parts with nothing
between them. A bare number is rejected before time.ParseDuration sees
it, since Go reads 0 and +0 as zero with no unit.

Judgement call: a space between number and unit (`30 days`) was
accepted and is now an error, matching Go's own units.

Model: opus-5-5
2026-10-06 06:12:07 +02:00
clawbot 35cf985c18 Re-chunk a known file whose chunks no uploaded blob holds (closes #214)
check / check (push) Successful in 6m53s
check / check (pull_request) Successful in 6m17s
File rows are shared by every snapshot and updated in place, while a
blob row is deleted once no snapshot references it. Removing the newest
snapshot, or the prune after an interrupted run, could drop the only
blob holding a changed file's current chunks while an older snapshot
kept the file row. The next backup compared metadata only, skipped the
file, and completed a snapshot that could not restore it.

The scanner now loads the IDs of known files that list a chunk no
uploaded blob holds and re-chunks them even when their metadata is
unchanged.

The tests append to a file, so the file keeps its first chunk in a blob
the first snapshot still references. Each backup run gets its own
snapshot name, so the second-precision snapshot IDs differ without
sleeping.

Model: opus-5-5
2026-10-06 04:46:16 +02:00
clawbot b30e79ee45 Run the disk-full restore test again (closes #207)
check / check (push) Successful in 4m52s
check / check (pull_request) Successful in 5m19s
TestRestoreReportsDiskFull was skipped pending
#163, which is closed. The skip
and its "skipped until" wording are removed.

Restore is unchanged. With the skip removed the test failed because
restore succeeded: its simulated full disk capped only Create, but
restore now opens each file with OpenFile, so nothing was capped. It now
caps OpenFile, and only for files under the restore target: restore also
writes the decrypted metadata database under $TMPDIR through the same
filesystem, and capping that would fail the restore before any file
reached the target.

Judgement call: the test was corrected, not restore; both assertions are
unchanged.

Model: opus-5-5
2026-10-01 20:24:30 +02:00
clawbot eed117fe25 Route direct-stdout command output through internal/ui (closes #149)
check / check (pull_request) Successful in 1m27s
check / check (push) Successful in 3m18s
version, info, remote info, config, and database delete wrote plain text straight to stdout, so they were unstyled and ignored --quiet. Output is now governed by internal/ui in two buckets. Status lines and confirmations (config init, config set, the database delete prompt) go through the ui message methods and are silenced by --quiet. The data a command exists to produce is written plain -- the version/info/remote-info reports, the snapshot list table, config get values, and the --json documents -- and is never suppressed, since a script depends on it and a marker would corrupt a table or document. The database delete confirmation prompt is always shown. Pure-cli commands reach ui through a small commandUI helper.

Model: opus-4-8
2026-09-22 20:28:50 +02:00