Fault-injection tests: prove backups survive interruption and corruption #72

New Issue

clawbot · 2026-08-09T03:43:30+02:00

clawbot commented

2026-08-09 03:43:30 +02:00

vaultik is a backup tool, so the failure modes that matter are the ones
where the process dies partway through or the destination lies. None of
those are currently tested. README:411-424 already names this gap; this
issue makes it actionable.

The existing suite covers the happy path well (internal/vaultik
3,114 test LOC, internal/database 4,203) but every test assumes writes
succeed and data comes back intact.

Scenarios to cover

Network failure mid-blob. Upload fails partway through a multi-part
blob. Assert: no partial blob is recorded as complete in the local
index; a retry produces a consistent state; no dangling reference
survives.
kill -9 between manifest write and db.zst.age upload. This is
the dangerous window — the destination has blobs and a manifest but no
index. Assert the next run detects and repairs the state, and that
snapshot list and verify both report it honestly rather than
crashing or silently ignoring it.
Corrupt blob on read. Flip bytes in a stored blob; assert restore
fails loudly with an actionable error naming the blob, and never
writes corrupt data to the restore target.
Truncated blob on read. Same, for short reads.
Disk full during restore. Assert a clear error and no
partially-written file left in place presenting as complete.
Destination returns success but stores nothing. A lying backend.
verify --deep must catch it.

Definition of done

Every scenario above has a test. Where a scenario cannot be simulated
without excessive machinery, say so explicitly on this issue with the
reason rather than silently skipping it.
Failures are injected through a seam, not by patching production
code paths at test time — a fault-injecting Storer wrapper is the
natural fit given the internal/storage interface. That wrapper
should be reusable for future tests.
Each test asserts on the observable end state (what is in the
index, what is at the destination, what the user is told), not merely
that an error was returned.
No production behavior changes. If a test exposes a real bug — which
is the likely outcome for scenarios 1 and 2 — file it as its own issue
linked here, and mark the test t.Skip with the issue number rather
than leaving a red suite or weakening the assertion.
These tests coordinate with #69: if integration build tags are
adopted, they belong in the tagged set.
make check green.

Out of scope

At-scale restore verification (multi-GB, thousands of files) is real
pre-1.0 work but belongs in its own issue — it is a performance and
endurance concern, not a correctness-under-failure one.

vaultik is a backup tool, so the failure modes that matter are the ones where the process dies partway through or the destination lies. None of those are currently tested. README:411-424 already names this gap; this issue makes it actionable. The existing suite covers the happy path well (`internal/vaultik` 3,114 test LOC, `internal/database` 4,203) but every test assumes writes succeed and data comes back intact. ## Scenarios to cover 1. **Network failure mid-blob.** Upload fails partway through a multi-part blob. Assert: no partial blob is recorded as complete in the local index; a retry produces a consistent state; no dangling reference survives. 2. **`kill -9` between manifest write and `db.zst.age` upload.** This is the dangerous window — the destination has blobs and a manifest but no index. Assert the next run detects and repairs the state, and that `snapshot list` and `verify` both report it honestly rather than crashing or silently ignoring it. 3. **Corrupt blob on read.** Flip bytes in a stored blob; assert restore fails loudly with an actionable error naming the blob, and never writes corrupt data to the restore target. 4. **Truncated blob on read.** Same, for short reads. 5. **Disk full during restore.** Assert a clear error and no partially-written file left in place presenting as complete. 6. **Destination returns success but stores nothing.** A lying backend. `verify --deep` must catch it. ## Definition of done 1. Every scenario above has a test. Where a scenario cannot be simulated without excessive machinery, say so explicitly on this issue with the reason rather than silently skipping it. 2. Failures are injected through a **seam**, not by patching production code paths at test time — a fault-injecting `Storer` wrapper is the natural fit given the `internal/storage` interface. That wrapper should be reusable for future tests. 3. Each test asserts on the **observable end state** (what is in the index, what is at the destination, what the user is told), not merely that an error was returned. 4. No production behavior changes. If a test exposes a real bug — which is the likely outcome for scenarios 1 and 2 — file it as its own issue linked here, and mark the test `t.Skip` with the issue number rather than leaving a red suite or weakening the assertion. 5. These tests coordinate with #69: if integration build tags are adopted, they belong in the tagged set. 6. `make check` green. ## Out of scope At-scale restore verification (multi-GB, thousands of files) is real pre-1.0 work but belongs in its own issue — it is a performance and endurance concern, not a correctness-under-failure one.

clawbot added this to the 1.0.0 milestone 2026-08-09 03:43:30 +02:00

clawbot referenced this issue

2026-08-09 10:13:00 +02:00

Decide whether feature/daemon-mode is 1.0 scope #94

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sneak/vaultik#72