Fix the review findings in the three design documents
check / check (push) Successful in 13m23s

ARCHITECTURE.md: the chunk size limits, the commit date, the cases in
which a backup skips its prune, and two sentences that read badly.

docs/DATAMODEL.md: which repository type serves which table, what the
metadata export deletes and keeps, the blobs table in a restore, and
when a chunk row is deleted.

docs/REPOSTRUCTURE.md: how an object's size relates to its contents,
what blob sizes can leak and when, and a sentence moved out of the
prune rules.

All three now say that prune drops a local snapshot whose manifest is
gone, not whose directory is gone.

Model: fable-5-1
This commit is contained in:
2026-11-06 19:34:28 +00:00
parent a52a8a781c
commit 252db3259a
3 changed files with 63 additions and 33 deletions
+22 -12
View File
@@ -58,8 +58,10 @@ source files
- `internal/blob`: the `Packer`, which collects chunks into blobs.
- `internal/blobgen`: the writer that hashes, compresses and encrypts a blob,
and the reader that reverses it.
- `internal/database`: the local index. It holds the schema and one repository
type for each table.
- `internal/database`: the local index. It holds the schema and the repository
types that read and write the tables.
[docs/DATAMODEL.md](docs/DATAMODEL.md#code) says which type serves which
table.
- `internal/storage`: the `Storer` interface and its three implementations, for
S3, a local directory and rclone.
- `internal/s3`: the S3 client that the S3 `Storer` uses.
@@ -68,7 +70,7 @@ source files
- `internal/log`: the logger. It writes to stderr.
- `internal/pidlock`: the lock that keeps two data-changing commands from
running at once.
- `internal/globals`: the version, commit and build date, set when the binary
- `internal/globals`: the version, commit and commit date, set when the binary
is built.
- `internal/types`: distinct types for IDs, hashes and paths.
@@ -127,6 +129,14 @@ build the application.
- `createNamedSnapshot` runs once for each snapshot name.
- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.
`CreateSnapshot` calls `skipPostBackupPrune` instead of `runPostBackupPrune` in
these cases:
- `--skip-errors` skipped a file or directory that could not be read.
- A configured path held nothing.
- An earlier snapshot of the same name found a mount point that this backup did
not find.
`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
left in the index would make the backup skip files that were never fully
stored.
@@ -186,17 +196,17 @@ A file's record matches when all of these are unchanged:
- the link target
One case overrides the file check. A file is read again when it lists a chunk
that no uploaded blob holds. This happens after the only snapshot that used
such a blob is removed. `loadFilesToRechunk` finds these files.
that no uploaded blob holds. This happens when the only snapshot that used the
blob holding that chunk is removed. `loadFilesToRechunk` finds these files.
### Chunks
`chunker.Chunker` cuts a file with the FastCDC algorithm. The cut points depend
on the content, so an insertion early in a file moves only nearby chunks.
`chunk_size` sets the average chunk size. A chunk is between a quarter of that
size and four times that size. With the default of 10 MB, a chunk is between 2.5
MB and 40 MB.
`chunk_size` sets the average chunk size. No chunk is larger than four times
this size. Only a file's last chunk can be smaller than a quarter of it. With
the default of 10 MB, those limits are 40 MB and 2.5 MB.
`ChunkReaderStreaming` passes each chunk to a callback as it is found. The whole
file is never held in memory.
@@ -324,8 +334,8 @@ time after the whole restore.
`Vaultik.Prune` does three steps:
- `CleanupLocalSnapshots` drops each snapshot from the local index whose
directory is gone from the destination.
- `CleanupLocalSnapshots` drops from the local index each snapshot whose
manifest is gone from the destination.
- `PruneDatabase` removes unfinished snapshots and unused records from the
local index.
- `PruneBlobs` deletes unused data at the destination.
@@ -375,8 +385,8 @@ Every command that changes data calls `EnsureStorageBinding` first.
## Concurrency
- The local index is opened with one connection, so all access to it is in
turn.
- The local index is opened with one connection, so its queries run one at a
time.
- The index runs in SQLite's WAL mode. A second process can therefore read it
while a backup writes to it.
- `Packer` guards its state with a mutex.