Fix the review findings in the three design documents
check / check (push) Waiting to run

ARCHITECTURE.md: the chunk size limits, the commit date, the cases in
which a backup skips its prune, and two sentences that read badly.

docs/DATAMODEL.md: which repository type serves which table, what the
metadata export deletes and keeps, the blobs table in a restore, and
when a chunk row is deleted.

docs/REPOSTRUCTURE.md: how an object's size relates to its contents,
what blob sizes can leak and when, and a sentence moved out of the
prune rules.

All three now say that prune drops a local snapshot whose manifest is
gone, not whose directory is gone.

Model: fable-5-1
This commit is contained in:
2026-11-06 19:34:28 +00:00
parent a52a8a781c
commit 252db3259a
3 changed files with 63 additions and 33 deletions
+22 -12
View File
@@ -58,8 +58,10 @@ source files
- `internal/blob`: the `Packer`, which collects chunks into blobs.
- `internal/blobgen`: the writer that hashes, compresses and encrypts a blob,
and the reader that reverses it.
- `internal/database`: the local index. It holds the schema and one repository
type for each table.
- `internal/database`: the local index. It holds the schema and the repository
types that read and write the tables.
[docs/DATAMODEL.md](docs/DATAMODEL.md#code) says which type serves which
table.
- `internal/storage`: the `Storer` interface and its three implementations, for
S3, a local directory and rclone.
- `internal/s3`: the S3 client that the S3 `Storer` uses.
@@ -68,7 +70,7 @@ source files
- `internal/log`: the logger. It writes to stderr.
- `internal/pidlock`: the lock that keeps two data-changing commands from
running at once.
- `internal/globals`: the version, commit and build date, set when the binary
- `internal/globals`: the version, commit and commit date, set when the binary
is built.
- `internal/types`: distinct types for IDs, hashes and paths.
@@ -127,6 +129,14 @@ build the application.
- `createNamedSnapshot` runs once for each snapshot name.
- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.
`CreateSnapshot` calls `skipPostBackupPrune` instead of `runPostBackupPrune` in
these cases:
- `--skip-errors` skipped a file or directory that could not be read.
- A configured path held nothing.
- An earlier snapshot of the same name found a mount point that this backup did
not find.
`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
left in the index would make the backup skip files that were never fully
stored.
@@ -186,17 +196,17 @@ A file's record matches when all of these are unchanged:
- the link target
One case overrides the file check. A file is read again when it lists a chunk
that no uploaded blob holds. This happens after the only snapshot that used
such a blob is removed. `loadFilesToRechunk` finds these files.
that no uploaded blob holds. This happens when the only snapshot that used the
blob holding that chunk is removed. `loadFilesToRechunk` finds these files.
### Chunks
`chunker.Chunker` cuts a file with the FastCDC algorithm. The cut points depend
on the content, so an insertion early in a file moves only nearby chunks.
`chunk_size` sets the average chunk size. A chunk is between a quarter of that
size and four times that size. With the default of 10 MB, a chunk is between 2.5
MB and 40 MB.
`chunk_size` sets the average chunk size. No chunk is larger than four times
this size. Only a file's last chunk can be smaller than a quarter of it. With
the default of 10 MB, those limits are 40 MB and 2.5 MB.
`ChunkReaderStreaming` passes each chunk to a callback as it is found. The whole
file is never held in memory.
@@ -324,8 +334,8 @@ time after the whole restore.
`Vaultik.Prune` does three steps:
- `CleanupLocalSnapshots` drops each snapshot from the local index whose
directory is gone from the destination.
- `CleanupLocalSnapshots` drops from the local index each snapshot whose
manifest is gone from the destination.
- `PruneDatabase` removes unfinished snapshots and unused records from the
local index.
- `PruneBlobs` deletes unused data at the destination.
@@ -375,8 +385,8 @@ Every command that changes data calls `EnsureStorageBinding` first.
## Concurrency
- The local index is opened with one connection, so all access to it is in
turn.
- The local index is opened with one connection, so its queries run one at a
time.
- The index runs in SQLite's WAL mode. A second process can therefore read it
while a backup writes to it.
- `Packer` guards its state with a mutex.
+28 -15
View File
@@ -327,7 +327,12 @@ holds the file's new chunks is stored.
The metadata of a snapshot is made from a copy of the index:
- The index is copied into a temporary directory with `VACUUM INTO`.
- Every row that does not belong to this snapshot is deleted from the copy.
- The rows of every other snapshot are deleted from the copy. So are the files
and blobs that this snapshot does not use. A chunk is then deleted when no
remaining file and no remaining blob uses it.
- The `local_meta` and `schema_migrations` tables are kept as they are. The
uploaded database therefore names the destination, and lists the migrations
that have run.
- `VACUUM` rewrites the copy, so that no deleted row is left in the file.
- The copy is compressed with zstd and encrypted with age.
- The manifest is built from the copy's `snapshot_blobs` table.
@@ -347,6 +352,7 @@ A restore does not use the local index.
- It reads the files to restore from `files`.
- It finds each file's chunks in `file_chunks`.
- It finds the blob and offset of each chunk in `blob_chunks`.
- It finds the name of each blob in `blobs`.
It does not read the manifest.
@@ -354,7 +360,7 @@ It does not read the manifest.
`prune` changes the local index in two ways:
- It deletes each snapshot whose directory is gone from the destination.
- It deletes each snapshot whose manifest is gone from the destination.
- It deletes unfinished snapshots, and the rows that no snapshot uses.
The second step removes unused rows in this order:
@@ -362,7 +368,7 @@ The second step removes unused rows in this order:
- files that are in no snapshot
- blobs that are in no snapshot
- `blob_chunks` rows whose chunk is gone
- chunks that no file uses
- chunks that no file and no blob uses
`prune` then deletes unused data at the destination. That step reads the
manifests, not the local index. It reads the local index only to keep a
@@ -370,19 +376,26 @@ directory under `metadata/` whose snapshot is still recorded there.
## Code
Each table has a repository type in `internal/database`:
Each repository type in `internal/database` serves one or more tables:
- `FileRepository`
- `ChunkRepository`
- `FileChunkRepository`
- `ChunkFileRepository`
- `BlobRepository`
- `BlobChunkRepository`
- `SnapshotRepository`
- `UploadRepository`
- `LocalMetaRepository`
- `FileRepository`: `files`
- `ChunkRepository`: `chunks`
- `FileChunkRepository`: `file_chunks`
- `ChunkFileRepository`: `chunk_files`
- `BlobRepository`: `blobs`
- `BlobChunkRepository`: `blob_chunks`
- `SnapshotRepository`:
- `snapshots`
- `snapshot_files`
- `snapshot_blobs`
- `snapshot_mount_points`
- `UploadRepository`: `uploads`
- `LocalMetaRepository`: `local_meta`
`Repositories` holds one of each.
No repository type serves `schema_migrations`. The code that applies the
migrations reads and writes it.
`Repositories` holds one of each type.
`Repositories.WithTx` runs a function in one transaction:
@@ -401,5 +414,5 @@ rolled back.
`snapshot list`, can read the index while a backup writes to it.
- A connection waits up to 10 seconds for a lock before it fails.
- Foreign keys are enforced.
- A process opens one connection, so all its access to the index is in turn.
- Each open database uses one connection, so its queries run one at a time.
- Queries pass their values as parameters.
+13 -6
View File
@@ -253,17 +253,23 @@ therefore does not stop overwrites.
### Sizes are visible
age adds no padding. The size of an object is the exact size of its compressed
contents.
An object's size follows exactly from the size of its compressed contents.
All the new chunks in one blob share one zstd stream. The stream looks back 8
MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at
`blob_size_limit` and at the end of each configured path. A chunk that is
already stored is never packed again.
Someone who can write into a backed-up file and watch blob sizes can learn
something in one case. Their data and a secret must land in the same chunk, in
a file that keeps changing.
Data that an outsider writes compresses smaller when it matches a secret in the
same chunk, so the blob gets smaller. The blob's size therefore shows the
outsider whether their data matched the secret.
This works only when all of these hold:
- The outsider can write into a backed-up file.
- The outsider can see the blob sizes at the destination.
- The outsider's data and the secret land in the same chunk.
- The file keeps changing.
Back up a directory that outsiders can write to as a snapshot of its own.
@@ -363,10 +369,11 @@ objects. An rclone remote that cannot move objects can keep a partial object.
- a blob that no manifest lists
- a directory under `metadata/` that has no manifest
It reads the manifests, which are not encrypted, so it needs no private key.
These rules keep it from deleting data that is in use:
- It keeps every blob that any manifest lists.
- It reads the manifests, which are not encrypted, so it needs no private key.
- It deletes nothing when any manifest cannot be downloaded or decoded.
- It deletes a blob only once the blob is older than `prune_grace_period`.
- It deletes a directory only once every object in it is that old.