Fix the review findings in the three design documents
check / check (push) Successful in 13m23s

ARCHITECTURE.md: the chunk size limits, the commit date, the cases in
which a backup skips its prune, and two sentences that read badly.

docs/DATAMODEL.md: which repository type serves which table, what the
metadata export deletes and keeps, the blobs table in a restore, and
when a chunk row is deleted.

docs/REPOSTRUCTURE.md: how an object's size relates to its contents,
what blob sizes can leak and when, and a sentence moved out of the
prune rules.

All three now say that prune drops a local snapshot whose manifest is
gone, not whose directory is gone.

Model: fable-5-1
This commit is contained in:
sneak committed 2026-11-06 19:34:28 +00:00
1 parent a52a8a781c
commit 252db3259a
3 files changed
+63 -33

No files matched your search

+22 -12
View File
@@ -58,8 +58,10 @@ source files
- `internal/blob`: the `Packer`, which collects chunks into blobs. - `internal/blob`: the `Packer`, which collects chunks into blobs.
- `internal/blobgen`: the writer that hashes, compresses and encrypts a blob, - `internal/blobgen`: the writer that hashes, compresses and encrypts a blob,
and the reader that reverses it. and the reader that reverses it.
- `internal/database`: the local index. It holds the schema and one repository - `internal/database`: the local index. It holds the schema and the repository
type for each table. types that read and write the tables.
[docs/DATAMODEL.md](docs/DATAMODEL.md#code) says which type serves which
table.
- `internal/storage`: the `Storer` interface and its three implementations, for - `internal/storage`: the `Storer` interface and its three implementations, for
S3, a local directory and rclone. S3, a local directory and rclone.
- `internal/s3`: the S3 client that the S3 `Storer` uses. - `internal/s3`: the S3 client that the S3 `Storer` uses.
@@ -68,7 +70,7 @@ source files
- `internal/log`: the logger. It writes to stderr. - `internal/log`: the logger. It writes to stderr.
- `internal/pidlock`: the lock that keeps two data-changing commands from - `internal/pidlock`: the lock that keeps two data-changing commands from
running at once. running at once.
- `internal/globals`: the version, commit and build date, set when the binary - `internal/globals`: the version, commit and commit date, set when the binary
is built. is built.
- `internal/types`: distinct types for IDs, hashes and paths. - `internal/types`: distinct types for IDs, hashes and paths.
@@ -127,6 +129,14 @@ build the application.
- `createNamedSnapshot` runs once for each snapshot name. - `createNamedSnapshot` runs once for each snapshot name.
- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data. - With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.
`CreateSnapshot` calls `skipPostBackupPrune` instead of `runPostBackupPrune` in
these cases:
- `--skip-errors` skipped a file or directory that could not be read.
- A configured path held nothing.
- An earlier snapshot of the same name found a mount point that this backup did
not find.
`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot `PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
left in the index would make the backup skip files that were never fully left in the index would make the backup skip files that were never fully
stored. stored.
@@ -186,17 +196,17 @@ A file's record matches when all of these are unchanged:
- the link target - the link target
One case overrides the file check. A file is read again when it lists a chunk One case overrides the file check. A file is read again when it lists a chunk
that no uploaded blob holds. This happens after the only snapshot that used that no uploaded blob holds. This happens when the only snapshot that used the
such a blob is removed. `loadFilesToRechunk` finds these files. blob holding that chunk is removed. `loadFilesToRechunk` finds these files.
### Chunks ### Chunks
`chunker.Chunker` cuts a file with the FastCDC algorithm. The cut points depend `chunker.Chunker` cuts a file with the FastCDC algorithm. The cut points depend
on the content, so an insertion early in a file moves only nearby chunks. on the content, so an insertion early in a file moves only nearby chunks.
`chunk_size` sets the average chunk size. A chunk is between a quarter of that `chunk_size` sets the average chunk size. No chunk is larger than four times
size and four times that size. With the default of 10 MB, a chunk is between 2.5 this size. Only a file's last chunk can be smaller than a quarter of it. With
MB and 40 MB. the default of 10 MB, those limits are 40 MB and 2.5 MB.
`ChunkReaderStreaming` passes each chunk to a callback as it is found. The whole `ChunkReaderStreaming` passes each chunk to a callback as it is found. The whole
file is never held in memory. file is never held in memory.
@@ -324,8 +334,8 @@ time after the whole restore.
`Vaultik.Prune` does three steps: `Vaultik.Prune` does three steps:
- `CleanupLocalSnapshots` drops each snapshot from the local index whose - `CleanupLocalSnapshots` drops from the local index each snapshot whose
directory is gone from the destination. manifest is gone from the destination.
- `PruneDatabase` removes unfinished snapshots and unused records from the - `PruneDatabase` removes unfinished snapshots and unused records from the
local index. local index.
- `PruneBlobs` deletes unused data at the destination. - `PruneBlobs` deletes unused data at the destination.
@@ -375,8 +385,8 @@ Every command that changes data calls `EnsureStorageBinding` first.
## Concurrency ## Concurrency
- The local index is opened with one connection, so all access to it is in - The local index is opened with one connection, so its queries run one at a
turn. time.
- The index runs in SQLite's WAL mode. A second process can therefore read it - The index runs in SQLite's WAL mode. A second process can therefore read it
while a backup writes to it. while a backup writes to it.
- `Packer` guards its state with a mutex. - `Packer` guards its state with a mutex.
+28 -15
View File
@@ -327,7 +327,12 @@ holds the file's new chunks is stored.
The metadata of a snapshot is made from a copy of the index: The metadata of a snapshot is made from a copy of the index:
- The index is copied into a temporary directory with `VACUUM INTO`. - The index is copied into a temporary directory with `VACUUM INTO`.
- Every row that does not belong to this snapshot is deleted from the copy. - The rows of every other snapshot are deleted from the copy. So are the files
and blobs that this snapshot does not use. A chunk is then deleted when no
remaining file and no remaining blob uses it.
- The `local_meta` and `schema_migrations` tables are kept as they are. The
uploaded database therefore names the destination, and lists the migrations
that have run.
- `VACUUM` rewrites the copy, so that no deleted row is left in the file. - `VACUUM` rewrites the copy, so that no deleted row is left in the file.
- The copy is compressed with zstd and encrypted with age. - The copy is compressed with zstd and encrypted with age.
- The manifest is built from the copy's `snapshot_blobs` table. - The manifest is built from the copy's `snapshot_blobs` table.
@@ -347,6 +352,7 @@ A restore does not use the local index.
- It reads the files to restore from `files`. - It reads the files to restore from `files`.
- It finds each file's chunks in `file_chunks`. - It finds each file's chunks in `file_chunks`.
- It finds the blob and offset of each chunk in `blob_chunks`. - It finds the blob and offset of each chunk in `blob_chunks`.
- It finds the name of each blob in `blobs`.
It does not read the manifest. It does not read the manifest.
@@ -354,7 +360,7 @@ It does not read the manifest.
`prune` changes the local index in two ways: `prune` changes the local index in two ways:
- It deletes each snapshot whose directory is gone from the destination. - It deletes each snapshot whose manifest is gone from the destination.
- It deletes unfinished snapshots, and the rows that no snapshot uses. - It deletes unfinished snapshots, and the rows that no snapshot uses.
The second step removes unused rows in this order: The second step removes unused rows in this order:
@@ -362,7 +368,7 @@ The second step removes unused rows in this order:
- files that are in no snapshot - files that are in no snapshot
- blobs that are in no snapshot - blobs that are in no snapshot
- `blob_chunks` rows whose chunk is gone - `blob_chunks` rows whose chunk is gone
- chunks that no file uses - chunks that no file and no blob uses
`prune` then deletes unused data at the destination. That step reads the `prune` then deletes unused data at the destination. That step reads the
manifests, not the local index. It reads the local index only to keep a manifests, not the local index. It reads the local index only to keep a
@@ -370,19 +376,26 @@ directory under `metadata/` whose snapshot is still recorded there.
## Code ## Code
Each table has a repository type in `internal/database`: Each repository type in `internal/database` serves one or more tables:
- `FileRepository` - `FileRepository`: `files`
- `ChunkRepository` - `ChunkRepository`: `chunks`
- `FileChunkRepository` - `FileChunkRepository`: `file_chunks`
- `ChunkFileRepository` - `ChunkFileRepository`: `chunk_files`
- `BlobRepository` - `BlobRepository`: `blobs`
- `BlobChunkRepository` - `BlobChunkRepository`: `blob_chunks`
- `SnapshotRepository` - `SnapshotRepository`:
- `UploadRepository` - `snapshots`
- `LocalMetaRepository` - `snapshot_files`
- `snapshot_blobs`
- `snapshot_mount_points`
- `UploadRepository`: `uploads`
- `LocalMetaRepository`: `local_meta`
`Repositories` holds one of each. No repository type serves `schema_migrations`. The code that applies the
migrations reads and writes it.
`Repositories` holds one of each type.
`Repositories.WithTx` runs a function in one transaction: `Repositories.WithTx` runs a function in one transaction:
@@ -401,5 +414,5 @@ rolled back.
`snapshot list`, can read the index while a backup writes to it. `snapshot list`, can read the index while a backup writes to it.
- A connection waits up to 10 seconds for a lock before it fails. - A connection waits up to 10 seconds for a lock before it fails.
- Foreign keys are enforced. - Foreign keys are enforced.
- A process opens one connection, so all its access to the index is in turn. - Each open database uses one connection, so its queries run one at a time.
- Queries pass their values as parameters. - Queries pass their values as parameters.
+13 -6
View File
@@ -253,17 +253,23 @@ therefore does not stop overwrites.
### Sizes are visible ### Sizes are visible
age adds no padding. The size of an object is the exact size of its compressed An object's size follows exactly from the size of its compressed contents.
contents.
All the new chunks in one blob share one zstd stream. The stream looks back 8 All the new chunks in one blob share one zstd stream. The stream looks back 8
MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at
`blob_size_limit` and at the end of each configured path. A chunk that is `blob_size_limit` and at the end of each configured path. A chunk that is
already stored is never packed again. already stored is never packed again.
Someone who can write into a backed-up file and watch blob sizes can learn Data that an outsider writes compresses smaller when it matches a secret in the
something in one case. Their data and a secret must land in the same chunk, in same chunk, so the blob gets smaller. The blob's size therefore shows the
a file that keeps changing. outsider whether their data matched the secret.
This works only when all of these hold:
- The outsider can write into a backed-up file.
- The outsider can see the blob sizes at the destination.
- The outsider's data and the secret land in the same chunk.
- The file keeps changing.
Back up a directory that outsiders can write to as a snapshot of its own. Back up a directory that outsiders can write to as a snapshot of its own.
@@ -363,10 +369,11 @@ objects. An rclone remote that cannot move objects can keep a partial object.
- a blob that no manifest lists - a blob that no manifest lists
- a directory under `metadata/` that has no manifest - a directory under `metadata/` that has no manifest
It reads the manifests, which are not encrypted, so it needs no private key.
These rules keep it from deleting data that is in use: These rules keep it from deleting data that is in use:
- It keeps every blob that any manifest lists. - It keeps every blob that any manifest lists.
- It reads the manifests, which are not encrypted, so it needs no private key.
- It deletes nothing when any manifest cannot be downloaded or decoded. - It deletes nothing when any manifest cannot be downloaded or decoded.
- It deletes a blob only once the blob is older than `prune_grace_period`. - It deletes a blob only once the blob is older than `prune_grace_period`.
- It deletes a directory only once every object in it is that old. - It deletes a directory only once every object in it is that old.