Fix the review findings in the three design documents
check / check (push) In progress

ARCHITECTURE.md: the chunk size limits, the commit date, the cases in
which a backup skips its prune, and two sentences that read badly.

docs/DATAMODEL.md: which repository type serves which table, what the
metadata export deletes and keeps, the blobs table in a restore, and
when a chunk row is deleted.

docs/REPOSTRUCTURE.md: how an object's size relates to its contents,
what blob sizes can leak and when, and a sentence moved out of the
prune rules.

All three now say that prune drops a local snapshot whose manifest is
gone, not whose directory is gone.

Model: fable-5-1
This commit is contained in:
2026-11-06 19:34:28 +00:00
parent a52a8a781c
commit 252db3259a
3 changed files with 63 additions and 33 deletions
+28 -15
View File
@@ -327,7 +327,12 @@ holds the file's new chunks is stored.
The metadata of a snapshot is made from a copy of the index:
- The index is copied into a temporary directory with `VACUUM INTO`.
- Every row that does not belong to this snapshot is deleted from the copy.
- The rows of every other snapshot are deleted from the copy. So are the files
and blobs that this snapshot does not use. A chunk is then deleted when no
remaining file and no remaining blob uses it.
- The `local_meta` and `schema_migrations` tables are kept as they are. The
uploaded database therefore names the destination, and lists the migrations
that have run.
- `VACUUM` rewrites the copy, so that no deleted row is left in the file.
- The copy is compressed with zstd and encrypted with age.
- The manifest is built from the copy's `snapshot_blobs` table.
@@ -347,6 +352,7 @@ A restore does not use the local index.
- It reads the files to restore from `files`.
- It finds each file's chunks in `file_chunks`.
- It finds the blob and offset of each chunk in `blob_chunks`.
- It finds the name of each blob in `blobs`.
It does not read the manifest.
@@ -354,7 +360,7 @@ It does not read the manifest.
`prune` changes the local index in two ways:
- It deletes each snapshot whose directory is gone from the destination.
- It deletes each snapshot whose manifest is gone from the destination.
- It deletes unfinished snapshots, and the rows that no snapshot uses.
The second step removes unused rows in this order:
@@ -362,7 +368,7 @@ The second step removes unused rows in this order:
- files that are in no snapshot
- blobs that are in no snapshot
- `blob_chunks` rows whose chunk is gone
- chunks that no file uses
- chunks that no file and no blob uses
`prune` then deletes unused data at the destination. That step reads the
manifests, not the local index. It reads the local index only to keep a
@@ -370,19 +376,26 @@ directory under `metadata/` whose snapshot is still recorded there.
## Code
Each table has a repository type in `internal/database`:
Each repository type in `internal/database` serves one or more tables:
- `FileRepository`
- `ChunkRepository`
- `FileChunkRepository`
- `ChunkFileRepository`
- `BlobRepository`
- `BlobChunkRepository`
- `SnapshotRepository`
- `UploadRepository`
- `LocalMetaRepository`
- `FileRepository`: `files`
- `ChunkRepository`: `chunks`
- `FileChunkRepository`: `file_chunks`
- `ChunkFileRepository`: `chunk_files`
- `BlobRepository`: `blobs`
- `BlobChunkRepository`: `blob_chunks`
- `SnapshotRepository`:
- `snapshots`
- `snapshot_files`
- `snapshot_blobs`
- `snapshot_mount_points`
- `UploadRepository`: `uploads`
- `LocalMetaRepository`: `local_meta`
`Repositories` holds one of each.
No repository type serves `schema_migrations`. The code that applies the
migrations reads and writes it.
`Repositories` holds one of each type.
`Repositories.WithTx` runs a function in one transaction:
@@ -401,5 +414,5 @@ rolled back.
`snapshot list`, can read the index while a backup writes to it.
- A connection waits up to 10 seconds for a lock before it fails.
- Foreign keys are enforced.
- A process opens one connection, so all its access to the index is in turn.
- Each open database uses one connection, so its queries run one at a time.
- Queries pass their values as parameters.
+13 -6
View File
@@ -253,17 +253,23 @@ therefore does not stop overwrites.
### Sizes are visible
age adds no padding. The size of an object is the exact size of its compressed
contents.
An object's size follows exactly from the size of its compressed contents.
All the new chunks in one blob share one zstd stream. The stream looks back 8
MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at
`blob_size_limit` and at the end of each configured path. A chunk that is
already stored is never packed again.
Someone who can write into a backed-up file and watch blob sizes can learn
something in one case. Their data and a secret must land in the same chunk, in
a file that keeps changing.
Data that an outsider writes compresses smaller when it matches a secret in the
same chunk, so the blob gets smaller. The blob's size therefore shows the
outsider whether their data matched the secret.
This works only when all of these hold:
- The outsider can write into a backed-up file.
- The outsider can see the blob sizes at the destination.
- The outsider's data and the secret land in the same chunk.
- The file keeps changing.
Back up a directory that outsiders can write to as a snapshot of its own.
@@ -363,10 +369,11 @@ objects. An rclone remote that cannot move objects can keep a partial object.
- a blob that no manifest lists
- a directory under `metadata/` that has no manifest
It reads the manifests, which are not encrypted, so it needs no private key.
These rules keep it from deleting data that is in use:
- It keeps every blob that any manifest lists.
- It reads the manifests, which are not encrypted, so it needs no private key.
- It deletes nothing when any manifest cannot be downloaded or decoded.
- It deletes a blob only once the blob is older than `prune_grace_period`.
- It deletes a directory only once every object in it is that old.