ARCHITECTURE.md: the chunk size limits, the commit date, the cases in which a backup skips its prune, and two sentences that read badly. docs/DATAMODEL.md: which repository type serves which table, what the metadata export deletes and keeps, the blobs table in a restore, and when a chunk row is deleted. docs/REPOSTRUCTURE.md: how an object's size relates to its contents, what blob sizes can leak and when, and a sentence moved out of the prune rules. All three now say that prune drops a local snapshot whose manifest is gone, not whose directory is gone. Model: fable-5-1
This commit is contained in:
+22
-12
@@ -58,8 +58,10 @@ source files
|
||||
- `internal/blob`: the `Packer`, which collects chunks into blobs.
|
||||
- `internal/blobgen`: the writer that hashes, compresses and encrypts a blob,
|
||||
and the reader that reverses it.
|
||||
- `internal/database`: the local index. It holds the schema and one repository
|
||||
type for each table.
|
||||
- `internal/database`: the local index. It holds the schema and the repository
|
||||
types that read and write the tables.
|
||||
[docs/DATAMODEL.md](docs/DATAMODEL.md#code) says which type serves which
|
||||
table.
|
||||
- `internal/storage`: the `Storer` interface and its three implementations, for
|
||||
S3, a local directory and rclone.
|
||||
- `internal/s3`: the S3 client that the S3 `Storer` uses.
|
||||
@@ -68,7 +70,7 @@ source files
|
||||
- `internal/log`: the logger. It writes to stderr.
|
||||
- `internal/pidlock`: the lock that keeps two data-changing commands from
|
||||
running at once.
|
||||
- `internal/globals`: the version, commit and build date, set when the binary
|
||||
- `internal/globals`: the version, commit and commit date, set when the binary
|
||||
is built.
|
||||
- `internal/types`: distinct types for IDs, hashes and paths.
|
||||
|
||||
@@ -127,6 +129,14 @@ build the application.
|
||||
- `createNamedSnapshot` runs once for each snapshot name.
|
||||
- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.
|
||||
|
||||
`CreateSnapshot` calls `skipPostBackupPrune` instead of `runPostBackupPrune` in
|
||||
these cases:
|
||||
|
||||
- `--skip-errors` skipped a file or directory that could not be read.
|
||||
- A configured path held nothing.
|
||||
- An earlier snapshot of the same name found a mount point that this backup did
|
||||
not find.
|
||||
|
||||
`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
|
||||
left in the index would make the backup skip files that were never fully
|
||||
stored.
|
||||
@@ -186,17 +196,17 @@ A file's record matches when all of these are unchanged:
|
||||
- the link target
|
||||
|
||||
One case overrides the file check. A file is read again when it lists a chunk
|
||||
that no uploaded blob holds. This happens after the only snapshot that used
|
||||
such a blob is removed. `loadFilesToRechunk` finds these files.
|
||||
that no uploaded blob holds. This happens when the only snapshot that used the
|
||||
blob holding that chunk is removed. `loadFilesToRechunk` finds these files.
|
||||
|
||||
### Chunks
|
||||
|
||||
`chunker.Chunker` cuts a file with the FastCDC algorithm. The cut points depend
|
||||
on the content, so an insertion early in a file moves only nearby chunks.
|
||||
|
||||
`chunk_size` sets the average chunk size. A chunk is between a quarter of that
|
||||
size and four times that size. With the default of 10 MB, a chunk is between 2.5
|
||||
MB and 40 MB.
|
||||
`chunk_size` sets the average chunk size. No chunk is larger than four times
|
||||
this size. Only a file's last chunk can be smaller than a quarter of it. With
|
||||
the default of 10 MB, those limits are 40 MB and 2.5 MB.
|
||||
|
||||
`ChunkReaderStreaming` passes each chunk to a callback as it is found. The whole
|
||||
file is never held in memory.
|
||||
@@ -324,8 +334,8 @@ time after the whole restore.
|
||||
|
||||
`Vaultik.Prune` does three steps:
|
||||
|
||||
- `CleanupLocalSnapshots` drops each snapshot from the local index whose
|
||||
directory is gone from the destination.
|
||||
- `CleanupLocalSnapshots` drops from the local index each snapshot whose
|
||||
manifest is gone from the destination.
|
||||
- `PruneDatabase` removes unfinished snapshots and unused records from the
|
||||
local index.
|
||||
- `PruneBlobs` deletes unused data at the destination.
|
||||
@@ -375,8 +385,8 @@ Every command that changes data calls `EnsureStorageBinding` first.
|
||||
|
||||
## Concurrency
|
||||
|
||||
- The local index is opened with one connection, so all access to it is in
|
||||
turn.
|
||||
- The local index is opened with one connection, so its queries run one at a
|
||||
time.
|
||||
- The index runs in SQLite's WAL mode. A second process can therefore read it
|
||||
while a backup writes to it.
|
||||
- `Packer` guards its state with a mutex.
|
||||
|
||||
+28
-15
@@ -327,7 +327,12 @@ holds the file's new chunks is stored.
|
||||
The metadata of a snapshot is made from a copy of the index:
|
||||
|
||||
- The index is copied into a temporary directory with `VACUUM INTO`.
|
||||
- Every row that does not belong to this snapshot is deleted from the copy.
|
||||
- The rows of every other snapshot are deleted from the copy. So are the files
|
||||
and blobs that this snapshot does not use. A chunk is then deleted when no
|
||||
remaining file and no remaining blob uses it.
|
||||
- The `local_meta` and `schema_migrations` tables are kept as they are. The
|
||||
uploaded database therefore names the destination, and lists the migrations
|
||||
that have run.
|
||||
- `VACUUM` rewrites the copy, so that no deleted row is left in the file.
|
||||
- The copy is compressed with zstd and encrypted with age.
|
||||
- The manifest is built from the copy's `snapshot_blobs` table.
|
||||
@@ -347,6 +352,7 @@ A restore does not use the local index.
|
||||
- It reads the files to restore from `files`.
|
||||
- It finds each file's chunks in `file_chunks`.
|
||||
- It finds the blob and offset of each chunk in `blob_chunks`.
|
||||
- It finds the name of each blob in `blobs`.
|
||||
|
||||
It does not read the manifest.
|
||||
|
||||
@@ -354,7 +360,7 @@ It does not read the manifest.
|
||||
|
||||
`prune` changes the local index in two ways:
|
||||
|
||||
- It deletes each snapshot whose directory is gone from the destination.
|
||||
- It deletes each snapshot whose manifest is gone from the destination.
|
||||
- It deletes unfinished snapshots, and the rows that no snapshot uses.
|
||||
|
||||
The second step removes unused rows in this order:
|
||||
@@ -362,7 +368,7 @@ The second step removes unused rows in this order:
|
||||
- files that are in no snapshot
|
||||
- blobs that are in no snapshot
|
||||
- `blob_chunks` rows whose chunk is gone
|
||||
- chunks that no file uses
|
||||
- chunks that no file and no blob uses
|
||||
|
||||
`prune` then deletes unused data at the destination. That step reads the
|
||||
manifests, not the local index. It reads the local index only to keep a
|
||||
@@ -370,19 +376,26 @@ directory under `metadata/` whose snapshot is still recorded there.
|
||||
|
||||
## Code
|
||||
|
||||
Each table has a repository type in `internal/database`:
|
||||
Each repository type in `internal/database` serves one or more tables:
|
||||
|
||||
- `FileRepository`
|
||||
- `ChunkRepository`
|
||||
- `FileChunkRepository`
|
||||
- `ChunkFileRepository`
|
||||
- `BlobRepository`
|
||||
- `BlobChunkRepository`
|
||||
- `SnapshotRepository`
|
||||
- `UploadRepository`
|
||||
- `LocalMetaRepository`
|
||||
- `FileRepository`: `files`
|
||||
- `ChunkRepository`: `chunks`
|
||||
- `FileChunkRepository`: `file_chunks`
|
||||
- `ChunkFileRepository`: `chunk_files`
|
||||
- `BlobRepository`: `blobs`
|
||||
- `BlobChunkRepository`: `blob_chunks`
|
||||
- `SnapshotRepository`:
|
||||
- `snapshots`
|
||||
- `snapshot_files`
|
||||
- `snapshot_blobs`
|
||||
- `snapshot_mount_points`
|
||||
- `UploadRepository`: `uploads`
|
||||
- `LocalMetaRepository`: `local_meta`
|
||||
|
||||
`Repositories` holds one of each.
|
||||
No repository type serves `schema_migrations`. The code that applies the
|
||||
migrations reads and writes it.
|
||||
|
||||
`Repositories` holds one of each type.
|
||||
|
||||
`Repositories.WithTx` runs a function in one transaction:
|
||||
|
||||
@@ -401,5 +414,5 @@ rolled back.
|
||||
`snapshot list`, can read the index while a backup writes to it.
|
||||
- A connection waits up to 10 seconds for a lock before it fails.
|
||||
- Foreign keys are enforced.
|
||||
- A process opens one connection, so all its access to the index is in turn.
|
||||
- Each open database uses one connection, so its queries run one at a time.
|
||||
- Queries pass their values as parameters.
|
||||
|
||||
+13
-6
@@ -253,17 +253,23 @@ therefore does not stop overwrites.
|
||||
|
||||
### Sizes are visible
|
||||
|
||||
age adds no padding. The size of an object is the exact size of its compressed
|
||||
contents.
|
||||
An object's size follows exactly from the size of its compressed contents.
|
||||
|
||||
All the new chunks in one blob share one zstd stream. The stream looks back 8
|
||||
MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at
|
||||
`blob_size_limit` and at the end of each configured path. A chunk that is
|
||||
already stored is never packed again.
|
||||
|
||||
Someone who can write into a backed-up file and watch blob sizes can learn
|
||||
something in one case. Their data and a secret must land in the same chunk, in
|
||||
a file that keeps changing.
|
||||
Data that an outsider writes compresses smaller when it matches a secret in the
|
||||
same chunk, so the blob gets smaller. The blob's size therefore shows the
|
||||
outsider whether their data matched the secret.
|
||||
|
||||
This works only when all of these hold:
|
||||
|
||||
- The outsider can write into a backed-up file.
|
||||
- The outsider can see the blob sizes at the destination.
|
||||
- The outsider's data and the secret land in the same chunk.
|
||||
- The file keeps changing.
|
||||
|
||||
Back up a directory that outsiders can write to as a snapshot of its own.
|
||||
|
||||
@@ -363,10 +369,11 @@ objects. An rclone remote that cannot move objects can keep a partial object.
|
||||
- a blob that no manifest lists
|
||||
- a directory under `metadata/` that has no manifest
|
||||
|
||||
It reads the manifests, which are not encrypted, so it needs no private key.
|
||||
|
||||
These rules keep it from deleting data that is in use:
|
||||
|
||||
- It keeps every blob that any manifest lists.
|
||||
- It reads the manifests, which are not encrypted, so it needs no private key.
|
||||
- It deletes nothing when any manifest cannot be downloaded or decoded.
|
||||
- It deletes a blob only once the blob is older than `prune_grace_period`.
|
||||
- It deletes a directory only once every object in it is that old.
|
||||
|
||||
Reference in New Issue
Block a user