Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c2ce69d258 |
+2
-2
@@ -84,11 +84,11 @@ The final storage unit uploaded to S3. Contains many compressed and encrypted ch
|
|||||||
Blob creation process:
|
Blob creation process:
|
||||||
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
|
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
|
||||||
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
|
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
|
||||||
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file in `$TMPDIR` (`/tmp` when unset)
|
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file
|
||||||
4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes
|
4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes
|
||||||
5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted
|
5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted
|
||||||
|
|
||||||
The metadata export at the end of a backup also uses `$TMPDIR`: it copies the local index there and runs `VACUUM` on the copy, which writes another temporary copy and a write-ahead log. A backup therefore needs free space in `$TMPDIR` of the larger of `blob_size_limit` and about three times the size of the local index.
|
A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp`.
|
||||||
|
|
||||||
#### BlobChunk (`database.BlobChunk`)
|
#### BlobChunk (`database.BlobChunk`)
|
||||||
Maps chunks to their position within blobs:
|
Maps chunks to their position within blobs:
|
||||||
|
|||||||
@@ -546,7 +546,7 @@ complete annotated example also lives in
|
|||||||
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
|
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
|
||||||
| `exclude` | | Global exclude patterns (applied to all snapshots) |
|
| `exclude` | | Global exclude patterns (applied to all snapshots) |
|
||||||
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
|
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
|
||||||
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. Each blob is written in full to a temporary file in `$TMPDIR` (`/tmp` when unset) before it is uploaded, and the metadata export at the end of a backup works on a copy of the local index there. A backup needs free space there of the larger of this limit and about three times the size of the local index |
|
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
|
||||||
| `compression_level` | `3` | zstd compression level (1-19) |
|
| `compression_level` | `3` | zstd compression level (1-19) |
|
||||||
| `hostname` | system hostname | Hostname used in snapshot IDs |
|
| `hostname` | system hostname | Hostname used in snapshot IDs |
|
||||||
| `index_path` | platform data dir | Local SQLite index path |
|
| `index_path` | platform data dir | Local SQLite index path |
|
||||||
|
|||||||
@@ -25,12 +25,14 @@ the tag exists and is exercised; what is left is merging `next` to
|
|||||||
- 2026-10-07: Corrected documentation, help text and comments that were
|
- 2026-10-07: Corrected documentation, help text and comments that were
|
||||||
false about the code
|
false about the code
|
||||||
([issue #233](https://git.eeqj.de/sneak/vaultik/issues/233)). A blob
|
([issue #233](https://git.eeqj.de/sneak/vaultik/issues/233)). A blob
|
||||||
is not streamed to storage; it is written in full to a temporary file
|
is not streamed to storage. The README, `ARCHITECTURE.md` and
|
||||||
in `$TMPDIR` and uploaded once finished, and the metadata export works
|
`config.example.yml` now say a backup needs free temporary space,
|
||||||
on a copy of the local index there. The README and
|
because each blob is written whole to a temporary file before it is
|
||||||
`config.example.yml` now say a backup needs free space there of the
|
uploaded (up to about `blob_size_limit`; an rclone destination that
|
||||||
larger of `blob_size_limit` and about three times the size of the
|
cannot stream uploads needs about twice that) and the metadata export
|
||||||
local index. Also corrected: the snapshot ID format, what restore
|
writes copies of the local index. Temporary files go to `$TMPDIR`
|
||||||
|
(default `/tmp`); with `TMPDIR` unset, SQLite writes one of those
|
||||||
|
copies to `/var/tmp`. Also corrected: the snapshot ID format, what restore
|
||||||
reads and how incomplete snapshots are removed in `docs/DATAMODEL.md`,
|
reads and how incomplete snapshots are removed in `docs/DATAMODEL.md`,
|
||||||
what `source_path` holds, the `index_path` and config file defaults,
|
what `source_path` holds, the `index_path` and config file defaults,
|
||||||
what `snapshot remove` cleans up, how the release gets its Go
|
what `snapshot remove` cleans up, how the release gets its Go
|
||||||
|
|||||||
+6
-5
@@ -313,11 +313,12 @@ storage_url: "rclone://myremote/path/to/backups"
|
|||||||
# Chunking uses no secret (the FastCDC parameters are fixed and public). At a
|
# Chunking uses no secret (the FastCDC parameters are fixed and public). At a
|
||||||
# large limit a blob holds hundreds of chunks, so individual chunk lengths are
|
# large limit a blob holds hundreds of chunks, so individual chunk lengths are
|
||||||
# not visible in its size; lowering the limit toward chunk_size exposes them.
|
# not visible in its size; lowering the limit toward chunk_size exposes them.
|
||||||
# Each blob is written in full to a temporary file in $TMPDIR (/tmp when
|
# A backup needs free temporary space, because each blob is written whole to
|
||||||
# unset) before it is uploaded, and the metadata export at the end of a
|
# a temporary file before it is uploaded (up to about blob_size_limit; an
|
||||||
# backup works on a copy of the local index there. A backup needs free space
|
# rclone destination that cannot stream uploads needs about twice that) and
|
||||||
# there of the larger of this limit and about three times the size of the
|
# the metadata export writes copies of the local index. Temporary files go to
|
||||||
# local index.
|
# $TMPDIR (default /tmp); with TMPDIR unset, SQLite writes one of those copies
|
||||||
|
# to /var/tmp.
|
||||||
# Supports: 1GB, 10G, 500MB, 1GiB, etc.
|
# Supports: 1GB, 10G, 500MB, 1GiB, etc.
|
||||||
# Default: 10GB
|
# Default: 10GB
|
||||||
#blob_size_limit: 10GB
|
#blob_size_limit: 10GB
|
||||||
|
|||||||
Reference in New Issue
Block a user