Compare commits
2
Commits
c2ce69d258
...
27109bfd89
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
27109bfd89 | ||
|
|
e161343eac |
+7
-3
@@ -54,10 +54,12 @@ The database tracks five primary entities and their relationships:
|
|||||||
|
|
||||||
#### File (`database.File`)
|
#### File (`database.File`)
|
||||||
Represents a file, directory, or symlink in the backup system. Stores metadata needed for restoration:
|
Represents a file, directory, or symlink in the backup system. Stores metadata needed for restoration:
|
||||||
- Path, source_path (for restore path stripping), mtime
|
- Path, mtime
|
||||||
- Size, mode, ownership (uid, gid)
|
- Size, mode, ownership (uid, gid)
|
||||||
- Symlink target (if applicable)
|
- Symlink target (if applicable)
|
||||||
|
|
||||||
|
It also stores `source_path`, the source directory the scan found it under, made absolute and with symlinks resolved. Restore does not read it.
|
||||||
|
|
||||||
#### Chunk (`database.Chunk`)
|
#### Chunk (`database.Chunk`)
|
||||||
A content-addressed unit of data. Files are split into variable-size chunks using the FastCDC algorithm:
|
A content-addressed unit of data. Files are split into variable-size chunks using the FastCDC algorithm:
|
||||||
- `ChunkHash`: SHA256 hash of chunk content (primary key)
|
- `ChunkHash`: SHA256 hash of chunk content (primary key)
|
||||||
@@ -82,9 +84,11 @@ The final storage unit uploaded to S3. Contains many compressed and encrypted ch
|
|||||||
Blob creation process:
|
Blob creation process:
|
||||||
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
|
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
|
||||||
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
|
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
|
||||||
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and streamed to storage
|
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file
|
||||||
4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes
|
4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes
|
||||||
5. Uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}`
|
5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted
|
||||||
|
|
||||||
|
A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp`.
|
||||||
|
|
||||||
#### BlobChunk (`database.BlobChunk`)
|
#### BlobChunk (`database.BlobChunk`)
|
||||||
Maps chunks to their position within blobs:
|
Maps chunks to their position within blobs:
|
||||||
|
|||||||
@@ -546,7 +546,7 @@ complete annotated example also lives in
|
|||||||
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
|
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
|
||||||
| `exclude` | | Global exclude patterns (applied to all snapshots) |
|
| `exclude` | | Global exclude patterns (applied to all snapshots) |
|
||||||
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
|
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
|
||||||
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit |
|
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
|
||||||
| `compression_level` | `3` | zstd compression level (1-19) |
|
| `compression_level` | `3` | zstd compression level (1-19) |
|
||||||
| `hostname` | system hostname | Hostname used in snapshot IDs |
|
| `hostname` | system hostname | Hostname used in snapshot IDs |
|
||||||
| `index_path` | platform data dir | Local SQLite index path |
|
| `index_path` | platform data dir | Local SQLite index path |
|
||||||
@@ -805,7 +805,8 @@ them. We provide:
|
|||||||
in the `Dockerfile` together.
|
in the `Dockerfile` together.
|
||||||
* `script/release` — cross-compile and publish the release artifacts
|
* `script/release` — cross-compile and publish the release artifacts
|
||||||
with the pinned `goreleaser`. Refuses a `goreleaser` on `PATH` whose
|
with the pinned `goreleaser`. Refuses a `goreleaser` on `PATH` whose
|
||||||
version is not the pinned one, on the same reasoning as `script/lint`.
|
version is not the pinned one, because a different version would build
|
||||||
|
a different release from the same tag.
|
||||||
* `script/release-snapshot` — the same build with no publishing and no
|
* `script/release-snapshot` — the same build with no publishing and no
|
||||||
tagging, into `./dist`
|
tagging, into `./dist`
|
||||||
* `script/test` — run the test suite by building the `test` phase of
|
* `script/test` — run the test suite by building the `test` phase of
|
||||||
@@ -915,14 +916,14 @@ It is passed to `goreleaser` as `GITEA_TOKEN`. The runner's automatic
|
|||||||
token is deliberately not used: it is not guaranteed to carry release
|
token is deliberately not used: it is not guaranteed to carry release
|
||||||
write access.
|
write access.
|
||||||
|
|
||||||
The Go toolchain that compiles the released binaries comes from an
|
The Go toolchain that compiles the released binaries is installed by
|
||||||
`actions/setup-go` step pinned by commit sha, reading its version from
|
`script/install-go`, which downloads the version named by `go.mod`
|
||||||
`go.mod` (currently `1.26.1`, the same version the `Dockerfile` builder
|
(currently `1.26.1`, the same version the `Dockerfile` builder stage
|
||||||
stage pins by digest). `goreleaser` shells out to `go` for every
|
pins by digest) and refuses the archive unless its sha256 matches the
|
||||||
|
value committed in the script. `goreleaser` shells out to `go` for every
|
||||||
cross-compile, so without that step the release would either fail
|
cross-compile, so without that step the release would either fail
|
||||||
outright or ship binaries built by whatever unpinned toolchain the
|
outright or ship binaries built by whatever unpinned toolchain the
|
||||||
runner happened to carry — the one unpinned thing in an otherwise
|
runner happened to carry.
|
||||||
hash-pinned release path.
|
|
||||||
|
|
||||||
To rehearse the whole build without publishing or tagging anything:
|
To rehearse the whole build without publishing or tagging anything:
|
||||||
|
|
||||||
|
|||||||
@@ -22,6 +22,32 @@ the tag exists and is exercised; what is left is merging `next` to
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-10-07: Corrected documentation, help text and comments that were
|
||||||
|
false about the code
|
||||||
|
([issue #233](https://git.eeqj.de/sneak/vaultik/issues/233)). A blob
|
||||||
|
is not streamed to storage. The README, `ARCHITECTURE.md` and
|
||||||
|
`config.example.yml` now say a backup needs free temporary space,
|
||||||
|
because each blob is written whole to a temporary file before it is
|
||||||
|
uploaded (up to about `blob_size_limit`; an rclone destination that
|
||||||
|
cannot stream uploads needs about twice that) and the metadata export
|
||||||
|
writes copies of the local index. Temporary files go to `$TMPDIR`
|
||||||
|
(default `/tmp`); with `TMPDIR` unset, SQLite writes one of those
|
||||||
|
copies to `/var/tmp`. Also corrected: the snapshot ID format, what restore
|
||||||
|
reads and how incomplete snapshots are removed in `docs/DATAMODEL.md`,
|
||||||
|
what `source_path` holds, the `index_path` and config file defaults,
|
||||||
|
what `snapshot remove` cleans up, how the release gets its Go
|
||||||
|
toolchain, and the `script/release` and `script/fmt-check` comments.
|
||||||
|
|
||||||
|
- 2026-10-07: Cut the time the `internal/vaultik` and `internal/database`
|
||||||
|
tests take ([issue #235](https://git.eeqj.de/sneak/vaultik/issues/235)).
|
||||||
|
Most of the `internal/vaultik` time went to 24 tests that ran one at a
|
||||||
|
time only because they call `log.Initialize`; they now call it before
|
||||||
|
`t.Parallel()`, as the package's other tests do. `TestLargeDatasets`
|
||||||
|
committed each of its 1,500 inserts on its own and now makes them in
|
||||||
|
one transaction. `TestDedupOnlySnapshotRestores` gives its second
|
||||||
|
backup its own snapshot name instead of sleeping past the one-second
|
||||||
|
timestamp in the snapshot ID.
|
||||||
|
|
||||||
- 2026-10-07: Made two messages say only what is true
|
- 2026-10-07: Made two messages say only what is true
|
||||||
([issue #240](https://git.eeqj.de/sneak/vaultik/issues/240)). A config
|
([issue #240](https://git.eeqj.de/sneak/vaultik/issues/240)). A config
|
||||||
file that others can read was warned about as containing S3
|
file that others can read was warned about as containing S3
|
||||||
|
|||||||
+10
-2
@@ -295,8 +295,10 @@ storage_url: "rclone://myremote/path/to/backups"
|
|||||||
|
|
||||||
# Path to local SQLite index database
|
# Path to local SQLite index database
|
||||||
# This database tracks file state for incremental backups
|
# This database tracks file state for incremental backups
|
||||||
# Default: /var/lib/vaultik/index.sqlite
|
# Default: the platform data directory, e.g.
|
||||||
#index_path: /var/lib/vaultik/index.sqlite
|
# macOS: ~/Library/Application Support/vaultik/index.sqlite
|
||||||
|
# Linux: ~/.local/share/vaultik/index.sqlite
|
||||||
|
#index_path: /path/to/index.sqlite
|
||||||
|
|
||||||
# Average chunk size for content-defined chunking
|
# Average chunk size for content-defined chunking
|
||||||
# Smaller chunks = better deduplication but more metadata
|
# Smaller chunks = better deduplication but more metadata
|
||||||
@@ -311,6 +313,12 @@ storage_url: "rclone://myremote/path/to/backups"
|
|||||||
# Chunking uses no secret (the FastCDC parameters are fixed and public). At a
|
# Chunking uses no secret (the FastCDC parameters are fixed and public). At a
|
||||||
# large limit a blob holds hundreds of chunks, so individual chunk lengths are
|
# large limit a blob holds hundreds of chunks, so individual chunk lengths are
|
||||||
# not visible in its size; lowering the limit toward chunk_size exposes them.
|
# not visible in its size; lowering the limit toward chunk_size exposes them.
|
||||||
|
# A backup needs free temporary space, because each blob is written whole to
|
||||||
|
# a temporary file before it is uploaded (up to about blob_size_limit; an
|
||||||
|
# rclone destination that cannot stream uploads needs about twice that) and
|
||||||
|
# the metadata export writes copies of the local index. Temporary files go to
|
||||||
|
# $TMPDIR (default /tmp); with TMPDIR unset, SQLite writes one of those copies
|
||||||
|
# to /var/tmp.
|
||||||
# Supports: 1GB, 10G, 500MB, 1GiB, etc.
|
# Supports: 1GB, 10G, 500MB, 1GiB, etc.
|
||||||
# Default: 10GB
|
# Default: 10GB
|
||||||
#blob_size_limit: 10GB
|
#blob_size_limit: 10GB
|
||||||
|
|||||||
+5
-6
@@ -111,7 +111,7 @@ Maps chunks to the blobs that contain them.
|
|||||||
Tracks backup snapshots.
|
Tracks backup snapshots.
|
||||||
|
|
||||||
**Columns:**
|
**Columns:**
|
||||||
- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: hostname-YYYYMMDD-HHMMSSZ)
|
- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: `hostname_name_timestamp`, e.g. `server1_home_2025-06-01T12:00:00Z`: the hostname up to its first `.`, the snapshot name, and an RFC 3339 UTC timestamp)
|
||||||
- `hostname` (TEXT) - Hostname where backup was created
|
- `hostname` (TEXT) - Hostname where backup was created
|
||||||
- `vaultik_version` (TEXT) - Version of Vaultik used
|
- `vaultik_version` (TEXT) - Version of Vaultik used
|
||||||
- `vaultik_git_revision` (TEXT) - Git revision of Vaultik used
|
- `vaultik_git_revision` (TEXT) - Git revision of Vaultik used
|
||||||
@@ -218,8 +218,8 @@ The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so
|
|||||||
### 4. Restore Process
|
### 4. Restore Process
|
||||||
|
|
||||||
The restore process doesn't use the local database. Instead:
|
The restore process doesn't use the local database. Instead:
|
||||||
1. Downloads snapshot metadata from S3
|
1. Downloads and decrypts the snapshot's metadata database (`db.zst.age`) from S3
|
||||||
2. Downloads required blobs based on manifest
|
2. Downloads the blobs holding the chunks of the files being restored, found through that database's `blob_chunks` table; the manifest is not read
|
||||||
3. Reconstructs files from decrypted and decompressed chunks
|
3. Reconstructs files from decrypted and decompressed chunks
|
||||||
|
|
||||||
### 5. Pruning
|
### 5. Pruning
|
||||||
@@ -232,9 +232,8 @@ The restore process doesn't use the local database. Instead:
|
|||||||
|
|
||||||
Before each backup:
|
Before each backup:
|
||||||
1. Query incomplete snapshots (where `completed_at IS NULL`)
|
1. Query incomplete snapshots (where `completed_at IS NULL`)
|
||||||
2. Check if metadata exists in S3
|
2. Delete each one and all its associations, without checking S3 for its metadata
|
||||||
3. If no metadata, delete snapshot and all associations
|
3. Clean up orphaned files, chunks, and blobs
|
||||||
4. Clean up orphaned files, chunks, and blobs
|
|
||||||
|
|
||||||
## Repository Pattern
|
## Repository Pattern
|
||||||
|
|
||||||
|
|||||||
@@ -22,9 +22,11 @@ scans every snapshot manifest in the destination store, builds the
|
|||||||
set of still-referenced blob hashes, and deletes any blob not in that
|
set of still-referenced blob hashes, and deletes any blob not in that
|
||||||
set.
|
set.
|
||||||
|
|
||||||
Snapshot create --prune and snapshot remove run the same cleanup
|
Snapshot create --prune runs the same cleanup automatically; this
|
||||||
automatically; this command is the manual entry point for the same
|
command is the manual entry point for the same work (e.g. after a
|
||||||
work (e.g. after a crashed backup or to reclaim storage).`,
|
crashed backup or to reclaim storage). Snapshot remove leaves blobs in
|
||||||
|
place; run this command afterwards to delete the ones no longer
|
||||||
|
referenced.`,
|
||||||
Args: cobra.NoArgs,
|
Args: cobra.NoArgs,
|
||||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||||
// Use unified config resolution
|
// Use unified config resolution
|
||||||
|
|||||||
@@ -66,8 +66,9 @@ func newSnapshotCreateCommand() *cobra.Command {
|
|||||||
If snapshot names are provided, only those snapshots are created.
|
If snapshot names are provided, only those snapshots are created.
|
||||||
If no names are provided, all configured snapshots are created.
|
If no names are provided, all configured snapshots are created.
|
||||||
|
|
||||||
Config is located at /etc/vaultik/config.yml by default, but can be overridden by
|
The config is read from the path given by --config or VAULTIK_CONFIG;
|
||||||
specifying a path using --config or by setting VAULTIK_CONFIG to a path.`,
|
otherwise from the platform config directory (~/.config/vaultik/config.yml
|
||||||
|
on Linux), then /etc/vaultik/config.yml.`,
|
||||||
Args: cobra.ArbitraryArgs,
|
Args: cobra.ArbitraryArgs,
|
||||||
RunE: func(cmd *cobra.Command, args []string) error {
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
// Pass snapshot names from args
|
// Pass snapshot names from args
|
||||||
|
|||||||
@@ -14,8 +14,7 @@ type File struct {
|
|||||||
ID types.FileID // UUID primary key
|
ID types.FileID // UUID primary key
|
||||||
Path types.FilePath // Absolute path of the file
|
Path types.FilePath // Absolute path of the file
|
||||||
|
|
||||||
// SourcePath is the source directory this file came from (used for
|
// SourcePath is the source directory this file came from.
|
||||||
// restore path stripping).
|
|
||||||
SourcePath types.SourcePath
|
SourcePath types.SourcePath
|
||||||
MTime time.Time
|
MTime time.Time
|
||||||
Size int64
|
Size int64
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package database
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"database/sql"
|
||||||
"fmt"
|
"fmt"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -367,7 +368,7 @@ func verifyBlobNullUploadTS(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// createLargeDatasetFiles creates fileCount files and adds every other
|
// createLargeDatasetFiles creates fileCount files and adds every other
|
||||||
// one to the snapshot.
|
// one to the snapshot, in one transaction as a backup writes them.
|
||||||
func createLargeDatasetFiles(
|
func createLargeDatasetFiles(
|
||||||
t *testing.T,
|
t *testing.T,
|
||||||
repos *Repositories,
|
repos *Repositories,
|
||||||
@@ -376,9 +377,10 @@ func createLargeDatasetFiles(
|
|||||||
) {
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
start := time.Now()
|
start := time.Now()
|
||||||
|
|
||||||
|
err := repos.WithTx(context.Background(),
|
||||||
|
func(ctx context.Context, tx *sql.Tx) error {
|
||||||
for i := range fileCount {
|
for i := range fileCount {
|
||||||
file := &File{
|
file := &File{
|
||||||
Path: types.FilePath(fmt.Sprintf("/large/file%05d.txt", i)),
|
Path: types.FilePath(fmt.Sprintf("/large/file%05d.txt", i)),
|
||||||
@@ -389,19 +391,25 @@ func createLargeDatasetFiles(
|
|||||||
GID: uint32(1000 + (i % 10)),
|
GID: uint32(1000 + (i % 10)),
|
||||||
}
|
}
|
||||||
|
|
||||||
err := repos.Files.Create(ctx, nil, file)
|
err := repos.Files.Create(ctx, tx, file)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("failed to create file %d: %v", i, err)
|
return fmt.Errorf("creating file %d: %w", i, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Add half to snapshot
|
// Add half to snapshot
|
||||||
if i%2 == 0 {
|
if i%2 == 0 {
|
||||||
err = repos.Snapshots.AddFileByID(ctx, nil, snapshotID, file.ID)
|
err = repos.Snapshots.AddFileByID(ctx, tx, snapshotID, file.ID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Logf("Created %d files in %v", fileCount, time.Since(start))
|
t.Logf("Created %d files in %v", fileCount, time.Since(start))
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,7 +5,7 @@
|
|||||||
CREATE TABLE IF NOT EXISTS files (
|
CREATE TABLE IF NOT EXISTS files (
|
||||||
id TEXT PRIMARY KEY, -- UUID
|
id TEXT PRIMARY KEY, -- UUID
|
||||||
path TEXT NOT NULL UNIQUE,
|
path TEXT NOT NULL UNIQUE,
|
||||||
source_path TEXT NOT NULL DEFAULT '', -- The source directory this file came from (for restore path stripping)
|
source_path TEXT NOT NULL DEFAULT '', -- The source directory this file came from
|
||||||
mtime INTEGER NOT NULL, -- whole seconds since the Unix epoch
|
mtime INTEGER NOT NULL, -- whole seconds since the Unix epoch
|
||||||
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
|
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
|
||||||
size INTEGER NOT NULL,
|
size INTEGER NOT NULL,
|
||||||
|
|||||||
@@ -58,8 +58,8 @@ type Scanner struct {
|
|||||||
compressionLevel int
|
compressionLevel int
|
||||||
ageRecipient string
|
ageRecipient string
|
||||||
snapshotID string // Current snapshot being processed
|
snapshotID string // Current snapshot being processed
|
||||||
// currentSourcePath is the source directory being scanned (used for
|
// currentSourcePath is the source directory being scanned, stored with
|
||||||
// restore path stripping).
|
// each file record.
|
||||||
currentSourcePath string
|
currentSourcePath string
|
||||||
exclude []string // Glob patterns for files/directories to exclude
|
exclude []string // Glob patterns for files/directories to exclude
|
||||||
compiledExclude []compiledPattern // Compiled glob patterns
|
compiledExclude []compiledPattern // Compiled glob patterns
|
||||||
@@ -211,7 +211,6 @@ func (s *Scanner) Scan(
|
|||||||
ctx context.Context, path string, snapshotID string,
|
ctx context.Context, path string, snapshotID string,
|
||||||
) (*ScanResult, error) {
|
) (*ScanResult, error) {
|
||||||
s.snapshotID = snapshotID
|
s.snapshotID = snapshotID
|
||||||
// Store source path for file records (used during restore)
|
|
||||||
s.currentSourcePath = path
|
s.currentSourcePath = path
|
||||||
result := &ScanResult{
|
result := &ScanResult{
|
||||||
StartTime: time.Now().UTC(),
|
StartTime: time.Now().UTC(),
|
||||||
@@ -1200,7 +1199,6 @@ func (s *Scanner) checkFileInMemory(
|
|||||||
file := &database.File{
|
file := &database.File{
|
||||||
ID: fileID,
|
ID: fileID,
|
||||||
Path: types.FilePath(path),
|
Path: types.FilePath(path),
|
||||||
// Store source directory for restore path stripping
|
|
||||||
SourcePath: types.SourcePath(s.currentSourcePath),
|
SourcePath: types.SourcePath(s.currentSourcePath),
|
||||||
MTime: info.ModTime(),
|
MTime: info.ModTime(),
|
||||||
Size: info.Size(),
|
Size: info.Size(),
|
||||||
|
|||||||
@@ -155,8 +155,8 @@ type BlobHash string
|
|||||||
// FilePath represents an absolute path to a file or directory.
|
// FilePath represents an absolute path to a file or directory.
|
||||||
type FilePath string
|
type FilePath string
|
||||||
|
|
||||||
// SourcePath represents the root directory from which files are backed up.
|
// SourcePath is the source directory a scan found a file under, made
|
||||||
// Used during restore to strip the source prefix from paths.
|
// absolute and with symlinks resolved.
|
||||||
type SourcePath string
|
type SourcePath string
|
||||||
|
|
||||||
// Hostname identifies a host machine.
|
// Hostname identifies a host machine.
|
||||||
|
|||||||
@@ -130,10 +130,9 @@ func assertThirdSnapshotRestores(
|
|||||||
// up, and that snapshot is removed. The first snapshot keeps the file row,
|
// up, and that snapshot is removed. The first snapshot keeps the file row,
|
||||||
// which now lists the appended content's chunks, while removal drops the
|
// which now lists the appended content's chunks, while removal drops the
|
||||||
// blob that held them.
|
// blob that held them.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestBackupAfterRemovingNewestSnapshotRestoresChangedFile(t *testing.T) {
|
func TestBackupAfterRemovingNewestSnapshotRestoresChangedFile(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
fs := afero.NewOsFs()
|
fs := afero.NewOsFs()
|
||||||
tempDir := t.TempDir()
|
tempDir := t.TempDir()
|
||||||
@@ -178,10 +177,9 @@ func TestBackupAfterRemovingNewestSnapshotRestoresChangedFile(t *testing.T) {
|
|||||||
// The next run's prune drops that incomplete snapshot and its blob, while
|
// The next run's prune drops that incomplete snapshot and its blob, while
|
||||||
// the first snapshot keeps the file row, which now lists the appended
|
// the first snapshot keeps the file row, which now lists the appended
|
||||||
// content's chunks.
|
// content's chunks.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestBackupAfterInterruptedRunRestoresChangedFile(t *testing.T) {
|
func TestBackupAfterInterruptedRunRestoresChangedFile(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
fs := afero.NewOsFs()
|
fs := afero.NewOsFs()
|
||||||
tempDir := t.TempDir()
|
tempDir := t.TempDir()
|
||||||
|
|||||||
@@ -38,11 +38,9 @@ import (
|
|||||||
// (https://git.eeqj.de/sneak/vaultik/issues/130) and is not re-tested
|
// (https://git.eeqj.de/sneak/vaultik/issues/130) and is not re-tested
|
||||||
// here; these tests target the layers above the backend.
|
// here; these tests target the layers above the backend.
|
||||||
//
|
//
|
||||||
// The tests run serially, not with t.Parallel: each calls
|
// log.Initialize replaces the package-global logger that a running
|
||||||
// log.Initialize, which replaces the package-global logger, and a
|
// backup or restore reads, so each test calls it before t.Parallel,
|
||||||
// backup or restore running concurrently reads that same logger. Under
|
// while no parallel test is running yet.
|
||||||
// -race the two collide. Running one at a time is the same choice
|
|
||||||
// prune_count_test.go already makes for the same reason.
|
|
||||||
|
|
||||||
const (
|
const (
|
||||||
faultChunkSize = int64(64 * 1024)
|
faultChunkSize = int64(64 * 1024)
|
||||||
@@ -165,17 +163,19 @@ func newReaderVaultik(
|
|||||||
// Scenario 3: a stored blob's bytes are flipped before restore reads
|
// Scenario 3: a stored blob's bytes are flipped before restore reads
|
||||||
// them. Restore must fail loudly, and no file must be left on the
|
// them. Restore must fail loudly, and no file must be left on the
|
||||||
// restore target holding corrupt content.
|
// restore target holding corrupt content.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestRestoreRejectsCorruptBlob(t *testing.T) {
|
func TestRestoreRejectsCorruptBlob(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
assertRestoreRejectsDamagedBlob(t, faultstore.GetCorrupt, "corrupt")
|
assertRestoreRejectsDamagedBlob(t, faultstore.GetCorrupt, "corrupt")
|
||||||
}
|
}
|
||||||
|
|
||||||
// Scenario 4: a stored blob is truncated before restore reads it. Same
|
// Scenario 4: a stored blob is truncated before restore reads it. Same
|
||||||
// contract as the corrupt case.
|
// contract as the corrupt case.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestRestoreRejectsTruncatedBlob(t *testing.T) {
|
func TestRestoreRejectsTruncatedBlob(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
assertRestoreRejectsDamagedBlob(t, faultstore.GetTruncate, "truncated")
|
assertRestoreRejectsDamagedBlob(t, faultstore.GetTruncate, "truncated")
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -188,7 +188,6 @@ func assertRestoreRejectsDamagedBlob(
|
|||||||
t *testing.T, fault faultstore.GetFault, name string,
|
t *testing.T, fault faultstore.GetFault, name string,
|
||||||
) {
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
log.Initialize(log.Config{})
|
|
||||||
|
|
||||||
fs := afero.NewOsFs()
|
fs := afero.NewOsFs()
|
||||||
tempDir := t.TempDir()
|
tempDir := t.TempDir()
|
||||||
@@ -232,10 +231,9 @@ func assertRestoreRejectsDamagedBlob(
|
|||||||
|
|
||||||
// Scenario 6: the backend accepts blob uploads and reports success but
|
// Scenario 6: the backend accepts blob uploads and reports success but
|
||||||
// stores nothing. verify --deep must catch it.
|
// stores nothing. verify --deep must catch it.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestDeepVerifyCatchesLyingBackend(t *testing.T) {
|
func TestDeepVerifyCatchesLyingBackend(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
fs := afero.NewOsFs()
|
fs := afero.NewOsFs()
|
||||||
tempDir := t.TempDir()
|
tempDir := t.TempDir()
|
||||||
@@ -285,10 +283,9 @@ func TestDeepVerifyCatchesLyingBackend(t *testing.T) {
|
|||||||
// Scenario 1a: a blob upload fails partway through. The interrupted run
|
// Scenario 1a: a blob upload fails partway through. The interrupted run
|
||||||
// must not record the blob as uploaded, must not reference it from the
|
// must not record the blob as uploaded, must not reference it from the
|
||||||
// snapshot, and must leave no blob object at the destination.
|
// snapshot, and must leave no blob object at the destination.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestInterruptedBlobUploadRecordsNoUploadedBlob(t *testing.T) {
|
func TestInterruptedBlobUploadRecordsNoUploadedBlob(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
fs := afero.NewOsFs()
|
fs := afero.NewOsFs()
|
||||||
tempDir := t.TempDir()
|
tempDir := t.TempDir()
|
||||||
@@ -361,10 +358,9 @@ func TestInterruptedBlobUploadRecordsNoUploadedBlob(t *testing.T) {
|
|||||||
// chunks in a blob that was actually uploaded, so the retry re-chunks and
|
// chunks in a blob that was actually uploaded, so the retry re-chunks and
|
||||||
// re-uploads the affected data instead of silently referencing data that
|
// re-uploads the affected data instead of silently referencing data that
|
||||||
// never reached storage.
|
// never reached storage.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestBackupRetryAfterInterruptedUploadIsRestorable(t *testing.T) {
|
func TestBackupRetryAfterInterruptedUploadIsRestorable(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
fs := afero.NewOsFs()
|
fs := afero.NewOsFs()
|
||||||
tempDir := t.TempDir()
|
tempDir := t.TempDir()
|
||||||
@@ -426,10 +422,9 @@ func TestBackupRetryAfterInterruptedUploadIsRestorable(t *testing.T) {
|
|||||||
// covered by TestBackupCompletesOnlyAfterMetadataExport
|
// covered by TestBackupCompletesOnlyAfterMetadataExport
|
||||||
// (https://git.eeqj.de/sneak/vaultik/issues/177); this test exercises the
|
// (https://git.eeqj.de/sneak/vaultik/issues/177); this test exercises the
|
||||||
// lower-level export path in isolation.
|
// lower-level export path in isolation.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestBackupSurvivesMetadataExportInterruption(t *testing.T) {
|
func TestBackupSurvivesMetadataExportInterruption(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
fs := afero.NewOsFs()
|
fs := afero.NewOsFs()
|
||||||
tempDir := t.TempDir()
|
tempDir := t.TempDir()
|
||||||
@@ -509,10 +504,9 @@ func TestBackupSurvivesMetadataExportInterruption(t *testing.T) {
|
|||||||
// destination. Rerunning the backup must then prune the incomplete
|
// destination. Rerunning the backup must then prune the incomplete
|
||||||
// snapshot, produce a snapshot whose destination metadata and local index
|
// snapshot, produce a snapshot whose destination metadata and local index
|
||||||
// agree, and restore. See https://git.eeqj.de/sneak/vaultik/issues/177.
|
// agree, and restore. See https://git.eeqj.de/sneak/vaultik/issues/177.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestBackupCompletesOnlyAfterMetadataExport(t *testing.T) {
|
func TestBackupCompletesOnlyAfterMetadataExport(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
fs := afero.NewOsFs()
|
fs := afero.NewOsFs()
|
||||||
tempDir := t.TempDir()
|
tempDir := t.TempDir()
|
||||||
@@ -685,10 +679,9 @@ func faultScannerFactory(
|
|||||||
// Scenario 5: the restore target runs out of space mid-file. Restore
|
// Scenario 5: the restore target runs out of space mid-file. Restore
|
||||||
// must fail with an out-of-space error, and must not leave a truncated
|
// must fail with an out-of-space error, and must not leave a truncated
|
||||||
// file at the target path presenting as a complete restore.
|
// file at the target path presenting as a complete restore.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestRestoreReportsDiskFull(t *testing.T) {
|
func TestRestoreReportsDiskFull(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
osFS := afero.NewOsFs()
|
osFS := afero.NewOsFs()
|
||||||
tempDir := t.TempDir()
|
tempDir := t.TempDir()
|
||||||
|
|||||||
@@ -928,17 +928,17 @@ func setupDedupBackupEnv(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// runDedupSnapshot creates a "dedup" snapshot, scans dataDir into it,
|
// runDedupSnapshot creates a snapshot with the given name, scans dataDir
|
||||||
// completes it, and exports its metadata, returning the snapshot ID and
|
// into it, completes it, and exports its metadata, returning the snapshot
|
||||||
// scan result.
|
// ID and scan result.
|
||||||
func runDedupSnapshot(
|
func runDedupSnapshot(
|
||||||
ctx context.Context, t *testing.T,
|
ctx context.Context, t *testing.T,
|
||||||
sm *snapshot.SnapshotManager, scanner *snapshot.Scanner,
|
sm *snapshot.SnapshotManager, scanner *snapshot.Scanner,
|
||||||
hostname, dataDir, dbPath string,
|
hostname, name, dataDir, dbPath string,
|
||||||
) (string, *snapshot.ScanResult) {
|
) (string, *snapshot.ScanResult) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
id, err := sm.CreateSnapshotWithName(ctx, hostname, "dedup", "v", "g")
|
id, err := sm.CreateSnapshotWithName(ctx, hostname, name, "v", "g")
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
result, err := scanner.Scan(ctx, dataDir, id)
|
result, err := scanner.Scan(ctx, dataDir, id)
|
||||||
@@ -980,16 +980,15 @@ func TestDedupOnlySnapshotRestores(t *testing.T) {
|
|||||||
|
|
||||||
// First snapshot — uploads all blobs.
|
// First snapshot — uploads all blobs.
|
||||||
_, r1 := runDedupSnapshot(ctx, t, sm, makeScanner(),
|
_, r1 := runDedupSnapshot(ctx, t, sm, makeScanner(),
|
||||||
cfg.Hostname, dataDir, dbPath)
|
cfg.Hostname, "first", dataDir, dbPath)
|
||||||
require.Positive(t, r1.BlobsCreated,
|
require.Positive(t, r1.BlobsCreated,
|
||||||
"first snapshot should upload at least one blob")
|
"first snapshot should upload at least one blob")
|
||||||
|
|
||||||
// Second snapshot — same data, every chunk dedups. Sleep past the
|
// Second snapshot — same data, every chunk dedups. Its own name gives
|
||||||
// second-precision timestamp so the snapshot IDs differ.
|
// it a different snapshot ID without waiting for the one-second
|
||||||
time.Sleep(1100 * time.Millisecond)
|
// timestamp in the ID to tick over.
|
||||||
|
|
||||||
id2, r2 := runDedupSnapshot(ctx, t, sm, makeScanner(),
|
id2, r2 := runDedupSnapshot(ctx, t, sm, makeScanner(),
|
||||||
cfg.Hostname, dataDir, dbPath)
|
cfg.Hostname, "second", dataDir, dbPath)
|
||||||
require.Equal(t, 0, r2.BlobsCreated,
|
require.Equal(t, 0, r2.BlobsCreated,
|
||||||
"second snapshot should upload zero new blobs (fully dedup'd)")
|
"second snapshot should upload zero new blobs (fully dedup'd)")
|
||||||
|
|
||||||
|
|||||||
@@ -78,10 +78,9 @@ func backUpThenUnplug(
|
|||||||
// TestFirstBackupCreatesDestinationDirectory checks that a first backup
|
// TestFirstBackupCreatesDestinationDirectory checks that a first backup
|
||||||
// to a destination directory that does not exist yet creates it, and
|
// to a destination directory that does not exist yet creates it, and
|
||||||
// that the destination can be listed afterwards.
|
// that the destination can be listed afterwards.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestFirstBackupCreatesDestinationDirectory(t *testing.T) {
|
func TestFirstBackupCreatesDestinationDirectory(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
storeDir := filepath.Join(t.TempDir(), "volume", "backup")
|
storeDir := filepath.Join(t.TempDir(), "volume", "backup")
|
||||||
@@ -96,10 +95,9 @@ func TestFirstBackupCreatesDestinationDirectory(t *testing.T) {
|
|||||||
// TestListSnapshotsWarnsWhenDestinationMissing checks that snapshot list
|
// TestListSnapshotsWarnsWhenDestinationMissing checks that snapshot list
|
||||||
// warns and shows the local index alone, without reporting the local
|
// warns and shows the local index alone, without reporting the local
|
||||||
// snapshot as missing from the destination.
|
// snapshot as missing from the destination.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestListSnapshotsWarnsWhenDestinationMissing(t *testing.T) {
|
func TestListSnapshotsWarnsWhenDestinationMissing(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
v, repos, out := backUpThenUnplug(ctx, t)
|
v, repos, out := backUpThenUnplug(ctx, t)
|
||||||
@@ -116,10 +114,9 @@ func TestListSnapshotsWarnsWhenDestinationMissing(t *testing.T) {
|
|||||||
// TestRemoveSnapshotWarnsWhenDestinationMissing checks that snapshot
|
// TestRemoveSnapshotWarnsWhenDestinationMissing checks that snapshot
|
||||||
// remove warns that the metadata could not be removed from the
|
// remove warns that the metadata could not be removed from the
|
||||||
// destination, instead of reporting that it was.
|
// destination, instead of reporting that it was.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestRemoveSnapshotWarnsWhenDestinationMissing(t *testing.T) {
|
func TestRemoveSnapshotWarnsWhenDestinationMissing(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
v, repos, out := backUpThenUnplug(ctx, t)
|
v, repos, out := backUpThenUnplug(ctx, t)
|
||||||
@@ -138,10 +135,9 @@ func TestRemoveSnapshotWarnsWhenDestinationMissing(t *testing.T) {
|
|||||||
// TestPruneKeepsLocalRecordsWhenDestinationMissing checks that prune
|
// TestPruneKeepsLocalRecordsWhenDestinationMissing checks that prune
|
||||||
// fails on a destination it cannot list and deletes no local snapshot
|
// fails on a destination it cannot list and deletes no local snapshot
|
||||||
// record.
|
// record.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestPruneKeepsLocalRecordsWhenDestinationMissing(t *testing.T) {
|
func TestPruneKeepsLocalRecordsWhenDestinationMissing(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
v, repos, _ := backUpThenUnplug(ctx, t)
|
v, repos, _ := backUpThenUnplug(ctx, t)
|
||||||
@@ -158,10 +154,9 @@ func TestPruneKeepsLocalRecordsWhenDestinationMissing(t *testing.T) {
|
|||||||
// TestPurgeSaysListingFailedOnceWhenDestinationMissing checks that
|
// TestPurgeSaysListingFailedOnceWhenDestinationMissing checks that
|
||||||
// snapshot purge fails on a destination it cannot list, with an error
|
// snapshot purge fails on a destination it cannot list, with an error
|
||||||
// that says "listing remote snapshots" once.
|
// that says "listing remote snapshots" once.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestPurgeSaysListingFailedOnceWhenDestinationMissing(t *testing.T) {
|
func TestPurgeSaysListingFailedOnceWhenDestinationMissing(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
v, _, _ := backUpThenUnplug(ctx, t)
|
v, _, _ := backUpThenUnplug(ctx, t)
|
||||||
|
|||||||
@@ -14,10 +14,9 @@ import (
|
|||||||
// the discarded-error bug: getTableCount for a table its query cannot
|
// the discarded-error bug: getTableCount for a table its query cannot
|
||||||
// resolve must not silently become 0. A count that could not be read is
|
// resolve must not silently become 0. A count that could not be read is
|
||||||
// reported as unknown, which a reader can tell apart from an empty table.
|
// reported as unknown, which a reader can tell apart from an empty table.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestTableCountForReportSurfacesReadFailure(t *testing.T) {
|
func TestTableCountForReportSurfacesReadFailure(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
|
|
||||||
|
|||||||
@@ -164,10 +164,9 @@ func scratchEntries(t *testing.T, dir string) []string {
|
|||||||
// restore while a blob download is in progress. The download fails only
|
// restore while a blob download is in progress. The download fails only
|
||||||
// because of the cancel, so Restore must return context.Canceled without
|
// because of the cancel, so Restore must return context.Canceled without
|
||||||
// reporting the file that needs the blob as failed.
|
// reporting the file that needs the blob as failed.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestRestoreSkipErrorsCancelDuringBlobDownload(t *testing.T) {
|
func TestRestoreSkipErrorsCancelDuringBlobDownload(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
fs := afero.NewOsFs()
|
fs := afero.NewOsFs()
|
||||||
tempDir := t.TempDir()
|
tempDir := t.TempDir()
|
||||||
|
|||||||
@@ -45,9 +45,10 @@ type missingBlobBackup struct {
|
|||||||
// after one blob of a two-blob snapshot was deleted. Every file stored in
|
// after one blob of a two-blob snapshot was deleted. Every file stored in
|
||||||
// that blob must be reported as failed and left absent, every other file
|
// that blob must be reported as failed and left absent, every other file
|
||||||
// must be restored intact, and Restore must still return an error.
|
// must be restored intact, and Restore must still return an error.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestRestoreSkipErrorsSkipsFilesOfMissingBlob(t *testing.T) {
|
func TestRestoreSkipErrorsSkipsFilesOfMissingBlob(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
backup := backupThenDeleteOneBlob(ctx, t)
|
backup := backupThenDeleteOneBlob(ctx, t)
|
||||||
|
|
||||||
@@ -85,9 +86,10 @@ func TestRestoreSkipErrorsSkipsFilesOfMissingBlob(t *testing.T) {
|
|||||||
|
|
||||||
// TestRestoreMissingBlobAbortsWithoutSkipErrors checks that a deleted blob
|
// TestRestoreMissingBlobAbortsWithoutSkipErrors checks that a deleted blob
|
||||||
// still ends the restore with an error when SkipErrors is not set.
|
// still ends the restore with an error when SkipErrors is not set.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestRestoreMissingBlobAbortsWithoutSkipErrors(t *testing.T) {
|
func TestRestoreMissingBlobAbortsWithoutSkipErrors(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
backup := backupThenDeleteOneBlob(ctx, t)
|
backup := backupThenDeleteOneBlob(ctx, t)
|
||||||
|
|
||||||
@@ -107,7 +109,6 @@ func backupThenDeleteOneBlob(
|
|||||||
ctx context.Context, t *testing.T,
|
ctx context.Context, t *testing.T,
|
||||||
) *missingBlobBackup {
|
) *missingBlobBackup {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
log.Initialize(log.Config{})
|
|
||||||
|
|
||||||
fs := afero.NewOsFs()
|
fs := afero.NewOsFs()
|
||||||
tempDir := t.TempDir()
|
tempDir := t.TempDir()
|
||||||
|
|||||||
@@ -18,10 +18,9 @@ import (
|
|||||||
// A file rewritten with its size unchanged and a new mtime in the same
|
// A file rewritten with its size unchanged and a new mtime in the same
|
||||||
// second as the mtime the index holds must still be backed up. See
|
// second as the mtime the index holds must still be backed up. See
|
||||||
// https://git.eeqj.de/sneak/vaultik/issues/226.
|
// https://git.eeqj.de/sneak/vaultik/issues/226.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestBackupOfSameSecondRewriteRestoresNewContent(t *testing.T) {
|
func TestBackupOfSameSecondRewriteRestoresNewContent(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
fs := afero.NewOsFs()
|
fs := afero.NewOsFs()
|
||||||
tempDir := t.TempDir()
|
tempDir := t.TempDir()
|
||||||
|
|||||||
@@ -54,8 +54,6 @@ type summaryEnv struct {
|
|||||||
func newSummaryEnv(t *testing.T) *summaryEnv {
|
func newSummaryEnv(t *testing.T) *summaryEnv {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
log.Initialize(log.Config{})
|
|
||||||
|
|
||||||
fs := afero.NewOsFs()
|
fs := afero.NewOsFs()
|
||||||
tempDir := t.TempDir()
|
tempDir := t.TempDir()
|
||||||
srcDir := filepath.Join(tempDir, "src")
|
srcDir := filepath.Join(tempDir, "src")
|
||||||
@@ -193,9 +191,10 @@ func (e *summaryEnv) dataLine(total, backedUp int64) string {
|
|||||||
// A first backup stores copy.bin's chunks while backing up a.bin, so
|
// A first backup stores copy.bin's chunks while backing up a.bin, so
|
||||||
// copy.bin's chunks are deduplicated within the run. Each file and byte
|
// copy.bin's chunks are deduplicated within the run. Each file and byte
|
||||||
// is still counted once.
|
// is still counted once.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestSnapshotSummaryFirstRun(t *testing.T) {
|
func TestSnapshotSummaryFirstRun(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
env := newSummaryEnv(t)
|
env := newSummaryEnv(t)
|
||||||
|
|
||||||
summary := env.backUp(t, "first", false)
|
summary := env.backUp(t, "first", false)
|
||||||
@@ -219,9 +218,10 @@ func TestSnapshotSummaryFirstRun(t *testing.T) {
|
|||||||
// An incremental backup where a.bin's mtime changed but its content did
|
// An incremental backup where a.bin's mtime changed but its content did
|
||||||
// not: a.bin is backed up again and every one of its chunks is already
|
// not: a.bin is backed up again and every one of its chunks is already
|
||||||
// stored.
|
// stored.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestSnapshotSummaryIncrementalRunWithDeduplicatedChunks(t *testing.T) {
|
func TestSnapshotSummaryIncrementalRunWithDeduplicatedChunks(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
env := newSummaryEnv(t)
|
env := newSummaryEnv(t)
|
||||||
|
|
||||||
env.backUp(t, "first", false)
|
env.backUp(t, "first", false)
|
||||||
@@ -251,9 +251,10 @@ func TestSnapshotSummaryIncrementalRunWithDeduplicatedChunks(t *testing.T) {
|
|||||||
// Under --cron the progress reporter is off; the upload figures must
|
// Under --cron the progress reporter is off; the upload figures must
|
||||||
// still reach the summary and the snapshots row. The snapshot has two
|
// still reach the summary and the snapshots row. The snapshot has two
|
||||||
// paths, each backed up by its own scan.
|
// paths, each backed up by its own scan.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestSnapshotSummaryCronRunRecordsUploads(t *testing.T) {
|
func TestSnapshotSummaryCronRunRecordsUploads(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
env := newSummaryEnv(t)
|
env := newSummaryEnv(t)
|
||||||
|
|
||||||
summary := env.backUp(t, "split", true)
|
summary := env.backUp(t, "split", true)
|
||||||
|
|||||||
@@ -18,10 +18,9 @@ import (
|
|||||||
// A backup without --cron runs the progress reporter while one scanner
|
// A backup without --cron runs the progress reporter while one scanner
|
||||||
// scans each path of the snapshot in turn. See
|
// scans each path of the snapshot in turn. See
|
||||||
// https://git.eeqj.de/sneak/vaultik/issues/253.
|
// https://git.eeqj.de/sneak/vaultik/issues/253.
|
||||||
//
|
|
||||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
|
||||||
func TestBackupWithoutCronOfTwoPathSnapshotRestoresBothPaths(t *testing.T) {
|
func TestBackupWithoutCronOfTwoPathSnapshotRestoresBothPaths(t *testing.T) {
|
||||||
log.Initialize(log.Config{})
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
const snapshotName = "data"
|
const snapshotName = "data"
|
||||||
|
|
||||||
|
|||||||
+5
-2
@@ -1,6 +1,9 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/fmt-check: check formatting (read-only). Same scope as
|
# script/fmt-check: check formatting (read-only). Fails instead of
|
||||||
# script/fmt, but fails instead of writing.
|
# writing. It checks every Go file outside .tool, which is more than
|
||||||
|
# script/fmt formats: `go fmt ./...` skips `testdata` directories and
|
||||||
|
# files and directories whose names start with `.` or `_`. Fix a file
|
||||||
|
# only this reports with `gofmt -w`.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|||||||
+3
-4
@@ -21,10 +21,9 @@ goreleaser_version() {
|
|||||||
head -n 1
|
head -n 1
|
||||||
}
|
}
|
||||||
|
|
||||||
# Resolve the goreleaser to run, on the same rule script/lint uses for
|
# Resolve the goreleaser to run. A binary on PATH is accepted only when
|
||||||
# golangci-lint: a binary on PATH is accepted only when it is exactly
|
# it is exactly the pinned version, because a differently versioned tool
|
||||||
# the pinned version, because a differently versioned tool would
|
# would produce a differently built release from the same tag. Anything else
|
||||||
# produce a differently built release from the same tag. Anything else
|
|
||||||
# comes from .tool/bin, and a missing one is a loud failure naming the
|
# comes from .tool/bin, and a missing one is a loud failure naming the
|
||||||
# script that installs it rather than a silent fallback.
|
# script that installs it rather than a silent fallback.
|
||||||
resolve_goreleaser() {
|
resolve_goreleaser() {
|
||||||
|
|||||||
Reference in New Issue
Block a user