Compare commits
4
Commits
a99f9877d6
..
next
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6fcd8e1668 | ||
|
|
aab6a87f8c | ||
|
|
c355ef4d25 | ||
|
|
5927e1aa3d |
@@ -104,7 +104,12 @@ Version: 2025-06-08
|
||||
|
||||
13. Pre-1.0: NEVER write database migrations. There are no live databases
|
||||
anywhere — every user's local index can be rebuilt from a fresh full
|
||||
backup. When the schema changes, just change `schema.sql` (and any code
|
||||
that touches the affected tables). The local index is disposable until
|
||||
1.0 ships and is tagged.
|
||||
backup. To change the schema, edit `internal/database/schema/001.sql`
|
||||
(and any code that touches the affected tables) directly; do not add new
|
||||
numbered schema files. Those numbered files and the `schema_migrations`
|
||||
table they populate only bootstrap a fresh database — they are not an
|
||||
upgrade path. The local index is disposable until 1.0 ships and is
|
||||
tagged; once 1.0 is tagged that clause expires and the question of
|
||||
upgrading existing indexes returns. See [`docs/DATAMODEL.md`](docs/DATAMODEL.md)
|
||||
for the full explanation.
|
||||
|
||||
|
||||
@@ -457,9 +457,13 @@ Key fields:
|
||||
sequentially. Restore speed is bound by single-stream throughput.
|
||||
* **Device nodes, named pipes, and sockets are silently skipped.** Only
|
||||
regular files, directories, and symlinks are backed up.
|
||||
* **No database migrations.** If the local SQLite schema changes between
|
||||
versions, delete the local database (`vaultik database delete`) and run
|
||||
a full backup. Remote storage is unaffected.
|
||||
* **No upgrade path between versions.** There is no supported way to carry
|
||||
an existing local index across a schema change; if the local SQLite
|
||||
schema changes between versions, delete the local database (`vaultik
|
||||
database delete`) and run a full backup. Remote storage is unaffected.
|
||||
(The binary does embed numbered schema files and a `schema_migrations`
|
||||
table to bootstrap a fresh database — see [`docs/DATAMODEL.md`](docs/DATAMODEL.md)
|
||||
— but that is not an upgrade path.)
|
||||
* **Files that change during backup may be inconsistent.** There is no
|
||||
filesystem snapshot or freeze. If a file is modified between the scan
|
||||
and chunk phases, the backed-up copy may reflect a partial write.
|
||||
@@ -529,10 +533,12 @@ priority.
|
||||
another host" workflow works but isn't documented as a
|
||||
first-class operation in this README. Worth a dedicated section
|
||||
once it's settled.
|
||||
* **Schema migrations.** Currently nonexistent — pre-1.0 schema
|
||||
changes are handled by `vaultik database delete` plus a full
|
||||
re-scan. Post-1.0 we'll need a migration story to keep existing
|
||||
index databases usable across upgrades.
|
||||
* **Cross-version schema upgrades.** There is no upgrade path between
|
||||
released versions — pre-1.0 schema changes are handled by `vaultik
|
||||
database delete` plus a full re-scan (see
|
||||
[`docs/DATAMODEL.md`](docs/DATAMODEL.md)). Post-1.0 we'll need a
|
||||
migration story to keep existing index databases usable across
|
||||
upgrades.
|
||||
* **Storage backend coverage tests.** S3, file://, and rclone://
|
||||
all share the Storer interface but the rclone path is the least
|
||||
exercised in CI.
|
||||
|
||||
@@ -25,6 +25,16 @@ release" is exactly the contradiction
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-09-21: Stopped `prune` from reporting a failed row count as 0
|
||||
([issue #96](https://git.eeqj.de/sneak/vaultik/issues/96)). The seven
|
||||
`getTableCount` reads in `PruneDatabase` discarded their error, so a
|
||||
query that could not run became a plausible `0` and the before/after
|
||||
delta computed from it looked like real work. Each read now logs at
|
||||
warn on failure and renders as `unknown`, never `0`, so an empty table
|
||||
is distinguishable from one that could not be queried. The counts have
|
||||
no `--json` representation — under `--json` the summary is suppressed
|
||||
entirely — so nothing there can show a false `0`.
|
||||
|
||||
- 2026-09-21: Made the s3 storage backend report a missing object as
|
||||
`storage.ErrNotFound`, like the `file` and `rclone` backends and as the
|
||||
`Storer` interface documents. `S3Storer.Get` and `Stat` returned the raw
|
||||
@@ -33,7 +43,6 @@ release" is exactly the contradiction
|
||||
helper (reused by `HeadObject`) and a test that a missing key maps to
|
||||
`ErrNotFound`
|
||||
([issue #129](https://git.eeqj.de/sneak/vaultik/issues/129)).
|
||||
|
||||
- 2026-09-21: Fixed `verify --deep` reporting healthy snapshots as
|
||||
corrupt. Its final blob-integrity check hashed the encrypted
|
||||
downloaded bytes with a single SHA256 and compared that to the blob
|
||||
|
||||
+24
-5
@@ -5,11 +5,30 @@
|
||||
Vaultik uses a local SQLite database to track file metadata, chunk mappings, and blob associations during the backup process. This database serves as an index for incremental backups and enables efficient deduplication.
|
||||
|
||||
**Important Notes:**
|
||||
- **No Migration Support (pre-1.0)**: Vaultik does not support database schema
|
||||
migrations. The local index is treated as disposable — if the schema changes,
|
||||
delete the local SQLite database (`vaultik database delete`) and run a full
|
||||
backup. The remote storage is unaffected; the new index will re-deduplicate
|
||||
against existing remote blobs.
|
||||
|
||||
This section is the authoritative explanation of the schema/migration story;
|
||||
other documents (the README and `AGENTS.md`) link here.
|
||||
|
||||
- **No upgrade path between versions (pre-1.0)**: Vaultik has no supported way to
|
||||
carry an existing local index across a schema change. The index is disposable
|
||||
— if the on-disk schema changes between versions, delete the local SQLite
|
||||
database (`vaultik database delete`) and run a full backup. Remote storage is
|
||||
unaffected; the new index re-deduplicates against existing remote blobs. This
|
||||
is the standing project policy, and it is separate from the schema bootstrap
|
||||
described next.
|
||||
- **Schema bootstrap**: a fresh database is populated from numbered SQL files
|
||||
embedded in the binary under `internal/database/schema/`. `000.sql` creates the
|
||||
`schema_migrations` table; `001.sql` creates the application tables. On opening
|
||||
a database the code applies each numbered file that has not yet run and records
|
||||
its version in `schema_migrations`. This bootstraps a new database; it does not
|
||||
upgrade an existing one between released versions.
|
||||
- **Changing the schema (pre-1.0)**: edit `internal/database/schema/001.sql` (and
|
||||
the code that touches the affected tables) directly. Do not add new numbered
|
||||
files — there is no installed base to migrate.
|
||||
- **Disposability expires at 1.0**: the index is treated as disposable only until
|
||||
1.0 ships and is tagged. Once 1.0 is tagged that clause expires and the
|
||||
question of upgrading existing indexes returns. It is deliberately left open
|
||||
here.
|
||||
- **Version Compatibility**: In rare cases, you may need to use the same version
|
||||
of Vaultik to restore a backup as was used to create it. This ensures
|
||||
compatibility with the metadata format stored in S3.
|
||||
|
||||
+79
-54
@@ -46,31 +46,18 @@ func (f *FileStorer) SetFilesystem(fs afero.Fs) {
|
||||
// storage base path.
|
||||
const storageDirPerm = 0o755
|
||||
|
||||
// tempSuffix marks a partially written object. writeAtomic streams into a
|
||||
// temp file carrying this suffix and only renames it onto the real key once
|
||||
// the whole object is on disk, so an interrupted write can never leave a
|
||||
// truncated object at the key a later run would Stat and trust as a complete
|
||||
// blob. List and ListStream skip these files, so a leftover from an
|
||||
// interrupted write is never listed or trusted as a blob; it is otherwise
|
||||
// harmless and is overwritten when the same key is written again.
|
||||
const tempSuffix = ".partial"
|
||||
|
||||
// Put stores data at the specified key.
|
||||
func (f *FileStorer) Put(_ context.Context, key string, data io.Reader) error {
|
||||
path := f.fullPath(key)
|
||||
|
||||
// Create parent directories
|
||||
dir := filepath.Dir(path)
|
||||
|
||||
err := f.fs.MkdirAll(dir, storageDirPerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating directories: %w", err)
|
||||
}
|
||||
|
||||
file, err := f.fs.Create(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating file: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = file.Close() }()
|
||||
|
||||
_, err = io.Copy(file, data)
|
||||
if err != nil {
|
||||
return fmt.Errorf("writing file: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
return f.writeAtomic(key, data, nil)
|
||||
}
|
||||
|
||||
// PutWithProgress stores data with progress reporting.
|
||||
@@ -78,35 +65,7 @@ func (f *FileStorer) PutWithProgress(
|
||||
_ context.Context, key string, data io.Reader,
|
||||
_ int64, progress ProgressCallback,
|
||||
) error {
|
||||
path := f.fullPath(key)
|
||||
|
||||
// Create parent directories
|
||||
dir := filepath.Dir(path)
|
||||
|
||||
err := f.fs.MkdirAll(dir, storageDirPerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating directories: %w", err)
|
||||
}
|
||||
|
||||
file, err := f.fs.Create(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating file: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = file.Close() }()
|
||||
|
||||
// Wrap with progress tracking
|
||||
pw := &progressWriter{
|
||||
writer: file,
|
||||
callback: progress,
|
||||
}
|
||||
|
||||
_, err = io.Copy(pw, data)
|
||||
if err != nil {
|
||||
return fmt.Errorf("writing file: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
return f.writeAtomic(key, data, progress)
|
||||
}
|
||||
|
||||
// Get retrieves data from the specified key.
|
||||
@@ -188,7 +147,7 @@ func (f *FileStorer) List(ctx context.Context, prefix string) ([]string, error)
|
||||
default:
|
||||
}
|
||||
|
||||
if !info.IsDir() {
|
||||
if !info.IsDir() && !strings.HasSuffix(info.Name(), tempSuffix) {
|
||||
// Convert back to key (relative path from basePath)
|
||||
relPath, err := filepath.Rel(f.basePath, path)
|
||||
if err != nil {
|
||||
@@ -245,7 +204,7 @@ func (f *FileStorer) ListStream(ctx context.Context, prefix string) <-chan Objec
|
||||
return nil //nolint:nilerr // continue walking despite errors
|
||||
}
|
||||
|
||||
if !info.IsDir() {
|
||||
if !info.IsDir() && !strings.HasSuffix(info.Name(), tempSuffix) {
|
||||
relPath, err := filepath.Rel(f.basePath, path)
|
||||
if err != nil {
|
||||
ch <- ObjectInfo{Err: fmt.Errorf("computing relative path: %w", err)}
|
||||
@@ -275,6 +234,72 @@ func (f *FileStorer) Info() Info {
|
||||
}
|
||||
}
|
||||
|
||||
// writeAtomic streams data into a temp file in the destination directory,
|
||||
// fsyncs it, and renames it onto the final key. The key therefore appears
|
||||
// only once the whole object has been durably written; a failure part-way
|
||||
// leaves a temp file (removed here on the failing path) rather than a
|
||||
// truncated object at the key.
|
||||
func (f *FileStorer) writeAtomic(
|
||||
key string, data io.Reader, progress ProgressCallback,
|
||||
) error {
|
||||
path := f.fullPath(key)
|
||||
dir := filepath.Dir(path)
|
||||
|
||||
err := f.fs.MkdirAll(dir, storageDirPerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating directories: %w", err)
|
||||
}
|
||||
|
||||
tmp, err := afero.TempFile(f.fs, dir, filepath.Base(path)+"-*"+tempSuffix)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating temp file: %w", err)
|
||||
}
|
||||
|
||||
tmpPath := tmp.Name()
|
||||
|
||||
// Remove the temp file unless the rename below claims it. On the success
|
||||
// path renamed is true, so the deferred Close and Remove are harmless
|
||||
// no-ops on a name that no longer exists.
|
||||
renamed := false
|
||||
|
||||
defer func() {
|
||||
_ = tmp.Close()
|
||||
|
||||
if !renamed {
|
||||
_ = f.fs.Remove(tmpPath)
|
||||
}
|
||||
}()
|
||||
|
||||
var w io.Writer = tmp
|
||||
if progress != nil {
|
||||
w = &progressWriter{writer: tmp, callback: progress}
|
||||
}
|
||||
|
||||
_, err = io.Copy(w, data)
|
||||
if err != nil {
|
||||
return fmt.Errorf("writing file: %w", err)
|
||||
}
|
||||
|
||||
err = tmp.Sync()
|
||||
if err != nil {
|
||||
return fmt.Errorf("syncing temp file: %w", err)
|
||||
}
|
||||
|
||||
err = tmp.Close()
|
||||
if err != nil {
|
||||
return fmt.Errorf("closing temp file: %w", err)
|
||||
}
|
||||
|
||||
err = f.fs.Rename(tmpPath, path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("renaming temp file: %w", err)
|
||||
}
|
||||
|
||||
renamed = true
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// fullPath returns the full filesystem path for a key.
|
||||
func (f *FileStorer) fullPath(key string) string {
|
||||
return filepath.Join(f.basePath, key)
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
package storage_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/storage"
|
||||
)
|
||||
|
||||
// errStreamInterrupted stands in for an upload cut off mid-stream.
|
||||
var errStreamInterrupted = errors.New("connection reset mid-upload")
|
||||
|
||||
// failingReader yields its data once, then fails.
|
||||
type failingReader struct {
|
||||
data []byte
|
||||
done bool
|
||||
}
|
||||
|
||||
func (r *failingReader) Read(p []byte) (int, error) {
|
||||
if r.done {
|
||||
return 0, errStreamInterrupted
|
||||
}
|
||||
|
||||
n := copy(p, r.data)
|
||||
r.done = true
|
||||
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// TestFileStorer_InterruptedWriteLeavesNoTrustedObject checks that a write
|
||||
// cut off mid-stream leaves nothing at the destination key, so a later run
|
||||
// cannot Stat a truncated object and trust it as a complete blob.
|
||||
func TestFileStorer_InterruptedWriteLeavesNoTrustedObject(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f, err := storage.NewFileStorer(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStorer: %v", err)
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
key := "blobs/aa/bb/aabbccddeeff"
|
||||
|
||||
err = f.PutWithProgress(ctx, key, &failingReader{data: []byte("partial")}, 4096, nil)
|
||||
if err == nil {
|
||||
t.Fatal("expected the interrupted write to fail, got nil")
|
||||
}
|
||||
|
||||
_, err = f.Stat(ctx, key)
|
||||
if !errors.Is(err, storage.ErrNotFound) {
|
||||
t.Fatalf("expected key absent after interrupted write, got Stat err %v", err)
|
||||
}
|
||||
|
||||
keys, err := f.List(ctx, "blobs/")
|
||||
if err != nil {
|
||||
t.Fatalf("List: %v", err)
|
||||
}
|
||||
|
||||
if len(keys) != 0 {
|
||||
t.Fatalf("expected no keys listed after interrupted write, got %v", keys)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFileStorer_ListSkipsPartialFiles checks that a leftover temp file (the
|
||||
// storage layer names them with a ".partial" suffix) is never surfaced as a
|
||||
// key by List or ListStream.
|
||||
func TestFileStorer_ListSkipsPartialFiles(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
base := t.TempDir()
|
||||
|
||||
f, err := storage.NewFileStorer(base)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileStorer: %v", err)
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
realKey := "blobs/aa/bb/aabbccddeeff"
|
||||
|
||||
err = f.Put(ctx, realKey, strings.NewReader("blob-bytes"))
|
||||
if err != nil {
|
||||
t.Fatalf("Put: %v", err)
|
||||
}
|
||||
|
||||
// A stray temp file, as an interrupted write would leave behind.
|
||||
leftover := filepath.Join(base, "blobs/aa/bb/aabbccddeeff-123456.partial")
|
||||
|
||||
err = os.WriteFile(leftover, []byte("half"), 0o600)
|
||||
if err != nil {
|
||||
t.Fatalf("writing leftover temp file: %v", err)
|
||||
}
|
||||
|
||||
keys, err := f.List(ctx, "blobs/")
|
||||
if err != nil {
|
||||
t.Fatalf("List: %v", err)
|
||||
}
|
||||
|
||||
if len(keys) != 1 || keys[0] != realKey {
|
||||
t.Fatalf("List should return only the real key, got %v", keys)
|
||||
}
|
||||
|
||||
var streamed []string
|
||||
|
||||
for obj := range f.ListStream(ctx, "blobs/") {
|
||||
if obj.Err != nil {
|
||||
t.Fatalf("ListStream: %v", obj.Err)
|
||||
}
|
||||
|
||||
streamed = append(streamed, obj.Key)
|
||||
}
|
||||
|
||||
if len(streamed) != 1 || streamed[0] != realKey {
|
||||
t.Fatalf("ListStream should return only the real key, got %v", streamed)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
package vaultik //nolint:testpackage // exercises unexported count helpers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/vaultik/internal/database"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
)
|
||||
|
||||
// TestTableCountForReportSurfacesReadFailure is the regression guard for
|
||||
// the discarded-error bug: getTableCount for a table its query cannot
|
||||
// resolve must not silently become 0. A count that could not be read is
|
||||
// reported as unknown, which a reader can tell apart from an empty table.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestTableCountForReportSurfacesReadFailure(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
db, err := database.New(ctx, ":memory:")
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
|
||||
v := &Vaultik{DB: db}
|
||||
v.SetContext(ctx)
|
||||
|
||||
// A table present in the schema reads as a real count.
|
||||
blobs := v.tableCountForReport("blobs")
|
||||
require.NotNil(t, blobs, "an existing table must read as a real count")
|
||||
assert.Equal(t, int64(0), *blobs)
|
||||
|
||||
// A syntactically valid name the sanitizer accepts but whose table
|
||||
// the query cannot resolve is the exact shape #96 describes: a
|
||||
// would-be loud failure that used to be discarded into a 0.
|
||||
_, err = v.getTableCount("snapshots_missing")
|
||||
require.Error(t, err, "a query against a nonexistent table must fail")
|
||||
|
||||
missing := v.tableCountForReport("snapshots_missing")
|
||||
assert.Nil(t, missing, "a failed read is unknown, not a count")
|
||||
|
||||
// The rendered count for a failed read must say unknown, never 0.
|
||||
assert.Equal(t, countUnknown, countText(missing))
|
||||
assert.NotEqual(t, "0", countText(missing))
|
||||
}
|
||||
|
||||
// TestCountTextDistinguishesEmptyFromUnknown pins the distinction the
|
||||
// output has to preserve: 0 means the table was empty, "unknown" means
|
||||
// the count could not be read.
|
||||
func TestCountTextDistinguishesEmptyFromUnknown(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
zero := int64(0)
|
||||
seven := int64(7)
|
||||
|
||||
assert.Equal(t, "0", countText(&zero))
|
||||
assert.Equal(t, "7", countText(&seven))
|
||||
assert.Equal(t, countUnknown, countText(nil))
|
||||
}
|
||||
|
||||
// TestCountDiffUnknownWhenEitherSideUnknown checks that a delta computed
|
||||
// from an unreadable count is itself unknown rather than a plausible
|
||||
// number.
|
||||
func TestCountDiffUnknownWhenEitherSideUnknown(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
before := int64(10)
|
||||
after := int64(3)
|
||||
|
||||
require.NotNil(t, countDiff(&before, &after))
|
||||
assert.Equal(t, int64(7), *countDiff(&before, &after))
|
||||
|
||||
assert.Nil(t, countDiff(nil, &after), "unknown before yields unknown delta")
|
||||
assert.Nil(t, countDiff(&before, nil), "unknown after yields unknown delta")
|
||||
assert.Nil(t, countDiff(nil, nil))
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -1540,12 +1541,17 @@ func (v *Vaultik) outputRemoveJSON(result *RemoveResult) error {
|
||||
return encoder.Encode(result)
|
||||
}
|
||||
|
||||
// PruneResult contains statistics about the prune operation
|
||||
// PruneResult contains statistics about the prune operation.
|
||||
// SnapshotsDeleted counts snapshots actually deleted. FilesDeleted,
|
||||
// ChunksDeleted, and BlobsDeleted are derived from before/after row
|
||||
// counts of the local index; each is nil when a count could not be read,
|
||||
// so an unreadable count is reported as unknown rather than silently
|
||||
// as 0.
|
||||
type PruneResult struct {
|
||||
SnapshotsDeleted int64
|
||||
FilesDeleted int64
|
||||
ChunksDeleted int64
|
||||
BlobsDeleted int64
|
||||
FilesDeleted *int64
|
||||
ChunksDeleted *int64
|
||||
BlobsDeleted *int64
|
||||
}
|
||||
|
||||
// PruneDatabase removes incomplete snapshots and orphaned files, chunks,
|
||||
@@ -1560,7 +1566,7 @@ func (v *Vaultik) PruneDatabase() (*PruneResult, error) {
|
||||
result := &PruneResult{}
|
||||
|
||||
// Snapshot counts before deletion of incompletes.
|
||||
snapshotCountBefore, _ := v.getTableCount("snapshots")
|
||||
snapshotCountBefore := v.tableCountForReport("snapshots")
|
||||
|
||||
// First, delete any incomplete snapshots
|
||||
incompleteSnapshots, err := v.Repositories.Snapshots.GetIncompleteSnapshots(v.ctx)
|
||||
@@ -1575,9 +1581,9 @@ func (v *Vaultik) PruneDatabase() (*PruneResult, error) {
|
||||
}
|
||||
|
||||
// Get counts before cleanup for reporting
|
||||
fileCountBefore, _ := v.getTableCount("files")
|
||||
chunkCountBefore, _ := v.getTableCount("chunks")
|
||||
blobCountBefore, _ := v.getTableCount("blobs")
|
||||
fileCountBefore := v.tableCountForReport("files")
|
||||
chunkCountBefore := v.tableCountForReport("chunks")
|
||||
blobCountBefore := v.tableCountForReport("blobs")
|
||||
|
||||
// Run the cleanup
|
||||
err = v.SnapshotManager.CleanupOrphanedData(v.ctx)
|
||||
@@ -1586,36 +1592,83 @@ func (v *Vaultik) PruneDatabase() (*PruneResult, error) {
|
||||
}
|
||||
|
||||
// Get counts after cleanup
|
||||
fileCountAfter, _ := v.getTableCount("files")
|
||||
chunkCountAfter, _ := v.getTableCount("chunks")
|
||||
blobCountAfter, _ := v.getTableCount("blobs")
|
||||
fileCountAfter := v.tableCountForReport("files")
|
||||
chunkCountAfter := v.tableCountForReport("chunks")
|
||||
blobCountAfter := v.tableCountForReport("blobs")
|
||||
|
||||
result.FilesDeleted = fileCountBefore - fileCountAfter
|
||||
result.ChunksDeleted = chunkCountBefore - chunkCountAfter
|
||||
result.BlobsDeleted = blobCountBefore - blobCountAfter
|
||||
result.FilesDeleted = countDiff(fileCountBefore, fileCountAfter)
|
||||
result.ChunksDeleted = countDiff(chunkCountBefore, chunkCountAfter)
|
||||
result.BlobsDeleted = countDiff(blobCountBefore, blobCountAfter)
|
||||
|
||||
log.Info("Local database prune complete",
|
||||
"incomplete_snapshots", result.SnapshotsDeleted,
|
||||
"orphaned_files", result.FilesDeleted,
|
||||
"orphaned_chunks", result.ChunksDeleted,
|
||||
"orphaned_blobs", result.BlobsDeleted,
|
||||
"orphaned_files", countText(result.FilesDeleted),
|
||||
"orphaned_chunks", countText(result.ChunksDeleted),
|
||||
"orphaned_blobs", countText(result.BlobsDeleted),
|
||||
)
|
||||
|
||||
snapshotCountAfter := snapshotCountBefore - result.SnapshotsDeleted
|
||||
// Snapshots remaining after removing the incomplete ones; unknown if
|
||||
// the pre-prune snapshot count could not be read.
|
||||
snapshotsRemain := countDiff(snapshotCountBefore, &result.SnapshotsDeleted)
|
||||
|
||||
v.UI.Completef("Pruned local index database.")
|
||||
v.UI.Detailf("Incomplete snapshots: %d removed (%d remain).",
|
||||
result.SnapshotsDeleted, snapshotCountAfter)
|
||||
v.UI.Detailf("Orphaned files: %d removed (%d remain).",
|
||||
result.FilesDeleted, fileCountAfter)
|
||||
v.UI.Detailf("Orphaned chunks: %d removed (%d remain).",
|
||||
result.ChunksDeleted, chunkCountAfter)
|
||||
v.UI.Detailf("Orphaned blobs: %d removed (%d remain).",
|
||||
result.BlobsDeleted, blobCountAfter)
|
||||
v.UI.Detailf("Incomplete snapshots: %s removed (%s remain).",
|
||||
countText(&result.SnapshotsDeleted), countText(snapshotsRemain))
|
||||
v.UI.Detailf("Orphaned files: %s removed (%s remain).",
|
||||
countText(result.FilesDeleted), countText(fileCountAfter))
|
||||
v.UI.Detailf("Orphaned chunks: %s removed (%s remain).",
|
||||
countText(result.ChunksDeleted), countText(chunkCountAfter))
|
||||
v.UI.Detailf("Orphaned blobs: %s removed (%s remain).",
|
||||
countText(result.BlobsDeleted), countText(blobCountAfter))
|
||||
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// countUnknown is what a count reads as when its query could not be run,
|
||||
// distinct from "0", which means the table really was empty.
|
||||
const countUnknown = "unknown"
|
||||
|
||||
// tableCountForReport returns the row count of a table for the prune
|
||||
// summary, or nil if the count could not be read. A read failure is
|
||||
// logged at warn — visible even under --json, which routes warnings to
|
||||
// stderr — and then rendered as unknown rather than silently becoming 0,
|
||||
// so a broken query is a visible failure instead of a plausible wrong
|
||||
// number.
|
||||
func (v *Vaultik) tableCountForReport(tableName string) *int64 {
|
||||
count, err := v.getTableCount(tableName)
|
||||
if err != nil {
|
||||
log.Warn("could not read table row count for prune summary",
|
||||
"table", tableName, "error", err)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
return &count
|
||||
}
|
||||
|
||||
// countDiff returns before-after, or nil if either count is unknown so
|
||||
// that an unreadable count does not collapse into a plausible delta.
|
||||
func countDiff(before, after *int64) *int64 {
|
||||
if before == nil || after == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
diff := *before - *after
|
||||
|
||||
return &diff
|
||||
}
|
||||
|
||||
// countText renders a count that may be unknown: nil (the read failed)
|
||||
// becomes "unknown", never "0", so a reader can tell an empty table from
|
||||
// one that could not be queried.
|
||||
func countText(count *int64) string {
|
||||
if count == nil {
|
||||
return countUnknown
|
||||
}
|
||||
|
||||
return strconv.FormatInt(*count, 10)
|
||||
}
|
||||
|
||||
// validTableNameRe matches table names containing only lowercase
|
||||
// alphanumeric characters and underscores.
|
||||
var validTableNameRe = regexp.MustCompile(`^[a-z0-9_]+$`)
|
||||
|
||||
Reference in New Issue
Block a user