Store and compare file mtimes in nanoseconds (closes #226)

The files table held mtime in whole seconds and the scanner compared
whole seconds. A file rewritten with its size unchanged and a new mtime
in the same second as the indexed one was treated as unchanged, and
every later snapshot restored the old content. mtime is now stored as
nanoseconds since the Unix epoch and compared at that precision.

A local index written before this change holds seconds in mtime: its
next backup re-chunks every file once, and a restore of a snapshot made
before it sets mtimes near 1970.

An mtime outside the years 1678 to 2262 does not fit in int64
nanoseconds; it is stored wrong and restored wrong.

Model: opus-5-5

README.md

@@ -352,9 +352,8 @@ may hold snapshots this host doesn't know about), which is what
 prune` invocation to run as a follow-up. Local row cleanup (files,
 chunks, blobs the snapshot was the last referrer for) runs
 automatically. If the destination store is unreachable, the local-DB
-removal still completes and a warning is emitted; run `vaultik snapshot
+removal still completes and a warning is emitted; rerun `vaultik prune`
-remove <snapshot-id>` again once the store is reachable to remove the
+once the store is reachable to finish remote cleanup. To wipe everything
-snapshot's metadata from it (`vaultik prune` does not). To wipe everything
 on the destination in one go, use `vaultik remote nuke --force`.
 * `--local-only`: Skip remote cleanup; only touch the local index
 * `--dry-run`: Show what would be deleted without deleting

TODO.md

@@ -26,22 +26,11 @@ the tag exists and is exercised; what is left is merging `next` to
   unchanged and a new mtime in the same second as the one in the index
   ([issue #226](https://git.eeqj.de/sneak/vaultik/issues/226)). The
   `files` table held mtime in whole seconds and the scanner compared
-  whole seconds, so every later snapshot kept the old content. A new
+  whole seconds, so every later snapshot kept the old content. `mtime`
-  `mtime_nsec` column holds the nanoseconds within the second that
+  now holds nanoseconds since the Unix epoch and is compared at that
-  `mtime` holds, and the scanner compares the full mtime. A local index
+  precision. A local index written before the change holds seconds
-  created before the change lacks the column and is rebuilt with
+  there, so its next backup re-chunks every file once; chunks it already
-  `vaultik database delete` and a full backup.
+  stored are not uploaded again.
-
-- 2026-10-06: Made `snapshot remove --json` write only its document to
-  stdout when the destination store cannot be reached
-  ([issue #251](https://git.eeqj.de/sneak/vaultik/issues/251)). Its
-  warning that the snapshot's metadata was left on the destination store
-  went to stdout ahead of the document, breaking `| jq` on a command that
-  exited 0. Under `--json` the warning now reaches stderr only, through
-  the logger. The warning, the README and the command's help said
-  `vaultik prune` would finish the cleanup, but `prune` never removes
-  snapshot metadata; they now say to run `vaultik snapshot remove` for the
-  snapshot again once the destination store is reachable.
 
 - 2026-10-06: Made the backup summary and the `snapshots` row count each
   file, byte and upload once

docs/DATAMODEL.md

@@ -36,8 +36,7 @@ Stores metadata about files in the filesystem being backed up.
 **Columns:**
 - `id` (TEXT PRIMARY KEY) - UUID for the file record
 - `path` (TEXT NOT NULL UNIQUE) - Absolute file path
-- `mtime` (INTEGER NOT NULL) - Modification time, whole seconds since the Unix epoch
+- `mtime` (INTEGER NOT NULL) - Modification time in nanoseconds since the Unix epoch
-- `mtime_nsec` (INTEGER NOT NULL) - Nanoseconds within that second, 0 to 999999999
 - `size` (INTEGER NOT NULL) - File size in bytes
 - `mode` (INTEGER NOT NULL) - Unix file permissions and type
 - `uid` (INTEGER NOT NULL) - User ID of file owner

internal/cli/entry_stdout_stderr_test.go

@@ -2,9 +2,7 @@ package cli //nolint:testpackage // shares hermeticConfig and the capture helper
 
 import (
 	"context"
-	"encoding/json"
 	"fmt"
-	"log/slog"
 	"os"
 	"path/filepath"
 	"strings"
@@ -89,45 +87,12 @@ func TestEntryJSONFailureIsReportedOnStderr(t *testing.T) {
 	}
 }
 
-// TestEntrySnapshotRemoveJSONWarningIsOnStderr runs `snapshot remove
+// writeUnusableDestinationConfig builds a config whose destination
-// --json` on a snapshot in the local index, against a destination
+// directory does not exist, which fails `remote info`, and whose local
-// directory that does not exist. The command removes the snapshot from
+// index is bound to another destination, which fails `prune` and
-// the local index and still exits 0. Its stdout must hold the document
+// `snapshot remove` (a missing destination alone only makes `snapshot
-// alone, with the warning about the destination store on stderr: the
+// remove` warn). Returns the config path.
-// command to run again once it is reachable, and the snapshot's ID in
+func writeUnusableDestinationConfig(t *testing.T) string {
-// the record's snapshot_id field.
-//
-//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
-func TestEntrySnapshotRemoveJSONWarningIsOnStderr(t *testing.T) {
-	configPath, indexPath := writeMissingDestinationConfig(t)
-	seedStaleSnapshotRecord(t, indexPath)
-
-	code, stdout, stderr := runEntry(t, flagConfig, configPath,
-		cmdSnapshot, cmdRemove, stalePruneSnapshotID, flagJSON)
-
-	require.Equal(t, 0, code)
-	requireExactlyOneJSONDocument(t, stdout)
-
-	// stderr is a pipe here, so the logger writes one JSON record a line.
-	var warning map[string]any
-
-	for line := range strings.Lines(stderr) {
-		if strings.Contains(line,
-			"Could not remove snapshot metadata from remote storage") {
-			require.NoError(t, json.Unmarshal([]byte(line), &warning))
-		}
-	}
-
-	require.NotNil(t, warning, "the warning must reach stderr")
-	assert.Contains(t, warning[slog.MessageKey],
-		"run 'vaultik snapshot remove' with the snapshot's ID again")
-	assert.Equal(t, stalePruneSnapshotID, warning["snapshot_id"])
-}
-
-// writeMissingDestinationConfig builds a config whose destination
-// directory does not exist. Returns the config path and the path of
-// its local index, which is not created here.
-func writeMissingDestinationConfig(t *testing.T) (string, string) {
 	t.Helper()
 
 	dir := t.TempDir()
@@ -149,19 +114,6 @@ func writeMissingDestinationConfig(t *testing.T) (string, string) {
 	xdg.Reload()
 	t.Cleanup(xdg.Reload)
 
-	return configPath, indexPath
-}
-
-// writeUnusableDestinationConfig builds a config whose destination
-// directory does not exist, which fails `remote info`, and whose local
-// index is bound to another destination, which fails `prune` and
-// `snapshot remove` (a missing destination alone only makes `snapshot
-// remove` warn). Returns the config path.
-func writeUnusableDestinationConfig(t *testing.T) string {
-	t.Helper()
-
-	configPath, indexPath := writeMissingDestinationConfig(t)
-
 	ctx := context.Background()
 
 	db, err := database.New(ctx, indexPath)
@@ -170,7 +122,7 @@ func writeUnusableDestinationConfig(t *testing.T) string {
 	defer func() { require.NoError(t, db.Close()) }()
 
 	require.NoError(t, database.NewRepositories(db).LocalMeta.Set(ctx,
-		database.LocalMetaKeyStorageURL, "file://"+t.TempDir()))
+		database.LocalMetaKeyStorageURL, "file://"+filepath.Join(dir, "other")))
 
 	return configPath
 }

internal/cli/snapshot.go

@@ -258,9 +258,8 @@ Use --local-only to skip the remote half (e.g. when you want to forget a
 snapshot locally without touching the destination store).
 
 If the remote is unreachable, the local-database removal still completes
-and a warning is emitted; run 'vaultik snapshot remove <snapshot-id>' again
+and a warning is emitted; rerun 'vaultik prune' once the destination store
-once the destination store is reachable to remove the snapshot's metadata
+is reachable to finish remote cleanup.
-from it ('vaultik prune' does not).
 
 To wipe the entire destination store and start over, use 'vaultik remote
 nuke --force' — it is the single supported entry point for that.`,

internal/database/files.go

@@ -33,13 +33,11 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
 	}
 
 	query := `
-		INSERT INTO files
+		INSERT INTO files (id, path, source_path, mtime, size, mode, uid, gid, link_target)
-			(id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target)
+		VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
-		VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
 		ON CONFLICT(path) DO UPDATE SET
 			source_path = excluded.source_path,
 			mtime = excluded.mtime,
-			mtime_nsec = excluded.mtime_nsec,
 			size = excluded.size,
 			mode = excluded.mode,
 			uid = excluded.uid,
@@ -56,19 +54,16 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
 	if tx != nil {
 		LogSQL("Execute", query,
 			file.ID.String(), file.Path.String(), file.SourcePath.String(),
-			file.MTime.Unix(), file.MTime.Nanosecond(),
+			file.MTime.UnixNano(), file.Size, file.Mode, file.UID, file.GID,
-			file.Size, file.Mode, file.UID, file.GID,
 			file.LinkTarget.String())
 		err = tx.QueryRowContext(ctx, query,
 			file.ID.String(), file.Path.String(), file.SourcePath.String(),
-			file.MTime.Unix(), file.MTime.Nanosecond(),
+			file.MTime.UnixNano(), file.Size, file.Mode, file.UID, file.GID,
-			file.Size, file.Mode, file.UID, file.GID,
 			file.LinkTarget.String()).Scan(&idStr)
 	} else {
 		err = r.db.QueryRowWithLog(ctx, query,
 			file.ID.String(), file.Path.String(), file.SourcePath.String(),
-			file.MTime.Unix(), file.MTime.Nanosecond(),
+			file.MTime.UnixNano(), file.Size, file.Mode, file.UID, file.GID,
-			file.Size, file.Mode, file.UID, file.GID,
 			file.LinkTarget.String()).Scan(&idStr)
 	}
 
@@ -89,7 +84,7 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
 // in the index.
 func (r *FileRepository) GetByPath(ctx context.Context, path string) (*File, error) {
 	query := `
-		SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
+		SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
 		FROM files
 		WHERE path = ?
 	`
@@ -109,7 +104,7 @@ func (r *FileRepository) GetByPath(ctx context.Context, path string) (*File, err
 // GetByID retrieves a file by its UUID
 func (r *FileRepository) GetByID(ctx context.Context, id types.FileID) (*File, error) {
 	query := `
-		SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
+		SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
 		FROM files
 		WHERE id = ?
 	`
@@ -132,7 +127,7 @@ func (r *FileRepository) GetByPathTx(
 	ctx context.Context, tx *sql.Tx, path string,
 ) (*File, error) {
 	query := `
-		SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
+		SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
 		FROM files
 		WHERE path = ?
 	`
@@ -163,14 +158,13 @@ func (r *FileRepository) ListModifiedSince(
 	ctx context.Context, since time.Time,
 ) ([]*File, error) {
 	query := `
-		SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
+		SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
 		FROM files
-		WHERE (mtime, mtime_nsec) >= (?, ?)
+		WHERE mtime >= ?
 		ORDER BY path
 	`
 
-	rows, err := r.db.conn.QueryContext(ctx, query,
+	rows, err := r.db.conn.QueryContext(ctx, query, since.UnixNano())
-		since.Unix(), since.Nanosecond())
 	if err != nil {
 		return nil, fmt.Errorf("querying files: %w", err)
 	}
@@ -245,7 +239,7 @@ func (r *FileRepository) ListUnderPath(
 
 	// LIKE would ignore ASCII case and treat _ and % in path as wildcards.
 	query := `
-		SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
+		SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
 		FROM files
 		WHERE path = ? OR substr(path, 1, length(?)) = ?
 		ORDER BY path
@@ -330,7 +324,7 @@ func (r *FileRepository) ListIDsWithChunksNotInUploadedBlobs(
 // ListAll returns all files in the database
 func (r *FileRepository) ListAll(ctx context.Context) ([]*File, error) {
 	query := `
-		SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
+		SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
 		FROM files
 		ORDER BY path
 	`
@@ -371,7 +365,7 @@ func (r *FileRepository) CreateBatch(
 	}
 
 	// Each files row binds this many SQL variables.
-	const fileCols = 10
+	const fileCols = 9
 
 	// Batch at 100 rows to be safe with SQLite's variable limit.
 	const batchSize = 100
@@ -382,7 +376,7 @@ func (r *FileRepository) CreateBatch(
 		batch := files[i:end]
 
 		query := `INSERT INTO files
-			(id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target)
+			(id, path, source_path, mtime, size, mode, uid, gid, link_target)
 			VALUES `
 
 		args := make([]any, 0, len(batch)*fileCols)
@@ -394,12 +388,11 @@ func (r *FileRepository) CreateBatch(
 				querySb325.WriteString(", ")
 			}
 
-			querySb325.WriteString("(?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")
+			querySb325.WriteString("(?, ?, ?, ?, ?, ?, ?, ?, ?)")
 
 			args = append(args,
 				f.ID.String(), f.Path.String(), f.SourcePath.String(),
-				f.MTime.Unix(), f.MTime.Nanosecond(),
+				f.MTime.UnixNano(), f.Size, f.Mode, f.UID, f.GID,
-				f.Size, f.Mode, f.UID, f.GID,
 				f.LinkTarget.String())
 		}
 
@@ -408,7 +401,6 @@ func (r *FileRepository) CreateBatch(
 		query += ` ON CONFLICT(path) DO UPDATE SET
 			source_path = excluded.source_path,
 			mtime = excluded.mtime,
-			mtime_nsec = excluded.mtime_nsec,
 			size = excluded.size,
 			mode = excluded.mode,
 			uid = excluded.uid,
@@ -468,7 +460,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
 	var (
 		file                          File
 		idStr, pathStr, sourcePathStr string
-		mtimeUnix, mtimeNsec          int64
+		mtimeUnixNano                 int64
 		linkTarget                    sql.NullString
 	)
 
@@ -476,8 +468,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
 		&idStr,
 		&pathStr,
 		&sourcePathStr,
-		&mtimeUnix,
+		&mtimeUnixNano,
-		&mtimeNsec,
 		&file.Size,
 		&file.Mode,
 		&file.UID,
@@ -496,7 +487,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
 	file.Path = types.FilePath(pathStr)
 	file.SourcePath = types.SourcePath(sourcePathStr)
 
-	file.MTime = time.Unix(mtimeUnix, mtimeNsec).UTC()
+	file.MTime = time.Unix(0, mtimeUnixNano).UTC()
 	if linkTarget.Valid {
 		file.LinkTarget = types.FilePath(linkTarget.String)
 	}

internal/database/files_test.go

@@ -252,58 +252,6 @@ func TestFileRepositorySymlink(t *testing.T) {
 	}
 }
 
-// An mtime after 2262 or before 1678 does not fit in int64 nanoseconds
-// since the epoch, and must still come back from the database unchanged.
-func TestFileRepositoryMTimeOutsideInt64NanosecondRange(t *testing.T) {
-	t.Parallel()
-
-	db, cleanup := setupTestDB(t)
-	defer cleanup()
-
-	ctx := context.Background()
-	repo := database.NewFileRepository(db)
-
-	mtimes := []time.Time{
-		time.Date(2300, time.January, 1, 0, 0, 0, 123456789, time.UTC),
-		time.Date(1601, time.January, 1, 0, 0, 0, 987654321, time.UTC),
-	}
-
-	for _, mtime := range mtimes {
-		created := &database.File{
-			Path:  types.FilePath("/created-" + mtime.Format(time.RFC3339Nano)),
-			MTime: mtime,
-		}
-
-		err := repo.Create(ctx, nil, created)
-		if err != nil {
-			t.Fatalf("failed to create file: %v", err)
-		}
-
-		batched := &database.File{
-			ID:    types.NewFileID(),
-			Path:  types.FilePath("/batched-" + mtime.Format(time.RFC3339Nano)),
-			MTime: mtime,
-		}
-
-		err = repo.CreateBatch(ctx, nil, []*database.File{batched})
-		if err != nil {
-			t.Fatalf("failed to batch create file: %v", err)
-		}
-
-		for _, path := range []types.FilePath{created.Path, batched.Path} {
-			retrieved, err := repo.GetByPath(ctx, path.String())
-			if err != nil {
-				t.Fatalf("failed to get file: %v", err)
-			}
-
-			if !retrieved.MTime.Equal(mtime) {
-				t.Errorf("%s: mtime got %v, want %v",
-					path, retrieved.MTime, mtime)
-			}
-		}
-	}
-}
-
 func TestFileRepositoryTransaction(t *testing.T) {
 	t.Parallel()
 

internal/database/schema/001.sql

@@ -6,8 +6,7 @@ CREATE TABLE IF NOT EXISTS files (
     id TEXT PRIMARY KEY,  -- UUID
     path TEXT NOT NULL UNIQUE,
     source_path TEXT NOT NULL DEFAULT '',  -- The source directory this file came from (for restore path stripping)
-    mtime INTEGER NOT NULL,  -- whole seconds since the Unix epoch
+    mtime INTEGER NOT NULL,  -- nanoseconds since the Unix epoch
-    mtime_nsec INTEGER NOT NULL,  -- nanoseconds within that second, 0 to 999999999
     size INTEGER NOT NULL,
     mode INTEGER NOT NULL,
     uid INTEGER NOT NULL,

internal/snapshot/scanner.go

@@ -1221,7 +1221,7 @@ func (s *Scanner) checkFileInMemory(
 
 	// Check if file has changed
 	if existingFile.Size != file.Size ||
-		!existingFile.MTime.Equal(file.MTime) ||
+		existingFile.MTime.UnixNano() != file.MTime.UnixNano() ||
 		existingFile.Mode != file.Mode ||
 		existingFile.UID != file.UID ||
 		existingFile.GID != file.GID {

internal/vaultik/snapshot.go

@@ -1110,12 +1110,6 @@ type RemoveResult struct {
 // just-removed snapshot left behind on the destination store.
 const pruneCommandHint = "vaultik prune"
 
-// snapshotRemoveCommandHint is the command suggested, with the
-// snapshot's ID, when a remove could not reach the destination store:
-// running it again removes the snapshot's metadata there, which
-// `vaultik prune` never does.
-const snapshotRemoveCommandHint = "vaultik snapshot remove"
-
 // RemoveSnapshot removes a snapshot from the local index database and,
 // unless LocalOnly is set, also strips the snapshot's metadata from the
 // destination store. Blobs are NOT touched: removing a snapshot's
@@ -1152,7 +1146,7 @@ func (v *Vaultik) RemoveSnapshot(
 	}
 
 	if !opts.LocalOnly {
-		result.RemoteRemoved = v.removeSnapshotRemote(snapshotID, opts)
+		result.RemoteRemoved = v.removeSnapshotRemote(snapshotID)
 	}
 
 	if v.SnapshotManager != nil {
@@ -1235,11 +1229,9 @@ func (v *Vaultik) confirmRemoveSnapshot(snapshotID string, opts *RemoveOptions)
 // removeSnapshotRemote strips the snapshot's metadata from the
 // destination store, warning and proceeding on failure: the local-DB
 // removal has already happened, so the user is told the remote half
-// didn't finish and to run `vaultik snapshot remove` for the snapshot
+// didn't finish and can retry with `vaultik prune` once the destination
-// again once the destination store is reachable (`vaultik prune` never
+// store is reachable. Returns true when the remote removal succeeded.
-// removes snapshot metadata). Returns true when the remote removal
+func (v *Vaultik) removeSnapshotRemote(snapshotID string) bool {
-// succeeded.
-func (v *Vaultik) removeSnapshotRemote(snapshotID string, opts *RemoveOptions) bool {
 	log.Info("Removing snapshot metadata from remote storage",
 		"snapshot_id", snapshotID)
 
@@ -1247,17 +1239,13 @@ func (v *Vaultik) removeSnapshotRemote(snapshotID string, opts *RemoveOptions) b
 
 	err := v.deleteRemoteSnapshotByKey(remoteKey)
 	if err != nil {
-		log.Warn("Could not remove snapshot metadata from remote storage; "+
+		log.Warn("Could not remove snapshot metadata from remote storage",
-			"run '"+snapshotRemoveCommandHint+"' with the snapshot's ID "+
+			"error", err)
-			"again once the remote is reachable",
-			"snapshot_id", snapshotID, "error", err)
 
-		// The UI writes to stdout, which under --json holds only the
+		if v.UI != nil {
-		// document; the log record above is the warning on stderr.
-		if v.UI != nil && !opts.JSON {
 			v.UI.Warningf("Could not remove snapshot metadata from remote: "+
-				"%v. Run '%s %s' again once the remote is reachable.",
+				"%v. Run '%s' once the remote is reachable to finish cleanup.",
-				err, snapshotRemoveCommandHint, snapshotID)
+				err, pruneCommandHint)
 		}
 
 		return false