sneak/vaultik
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

snapshot create fails on any host without the sqlite3 CLI: VACUUM shells out instead of using the driver #120

New Issue
Closed
opened 2026-09-21 09:19:43 +02:00 by clawbot · 1 comment
clawbot commented 2026-09-21 09:19:43 +02:00
Collaborator
Copy Link
Copy Source

snapshot create uploads every blob and then shells out to a sqlite3 binary to VACUUM the per-snapshot metadata database before encrypting and uploading it (internal/snapshot/snapshot.go:672-683, exec.CommandContext(ctx, "sqlite3", dbPath, "VACUUM;")). On a host without that binary the backup fails at the very end, after the expensive work, with running VACUUM: exec: "sqlite3": executable file not found in $PATH.

The README advertises a pure-Go binary installed with go install, and nothing in the README or config init output says the sqlite CLI must be present. script/bootstrap:117 installs it "for the test suite", which is why CI is green; on this host, which has no sqlite3, make check on next fails TestRestoreLocalityAndReadAt and TestRestoreSweeperEvictsBlobs with exactly that error. So the gate depends on the environment, and a fresh go install user hits a defect the suite cannot see.

The VACUUM itself is right and stays (the comment explains why: deleted pages must not be uploaded). It just has to run through the driver.

Definition of done

  1. vacuumDatabase executes VACUUM through the open modernc.org/sqlite connection (db.ExecContext(ctx, "VACUUM"), outside any transaction) and os/exec is no longer imported by internal/snapshot. The existing comment on why VACUUM matters is kept.
  2. The upload path still uploads the vacuumed file: whatever ordering the current code uses (close, vacuum, reopen, or vacuum before close) is preserved or made correct, and a test asserts the uploaded database contains no pages from deleted rows (for example: insert rows, delete them, vacuum, and assert the file shrank or that a byte pattern from the deleted rows is absent).
  3. script/bootstrap no longer installs the sqlite3 CLI and its comment is removed; grep -rn sqlite3 outside of Go import paths and driver names returns nothing.
  4. make check is green on a host with no sqlite3 on PATH. State in the PR that it was run that way (PATH without the binary, or command -v sqlite3 returning nothing).
  5. No other behavior change. Landing commit title ends (closes #N).

Model: fable-5-1

`snapshot create` uploads every blob and then shells out to a `sqlite3` binary to VACUUM the per-snapshot metadata database before encrypting and uploading it (`internal/snapshot/snapshot.go:672-683`, `exec.CommandContext(ctx, "sqlite3", dbPath, "VACUUM;")`). On a host without that binary the backup fails at the very end, after the expensive work, with `running VACUUM: exec: "sqlite3": executable file not found in $PATH`. The README advertises a pure-Go binary installed with `go install`, and nothing in the README or `config init` output says the sqlite CLI must be present. `script/bootstrap:117` installs it "for the test suite", which is why CI is green; on this host, which has no `sqlite3`, `make check` on `next` fails `TestRestoreLocalityAndReadAt` and `TestRestoreSweeperEvictsBlobs` with exactly that error. So the gate depends on the environment, and a fresh `go install` user hits a defect the suite cannot see. The VACUUM itself is right and stays (the comment explains why: deleted pages must not be uploaded). It just has to run through the driver. ## Definition of done 1. `vacuumDatabase` executes `VACUUM` through the open `modernc.org/sqlite` connection (`db.ExecContext(ctx, "VACUUM")`, outside any transaction) and `os/exec` is no longer imported by `internal/snapshot`. The existing comment on why VACUUM matters is kept. 2. The upload path still uploads the vacuumed file: whatever ordering the current code uses (close, vacuum, reopen, or vacuum before close) is preserved or made correct, and a test asserts the uploaded database contains no pages from deleted rows (for example: insert rows, delete them, vacuum, and assert the file shrank or that a byte pattern from the deleted rows is absent). 3. `script/bootstrap` no longer installs the `sqlite3` CLI and its comment is removed; `grep -rn sqlite3` outside of Go import paths and driver names returns nothing. 4. `make check` is green on a host with no `sqlite3` on `PATH`. State in the PR that it was run that way (`PATH` without the binary, or `command -v sqlite3` returning nothing). 5. No other behavior change. Landing commit title ends ` (closes #N)`. Model: fable-5-1
clawbot added this to the 1.0.0 milestone 2026-09-21 09:19:43 +02:00
clawbot commented 2026-09-21 15:12:09 +02:00
Author
Collaborator
Copy Link
Copy Source

Fixed in #136.

vacuumDatabase now runs VACUUM through the modernc.org/sqlite driver (open, ExecWithLog(ctx, "VACUUM") outside any transaction, close) instead of shelling out; internal/snapshot no longer imports os/exec. Closing the connection checkpoints the WAL into the main file that is compressed and uploaded, so the uploaded database is the vacuumed one; a new test asserts a deleted-row byte pattern is absent and the file shrank. script/bootstrap and the Dockerfile test image no longer install the CLI, and a repo-wide grep for sqlite3 outside the Go driver name is empty. make check is green with no sqlite3 on PATH (command -v sqlite3 printed nothing).

Model: opus-4-8

Fixed in https://git.eeqj.de/sneak/vaultik/pulls/136. `vacuumDatabase` now runs `VACUUM` through the `modernc.org/sqlite` driver (open, `ExecWithLog(ctx, "VACUUM")` outside any transaction, close) instead of shelling out; `internal/snapshot` no longer imports `os/exec`. Closing the connection checkpoints the WAL into the main file that is compressed and uploaded, so the uploaded database is the vacuumed one; a new test asserts a deleted-row byte pattern is absent and the file shrank. `script/bootstrap` and the `Dockerfile` test image no longer install the CLI, and a repo-wide grep for `sqlite3` outside the Go driver name is empty. `make check` is green with no `sqlite3` on `PATH` (`command -v sqlite3` printed nothing). Model: opus-4-8
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
critical
Extremely important: data loss, a security hole, production down, or basic use impossible
 merge-ready
All checks pass, reviewed, rebased, ready for merge
 needs-checks
make check does not pass cleanly
 needs-rebase
PR has merge conflicts or is behind main
 needs-review
Code review not yet done
 needs-rework
Code review found issues to fix
No labels
Milestone
No items
No Milestone 1.0.0
Projects
Clear projects
No projects
Assignees
Clear assignees
clawbot sneak (Jeffrey Paul)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/vaultik#120