From aab6a87f8ca95f78cbdb3aceba745cb33708d7be Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Mon, 21 Sep 2026 21:58:37 +0200 Subject: [PATCH] Reconcile the schema/migration docs with the code (closes #68) Four documents told different stories about the database schema. docs/DATAMODEL.md now owns the explanation and separates two things: the policy, which is unchanged (no supported upgrade path between versions; delete the local index with vaultik database delete and back up again), and the schema bootstrap that does exist (numbered files in internal/database/schema applied to a fresh database and recorded in schema_migrations). README.md and AGENTS.md are reworded to match and link there. AGENTS.md names the real file to edit, internal/database/schema/001.sql, and notes that the pre-1.0 disposability clause expires on tagging. No code changed. Judgement call: REPO_POLICIES.md still names a different schema file; it is cross-project policy and was left alone. model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge) --- AGENTS.md | 11 ++++++++--- README.md | 20 +++++++++++++------- docs/DATAMODEL.md | 29 ++++++++++++++++++++++++----- 3 files changed, 45 insertions(+), 15 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 2dc6fc7..66b9c69 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -104,7 +104,12 @@ Version: 2025-06-08 13. Pre-1.0: NEVER write database migrations. There are no live databases anywhere — every user's local index can be rebuilt from a fresh full - backup. When the schema changes, just change `schema.sql` (and any code - that touches the affected tables). The local index is disposable until - 1.0 ships and is tagged. + backup. To change the schema, edit `internal/database/schema/001.sql` + (and any code that touches the affected tables) directly; do not add new + numbered schema files. Those numbered files and the `schema_migrations` + table they populate only bootstrap a fresh database — they are not an + upgrade path. The local index is disposable until 1.0 ships and is + tagged; once 1.0 is tagged that clause expires and the question of + upgrading existing indexes returns. See [`docs/DATAMODEL.md`](docs/DATAMODEL.md) + for the full explanation. diff --git a/README.md b/README.md index b411fd4..8b1702f 100644 --- a/README.md +++ b/README.md @@ -457,9 +457,13 @@ Key fields: sequentially. Restore speed is bound by single-stream throughput. * **Device nodes, named pipes, and sockets are silently skipped.** Only regular files, directories, and symlinks are backed up. -* **No database migrations.** If the local SQLite schema changes between - versions, delete the local database (`vaultik database delete`) and run - a full backup. Remote storage is unaffected. +* **No upgrade path between versions.** There is no supported way to carry + an existing local index across a schema change; if the local SQLite + schema changes between versions, delete the local database (`vaultik + database delete`) and run a full backup. Remote storage is unaffected. + (The binary does embed numbered schema files and a `schema_migrations` + table to bootstrap a fresh database — see [`docs/DATAMODEL.md`](docs/DATAMODEL.md) + — but that is not an upgrade path.) * **Files that change during backup may be inconsistent.** There is no filesystem snapshot or freeze. If a file is modified between the scan and chunk phases, the backed-up copy may reflect a partial write. @@ -529,10 +533,12 @@ priority. another host" workflow works but isn't documented as a first-class operation in this README. Worth a dedicated section once it's settled. -* **Schema migrations.** Currently nonexistent — pre-1.0 schema - changes are handled by `vaultik database delete` plus a full - re-scan. Post-1.0 we'll need a migration story to keep existing - index databases usable across upgrades. +* **Cross-version schema upgrades.** There is no upgrade path between + released versions — pre-1.0 schema changes are handled by `vaultik + database delete` plus a full re-scan (see + [`docs/DATAMODEL.md`](docs/DATAMODEL.md)). Post-1.0 we'll need a + migration story to keep existing index databases usable across + upgrades. * **Storage backend coverage tests.** S3, file://, and rclone:// all share the Storer interface but the rclone path is the least exercised in CI. diff --git a/docs/DATAMODEL.md b/docs/DATAMODEL.md index 14f2435..bde2012 100644 --- a/docs/DATAMODEL.md +++ b/docs/DATAMODEL.md @@ -5,11 +5,30 @@ Vaultik uses a local SQLite database to track file metadata, chunk mappings, and blob associations during the backup process. This database serves as an index for incremental backups and enables efficient deduplication. **Important Notes:** -- **No Migration Support (pre-1.0)**: Vaultik does not support database schema - migrations. The local index is treated as disposable — if the schema changes, - delete the local SQLite database (`vaultik database delete`) and run a full - backup. The remote storage is unaffected; the new index will re-deduplicate - against existing remote blobs. + +This section is the authoritative explanation of the schema/migration story; +other documents (the README and `AGENTS.md`) link here. + +- **No upgrade path between versions (pre-1.0)**: Vaultik has no supported way to + carry an existing local index across a schema change. The index is disposable + — if the on-disk schema changes between versions, delete the local SQLite + database (`vaultik database delete`) and run a full backup. Remote storage is + unaffected; the new index re-deduplicates against existing remote blobs. This + is the standing project policy, and it is separate from the schema bootstrap + described next. +- **Schema bootstrap**: a fresh database is populated from numbered SQL files + embedded in the binary under `internal/database/schema/`. `000.sql` creates the + `schema_migrations` table; `001.sql` creates the application tables. On opening + a database the code applies each numbered file that has not yet run and records + its version in `schema_migrations`. This bootstraps a new database; it does not + upgrade an existing one between released versions. +- **Changing the schema (pre-1.0)**: edit `internal/database/schema/001.sql` (and + the code that touches the affected tables) directly. Do not add new numbered + files — there is no installed base to migrate. +- **Disposability expires at 1.0**: the index is treated as disposable only until + 1.0 ships and is tagged. Once 1.0 is tagged that clause expires and the + question of upgrading existing indexes returns. It is deliberately left open + here. - **Version Compatibility**: In rare cases, you may need to use the same version of Vaultik to restore a backup as was used to create it. This ensures compatibility with the metadata format stored in S3.