From 714eea086b8e1b031135dfe14856818d45af7a06 Mon Sep 17 00:00:00 2001 From: sneak Date: Mon, 28 Sep 2026 18:09:48 +0000 Subject: [PATCH] Document that migrations are supported and none are added before 1.0 (closes #68) https://git.eeqj.de/sneak/vaultik/pulls/146 called the numbered schema files a bootstrap and said vaultik has no upgrade path between versions. That was wrong: they are migrations, and database.New applies any the database has not recorded. None are added before 1.0 because nothing is installed anywhere yet; after 1.0 each schema change is a new numbered file and the local database is migrated on update. docs/DATAMODEL.md gets a Schema Migrations section that owns the explanation. The README caveat and roadmap entry and AGENTS.md policy 13 say the same and link to it. CLAUDE.md, the owner's file, gets a one-line edit so it no longer says migrations are not needed. Model: opus-5-5 --- AGENTS.md | 18 ++++++++---------- CLAUDE.md | 2 +- README.md | 25 ++++++++++++------------- docs/DATAMODEL.md | 41 ++++++++++++++++++----------------------- 4 files changed, 39 insertions(+), 47 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 66b9c69..2f93504 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -102,14 +102,12 @@ Version: 2025-06-08 build files are acceptable in the root, but source code and other files should be organized in appropriate subdirectories. -13. Pre-1.0: NEVER write database migrations. There are no live databases - anywhere — every user's local index can be rebuilt from a fresh full - backup. To change the schema, edit `internal/database/schema/001.sql` - (and any code that touches the affected tables) directly; do not add new - numbered schema files. Those numbered files and the `schema_migrations` - table they populate only bootstrap a fresh database — they are not an - upgrade path. The local index is disposable until 1.0 ships and is - tagged; once 1.0 is tagged that clause expires and the question of - upgrading existing indexes returns. See [`docs/DATAMODEL.md`](docs/DATAMODEL.md) - for the full explanation. +13. Pre-1.0: NEVER add a database migration. Migrations are supported, but + nothing is installed anywhere yet, so there is nothing to migrate. To + change the schema, edit `internal/database/schema/001.sql` (and any + code that touches the affected tables) directly. After 1.0, each schema + change is a new numbered file in that directory and a released file is + never edited; an existing local database is then migrated when vaultik + is updated. See + [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations). diff --git a/CLAUDE.md b/CLAUDE.md index cec4213..7b5788e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -30,7 +30,7 @@ Read the rules in AGENTS.md and follow them. done provided to you in the initial instruction. Don't do part or most of the work, do all of the work until the criteria for done are met. -* We do not need to support migrations; schema upgrades can be handled by +* We do not add migrations before 1.0; schema upgrades can be handled by deleting the local state file and doing a full backup to re-create it. * When testing on a 2.5Gbit/s ethernet to an s3 server backed by 2000MB/sec SSD, diff --git a/README.md b/README.md index 9e8f862..6fb2d9c 100644 --- a/README.md +++ b/README.md @@ -559,13 +559,13 @@ complete annotated example also lives in sequentially. Restore speed is bound by single-stream throughput. * **Device nodes, named pipes, and sockets are silently skipped.** Only regular files, directories, and symlinks are backed up. -* **No upgrade path between versions.** There is no supported way to carry - an existing local index across a schema change; if the local SQLite - schema changes between versions, delete the local database (`vaultik - database delete`) and run a full backup. Remote storage is unaffected. - (The binary does embed numbered schema files and a `schema_migrations` - table to bootstrap a fresh database — see [`docs/DATAMODEL.md`](docs/DATAMODEL.md) - — but that is not an upgrade path.) +* **Before 1.0, an update can make the local index unusable.** Vaultik + supports schema migrations, but none are added before 1.0 because + there is no installed base yet. If an update leaves your local index + unusable, run `vaultik database delete` and then a full backup; remote + storage is unaffected. After 1.0, the local index is migrated when + vaultik is updated. See + [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations). * **Files that change during backup may be inconsistent.** There is no filesystem snapshot or freeze. If a file is modified between the scan and chunk phases, the backed-up copy may reflect a partial write. @@ -631,12 +631,11 @@ priority. ### infrastructure -* **Cross-version schema upgrades.** There is no upgrade path between - released versions — pre-1.0 schema changes are handled by `vaultik - database delete` plus a full re-scan (see - [`docs/DATAMODEL.md`](docs/DATAMODEL.md)). Post-1.0 we'll need a - migration story to keep existing index databases usable across - upgrades. +* **Schema migrations after 1.0.** Migrations are supported, but none + are added before 1.0 because there is no installed base yet. After + 1.0, each schema change is a new migration, so an existing local + index is migrated when vaultik is updated (see + [`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations)). * **Storage backend coverage tests.** S3, file://, and rclone:// all share the Storer interface but the rclone path is the least exercised in CI. diff --git a/docs/DATAMODEL.md b/docs/DATAMODEL.md index fdb86cf..7338c0d 100644 --- a/docs/DATAMODEL.md +++ b/docs/DATAMODEL.md @@ -6,33 +6,28 @@ Vaultik uses a local SQLite database to track file metadata, chunk mappings, and **Important Notes:** -This section is the authoritative explanation of the schema/migration story; -other documents (the README and `AGENTS.md`) link here. - -- **No upgrade path between versions (pre-1.0)**: Vaultik has no supported way to - carry an existing local index across a schema change. The index is disposable - — if the on-disk schema changes between versions, delete the local SQLite - database (`vaultik database delete`) and run a full backup. Remote storage is - unaffected; the new index re-deduplicates against existing remote blobs. This - is the standing project policy, and it is separate from the schema bootstrap - described next. -- **Schema bootstrap**: a fresh database is populated from numbered SQL files - embedded in the binary under `internal/database/schema/`. `000.sql` creates the - `schema_migrations` table; `001.sql` creates the application tables. On opening - a database the code applies each numbered file that has not yet run and records - its version in `schema_migrations`. This bootstraps a new database; it does not - upgrade an existing one between released versions. -- **Changing the schema (pre-1.0)**: edit `internal/database/schema/001.sql` (and - the code that touches the affected tables) directly. Do not add new numbered - files — there is no installed base to migrate. -- **Disposability expires at 1.0**: the index is treated as disposable only until - 1.0 ships and is tagged. Once 1.0 is tagged that clause expires and the - question of upgrading existing indexes returns. It is deliberately left open - here. - **Version Compatibility**: In rare cases, you may need to use the same version of Vaultik to restore a backup as was used to create it. This ensures compatibility with the metadata format stored in S3. +## Schema Migrations + +Vaultik supports schema migrations. They are the numbered SQL files in +`internal/database/schema/`, embedded in the binary: `000.sql` creates the +`schema_migrations` table, which records each migration that has run, and +`001.sql` creates the application tables. `database.New` opens a database and +applies, in order, every migration that database has not yet recorded. + +**Before 1.0** no migrations are added, because nothing is installed anywhere +yet. A schema change edits `001.sql` (and the code that uses the affected +tables) directly. A local database created before the change has already +recorded `001.sql` as run, so it keeps the old schema and can become unusable; +`vaultik database delete` followed by a full backup rebuilds it. + +**After 1.0** each schema change is a new numbered file, so an existing local +database is migrated the first time an updated vaultik opens it. A file that +has shipped in a release is never edited. + ## Database Tables ### 1. `files` -- 2.54.0