Document that migrations are supported and none are added before 1.0 (closes #68) #205
@@ -102,14 +102,12 @@ Version: 2025-06-08
|
|||||||
build files are acceptable in the root, but source code and other files
|
build files are acceptable in the root, but source code and other files
|
||||||
should be organized in appropriate subdirectories.
|
should be organized in appropriate subdirectories.
|
||||||
|
|
||||||
13. Pre-1.0: NEVER write database migrations. There are no live databases
|
13. Pre-1.0: NEVER add a database migration. Migrations are supported, but
|
||||||
anywhere — every user's local index can be rebuilt from a fresh full
|
nothing is installed anywhere yet, so there is nothing to migrate. To
|
||||||
backup. To change the schema, edit `internal/database/schema/001.sql`
|
change the schema, edit `internal/database/schema/001.sql` (and any
|
||||||
(and any code that touches the affected tables) directly; do not add new
|
code that touches the affected tables) directly. After 1.0, each schema
|
||||||
numbered schema files. Those numbered files and the `schema_migrations`
|
change is a new numbered file in that directory and a released file is
|
||||||
table they populate only bootstrap a fresh database — they are not an
|
never edited; an existing local database is then migrated when vaultik
|
||||||
upgrade path. The local index is disposable until 1.0 ships and is
|
is updated. See
|
||||||
tagged; once 1.0 is tagged that clause expires and the question of
|
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations).
|
||||||
upgrading existing indexes returns. See [`docs/DATAMODEL.md`](docs/DATAMODEL.md)
|
|
||||||
for the full explanation.
|
|
||||||
|
|
||||||
|
|||||||
@@ -30,7 +30,7 @@ Read the rules in AGENTS.md and follow them.
|
|||||||
done provided to you in the initial instruction. Don't do part or most of
|
done provided to you in the initial instruction. Don't do part or most of
|
||||||
the work, do all of the work until the criteria for done are met.
|
the work, do all of the work until the criteria for done are met.
|
||||||
|
|
||||||
* We do not need to support migrations; schema upgrades can be handled by
|
* We do not add migrations before 1.0; schema upgrades can be handled by
|
||||||
deleting the local state file and doing a full backup to re-create it.
|
deleting the local state file and doing a full backup to re-create it.
|
||||||
|
|
||||||
* When testing on a 2.5Gbit/s ethernet to an s3 server backed by 2000MB/sec SSD,
|
* When testing on a 2.5Gbit/s ethernet to an s3 server backed by 2000MB/sec SSD,
|
||||||
|
|||||||
@@ -559,13 +559,13 @@ complete annotated example also lives in
|
|||||||
sequentially. Restore speed is bound by single-stream throughput.
|
sequentially. Restore speed is bound by single-stream throughput.
|
||||||
* **Device nodes, named pipes, and sockets are silently skipped.** Only
|
* **Device nodes, named pipes, and sockets are silently skipped.** Only
|
||||||
regular files, directories, and symlinks are backed up.
|
regular files, directories, and symlinks are backed up.
|
||||||
* **No upgrade path between versions.** There is no supported way to carry
|
* **Before 1.0, an update can make the local index unusable.** Vaultik
|
||||||
an existing local index across a schema change; if the local SQLite
|
supports schema migrations, but none are added before 1.0 because
|
||||||
schema changes between versions, delete the local database (`vaultik
|
there is no installed base yet. If an update leaves your local index
|
||||||
database delete`) and run a full backup. Remote storage is unaffected.
|
unusable, run `vaultik database delete` and then a full backup; remote
|
||||||
(The binary does embed numbered schema files and a `schema_migrations`
|
storage is unaffected. After 1.0, the local index is migrated when
|
||||||
table to bootstrap a fresh database — see [`docs/DATAMODEL.md`](docs/DATAMODEL.md)
|
vaultik is updated. See
|
||||||
— but that is not an upgrade path.)
|
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations).
|
||||||
* **Files that change during backup may be inconsistent.** There is no
|
* **Files that change during backup may be inconsistent.** There is no
|
||||||
filesystem snapshot or freeze. If a file is modified between the scan
|
filesystem snapshot or freeze. If a file is modified between the scan
|
||||||
and chunk phases, the backed-up copy may reflect a partial write.
|
and chunk phases, the backed-up copy may reflect a partial write.
|
||||||
@@ -631,12 +631,11 @@ priority.
|
|||||||
|
|
||||||
### infrastructure
|
### infrastructure
|
||||||
|
|
||||||
* **Cross-version schema upgrades.** There is no upgrade path between
|
* **Schema migrations after 1.0.** Migrations are supported, but none
|
||||||
released versions — pre-1.0 schema changes are handled by `vaultik
|
are added before 1.0 because there is no installed base yet. After
|
||||||
database delete` plus a full re-scan (see
|
1.0, each schema change is a new migration, so an existing local
|
||||||
[`docs/DATAMODEL.md`](docs/DATAMODEL.md)). Post-1.0 we'll need a
|
index is migrated when vaultik is updated (see
|
||||||
migration story to keep existing index databases usable across
|
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations)).
|
||||||
upgrades.
|
|
||||||
* **Storage backend coverage tests.** S3, file://, and rclone://
|
* **Storage backend coverage tests.** S3, file://, and rclone://
|
||||||
all share the Storer interface but the rclone path is the least
|
all share the Storer interface but the rclone path is the least
|
||||||
exercised in CI.
|
exercised in CI.
|
||||||
|
|||||||
+18
-23
@@ -6,33 +6,28 @@ Vaultik uses a local SQLite database to track file metadata, chunk mappings, and
|
|||||||
|
|
||||||
**Important Notes:**
|
**Important Notes:**
|
||||||
|
|
||||||
This section is the authoritative explanation of the schema/migration story;
|
|
||||||
other documents (the README and `AGENTS.md`) link here.
|
|
||||||
|
|
||||||
- **No upgrade path between versions (pre-1.0)**: Vaultik has no supported way to
|
|
||||||
carry an existing local index across a schema change. The index is disposable
|
|
||||||
— if the on-disk schema changes between versions, delete the local SQLite
|
|
||||||
database (`vaultik database delete`) and run a full backup. Remote storage is
|
|
||||||
unaffected; the new index re-deduplicates against existing remote blobs. This
|
|
||||||
is the standing project policy, and it is separate from the schema bootstrap
|
|
||||||
described next.
|
|
||||||
- **Schema bootstrap**: a fresh database is populated from numbered SQL files
|
|
||||||
embedded in the binary under `internal/database/schema/`. `000.sql` creates the
|
|
||||||
`schema_migrations` table; `001.sql` creates the application tables. On opening
|
|
||||||
a database the code applies each numbered file that has not yet run and records
|
|
||||||
its version in `schema_migrations`. This bootstraps a new database; it does not
|
|
||||||
upgrade an existing one between released versions.
|
|
||||||
- **Changing the schema (pre-1.0)**: edit `internal/database/schema/001.sql` (and
|
|
||||||
the code that touches the affected tables) directly. Do not add new numbered
|
|
||||||
files — there is no installed base to migrate.
|
|
||||||
- **Disposability expires at 1.0**: the index is treated as disposable only until
|
|
||||||
1.0 ships and is tagged. Once 1.0 is tagged that clause expires and the
|
|
||||||
question of upgrading existing indexes returns. It is deliberately left open
|
|
||||||
here.
|
|
||||||
- **Version Compatibility**: In rare cases, you may need to use the same version
|
- **Version Compatibility**: In rare cases, you may need to use the same version
|
||||||
of Vaultik to restore a backup as was used to create it. This ensures
|
of Vaultik to restore a backup as was used to create it. This ensures
|
||||||
compatibility with the metadata format stored in S3.
|
compatibility with the metadata format stored in S3.
|
||||||
|
|
||||||
|
## Schema Migrations
|
||||||
|
|
||||||
|
Vaultik supports schema migrations. They are the numbered SQL files in
|
||||||
|
`internal/database/schema/`, embedded in the binary: `000.sql` creates the
|
||||||
|
`schema_migrations` table, which records each migration that has run, and
|
||||||
|
`001.sql` creates the application tables. `database.New` opens a database and
|
||||||
|
applies, in order, every migration that database has not yet recorded.
|
||||||
|
|
||||||
|
**Before 1.0** no migrations are added, because nothing is installed anywhere
|
||||||
|
yet. A schema change edits `001.sql` (and the code that uses the affected
|
||||||
|
tables) directly. A local database created before the change has already
|
||||||
|
recorded `001.sql` as run, so it keeps the old schema and can become unusable;
|
||||||
|
`vaultik database delete` followed by a full backup rebuilds it.
|
||||||
|
|
||||||
|
**After 1.0** each schema change is a new numbered file, so an existing local
|
||||||
|
database is migrated the first time an updated vaultik opens it. A file that
|
||||||
|
has shipped in a release is never edited.
|
||||||
|
|
||||||
## Database Tables
|
## Database Tables
|
||||||
|
|
||||||
### 1. `files`
|
### 1. `files`
|
||||||
|
|||||||
Reference in New Issue
Block a user