Build every database's schema from embedded SQL files (closes #56)
check / check (push) Waiting to run
check / check (push) Waiting to run
The main database, each webhook's event database and each archive file get their schema from SQL files under internal/db/migrations/, one directory per kind. Opening a file runs, in one transaction, each file that schema_migrations does not list. GORM stays for queries. AutoMigrate is gone, tests included. An event database holds only its own tables now, so the sweep that removed target rows copied into it is deleted. The association guard stays: without it a delivery write fails on the missing targets table. AutoMigrate also had GORM parse every model before a handle was shared. That is now done on its own: GORM races when two goroutines first use models that refer to each other. Model: opus-5-5
This commit is contained in:
@@ -1071,23 +1071,27 @@ beside it, or wait until there are none.
|
||||
|
||||
4. Start the service. The container gives the directory and the restored files
|
||||
to the `webhooker` user before the app starts, whoever restored them (see
|
||||
[Running with Docker](#running-with-docker)). `AutoMigrate` runs against each
|
||||
restored database as it is opened.
|
||||
[Running with Docker](#running-with-docker)). Each restored database gets any
|
||||
of its SQL migrations it has not had as it is opened (see
|
||||
[Upgrades](#upgrades)).
|
||||
|
||||
### Upgrades
|
||||
|
||||
**Back up before every upgrade.** Every start runs GORM `AutoMigrate`
|
||||
unconditionally against whatever files it finds:
|
||||
**Back up before every upgrade.** Each kind of database file has its schema in
|
||||
SQL files built into the binary, one directory per kind under
|
||||
`internal/db/migrations/`. In each, `000_migration.sql` creates the
|
||||
`schema_migrations` table, which records the number of every file the database
|
||||
has had, and `001_schema.sql` holds the whole schema. Opening a database runs,
|
||||
in one transaction, every file in its directory that `schema_migrations` does
|
||||
not list:
|
||||
|
||||
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
|
||||
`Entrypoint`, `Target`
|
||||
- each event database when it is lazily opened — `Event`, `Delivery`,
|
||||
`DeliveryResult`, `EventTotals`, `TargetTotals`, `EntrypointTotals`
|
||||
- each archive database on every open and reopen
|
||||
- `main/` for the main database, on connect — settings, users, API keys,
|
||||
webhooks, entrypoints and targets
|
||||
- `events/` for each event database, when it is lazily opened — events,
|
||||
deliveries, delivery results and their running totals
|
||||
- `archive/` for each archive database, on every open and reopen
|
||||
|
||||
There is no schema version table, no migration ledger, and no down migrations.
|
||||
Nothing in the files records which version wrote them, and no code path undoes a
|
||||
migration.
|
||||
There are no down migrations.
|
||||
|
||||
Upgrade procedure:
|
||||
|
||||
@@ -1112,12 +1116,9 @@ connects to. Check the `bindAddress` field of the startup log to see what a
|
||||
running process bound. See [Bind address](#bind-address).
|
||||
|
||||
**Downgrade is unsupported.** Once a newer binary has migrated the files there
|
||||
is no way to move them back. `AutoMigrate` is additive — it adds tables, columns
|
||||
and indexes and never drops or rewrites them — so an older binary will generally
|
||||
open migrated files and appear to work while writing against a schema it does
|
||||
not know about. The failure mode is silent divergence, not a startup error. The
|
||||
only supported way back to an older version is restoring the pre-upgrade backup,
|
||||
which discards everything received since that backup was taken.
|
||||
is no way to move them back. The only supported way back to an older version is
|
||||
restoring the pre-upgrade backup, which discards everything received since that
|
||||
backup was taken.
|
||||
|
||||
### Version stamping
|
||||
|
||||
@@ -1182,26 +1183,6 @@ at rest and restrict who can read them.
|
||||
- `events-{uuid}.db` and `archive-….db` hold the **full payload body and
|
||||
headers** of every event as received, including whatever the sending service
|
||||
put in them — tokens, signatures, personal data.
|
||||
- Event databases written before
|
||||
[issue #206](https://git.eeqj.de/sneak/webhooker/issues/206) was fixed **also
|
||||
contain target credentials**: a GORM association upsert on the delivery and
|
||||
retry write path copied `targets` rows, `config` included, into the
|
||||
per-webhook database. For a Slack target the `webhookUrl` _is_ the bearer
|
||||
credential, and an `http` target's URL can embed userinfo. This version never
|
||||
writes those rows; the first time it opens such a file it deletes them and
|
||||
vacuums the file, which removes the credential bytes rather than only
|
||||
unlinking the rows. Deleting alone would not: the bytes stay readable in the
|
||||
file's free pages until it is rewritten. The sweep is recorded in the file's
|
||||
`user_version` only once the vacuum returns, so a sweep that fails or is
|
||||
interrupted fails the open and is retried on the next one, and a file this
|
||||
version has opened without error holds no leaked rows and no recoverable bytes
|
||||
from them. On upgrade this rewrites each existing `events-{uuid}.db` once, on
|
||||
its first open. Two cases still hand over live delivery destinations: a backup
|
||||
taken from an older build, and a backup of a file this version has not yet
|
||||
opened successfully. Copies already made stay affected — the sweep only
|
||||
rewrites the file it opens, and freed blocks may persist in filesystem
|
||||
snapshots and on the underlying storage. Rotate any target credential that was
|
||||
in a backup you cannot account for.
|
||||
- `webhooker.db` stores target config **unencrypted**, tracked at
|
||||
[issue #212](https://git.eeqj.de/sneak/webhooker/issues/212), next to the
|
||||
session encryption key and the Argon2id password hashes.
|
||||
@@ -1919,8 +1900,9 @@ Deliveries still `pending` or `retrying` count in neither.
|
||||
|
||||
#### Event-tier indexes
|
||||
|
||||
These indexes on the per-webhook event databases are declared in the model tags,
|
||||
so `AutoMigrate` creates them on a fresh database:
|
||||
These indexes on the per-webhook event databases are created by
|
||||
`internal/db/migrations/events/001_schema.sql`, and declared in the model tags
|
||||
to match:
|
||||
|
||||
| Table | Columns | Serves |
|
||||
| ------------------ | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
@@ -1984,8 +1966,9 @@ and application state:
|
||||
- **Targets** — delivery destination configurations
|
||||
- **APIKeys** — programmatic access credentials
|
||||
|
||||
On first startup the main database is auto-migrated, a session encryption key is
|
||||
generated and stored, and an `admin` user is created.
|
||||
On first startup the main database is created from its SQL files (see
|
||||
[Upgrades](#upgrades)), a session encryption key is generated and stored, and an
|
||||
`admin` user is created.
|
||||
|
||||
**Per-Webhook Event Databases** (`{DATA_DIR}/events-{webhook_uuid}.db`) — each
|
||||
webhook gets its own dedicated SQLite file containing:
|
||||
@@ -1997,9 +1980,10 @@ webhook gets its own dedicated SQLite file containing:
|
||||
the above, the deliveries per target, and each entrypoint's last event, kept
|
||||
through retention
|
||||
|
||||
Per-webhook databases are created automatically when a webhook is created. They
|
||||
are managed by the `WebhookDBManager` component, which handles connection
|
||||
pooling, lazy opening, migrations, and cleanup.
|
||||
Per-webhook databases are created automatically when a webhook is created, from
|
||||
the SQL files in `internal/db/migrations/events/`. They hold no copy of the main
|
||||
database's tables. They are managed by the `WebhookDBManager` component, which
|
||||
handles connection pooling, lazy opening, migrations, and cleanup.
|
||||
|
||||
A per-webhook database that is missing or zero-length later means its webhook's
|
||||
events and pending deliveries are gone. The next time it is opened, an empty one
|
||||
@@ -2083,14 +2067,15 @@ the service again.
|
||||
After each write the archive handle is closed and reopened, debounced to at most
|
||||
once per second, so an operator can move the archive file away for offline
|
||||
archiving without stopping the service; a moved or removed archive file is
|
||||
recreated automatically on the next write. A zero-length archive file is written
|
||||
to as a new archive: SQLite opens it as an empty database, so it holds nothing
|
||||
to lose. An optional `expiry` in the target's config JSON (e.g.
|
||||
`{"expiry":"720h"}`) is validated when the target is created — the default
|
||||
(unset or the literal `never`) keeps rows forever — and rows older than the
|
||||
expiry are pruned each time the archive is (re)opened. An archive write failure
|
||||
is never silent success: the delivery records a failed attempt with the error
|
||||
and is marked failed.
|
||||
recreated automatically on the next write, from the SQL files in
|
||||
`internal/db/migrations/archive/`. A zero-length archive file is written to as a
|
||||
new archive: SQLite opens it as an empty database, so it holds nothing to lose.
|
||||
An optional `expiry` in the target's config JSON (e.g. `{"expiry":"720h"}`) is
|
||||
validated when the target is created — the default (unset or the literal
|
||||
`never`) keeps rows forever — and rows older than the expiry are pruned each
|
||||
time the archive is (re)opened. An archive write failure is never silent
|
||||
success: the delivery records a failed attempt with the error and is marked
|
||||
failed.
|
||||
|
||||
Because reopens only happen on writes, an archive whose target has stopped
|
||||
receiving events would never be pruned. A background **archive sweeper** closes
|
||||
@@ -3019,7 +3004,6 @@ webhooker/
|
||||
│ ├── database/
|
||||
│ │ ├── base_model.go # BaseModel with UUID primary keys
|
||||
│ │ ├── database.go # GORM connection, migrations, admin seed
|
||||
│ │ ├── models.go # AutoMigrate for config-tier models
|
||||
│ │ ├── model_setting.go # Setting entity (key-value app config)
|
||||
│ │ ├── model_user.go # User entity
|
||||
│ │ ├── model_webhook.go # Webhook entity
|
||||
@@ -3036,6 +3020,11 @@ webhooker/
|
||||
│ │ └── databasetest/ # Test support: a WebhookDBManager for tests in other packages
|
||||
│ ├── datadir/
|
||||
│ │ └── lock.go # Exclusive advisory lock on DATA_DIR (one instance)
|
||||
│ ├── db/migrations/
|
||||
│ │ ├── migrations.go # Applies a database file's SQL files when it is opened
|
||||
│ │ ├── main/ # 000_migration.sql and 001_schema.sql for webhooker.db
|
||||
│ │ ├── events/ # The same for each events-{uuid}.db
|
||||
│ │ └── archive/ # The same for each archive file
|
||||
│ ├── globals/
|
||||
│ │ └── globals.go # Build-time variables (appname, version, arch)
|
||||
│ ├── gormlog/
|
||||
|
||||
Reference in New Issue
Block a user