Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f86ab6321d |
@@ -1071,27 +1071,23 @@ beside it, or wait until there are none.
|
||||
|
||||
4. Start the service. The container gives the directory and the restored files
|
||||
to the `webhooker` user before the app starts, whoever restored them (see
|
||||
[Running with Docker](#running-with-docker)). Each restored database gets any
|
||||
of its SQL migrations it has not had as it is opened (see
|
||||
[Upgrades](#upgrades)).
|
||||
[Running with Docker](#running-with-docker)). `AutoMigrate` runs against each
|
||||
restored database as it is opened.
|
||||
|
||||
### Upgrades
|
||||
|
||||
**Back up before every upgrade.** Each kind of database file has its schema in
|
||||
SQL files built into the binary, one directory per kind under
|
||||
`internal/db/migrations/`. In each, `000_migration.sql` creates the
|
||||
`schema_migrations` table, which records the number of every file the database
|
||||
has had, and `001_schema.sql` holds the whole schema. Opening a database runs,
|
||||
in one transaction, every file in its directory that `schema_migrations` does
|
||||
not list:
|
||||
**Back up before every upgrade.** Every start runs GORM `AutoMigrate`
|
||||
unconditionally against whatever files it finds:
|
||||
|
||||
- `main/` for the main database, on connect — settings, users, API keys,
|
||||
webhooks, entrypoints and targets
|
||||
- `events/` for each event database, when it is lazily opened — events,
|
||||
deliveries, delivery results and their running totals
|
||||
- `archive/` for each archive database, on every open and reopen
|
||||
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
|
||||
`Entrypoint`, `Target`
|
||||
- each event database when it is lazily opened — `Event`, `Delivery`,
|
||||
`DeliveryResult`, `EventTotals`, `TargetTotals`, `EntrypointTotals`
|
||||
- each archive database on every open and reopen
|
||||
|
||||
There are no down migrations.
|
||||
There is no schema version table, no migration ledger, and no down migrations.
|
||||
Nothing in the files records which version wrote them, and no code path undoes a
|
||||
migration.
|
||||
|
||||
Upgrade procedure:
|
||||
|
||||
@@ -1116,9 +1112,12 @@ connects to. Check the `bindAddress` field of the startup log to see what a
|
||||
running process bound. See [Bind address](#bind-address).
|
||||
|
||||
**Downgrade is unsupported.** Once a newer binary has migrated the files there
|
||||
is no way to move them back. The only supported way back to an older version is
|
||||
restoring the pre-upgrade backup, which discards everything received since that
|
||||
backup was taken.
|
||||
is no way to move them back. `AutoMigrate` is additive — it adds tables, columns
|
||||
and indexes and never drops or rewrites them — so an older binary will generally
|
||||
open migrated files and appear to work while writing against a schema it does
|
||||
not know about. The failure mode is silent divergence, not a startup error. The
|
||||
only supported way back to an older version is restoring the pre-upgrade backup,
|
||||
which discards everything received since that backup was taken.
|
||||
|
||||
### Version stamping
|
||||
|
||||
@@ -1183,6 +1182,26 @@ at rest and restrict who can read them.
|
||||
- `events-{uuid}.db` and `archive-….db` hold the **full payload body and
|
||||
headers** of every event as received, including whatever the sending service
|
||||
put in them — tokens, signatures, personal data.
|
||||
- Event databases written before
|
||||
[issue #206](https://git.eeqj.de/sneak/webhooker/issues/206) was fixed **also
|
||||
contain target credentials**: a GORM association upsert on the delivery and
|
||||
retry write path copied `targets` rows, `config` included, into the
|
||||
per-webhook database. For a Slack target the `webhookUrl` _is_ the bearer
|
||||
credential, and an `http` target's URL can embed userinfo. This version never
|
||||
writes those rows; the first time it opens such a file it deletes them and
|
||||
vacuums the file, which removes the credential bytes rather than only
|
||||
unlinking the rows. Deleting alone would not: the bytes stay readable in the
|
||||
file's free pages until it is rewritten. The sweep is recorded in the file's
|
||||
`user_version` only once the vacuum returns, so a sweep that fails or is
|
||||
interrupted fails the open and is retried on the next one, and a file this
|
||||
version has opened without error holds no leaked rows and no recoverable bytes
|
||||
from them. On upgrade this rewrites each existing `events-{uuid}.db` once, on
|
||||
its first open. Two cases still hand over live delivery destinations: a backup
|
||||
taken from an older build, and a backup of a file this version has not yet
|
||||
opened successfully. Copies already made stay affected — the sweep only
|
||||
rewrites the file it opens, and freed blocks may persist in filesystem
|
||||
snapshots and on the underlying storage. Rotate any target credential that was
|
||||
in a backup you cannot account for.
|
||||
- `webhooker.db` stores target config **unencrypted**, tracked at
|
||||
[issue #212](https://git.eeqj.de/sneak/webhooker/issues/212), next to the
|
||||
session encryption key and the Argon2id password hashes.
|
||||
@@ -1900,9 +1919,8 @@ Deliveries still `pending` or `retrying` count in neither.
|
||||
|
||||
#### Event-tier indexes
|
||||
|
||||
These indexes on the per-webhook event databases are created by
|
||||
`internal/db/migrations/events/001_schema.sql`, and declared in the model tags
|
||||
to match:
|
||||
These indexes on the per-webhook event databases are declared in the model tags,
|
||||
so `AutoMigrate` creates them on a fresh database:
|
||||
|
||||
| Table | Columns | Serves |
|
||||
| ------------------ | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
@@ -1966,9 +1984,8 @@ and application state:
|
||||
- **Targets** — delivery destination configurations
|
||||
- **APIKeys** — programmatic access credentials
|
||||
|
||||
On first startup the main database is created from its SQL files (see
|
||||
[Upgrades](#upgrades)), a session encryption key is generated and stored, and an
|
||||
`admin` user is created.
|
||||
On first startup the main database is auto-migrated, a session encryption key is
|
||||
generated and stored, and an `admin` user is created.
|
||||
|
||||
**Per-Webhook Event Databases** (`{DATA_DIR}/events-{webhook_uuid}.db`) — each
|
||||
webhook gets its own dedicated SQLite file containing:
|
||||
@@ -1980,10 +1997,9 @@ webhook gets its own dedicated SQLite file containing:
|
||||
the above, the deliveries per target, and each entrypoint's last event, kept
|
||||
through retention
|
||||
|
||||
Per-webhook databases are created automatically when a webhook is created, from
|
||||
the SQL files in `internal/db/migrations/events/`. They hold no copy of the main
|
||||
database's tables. They are managed by the `WebhookDBManager` component, which
|
||||
handles connection pooling, lazy opening, migrations, and cleanup.
|
||||
Per-webhook databases are created automatically when a webhook is created. They
|
||||
are managed by the `WebhookDBManager` component, which handles connection
|
||||
pooling, lazy opening, migrations, and cleanup.
|
||||
|
||||
A per-webhook database that is missing or zero-length later means its webhook's
|
||||
events and pending deliveries are gone. The next time it is opened, an empty one
|
||||
@@ -2067,15 +2083,14 @@ the service again.
|
||||
After each write the archive handle is closed and reopened, debounced to at most
|
||||
once per second, so an operator can move the archive file away for offline
|
||||
archiving without stopping the service; a moved or removed archive file is
|
||||
recreated automatically on the next write, from the SQL files in
|
||||
`internal/db/migrations/archive/`. A zero-length archive file is written to as a
|
||||
new archive: SQLite opens it as an empty database, so it holds nothing to lose.
|
||||
An optional `expiry` in the target's config JSON (e.g. `{"expiry":"720h"}`) is
|
||||
validated when the target is created — the default (unset or the literal
|
||||
`never`) keeps rows forever — and rows older than the expiry are pruned each
|
||||
time the archive is (re)opened. An archive write failure is never silent
|
||||
success: the delivery records a failed attempt with the error and is marked
|
||||
failed.
|
||||
recreated automatically on the next write. A zero-length archive file is written
|
||||
to as a new archive: SQLite opens it as an empty database, so it holds nothing
|
||||
to lose. An optional `expiry` in the target's config JSON (e.g.
|
||||
`{"expiry":"720h"}`) is validated when the target is created — the default
|
||||
(unset or the literal `never`) keeps rows forever — and rows older than the
|
||||
expiry are pruned each time the archive is (re)opened. An archive write failure
|
||||
is never silent success: the delivery records a failed attempt with the error
|
||||
and is marked failed.
|
||||
|
||||
Because reopens only happen on writes, an archive whose target has stopped
|
||||
receiving events would never be pruned. A background **archive sweeper** closes
|
||||
@@ -2536,10 +2551,23 @@ Where the pattern is out of reach — the transaction dispatch, an event capture
|
||||
outside the router, or a request that matched no route — the fallback is never
|
||||
the concrete path. The path becomes the literal `/(redacted)`, so the URL reads
|
||||
`http://host/(redacted)`; a URL the rewrite cannot parse into a scheme is
|
||||
withheld whole. A transaction event additionally carries the SDK's own
|
||||
`METHOD /path` name, built from the concrete path as well; it is rewritten on
|
||||
the same terms, to `POST /h/{uuid}` where the pattern is known and
|
||||
`POST /(redacted)` where it is not.
|
||||
withheld whole.
|
||||
|
||||
A transaction event also carries a name, which the SDK builds as `METHOD /path`
|
||||
from the concrete path and gives the transaction source `url`. Sentry groups
|
||||
transactions by name, so withholding it would put every request in the service
|
||||
into one group per method, such as `POST /(redacted)`. Inside each Sentry
|
||||
handler, a second middleware renames the transaction once the request has been
|
||||
routed, to the method and the route pattern (`POST /h/{uuid}`), and sets its
|
||||
source to `route`. The hook keeps a name whose source is `route` and rewrites
|
||||
any other to `METHOD /(redacted)`. A path that matches nothing at the top level
|
||||
has no pattern, so its transaction keeps the SDK's name and the hook withholds
|
||||
it. Inside a route group, a path that matches no route in the group, and a
|
||||
request the group's own middleware turns away before routing (a CSRF refusal,
|
||||
the redirect to `/pages/login`), are named after the group's pattern, such as
|
||||
`GET /pages/*` or `POST /hook/{sourceID}/*`, and the hook keeps that name. The
|
||||
URL on a transaction event still falls back to `http://host/(redacted)`, because
|
||||
the hook has no request there.
|
||||
|
||||
The headers are an allowlist for the same reason the rules above are
|
||||
unconditional: the SDK's own filter removes four names and passes everything
|
||||
@@ -3004,6 +3032,7 @@ webhooker/
|
||||
│ ├── database/
|
||||
│ │ ├── base_model.go # BaseModel with UUID primary keys
|
||||
│ │ ├── database.go # GORM connection, migrations, admin seed
|
||||
│ │ ├── models.go # AutoMigrate for config-tier models
|
||||
│ │ ├── model_setting.go # Setting entity (key-value app config)
|
||||
│ │ ├── model_user.go # User entity
|
||||
│ │ ├── model_webhook.go # Webhook entity
|
||||
@@ -3020,11 +3049,6 @@ webhooker/
|
||||
│ │ └── databasetest/ # Test support: a WebhookDBManager for tests in other packages
|
||||
│ ├── datadir/
|
||||
│ │ └── lock.go # Exclusive advisory lock on DATA_DIR (one instance)
|
||||
│ ├── db/migrations/
|
||||
│ │ ├── migrations.go # Applies a database file's SQL files when it is opened
|
||||
│ │ ├── main/ # 000_migration.sql and 001_schema.sql for webhooker.db
|
||||
│ │ ├── events/ # The same for each events-{uuid}.db
|
||||
│ │ └── archive/ # The same for each archive file
|
||||
│ ├── globals/
|
||||
│ │ └── globals.go # Build-time variables (appname, version, arch)
|
||||
│ ├── gormlog/
|
||||
@@ -3164,7 +3188,8 @@ Applied to all routes in this order:
|
||||
7. **Recoverer** — Panic recovery: one `ERROR` record through `internal/logger`
|
||||
and a `500`
|
||||
8. **Sentry** — Error reporting to Sentry (if `SENTRY_DSN` is set; configured
|
||||
with `Repanic: true` so panics still reach Recoverer)
|
||||
with `Repanic: true` so panics still reach Recoverer), and inside it the
|
||||
naming of the request's Sentry transaction after its route pattern
|
||||
|
||||
Recoverer sits seventh rather than first, and both neighbours are the reason. It
|
||||
runs **inside** everything that observes the response, so the `500` it writes
|
||||
@@ -3179,9 +3204,9 @@ set a header or start a timer.
|
||||
|
||||
Each admin page route group (`/pages`, `/user/*`, `/settings`, `/hooks`,
|
||||
`/hook/*`) starts with its own **Recoverer** and, if `SENTRY_DSN` is set, its
|
||||
own **Sentry** error reporting. That Recoverer answers a panic with the `500`
|
||||
error page in the normal layout; the global one keeps the plain-text `500` for
|
||||
every other route.
|
||||
own **Sentry** error reporting and transaction naming. That Recoverer answers a
|
||||
panic with the `500` error page in the normal layout; the global one keeps the
|
||||
plain-text `500` for every other route.
|
||||
|
||||
Additionally, form endpoints (`/pages`, `/user/*`, `/settings`, `/hooks`,
|
||||
`/hook/*`) apply a **MaxBodySize** middleware that limits POST/PUT/PATCH request
|
||||
|
||||
@@ -18,7 +18,6 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/banner"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/datadir"
|
||||
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
"sneak.berlin/go/webhooker/internal/logger"
|
||||
)
|
||||
@@ -250,21 +249,12 @@ func (d *Database) connectTo(dataDir string) error {
|
||||
return fmt.Errorf("migrating %s: %w", dbPath, err)
|
||||
}
|
||||
|
||||
err = parseModels(
|
||||
db, &Setting{}, &User{}, &APIKey{},
|
||||
&Webhook{}, &Entrypoint{}, &Target{},
|
||||
)
|
||||
if err != nil {
|
||||
return fmt.Errorf("connecting to %s: %w", dbPath, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// migrate applies the main database's SQL files. See
|
||||
// internal/db/migrations.
|
||||
func (d *Database) migrate() error {
|
||||
err := migrations.Apply(d.db, migrations.Main)
|
||||
// Run GORM auto-migrations
|
||||
err := d.Migrate()
|
||||
if err != nil {
|
||||
d.log.Error(
|
||||
"failed to run database migrations",
|
||||
@@ -279,21 +269,6 @@ func (d *Database) migrate() error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// parseModels has GORM parse each model for db before the handle is
|
||||
// shared. GORM parses a model the first time a handle uses it, and two
|
||||
// goroutines doing that at once for models that refer to each other,
|
||||
// as Webhook and Target do, race inside GORM.
|
||||
func parseModels(db *gorm.DB, models ...any) error {
|
||||
for _, model := range models {
|
||||
err := (&gorm.Statement{DB: db}).Parse(model)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parsing model %T: %w", model, err)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// ensureAdminUser creates the bootstrap admin account when the
|
||||
// deployment has no users at all.
|
||||
func (d *Database) ensureAdminUser() error {
|
||||
|
||||
@@ -2,6 +2,7 @@ package database
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
@@ -17,14 +18,14 @@ const omitAssociationsCallback = "webhooker:omit_associations"
|
||||
//
|
||||
// A per-webhook database holds the event tier only, but Delivery
|
||||
// declares belongs-to Event and Target and the delivery engine fills
|
||||
// both in memory before writing. Without this guard GORM upserts those
|
||||
// parent rows here on the delivery and retry write paths, and the
|
||||
// target, whose config holds destination URLs and bearer credentials,
|
||||
// goes to a targets table this file does not have, which fails the
|
||||
// write. Registering the guard on the connection
|
||||
// covers every write path, including writes inside a transaction and
|
||||
// write paths added later. Every event-tier row this file holds is
|
||||
// written explicitly, so nothing depends on the automatic save.
|
||||
// both in memory before writing. Without this guard GORM upserts
|
||||
// those parent rows here on the delivery and retry write paths,
|
||||
// copying targets.config, which holds destination URLs and bearer
|
||||
// credentials, into the file most likely to be backed up or handed
|
||||
// to someone else. Registering the guard on the connection covers
|
||||
// every write path, including writes inside a transaction and write
|
||||
// paths added later. Every event-tier row this file holds is written
|
||||
// explicitly, so nothing depends on the automatic save.
|
||||
func omitAssociations(db *gorm.DB) error {
|
||||
omit := func(tx *gorm.DB) {
|
||||
tx.Statement.Omits = append(
|
||||
@@ -52,3 +53,107 @@ func omitAssociations(db *gorm.DB) error {
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// eventDBSweptVersion is the PRAGMA user_version purgeTargetRows
|
||||
// stamps into a per-webhook database once it has removed any leaked
|
||||
// target rows *and* the VACUUM that removes their bytes has returned.
|
||||
// Nothing else in the tree uses user_version, so 0 means "not swept
|
||||
// by this build".
|
||||
//
|
||||
// The stamp, not the DELETE, is what records that a file is done. A
|
||||
// DELETE commits on its own, so a sweep that is interrupted or whose
|
||||
// VACUUM fails leaves a file whose rows are gone but whose credential
|
||||
// bytes are still in the free pages -- indistinguishable, by row
|
||||
// count, from a file that never leaked. Both leave the stamp unset,
|
||||
// so the next open sweeps again.
|
||||
const eventDBSweptVersion = 1
|
||||
|
||||
// purgeTargetRows deletes target rows that an earlier build's
|
||||
// association upsert wrote into a per-webhook database, and rewrites
|
||||
// the file so their bytes are gone with them. AutoMigrate creates a
|
||||
// targets table in every one of these files because Delivery declares
|
||||
// a belongs-to Target, but nothing in the event tier may put rows in
|
||||
// it. The rows it did put there are junk, not history: they carry an
|
||||
// empty webhook_id, and delivery rows resolve their target against
|
||||
// the main database, so nothing here refers to them.
|
||||
//
|
||||
// The DELETE only unlinks the rows: modernc.org/sqlite leaves
|
||||
// secure_delete at SQLite's default of off, so the credential bytes
|
||||
// stay readable in the file's free pages and a backup of a swept file
|
||||
// would still hand them over. VACUUM rewrites the file without them.
|
||||
//
|
||||
// This runs before every migration and is gated on
|
||||
// eventDBSweptVersion, so a file pays for the rewrite once, on the
|
||||
// first open that finds it unstamped, and every open after that is a
|
||||
// PRAGMA read. A file this build created is stamped before its
|
||||
// targets table exists, so it never vacuums at all. A failure here
|
||||
// fails the open with the stamp left unset, so the sweep is retried
|
||||
// rather than skipped -- a webhook whose file cannot be swept stays
|
||||
// unusable instead of quietly serving from a file that still holds
|
||||
// recoverable credentials.
|
||||
func purgeTargetRows(
|
||||
db *gorm.DB, log *slog.Logger, webhookID string,
|
||||
) error {
|
||||
var version int
|
||||
|
||||
// Row().Scan, not (*gorm.DB).Scan: see internal/gormlog.
|
||||
err := db.Raw("PRAGMA user_version").Row().Scan(&version)
|
||||
if err != nil {
|
||||
return fmt.Errorf(
|
||||
"reading sweep marker of webhook database %s: %w",
|
||||
webhookID, err,
|
||||
)
|
||||
}
|
||||
|
||||
if version >= eventDBSweptVersion {
|
||||
return nil
|
||||
}
|
||||
|
||||
var purged int64
|
||||
|
||||
if db.Migrator().HasTable("targets") {
|
||||
res := db.Exec("DELETE FROM targets")
|
||||
if res.Error != nil {
|
||||
return fmt.Errorf(
|
||||
"purging target rows from webhook database %s: %w",
|
||||
webhookID, res.Error,
|
||||
)
|
||||
}
|
||||
|
||||
purged = res.RowsAffected
|
||||
|
||||
// Unconditional: a zero row count here does not mean there is
|
||||
// nothing to remove, only that no *live* row is left. See
|
||||
// eventDBSweptVersion.
|
||||
err = db.Exec("VACUUM").Error
|
||||
if err != nil {
|
||||
return fmt.Errorf(
|
||||
"purged %d leaked target rows from webhook database "+
|
||||
"%s but vacuuming it failed, so the deleted "+
|
||||
"target credentials are still recoverable from "+
|
||||
"the file; it stays marked unswept and the next "+
|
||||
"open retries: %w",
|
||||
purged, webhookID, err,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
err = db.Exec(fmt.Sprintf(
|
||||
"PRAGMA user_version = %d", eventDBSweptVersion,
|
||||
)).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf(
|
||||
"marking webhook database %s swept: %w", webhookID, err,
|
||||
)
|
||||
}
|
||||
|
||||
if purged > 0 {
|
||||
log.Warn(
|
||||
"purged leaked target rows from per-webhook database",
|
||||
"webhook_id", webhookID,
|
||||
"rows", purged,
|
||||
)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -1,15 +1,392 @@
|
||||
package database_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
_ "modernc.org/sqlite"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||
)
|
||||
|
||||
// testDataDirPerm is the mode the test data directory is created
|
||||
// with.
|
||||
const testDataDirPerm = 0o750
|
||||
|
||||
// eventDBDataDir returns a data directory that a WebhookDBManager
|
||||
// can be pointed at.
|
||||
func eventDBDataDir(t *testing.T) string {
|
||||
t.Helper()
|
||||
|
||||
dir := filepath.Join(t.TempDir(), "events")
|
||||
require.NoError(t, os.MkdirAll(dir, testDataDirPerm))
|
||||
|
||||
return dir
|
||||
}
|
||||
|
||||
// openRawEventDB opens the per-webhook database file directly,
|
||||
// without the manager, so a test can put a file on disk in a state
|
||||
// the manager has to cope with, or inspect one afterwards.
|
||||
func openRawEventDB(
|
||||
t *testing.T, dataDir, webhookID string,
|
||||
) *sql.DB {
|
||||
t.Helper()
|
||||
|
||||
path := filepath.Join(
|
||||
dataDir, fmt.Sprintf("events-%s.db", webhookID),
|
||||
)
|
||||
|
||||
sqlDB, err := sql.Open(
|
||||
"sqlite",
|
||||
fmt.Sprintf("file:%s?mode=rwc", path),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
t.Cleanup(func() { _ = sqlDB.Close() })
|
||||
|
||||
return sqlDB
|
||||
}
|
||||
|
||||
// eventDBFileBytes reads a per-webhook database file off disk, so a
|
||||
// test can assert on what the file itself still holds rather than on
|
||||
// what a query returns.
|
||||
func eventDBFileBytes(t *testing.T, dataDir, webhookID string) []byte {
|
||||
t.Helper()
|
||||
|
||||
//nolint:gosec // reads a file the test just created under t.TempDir()
|
||||
raw, err := os.ReadFile(filepath.Join(
|
||||
dataDir, fmt.Sprintf("events-%s.db", webhookID),
|
||||
))
|
||||
require.NoError(t, err)
|
||||
|
||||
return raw
|
||||
}
|
||||
|
||||
// eventDBUserVersion returns the PRAGMA user_version of a per-webhook
|
||||
// database file, which is the marker purgeTargetRows stamps once it
|
||||
// has swept and vacuumed.
|
||||
func eventDBUserVersion(t *testing.T, sqlDB *sql.DB) int {
|
||||
t.Helper()
|
||||
|
||||
var version int
|
||||
|
||||
require.NoError(t, sqlDB.QueryRowContext(
|
||||
t.Context(), "PRAGMA user_version",
|
||||
).Scan(&version))
|
||||
|
||||
return version
|
||||
}
|
||||
|
||||
// clearEventDBSweptMarker resets the sweep marker to 0, which is what
|
||||
// a file written by a build without the sweep looks like. Tests that
|
||||
// seed a leaked row have to create the file through the manager to
|
||||
// get the real targets table shape, and that stamps it.
|
||||
func clearEventDBSweptMarker(t *testing.T, sqlDB *sql.DB) {
|
||||
t.Helper()
|
||||
|
||||
_, err := sqlDB.ExecContext(t.Context(), "PRAGMA user_version = 0")
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// countTargetRows returns the number of rows in the targets table of
|
||||
// a per-webhook database file, or -1 if the table does not exist.
|
||||
func countTargetRows(t *testing.T, sqlDB *sql.DB) int {
|
||||
t.Helper()
|
||||
|
||||
var tables int
|
||||
|
||||
require.NoError(t, sqlDB.QueryRowContext(
|
||||
t.Context(),
|
||||
"SELECT count(*) FROM sqlite_master "+
|
||||
"WHERE type = 'table' AND name = 'targets'",
|
||||
).Scan(&tables))
|
||||
|
||||
if tables == 0 {
|
||||
return -1
|
||||
}
|
||||
|
||||
var rows int
|
||||
|
||||
require.NoError(t, sqlDB.QueryRowContext(
|
||||
t.Context(), "SELECT count(*) FROM targets",
|
||||
).Scan(&rows))
|
||||
|
||||
return rows
|
||||
}
|
||||
|
||||
// TestOpenPurgesLeakedTargetRows covers the sweep for event
|
||||
// databases written by a build that let GORM upsert target rows
|
||||
// into them: opening the database clears them, and opening it again
|
||||
// is a no-op.
|
||||
func TestOpenPurgesLeakedTargetRows(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dataDir := eventDBDataDir(t)
|
||||
webhookID := uuid.New().String()
|
||||
|
||||
// Create the file the way the application does, so the targets
|
||||
// table has exactly the shape AutoMigrate gives it, then write
|
||||
// a leaked row into it the way the association upsert did.
|
||||
initial := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err := initial.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, initial.CloseAll())
|
||||
|
||||
seed := openRawEventDB(t, dataDir, webhookID)
|
||||
|
||||
_, err = seed.ExecContext(
|
||||
t.Context(),
|
||||
"INSERT INTO targets "+
|
||||
"(id, webhook_id, name, type, config) "+
|
||||
"VALUES (?, '', ?, ?, ?)",
|
||||
uuid.New().String(),
|
||||
"leaked-target",
|
||||
"slack",
|
||||
`{"webhookUrl":"https://hooks.example/T000/B000/secret"}`,
|
||||
)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, 1, countTargetRows(t, seed))
|
||||
clearEventDBSweptMarker(t, seed)
|
||||
require.NoError(t, seed.Close())
|
||||
|
||||
mgr := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err = mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, mgr.CloseAll())
|
||||
|
||||
check := openRawEventDB(t, dataDir, webhookID)
|
||||
assert.Zero(t, countTargetRows(t, check))
|
||||
assert.Equal(
|
||||
t, 1, eventDBUserVersion(t, check),
|
||||
"a completed sweep must mark the file so later opens skip it",
|
||||
)
|
||||
require.NoError(t, check.Close())
|
||||
|
||||
// Idempotent: a second open leaves it at zero and does not
|
||||
// error.
|
||||
again := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err = again.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, again.CloseAll())
|
||||
|
||||
recheck := openRawEventDB(t, dataDir, webhookID)
|
||||
assert.Zero(t, countTargetRows(t, recheck))
|
||||
}
|
||||
|
||||
// TestOpenPurgeRemovesCredentialBytes covers the sweep at the level
|
||||
// that matters for a backup handed to someone else: the leaked
|
||||
// credential must be gone from the raw bytes of the file, not merely
|
||||
// unreachable by query. A bare DELETE unlinks the row and leaves the
|
||||
// bytes readable in the free pages, so this fails without the VACUUM
|
||||
// in purgeTargetRows.
|
||||
func TestOpenPurgeRemovesCredentialBytes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dataDir := eventDBDataDir(t)
|
||||
webhookID := uuid.New().String()
|
||||
credential := "T00000000/B00000000/" + uuid.New().String()
|
||||
|
||||
initial := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err := initial.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, initial.CloseAll())
|
||||
|
||||
seed := openRawEventDB(t, dataDir, webhookID)
|
||||
|
||||
_, err = seed.ExecContext(
|
||||
t.Context(),
|
||||
"INSERT INTO targets "+
|
||||
"(id, webhook_id, name, type, config) "+
|
||||
"VALUES (?, '', ?, ?, ?)",
|
||||
uuid.New().String(),
|
||||
"leaked-target",
|
||||
"slack",
|
||||
fmt.Sprintf(
|
||||
`{"webhookUrl":"https://hooks.example/%s"}`, credential,
|
||||
),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
clearEventDBSweptMarker(t, seed)
|
||||
require.NoError(t, seed.Close())
|
||||
|
||||
// The seed has to be in the file for its absence later to mean
|
||||
// anything.
|
||||
require.True(
|
||||
t,
|
||||
bytes.Contains(
|
||||
eventDBFileBytes(t, dataDir, webhookID),
|
||||
[]byte(credential),
|
||||
),
|
||||
"seeded credential is not in the file, so this test proves nothing",
|
||||
)
|
||||
|
||||
mgr := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err = mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, mgr.CloseAll())
|
||||
|
||||
assert.NotContains(
|
||||
t,
|
||||
string(eventDBFileBytes(t, dataDir, webhookID)),
|
||||
credential,
|
||||
"leaked credential is still recoverable from the raw file",
|
||||
)
|
||||
}
|
||||
|
||||
// TestOpenRevacuumsAfterIncompleteSweep covers the case a row count
|
||||
// cannot see: the rows are already deleted but the file was never
|
||||
// vacuumed, because an earlier sweep died between the two or its
|
||||
// VACUUM failed. The credential bytes are still recoverable, and the
|
||||
// unset marker is the only thing that says so, so the next open must
|
||||
// vacuum rather than conclude from the empty table that there is
|
||||
// nothing to do.
|
||||
func TestOpenRevacuumsAfterIncompleteSweep(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dataDir := eventDBDataDir(t)
|
||||
webhookID := uuid.New().String()
|
||||
credential := "T00000000/B00000000/" + uuid.New().String()
|
||||
|
||||
initial := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err := initial.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, initial.CloseAll())
|
||||
|
||||
seed := openRawEventDB(t, dataDir, webhookID)
|
||||
|
||||
_, err = seed.ExecContext(
|
||||
t.Context(),
|
||||
"INSERT INTO targets "+
|
||||
"(id, webhook_id, name, type, config) "+
|
||||
"VALUES (?, '', ?, ?, ?)",
|
||||
uuid.New().String(),
|
||||
"leaked-target",
|
||||
"slack",
|
||||
fmt.Sprintf(
|
||||
`{"webhookUrl":"https://hooks.example/%s"}`, credential,
|
||||
),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Exactly the state an interrupted sweep leaves: rows gone,
|
||||
// marker unset, bytes still in the free pages.
|
||||
_, err = seed.ExecContext(t.Context(), "DELETE FROM targets")
|
||||
require.NoError(t, err)
|
||||
require.Zero(t, countTargetRows(t, seed))
|
||||
clearEventDBSweptMarker(t, seed)
|
||||
require.NoError(t, seed.Close())
|
||||
|
||||
require.True(
|
||||
t,
|
||||
bytes.Contains(
|
||||
eventDBFileBytes(t, dataDir, webhookID),
|
||||
[]byte(credential),
|
||||
),
|
||||
"the deleted row's bytes must still be in the file, or this "+
|
||||
"test proves nothing",
|
||||
)
|
||||
|
||||
mgr := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err = mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, mgr.CloseAll())
|
||||
|
||||
assert.NotContains(
|
||||
t,
|
||||
string(eventDBFileBytes(t, dataDir, webhookID)),
|
||||
credential,
|
||||
"an interrupted sweep was not retried, so the credential is "+
|
||||
"still recoverable from the raw file",
|
||||
)
|
||||
|
||||
check := openRawEventDB(t, dataDir, webhookID)
|
||||
assert.Equal(t, 1, eventDBUserVersion(t, check))
|
||||
}
|
||||
|
||||
// TestOpenSkipsSweptDatabase covers the other half of the marker: a
|
||||
// file this build created is marked without ever being vacuumed, and
|
||||
// a marked file is not swept again.
|
||||
func TestOpenSkipsSweptDatabase(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dataDir := eventDBDataDir(t)
|
||||
webhookID := uuid.New().String()
|
||||
|
||||
mgr := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err := mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, mgr.CloseAll())
|
||||
|
||||
marked := openRawEventDB(t, dataDir, webhookID)
|
||||
assert.Equal(t, 1, eventDBUserVersion(t, marked))
|
||||
|
||||
// A marked file is left alone, so a row written into it survives
|
||||
// a reopen. Nothing writes target rows any more; this stands in
|
||||
// for the sweep having run.
|
||||
_, err = marked.ExecContext(
|
||||
t.Context(),
|
||||
"INSERT INTO targets "+
|
||||
"(id, webhook_id, name, type, config) "+
|
||||
"VALUES (?, '', ?, ?, ?)",
|
||||
uuid.New().String(), "sentinel", "slack", `{}`,
|
||||
)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, marked.Close())
|
||||
|
||||
again := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err = again.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, again.CloseAll())
|
||||
|
||||
check := openRawEventDB(t, dataDir, webhookID)
|
||||
assert.Equal(
|
||||
t, 1, countTargetRows(t, check),
|
||||
"a marked file must not be swept again",
|
||||
)
|
||||
}
|
||||
|
||||
// TestOpenSucceedsWithoutTargetsTable covers an existing event
|
||||
// database that never grew a targets table. The sweep must not fail
|
||||
// startup on it.
|
||||
func TestOpenSucceedsWithoutTargetsTable(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dataDir := eventDBDataDir(t)
|
||||
webhookID := uuid.New().String()
|
||||
|
||||
seed := openRawEventDB(t, dataDir, webhookID)
|
||||
|
||||
_, err := seed.ExecContext(
|
||||
t.Context(),
|
||||
"CREATE TABLE events (id text PRIMARY KEY)",
|
||||
)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, seed.Close())
|
||||
|
||||
mgr := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
db, err := mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
assert.NotNil(t, db)
|
||||
require.NoError(t, mgr.CloseAll())
|
||||
}
|
||||
|
||||
// TestEventDBCreateOmitsAssociations covers the connection-level
|
||||
// guard directly: a Delivery carrying its Event and Target in
|
||||
// memory, written through the manager's handle, must store only the
|
||||
@@ -17,9 +394,10 @@ import (
|
||||
func TestEventDBCreateOmitsAssociations(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dataDir := eventDBDataDir(t)
|
||||
webhookID := uuid.New().String()
|
||||
|
||||
mgr := databasetest.NewWebhookDBManager(t, t.TempDir())
|
||||
mgr := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
db, err := mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
@@ -54,9 +432,8 @@ func TestEventDBCreateOmitsAssociations(t *testing.T) {
|
||||
require.NoError(t, db.Model(d).
|
||||
Update("status", database.DeliveryStatusDelivered).
|
||||
Error)
|
||||
|
||||
// The file has no targets table, so the create and update above
|
||||
// would have failed had GORM tried to save the target.
|
||||
assert.False(t, db.Migrator().HasTable(&database.Target{}))
|
||||
require.NoError(t, mgr.CloseAll())
|
||||
|
||||
check := openRawEventDB(t, dataDir, webhookID)
|
||||
assert.Zero(t, countTargetRows(t, check))
|
||||
}
|
||||
|
||||
@@ -13,8 +13,10 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// TestWebhookDBManager_OpenAddsEventTierIndexes verifies that a new
|
||||
// per-webhook database has these indexes.
|
||||
// TestWebhookDBManager_OpenAddsEventTierIndexes verifies that opening a
|
||||
// per-webhook database that predates these indexes creates them. It
|
||||
// stands in for an older database file by dropping the indexes
|
||||
// AutoMigrate just created, then reopening the same file.
|
||||
func TestWebhookDBManager_OpenAddsEventTierIndexes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -40,9 +42,27 @@ func TestWebhookDBManager_OpenAddsEventTierIndexes(t *testing.T) {
|
||||
db, err := mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// A fresh database has them.
|
||||
for _, ix := range indexes {
|
||||
require.True(t, db.Migrator().HasIndex(ix.model, ix.name))
|
||||
}
|
||||
|
||||
// Stand in for a database file created before the indexes existed.
|
||||
for _, ix := range indexes {
|
||||
require.NoError(t, db.Migrator().DropIndex(ix.model, ix.name))
|
||||
require.False(t, db.Migrator().HasIndex(ix.model, ix.name))
|
||||
}
|
||||
|
||||
// Drop the cached connection so the next open reopens the file and
|
||||
// runs AutoMigrate against it, as a restart would.
|
||||
require.NoError(t, mgr.CloseAll())
|
||||
|
||||
db, err = mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
|
||||
for _, ix := range indexes {
|
||||
assert.True(t, db.Migrator().HasIndex(ix.model, ix.name),
|
||||
"a new database should have %s", ix.name)
|
||||
"opening the existing database should create %s", ix.name)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
package database
|
||||
|
||||
// Migrate runs database migrations for the main application database.
|
||||
// Only configuration-tier models are stored in the main database.
|
||||
// Event-tier models (Event, Delivery, DeliveryResult, EventTotals,
|
||||
// TargetTotals, EntrypointTotals) live in
|
||||
// per-webhook dedicated databases managed by WebhookDBManager.
|
||||
func (d *Database) Migrate() error {
|
||||
return d.db.AutoMigrate(
|
||||
&Setting{},
|
||||
&User{},
|
||||
&APIKey{},
|
||||
&Webhook{},
|
||||
&Entrypoint{},
|
||||
&Target{},
|
||||
)
|
||||
}
|
||||
@@ -1,168 +0,0 @@
|
||||
package database_test
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||
)
|
||||
|
||||
// withFixedNow returns db with GORM's clock stopped at a time in UTC,
|
||||
// so that the created and updated times it writes come back from
|
||||
// SQLite equal to the row written, which holds them in UTC.
|
||||
func withFixedNow(db *gorm.DB) (*gorm.DB, time.Time) {
|
||||
at := time.Date(2026, 10, 13, 12, 30, 45, 123456789, time.UTC)
|
||||
|
||||
return db.Session(&gorm.Session{
|
||||
NowFunc: func() time.Time { return at },
|
||||
}), at
|
||||
}
|
||||
|
||||
// newBaseModel returns a BaseModel with a new ID.
|
||||
func newBaseModel() database.BaseModel {
|
||||
return database.BaseModel{ID: uuid.New().String()}
|
||||
}
|
||||
|
||||
// assertRoundTrip writes row, which must be the only row of its
|
||||
// table, and reads back the same row.
|
||||
func assertRoundTrip[T any](t *testing.T, db *gorm.DB, row *T) {
|
||||
t.Helper()
|
||||
|
||||
require.NoError(t, db.Create(row).Error)
|
||||
|
||||
var got T
|
||||
|
||||
require.NoError(t, db.First(&got).Error)
|
||||
assert.Equal(t, *row, got)
|
||||
}
|
||||
|
||||
// TestMainSchemaHoldsEveryModel writes a row of each main database
|
||||
// model, every field set, into a database made from its SQL files.
|
||||
func TestMainSchemaHoldsEveryModel(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
mainDB, err := database.Open(
|
||||
t.TempDir(), slog.New(slog.DiscardHandler),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
t.Cleanup(func() { _ = mainDB.Close() })
|
||||
|
||||
db, at := withFixedNow(mainDB.DB())
|
||||
|
||||
assertRoundTrip(t, db, &database.Setting{
|
||||
Key: "session_key", Value: "value",
|
||||
})
|
||||
assertRoundTrip(t, db, &database.User{
|
||||
BaseModel: newBaseModel(),
|
||||
Username: "user",
|
||||
Password: "password hash",
|
||||
})
|
||||
assertRoundTrip(t, db, &database.APIKey{
|
||||
BaseModel: newBaseModel(),
|
||||
UserID: uuid.New().String(),
|
||||
Key: "key",
|
||||
Description: "API key description",
|
||||
LastUsedAt: &at,
|
||||
})
|
||||
assertRoundTrip(t, db, &database.Webhook{
|
||||
BaseModel: newBaseModel(),
|
||||
UserID: uuid.New().String(),
|
||||
Name: "webhook",
|
||||
Description: "webhook description",
|
||||
RetentionDays: 7,
|
||||
})
|
||||
assertRoundTrip(t, db, &database.Entrypoint{
|
||||
BaseModel: newBaseModel(),
|
||||
WebhookID: uuid.New().String(),
|
||||
Path: uuid.New().String(),
|
||||
Description: "entrypoint description",
|
||||
Active: true,
|
||||
})
|
||||
assertRoundTrip(t, db, &database.Target{
|
||||
BaseModel: newBaseModel(),
|
||||
WebhookID: uuid.New().String(),
|
||||
Name: "target",
|
||||
Type: database.TargetTypeHTTP,
|
||||
Active: true,
|
||||
Config: `{"url":"https://example.com/"}`,
|
||||
MaxRetries: 3,
|
||||
})
|
||||
}
|
||||
|
||||
// TestEventSchemaHoldsEveryModel does the same for a webhook's event
|
||||
// database, which the manager makes from its SQL files.
|
||||
func TestEventSchemaHoldsEveryModel(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
mgr := databasetest.NewWebhookDBManager(t, t.TempDir())
|
||||
|
||||
t.Cleanup(func() { _ = mgr.CloseAll() })
|
||||
|
||||
events, err := mgr.GetDB(uuid.New().String())
|
||||
require.NoError(t, err)
|
||||
|
||||
db, at := withFixedNow(events)
|
||||
source := uuid.New().String()
|
||||
|
||||
assertRoundTrip(t, db, &database.Event{
|
||||
BaseModel: newBaseModel(),
|
||||
WebhookID: uuid.New().String(),
|
||||
EntrypointID: uuid.New().String(),
|
||||
Method: http.MethodPost,
|
||||
RawQuery: "a=1",
|
||||
Headers: `{"X":"Y"}`,
|
||||
Body: "body",
|
||||
ContentType: "text/plain",
|
||||
BodyBytes: 4,
|
||||
ResubmittedFromID: &source,
|
||||
})
|
||||
assertRoundTrip(t, db, &database.Delivery{
|
||||
BaseModel: newBaseModel(),
|
||||
EventID: uuid.New().String(),
|
||||
TargetID: uuid.New().String(),
|
||||
Status: database.DeliveryStatusDelivered,
|
||||
FinishedAt: &at,
|
||||
Replay: true,
|
||||
})
|
||||
assertRoundTrip(t, db, &database.DeliveryResult{
|
||||
BaseModel: newBaseModel(),
|
||||
DeliveryID: uuid.New().String(),
|
||||
AttemptNum: 2,
|
||||
Success: true,
|
||||
StatusCode: 200,
|
||||
ResponseBody: "ok",
|
||||
Error: "error",
|
||||
Duration: 15,
|
||||
})
|
||||
assertRoundTrip(t, db, &database.TargetTotals{
|
||||
TargetID: uuid.New().String(),
|
||||
Deliveries: 5,
|
||||
Delivered: 3,
|
||||
Failed: 1,
|
||||
DeliveriesRemoved: 2,
|
||||
FailedRemoved: 1,
|
||||
})
|
||||
assertRoundTrip(t, db, &database.EntrypointTotals{
|
||||
EntrypointID: uuid.New().String(), LastEventAt: at,
|
||||
})
|
||||
|
||||
// A new event database has its one row of event totals already.
|
||||
require.NoError(t, database.AddEventTotals(db, database.EventTotals{
|
||||
Events: 3, EventsRemoved: 1, LastEventAt: &at,
|
||||
}))
|
||||
|
||||
var totals []database.EventTotals
|
||||
|
||||
require.NoError(t, db.Find(&totals).Error)
|
||||
assert.Equal(t, []database.EventTotals{{
|
||||
ID: 1, Events: 3, EventsRemoved: 1, LastEventAt: &at,
|
||||
}}, totals)
|
||||
}
|
||||
@@ -14,7 +14,6 @@ import (
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/datadir"
|
||||
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
)
|
||||
|
||||
@@ -371,7 +370,18 @@ func (m *WebhookDBManager) openDB(
|
||||
)
|
||||
}
|
||||
|
||||
err = migrations.Apply(db, migrations.Events)
|
||||
err = purgeTargetRows(db, m.log, webhookID)
|
||||
if err != nil {
|
||||
_ = sqlDB.Close()
|
||||
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Run migrations for event-tier models only
|
||||
err = db.AutoMigrate(
|
||||
&Event{}, &Delivery{}, &DeliveryResult{},
|
||||
&EventTotals{}, &TargetTotals{}, &EntrypointTotals{},
|
||||
)
|
||||
if err != nil {
|
||||
_ = sqlDB.Close()
|
||||
|
||||
@@ -381,15 +391,14 @@ func (m *WebhookDBManager) openDB(
|
||||
)
|
||||
}
|
||||
|
||||
err = parseModels(
|
||||
db, &Event{}, &Delivery{}, &DeliveryResult{},
|
||||
&EventTotals{}, &TargetTotals{}, &EntrypointTotals{},
|
||||
)
|
||||
// A new database gets its row of event totals, all zero. Target
|
||||
// totals rows are created by the first delivery to each target.
|
||||
err = db.FirstOrCreate(&EventTotals{}).Error
|
||||
if err != nil {
|
||||
_ = sqlDB.Close()
|
||||
|
||||
return nil, fmt.Errorf(
|
||||
"connecting to webhook database %s: %w",
|
||||
"creating event totals for webhook database %s: %w",
|
||||
webhookID, err,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -1,9 +0,0 @@
|
||||
-- Migration 000: the table recording which migrations this file has
|
||||
-- had. Applied before the others; it records itself.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
version INTEGER PRIMARY KEY,
|
||||
applied_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
INSERT OR IGNORE INTO schema_migrations (version) VALUES (0);
|
||||
@@ -1,18 +0,0 @@
|
||||
-- Migration 001: the schema of a database target's archive file. The
|
||||
-- archivedEvent model in internal/delivery must agree with it.
|
||||
|
||||
CREATE TABLE archived_events (
|
||||
id integer PRIMARY KEY AUTOINCREMENT,
|
||||
event_id text,
|
||||
webhook_id text,
|
||||
entrypoint_id text,
|
||||
method text,
|
||||
raw_query text,
|
||||
headers text,
|
||||
body text,
|
||||
content_type text,
|
||||
archived_at datetime
|
||||
);
|
||||
|
||||
CREATE INDEX idx_archived_events_event_id ON archived_events (event_id);
|
||||
CREATE INDEX idx_archived_events_archived_at ON archived_events (archived_at);
|
||||
@@ -1,9 +0,0 @@
|
||||
-- Migration 000: the table recording which migrations this file has
|
||||
-- had. Applied before the others; it records itself.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
version INTEGER PRIMARY KEY,
|
||||
applied_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
INSERT OR IGNORE INTO schema_migrations (version) VALUES (0);
|
||||
@@ -1,98 +0,0 @@
|
||||
-- Migration 001: the schema of a webhook's event database. The models
|
||||
-- in internal/database must agree with it. The webhook, entrypoint and
|
||||
-- target ids here name rows in the main database, which has those
|
||||
-- tables; this file does not.
|
||||
|
||||
CREATE TABLE events (
|
||||
id uuid,
|
||||
created_at datetime,
|
||||
updated_at datetime,
|
||||
deleted_at datetime,
|
||||
webhook_id uuid NOT NULL,
|
||||
entrypoint_id uuid NOT NULL,
|
||||
method text NOT NULL,
|
||||
raw_query text,
|
||||
headers text,
|
||||
body text,
|
||||
content_type text,
|
||||
body_bytes integer NOT NULL,
|
||||
resubmitted_from_id uuid,
|
||||
PRIMARY KEY (id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_events_created_at ON events (created_at);
|
||||
CREATE INDEX idx_events_deleted_at ON events (deleted_at);
|
||||
CREATE INDEX idx_events_deleted_at_created_at
|
||||
ON events (deleted_at, created_at);
|
||||
CREATE INDEX idx_events_resubmitted_from_id
|
||||
ON events (resubmitted_from_id, deleted_at);
|
||||
CREATE INDEX idx_events_entrypoint_id
|
||||
ON events (entrypoint_id, deleted_at, resubmitted_from_id, created_at);
|
||||
|
||||
CREATE TABLE deliveries (
|
||||
id uuid,
|
||||
created_at datetime,
|
||||
updated_at datetime,
|
||||
deleted_at datetime,
|
||||
event_id uuid NOT NULL,
|
||||
target_id uuid NOT NULL,
|
||||
status text NOT NULL DEFAULT 'pending',
|
||||
finished_at datetime,
|
||||
replay numeric NOT NULL DEFAULT false,
|
||||
PRIMARY KEY (id),
|
||||
CONSTRAINT fk_events_deliveries
|
||||
FOREIGN KEY (event_id) REFERENCES events (id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_deliveries_deleted_at ON deliveries (deleted_at);
|
||||
CREATE INDEX idx_deliveries_event_id ON deliveries (event_id, deleted_at);
|
||||
CREATE INDEX idx_deliveries_status
|
||||
ON deliveries (status, deleted_at, finished_at, target_id);
|
||||
|
||||
CREATE TABLE delivery_results (
|
||||
id uuid,
|
||||
created_at datetime,
|
||||
updated_at datetime,
|
||||
deleted_at datetime,
|
||||
delivery_id uuid NOT NULL,
|
||||
attempt_num integer NOT NULL,
|
||||
success numeric,
|
||||
status_code integer,
|
||||
response_body text,
|
||||
error text,
|
||||
duration integer,
|
||||
PRIMARY KEY (id),
|
||||
CONSTRAINT fk_deliveries_delivery_results
|
||||
FOREIGN KEY (delivery_id) REFERENCES deliveries (id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_delivery_results_deleted_at
|
||||
ON delivery_results (deleted_at);
|
||||
CREATE INDEX idx_delivery_results_delivery_id
|
||||
ON delivery_results (delivery_id, deleted_at);
|
||||
|
||||
CREATE TABLE event_totals (
|
||||
id integer PRIMARY KEY AUTOINCREMENT,
|
||||
events integer NOT NULL,
|
||||
events_removed integer NOT NULL,
|
||||
last_event_at datetime
|
||||
);
|
||||
|
||||
-- The one row of event totals, all zero.
|
||||
INSERT INTO event_totals (events, events_removed) VALUES (0, 0);
|
||||
|
||||
CREATE TABLE target_totals (
|
||||
target_id uuid,
|
||||
deliveries integer NOT NULL,
|
||||
delivered integer NOT NULL,
|
||||
failed integer NOT NULL,
|
||||
deliveries_removed integer NOT NULL,
|
||||
failed_removed integer NOT NULL,
|
||||
PRIMARY KEY (target_id)
|
||||
);
|
||||
|
||||
CREATE TABLE entrypoint_totals (
|
||||
entrypoint_id uuid,
|
||||
last_event_at datetime NOT NULL,
|
||||
PRIMARY KEY (entrypoint_id)
|
||||
);
|
||||
@@ -1,9 +0,0 @@
|
||||
-- Migration 000: the table recording which migrations this file has
|
||||
-- had. Applied before the others; it records itself.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
version INTEGER PRIMARY KEY,
|
||||
applied_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
INSERT OR IGNORE INTO schema_migrations (version) VALUES (0);
|
||||
@@ -1,89 +0,0 @@
|
||||
-- Migration 001: the main database's schema. The models in
|
||||
-- internal/database must agree with it.
|
||||
|
||||
CREATE TABLE settings (
|
||||
"key" text,
|
||||
value text NOT NULL,
|
||||
PRIMARY KEY ("key")
|
||||
);
|
||||
|
||||
CREATE TABLE users (
|
||||
id uuid,
|
||||
created_at datetime,
|
||||
updated_at datetime,
|
||||
deleted_at datetime,
|
||||
username text NOT NULL,
|
||||
password text NOT NULL,
|
||||
PRIMARY KEY (id),
|
||||
CONSTRAINT chk_users_username
|
||||
CHECK (length(CAST(username AS BLOB)) <= 1024)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_users_deleted_at ON users (deleted_at);
|
||||
CREATE UNIQUE INDEX idx_users_username ON users (username);
|
||||
|
||||
CREATE TABLE api_keys (
|
||||
id uuid,
|
||||
created_at datetime,
|
||||
updated_at datetime,
|
||||
deleted_at datetime,
|
||||
user_id uuid NOT NULL,
|
||||
"key" text NOT NULL,
|
||||
description text,
|
||||
last_used_at datetime,
|
||||
PRIMARY KEY (id),
|
||||
CONSTRAINT fk_users_api_keys FOREIGN KEY (user_id) REFERENCES users (id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_api_keys_deleted_at ON api_keys (deleted_at);
|
||||
CREATE UNIQUE INDEX idx_api_keys_key ON api_keys ("key");
|
||||
|
||||
CREATE TABLE webhooks (
|
||||
id uuid,
|
||||
created_at datetime,
|
||||
updated_at datetime,
|
||||
deleted_at datetime,
|
||||
user_id uuid NOT NULL,
|
||||
name text NOT NULL,
|
||||
description text,
|
||||
retention_days integer DEFAULT 30,
|
||||
PRIMARY KEY (id),
|
||||
CONSTRAINT fk_users_webhooks FOREIGN KEY (user_id) REFERENCES users (id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_webhooks_deleted_at ON webhooks (deleted_at);
|
||||
|
||||
CREATE TABLE entrypoints (
|
||||
id uuid,
|
||||
created_at datetime,
|
||||
updated_at datetime,
|
||||
deleted_at datetime,
|
||||
webhook_id uuid NOT NULL,
|
||||
path text NOT NULL,
|
||||
description text,
|
||||
active numeric DEFAULT true,
|
||||
PRIMARY KEY (id),
|
||||
CONSTRAINT fk_webhooks_entrypoints
|
||||
FOREIGN KEY (webhook_id) REFERENCES webhooks (id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_entrypoints_deleted_at ON entrypoints (deleted_at);
|
||||
CREATE UNIQUE INDEX idx_entrypoints_path ON entrypoints (path);
|
||||
|
||||
CREATE TABLE targets (
|
||||
id uuid,
|
||||
created_at datetime,
|
||||
updated_at datetime,
|
||||
deleted_at datetime,
|
||||
webhook_id uuid NOT NULL,
|
||||
name text NOT NULL,
|
||||
type text NOT NULL,
|
||||
active numeric DEFAULT true,
|
||||
config text,
|
||||
max_retries integer,
|
||||
PRIMARY KEY (id),
|
||||
CONSTRAINT fk_webhooks_targets
|
||||
FOREIGN KEY (webhook_id) REFERENCES webhooks (id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_targets_deleted_at ON targets (deleted_at);
|
||||
@@ -1,135 +0,0 @@
|
||||
// Package migrations holds the schema of each kind of SQLite file
|
||||
// webhooker writes, as numbered SQL files embedded in the binary, and
|
||||
// applies it when a file is opened.
|
||||
//
|
||||
// Each kind of file has its own directory. In each, 000_migration.sql
|
||||
// creates only the schema_migrations table, and 001_schema.sql holds
|
||||
// the whole schema. Before 1.0 a schema change is an edit to
|
||||
// 001_schema.sql, never a new file.
|
||||
package migrations
|
||||
|
||||
import (
|
||||
"embed"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"path"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// The directories, one for each kind of database file.
|
||||
const (
|
||||
// Main is the main database, webhooker.db.
|
||||
Main = "main"
|
||||
|
||||
// Events is a webhook's event database, events-{uuid}.db.
|
||||
Events = "events"
|
||||
|
||||
// Archive is a database target's archive file.
|
||||
Archive = "archive"
|
||||
)
|
||||
|
||||
//go:embed main/*.sql events/*.sql archive/*.sql
|
||||
var files embed.FS
|
||||
|
||||
// Apply runs each SQL file in dir, which is Main, Events or Archive,
|
||||
// that the database has not had yet, in the order of the number its
|
||||
// name starts with, and records that number in schema_migrations.
|
||||
// 000_migration.sql creates that table and records itself, so it runs
|
||||
// first, and only when the table is missing.
|
||||
//
|
||||
// It all runs in one transaction, so a failure leaves the file as it
|
||||
// was.
|
||||
func Apply(db *gorm.DB, dir string) error {
|
||||
return db.Transaction(func(tx *gorm.DB) error {
|
||||
var tables int
|
||||
|
||||
// Row().Scan, not (*gorm.DB).Scan: see internal/gormlog.
|
||||
err := tx.Raw(
|
||||
"SELECT count(*) FROM sqlite_master "+
|
||||
"WHERE type = 'table' AND name = ?",
|
||||
"schema_migrations",
|
||||
).Row().Scan(&tables)
|
||||
if err != nil {
|
||||
return fmt.Errorf("looking for schema_migrations: %w", err)
|
||||
}
|
||||
|
||||
if tables == 0 {
|
||||
err = run(tx, path.Join(dir, "000_migration.sql"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// fs.Glob lists the files in name order, which is number order.
|
||||
names, err := fs.Glob(files, path.Join(dir, "*.sql"))
|
||||
if err != nil {
|
||||
return fmt.Errorf("listing migrations: %w", err)
|
||||
}
|
||||
|
||||
for _, name := range names {
|
||||
err = applyOnce(tx, name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// applyOnce runs the migration file name and records its number,
|
||||
// unless schema_migrations already has that number.
|
||||
func applyOnce(tx *gorm.DB, name string) error {
|
||||
number, _, _ := strings.Cut(path.Base(name), "_")
|
||||
|
||||
version, err := strconv.Atoi(number)
|
||||
if err != nil {
|
||||
return fmt.Errorf("numbering migration %s: %w", name, err)
|
||||
}
|
||||
|
||||
var applied int
|
||||
|
||||
err = tx.Raw(
|
||||
"SELECT count(*) FROM schema_migrations WHERE version = ?",
|
||||
version,
|
||||
).Row().Scan(&applied)
|
||||
if err != nil {
|
||||
return fmt.Errorf("checking migration %s: %w", name, err)
|
||||
}
|
||||
|
||||
if applied > 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
err = run(tx, name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = tx.Exec(
|
||||
"INSERT INTO schema_migrations (version) VALUES (?)", version,
|
||||
).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("recording migration %s: %w", name, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// run executes every statement in the migration file name.
|
||||
func run(tx *gorm.DB, name string) error {
|
||||
script, err := files.ReadFile(name)
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading migration %s: %w", name, err)
|
||||
}
|
||||
|
||||
err = tx.Exec(string(script)).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("applying migration %s: %w", name, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -1,57 +0,0 @@
|
||||
package migrations_test
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"log/slog"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/driver/sqlite"
|
||||
"gorm.io/gorm"
|
||||
_ "modernc.org/sqlite"
|
||||
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
)
|
||||
|
||||
// TestApplyRunsEachFileOnce applies each kind of database file's
|
||||
// migrations to a new file twice. The first time records every file.
|
||||
// The second finds them all recorded and runs none, which is the only
|
||||
// way it can succeed: 001_schema.sql creates tables that then exist.
|
||||
func TestApplyRunsEachFileOnce(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, dir := range []string{
|
||||
migrations.Main, migrations.Events, migrations.Archive,
|
||||
} {
|
||||
t.Run(dir, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
sqlDB, err := sql.Open(
|
||||
"sqlite", filepath.Join(t.TempDir(), "test.db"),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
t.Cleanup(func() { _ = sqlDB.Close() })
|
||||
|
||||
db, err := gorm.Open(
|
||||
sqlite.Dialector{Conn: sqlDB},
|
||||
&gorm.Config{
|
||||
Logger: gormlog.New(slog.New(slog.DiscardHandler)),
|
||||
},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, migrations.Apply(db, dir))
|
||||
require.NoError(t, migrations.Apply(db, dir))
|
||||
|
||||
var versions string
|
||||
|
||||
require.NoError(t, sqlDB.QueryRowContext(t.Context(),
|
||||
"SELECT group_concat(version) FROM schema_migrations",
|
||||
).Scan(&versions))
|
||||
assert.Equal(t, "0,1", versions)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -22,7 +22,6 @@ import (
|
||||
_ "modernc.org/sqlite" // Pure Go SQLite driver.
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
)
|
||||
@@ -173,7 +172,9 @@ func seedArchiveFile(
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, migrations.Apply(gdb, migrations.Archive))
|
||||
require.NoError(
|
||||
t, gdb.AutoMigrate(&delivery.ExportArchivedEvent{}),
|
||||
)
|
||||
|
||||
for i, at := range archivedAt {
|
||||
row := delivery.ExportArchivedEvent{
|
||||
@@ -780,9 +781,9 @@ func TestArchiveSweep_NeverExpiryUntouched(t *testing.T) {
|
||||
//
|
||||
// The spec is stronger than that — a "never" archive is skipped
|
||||
// before any file is touched — so the archive here exists but has
|
||||
// never been migrated. Opening it at all would apply the archive
|
||||
// schema and create the archive table, which is exactly what must
|
||||
// not happen.
|
||||
// never been migrated. Opening it at all would run AutoMigrate
|
||||
// and create the archive table, which is exactly what must not
|
||||
// happen.
|
||||
func TestArchiveSweep_NeverExpirySkipsBeforeOpening(
|
||||
t *testing.T,
|
||||
) {
|
||||
@@ -806,7 +807,7 @@ func TestArchiveSweep_NeverExpirySkipsBeforeOpening(
|
||||
|
||||
// seedUnmigratedArchive creates an archive file that exists but
|
||||
// carries no archive schema, so any open of it is observable: the
|
||||
// archive table appears only if something applied the schema.
|
||||
// archive table appears only if something ran AutoMigrate.
|
||||
func seedUnmigratedArchive(t *testing.T, path string) {
|
||||
t.Helper()
|
||||
|
||||
|
||||
@@ -963,12 +963,29 @@ func TestFailUnretryableRetry_WritesNoTargetRow(
|
||||
database.DeliveryStatusFailed,
|
||||
)
|
||||
|
||||
// The per-webhook database has no targets table, so writing a
|
||||
// target row there would have failed the status write above.
|
||||
assert.False(t, s.WebhookDB.Migrator().HasTable("targets"),
|
||||
// The table exists in the per-webhook database because GORM
|
||||
// migrates the Delivery relation's model alongside it. It must
|
||||
// stay empty.
|
||||
var targetRows int64
|
||||
|
||||
require.NoError(t, s.WebhookDB.
|
||||
Table("targets").
|
||||
Count(&targetRows).Error)
|
||||
|
||||
assert.Zero(t, targetRows,
|
||||
"orphaned-retry terminal failure wrote a target row "+
|
||||
"into the per-webhook event database",
|
||||
)
|
||||
|
||||
var configs []string
|
||||
|
||||
require.NoError(t, s.WebhookDB.
|
||||
Table("targets").
|
||||
Pluck("config", &configs).Error)
|
||||
|
||||
assert.NotContains(
|
||||
t, strings.Join(configs, " "), hookURL,
|
||||
)
|
||||
}
|
||||
|
||||
func TestRecoverSingleRetry_UnknownTargetType(
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
@@ -21,11 +22,12 @@ import (
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/driver/sqlite"
|
||||
"gorm.io/gorm"
|
||||
_ "modernc.org/sqlite"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
"sneak.berlin/go/webhooker/internal/metrics"
|
||||
)
|
||||
|
||||
@@ -35,18 +37,35 @@ const testContentType = "application/json"
|
||||
func testWebhookDB(t *testing.T) *gorm.DB {
|
||||
t.Helper()
|
||||
|
||||
// Opened by the manager the service uses, so these tests cannot
|
||||
// pass against a schema, journal and locking settings or write
|
||||
// guard production does not use.
|
||||
mgr := databasetest.NewWebhookDBManagerWithLogger(
|
||||
t, t.TempDir(), slog.New(slog.DiscardHandler),
|
||||
dbPath := filepath.Join(
|
||||
t.TempDir(), "events-test.db",
|
||||
)
|
||||
|
||||
t.Cleanup(func() { _ = mgr.CloseAll() })
|
||||
|
||||
db, err := mgr.GetDB(uuid.New().String())
|
||||
// Opened the way the service opens a per-webhook database, so
|
||||
// these tests cannot pass against journal and locking settings
|
||||
// production does not use.
|
||||
sqlDB, err := database.OpenSQLite(
|
||||
dbPath, database.SQLiteModeCreate,
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
t.Cleanup(func() { _ = sqlDB.Close() })
|
||||
|
||||
db, err := gorm.Open(
|
||||
sqlite.Dialector{Conn: sqlDB},
|
||||
&gorm.Config{Logger: gormlog.New(slog.New(slog.DiscardHandler))},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, db.AutoMigrate(
|
||||
&database.Event{},
|
||||
&database.Delivery{},
|
||||
&database.DeliveryResult{},
|
||||
&database.EventTotals{},
|
||||
&database.TargetTotals{},
|
||||
))
|
||||
require.NoError(t, db.Create(&database.EventTotals{}).Error)
|
||||
|
||||
return db
|
||||
}
|
||||
|
||||
|
||||
@@ -14,7 +14,6 @@ import (
|
||||
"gorm.io/driver/sqlite"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
)
|
||||
|
||||
@@ -336,7 +335,7 @@ func (w *archiveWriter) openMode(
|
||||
)
|
||||
}
|
||||
|
||||
err = migrations.Apply(gdb, migrations.Archive)
|
||||
err = gdb.AutoMigrate(&archivedEvent{})
|
||||
if err != nil {
|
||||
_ = sqlDB.Close()
|
||||
|
||||
|
||||
@@ -312,9 +312,13 @@ func TestFailMissingTarget_WritesNoTargetRow(
|
||||
database.DeliveryStatusFailed,
|
||||
)
|
||||
|
||||
// The per-webhook database has no targets table, so writing a
|
||||
// target row there would have failed the status write above.
|
||||
assert.False(t, s.WebhookDB.Migrator().HasTable("targets"),
|
||||
var configs []string
|
||||
|
||||
require.NoError(t, s.WebhookDB.
|
||||
Table("targets").
|
||||
Pluck("config", &configs).Error)
|
||||
|
||||
assert.Empty(t, configs,
|
||||
"the deleted-target terminal path wrote a target row "+
|
||||
"into the per-webhook event database",
|
||||
)
|
||||
|
||||
@@ -135,11 +135,9 @@ func openDB(
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, gdb.Exec(
|
||||
"CREATE TABLE things (id text PRIMARY KEY, name text)",
|
||||
).Error)
|
||||
require.NoError(t, gdb.AutoMigrate(&thing{}))
|
||||
|
||||
// Creating the table is not what any of these cases is about.
|
||||
// Migration chatter is not what any of these cases is about.
|
||||
buf.Reset()
|
||||
|
||||
return gdb
|
||||
|
||||
@@ -115,8 +115,8 @@ func skipDir(name string) bool {
|
||||
}
|
||||
|
||||
// walkNonTestGo parses every non-test .go file under root. It returns
|
||||
// the directories, relative to root, it parsed a file in or below,
|
||||
// along with every unguarded Scan it found.
|
||||
// the directories, relative to root, it parsed a file in, along with
|
||||
// every unguarded Scan it found.
|
||||
func walkNonTestGo(t *testing.T, root string) (map[string]bool, []string) {
|
||||
t.Helper()
|
||||
|
||||
@@ -155,12 +155,7 @@ func walkNonTestGo(t *testing.T, root string) (map[string]bool, []string) {
|
||||
return err
|
||||
}
|
||||
|
||||
// The directories above count too, so that one holding
|
||||
// only a nested package, as internal/db does, is not
|
||||
// taken for one the walk skipped.
|
||||
for ; dir != "."; dir = filepath.Dir(dir) {
|
||||
walked[dir] = true
|
||||
}
|
||||
walked[dir] = true
|
||||
|
||||
for _, pos := range unguardedScans(fset, file) {
|
||||
hits = append(hits, relPosition(root, pos))
|
||||
@@ -210,9 +205,9 @@ func TestGormScanIsNeverCalledOutsideTests(t *testing.T) {
|
||||
walked, offenders := walkNonTestGo(t, root)
|
||||
|
||||
// The module's packages are static, templates, and every directory
|
||||
// directly under cmd and internal. Each holds non-test code, itself
|
||||
// or in a package below it, so one the walk parsed nothing in or
|
||||
// below was skipped, and a Scan there would pass unseen.
|
||||
// directly under cmd and internal. Each holds non-test code, so one
|
||||
// the walk parsed nothing in was skipped, and a Scan there would
|
||||
// pass unseen.
|
||||
packages := []string{"static", "templates"}
|
||||
|
||||
for _, parent := range []string{"cmd", "internal"} {
|
||||
|
||||
@@ -42,11 +42,8 @@ func TestDeliveryAttempts_ReadInTheTargetTypesOwnTerms(t *testing.T) {
|
||||
{database.TargetTypeLog, true, 0, "", "written to the log", ""},
|
||||
}
|
||||
|
||||
// Named for type and outcome, so that no two cases share a name:
|
||||
// Go adds "#01" to a repeated one, and a "#" in the temporary
|
||||
// directory's path cuts the path SQLiteDSN gives SQLite short.
|
||||
for _, tc := range cases {
|
||||
t.Run(string(tc.targetType)+" "+tc.outcome, func(t *testing.T) {
|
||||
t.Run(string(tc.targetType), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newRecentEventsFixture(t)
|
||||
|
||||
@@ -314,15 +314,21 @@ func assertReplayTask(
|
||||
)
|
||||
}
|
||||
|
||||
// assertNoLeakedTarget proves the per-webhook database holds no
|
||||
// target: an association write would have upserted the whole target,
|
||||
// plaintext config and all, into the event database. See
|
||||
// assertNoLeakedTarget proves the per-webhook database holds no target
|
||||
// rows. AutoMigrate creates the table there because Delivery declares
|
||||
// the relation, so it is a ROW that signals a leak: an association
|
||||
// write would have upserted the whole target, plaintext config and
|
||||
// all, into the event database. See
|
||||
// https://git.eeqj.de/sneak/webhooker/issues/206.
|
||||
func assertNoLeakedTarget(t *testing.T, webhookDB *gorm.DB) {
|
||||
t.Helper()
|
||||
|
||||
assert.False(
|
||||
t, webhookDB.Migrator().HasTable(&database.Target{}),
|
||||
var leaked int64
|
||||
|
||||
require.NoError(t, webhookDB.Unscoped().
|
||||
Model(&database.Target{}).Count(&leaked).Error)
|
||||
assert.Zero(
|
||||
t, leaked,
|
||||
"replay must not write the target into the event database",
|
||||
)
|
||||
}
|
||||
|
||||
@@ -91,6 +91,24 @@ func NewRouterForTest(
|
||||
return s.router
|
||||
}
|
||||
|
||||
// NewRouterWithSentryForTest is NewRouterForTest with the sentryhttp
|
||||
// handler registered, as a set SENTRY_DSN does in production.
|
||||
func NewRouterWithSentryForTest(
|
||||
t *testing.T,
|
||||
log *logger.Logger,
|
||||
cfg *config.Config,
|
||||
mw *middleware.Middleware,
|
||||
h *handlers.Handlers,
|
||||
) http.Handler {
|
||||
t.Helper()
|
||||
|
||||
s := newServerForTest(t, log, cfg, mw, h)
|
||||
s.sentryEnabled.Store(true)
|
||||
s.SetupRoutes()
|
||||
|
||||
return s.router
|
||||
}
|
||||
|
||||
// ListenAddrForTest exposes the address the HTTP listener binds for
|
||||
// a given Config, so the rendering of host and port — IPv6
|
||||
// bracketing above all — can be pinned without standing up a
|
||||
|
||||
@@ -90,8 +90,8 @@ func (s *Server) setupGlobalMiddleware() {
|
||||
|
||||
// recoverPanics installs on r the recoverer, answering a panic with
|
||||
// page (a plain 500 when page is nil), and inside it the Sentry error
|
||||
// reporting (if SENTRY_DSN is set). Repanic is true so panics still
|
||||
// bubble up to the recoverer.
|
||||
// reporting and transaction naming (if SENTRY_DSN is set). Repanic is
|
||||
// true so panics still bubble up to the recoverer.
|
||||
//
|
||||
// Each admin page route group installs its own, with the error page,
|
||||
// as its first middleware. A panic there is logged, reported and
|
||||
@@ -105,6 +105,11 @@ func (s *Server) recoverPanics(r chi.Router, page http.Handler) {
|
||||
Repanic: true,
|
||||
})
|
||||
r.Use(sentryHandler.Handle)
|
||||
// Installed with every sentryhttp handler rather than once
|
||||
// globally: in an admin page route group, the group's own
|
||||
// handler finishes the transaction as the group returns, so
|
||||
// a rename further out would come too late.
|
||||
r.Use(nameSentryTransaction)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
@@ -71,10 +72,15 @@ func scrubSentryRequest(
|
||||
|
||||
pattern := sentryRoutePattern(hint)
|
||||
|
||||
// Only transaction events carry a Transaction name, and the SDK
|
||||
// Only transaction events carry a Transaction name. sentryhttp
|
||||
// builds it from the concrete path too (sentryhttp.go:105 via
|
||||
// tracing.go:553). Rewritten on the same terms.
|
||||
if event.Transaction != "" {
|
||||
// tracing.go:553) and marks it SourceURL, so it is rewritten on
|
||||
// the same terms. Only a name nameSentryTransaction set from the
|
||||
// route pattern, which it marks SourceRoute, is kept.
|
||||
namedByRoute := event.TransactionInfo != nil &&
|
||||
event.TransactionInfo.Source == sentry.SourceRoute
|
||||
|
||||
if event.Transaction != "" && !namedByRoute {
|
||||
event.Transaction = sentryTransactionName(
|
||||
event.Transaction, pattern,
|
||||
)
|
||||
@@ -120,7 +126,8 @@ func scrubSentryRequest(
|
||||
// The transaction dispatch has no such request: Span.doFinish calls
|
||||
// hub.CaptureEvent (tracing.go:356), which passes a nil hint that the
|
||||
// client replaces with an empty one (client.go:620-622). The pattern
|
||||
// is therefore always "" there, and the callers fall back.
|
||||
// is therefore always "" there, and the callers fall back. A routed
|
||||
// transaction is named by nameSentryTransaction instead.
|
||||
func sentryRoutePattern(hint *sentry.EventHint) string {
|
||||
if hint == nil || hint.Context == nil {
|
||||
return ""
|
||||
@@ -142,6 +149,52 @@ func sentryRoutePattern(hint *sentry.EventHint) string {
|
||||
return rctx.RoutePattern()
|
||||
}
|
||||
|
||||
// nameSentryTransaction renames the request's Sentry transaction to
|
||||
// "METHOD <route pattern>", such as "POST /h/{uuid}". sentryhttp names
|
||||
// it after the concrete path, which scrubSentryRequest withholds;
|
||||
// since Sentry groups transactions by name, every transaction in the
|
||||
// service would otherwise land in one group per method, such as
|
||||
// "POST /(redacted)".
|
||||
//
|
||||
// It has to run inside the sentryhttp handler, where the transaction
|
||||
// is on the request context. It renames on the way out, once chi has
|
||||
// routed the request, and does so in a defer so that a request whose
|
||||
// handler panics is named too. The context is taken when the defer is
|
||||
// set up, before routing, but the chi routing context on it is a
|
||||
// pointer that chi fills in as it routes.
|
||||
func nameSentryTransaction(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
defer setSentryTransactionName(r.Context(), r.Method)
|
||||
|
||||
next.ServeHTTP(w, r)
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
// setSentryTransactionName does the rename for nameSentryTransaction.
|
||||
// A path that matches nothing at the top level has no pattern; its
|
||||
// name is left as the SDK set it, for scrubSentryRequest to withhold.
|
||||
// Inside a route group, a path that matches no route in the group, or
|
||||
// a request the group's own middleware turns away before routing, has
|
||||
// the group's pattern, such as "/pages/*", and is named after it.
|
||||
func setSentryTransactionName(ctx context.Context, method string) {
|
||||
transaction := sentry.TransactionFromContext(ctx)
|
||||
rctx := chi.RouteContext(ctx)
|
||||
|
||||
if transaction == nil || rctx == nil {
|
||||
return
|
||||
}
|
||||
|
||||
pattern := rctx.RoutePattern()
|
||||
if pattern == "" {
|
||||
return
|
||||
}
|
||||
|
||||
transaction.Name = method + " " + pattern
|
||||
transaction.Source = sentry.SourceRoute
|
||||
}
|
||||
|
||||
// sentryRouteURL rebuilds an event's request URL with the route
|
||||
// pattern in place of the concrete path.
|
||||
//
|
||||
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
"github.com/go-chi/chi"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/server"
|
||||
)
|
||||
|
||||
@@ -130,11 +131,15 @@ func (c sentryCase) capture(t *testing.T) []*sentry.Event {
|
||||
// router mirrors the one ordering these tests depend on, over the two
|
||||
// route patterns they need: a recovering middleware outside, then the
|
||||
// sentryhttp handler registered with Use and Repanic set, exactly as
|
||||
// routes.go orders the two. The bare recover stands in for
|
||||
// Middleware.Recoverer, which holds that outer slot in production; it
|
||||
// is here only to keep panic stacks out of the test output. That the
|
||||
// production one really does catch what sentryhttp re-raises is
|
||||
// pinned separately, by TestSentryStillSeesAPanic.
|
||||
// routes.go orders the two. It leaves out nameSentryTransaction,
|
||||
// which routes.go installs inside the sentryhttp handler, so a
|
||||
// transaction captured here keeps the SDK's name and shows what the
|
||||
// hook does with a transaction that missed the rename. The bare
|
||||
// recover stands in for Middleware.Recoverer, which holds that outer
|
||||
// slot in production; it is here only to keep panic stacks out of the
|
||||
// test output. That the production one really does catch what
|
||||
// sentryhttp re-raises is pinned separately, by
|
||||
// TestSentryStillSeesAPanic.
|
||||
func (c sentryCase) router() http.Handler {
|
||||
handler := func(_ http.ResponseWriter, r *http.Request) {
|
||||
// This call is what drains the body tee and fills the
|
||||
@@ -353,8 +358,9 @@ func TestSentryScrub_KeepsTheRoutingContext(t *testing.T) {
|
||||
// Span.doFinish captures with a nil hint, so BeforeSendTransaction
|
||||
// gets one with no context and no request: the route pattern is out of
|
||||
// reach and both the URL and the SDK-built transaction name have to
|
||||
// fall back. Tracing is off in this service, so no transaction event
|
||||
// is produced today; the hook is a floor against that changing.
|
||||
// fall back. This router has no nameSentryTransaction, so this is the
|
||||
// floor under a transaction that missed the rename;
|
||||
// TestSentryTransaction_NamedByRoute covers the routes as they ship.
|
||||
func TestSentryScrub_RedactsTheTransactionDispatch(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -376,6 +382,83 @@ func TestSentryScrub_RedactsTheTransactionDispatch(t *testing.T) {
|
||||
assert.Equal(t, "POST /(redacted)", event.Transaction)
|
||||
}
|
||||
|
||||
// TestSentryTransaction_NamedByRoute serves requests through the real
|
||||
// route tree with tracing on, which the service leaves off. A routed
|
||||
// request's transaction is named after its route pattern, so Sentry
|
||||
// can group transactions by route, and the entrypoint UUID in the
|
||||
// receiver path appears nowhere in the event. A path that matches
|
||||
// nothing at the top level has no pattern, and its name is still
|
||||
// withheld; one that matches no route inside a route group is named
|
||||
// after the group's pattern.
|
||||
func TestSentryTransaction_NamedByRoute(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// newTestEnv leaves the receiver rate limit at zero, which
|
||||
// refuses every request before it reaches the receiver.
|
||||
env := newTestEnvWithConfig(t, &config.Config{
|
||||
DataDir: t.TempDir(),
|
||||
Environment: config.EnvironmentDev,
|
||||
ReceiverRateLimit: 10,
|
||||
})
|
||||
|
||||
router := server.NewRouterWithSentryForTest(
|
||||
t, env.log, env.cfg, env.mw, env.hnd,
|
||||
)
|
||||
|
||||
for _, tc := range []struct {
|
||||
method, path, want string
|
||||
}{
|
||||
{
|
||||
http.MethodPost,
|
||||
"/h/" + sentryReceiverUUID,
|
||||
"POST /h/{uuid}",
|
||||
},
|
||||
// An admin page route group, which installs its own
|
||||
// sentryhttp handler.
|
||||
{http.MethodGet, "/pages/login", "GET /pages/login"},
|
||||
{http.MethodGet, "/pages/unrouted", "GET /pages/*"},
|
||||
{
|
||||
http.MethodPost,
|
||||
"/h/" + sentryReceiverUUID + "/unrouted",
|
||||
"POST /(redacted)",
|
||||
},
|
||||
} {
|
||||
t.Run(tc.want, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
transport := &captureTransport{}
|
||||
|
||||
opts := server.SentryClientOptionsForTest(
|
||||
"https://public@sentry.invalid/1",
|
||||
"webhooker-test",
|
||||
)
|
||||
opts.Transport = transport
|
||||
opts.EnableTracing = true
|
||||
opts.TracesSampleRate = 1.0
|
||||
|
||||
client, err := sentry.NewClient(opts)
|
||||
require.NoError(t, err)
|
||||
|
||||
req := httptest.NewRequestWithContext(
|
||||
sentry.SetHubOnContext(
|
||||
context.Background(),
|
||||
sentry.NewHub(client, sentry.NewScope()),
|
||||
),
|
||||
tc.method, tc.path, nil,
|
||||
)
|
||||
router.ServeHTTP(httptest.NewRecorder(), req)
|
||||
|
||||
event := onlyEvent(t, transport.events)
|
||||
require.Equal(t, "transaction", event.Type)
|
||||
|
||||
assert.Equal(t, tc.want, event.Transaction)
|
||||
assert.NotContains(
|
||||
t, marshalEvent(t, event), sentryReceiverUUID,
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestSentryScrub_TransactionDispatchIsUnscrubbedWithoutTheHook pins
|
||||
// that dispatch's premise the same way, since it is the one the
|
||||
// service does not exercise today.
|
||||
|
||||
Reference in New Issue
Block a user