Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 clawbot
|
677461448f |
@@ -1071,27 +1071,23 @@ beside it, or wait until there are none.
|
||||
|
||||
4. Start the service. The container gives the directory and the restored files
|
||||
to the `webhooker` user before the app starts, whoever restored them (see
|
||||
[Running with Docker](#running-with-docker)). Each restored database gets any
|
||||
of its SQL migrations it has not had as it is opened (see
|
||||
[Upgrades](#upgrades)).
|
||||
[Running with Docker](#running-with-docker)). `AutoMigrate` runs against each
|
||||
restored database as it is opened.
|
||||
|
||||
### Upgrades
|
||||
|
||||
**Back up before every upgrade.** Each kind of database file has its schema in
|
||||
SQL files built into the binary, one directory per kind under
|
||||
`internal/db/migrations/`. In each, `000_migration.sql` creates the
|
||||
`schema_migrations` table, which records the number of every file the database
|
||||
has had, and `001_schema.sql` holds the whole schema. Opening a database runs,
|
||||
in one transaction, every file in its directory that `schema_migrations` does
|
||||
not list:
|
||||
**Back up before every upgrade.** Every start runs GORM `AutoMigrate`
|
||||
unconditionally against whatever files it finds:
|
||||
|
||||
- `main/` for the main database, on connect — settings, users, API keys,
|
||||
webhooks, entrypoints and targets
|
||||
- `events/` for each event database, when it is lazily opened — events,
|
||||
deliveries, delivery results and their running totals
|
||||
- `archive/` for each archive database, on every open and reopen
|
||||
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
|
||||
`Entrypoint`, `Target`
|
||||
- each event database when it is lazily opened — `Event`, `Delivery`,
|
||||
`DeliveryResult`, `EventTotals`, `TargetTotals`, `EntrypointTotals`
|
||||
- each archive database on every open and reopen
|
||||
|
||||
There are no down migrations.
|
||||
There is no schema version table, no migration ledger, and no down migrations.
|
||||
Nothing in the files records which version wrote them, and no code path undoes a
|
||||
migration.
|
||||
|
||||
Upgrade procedure:
|
||||
|
||||
@@ -1116,9 +1112,12 @@ connects to. Check the `bindAddress` field of the startup log to see what a
|
||||
running process bound. See [Bind address](#bind-address).
|
||||
|
||||
**Downgrade is unsupported.** Once a newer binary has migrated the files there
|
||||
is no way to move them back. The only supported way back to an older version is
|
||||
restoring the pre-upgrade backup, which discards everything received since that
|
||||
backup was taken.
|
||||
is no way to move them back. `AutoMigrate` is additive — it adds tables, columns
|
||||
and indexes and never drops or rewrites them — so an older binary will generally
|
||||
open migrated files and appear to work while writing against a schema it does
|
||||
not know about. The failure mode is silent divergence, not a startup error. The
|
||||
only supported way back to an older version is restoring the pre-upgrade backup,
|
||||
which discards everything received since that backup was taken.
|
||||
|
||||
### Version stamping
|
||||
|
||||
@@ -1183,6 +1182,26 @@ at rest and restrict who can read them.
|
||||
- `events-{uuid}.db` and `archive-….db` hold the **full payload body and
|
||||
headers** of every event as received, including whatever the sending service
|
||||
put in them — tokens, signatures, personal data.
|
||||
- Event databases written before
|
||||
[issue #206](https://git.eeqj.de/sneak/webhooker/issues/206) was fixed **also
|
||||
contain target credentials**: a GORM association upsert on the delivery and
|
||||
retry write path copied `targets` rows, `config` included, into the
|
||||
per-webhook database. For a Slack target the `webhookUrl` _is_ the bearer
|
||||
credential, and an `http` target's URL can embed userinfo. This version never
|
||||
writes those rows; the first time it opens such a file it deletes them and
|
||||
vacuums the file, which removes the credential bytes rather than only
|
||||
unlinking the rows. Deleting alone would not: the bytes stay readable in the
|
||||
file's free pages until it is rewritten. The sweep is recorded in the file's
|
||||
`user_version` only once the vacuum returns, so a sweep that fails or is
|
||||
interrupted fails the open and is retried on the next one, and a file this
|
||||
version has opened without error holds no leaked rows and no recoverable bytes
|
||||
from them. On upgrade this rewrites each existing `events-{uuid}.db` once, on
|
||||
its first open. Two cases still hand over live delivery destinations: a backup
|
||||
taken from an older build, and a backup of a file this version has not yet
|
||||
opened successfully. Copies already made stay affected — the sweep only
|
||||
rewrites the file it opens, and freed blocks may persist in filesystem
|
||||
snapshots and on the underlying storage. Rotate any target credential that was
|
||||
in a backup you cannot account for.
|
||||
- `webhooker.db` stores target config **unencrypted**, tracked at
|
||||
[issue #212](https://git.eeqj.de/sneak/webhooker/issues/212), next to the
|
||||
session encryption key and the Argon2id password hashes.
|
||||
@@ -1900,9 +1919,8 @@ Deliveries still `pending` or `retrying` count in neither.
|
||||
|
||||
#### Event-tier indexes
|
||||
|
||||
These indexes on the per-webhook event databases are created by
|
||||
`internal/db/migrations/events/001_schema.sql`, and declared in the model tags
|
||||
to match:
|
||||
These indexes on the per-webhook event databases are declared in the model tags,
|
||||
so `AutoMigrate` creates them on a fresh database:
|
||||
|
||||
| Table | Columns | Serves |
|
||||
| ------------------ | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
@@ -1966,9 +1984,8 @@ and application state:
|
||||
- **Targets** — delivery destination configurations
|
||||
- **APIKeys** — programmatic access credentials
|
||||
|
||||
On first startup the main database is created from its SQL files (see
|
||||
[Upgrades](#upgrades)), a session encryption key is generated and stored, and an
|
||||
`admin` user is created.
|
||||
On first startup the main database is auto-migrated, a session encryption key is
|
||||
generated and stored, and an `admin` user is created.
|
||||
|
||||
**Per-Webhook Event Databases** (`{DATA_DIR}/events-{webhook_uuid}.db`) — each
|
||||
webhook gets its own dedicated SQLite file containing:
|
||||
@@ -1980,10 +1997,9 @@ webhook gets its own dedicated SQLite file containing:
|
||||
the above, the deliveries per target, and each entrypoint's last event, kept
|
||||
through retention
|
||||
|
||||
Per-webhook databases are created automatically when a webhook is created, from
|
||||
the SQL files in `internal/db/migrations/events/`. They hold no copy of the main
|
||||
database's tables. They are managed by the `WebhookDBManager` component, which
|
||||
handles connection pooling, lazy opening, migrations, and cleanup.
|
||||
Per-webhook databases are created automatically when a webhook is created. They
|
||||
are managed by the `WebhookDBManager` component, which handles connection
|
||||
pooling, lazy opening, migrations, and cleanup.
|
||||
|
||||
A per-webhook database that is missing or zero-length later means its webhook's
|
||||
events and pending deliveries are gone. The next time it is opened, an empty one
|
||||
@@ -2067,15 +2083,14 @@ the service again.
|
||||
After each write the archive handle is closed and reopened, debounced to at most
|
||||
once per second, so an operator can move the archive file away for offline
|
||||
archiving without stopping the service; a moved or removed archive file is
|
||||
recreated automatically on the next write, from the SQL files in
|
||||
`internal/db/migrations/archive/`. A zero-length archive file is written to as a
|
||||
new archive: SQLite opens it as an empty database, so it holds nothing to lose.
|
||||
An optional `expiry` in the target's config JSON (e.g. `{"expiry":"720h"}`) is
|
||||
validated when the target is created — the default (unset or the literal
|
||||
`never`) keeps rows forever — and rows older than the expiry are pruned each
|
||||
time the archive is (re)opened. An archive write failure is never silent
|
||||
success: the delivery records a failed attempt with the error and is marked
|
||||
failed.
|
||||
recreated automatically on the next write. A zero-length archive file is written
|
||||
to as a new archive: SQLite opens it as an empty database, so it holds nothing
|
||||
to lose. An optional `expiry` in the target's config JSON (e.g.
|
||||
`{"expiry":"720h"}`) is validated when the target is created — the default
|
||||
(unset or the literal `never`) keeps rows forever — and rows older than the
|
||||
expiry are pruned each time the archive is (re)opened. An archive write failure
|
||||
is never silent success: the delivery records a failed attempt with the error
|
||||
and is marked failed.
|
||||
|
||||
Because reopens only happen on writes, an archive whose target has stopped
|
||||
receiving events would never be pruned. A background **archive sweeper** closes
|
||||
@@ -3004,6 +3019,7 @@ webhooker/
|
||||
│ ├── database/
|
||||
│ │ ├── base_model.go # BaseModel with UUID primary keys
|
||||
│ │ ├── database.go # GORM connection, migrations, admin seed
|
||||
│ │ ├── models.go # AutoMigrate for config-tier models
|
||||
│ │ ├── model_setting.go # Setting entity (key-value app config)
|
||||
│ │ ├── model_user.go # User entity
|
||||
│ │ ├── model_webhook.go # Webhook entity
|
||||
@@ -3020,11 +3036,6 @@ webhooker/
|
||||
│ │ └── databasetest/ # Test support: a WebhookDBManager for tests in other packages
|
||||
│ ├── datadir/
|
||||
│ │ └── lock.go # Exclusive advisory lock on DATA_DIR (one instance)
|
||||
│ ├── db/migrations/
|
||||
│ │ ├── migrations.go # Applies a database file's SQL files when it is opened
|
||||
│ │ ├── main/ # 000_migration.sql and 001_schema.sql for webhooker.db
|
||||
│ │ ├── events/ # The same for each events-{uuid}.db
|
||||
│ │ └── archive/ # The same for each archive file
|
||||
│ ├── globals/
|
||||
│ │ └── globals.go # Build-time variables (appname, version, arch)
|
||||
│ ├── gormlog/
|
||||
|
||||
@@ -12,24 +12,14 @@ One issue per unit of work, one branch and one PR per issue:
|
||||
`main` without notice. One `next` -> `main` PR accumulates the milestone;
|
||||
releases are cut from `main` separately.
|
||||
|
||||
Issue branches do NOT touch this file — the manager maintains it on `next`.
|
||||
Every branch editing `TODO.md` conflicts with every other (#112).
|
||||
The tracker (https://git.eeqj.de/sneak/webhooker/issues) is the record of what
|
||||
is done and what is under way. This file keeps no list of either.
|
||||
|
||||
# Status
|
||||
|
||||
The milestone (https://git.eeqj.de/sneak/webhooker/milestone/9) is the
|
||||
authoritative list, and the only place to read a count or a state of play from.
|
||||
This file records where the project is, not what is in flight: a sentence whose
|
||||
truth depends on a branch being unmerged is wrong the moment it merges, and this
|
||||
file has been wrong that way before.
|
||||
|
||||
The durability defect that held the tag has landed
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/256, commit `8d64259`). Every SQLite
|
||||
handle opens with WAL journaling and a busy timeout, a bookkeeping write that
|
||||
fails leaves its delivery in a recoverable state rather than a lying one, and
|
||||
recovery skips a delivery that already has a successful result row. Final
|
||||
pre-tag verification exercised it and confirmed it holds. Whatever the milestone
|
||||
still shows open is what remains before `v1.0.0`.
|
||||
What is still open for 1.0.0 is in its milestone
|
||||
(https://git.eeqj.de/sneak/webhooker/milestone/9). `v1.0.0` is not tagged, and
|
||||
only @sneak tags it.
|
||||
|
||||
Delivery is at-least-once by design, not by accident: a send whose result row
|
||||
does not land is attempted again, so a receiver can see a duplicate. That is
|
||||
@@ -38,338 +28,31 @@ under Rationale. It is not a defect to re-file.
|
||||
|
||||
# Next Step
|
||||
|
||||
Clear the rest of the open 1.0.0 milestone
|
||||
(https://git.eeqj.de/sneak/webhooker/milestone/9) and tag `v1.0.0`. Merging
|
||||
`next` into `main` is a separate act from tagging and waits on neither of those:
|
||||
`next` is kept mergeable at all times, which is the point of the branch.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-08-24 Bind the plaintext HTTP listener deliberately, via `BIND_ADDRESS`
|
||||
defaulting to `127.0.0.1`, and document the reverse-proxy deployment. A
|
||||
hostname, an empty value or a value carrying a port is a startup error, and
|
||||
the `Dockerfile` sets `0.0.0.0` because a loopback bind inside a container is
|
||||
unreachable (https://git.eeqj.de/sneak/webhooker/issues/268). The same commit
|
||||
removed the shutdown race: `httpServer` is built in the constructor rather
|
||||
than assigned from the serving goroutine, which orders the write before every
|
||||
fx hook and rules out the nil dereference a SIGTERM arriving first would have
|
||||
caused, and `sentryEnabled` is an `atomic.Bool`
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/226)
|
||||
- 2026-08-24 Remove inbound request signature verification. The entrypoint UUID
|
||||
is the authentication secret, so the per-entrypoint shared secret, the
|
||||
`internal/signature` package, the receiver check, the model fields and the
|
||||
forms are all gone. This reverses the feature that landed earlier in the same
|
||||
milestone (https://git.eeqj.de/sneak/webhooker/issues/67,
|
||||
https://git.eeqj.de/sneak/webhooker/issues/279)
|
||||
- 2026-08-24 Stamp the build version into the binary and render it in the UI
|
||||
footer. `script/version` is the single source — `$VERSION`, else
|
||||
`git describe --tags --always --dirty`, else `unknown` — so a `make build`
|
||||
binary and a `make docker` image from one checkout report the same thing, and
|
||||
nothing in it varies between two builds of the same commit, which the release
|
||||
gate's byte-identical assertion would catch
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/253)
|
||||
- 2026-08-24 Derive cookie `Secure` and CSRF strictness from the request
|
||||
transport rather than from `WEBHOOKER_ENVIRONMENT`. Behind a real TLS proxy
|
||||
with the environment left at its `dev` default, the session cookie silently
|
||||
lost `Secure` while the CSRF cookie on the same response kept it.
|
||||
`X-Forwarded-Proto` is now matched case-insensitively on its first
|
||||
comma-separated element, so `HTTPS` and `https, http` no longer fall to the
|
||||
relaxed CSRF path (https://git.eeqj.de/sneak/webhooker/issues/269)
|
||||
- 2026-08-24 Roll back a failed webhook deletion instead of committing it. A
|
||||
failing delete committed whatever had already succeeded, hard-deleted the
|
||||
per-webhook event database anyway, and redirected as though it had worked —
|
||||
orphaned config plus permanently destroyed history, reported as success. All
|
||||
three delete positions now roll back with the event database intact
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/262)
|
||||
- 2026-08-24 Name a deleted target on its historical deliveries, marked
|
||||
`(deleted)`, rather than leaving the event log unable to say where a delivery
|
||||
went. A deleted target's credentials stay masked exactly as a live one's, and
|
||||
it cannot become deliverable again through the receiver, resubmit, replay, the
|
||||
edit form or the toggle (https://git.eeqj.de/sneak/webhooker/issues/211)
|
||||
- 2026-08-24 Bound both request-controlled `/metrics` label dimensions, so the
|
||||
unauthenticated receiver is no longer a memory-exhaustion vector: `handler`
|
||||
carries the chi route pattern, and `method` folds anything chi cannot route
|
||||
onto a single `(unmatched)` sentinel. Both were reproduced before the fix —
|
||||
300 random method tokens took the series count from 106 to 7,631, and path
|
||||
flooding reached 62,532 — and a label audit across a live scrape found no
|
||||
third unbounded dimension (https://git.eeqj.de/sneak/webhooker/issues/254,
|
||||
https://git.eeqj.de/sneak/webhooker/issues/261)
|
||||
- 2026-08-24 Validate `max_retries` on both target forms. `abc`, `2.7` and `-5`
|
||||
silently became 0 — fire-and-forget — including on the edit path, where it
|
||||
destroyed a working value, and `999999999` stored verbatim. The ceiling of 20
|
||||
is the `max` both templates already declared
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/221)
|
||||
- 2026-08-24 Resubmit a stored event as a new undelivered event, so a backend
|
||||
under development can be tested against real captured traffic. Per-delivery
|
||||
replay cannot serve that: it re-sends one finished delivery to its own
|
||||
original target, and a target created for a dev backend has no prior delivery
|
||||
to replay. Resubmit re-injects the stored event at the top of the receiver
|
||||
path and fans it out to whatever targets are active now
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/250)
|
||||
- 2026-08-20 Take an exclusive lock on `DATA_DIR` at startup, so two instances
|
||||
on one directory cannot both deliver
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/201)
|
||||
- 2026-08-20 Shut down the app when the HTTP listener fails. The `OnStart` hook
|
||||
returned as soon as the serving goroutine was spawned, so a failed listen left
|
||||
fx reporting RUNNING and a live process with nothing bound — invisible to
|
||||
systemd and Docker restart policies
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/200)
|
||||
- 2026-08-20 Stop target credentials leaking into the per-webhook event
|
||||
databases (https://git.eeqj.de/sneak/webhooker/issues/206), log SQL with
|
||||
placeholders rather than bound values
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/207), and fail loudly on half-set
|
||||
metrics auth credentials (https://git.eeqj.de/sneak/webhooker/issues/205)
|
||||
- 2026-08-20 Read queue depths with `Find`, not `Scan`. `Scan` swaps GORM's own
|
||||
trace recorder in for the logging adapter, and that recorder does not
|
||||
implement `gorm.ParamsFilter`, so those statements logged their bound values
|
||||
interpolated and bypassed the suppression above. The two units gated green
|
||||
against a `next` that lacked the other, and `next` went red when both landed
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/234)
|
||||
- 2026-08-20 Render per-attempt delivery detail in the event log
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/202) and add replay of a
|
||||
terminally failed delivery (https://git.eeqj.de/sneak/webhooker/issues/203)
|
||||
- 2026-08-20 Expose delivery metrics on `/metrics`
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/209) and document the backup,
|
||||
restore and upgrade procedures
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/210)
|
||||
- 2026-08-20 Add a `webhooker resetpw` subcommand and a bootstrap banner. The
|
||||
admin bootstrap password was printed once among roughly 45 fx lines, and under
|
||||
`docker run -d` went to container logs subject to rotation; there was no reset
|
||||
path at all, so recovery meant hand-deleting the users row, documented
|
||||
nowhere. The password is read from stdin or generated, never from argv where
|
||||
`/proc` would publish it (https://git.eeqj.de/sneak/webhooker/issues/208)
|
||||
- 2026-08-20 Add `ALLOWED_EGRESS_CIDRS`, an allowlist-only escape hatch for the
|
||||
SSRF guard, so a self-hosted proxy can forward into the operator's own
|
||||
network. The guard's always-blocked set cannot be reopened by configuration
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/204)
|
||||
- 2026-08-20 Harden operator-set target headers, which were carried unsafely
|
||||
across a redirect (https://git.eeqj.de/sneak/webhooker/issues/233)
|
||||
- 2026-08-20 Add a target edit form with headers and timeout fields
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/127)
|
||||
- 2026-08-18 Raise `script/test`'s per-package timeout from 30s to 90s, matching
|
||||
the org-wide backstop. `go test` applies `-timeout` per package, and
|
||||
`internal/handlers` had grown past the old budget: a cache-defeated build
|
||||
failed outright at `GOMAXPROCS=4`, and every run under deliberate host load
|
||||
breached 30s. The measurement table lives in the script (#194)
|
||||
- 2026-08-18 Re-sync `REPO_POLICIES.md` from `prompts`. The local copy was stale
|
||||
and still mandated a 20s test target with a 30s timeout, which the org
|
||||
replaced with a 60s cap and a 90s backstop. A synced copy is not a source;
|
||||
reading it as one nearly produced a PR against `prompts` proposing a change
|
||||
already merged there (#196)
|
||||
- 2026-08-18 Report handler panics through the logger and answer 500. chi
|
||||
v1.5.5's `Recoverer` scans for a `panic(0x` frame the runtime no longer emits,
|
||||
then indexes `pkg[-1:]`, so it panicked inside its own stack printer before
|
||||
writing a byte: the recovery never ran, the client got a dropped connection
|
||||
instead of a 500, and the original panic was lost. A local middleware replaces
|
||||
it, bounded by `MaxPanicLogLineBytes` (#187)
|
||||
- 2026-08-18 Route GORM's logger through `slog` and bound it. Every `gorm.Open`
|
||||
left `logger.Default` in place at `Warn` with `IgnoreRecordNotFoundError`
|
||||
false, so **every record-not-found printed the fully interpolated SQL to
|
||||
stdout** — including the client-chosen path on `/webhook/{uuid}` and the
|
||||
submitted username on the login form, at no level the operator set and outside
|
||||
`internal/logger` entirely. Three call sites, not the two the issue named
|
||||
(#178)
|
||||
- 2026-08-18 Bound every `slog` line against client-chosen text. Eight sites
|
||||
reachable unauthenticated, found by reading every `slog` call in the tree
|
||||
rather than only the one reported; the budget moved to a shared
|
||||
`internal/logfield` so no second truncation exists. `DEBUG` being off by
|
||||
default is not a bound and is not treated as one (#176)
|
||||
- 2026-08-18 Stop a slow host turning a login-guard test into a segfault. A
|
||||
non-fatal `assert` on an acquire result was dereferenced on the next line, so
|
||||
one timing miss killed the whole `internal/middleware` binary and reddened CI
|
||||
for unrelated PRs. The fix also removed a real production race — `acquire`
|
||||
could shed a request with a slot standing free, because Go picks uniformly
|
||||
among ready `select` cases (#186)
|
||||
- 2026-08-18 Send the chi route pattern to Sentry rather than the concrete path.
|
||||
The receiver's path carries the entrypoint capability token, so every Sentry
|
||||
event from `/webhook/{uuid}` shipped a live credential to a third party.
|
||||
Request `Data`, `QueryString`, `Cookies` and `Env` are dropped and headers
|
||||
reduced to an allowlist (#179)
|
||||
- 2026-08-18 Read form fields from the POST body only. `r.FormValue` merges the
|
||||
query string, so a login could be driven by URL parameters — putting the
|
||||
password somewhere that lands in access logs, proxy logs and browser history
|
||||
(#160)
|
||||
- 2026-08-18 Verify login credentials before spending rate-limit budget, so a
|
||||
flood of wrong passwords cannot lock out the account it is guessing at. The
|
||||
manager took this decision rather than stall the queue; it is flagged on the
|
||||
issue for reversal (#150)
|
||||
- 2026-08-18 Run all linting in Docker via `Dockerfile.lint`. Host lint was
|
||||
wrong in both directions from version skew and shared caches. `script/lint`
|
||||
asserts the summary line, because `--no-cache-filter` silently ignores a stage
|
||||
name it does not match — the flag that makes the gate meaningful fails open
|
||||
(#109)
|
||||
- 2026-08-18 Serve an event's full stored body over HTTP. The list query
|
||||
truncates for rendering, and that truncated value was the only way to read a
|
||||
body, so the full payload was unreachable (#157)
|
||||
- 2026-08-18 Bound the access log line against client-chosen text.
|
||||
`internal/logfield` budgets by _encoded_ bytes, not runes, so a handler's JSON
|
||||
escaping cannot multiply a field past its allowance (#146)
|
||||
- 2026-08-18 Mark superseded CI commits `failure` rather than `skipped`. A
|
||||
skipped run rolls up green, so a commit that was never tested reported success
|
||||
(#152)
|
||||
- 2026-08-18 Set `fx.StopTimeout` inside the container stop grace, so shutdown
|
||||
hooks are bounded by a deadline the orchestrator will actually honour rather
|
||||
than being killed mid-flush (#134)
|
||||
- 2026-08-17 Bucket IPv6 rate-limit keys by `/64`. A single allocation hands out
|
||||
2^64 addresses, so per-address keying let one client mint unlimited buckets.
|
||||
Manager decision, recorded on the issue (#125)
|
||||
- 2026-08-17 Correct release-blocking README and startup-warning inaccuracies,
|
||||
including claims about behaviour the code does not have (#151)
|
||||
- 2026-08-17 Fetch and verify Alpine.js at build time against
|
||||
`static/vendor.sha256` instead of committing the minified blob, so the
|
||||
dependency is pinned by hash rather than by trust (#145)
|
||||
- 2026-08-17 Bound the event log's rendered bodies in the query itself, so a
|
||||
large stored payload cannot be read into memory just to be truncated for
|
||||
display (#135)
|
||||
- 2026-08-17 Mask the `http` target's destination URL in the UI: it can carry a
|
||||
bearer credential in its path or query, and was rendered verbatim. Manager
|
||||
decision to mask unconditionally (#115)
|
||||
- 2026-08-14 Bound shutdown hooks by their stop context, so a hook that hangs
|
||||
cannot hold the process past its grace period (#102)
|
||||
- 2026-08-14 Render templates via a buffer rather than the `ResponseWriter`, so
|
||||
a template error part-way through cannot commit a 200 and then fail — the
|
||||
response is written only once it is whole (#123)
|
||||
- 2026-08-14 Align the session codec's max-age with the 7-day absolute cap. The
|
||||
codec accepted cookies the session layer considered expired, so the cap was
|
||||
enforced in one place and not the other (#108)
|
||||
- 2026-08-12 Warn when `TRUSTED_PROXIES` is empty in production, where the safe
|
||||
default silently discards forwarded headers and every client rate-limits as
|
||||
the proxy's address (#149)
|
||||
- 2026-08-12 Bound the receiver rate limit per client IP across the whole
|
||||
`/webhook/*` route. The existing limiter keyed on the request path and
|
||||
`/webhook/{uuid}` matches any single segment, so a client that invented a
|
||||
fresh path per request minted a fresh bucket per request: the limit on the
|
||||
only unauthenticated endpoint bounded nothing in aggregate, and every request
|
||||
still cost an entrypoint lookup before it 404ed. An outer limiter keyed on the
|
||||
client address alone now bounds that, chained in front of the unchanged
|
||||
per-entrypoint limiter (#139)
|
||||
- 2026-08-12 Correct release-blocking documentation inaccuracies: the README
|
||||
promised manual redelivery in the present tense in three places when nothing
|
||||
implements it (the same false claim also sat in the doc comment that was its
|
||||
source text), the env table omitted `RETENTION_SWEEP_INTERVAL`, and `TODO.md`
|
||||
itself omitted five landed units (#141)
|
||||
- 2026-08-12 Make the CI gate execute the checks it reports on. The workflow now
|
||||
writes a build-context fingerprint before calling `script/cibuild`, so a code
|
||||
commit invalidates the `COPY` layer of the lint and builder stages while a
|
||||
docs-only commit still replays from cache; a superseding run also rewrites the
|
||||
`failure` status Gitea leaves on commits it cancelled and never tested.
|
||||
Verified by pushing a deliberately broken test and watching CI go red (#119)
|
||||
- 2026-08-12 Require a positive `RETENTION_SWEEP_INTERVAL`: a non-positive value
|
||||
reached `time.NewTicker` in both the retention reaper and the archive sweeper,
|
||||
panicking two goroutines with no recover after startup had already reported
|
||||
success (#140)
|
||||
- 2026-08-12 Bound the `X-Forwarded-For` scan's allocation to the hop cap: the
|
||||
reverse walk cuts entries with `strings.LastIndexByte` instead of joining and
|
||||
splitting, so a 1 MB header allocates 16 bytes rather than 1.6 MB per request
|
||||
on the unauthenticated receiver. Semantics proven unchanged by differential
|
||||
testing against the previous implementation (#133)
|
||||
- 2026-08-12 Cap the `X-Forwarded-For` hop walk at 64 entries, so an
|
||||
attacker-supplied chain cannot burn unbounded CPU in the rate-limit key
|
||||
function; running off the end falls back to the peer address (#124)
|
||||
- 2026-08-12 Gate forwarded-header trust behind a `TRUSTED_PROXIES` CIDR list:
|
||||
all three rate limiters key on the connection's own address unless the direct
|
||||
peer is a configured proxy, in which case `X-Forwarded-For` is walked right to
|
||||
left for the first non-proxy hop. Default trusts nothing, and a
|
||||
set-but-unparseable value aborts startup. Before this, any client could mint a
|
||||
fresh bucket or drain another's by rotating a spoofed header (#88)
|
||||
- 2026-08-11 Web UI cleanup: nav terminology unified on Webhooks, the Profile
|
||||
settings placeholder removed, a progressive-enhancement copy button for the
|
||||
entrypoint URL, and retention form copy that states the actual policy
|
||||
(deletion by the reaper, 0 retains forever) (#57)
|
||||
- 2026-08-11 Mask the webhook credential in delivery errors and logs: Go embeds
|
||||
the request URL in `*url.Error`, so every transport failure persisted the full
|
||||
Slack webhook URL into the per-webhook event database via
|
||||
`DeliveryResult.Error`, a field a future REST API would have served.
|
||||
`maskURLError` drops path, query and userinfo while preserving the wrapped
|
||||
cause, so `errors.Is`/`As` and `Timeout()` still work and DNS, TLS and timeout
|
||||
failures still read differently (#118)
|
||||
- 2026-08-11 Rate-limit the public webhook receiver endpoint
|
||||
(`RECEIVER_RATE_LIMIT`, default 120/min), keyed on client IP plus entrypoint
|
||||
path so one entrypoint cannot exhaust another's budget; over-limit requests
|
||||
get 429 with `Retry-After`. It was the one unauthenticated, internet-facing
|
||||
endpoint with no limit at all (#64)
|
||||
- 2026-08-11 Enforce the body size limit before CSRF parses the form:
|
||||
`MaxBodySize` is now first in all four form-parsing route groups, so an
|
||||
oversized request is rejected with 413 instead of being read in full by the
|
||||
CSRF middleware before any cap applied (#90)
|
||||
- 2026-08-11 Mask target config on the source detail page, which rendered the
|
||||
stored blob verbatim and so exposed the Slack incoming-webhook URL — a bearer
|
||||
credential that cannot be revoked per-holder. Config reaches the template only
|
||||
as a `TargetView` of labelled fields, and header values are rendered as a
|
||||
count (#113)
|
||||
- 2026-08-11 Allow `retention_days` of 0 to mean retain forever, via a sentinel
|
||||
written in `BeforeSave` so the GORM column default cannot win the race. Also
|
||||
bounds the reaper's cutoff arithmetic: day counts above 106751 overflowed
|
||||
`time.Duration` and wrapped the cutoff into the future, where every row
|
||||
matched and the sweep deleted everything (#79)
|
||||
- 2026-08-09 Inactivity-based session timeout: sliding idle expiry
|
||||
(`SESSION_IDLE_TIMEOUT`, default `24h`) refreshed on authenticated requests,
|
||||
with the 7-day absolute cap kept as an independent backstop that activity
|
||||
never extends (#66)
|
||||
- 2026-08-09 Restart recovery and the 60s retry sweep terminally fail an
|
||||
orphaned `retrying` delivery whose target type no longer supports retries,
|
||||
recording a `DeliveryResult` with the reason instead of leaving the delivery
|
||||
stuck forever (#82)
|
||||
- 2026-08-09 Root the delivery engine's worker pool and the retention reaper's
|
||||
sweep loop at `context.Background()` rather than the fx `OnStart` hook context
|
||||
(#97), which carries fx's 15s start timeout and killed both roughly fifteen
|
||||
seconds after boot: the proxy silently stopped delivering webhooks entirely,
|
||||
and the reaper never ran a single sweep under its default one-hour interval
|
||||
- 2026-08-09 Archive writer lifecycle (#89): deleting a webhook (or its last
|
||||
`database` target) evicts the cached archive writer and closes its handle
|
||||
while deliberately leaving `archive-{webhookID}.db` on disk, and a new
|
||||
`ArchiveSweeper` prunes idle archives on the existing
|
||||
`RETENTION_SWEEP_INTERVAL` without ever creating an archive file
|
||||
- 2026-08-09 Configuration parsing fails loudly on set-but-unparseable
|
||||
environment values: `envInt` removed in favour of `envPositiveInt` plus a
|
||||
`PORT` range check, `envBool` now parses with `strconv.ParseBool`, and
|
||||
defaults apply only to unset variables (#80)
|
||||
- 2026-08-07 Automatic event retention cleanup based on `retention_days`,
|
||||
deleting expired events, deliveries, and delivery results from each
|
||||
per-webhook event database (#63)
|
||||
- 2026-08-07 Update golangci-lint to v2.12.2 (Docker image digest in
|
||||
`Dockerfile`, release-archive sha256 pins in `script/bootstrap`), adopt the
|
||||
canonical `.golangci.yml` (v2 `linters.settings` layout so
|
||||
`lll`/`funlen`/`cyclop`/`dupl` thresholds actually apply), and fix all newly
|
||||
surfaced lint findings
|
||||
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
|
||||
shims, README Entrypoints section
|
||||
- 2026-03-25 pin golangci-lint Docker image for linting (#55)
|
||||
- 2026-03-18 CSRF middleware detects TLS per-request, fixing login over plain
|
||||
HTTP and behind reverse proxies (#54)
|
||||
- 2026-03-17 root path redirects based on auth state (#52)
|
||||
- 2026-03-17 CSRF protection, SSRF prevention for HTTP delivery targets with DNS
|
||||
rebinding defense, and per-IP login rate limiting (#42)
|
||||
- 2026-03-17 Slack target type for incoming webhook notifications (#47)
|
||||
- 2026-03-17 Dockerfile absolute paths and static linking (#49); absolute dev
|
||||
DATA_DIR default and clarified env docs (#46)
|
||||
- 2026-03-05 security headers middleware, session regeneration on login, request
|
||||
body size limits (#41)
|
||||
- 2026-03-04 tests for delivery, middleware, and session packages (#32); removed
|
||||
the build-architecture global (#31)
|
||||
- 2026-03-04 1.0 MVP merge: Webhook/Entrypoint/Target rename, core delivery
|
||||
engine with bounded worker pool and circuit breaker, parallel fan-out,
|
||||
per-webhook event databases, management UI (#16)
|
||||
- 2026-03-01 repo brought to REPO_POLICIES standards; TODO.md folded into README
|
||||
(#6)
|
||||
@sneak runs webhooker in limited production, then tags `v1.0.0`. Merging `next`
|
||||
into `main` is a separate act from tagging and waits on neither of those: `next`
|
||||
is kept mergeable at all times, which is the point of the branch.
|
||||
|
||||
# Future Steps
|
||||
|
||||
- Delivery status and retry management UI. Replay of a terminally failed
|
||||
delivery and per-attempt detail already landed
|
||||
(https://git.eeqj.de/sneak/webhooker/issues/203,
|
||||
https://git.eeqj.de/sneak/webhooker/issues/202)
|
||||
- Per-webhook rate limiting in the receiver handler (per-webhook config plus
|
||||
handler enforcement; global limits must not apply to receiver endpoints)
|
||||
- API key authentication for programmatic access (APIKey model exists; Bearer
|
||||
token middleware does not)
|
||||
- Per-webhook rate limits, set in the web UI and enforced in the receiver
|
||||
handler, on top of the receiver's existing limits, which `RECEIVER_RATE_LIMIT`
|
||||
sets for the whole server (see Rate Limiting in the README)
|
||||
- API key authentication for programmatic access (the `APIKey` model exists, but
|
||||
nothing creates a key and no middleware checks one)
|
||||
- REST API v1
|
||||
- CRUD for webhooks, entrypoints, targets
|
||||
- event viewing and filtering endpoints
|
||||
- event redelivery endpoint
|
||||
- OpenAPI specification
|
||||
- Analytics dashboard: success rates, response times, volume
|
||||
- A remember-me option at login
|
||||
- Analytics in the web UI: response times summarised per webhook or per target,
|
||||
and figures for all webhooks together. The webhook page already shows event,
|
||||
delivery and failure counts and failure percentages, the target list counts
|
||||
each target's deliveries, the event log and each event's page show every
|
||||
attempt's duration, and `/metrics` carries server-wide counts and delivery
|
||||
durations
|
||||
- A remember-me option at login. Every session already survives a browser
|
||||
restart: it lasts up to 7 days, and ends sooner after `SESSION_IDLE_TIMEOUT`
|
||||
(default 24 hours) without activity
|
||||
- Password reset flow for a forgotten password over the web. The authenticated
|
||||
password _change_ flow already landed, and a lost password is recoverable from
|
||||
the console with `webhooker resetpw`
|
||||
@@ -377,7 +60,13 @@ Clear the rest of the open 1.0.0 milestone
|
||||
- Later, nice to have
|
||||
- email delivery target type
|
||||
- SNS and S3 delivery targets
|
||||
- data transformations (e.g. webhook to Slack message formatting)
|
||||
- JSONL file delivery with periodic S3 upload
|
||||
- webhook event search and filtering
|
||||
- multi-user with role-based access control
|
||||
- data transformations (e.g. a configurable Slack message format; the Slack
|
||||
target sends one fixed format today)
|
||||
- JSONL file delivery with periodic S3 upload (the `database` target already
|
||||
archives events to SQLite files, rotated hourly, daily or monthly if set,
|
||||
and the webhook page offers them as one gzipped JSON download)
|
||||
- webhook event search, and filtering beyond the event log's failed and
|
||||
pending views
|
||||
- multi-user with role-based access control (each webhook already belongs to
|
||||
one account, but there is only the account made at first boot and no way
|
||||
to add another)
|
||||
|
||||
@@ -18,7 +18,6 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/banner"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/datadir"
|
||||
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
"sneak.berlin/go/webhooker/internal/logger"
|
||||
)
|
||||
@@ -250,21 +249,12 @@ func (d *Database) connectTo(dataDir string) error {
|
||||
return fmt.Errorf("migrating %s: %w", dbPath, err)
|
||||
}
|
||||
|
||||
err = parseModels(
|
||||
db, &Setting{}, &User{}, &APIKey{},
|
||||
&Webhook{}, &Entrypoint{}, &Target{},
|
||||
)
|
||||
if err != nil {
|
||||
return fmt.Errorf("connecting to %s: %w", dbPath, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// migrate applies the main database's SQL files. See
|
||||
// internal/db/migrations.
|
||||
func (d *Database) migrate() error {
|
||||
err := migrations.Apply(d.db, migrations.Main)
|
||||
// Run GORM auto-migrations
|
||||
err := d.Migrate()
|
||||
if err != nil {
|
||||
d.log.Error(
|
||||
"failed to run database migrations",
|
||||
@@ -279,21 +269,6 @@ func (d *Database) migrate() error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// parseModels has GORM parse each model for db before the handle is
|
||||
// shared. GORM parses a model the first time a handle uses it, and two
|
||||
// goroutines doing that at once for models that refer to each other,
|
||||
// as Webhook and Target do, race inside GORM.
|
||||
func parseModels(db *gorm.DB, models ...any) error {
|
||||
for _, model := range models {
|
||||
err := (&gorm.Statement{DB: db}).Parse(model)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parsing model %T: %w", model, err)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// ensureAdminUser creates the bootstrap admin account when the
|
||||
// deployment has no users at all.
|
||||
func (d *Database) ensureAdminUser() error {
|
||||
|
||||
@@ -2,6 +2,7 @@ package database
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
@@ -17,14 +18,14 @@ const omitAssociationsCallback = "webhooker:omit_associations"
|
||||
//
|
||||
// A per-webhook database holds the event tier only, but Delivery
|
||||
// declares belongs-to Event and Target and the delivery engine fills
|
||||
// both in memory before writing. Without this guard GORM upserts those
|
||||
// parent rows here on the delivery and retry write paths, and the
|
||||
// target, whose config holds destination URLs and bearer credentials,
|
||||
// goes to a targets table this file does not have, which fails the
|
||||
// write. Registering the guard on the connection
|
||||
// covers every write path, including writes inside a transaction and
|
||||
// write paths added later. Every event-tier row this file holds is
|
||||
// written explicitly, so nothing depends on the automatic save.
|
||||
// both in memory before writing. Without this guard GORM upserts
|
||||
// those parent rows here on the delivery and retry write paths,
|
||||
// copying targets.config, which holds destination URLs and bearer
|
||||
// credentials, into the file most likely to be backed up or handed
|
||||
// to someone else. Registering the guard on the connection covers
|
||||
// every write path, including writes inside a transaction and write
|
||||
// paths added later. Every event-tier row this file holds is written
|
||||
// explicitly, so nothing depends on the automatic save.
|
||||
func omitAssociations(db *gorm.DB) error {
|
||||
omit := func(tx *gorm.DB) {
|
||||
tx.Statement.Omits = append(
|
||||
@@ -52,3 +53,107 @@ func omitAssociations(db *gorm.DB) error {
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// eventDBSweptVersion is the PRAGMA user_version purgeTargetRows
|
||||
// stamps into a per-webhook database once it has removed any leaked
|
||||
// target rows *and* the VACUUM that removes their bytes has returned.
|
||||
// Nothing else in the tree uses user_version, so 0 means "not swept
|
||||
// by this build".
|
||||
//
|
||||
// The stamp, not the DELETE, is what records that a file is done. A
|
||||
// DELETE commits on its own, so a sweep that is interrupted or whose
|
||||
// VACUUM fails leaves a file whose rows are gone but whose credential
|
||||
// bytes are still in the free pages -- indistinguishable, by row
|
||||
// count, from a file that never leaked. Both leave the stamp unset,
|
||||
// so the next open sweeps again.
|
||||
const eventDBSweptVersion = 1
|
||||
|
||||
// purgeTargetRows deletes target rows that an earlier build's
|
||||
// association upsert wrote into a per-webhook database, and rewrites
|
||||
// the file so their bytes are gone with them. AutoMigrate creates a
|
||||
// targets table in every one of these files because Delivery declares
|
||||
// a belongs-to Target, but nothing in the event tier may put rows in
|
||||
// it. The rows it did put there are junk, not history: they carry an
|
||||
// empty webhook_id, and delivery rows resolve their target against
|
||||
// the main database, so nothing here refers to them.
|
||||
//
|
||||
// The DELETE only unlinks the rows: modernc.org/sqlite leaves
|
||||
// secure_delete at SQLite's default of off, so the credential bytes
|
||||
// stay readable in the file's free pages and a backup of a swept file
|
||||
// would still hand them over. VACUUM rewrites the file without them.
|
||||
//
|
||||
// This runs before every migration and is gated on
|
||||
// eventDBSweptVersion, so a file pays for the rewrite once, on the
|
||||
// first open that finds it unstamped, and every open after that is a
|
||||
// PRAGMA read. A file this build created is stamped before its
|
||||
// targets table exists, so it never vacuums at all. A failure here
|
||||
// fails the open with the stamp left unset, so the sweep is retried
|
||||
// rather than skipped -- a webhook whose file cannot be swept stays
|
||||
// unusable instead of quietly serving from a file that still holds
|
||||
// recoverable credentials.
|
||||
func purgeTargetRows(
|
||||
db *gorm.DB, log *slog.Logger, webhookID string,
|
||||
) error {
|
||||
var version int
|
||||
|
||||
// Row().Scan, not (*gorm.DB).Scan: see internal/gormlog.
|
||||
err := db.Raw("PRAGMA user_version").Row().Scan(&version)
|
||||
if err != nil {
|
||||
return fmt.Errorf(
|
||||
"reading sweep marker of webhook database %s: %w",
|
||||
webhookID, err,
|
||||
)
|
||||
}
|
||||
|
||||
if version >= eventDBSweptVersion {
|
||||
return nil
|
||||
}
|
||||
|
||||
var purged int64
|
||||
|
||||
if db.Migrator().HasTable("targets") {
|
||||
res := db.Exec("DELETE FROM targets")
|
||||
if res.Error != nil {
|
||||
return fmt.Errorf(
|
||||
"purging target rows from webhook database %s: %w",
|
||||
webhookID, res.Error,
|
||||
)
|
||||
}
|
||||
|
||||
purged = res.RowsAffected
|
||||
|
||||
// Unconditional: a zero row count here does not mean there is
|
||||
// nothing to remove, only that no *live* row is left. See
|
||||
// eventDBSweptVersion.
|
||||
err = db.Exec("VACUUM").Error
|
||||
if err != nil {
|
||||
return fmt.Errorf(
|
||||
"purged %d leaked target rows from webhook database "+
|
||||
"%s but vacuuming it failed, so the deleted "+
|
||||
"target credentials are still recoverable from "+
|
||||
"the file; it stays marked unswept and the next "+
|
||||
"open retries: %w",
|
||||
purged, webhookID, err,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
err = db.Exec(fmt.Sprintf(
|
||||
"PRAGMA user_version = %d", eventDBSweptVersion,
|
||||
)).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf(
|
||||
"marking webhook database %s swept: %w", webhookID, err,
|
||||
)
|
||||
}
|
||||
|
||||
if purged > 0 {
|
||||
log.Warn(
|
||||
"purged leaked target rows from per-webhook database",
|
||||
"webhook_id", webhookID,
|
||||
"rows", purged,
|
||||
)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -1,15 +1,392 @@
|
||||
package database_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
_ "modernc.org/sqlite"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||
)
|
||||
|
||||
// testDataDirPerm is the mode the test data directory is created
|
||||
// with.
|
||||
const testDataDirPerm = 0o750
|
||||
|
||||
// eventDBDataDir returns a data directory that a WebhookDBManager
|
||||
// can be pointed at.
|
||||
func eventDBDataDir(t *testing.T) string {
|
||||
t.Helper()
|
||||
|
||||
dir := filepath.Join(t.TempDir(), "events")
|
||||
require.NoError(t, os.MkdirAll(dir, testDataDirPerm))
|
||||
|
||||
return dir
|
||||
}
|
||||
|
||||
// openRawEventDB opens the per-webhook database file directly,
|
||||
// without the manager, so a test can put a file on disk in a state
|
||||
// the manager has to cope with, or inspect one afterwards.
|
||||
func openRawEventDB(
|
||||
t *testing.T, dataDir, webhookID string,
|
||||
) *sql.DB {
|
||||
t.Helper()
|
||||
|
||||
path := filepath.Join(
|
||||
dataDir, fmt.Sprintf("events-%s.db", webhookID),
|
||||
)
|
||||
|
||||
sqlDB, err := sql.Open(
|
||||
"sqlite",
|
||||
fmt.Sprintf("file:%s?mode=rwc", path),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
t.Cleanup(func() { _ = sqlDB.Close() })
|
||||
|
||||
return sqlDB
|
||||
}
|
||||
|
||||
// eventDBFileBytes reads a per-webhook database file off disk, so a
|
||||
// test can assert on what the file itself still holds rather than on
|
||||
// what a query returns.
|
||||
func eventDBFileBytes(t *testing.T, dataDir, webhookID string) []byte {
|
||||
t.Helper()
|
||||
|
||||
//nolint:gosec // reads a file the test just created under t.TempDir()
|
||||
raw, err := os.ReadFile(filepath.Join(
|
||||
dataDir, fmt.Sprintf("events-%s.db", webhookID),
|
||||
))
|
||||
require.NoError(t, err)
|
||||
|
||||
return raw
|
||||
}
|
||||
|
||||
// eventDBUserVersion returns the PRAGMA user_version of a per-webhook
|
||||
// database file, which is the marker purgeTargetRows stamps once it
|
||||
// has swept and vacuumed.
|
||||
func eventDBUserVersion(t *testing.T, sqlDB *sql.DB) int {
|
||||
t.Helper()
|
||||
|
||||
var version int
|
||||
|
||||
require.NoError(t, sqlDB.QueryRowContext(
|
||||
t.Context(), "PRAGMA user_version",
|
||||
).Scan(&version))
|
||||
|
||||
return version
|
||||
}
|
||||
|
||||
// clearEventDBSweptMarker resets the sweep marker to 0, which is what
|
||||
// a file written by a build without the sweep looks like. Tests that
|
||||
// seed a leaked row have to create the file through the manager to
|
||||
// get the real targets table shape, and that stamps it.
|
||||
func clearEventDBSweptMarker(t *testing.T, sqlDB *sql.DB) {
|
||||
t.Helper()
|
||||
|
||||
_, err := sqlDB.ExecContext(t.Context(), "PRAGMA user_version = 0")
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// countTargetRows returns the number of rows in the targets table of
|
||||
// a per-webhook database file, or -1 if the table does not exist.
|
||||
func countTargetRows(t *testing.T, sqlDB *sql.DB) int {
|
||||
t.Helper()
|
||||
|
||||
var tables int
|
||||
|
||||
require.NoError(t, sqlDB.QueryRowContext(
|
||||
t.Context(),
|
||||
"SELECT count(*) FROM sqlite_master "+
|
||||
"WHERE type = 'table' AND name = 'targets'",
|
||||
).Scan(&tables))
|
||||
|
||||
if tables == 0 {
|
||||
return -1
|
||||
}
|
||||
|
||||
var rows int
|
||||
|
||||
require.NoError(t, sqlDB.QueryRowContext(
|
||||
t.Context(), "SELECT count(*) FROM targets",
|
||||
).Scan(&rows))
|
||||
|
||||
return rows
|
||||
}
|
||||
|
||||
// TestOpenPurgesLeakedTargetRows covers the sweep for event
|
||||
// databases written by a build that let GORM upsert target rows
|
||||
// into them: opening the database clears them, and opening it again
|
||||
// is a no-op.
|
||||
func TestOpenPurgesLeakedTargetRows(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dataDir := eventDBDataDir(t)
|
||||
webhookID := uuid.New().String()
|
||||
|
||||
// Create the file the way the application does, so the targets
|
||||
// table has exactly the shape AutoMigrate gives it, then write
|
||||
// a leaked row into it the way the association upsert did.
|
||||
initial := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err := initial.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, initial.CloseAll())
|
||||
|
||||
seed := openRawEventDB(t, dataDir, webhookID)
|
||||
|
||||
_, err = seed.ExecContext(
|
||||
t.Context(),
|
||||
"INSERT INTO targets "+
|
||||
"(id, webhook_id, name, type, config) "+
|
||||
"VALUES (?, '', ?, ?, ?)",
|
||||
uuid.New().String(),
|
||||
"leaked-target",
|
||||
"slack",
|
||||
`{"webhookUrl":"https://hooks.example/T000/B000/secret"}`,
|
||||
)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, 1, countTargetRows(t, seed))
|
||||
clearEventDBSweptMarker(t, seed)
|
||||
require.NoError(t, seed.Close())
|
||||
|
||||
mgr := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err = mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, mgr.CloseAll())
|
||||
|
||||
check := openRawEventDB(t, dataDir, webhookID)
|
||||
assert.Zero(t, countTargetRows(t, check))
|
||||
assert.Equal(
|
||||
t, 1, eventDBUserVersion(t, check),
|
||||
"a completed sweep must mark the file so later opens skip it",
|
||||
)
|
||||
require.NoError(t, check.Close())
|
||||
|
||||
// Idempotent: a second open leaves it at zero and does not
|
||||
// error.
|
||||
again := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err = again.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, again.CloseAll())
|
||||
|
||||
recheck := openRawEventDB(t, dataDir, webhookID)
|
||||
assert.Zero(t, countTargetRows(t, recheck))
|
||||
}
|
||||
|
||||
// TestOpenPurgeRemovesCredentialBytes covers the sweep at the level
|
||||
// that matters for a backup handed to someone else: the leaked
|
||||
// credential must be gone from the raw bytes of the file, not merely
|
||||
// unreachable by query. A bare DELETE unlinks the row and leaves the
|
||||
// bytes readable in the free pages, so this fails without the VACUUM
|
||||
// in purgeTargetRows.
|
||||
func TestOpenPurgeRemovesCredentialBytes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dataDir := eventDBDataDir(t)
|
||||
webhookID := uuid.New().String()
|
||||
credential := "T00000000/B00000000/" + uuid.New().String()
|
||||
|
||||
initial := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err := initial.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, initial.CloseAll())
|
||||
|
||||
seed := openRawEventDB(t, dataDir, webhookID)
|
||||
|
||||
_, err = seed.ExecContext(
|
||||
t.Context(),
|
||||
"INSERT INTO targets "+
|
||||
"(id, webhook_id, name, type, config) "+
|
||||
"VALUES (?, '', ?, ?, ?)",
|
||||
uuid.New().String(),
|
||||
"leaked-target",
|
||||
"slack",
|
||||
fmt.Sprintf(
|
||||
`{"webhookUrl":"https://hooks.example/%s"}`, credential,
|
||||
),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
clearEventDBSweptMarker(t, seed)
|
||||
require.NoError(t, seed.Close())
|
||||
|
||||
// The seed has to be in the file for its absence later to mean
|
||||
// anything.
|
||||
require.True(
|
||||
t,
|
||||
bytes.Contains(
|
||||
eventDBFileBytes(t, dataDir, webhookID),
|
||||
[]byte(credential),
|
||||
),
|
||||
"seeded credential is not in the file, so this test proves nothing",
|
||||
)
|
||||
|
||||
mgr := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err = mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, mgr.CloseAll())
|
||||
|
||||
assert.NotContains(
|
||||
t,
|
||||
string(eventDBFileBytes(t, dataDir, webhookID)),
|
||||
credential,
|
||||
"leaked credential is still recoverable from the raw file",
|
||||
)
|
||||
}
|
||||
|
||||
// TestOpenRevacuumsAfterIncompleteSweep covers the case a row count
|
||||
// cannot see: the rows are already deleted but the file was never
|
||||
// vacuumed, because an earlier sweep died between the two or its
|
||||
// VACUUM failed. The credential bytes are still recoverable, and the
|
||||
// unset marker is the only thing that says so, so the next open must
|
||||
// vacuum rather than conclude from the empty table that there is
|
||||
// nothing to do.
|
||||
func TestOpenRevacuumsAfterIncompleteSweep(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dataDir := eventDBDataDir(t)
|
||||
webhookID := uuid.New().String()
|
||||
credential := "T00000000/B00000000/" + uuid.New().String()
|
||||
|
||||
initial := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err := initial.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, initial.CloseAll())
|
||||
|
||||
seed := openRawEventDB(t, dataDir, webhookID)
|
||||
|
||||
_, err = seed.ExecContext(
|
||||
t.Context(),
|
||||
"INSERT INTO targets "+
|
||||
"(id, webhook_id, name, type, config) "+
|
||||
"VALUES (?, '', ?, ?, ?)",
|
||||
uuid.New().String(),
|
||||
"leaked-target",
|
||||
"slack",
|
||||
fmt.Sprintf(
|
||||
`{"webhookUrl":"https://hooks.example/%s"}`, credential,
|
||||
),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Exactly the state an interrupted sweep leaves: rows gone,
|
||||
// marker unset, bytes still in the free pages.
|
||||
_, err = seed.ExecContext(t.Context(), "DELETE FROM targets")
|
||||
require.NoError(t, err)
|
||||
require.Zero(t, countTargetRows(t, seed))
|
||||
clearEventDBSweptMarker(t, seed)
|
||||
require.NoError(t, seed.Close())
|
||||
|
||||
require.True(
|
||||
t,
|
||||
bytes.Contains(
|
||||
eventDBFileBytes(t, dataDir, webhookID),
|
||||
[]byte(credential),
|
||||
),
|
||||
"the deleted row's bytes must still be in the file, or this "+
|
||||
"test proves nothing",
|
||||
)
|
||||
|
||||
mgr := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err = mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, mgr.CloseAll())
|
||||
|
||||
assert.NotContains(
|
||||
t,
|
||||
string(eventDBFileBytes(t, dataDir, webhookID)),
|
||||
credential,
|
||||
"an interrupted sweep was not retried, so the credential is "+
|
||||
"still recoverable from the raw file",
|
||||
)
|
||||
|
||||
check := openRawEventDB(t, dataDir, webhookID)
|
||||
assert.Equal(t, 1, eventDBUserVersion(t, check))
|
||||
}
|
||||
|
||||
// TestOpenSkipsSweptDatabase covers the other half of the marker: a
|
||||
// file this build created is marked without ever being vacuumed, and
|
||||
// a marked file is not swept again.
|
||||
func TestOpenSkipsSweptDatabase(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dataDir := eventDBDataDir(t)
|
||||
webhookID := uuid.New().String()
|
||||
|
||||
mgr := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err := mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, mgr.CloseAll())
|
||||
|
||||
marked := openRawEventDB(t, dataDir, webhookID)
|
||||
assert.Equal(t, 1, eventDBUserVersion(t, marked))
|
||||
|
||||
// A marked file is left alone, so a row written into it survives
|
||||
// a reopen. Nothing writes target rows any more; this stands in
|
||||
// for the sweep having run.
|
||||
_, err = marked.ExecContext(
|
||||
t.Context(),
|
||||
"INSERT INTO targets "+
|
||||
"(id, webhook_id, name, type, config) "+
|
||||
"VALUES (?, '', ?, ?, ?)",
|
||||
uuid.New().String(), "sentinel", "slack", `{}`,
|
||||
)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, marked.Close())
|
||||
|
||||
again := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
_, err = again.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, again.CloseAll())
|
||||
|
||||
check := openRawEventDB(t, dataDir, webhookID)
|
||||
assert.Equal(
|
||||
t, 1, countTargetRows(t, check),
|
||||
"a marked file must not be swept again",
|
||||
)
|
||||
}
|
||||
|
||||
// TestOpenSucceedsWithoutTargetsTable covers an existing event
|
||||
// database that never grew a targets table. The sweep must not fail
|
||||
// startup on it.
|
||||
func TestOpenSucceedsWithoutTargetsTable(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dataDir := eventDBDataDir(t)
|
||||
webhookID := uuid.New().String()
|
||||
|
||||
seed := openRawEventDB(t, dataDir, webhookID)
|
||||
|
||||
_, err := seed.ExecContext(
|
||||
t.Context(),
|
||||
"CREATE TABLE events (id text PRIMARY KEY)",
|
||||
)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, seed.Close())
|
||||
|
||||
mgr := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
db, err := mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
assert.NotNil(t, db)
|
||||
require.NoError(t, mgr.CloseAll())
|
||||
}
|
||||
|
||||
// TestEventDBCreateOmitsAssociations covers the connection-level
|
||||
// guard directly: a Delivery carrying its Event and Target in
|
||||
// memory, written through the manager's handle, must store only the
|
||||
@@ -17,9 +394,10 @@ import (
|
||||
func TestEventDBCreateOmitsAssociations(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dataDir := eventDBDataDir(t)
|
||||
webhookID := uuid.New().String()
|
||||
|
||||
mgr := databasetest.NewWebhookDBManager(t, t.TempDir())
|
||||
mgr := databasetest.NewWebhookDBManager(t, dataDir)
|
||||
|
||||
db, err := mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
@@ -54,9 +432,8 @@ func TestEventDBCreateOmitsAssociations(t *testing.T) {
|
||||
require.NoError(t, db.Model(d).
|
||||
Update("status", database.DeliveryStatusDelivered).
|
||||
Error)
|
||||
|
||||
// The file has no targets table, so the create and update above
|
||||
// would have failed had GORM tried to save the target.
|
||||
assert.False(t, db.Migrator().HasTable(&database.Target{}))
|
||||
require.NoError(t, mgr.CloseAll())
|
||||
|
||||
check := openRawEventDB(t, dataDir, webhookID)
|
||||
assert.Zero(t, countTargetRows(t, check))
|
||||
}
|
||||
|
||||
@@ -13,8 +13,10 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// TestWebhookDBManager_OpenAddsEventTierIndexes verifies that a new
|
||||
// per-webhook database has these indexes.
|
||||
// TestWebhookDBManager_OpenAddsEventTierIndexes verifies that opening a
|
||||
// per-webhook database that predates these indexes creates them. It
|
||||
// stands in for an older database file by dropping the indexes
|
||||
// AutoMigrate just created, then reopening the same file.
|
||||
func TestWebhookDBManager_OpenAddsEventTierIndexes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -40,9 +42,27 @@ func TestWebhookDBManager_OpenAddsEventTierIndexes(t *testing.T) {
|
||||
db, err := mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// A fresh database has them.
|
||||
for _, ix := range indexes {
|
||||
require.True(t, db.Migrator().HasIndex(ix.model, ix.name))
|
||||
}
|
||||
|
||||
// Stand in for a database file created before the indexes existed.
|
||||
for _, ix := range indexes {
|
||||
require.NoError(t, db.Migrator().DropIndex(ix.model, ix.name))
|
||||
require.False(t, db.Migrator().HasIndex(ix.model, ix.name))
|
||||
}
|
||||
|
||||
// Drop the cached connection so the next open reopens the file and
|
||||
// runs AutoMigrate against it, as a restart would.
|
||||
require.NoError(t, mgr.CloseAll())
|
||||
|
||||
db, err = mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
|
||||
for _, ix := range indexes {
|
||||
assert.True(t, db.Migrator().HasIndex(ix.model, ix.name),
|
||||
"a new database should have %s", ix.name)
|
||||
"opening the existing database should create %s", ix.name)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
package database
|
||||
|
||||
// Migrate runs database migrations for the main application database.
|
||||
// Only configuration-tier models are stored in the main database.
|
||||
// Event-tier models (Event, Delivery, DeliveryResult, EventTotals,
|
||||
// TargetTotals, EntrypointTotals) live in
|
||||
// per-webhook dedicated databases managed by WebhookDBManager.
|
||||
func (d *Database) Migrate() error {
|
||||
return d.db.AutoMigrate(
|
||||
&Setting{},
|
||||
&User{},
|
||||
&APIKey{},
|
||||
&Webhook{},
|
||||
&Entrypoint{},
|
||||
&Target{},
|
||||
)
|
||||
}
|
||||
@@ -1,168 +0,0 @@
|
||||
package database_test
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||
)
|
||||
|
||||
// withFixedNow returns db with GORM's clock stopped at a time in UTC,
|
||||
// so that the created and updated times it writes come back from
|
||||
// SQLite equal to the row written, which holds them in UTC.
|
||||
func withFixedNow(db *gorm.DB) (*gorm.DB, time.Time) {
|
||||
at := time.Date(2026, 10, 13, 12, 30, 45, 123456789, time.UTC)
|
||||
|
||||
return db.Session(&gorm.Session{
|
||||
NowFunc: func() time.Time { return at },
|
||||
}), at
|
||||
}
|
||||
|
||||
// newBaseModel returns a BaseModel with a new ID.
|
||||
func newBaseModel() database.BaseModel {
|
||||
return database.BaseModel{ID: uuid.New().String()}
|
||||
}
|
||||
|
||||
// assertRoundTrip writes row, which must be the only row of its
|
||||
// table, and reads back the same row.
|
||||
func assertRoundTrip[T any](t *testing.T, db *gorm.DB, row *T) {
|
||||
t.Helper()
|
||||
|
||||
require.NoError(t, db.Create(row).Error)
|
||||
|
||||
var got T
|
||||
|
||||
require.NoError(t, db.First(&got).Error)
|
||||
assert.Equal(t, *row, got)
|
||||
}
|
||||
|
||||
// TestMainSchemaHoldsEveryModel writes a row of each main database
|
||||
// model, every field set, into a database made from its SQL files.
|
||||
func TestMainSchemaHoldsEveryModel(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
mainDB, err := database.Open(
|
||||
t.TempDir(), slog.New(slog.DiscardHandler),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
t.Cleanup(func() { _ = mainDB.Close() })
|
||||
|
||||
db, at := withFixedNow(mainDB.DB())
|
||||
|
||||
assertRoundTrip(t, db, &database.Setting{
|
||||
Key: "session_key", Value: "value",
|
||||
})
|
||||
assertRoundTrip(t, db, &database.User{
|
||||
BaseModel: newBaseModel(),
|
||||
Username: "user",
|
||||
Password: "password hash",
|
||||
})
|
||||
assertRoundTrip(t, db, &database.APIKey{
|
||||
BaseModel: newBaseModel(),
|
||||
UserID: uuid.New().String(),
|
||||
Key: "key",
|
||||
Description: "API key description",
|
||||
LastUsedAt: &at,
|
||||
})
|
||||
assertRoundTrip(t, db, &database.Webhook{
|
||||
BaseModel: newBaseModel(),
|
||||
UserID: uuid.New().String(),
|
||||
Name: "webhook",
|
||||
Description: "webhook description",
|
||||
RetentionDays: 7,
|
||||
})
|
||||
assertRoundTrip(t, db, &database.Entrypoint{
|
||||
BaseModel: newBaseModel(),
|
||||
WebhookID: uuid.New().String(),
|
||||
Path: uuid.New().String(),
|
||||
Description: "entrypoint description",
|
||||
Active: true,
|
||||
})
|
||||
assertRoundTrip(t, db, &database.Target{
|
||||
BaseModel: newBaseModel(),
|
||||
WebhookID: uuid.New().String(),
|
||||
Name: "target",
|
||||
Type: database.TargetTypeHTTP,
|
||||
Active: true,
|
||||
Config: `{"url":"https://example.com/"}`,
|
||||
MaxRetries: 3,
|
||||
})
|
||||
}
|
||||
|
||||
// TestEventSchemaHoldsEveryModel does the same for a webhook's event
|
||||
// database, which the manager makes from its SQL files.
|
||||
func TestEventSchemaHoldsEveryModel(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
mgr := databasetest.NewWebhookDBManager(t, t.TempDir())
|
||||
|
||||
t.Cleanup(func() { _ = mgr.CloseAll() })
|
||||
|
||||
events, err := mgr.GetDB(uuid.New().String())
|
||||
require.NoError(t, err)
|
||||
|
||||
db, at := withFixedNow(events)
|
||||
source := uuid.New().String()
|
||||
|
||||
assertRoundTrip(t, db, &database.Event{
|
||||
BaseModel: newBaseModel(),
|
||||
WebhookID: uuid.New().String(),
|
||||
EntrypointID: uuid.New().String(),
|
||||
Method: http.MethodPost,
|
||||
RawQuery: "a=1",
|
||||
Headers: `{"X":"Y"}`,
|
||||
Body: "body",
|
||||
ContentType: "text/plain",
|
||||
BodyBytes: 4,
|
||||
ResubmittedFromID: &source,
|
||||
})
|
||||
assertRoundTrip(t, db, &database.Delivery{
|
||||
BaseModel: newBaseModel(),
|
||||
EventID: uuid.New().String(),
|
||||
TargetID: uuid.New().String(),
|
||||
Status: database.DeliveryStatusDelivered,
|
||||
FinishedAt: &at,
|
||||
Replay: true,
|
||||
})
|
||||
assertRoundTrip(t, db, &database.DeliveryResult{
|
||||
BaseModel: newBaseModel(),
|
||||
DeliveryID: uuid.New().String(),
|
||||
AttemptNum: 2,
|
||||
Success: true,
|
||||
StatusCode: 200,
|
||||
ResponseBody: "ok",
|
||||
Error: "error",
|
||||
Duration: 15,
|
||||
})
|
||||
assertRoundTrip(t, db, &database.TargetTotals{
|
||||
TargetID: uuid.New().String(),
|
||||
Deliveries: 5,
|
||||
Delivered: 3,
|
||||
Failed: 1,
|
||||
DeliveriesRemoved: 2,
|
||||
FailedRemoved: 1,
|
||||
})
|
||||
assertRoundTrip(t, db, &database.EntrypointTotals{
|
||||
EntrypointID: uuid.New().String(), LastEventAt: at,
|
||||
})
|
||||
|
||||
// A new event database has its one row of event totals already.
|
||||
require.NoError(t, database.AddEventTotals(db, database.EventTotals{
|
||||
Events: 3, EventsRemoved: 1, LastEventAt: &at,
|
||||
}))
|
||||
|
||||
var totals []database.EventTotals
|
||||
|
||||
require.NoError(t, db.Find(&totals).Error)
|
||||
assert.Equal(t, []database.EventTotals{{
|
||||
ID: 1, Events: 3, EventsRemoved: 1, LastEventAt: &at,
|
||||
}}, totals)
|
||||
}
|
||||
@@ -14,7 +14,6 @@ import (
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/datadir"
|
||||
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
)
|
||||
|
||||
@@ -371,7 +370,18 @@ func (m *WebhookDBManager) openDB(
|
||||
)
|
||||
}
|
||||
|
||||
err = migrations.Apply(db, migrations.Events)
|
||||
err = purgeTargetRows(db, m.log, webhookID)
|
||||
if err != nil {
|
||||
_ = sqlDB.Close()
|
||||
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Run migrations for event-tier models only
|
||||
err = db.AutoMigrate(
|
||||
&Event{}, &Delivery{}, &DeliveryResult{},
|
||||
&EventTotals{}, &TargetTotals{}, &EntrypointTotals{},
|
||||
)
|
||||
if err != nil {
|
||||
_ = sqlDB.Close()
|
||||
|
||||
@@ -381,15 +391,14 @@ func (m *WebhookDBManager) openDB(
|
||||
)
|
||||
}
|
||||
|
||||
err = parseModels(
|
||||
db, &Event{}, &Delivery{}, &DeliveryResult{},
|
||||
&EventTotals{}, &TargetTotals{}, &EntrypointTotals{},
|
||||
)
|
||||
// A new database gets its row of event totals, all zero. Target
|
||||
// totals rows are created by the first delivery to each target.
|
||||
err = db.FirstOrCreate(&EventTotals{}).Error
|
||||
if err != nil {
|
||||
_ = sqlDB.Close()
|
||||
|
||||
return nil, fmt.Errorf(
|
||||
"connecting to webhook database %s: %w",
|
||||
"creating event totals for webhook database %s: %w",
|
||||
webhookID, err,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -1,9 +0,0 @@
|
||||
-- Migration 000: the table recording which migrations this file has
|
||||
-- had. Applied before the others; it records itself.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
version INTEGER PRIMARY KEY,
|
||||
applied_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
INSERT OR IGNORE INTO schema_migrations (version) VALUES (0);
|
||||
@@ -1,18 +0,0 @@
|
||||
-- Migration 001: the schema of a database target's archive file. The
|
||||
-- archivedEvent model in internal/delivery must agree with it.
|
||||
|
||||
CREATE TABLE archived_events (
|
||||
id integer PRIMARY KEY AUTOINCREMENT,
|
||||
event_id text,
|
||||
webhook_id text,
|
||||
entrypoint_id text,
|
||||
method text,
|
||||
raw_query text,
|
||||
headers text,
|
||||
body text,
|
||||
content_type text,
|
||||
archived_at datetime
|
||||
);
|
||||
|
||||
CREATE INDEX idx_archived_events_event_id ON archived_events (event_id);
|
||||
CREATE INDEX idx_archived_events_archived_at ON archived_events (archived_at);
|
||||
@@ -1,9 +0,0 @@
|
||||
-- Migration 000: the table recording which migrations this file has
|
||||
-- had. Applied before the others; it records itself.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
version INTEGER PRIMARY KEY,
|
||||
applied_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
INSERT OR IGNORE INTO schema_migrations (version) VALUES (0);
|
||||
@@ -1,98 +0,0 @@
|
||||
-- Migration 001: the schema of a webhook's event database. The models
|
||||
-- in internal/database must agree with it. The webhook, entrypoint and
|
||||
-- target ids here name rows in the main database, which has those
|
||||
-- tables; this file does not.
|
||||
|
||||
CREATE TABLE events (
|
||||
id uuid,
|
||||
created_at datetime,
|
||||
updated_at datetime,
|
||||
deleted_at datetime,
|
||||
webhook_id uuid NOT NULL,
|
||||
entrypoint_id uuid NOT NULL,
|
||||
method text NOT NULL,
|
||||
raw_query text,
|
||||
headers text,
|
||||
body text,
|
||||
content_type text,
|
||||
body_bytes integer NOT NULL,
|
||||
resubmitted_from_id uuid,
|
||||
PRIMARY KEY (id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_events_created_at ON events (created_at);
|
||||
CREATE INDEX idx_events_deleted_at ON events (deleted_at);
|
||||
CREATE INDEX idx_events_deleted_at_created_at
|
||||
ON events (deleted_at, created_at);
|
||||
CREATE INDEX idx_events_resubmitted_from_id
|
||||
ON events (resubmitted_from_id, deleted_at);
|
||||
CREATE INDEX idx_events_entrypoint_id
|
||||
ON events (entrypoint_id, deleted_at, resubmitted_from_id, created_at);
|
||||
|
||||
CREATE TABLE deliveries (
|
||||
id uuid,
|
||||
created_at datetime,
|
||||
updated_at datetime,
|
||||
deleted_at datetime,
|
||||
event_id uuid NOT NULL,
|
||||
target_id uuid NOT NULL,
|
||||
status text NOT NULL DEFAULT 'pending',
|
||||
finished_at datetime,
|
||||
replay numeric NOT NULL DEFAULT false,
|
||||
PRIMARY KEY (id),
|
||||
CONSTRAINT fk_events_deliveries
|
||||
FOREIGN KEY (event_id) REFERENCES events (id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_deliveries_deleted_at ON deliveries (deleted_at);
|
||||
CREATE INDEX idx_deliveries_event_id ON deliveries (event_id, deleted_at);
|
||||
CREATE INDEX idx_deliveries_status
|
||||
ON deliveries (status, deleted_at, finished_at, target_id);
|
||||
|
||||
CREATE TABLE delivery_results (
|
||||
id uuid,
|
||||
created_at datetime,
|
||||
updated_at datetime,
|
||||
deleted_at datetime,
|
||||
delivery_id uuid NOT NULL,
|
||||
attempt_num integer NOT NULL,
|
||||
success numeric,
|
||||
status_code integer,
|
||||
response_body text,
|
||||
error text,
|
||||
duration integer,
|
||||
PRIMARY KEY (id),
|
||||
CONSTRAINT fk_deliveries_delivery_results
|
||||
FOREIGN KEY (delivery_id) REFERENCES deliveries (id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_delivery_results_deleted_at
|
||||
ON delivery_results (deleted_at);
|
||||
CREATE INDEX idx_delivery_results_delivery_id
|
||||
ON delivery_results (delivery_id, deleted_at);
|
||||
|
||||
CREATE TABLE event_totals (
|
||||
id integer PRIMARY KEY AUTOINCREMENT,
|
||||
events integer NOT NULL,
|
||||
events_removed integer NOT NULL,
|
||||
last_event_at datetime
|
||||
);
|
||||
|
||||
-- The one row of event totals, all zero.
|
||||
INSERT INTO event_totals (events, events_removed) VALUES (0, 0);
|
||||
|
||||
CREATE TABLE target_totals (
|
||||
target_id uuid,
|
||||
deliveries integer NOT NULL,
|
||||
delivered integer NOT NULL,
|
||||
failed integer NOT NULL,
|
||||
deliveries_removed integer NOT NULL,
|
||||
failed_removed integer NOT NULL,
|
||||
PRIMARY KEY (target_id)
|
||||
);
|
||||
|
||||
CREATE TABLE entrypoint_totals (
|
||||
entrypoint_id uuid,
|
||||
last_event_at datetime NOT NULL,
|
||||
PRIMARY KEY (entrypoint_id)
|
||||
);
|
||||
@@ -1,9 +0,0 @@
|
||||
-- Migration 000: the table recording which migrations this file has
|
||||
-- had. Applied before the others; it records itself.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
version INTEGER PRIMARY KEY,
|
||||
applied_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
INSERT OR IGNORE INTO schema_migrations (version) VALUES (0);
|
||||
@@ -1,89 +0,0 @@
|
||||
-- Migration 001: the main database's schema. The models in
|
||||
-- internal/database must agree with it.
|
||||
|
||||
CREATE TABLE settings (
|
||||
"key" text,
|
||||
value text NOT NULL,
|
||||
PRIMARY KEY ("key")
|
||||
);
|
||||
|
||||
CREATE TABLE users (
|
||||
id uuid,
|
||||
created_at datetime,
|
||||
updated_at datetime,
|
||||
deleted_at datetime,
|
||||
username text NOT NULL,
|
||||
password text NOT NULL,
|
||||
PRIMARY KEY (id),
|
||||
CONSTRAINT chk_users_username
|
||||
CHECK (length(CAST(username AS BLOB)) <= 1024)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_users_deleted_at ON users (deleted_at);
|
||||
CREATE UNIQUE INDEX idx_users_username ON users (username);
|
||||
|
||||
CREATE TABLE api_keys (
|
||||
id uuid,
|
||||
created_at datetime,
|
||||
updated_at datetime,
|
||||
deleted_at datetime,
|
||||
user_id uuid NOT NULL,
|
||||
"key" text NOT NULL,
|
||||
description text,
|
||||
last_used_at datetime,
|
||||
PRIMARY KEY (id),
|
||||
CONSTRAINT fk_users_api_keys FOREIGN KEY (user_id) REFERENCES users (id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_api_keys_deleted_at ON api_keys (deleted_at);
|
||||
CREATE UNIQUE INDEX idx_api_keys_key ON api_keys ("key");
|
||||
|
||||
CREATE TABLE webhooks (
|
||||
id uuid,
|
||||
created_at datetime,
|
||||
updated_at datetime,
|
||||
deleted_at datetime,
|
||||
user_id uuid NOT NULL,
|
||||
name text NOT NULL,
|
||||
description text,
|
||||
retention_days integer DEFAULT 30,
|
||||
PRIMARY KEY (id),
|
||||
CONSTRAINT fk_users_webhooks FOREIGN KEY (user_id) REFERENCES users (id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_webhooks_deleted_at ON webhooks (deleted_at);
|
||||
|
||||
CREATE TABLE entrypoints (
|
||||
id uuid,
|
||||
created_at datetime,
|
||||
updated_at datetime,
|
||||
deleted_at datetime,
|
||||
webhook_id uuid NOT NULL,
|
||||
path text NOT NULL,
|
||||
description text,
|
||||
active numeric DEFAULT true,
|
||||
PRIMARY KEY (id),
|
||||
CONSTRAINT fk_webhooks_entrypoints
|
||||
FOREIGN KEY (webhook_id) REFERENCES webhooks (id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_entrypoints_deleted_at ON entrypoints (deleted_at);
|
||||
CREATE UNIQUE INDEX idx_entrypoints_path ON entrypoints (path);
|
||||
|
||||
CREATE TABLE targets (
|
||||
id uuid,
|
||||
created_at datetime,
|
||||
updated_at datetime,
|
||||
deleted_at datetime,
|
||||
webhook_id uuid NOT NULL,
|
||||
name text NOT NULL,
|
||||
type text NOT NULL,
|
||||
active numeric DEFAULT true,
|
||||
config text,
|
||||
max_retries integer,
|
||||
PRIMARY KEY (id),
|
||||
CONSTRAINT fk_webhooks_targets
|
||||
FOREIGN KEY (webhook_id) REFERENCES webhooks (id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_targets_deleted_at ON targets (deleted_at);
|
||||
@@ -1,135 +0,0 @@
|
||||
// Package migrations holds the schema of each kind of SQLite file
|
||||
// webhooker writes, as numbered SQL files embedded in the binary, and
|
||||
// applies it when a file is opened.
|
||||
//
|
||||
// Each kind of file has its own directory. In each, 000_migration.sql
|
||||
// creates only the schema_migrations table, and 001_schema.sql holds
|
||||
// the whole schema. Before 1.0 a schema change is an edit to
|
||||
// 001_schema.sql, never a new file.
|
||||
package migrations
|
||||
|
||||
import (
|
||||
"embed"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"path"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// The directories, one for each kind of database file.
|
||||
const (
|
||||
// Main is the main database, webhooker.db.
|
||||
Main = "main"
|
||||
|
||||
// Events is a webhook's event database, events-{uuid}.db.
|
||||
Events = "events"
|
||||
|
||||
// Archive is a database target's archive file.
|
||||
Archive = "archive"
|
||||
)
|
||||
|
||||
//go:embed main/*.sql events/*.sql archive/*.sql
|
||||
var files embed.FS
|
||||
|
||||
// Apply runs each SQL file in dir, which is Main, Events or Archive,
|
||||
// that the database has not had yet, in the order of the number its
|
||||
// name starts with, and records that number in schema_migrations.
|
||||
// 000_migration.sql creates that table and records itself, so it runs
|
||||
// first, and only when the table is missing.
|
||||
//
|
||||
// It all runs in one transaction, so a failure leaves the file as it
|
||||
// was.
|
||||
func Apply(db *gorm.DB, dir string) error {
|
||||
return db.Transaction(func(tx *gorm.DB) error {
|
||||
var tables int
|
||||
|
||||
// Row().Scan, not (*gorm.DB).Scan: see internal/gormlog.
|
||||
err := tx.Raw(
|
||||
"SELECT count(*) FROM sqlite_master "+
|
||||
"WHERE type = 'table' AND name = ?",
|
||||
"schema_migrations",
|
||||
).Row().Scan(&tables)
|
||||
if err != nil {
|
||||
return fmt.Errorf("looking for schema_migrations: %w", err)
|
||||
}
|
||||
|
||||
if tables == 0 {
|
||||
err = run(tx, path.Join(dir, "000_migration.sql"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// fs.Glob lists the files in name order, which is number order.
|
||||
names, err := fs.Glob(files, path.Join(dir, "*.sql"))
|
||||
if err != nil {
|
||||
return fmt.Errorf("listing migrations: %w", err)
|
||||
}
|
||||
|
||||
for _, name := range names {
|
||||
err = applyOnce(tx, name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// applyOnce runs the migration file name and records its number,
|
||||
// unless schema_migrations already has that number.
|
||||
func applyOnce(tx *gorm.DB, name string) error {
|
||||
number, _, _ := strings.Cut(path.Base(name), "_")
|
||||
|
||||
version, err := strconv.Atoi(number)
|
||||
if err != nil {
|
||||
return fmt.Errorf("numbering migration %s: %w", name, err)
|
||||
}
|
||||
|
||||
var applied int
|
||||
|
||||
err = tx.Raw(
|
||||
"SELECT count(*) FROM schema_migrations WHERE version = ?",
|
||||
version,
|
||||
).Row().Scan(&applied)
|
||||
if err != nil {
|
||||
return fmt.Errorf("checking migration %s: %w", name, err)
|
||||
}
|
||||
|
||||
if applied > 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
err = run(tx, name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = tx.Exec(
|
||||
"INSERT INTO schema_migrations (version) VALUES (?)", version,
|
||||
).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("recording migration %s: %w", name, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// run executes every statement in the migration file name.
|
||||
func run(tx *gorm.DB, name string) error {
|
||||
script, err := files.ReadFile(name)
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading migration %s: %w", name, err)
|
||||
}
|
||||
|
||||
err = tx.Exec(string(script)).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("applying migration %s: %w", name, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -1,57 +0,0 @@
|
||||
package migrations_test
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"log/slog"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/driver/sqlite"
|
||||
"gorm.io/gorm"
|
||||
_ "modernc.org/sqlite"
|
||||
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
)
|
||||
|
||||
// TestApplyRunsEachFileOnce applies each kind of database file's
|
||||
// migrations to a new file twice. The first time records every file.
|
||||
// The second finds them all recorded and runs none, which is the only
|
||||
// way it can succeed: 001_schema.sql creates tables that then exist.
|
||||
func TestApplyRunsEachFileOnce(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, dir := range []string{
|
||||
migrations.Main, migrations.Events, migrations.Archive,
|
||||
} {
|
||||
t.Run(dir, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
sqlDB, err := sql.Open(
|
||||
"sqlite", filepath.Join(t.TempDir(), "test.db"),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
t.Cleanup(func() { _ = sqlDB.Close() })
|
||||
|
||||
db, err := gorm.Open(
|
||||
sqlite.Dialector{Conn: sqlDB},
|
||||
&gorm.Config{
|
||||
Logger: gormlog.New(slog.New(slog.DiscardHandler)),
|
||||
},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, migrations.Apply(db, dir))
|
||||
require.NoError(t, migrations.Apply(db, dir))
|
||||
|
||||
var versions string
|
||||
|
||||
require.NoError(t, sqlDB.QueryRowContext(t.Context(),
|
||||
"SELECT group_concat(version) FROM schema_migrations",
|
||||
).Scan(&versions))
|
||||
assert.Equal(t, "0,1", versions)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -22,7 +22,6 @@ import (
|
||||
_ "modernc.org/sqlite" // Pure Go SQLite driver.
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
)
|
||||
@@ -173,7 +172,9 @@ func seedArchiveFile(
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, migrations.Apply(gdb, migrations.Archive))
|
||||
require.NoError(
|
||||
t, gdb.AutoMigrate(&delivery.ExportArchivedEvent{}),
|
||||
)
|
||||
|
||||
for i, at := range archivedAt {
|
||||
row := delivery.ExportArchivedEvent{
|
||||
@@ -780,9 +781,9 @@ func TestArchiveSweep_NeverExpiryUntouched(t *testing.T) {
|
||||
//
|
||||
// The spec is stronger than that — a "never" archive is skipped
|
||||
// before any file is touched — so the archive here exists but has
|
||||
// never been migrated. Opening it at all would apply the archive
|
||||
// schema and create the archive table, which is exactly what must
|
||||
// not happen.
|
||||
// never been migrated. Opening it at all would run AutoMigrate
|
||||
// and create the archive table, which is exactly what must not
|
||||
// happen.
|
||||
func TestArchiveSweep_NeverExpirySkipsBeforeOpening(
|
||||
t *testing.T,
|
||||
) {
|
||||
@@ -806,7 +807,7 @@ func TestArchiveSweep_NeverExpirySkipsBeforeOpening(
|
||||
|
||||
// seedUnmigratedArchive creates an archive file that exists but
|
||||
// carries no archive schema, so any open of it is observable: the
|
||||
// archive table appears only if something applied the schema.
|
||||
// archive table appears only if something ran AutoMigrate.
|
||||
func seedUnmigratedArchive(t *testing.T, path string) {
|
||||
t.Helper()
|
||||
|
||||
|
||||
@@ -963,12 +963,29 @@ func TestFailUnretryableRetry_WritesNoTargetRow(
|
||||
database.DeliveryStatusFailed,
|
||||
)
|
||||
|
||||
// The per-webhook database has no targets table, so writing a
|
||||
// target row there would have failed the status write above.
|
||||
assert.False(t, s.WebhookDB.Migrator().HasTable("targets"),
|
||||
// The table exists in the per-webhook database because GORM
|
||||
// migrates the Delivery relation's model alongside it. It must
|
||||
// stay empty.
|
||||
var targetRows int64
|
||||
|
||||
require.NoError(t, s.WebhookDB.
|
||||
Table("targets").
|
||||
Count(&targetRows).Error)
|
||||
|
||||
assert.Zero(t, targetRows,
|
||||
"orphaned-retry terminal failure wrote a target row "+
|
||||
"into the per-webhook event database",
|
||||
)
|
||||
|
||||
var configs []string
|
||||
|
||||
require.NoError(t, s.WebhookDB.
|
||||
Table("targets").
|
||||
Pluck("config", &configs).Error)
|
||||
|
||||
assert.NotContains(
|
||||
t, strings.Join(configs, " "), hookURL,
|
||||
)
|
||||
}
|
||||
|
||||
func TestRecoverSingleRetry_UnknownTargetType(
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
@@ -21,11 +22,12 @@ import (
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/driver/sqlite"
|
||||
"gorm.io/gorm"
|
||||
_ "modernc.org/sqlite"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
"sneak.berlin/go/webhooker/internal/metrics"
|
||||
)
|
||||
|
||||
@@ -35,18 +37,35 @@ const testContentType = "application/json"
|
||||
func testWebhookDB(t *testing.T) *gorm.DB {
|
||||
t.Helper()
|
||||
|
||||
// Opened by the manager the service uses, so these tests cannot
|
||||
// pass against a schema, journal and locking settings or write
|
||||
// guard production does not use.
|
||||
mgr := databasetest.NewWebhookDBManagerWithLogger(
|
||||
t, t.TempDir(), slog.New(slog.DiscardHandler),
|
||||
dbPath := filepath.Join(
|
||||
t.TempDir(), "events-test.db",
|
||||
)
|
||||
|
||||
t.Cleanup(func() { _ = mgr.CloseAll() })
|
||||
|
||||
db, err := mgr.GetDB(uuid.New().String())
|
||||
// Opened the way the service opens a per-webhook database, so
|
||||
// these tests cannot pass against journal and locking settings
|
||||
// production does not use.
|
||||
sqlDB, err := database.OpenSQLite(
|
||||
dbPath, database.SQLiteModeCreate,
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
t.Cleanup(func() { _ = sqlDB.Close() })
|
||||
|
||||
db, err := gorm.Open(
|
||||
sqlite.Dialector{Conn: sqlDB},
|
||||
&gorm.Config{Logger: gormlog.New(slog.New(slog.DiscardHandler))},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, db.AutoMigrate(
|
||||
&database.Event{},
|
||||
&database.Delivery{},
|
||||
&database.DeliveryResult{},
|
||||
&database.EventTotals{},
|
||||
&database.TargetTotals{},
|
||||
))
|
||||
require.NoError(t, db.Create(&database.EventTotals{}).Error)
|
||||
|
||||
return db
|
||||
}
|
||||
|
||||
|
||||
@@ -14,7 +14,6 @@ import (
|
||||
"gorm.io/driver/sqlite"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
)
|
||||
|
||||
@@ -336,7 +335,7 @@ func (w *archiveWriter) openMode(
|
||||
)
|
||||
}
|
||||
|
||||
err = migrations.Apply(gdb, migrations.Archive)
|
||||
err = gdb.AutoMigrate(&archivedEvent{})
|
||||
if err != nil {
|
||||
_ = sqlDB.Close()
|
||||
|
||||
|
||||
@@ -312,9 +312,13 @@ func TestFailMissingTarget_WritesNoTargetRow(
|
||||
database.DeliveryStatusFailed,
|
||||
)
|
||||
|
||||
// The per-webhook database has no targets table, so writing a
|
||||
// target row there would have failed the status write above.
|
||||
assert.False(t, s.WebhookDB.Migrator().HasTable("targets"),
|
||||
var configs []string
|
||||
|
||||
require.NoError(t, s.WebhookDB.
|
||||
Table("targets").
|
||||
Pluck("config", &configs).Error)
|
||||
|
||||
assert.Empty(t, configs,
|
||||
"the deleted-target terminal path wrote a target row "+
|
||||
"into the per-webhook event database",
|
||||
)
|
||||
|
||||
@@ -135,11 +135,9 @@ func openDB(
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, gdb.Exec(
|
||||
"CREATE TABLE things (id text PRIMARY KEY, name text)",
|
||||
).Error)
|
||||
require.NoError(t, gdb.AutoMigrate(&thing{}))
|
||||
|
||||
// Creating the table is not what any of these cases is about.
|
||||
// Migration chatter is not what any of these cases is about.
|
||||
buf.Reset()
|
||||
|
||||
return gdb
|
||||
|
||||
@@ -115,8 +115,8 @@ func skipDir(name string) bool {
|
||||
}
|
||||
|
||||
// walkNonTestGo parses every non-test .go file under root. It returns
|
||||
// the directories, relative to root, it parsed a file in or below,
|
||||
// along with every unguarded Scan it found.
|
||||
// the directories, relative to root, it parsed a file in, along with
|
||||
// every unguarded Scan it found.
|
||||
func walkNonTestGo(t *testing.T, root string) (map[string]bool, []string) {
|
||||
t.Helper()
|
||||
|
||||
@@ -155,12 +155,7 @@ func walkNonTestGo(t *testing.T, root string) (map[string]bool, []string) {
|
||||
return err
|
||||
}
|
||||
|
||||
// The directories above count too, so that one holding
|
||||
// only a nested package, as internal/db does, is not
|
||||
// taken for one the walk skipped.
|
||||
for ; dir != "."; dir = filepath.Dir(dir) {
|
||||
walked[dir] = true
|
||||
}
|
||||
walked[dir] = true
|
||||
|
||||
for _, pos := range unguardedScans(fset, file) {
|
||||
hits = append(hits, relPosition(root, pos))
|
||||
@@ -210,9 +205,9 @@ func TestGormScanIsNeverCalledOutsideTests(t *testing.T) {
|
||||
walked, offenders := walkNonTestGo(t, root)
|
||||
|
||||
// The module's packages are static, templates, and every directory
|
||||
// directly under cmd and internal. Each holds non-test code, itself
|
||||
// or in a package below it, so one the walk parsed nothing in or
|
||||
// below was skipped, and a Scan there would pass unseen.
|
||||
// directly under cmd and internal. Each holds non-test code, so one
|
||||
// the walk parsed nothing in was skipped, and a Scan there would
|
||||
// pass unseen.
|
||||
packages := []string{"static", "templates"}
|
||||
|
||||
for _, parent := range []string{"cmd", "internal"} {
|
||||
|
||||
@@ -42,11 +42,8 @@ func TestDeliveryAttempts_ReadInTheTargetTypesOwnTerms(t *testing.T) {
|
||||
{database.TargetTypeLog, true, 0, "", "written to the log", ""},
|
||||
}
|
||||
|
||||
// Named for type and outcome, so that no two cases share a name:
|
||||
// Go adds "#01" to a repeated one, and a "#" in the temporary
|
||||
// directory's path cuts the path SQLiteDSN gives SQLite short.
|
||||
for _, tc := range cases {
|
||||
t.Run(string(tc.targetType)+" "+tc.outcome, func(t *testing.T) {
|
||||
t.Run(string(tc.targetType), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newRecentEventsFixture(t)
|
||||
|
||||
@@ -314,15 +314,21 @@ func assertReplayTask(
|
||||
)
|
||||
}
|
||||
|
||||
// assertNoLeakedTarget proves the per-webhook database holds no
|
||||
// target: an association write would have upserted the whole target,
|
||||
// plaintext config and all, into the event database. See
|
||||
// assertNoLeakedTarget proves the per-webhook database holds no target
|
||||
// rows. AutoMigrate creates the table there because Delivery declares
|
||||
// the relation, so it is a ROW that signals a leak: an association
|
||||
// write would have upserted the whole target, plaintext config and
|
||||
// all, into the event database. See
|
||||
// https://git.eeqj.de/sneak/webhooker/issues/206.
|
||||
func assertNoLeakedTarget(t *testing.T, webhookDB *gorm.DB) {
|
||||
t.Helper()
|
||||
|
||||
assert.False(
|
||||
t, webhookDB.Migrator().HasTable(&database.Target{}),
|
||||
var leaked int64
|
||||
|
||||
require.NoError(t, webhookDB.Unscoped().
|
||||
Model(&database.Target{}).Count(&leaked).Error)
|
||||
assert.Zero(
|
||||
t, leaked,
|
||||
"replay must not write the target into the event database",
|
||||
)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user