Rotate a database target's archive monthly, daily or hourly (closes #379)
check / check (push) Successful in 3m42s

A database target's rotation (none, monthly, daily or hourly) puts the
UTC period of each event's receive time in its archive file name, so
each file holds exactly its period's events. It is on the new webhook
page, the add target form and the target edit form, and shown in the
target list.

Renames move every one of a target's files, the sweep prunes every file
and deletes a rotated file it leaves empty, Download exports every file
oldest first with each row's period, and the target list names the
current file and totals the size of all of them.

Model: opus-5-5
This commit is contained in:
2026-10-02 23:51:42 +00:00
parent 22fa502638
commit 13f881007a
33 changed files with 1753 additions and 328 deletions
+78 -41
View File
@@ -736,7 +736,8 @@ The app runs as a non-root user (`webhooker`, UID 1000), exposes port
The `/var/lib/webhooker` volume holds all SQLite databases: the main
application database (`webhooker.db`), the per-webhook event databases
(`events-{uuid}.db`), and any archive databases written by `database`
targets (`archive-{webhook_name}-{target_name}-{target_uuid}.db`). Mount
targets (`archive-{webhook_name}-{target_name}-{target_uuid}.db`, with
`-{period}` before `.db` for a target that rotates). Mount
this as a persistent volume to
preserve data across container restarts.
@@ -976,9 +977,11 @@ is both the simplest and the only complete rule:
- `events-{webhook_uuid}.db` — **one per webhook**. Events, deliveries,
delivery results.
- `archive-{webhook_name}-{target_name}-{target_uuid}.db` — **one per
`database` target**. Archived events. The two names are made safe for
a file name, and the file is renamed when the webhook or the target is
(see [Database Architecture](#database-architecture)).
`database` target**, or one per month, day or hour for a target that
rotates, with `-{period}` before `.db`. Archived events. The two names
are made safe for a file name, and the files are renamed when the
webhook or the target is (see
[Database Architecture](#database-architecture)).
`{webhook_uuid}` and `{target_uuid}` are UUID primary keys in their
canonical 36-character hyphenated form, so a real filename looks like
@@ -1610,8 +1613,9 @@ The new webhook form can also give the webhook its first targets: an
optional HTTP target URL creates an `http` target named `HTTP`, and the
archive checkbox creates a `database` target named `Archive` whose
`expiry` is the pruning chosen beside it (never, 1h, 12h, 24h, 30d, 90d
or 365d). Both are validated as on the add target form, and the webhook
and its targets are created together or not at all.
or 365d) and whose `rotation` is the rotation chosen below that (none,
monthly, daily or hourly). Both are validated as on the add target form,
and the webhook and its targets are created together or not at all.
| Field | Type | Description |
| ---------------- | ------- | ----------- |
@@ -1720,12 +1724,14 @@ events should be forwarded.
own archive database
(`archive-{webhook_name}-{target_name}-{target_uuid}.db`) for long-term
retention, with an optional creation-validated expiry (default: keep
forever). The new webhook form, the add target form and the target edit
form all offer the same expiries: never, 1h, 12h, 24h, 30d, 90d or 365d.
The target list shows the expiry in plain units, such as "30 days". No
external delivery and no retries; an archive write failure fails the
delivery. See the database target section under
"Per-Webhook Event Databases" for the full semantics.
forever) and rotation (default: none, one file). The new webhook form,
the add target form and the target edit form all offer the same
expiries: never, 1h, 12h, 24h, 30d, 90d or 365d, and the same
rotations: none, monthly, daily or hourly. The target list shows the
expiry in plain units, such as "30 days", and the rotation. No external
delivery and no retries; an archive write failure fails the delivery.
See the database target section under "Per-Webhook Event Databases"
for the full semantics.
- **`log`** — Write the event to the application log (stdout). Useful
for debugging.
@@ -2088,15 +2094,31 @@ single `-`, no `-` at either end, cut to 40 characters, and `unnamed`
when nothing is left. The target UUID keeps the file name unique. A
webhook named `Orders (EU)` with a target named `Long-term archive`
archives into `archive-orders-eu-long-term-archive-{target_uuid}.db`.
Renaming the webhook or the target renames the file, under the same
lock the archive writes and the archive sweeper take. Webhook edits,
target edits and target creation run one at a time, so no edit can
rename the file between another's rename and save, and the name on disk
matches the UI. A rename never replaces a file: if one already has
the new name, the edit is refused with an error naming that file, and
the stored name stays. If the archive is not there (the operator moved
it away), the rename is not an error, and the next write creates the
file under the new name.
An optional `rotation` in the target's config JSON (e.g.
`{"rotation":"daily"}`) is `none`, the default, which keeps the one
file, or `monthly`, `daily` or `hourly`. A target that rotates writes
each event to a file named for the period of the event's receive time,
in UTC, put before the `.db`:
`archive-orders-eu-long-term-archive-{target_uuid}-2026-10.db` monthly,
`…-2026-10-01.db` daily and `…-2026-10-01-19.db` hourly. Each file holds
exactly its period's events, and the first event of a new period starts
the next file, so a finished period's file can be moved away like any
archive. A changed rotation applies from the next event: the files
already written keep their names and stay, pruned, shown and downloaded
with the rest, since every file named for the target is its archive,
whichever rotation wrote it.
Renaming the webhook or the target renames every one of the target's
files, each keeping its period, under the same lock the archive writes
and the archive sweeper take. Webhook edits, target edits and target
creation run one at a time, so no edit can rename the files between
another's rename and save, and the names on disk match the UI. A rename
never replaces a file: if one already has a new name, the edit is
refused with an error naming that file, nothing is moved, and the
stored name stays. If the archive is not there (the operator moved it
away), the rename is not an error, and the next write creates the file
under the new name.
The file is moved just before the new name is saved. If the process
stops between the two, the archive is left under the new name while the
@@ -2135,18 +2157,27 @@ interleave with a write, and it leaves the archive closed afterwards so
the move-the-file-away workflow keeps working. Archives with no expiry,
or the expiry `never`, are not touched by the sweep at all.
For a target with several files, the write path prunes only the file it
writes to, and the sweep prunes every one of them. A file named for a
period that the sweep leaves empty is deleted, with any `-wal` and
`-shm` beside it; the file without a period is kept even when empty, as
it always has been.
Because each `database` target has its own archive file, a target's
`expiry` governs only its own archive. Two `database` targets on one
webhook with different expiries keep two archives, each pruned on its
own schedule.
The webhook page shows, for each `database` target, its archive file's
name, its size on disk and when it was last written. The size counts
the `.db` and its `-wal` together, and the last write is the later of
their two modification times, since a write lands in the `-wal` first.
Both are read from the files' metadata; the archive is never opened.
Before the first write, and after the file has been moved away, the page
shows `not created yet` beside the name.
The webhook page shows, for each `database` target, the name of the
archive file an event received now would go to, the size on disk of all
the target's archive files together, how many there are when there is
more than one, and when the latest of them was last written. The size
counts each `.db` and its `-wal` together, and the last write is the
latest of their modification times, since a write lands in the `-wal`
first. All are read from the files' metadata; the archive is never
opened. While the named file does not exist — before the first write,
before the first event of a new period, and after the file has been
moved away — the page shows `not created yet` beside the name.
Each `database` target on the webhook page has a **Download** button,
which returns its archive as one gzipped JSON file,
@@ -2154,22 +2185,27 @@ which returns its archive as one gzipped JSON file,
names made safe as above and the time in UTC. The file holds one
object: `webhook` and `target`, each an `id` and a `name`;
`exported_at`; and `archived_events`, one object per archived row with
every column, keyed by column name. A body that is not valid UTF-8 is
written in base64, with `"body_encoding": "base64"` beside it. An
archive that does not exist yet, or was moved away, downloads with an
empty `archived_events`; the download never creates the file.
every column, keyed by column name. The rows come from every one of the
target's files: the file without a period first, then the others in
the order of their periods, oldest first, and each row from a file named
for a period has that `period` beside its columns. A body that is not
valid UTF-8 is written in base64, with `"body_encoding": "base64"`
beside it. An archive that does not exist yet, or was moved away,
downloads with an empty `archived_events`; the download never creates a
file.
The download streams: each row is read and written out compressed
before the next is read, so neither the archive nor the JSON is held in
memory. It reads on a connection of its own, inside one read-only
transaction, so the file holds the archive as it stood when the
download started, and archive writes go on meanwhile, since under WAL a
reader never blocks a writer. While it runs, the `-wal` cannot be
checkpointed past what it reads, so a long download lets the `-wal`
grow. It finds the file by the stored names under the lock that webhook
edits, target edits and target creation hold, and lets go once the file
is open: a rename during the download moves the file without affecting
it.
memory. When it starts it opens every one of the target's files, each on
a connection of its own inside one read-only transaction, so the
download holds the archive as it stood then, and archive writes go on
meanwhile, since under WAL a reader never blocks a writer. Each file is
closed once its rows are written out; until then its `-wal` cannot be
checkpointed past what the download reads, so a long download lets the
`-wal` grow. It finds the files by the stored names under the lock that
webhook edits, target edits and target creation hold, and lets go once
the files are open: a rename during the download moves the files without
affecting it.
Deleting a webhook releases its archives: the delivery engine's cached
archive writers are dropped and their file handles closed, so nothing
@@ -3121,6 +3157,7 @@ webhooker/
│ │ ├── target_slack.go # Slack/Mattermost incoming-webhook target
│ │ ├── target_database.go # Database archive target
│ │ ├── target_database_archive.go # Archive file lifecycle and pruning
│ │ ├── target_database_rotation.go # Archive rotation and file names
│ │ ├── target_database_export.go # Archive download as gzipped JSON
│ │ ├── target_log.go # Log target (stdout)
│ │ ├── target_config_view.go # Masked target config for templates