Rotate a database target's archive monthly, daily or hourly (closes #379)
check / check (push) Successful in 3m31s

A database target's rotation (none, monthly, daily or hourly) puts the
UTC period of each event's receive time in its archive file name, so
each file holds exactly its period's events. It is on the new webhook
page, the add target form and the target edit form, and shown in the
target list.

Renames move every one of a target's files and move them back if one
fails. The sweep prunes every file, one at a time under the target's
lock, and deletes a rotated file it leaves empty. Download lists the
files, then opens one at a time, oldest first, finding each again under
the target's current names, and gives each row its period. The target
list names the current file and totals the size of all of them.

Model: opus-5-5
This commit is contained in:
2026-10-03 00:47:31 +00:00
parent bcdd4791ec
commit e55134532c
36 changed files with 2120 additions and 406 deletions
+84 -41
View File
@@ -736,7 +736,8 @@ The app runs as a non-root user (`webhooker`, UID 1000), exposes port
The `/var/lib/webhooker` volume holds all SQLite databases: the main
application database (`webhooker.db`), the per-webhook event databases
(`events-{uuid}.db`), and any archive databases written by `database`
targets (`archive-{webhook_name}-{target_name}-{target_uuid}.db`). Mount
targets (`archive-{webhook_name}-{target_name}-{target_uuid}.db`, with
`-{period}` before `.db` for a target that rotates). Mount
this as a persistent volume to
preserve data across container restarts.
@@ -976,9 +977,11 @@ is both the simplest and the only complete rule:
- `events-{webhook_uuid}.db` — **one per webhook**. Events, deliveries,
delivery results.
- `archive-{webhook_name}-{target_name}-{target_uuid}.db` — **one per
`database` target**. Archived events. The two names are made safe for
a file name, and the file is renamed when the webhook or the target is
(see [Database Architecture](#database-architecture)).
`database` target**, or one per month, day or hour for a target that
rotates, with `-{period}` before `.db`. Archived events. The two names
are made safe for a file name, and the files are renamed when the
webhook or the target is (see
[Database Architecture](#database-architecture)).
`{webhook_uuid}` and `{target_uuid}` are UUID primary keys in their
canonical 36-character hyphenated form, so a real filename looks like
@@ -1611,8 +1614,9 @@ The new webhook form can also give the webhook its first targets: an
optional HTTP target URL creates an `http` target named `HTTP`, and the
archive checkbox creates a `database` target named `Archive` whose
`expiry` is the pruning chosen beside it (never, 1h, 12h, 24h, 30d, 90d
or 365d). Both are validated as on the add target form, and the webhook
and its targets are created together or not at all.
or 365d) and whose `rotation` is the rotation chosen below that (none,
monthly, daily or hourly). Both are validated as on the add target form,
and the webhook and its targets are created together or not at all.
| Field | Type | Description |
| ---------------- | ------- | ----------- |
@@ -1721,12 +1725,14 @@ events should be forwarded.
own archive database
(`archive-{webhook_name}-{target_name}-{target_uuid}.db`) for long-term
retention, with an optional creation-validated expiry (default: keep
forever). The new webhook form, the add target form and the target edit
form all offer the same expiries: never, 1h, 12h, 24h, 30d, 90d or 365d.
The target list shows the expiry in plain units, such as "30 days". No
external delivery and no retries; an archive write failure fails the
delivery. See the database target section under
"Per-Webhook Event Databases" for the full semantics.
forever) and rotation (default: none, one file). The new webhook form,
the add target form and the target edit form all offer the same
expiries: never, 1h, 12h, 24h, 30d, 90d or 365d, and the same
rotations: none, monthly, daily or hourly. The target list shows the
expiry in plain units, such as "30 days", and the rotation. No external
delivery and no retries; an archive write failure fails the delivery.
See the database target section under "Per-Webhook Event Databases"
for the full semantics.
- **`log`** — Write the event to the application log (stdout). Useful
for debugging.
@@ -2097,15 +2103,31 @@ single `-`, no `-` at either end, cut to 40 characters, and `unnamed`
when nothing is left. The target UUID keeps the file name unique. A
webhook named `Orders (EU)` with a target named `Long-term archive`
archives into `archive-orders-eu-long-term-archive-{target_uuid}.db`.
Renaming the webhook or the target renames the file, under the same
lock the archive writes and the archive sweeper take. Webhook edits,
target edits and target creation run one at a time, so no edit can
rename the file between another's rename and save, and the name on disk
matches the UI. A rename never replaces a file: if one already has
the new name, the edit is refused with an error naming that file, and
the stored name stays. If the archive is not there (the operator moved
it away), the rename is not an error, and the next write creates the
file under the new name.
An optional `rotation` in the target's config JSON (e.g.
`{"rotation":"daily"}`) is `none`, the default, which keeps the one
file, or `monthly`, `daily` or `hourly`. A target that rotates writes
each event to a file named for the period of the event's receive time,
in UTC, put before the `.db`:
`archive-orders-eu-long-term-archive-{target_uuid}-2026-10.db` monthly,
`…-2026-10-01.db` daily and `…-2026-10-01-19.db` hourly. Each file holds
exactly its period's events, and the first event of a new period starts
the next file, so a finished period's file can be moved away like any
archive. A changed rotation applies from the next event: the files
already written keep their names and stay, pruned, shown and downloaded
with the rest, since every file named for the target is its archive,
whichever rotation wrote it.
Renaming the webhook or the target renames every one of the target's
files, each keeping its period, under the same lock the archive writes
and the archive sweeper take. Webhook edits, target edits and target
creation run one at a time, so no edit can rename the files between
another's rename and save, and the names on disk match the UI. A rename
never replaces a file: if one already has a new name, the edit is
refused with an error naming that file, nothing is moved, and the
stored name stays. If the archive is not there (the operator moved it
away), the rename is not an error, and the next write creates the file
under the new name.
The file is moved just before the new name is saved. If the process
stops between the two, the archive is left under the new name while the
@@ -2144,18 +2166,29 @@ interleave with a write, and it leaves the archive closed afterwards so
the move-the-file-away workflow keeps working. Archives with no expiry,
or the expiry `never`, are not touched by the sweep at all.
For a target with several files, the write path prunes only the file it
writes to, and the sweep prunes every one of them, taking the target's
lock for one file at a time, so a write to the target waits for at most
one file's prune. A file that is gone by the time the sweep reaches it
is skipped. A file named for a period that the sweep leaves empty is
deleted, with any `-wal` and `-shm` beside it; the file without a period
is kept even when empty, as it always has been.
Because each `database` target has its own archive file, a target's
`expiry` governs only its own archive. Two `database` targets on one
webhook with different expiries keep two archives, each pruned on its
own schedule.
The webhook page shows, for each `database` target, its archive file's
name, its size on disk and when it was last written. The size counts
the `.db` and its `-wal` together, and the last write is the later of
their two modification times, since a write lands in the `-wal` first.
Both are read from the files' metadata; the archive is never opened.
Before the first write, and after the file has been moved away, the page
shows `not created yet` beside the name.
The webhook page shows, for each `database` target, the name of the
archive file an event received now would go to, the size on disk of all
the target's archive files together, how many there are when there is
more than one, and when the latest of them was last written. The size
counts each `.db` and its `-wal` together, and the last write is the
latest of their modification times, since a write lands in the `-wal`
first. All are read from the files' metadata; the archive is never
opened. While the named file does not exist — before the first write,
before the first event of a new period, and after the file has been
moved away — the page shows `not created yet` beside the name.
Each `database` target on the webhook page has a **Download** button,
which returns its archive as one gzipped JSON file,
@@ -2163,22 +2196,31 @@ which returns its archive as one gzipped JSON file,
names made safe as above and the time in UTC. The file holds one
object: `webhook` and `target`, each an `id` and a `name`;
`exported_at`; and `archived_events`, one object per archived row with
every column, keyed by column name. A body that is not valid UTF-8 is
written in base64, with `"body_encoding": "base64"` beside it. An
archive that does not exist yet, or was moved away, downloads with an
empty `archived_events`; the download never creates the file.
every column, keyed by column name. The rows come from every one of the
target's files: the file without a period first, then the others in
the order of their periods, oldest first, and each row from a file named
for a period has that `period` beside its columns. A body that is not
valid UTF-8 is written in base64, with `"body_encoding": "base64"`
beside it. An archive that does not exist yet, or was moved away,
downloads with an empty `archived_events`; the download never creates a
file.
The download streams: each row is read and written out compressed
before the next is read, so neither the archive nor the JSON is held in
memory. It reads on a connection of its own, inside one read-only
transaction, so the file holds the archive as it stood when the
download started, and archive writes go on meanwhile, since under WAL a
reader never blocks a writer. While it runs, the `-wal` cannot be
checkpointed past what it reads, so a long download lets the `-wal`
grow. It finds the file by the stored names under the lock that webhook
edits, target edits and target creation hold, and lets go once the file
is open: a rename during the download moves the file without affecting
it.
memory. When it starts it lists the target's files by the stored names,
under the lock that webhook edits, target edits and target creation
hold, and lets go. It then opens one file at a time, only when its rows
are about to be written out, and closes it before it opens the next, so
it never has more than one of the target's files open. To open each, it
takes the lock again just long enough to find the file by its period
under the names stored then, so a rename during the download loses no
file; a file that is gone by then, emptied by the sweep or moved away,
is skipped. Each file is read on a connection of its own inside one
read-only transaction, so it is written out as it stood when it was
opened, and archive writes go on meanwhile, since under WAL a reader
never blocks a writer. Until the open file is closed its `-wal` cannot
be checkpointed past what the download reads, so a long download lets
that `-wal` grow.
Deleting a webhook releases its archives: the delivery engine's cached
archive writers are dropped and their file handles closed, so nothing
@@ -3144,6 +3186,7 @@ webhooker/
│ │ ├── target_slack.go # Slack/Mattermost incoming-webhook target
│ │ ├── target_database.go # Database archive target
│ │ ├── target_database_archive.go # Archive file lifecycle and pruning
│ │ ├── target_database_rotation.go # Archive rotation and file names
│ │ ├── target_database_export.go # Archive download as gzipped JSON
│ │ ├── target_log.go # Log target (stdout)
│ │ ├── target_config_view.go # Masked target config for templates