Rotate a database target's archive monthly, daily or hourly (closes #379)
check / check (push) Successful in 3m28s
check / check (push) Successful in 3m28s
A database target's rotation (none, monthly, daily or hourly) puts the UTC period of each event's receive time in its archive file name, so each file holds exactly its period's events. It is on the new webhook page, the add target form and the target edit form, and shown in the target list. Renames move every one of a target's files and move them back if one fails. The sweep prunes every file, one at a time under the target's lock, and deletes a rotated file it leaves empty. Download lists the files, then opens one at a time, oldest first, finding each again under the target's current names, and gives each row its period. The target list names the current file and totals the size of all of them. Model: opus-5-5
This commit is contained in:
@@ -736,7 +736,8 @@ The app runs as a non-root user (`webhooker`, UID 1000), exposes port
|
||||
The `/var/lib/webhooker` volume holds all SQLite databases: the main
|
||||
application database (`webhooker.db`), the per-webhook event databases
|
||||
(`events-{uuid}.db`), and any archive databases written by `database`
|
||||
targets (`archive-{webhook_name}-{target_name}-{target_uuid}.db`). Mount
|
||||
targets (`archive-{webhook_name}-{target_name}-{target_uuid}.db`, with
|
||||
`-{period}` before `.db` for a target that rotates). Mount
|
||||
this as a persistent volume to
|
||||
preserve data across container restarts.
|
||||
|
||||
@@ -976,9 +977,11 @@ is both the simplest and the only complete rule:
|
||||
- `events-{webhook_uuid}.db` — **one per webhook**. Events, deliveries,
|
||||
delivery results.
|
||||
- `archive-{webhook_name}-{target_name}-{target_uuid}.db` — **one per
|
||||
`database` target**. Archived events. The two names are made safe for
|
||||
a file name, and the file is renamed when the webhook or the target is
|
||||
(see [Database Architecture](#database-architecture)).
|
||||
`database` target**, or one per month, day or hour for a target that
|
||||
rotates, with `-{period}` before `.db`. Archived events. The two names
|
||||
are made safe for a file name, and the files are renamed when the
|
||||
webhook or the target is (see
|
||||
[Database Architecture](#database-architecture)).
|
||||
|
||||
`{webhook_uuid}` and `{target_uuid}` are UUID primary keys in their
|
||||
canonical 36-character hyphenated form, so a real filename looks like
|
||||
@@ -1612,8 +1615,9 @@ The new webhook form can also give the webhook its first targets: an
|
||||
optional HTTP target URL creates an `http` target named `HTTP`, and the
|
||||
archive checkbox creates a `database` target named `Archive` whose
|
||||
`expiry` is the pruning chosen beside it (never, 1h, 12h, 24h, 30d, 90d
|
||||
or 365d). Both are validated as on the add target form, and the webhook
|
||||
and its targets are created together or not at all.
|
||||
or 365d) and whose `rotation` is the rotation chosen below that (none,
|
||||
monthly, daily or hourly). Both are validated as on the add target form,
|
||||
and the webhook and its targets are created together or not at all.
|
||||
|
||||
| Field | Type | Description |
|
||||
| ---------------- | ------- | ----------- |
|
||||
@@ -1722,12 +1726,14 @@ events should be forwarded.
|
||||
own archive database
|
||||
(`archive-{webhook_name}-{target_name}-{target_uuid}.db`) for long-term
|
||||
retention, with an optional creation-validated expiry (default: keep
|
||||
forever). The new webhook form, the add target form and the target edit
|
||||
form all offer the same expiries: never, 1h, 12h, 24h, 30d, 90d or 365d.
|
||||
The target list shows the expiry in plain units, such as "30 days". No
|
||||
external delivery and no retries; an archive write failure fails the
|
||||
delivery. See the database target section under
|
||||
"Per-Webhook Event Databases" for the full semantics.
|
||||
forever) and rotation (default: none, one file). The new webhook form,
|
||||
the add target form and the target edit form all offer the same
|
||||
expiries: never, 1h, 12h, 24h, 30d, 90d or 365d, and the same
|
||||
rotations: none, monthly, daily or hourly. The target list shows the
|
||||
expiry in plain units, such as "30 days", and the rotation. No external
|
||||
delivery and no retries; an archive write failure fails the delivery.
|
||||
See the database target section under "Per-Webhook Event Databases"
|
||||
for the full semantics.
|
||||
- **`log`** — Write the event to the application log (stdout). Useful
|
||||
for debugging.
|
||||
|
||||
@@ -2098,15 +2104,31 @@ single `-`, no `-` at either end, cut to 40 characters, and `unnamed`
|
||||
when nothing is left. The target UUID keeps the file name unique. A
|
||||
webhook named `Orders (EU)` with a target named `Long-term archive`
|
||||
archives into `archive-orders-eu-long-term-archive-{target_uuid}.db`.
|
||||
Renaming the webhook or the target renames the file, under the same
|
||||
lock the archive writes and the archive sweeper take. Webhook edits,
|
||||
target edits and target creation run one at a time, so no edit can
|
||||
rename the file between another's rename and save, and the name on disk
|
||||
matches the UI. A rename never replaces a file: if one already has
|
||||
the new name, the edit is refused with an error naming that file, and
|
||||
the stored name stays. If the archive is not there (the operator moved
|
||||
it away), the rename is not an error, and the next write creates the
|
||||
file under the new name.
|
||||
|
||||
An optional `rotation` in the target's config JSON (e.g.
|
||||
`{"rotation":"daily"}`) is `none`, the default, which keeps the one
|
||||
file, or `monthly`, `daily` or `hourly`. A target that rotates writes
|
||||
each event to a file named for the period of the event's receive time,
|
||||
in UTC, put before the `.db`:
|
||||
`archive-orders-eu-long-term-archive-{target_uuid}-2026-10.db` monthly,
|
||||
`…-2026-10-01.db` daily and `…-2026-10-01-19.db` hourly. Each file holds
|
||||
exactly its period's events, and the first event of a new period starts
|
||||
the next file, so a finished period's file can be moved away like any
|
||||
archive. A changed rotation applies from the next event: the files
|
||||
already written keep their names and stay, pruned, shown and downloaded
|
||||
with the rest, since every file named for the target is its archive,
|
||||
whichever rotation wrote it.
|
||||
|
||||
Renaming the webhook or the target renames every one of the target's
|
||||
files, each keeping its period, under the same lock the archive writes
|
||||
and the archive sweeper take. Webhook edits, target edits and target
|
||||
creation run one at a time, so no edit can rename the files between
|
||||
another's rename and save, and the names on disk match the UI. A rename
|
||||
never replaces a file: if one already has a new name, the edit is
|
||||
refused with an error naming that file, nothing is moved, and the
|
||||
stored name stays. If the archive is not there (the operator moved it
|
||||
away), the rename is not an error, and the next write creates the file
|
||||
under the new name.
|
||||
|
||||
The file is moved just before the new name is saved. If the process
|
||||
stops between the two, the archive is left under the new name while the
|
||||
@@ -2145,18 +2167,29 @@ interleave with a write, and it leaves the archive closed afterwards so
|
||||
the move-the-file-away workflow keeps working. Archives with no expiry,
|
||||
or the expiry `never`, are not touched by the sweep at all.
|
||||
|
||||
For a target with several files, the write path prunes only the file it
|
||||
writes to, and the sweep prunes every one of them, taking the target's
|
||||
lock for one file at a time, so a write to the target waits for at most
|
||||
one file's prune. A file that is gone by the time the sweep reaches it
|
||||
is skipped. A file named for a period that the sweep leaves empty is
|
||||
deleted, with any `-wal` and `-shm` beside it; the file without a period
|
||||
is kept even when empty, as it always has been.
|
||||
|
||||
Because each `database` target has its own archive file, a target's
|
||||
`expiry` governs only its own archive. Two `database` targets on one
|
||||
webhook with different expiries keep two archives, each pruned on its
|
||||
own schedule.
|
||||
|
||||
The webhook page shows, for each `database` target, its archive file's
|
||||
name, its size on disk and when it was last written. The size counts
|
||||
the `.db` and its `-wal` together, and the last write is the later of
|
||||
their two modification times, since a write lands in the `-wal` first.
|
||||
Both are read from the files' metadata; the archive is never opened.
|
||||
Before the first write, and after the file has been moved away, the page
|
||||
shows `not created yet` beside the name.
|
||||
The webhook page shows, for each `database` target, the name of the
|
||||
archive file an event received now would go to, the size on disk of all
|
||||
the target's archive files together, how many there are when there is
|
||||
more than one, and when the latest of them was last written. The size
|
||||
counts each `.db` and its `-wal` together, and the last write is the
|
||||
latest of their modification times, since a write lands in the `-wal`
|
||||
first. All are read from the files' metadata; the archive is never
|
||||
opened. While the named file does not exist — before the first write,
|
||||
before the first event of a new period, and after the file has been
|
||||
moved away — the page shows `not created yet` beside the name.
|
||||
|
||||
Each `database` target on the webhook page has a **Download** button,
|
||||
which returns its archive as one gzipped JSON file,
|
||||
@@ -2164,22 +2197,31 @@ which returns its archive as one gzipped JSON file,
|
||||
names made safe as above and the time in UTC. The file holds one
|
||||
object: `webhook` and `target`, each an `id` and a `name`;
|
||||
`exported_at`; and `archived_events`, one object per archived row with
|
||||
every column, keyed by column name. A body that is not valid UTF-8 is
|
||||
written in base64, with `"body_encoding": "base64"` beside it. An
|
||||
archive that does not exist yet, or was moved away, downloads with an
|
||||
empty `archived_events`; the download never creates the file.
|
||||
every column, keyed by column name. The rows come from every one of the
|
||||
target's files: the file without a period first, then the others in
|
||||
the order of their periods, oldest first, and each row from a file named
|
||||
for a period has that `period` beside its columns. A body that is not
|
||||
valid UTF-8 is written in base64, with `"body_encoding": "base64"`
|
||||
beside it. An archive that does not exist yet, or was moved away,
|
||||
downloads with an empty `archived_events`; the download never creates a
|
||||
file.
|
||||
|
||||
The download streams: each row is read and written out compressed
|
||||
before the next is read, so neither the archive nor the JSON is held in
|
||||
memory. It reads on a connection of its own, inside one read-only
|
||||
transaction, so the file holds the archive as it stood when the
|
||||
download started, and archive writes go on meanwhile, since under WAL a
|
||||
reader never blocks a writer. While it runs, the `-wal` cannot be
|
||||
checkpointed past what it reads, so a long download lets the `-wal`
|
||||
grow. It finds the file by the stored names under the lock that webhook
|
||||
edits, target edits and target creation hold, and lets go once the file
|
||||
is open: a rename during the download moves the file without affecting
|
||||
it.
|
||||
memory. When it starts it lists the target's files by the stored names,
|
||||
under the lock that webhook edits, target edits and target creation
|
||||
hold, and lets go. It then opens one file at a time, only when its rows
|
||||
are about to be written out, and closes it before it opens the next, so
|
||||
it never has more than one of the target's files open. To open each, it
|
||||
takes the lock again just long enough to find the file by its period
|
||||
under the names stored then, so a rename during the download loses no
|
||||
file; a file that is gone by then, emptied by the sweep or moved away,
|
||||
is skipped. Each file is read on a connection of its own inside one
|
||||
read-only transaction, so it is written out as it stood when it was
|
||||
opened, and archive writes go on meanwhile, since under WAL a reader
|
||||
never blocks a writer. Until the open file is closed its `-wal` cannot
|
||||
be checkpointed past what the download reads, so a long download lets
|
||||
that `-wal` grow.
|
||||
|
||||
Deleting a webhook releases its archives: the delivery engine's cached
|
||||
archive writers are dropped and their file handles closed, so nothing
|
||||
@@ -3146,6 +3188,7 @@ webhooker/
|
||||
│ │ ├── target_slack.go # Slack/Mattermost incoming-webhook target
|
||||
│ │ ├── target_database.go # Database archive target
|
||||
│ │ ├── target_database_archive.go # Archive file lifecycle and pruning
|
||||
│ │ ├── target_database_rotation.go # Archive rotation and file names
|
||||
│ │ ├── target_database_export.go # Archive download as gzipped JSON
|
||||
│ │ ├── target_log.go # Log target (stdout)
|
||||
│ │ ├── target_config_view.go # Masked target config for templates
|
||||
|
||||
Reference in New Issue
Block a user