Name each database target's archive for its webhook and target (closes #376)
check / check (push) Successful in 3m16s
check / check (push) Successful in 3m16s
Each database target now has its own archive file, archive-WEBHOOKNAME-TARGETNAME-TARGETID.db, instead of one archive-WEBHOOKID.db per webhook. delivery.ArchiveFileName builds the name. A change of webhook or target name renames its archive files under the archive writer's lock, before the new name is saved, and back again if the save fails. Webhook edits, target edits and target creation run one at a time, so no two of them interleave. A rename never replaces a file, and one that fails part way moves back what it moved. Deleting a target evicts only that target's writer. Archive files are never deleted, and nothing looks for files under the old name. Model: opus-5-5
This commit is contained in:
@@ -715,7 +715,8 @@ The app runs as a non-root user (`webhooker`, UID 1000), exposes port
|
||||
The `/var/lib/webhooker` volume holds all SQLite databases: the main
|
||||
application database (`webhooker.db`), the per-webhook event databases
|
||||
(`events-{uuid}.db`), and any archive databases written by `database`
|
||||
targets (`archive-{uuid}.db`). Mount this as a persistent volume to
|
||||
targets (`archive-{webhook_name}-{target_name}-{target_uuid}.db`). Mount
|
||||
this as a persistent volume to
|
||||
preserve data across container restarts.
|
||||
|
||||
**The container sets its data directory's owner and mode itself
|
||||
@@ -954,13 +955,13 @@ is both the simplest and the only complete rule:
|
||||
encryption key), users, API keys, webhooks, entrypoints, targets.
|
||||
- `events-{webhook_uuid}.db` — **one per webhook**. Events, deliveries,
|
||||
delivery results.
|
||||
- `archive-{webhook_uuid}.db` — **one per webhook that has a `database`
|
||||
target**. Archived events. Keyed on the webhook UUID, not the target
|
||||
UUID: a webhook with several `database` targets still has exactly one
|
||||
archive file.
|
||||
- `archive-{webhook_name}-{target_name}-{target_uuid}.db` — **one per
|
||||
`database` target**. Archived events. The two names are made safe for
|
||||
a file name, and the file is renamed when the webhook or the target is
|
||||
(see [Database Architecture](#database-architecture)).
|
||||
|
||||
`{webhook_uuid}` is the webhook's UUID primary key in its canonical
|
||||
36-character hyphenated form, so a real filename looks like
|
||||
`{webhook_uuid}` and `{target_uuid}` are UUID primary keys in their
|
||||
canonical 36-character hyphenated form, so a real filename looks like
|
||||
`events-3f2a1c9e-....db`. The only other file is `webhooker.lock`, the
|
||||
always-empty [single-instance lock](#single-instance-lock); it holds no
|
||||
state and is not part of the backup set — a copied one is stale and
|
||||
@@ -1034,8 +1035,8 @@ stopped copy.
|
||||
|
||||
Archive databases are the one exception the service is built for: the
|
||||
archive writer closes and reopens its handle around writes (debounced
|
||||
to at most one reopen per second), so an operator can move
|
||||
`archive-{uuid}.db` away for offline retention while the service runs,
|
||||
to at most one reopen per second), so an operator can move an
|
||||
`archive-….db` away for offline retention while the service runs,
|
||||
and it is recreated on the next write. See
|
||||
[Database Architecture](#database-architecture). That is a
|
||||
move-the-file-away workflow, not a substitute for the backup procedures
|
||||
@@ -1053,7 +1054,7 @@ happens on the next write past the debounce window, when the connection
|
||||
pool retires the idle connection (about a minute after the last write),
|
||||
or at the idle archive sweep — measured, the same file was a complete
|
||||
20 KB `.db` with no sidecars about a minute after its last write. A
|
||||
clean stop closes it too. So either move `archive-{uuid}.db` together
|
||||
clean stop closes it too. So either move the `archive-….db` together
|
||||
with any `-wal`/`-shm` beside it, or wait until there are none.
|
||||
|
||||
### Restore
|
||||
@@ -1188,7 +1189,7 @@ commit still produce a byte-identical binary.
|
||||
Treat a backup with the same care as the credentials inside it. Encrypt
|
||||
backups at rest and restrict who can read them.
|
||||
|
||||
- `events-{uuid}.db` and `archive-{uuid}.db` hold the **full payload
|
||||
- `events-{uuid}.db` and `archive-….db` hold the **full payload
|
||||
body and headers** of every event as received, including whatever the
|
||||
sending service put in them — tokens, signatures, personal data.
|
||||
- Event databases written before
|
||||
@@ -1606,8 +1607,9 @@ events should be forwarded.
|
||||
is built on the same HTTP core as `http` and honours `max_retries`
|
||||
identically, circuit breaker included. See the Slack target section
|
||||
under "Per-Webhook Event Databases" for the message format.
|
||||
- **`database`** — Archive the full event as a row into a separate
|
||||
per-webhook archive database (`archive-{webhookID}.db`) for long-term
|
||||
- **`database`** — Archive the full event as a row into the target's
|
||||
own archive database
|
||||
(`archive-{webhook_name}-{target_name}-{target_uuid}.db`) for long-term
|
||||
retention, with an optional creation-validated expiry (default: keep
|
||||
forever). No external delivery and no retries; an archive write
|
||||
failure fails the delivery. See the database target section under
|
||||
@@ -1921,9 +1923,41 @@ The **database target type** builds on this architecture to provide
|
||||
long-term archiving, separate from the per-webhook event database (which
|
||||
may prune events under its own retention). Delivering to a database
|
||||
target writes the full event — body, headers, method, content type, and
|
||||
webhook/entrypoint/event identifiers — as a row into a dedicated archive
|
||||
database, `archive-{webhookID}.db`, stored under the data directory
|
||||
beside the event database. After each write the archive handle is closed
|
||||
webhook/entrypoint/event identifiers — as a row into the target's own
|
||||
archive database, `archive-{webhook_name}-{target_name}-{target_uuid}.db`,
|
||||
stored under the data directory beside the event database. Each
|
||||
`database` target has its own archive file, even when one webhook has
|
||||
several.
|
||||
|
||||
Both names are made safe for a file name the same way: lowercased, ASCII
|
||||
letters and digits kept, every other run of characters turned into a
|
||||
single `-`, no `-` at either end, cut to 40 characters, and `unnamed`
|
||||
when nothing is left. The target UUID keeps the file name unique. A
|
||||
webhook named `Orders (EU)` with a target named `Long-term archive`
|
||||
archives into `archive-orders-eu-long-term-archive-{target_uuid}.db`.
|
||||
Renaming the webhook or the target renames the file, under the same
|
||||
lock the archive writes and the archive sweeper take. Webhook edits,
|
||||
target edits and target creation run one at a time, so no edit can
|
||||
rename the file between another's rename and save, and the name on disk
|
||||
matches the UI. A rename never replaces a file: if one already has
|
||||
the new name, the edit is refused with an error naming that file, and
|
||||
the stored name stays. If the archive is not there (the operator moved
|
||||
it away), the rename is not an error, and the next write creates the
|
||||
file under the new name.
|
||||
|
||||
The file is moved just before the new name is saved. If the process
|
||||
stops between the two, the archive is left under the new name while the
|
||||
UI still shows the old one, and the next delivery starts a second
|
||||
archive under the name shown. To bring them back together, stop the
|
||||
service before moving anything, and move each archive as its `.db`
|
||||
together with any `-wal` and `-shm` beside it, since the `-wal` can hold
|
||||
rows that are not yet in the `.db`. If no file has the name shown, move
|
||||
the archive under the new name back to it. If a second archive already
|
||||
has the name shown, move the archive under the new name out of the data
|
||||
directory instead and keep it as you would any archive moved away. Then
|
||||
start the service again.
|
||||
|
||||
After each write the archive handle is closed
|
||||
and reopened, debounced to at most once per second, so an operator can
|
||||
move the archive file away for offline archiving without stopping the
|
||||
service; a moved or removed archive file is recreated automatically on
|
||||
@@ -1934,35 +1968,33 @@ older than the expiry are pruned each time the archive is (re)opened. An
|
||||
archive write failure is never silent success: the delivery records a
|
||||
failed attempt with the error and is marked failed.
|
||||
|
||||
Because reopens only happen on writes, an archive belonging to a webhook
|
||||
that has stopped receiving events would never be pruned. A background
|
||||
**archive sweeper** closes that gap: on the same interval as the event
|
||||
retention reaper (`RETENTION_SWEEP_INTERVAL`) it prunes every archive
|
||||
whose database target declares a positive expiry, whether or not the
|
||||
webhook is still receiving traffic. The sweep never creates an archive —
|
||||
a webhook whose archive file does not yet exist is skipped, not
|
||||
initialised — it takes the same per-webhook lock the write path uses, so
|
||||
it can never interleave with a write, and it leaves the archive closed
|
||||
afterwards so the move-the-file-away workflow keeps working. Archives
|
||||
with no expiry, or the expiry `never`, are not touched by the sweep at
|
||||
all.
|
||||
Because reopens only happen on writes, an archive whose target has
|
||||
stopped receiving events would never be pruned. A background **archive
|
||||
sweeper** closes that gap: on the same interval as the event retention
|
||||
reaper (`RETENTION_SWEEP_INTERVAL`) it prunes every archive whose
|
||||
database target declares a positive expiry, whether or not the target
|
||||
is still receiving traffic. The sweep never creates an archive — a
|
||||
target whose archive file does not yet exist is skipped, not initialised
|
||||
— it takes the same per-target lock the write path uses, so it can never
|
||||
interleave with a write, and it leaves the archive closed afterwards so
|
||||
the move-the-file-away workflow keeps working. Archives with no expiry,
|
||||
or the expiry `never`, are not touched by the sweep at all.
|
||||
|
||||
Note that a webhook has one archive file but may carry more than one
|
||||
`database` target, each with its own `expiry`. The shortest expiry
|
||||
configured on any of them therefore governs the whole archive, and the
|
||||
sweep applies it whether or not the webhook is still receiving events.
|
||||
Configure a single `database` target per webhook unless you intend that.
|
||||
Because each `database` target has its own archive file, a target's
|
||||
`expiry` governs only its own archive. Two `database` targets on one
|
||||
webhook with different expiries keep two archives, each pruned on its
|
||||
own schedule.
|
||||
|
||||
Deleting a webhook releases its archive: the delivery engine's cached
|
||||
archive writer is dropped and its file handle closed, so nothing lingers
|
||||
after the webhook is gone. The archive **file itself is deliberately
|
||||
left on disk**. Unlike the event database — per-webhook working storage
|
||||
that is hard-deleted with the webhook — an archive is long-term storage
|
||||
an operator may still want to keep or move away for offline retention,
|
||||
and destroying it as a side effect of deleting a webhook would be
|
||||
unrecoverable. Removing `archive-{webhookID}.db` is the operator's call.
|
||||
Deleting a webhook's last `database` target releases the writer the same
|
||||
way, and for the same reason leaves the file alone.
|
||||
Deleting a webhook releases its archives: the delivery engine's cached
|
||||
archive writers are dropped and their file handles closed, so nothing
|
||||
lingers after the webhook is gone. The archive **files themselves are
|
||||
deliberately left on disk**. Unlike the event database — per-webhook
|
||||
working storage that is hard-deleted with the webhook — an archive is
|
||||
long-term storage an operator may still want to keep or move away for
|
||||
offline retention, and destroying it as a side effect of deleting a
|
||||
webhook would be unrecoverable. Removing an `archive-….db` is the
|
||||
operator's call. Deleting a `database` target releases its writer the
|
||||
same way, and for the same reason leaves its file alone.
|
||||
|
||||
The **Slack target type** sends webhook events as formatted messages to
|
||||
any Slack-compatible incoming webhook URL (works with Slack, Mattermost,
|
||||
@@ -2976,8 +3008,8 @@ Components are wired via Uber fx in this order:
|
||||
13. `delivery.New` — Event-driven delivery engine
|
||||
14. `delivery.NewArchiveSweeper` — Periodic pruning of idle archives
|
||||
15. `delivery.Engine` → `delivery.Notifier` — interface bridge
|
||||
16. `delivery.Engine` → `delivery.WebhookEvictor` — interface bridge so
|
||||
deleting a webhook releases its archive writer
|
||||
16. `delivery.Engine` → `delivery.Archives` — interface bridge so
|
||||
deleting or renaming a webhook or target reaches its archive files
|
||||
17. `server.New` — HTTP server and router
|
||||
|
||||
The server starts via `fx.Invoke(func(*server.Server, *delivery.Engine,
|
||||
|
||||
Reference in New Issue
Block a user