Name each database target's archive for its webhook and target (closes #376)
check / check (push) Waiting to run
check / check (push) Waiting to run
Each database target now has its own archive file, archive-WEBHOOKNAME-TARGETNAME-TARGETID.db, instead of one archive-WEBHOOKID.db per webhook. delivery.ArchiveFileName builds the name: each name is lowercased, keeps ASCII letters and digits, turns every other run of characters into one dash, and is cut to 40 characters. A change of webhook or target name renames its archive files under the archive writer's lock, before the new name is saved, and back again if the save fails. A rename never replaces a file: if one already has the new name, the edit is refused. Deleting a target evicts only that target's writer. Archive files are never deleted, and nothing looks for files under the old name. Model: opus-5-5
This commit is contained in:
@@ -698,7 +698,8 @@ The app runs as a non-root user (`webhooker`, UID 1000), exposes port
|
||||
The `/var/lib/webhooker` volume holds all SQLite databases: the main
|
||||
application database (`webhooker.db`), the per-webhook event databases
|
||||
(`events-{uuid}.db`), and any archive databases written by `database`
|
||||
targets (`archive-{uuid}.db`). Mount this as a persistent volume to
|
||||
targets (`archive-{webhook_name}-{target_name}-{target_uuid}.db`). Mount
|
||||
this as a persistent volume to
|
||||
preserve data across container restarts.
|
||||
|
||||
**The container sets its data directory's owner and mode itself
|
||||
@@ -937,13 +938,13 @@ is both the simplest and the only complete rule:
|
||||
encryption key), users, API keys, webhooks, entrypoints, targets.
|
||||
- `events-{webhook_uuid}.db` — **one per webhook**. Events, deliveries,
|
||||
delivery results.
|
||||
- `archive-{webhook_uuid}.db` — **one per webhook that has a `database`
|
||||
target**. Archived events. Keyed on the webhook UUID, not the target
|
||||
UUID: a webhook with several `database` targets still has exactly one
|
||||
archive file.
|
||||
- `archive-{webhook_name}-{target_name}-{target_uuid}.db` — **one per
|
||||
`database` target**. Archived events. The two names are made safe for
|
||||
a file name, and the file is renamed when the webhook or the target is
|
||||
(see [Database Architecture](#database-architecture)).
|
||||
|
||||
`{webhook_uuid}` is the webhook's UUID primary key in its canonical
|
||||
36-character hyphenated form, so a real filename looks like
|
||||
`{webhook_uuid}` and `{target_uuid}` are UUID primary keys in their
|
||||
canonical 36-character hyphenated form, so a real filename looks like
|
||||
`events-3f2a1c9e-....db`. The only other file is `webhooker.lock`, the
|
||||
always-empty [single-instance lock](#single-instance-lock); it holds no
|
||||
state and is not part of the backup set — a copied one is stale and
|
||||
@@ -1017,8 +1018,8 @@ stopped copy.
|
||||
|
||||
Archive databases are the one exception the service is built for: the
|
||||
archive writer closes and reopens its handle around writes (debounced
|
||||
to at most one reopen per second), so an operator can move
|
||||
`archive-{uuid}.db` away for offline retention while the service runs,
|
||||
to at most one reopen per second), so an operator can move an
|
||||
`archive-….db` away for offline retention while the service runs,
|
||||
and it is recreated on the next write. See
|
||||
[Database Architecture](#database-architecture). That is a
|
||||
move-the-file-away workflow, not a substitute for the backup procedures
|
||||
@@ -1036,7 +1037,7 @@ happens on the next write past the debounce window, when the connection
|
||||
pool retires the idle connection (about a minute after the last write),
|
||||
or at the idle archive sweep — measured, the same file was a complete
|
||||
20 KB `.db` with no sidecars about a minute after its last write. A
|
||||
clean stop closes it too. So either move `archive-{uuid}.db` together
|
||||
clean stop closes it too. So either move the `archive-….db` together
|
||||
with any `-wal`/`-shm` beside it, or wait until there are none.
|
||||
|
||||
### Restore
|
||||
@@ -1171,7 +1172,7 @@ commit still produce a byte-identical binary.
|
||||
Treat a backup with the same care as the credentials inside it. Encrypt
|
||||
backups at rest and restrict who can read them.
|
||||
|
||||
- `events-{uuid}.db` and `archive-{uuid}.db` hold the **full payload
|
||||
- `events-{uuid}.db` and `archive-….db` hold the **full payload
|
||||
body and headers** of every event as received, including whatever the
|
||||
sending service put in them — tokens, signatures, personal data.
|
||||
- Event databases written before
|
||||
@@ -1589,8 +1590,9 @@ events should be forwarded.
|
||||
is built on the same HTTP core as `http` and honours `max_retries`
|
||||
identically, circuit breaker included. See the Slack target section
|
||||
under "Per-Webhook Event Databases" for the message format.
|
||||
- **`database`** — Archive the full event as a row into a separate
|
||||
per-webhook archive database (`archive-{webhookID}.db`) for long-term
|
||||
- **`database`** — Archive the full event as a row into the target's
|
||||
own archive database
|
||||
(`archive-{webhook_name}-{target_name}-{target_uuid}.db`) for long-term
|
||||
retention, with an optional creation-validated expiry (default: keep
|
||||
forever). No external delivery and no retries; an archive write
|
||||
failure fails the delivery. See the database target section under
|
||||
@@ -1904,9 +1906,35 @@ The **database target type** builds on this architecture to provide
|
||||
long-term archiving, separate from the per-webhook event database (which
|
||||
may prune events under its own retention). Delivering to a database
|
||||
target writes the full event — body, headers, method, content type, and
|
||||
webhook/entrypoint/event identifiers — as a row into a dedicated archive
|
||||
database, `archive-{webhookID}.db`, stored under the data directory
|
||||
beside the event database. After each write the archive handle is closed
|
||||
webhook/entrypoint/event identifiers — as a row into the target's own
|
||||
archive database, `archive-{webhook_name}-{target_name}-{target_uuid}.db`,
|
||||
stored under the data directory beside the event database. Each
|
||||
`database` target has its own archive file, even when one webhook has
|
||||
several.
|
||||
|
||||
Both names are made safe for a file name the same way: lowercased, ASCII
|
||||
letters and digits kept, every other run of characters turned into a
|
||||
single `-`, no `-` at either end, cut to 40 characters, and `unnamed`
|
||||
when nothing is left. The target UUID keeps the file name unique. A
|
||||
webhook named `Orders (EU)` with a target named `Long-term archive`
|
||||
archives into `archive-orders-eu-long-term-archive-{target_uuid}.db`.
|
||||
Renaming the webhook or the target renames the file, under the same
|
||||
lock the archive writes and the archive sweeper take, so the name on
|
||||
disk matches the UI. A rename never replaces a file: if one already has
|
||||
the new name, the edit is refused with an error naming that file, and
|
||||
the stored name stays. If the archive is not there (the operator moved
|
||||
it away), the rename is not an error, and the next write creates the
|
||||
file under the new name.
|
||||
|
||||
The file is moved just before the new name is saved. If the process
|
||||
stops between the two, the archive is left under the new name while the
|
||||
UI still shows the old one, and the next delivery starts a second
|
||||
archive under the name shown. To bring them back together, move the
|
||||
file under the new name back to the name shown; if a second archive is
|
||||
already there, move the older file out of the data directory instead
|
||||
and keep it as you would any archive moved away.
|
||||
|
||||
After each write the archive handle is closed
|
||||
and reopened, debounced to at most once per second, so an operator can
|
||||
move the archive file away for offline archiving without stopping the
|
||||
service; a moved or removed archive file is recreated automatically on
|
||||
@@ -1917,35 +1945,33 @@ older than the expiry are pruned each time the archive is (re)opened. An
|
||||
archive write failure is never silent success: the delivery records a
|
||||
failed attempt with the error and is marked failed.
|
||||
|
||||
Because reopens only happen on writes, an archive belonging to a webhook
|
||||
that has stopped receiving events would never be pruned. A background
|
||||
**archive sweeper** closes that gap: on the same interval as the event
|
||||
retention reaper (`RETENTION_SWEEP_INTERVAL`) it prunes every archive
|
||||
whose database target declares a positive expiry, whether or not the
|
||||
webhook is still receiving traffic. The sweep never creates an archive —
|
||||
a webhook whose archive file does not yet exist is skipped, not
|
||||
initialised — it takes the same per-webhook lock the write path uses, so
|
||||
it can never interleave with a write, and it leaves the archive closed
|
||||
afterwards so the move-the-file-away workflow keeps working. Archives
|
||||
with no expiry, or the expiry `never`, are not touched by the sweep at
|
||||
all.
|
||||
Because reopens only happen on writes, an archive whose target has
|
||||
stopped receiving events would never be pruned. A background **archive
|
||||
sweeper** closes that gap: on the same interval as the event retention
|
||||
reaper (`RETENTION_SWEEP_INTERVAL`) it prunes every archive whose
|
||||
database target declares a positive expiry, whether or not the target
|
||||
is still receiving traffic. The sweep never creates an archive — a
|
||||
target whose archive file does not yet exist is skipped, not initialised
|
||||
— it takes the same per-target lock the write path uses, so it can never
|
||||
interleave with a write, and it leaves the archive closed afterwards so
|
||||
the move-the-file-away workflow keeps working. Archives with no expiry,
|
||||
or the expiry `never`, are not touched by the sweep at all.
|
||||
|
||||
Note that a webhook has one archive file but may carry more than one
|
||||
`database` target, each with its own `expiry`. The shortest expiry
|
||||
configured on any of them therefore governs the whole archive, and the
|
||||
sweep applies it whether or not the webhook is still receiving events.
|
||||
Configure a single `database` target per webhook unless you intend that.
|
||||
Because each `database` target has its own archive file, a target's
|
||||
`expiry` governs only its own archive. Two `database` targets on one
|
||||
webhook with different expiries keep two archives, each pruned on its
|
||||
own schedule.
|
||||
|
||||
Deleting a webhook releases its archive: the delivery engine's cached
|
||||
archive writer is dropped and its file handle closed, so nothing lingers
|
||||
after the webhook is gone. The archive **file itself is deliberately
|
||||
left on disk**. Unlike the event database — per-webhook working storage
|
||||
that is hard-deleted with the webhook — an archive is long-term storage
|
||||
an operator may still want to keep or move away for offline retention,
|
||||
and destroying it as a side effect of deleting a webhook would be
|
||||
unrecoverable. Removing `archive-{webhookID}.db` is the operator's call.
|
||||
Deleting a webhook's last `database` target releases the writer the same
|
||||
way, and for the same reason leaves the file alone.
|
||||
Deleting a webhook releases its archives: the delivery engine's cached
|
||||
archive writers are dropped and their file handles closed, so nothing
|
||||
lingers after the webhook is gone. The archive **files themselves are
|
||||
deliberately left on disk**. Unlike the event database — per-webhook
|
||||
working storage that is hard-deleted with the webhook — an archive is
|
||||
long-term storage an operator may still want to keep or move away for
|
||||
offline retention, and destroying it as a side effect of deleting a
|
||||
webhook would be unrecoverable. Removing an `archive-….db` is the
|
||||
operator's call. Deleting a `database` target releases its writer the
|
||||
same way, and for the same reason leaves its file alone.
|
||||
|
||||
The **Slack target type** sends webhook events as formatted messages to
|
||||
any Slack-compatible incoming webhook URL (works with Slack, Mattermost,
|
||||
@@ -2955,8 +2981,8 @@ Components are wired via Uber fx in this order:
|
||||
11. `delivery.New` — Event-driven delivery engine
|
||||
12. `delivery.NewArchiveSweeper` — Periodic pruning of idle archives
|
||||
13. `delivery.Engine` → `delivery.Notifier` — interface bridge
|
||||
14. `delivery.Engine` → `delivery.WebhookEvictor` — interface bridge so
|
||||
deleting a webhook releases its archive writer
|
||||
14. `delivery.Engine` → `delivery.Archives` — interface bridge so
|
||||
deleting or renaming a webhook or target reaches its archive files
|
||||
15. `server.New` — HTTP server and router
|
||||
|
||||
The server starts via `fx.Invoke(func(*server.Server, *delivery.Engine,
|
||||
|
||||
Reference in New Issue
Block a user