Files

check / check (push) Successful in 2m47s

Details

Evict archive writers on deletion and sweep idle archives (closes #89 )

The per-webhook archiveWriter registry in the database delivery target
was never evicted, so a deleted webhook's writer -- and any archive
file handle open within its debounce window -- lingered for the process
lifetime. Separately, expiry pruning ran only when an archive was
(re)opened, and reopens only happen on writes, so an archive belonging
to a webhook that stopped receiving events kept its expired rows
forever.

Eviction: a new one-method delivery.WebhookEvictor interface (kept
separate from Notifier: archiving lifecycle is not notification) is
implemented by the Engine and injected into the handlers. Deleting a
webhook, or deleting its last database target, drops the writer from
the registry and closes its handle under the writer's own mutex, so
eviction can never race an in-flight write. An evicted writer refuses
further writes rather than reopening a file nothing holds. The archive
file is deliberately left on disk: it is long-term storage an operator
may want to keep or move away, and destroying it as a side effect of
deleting a webhook would be unrecoverable.

Idle sweep: a new ArchiveSweeper, modelled on the event RetentionReaper
(fx lifecycle hooks, cancellable context, WaitGroup, ticker loop),
prunes archives whose database target declares a positive expiry. It
reuses the existing RETENTION_SWEEP_INTERVAL rather than adding a
config key. It never creates an archive -- a missing file is skipped,
and the reopen uses SQLite mode=rw so the file cannot be conjured even
if it disappears mid-sweep -- routes the prune through the per-webhook
writer so its mutex orders the sweep against concurrent writes, and
leaves the archive closed so the move-the-file-away workflow keeps
working. A failure for one webhook is logged and the sweep continues.
Archives with no expiry or the expiry "never" are untouched.

The sweep loop's context is rooted at context.Background(), not at the
fx OnStart hook context. The hook context carries fx's 15 second start
timeout, so a loop derived from it is cancelled three quarters of an
hour before the first tick under the default one-hour interval, giving
a sweeper that never sweeps. OnStop still cancels the loop and waits on
the WaitGroup, so shutdown is unchanged.

The sweep also never leaves a registry entry behind. Reaching the
writer through the ordinary create-and-cache accessor would let a sweep
that raced a webhook deletion re-insert a writer for a webhook that no
longer exists, which nothing would ever evict again -- the very leak
this change closes. An entry the sweep has to create is marked
sweep-owned and released when the prune finishes, unless a delivery
claimed it meanwhile, in which case it belongs to the registry and an
eviction can still reach it. A writer evicted underneath a sweep is an
ordinary interleaving and is logged at debug, not error.

Each of those guards is pinned by a test that fails when the guard is
removed. The adopt-during-sweep window -- a delivery claiming the
sweep's own registry entry while that sweep is still running -- is
driven directly against the registry, because a delivery placed between
two sweeps never reaches the release path at all. The requirement that
an idle archive ends the sweep closed is asserted both on a writer
proven to hold an open handle beforehand and, end to end, on a
delivery-owned entry the sweep keeps, rather than on an entry the sweep
has already released and which therefore reports "not open" either
way. The "never" expiry short circuit is checked against an archive
file that has never been migrated, so any open of it would be
observable as a created table.

2026-08-09 05:22:49 +00:00

4.1 KiB

Raw Permalink Blame History

Workflow

branch (from main)
do the work in Next Step
move Next Step to the top of Completed Steps
move the top item of Future Steps into Next Step
commit (TODO.md changes in the same commit as the work)
merge to main if the branch is not protected, otherwise open a PR
push

Status

pre-1.0. No git tags exist. main (4f5ecb1) is a working webhook proxy with auth, CSRF/SSRF protections, login rate limiting, Slack target, event retention (#63), the database archiving target (#43), the admin password change flow (#65), policy compliance (#6), and pinned lint tooling (#55). Note: TODO.md was deliberately deleted from this repo in f9a9569 (2026-03-01, #6); its content was folded into the README TODO section, which this draft reconstructs as of 2026-07-06.

Next Step

Implement automatic event retention cleanup based on retention_days: a periodic maintenance job that deletes Events, Deliveries, and DeliveryResults older than the parent webhook's retention_days from each per-webhook event database. The field exists on the Webhook model and the README promises the behavior, but nothing enforces it, so event databases currently grow without bound.

Completed Steps

2026-08-09 Archive writer lifecycle (#89): deleting a webhook (or its last database target) evicts the cached archive writer and closes its handle while deliberately leaving archive-{webhookID}.db on disk, and a new ArchiveSweeper prunes idle archives on the existing RETENTION_SWEEP_INTERVAL without ever creating an archive file
2026-08-07 Update golangci-lint to v2.12.2 (Docker image digest in Dockerfile, release-archive sha256 pins in script/bootstrap), adopt the canonical .golangci.yml (v2 linters.settings layout so lll/funlen/cyclop/dupl thresholds actually apply), and fix all newly surfaced lint findings
2026-07-07 Adopted scripts-to-rule-them-all: script/ entrypoints, Makefile shims, README Entrypoints section
2026-03-25 pin golangci-lint Docker image for linting (#55)
2026-03-18 CSRF middleware detects TLS per-request, fixing login over plain HTTP and behind reverse proxies (#54)
2026-03-17 root path redirects based on auth state (#52)
2026-03-17 CSRF protection, SSRF prevention for HTTP delivery targets with DNS rebinding defense, and per-IP login rate limiting (#42)
2026-03-17 Slack target type for incoming webhook notifications (#47)
2026-03-17 Dockerfile absolute paths and static linking (#49); absolute dev DATA_DIR default and clarified env docs (#46)
2026-03-05 security headers middleware, session regeneration on login, request body size limits (#41)
2026-03-04 tests for delivery, middleware, and session packages (#32); removed globals.Buildarch (#31)
2026-03-04 1.0 MVP merge: Webhook/Entrypoint/Target rename, core delivery engine with bounded worker pool and circuit breaker, parallel fan-out, per-webhook event databases, management UI (#16)
2026-03-01 repo brought to REPO_POLICIES standards; TODO.md folded into README (#6)

Future Steps

Manual event redelivery from the web UI (replay is a core promised capability in the README rationale)
Delivery status and retry management UI
Per-webhook rate limiting in the receiver handler (per-webhook config plus handler enforcement; global limits must not apply to receiver endpoints)
Webhook signature verification for GitHub and Stripe HMAC formats
API key authentication for programmatic access (APIKey model exists; Bearer token middleware does not)
REST API v1
- CRUD for webhooks, entrypoints, targets
- event viewing and filtering endpoints
- event redelivery endpoint
- OpenAPI specification
Analytics dashboard: success rates, response times, volume
Session expiration tuning and a remember-me option
Password change and reset flow
Later, nice to have
- email delivery target type
- SNS and S3 delivery targets
- data transformations (e.g. webhook to Slack message formatting)
- JSONL file delivery with periodic S3 upload
- webhook event search and filtering
- multi-user with role-based access control