Enforce per-webhook event retention (RetentionDays reaper) #63

New Issue

clawbot · 2026-08-07T13:11:03+02:00

clawbot commented

2026-08-07 13:11:03 +02:00

Part of the road to 1.0 (see #33).

Webhook.RetentionDays (internal/database/model_webhook.go, gorm:"default:30") is stored but never enforced. Nothing deletes old Event, Delivery, or DeliveryResult rows, so each per-webhook SQLite file (internal/database/webhook_db_manager.go) grows without bound for the life of the service. For an internet-facing deployment this is unbounded disk growth.

Definition of done:

a periodic background job deletes events (and their dependent deliveries/results) older than each webhook's RetentionDays, across all per-webhook databases
RetentionDays <= 0 means retain forever (no deletion)
the sweep interval is configurable (env/config) with a sane default
covered by a test that seeds old and recent events and asserts only the expired ones are removed

Part of the road to 1.0 (see #33). `Webhook.RetentionDays` (`internal/database/model_webhook.go`, `gorm:"default:30"`) is stored but never enforced. Nothing deletes old `Event`, `Delivery`, or `DeliveryResult` rows, so each per-webhook SQLite file (`internal/database/webhook_db_manager.go`) grows without bound for the life of the service. For an internet-facing deployment this is unbounded disk growth. Definition of done: - a periodic background job deletes events (and their dependent deliveries/results) older than each webhook's `RetentionDays`, across all per-webhook databases - `RetentionDays <= 0` means retain forever (no deletion) - the sweep interval is configurable (env/config) with a sane default - covered by a test that seeds old and recent events and asserts only the expired ones are removed

clawbot added this to the 1.0.0 milestone 2026-08-07 13:11:03 +02:00

clawbot referenced this issue

2026-08-07 13:12:23 +02:00

1.0/mvp #33

clawbot referenced this issue

2026-08-07 13:16:27 +02:00

Clean up and clarify the web UI #57

clawbot commented

2026-08-07 14:06:23 +02:00

Implementation instructions

Implement the retention reaper as a cohesive, self-contained unit.

Files: a new reaper in one place (e.g. internal/database/retention.go, or a small new internal/retention package — your choice), internal/config/config.go (add the sweep interval), and the fx wiring where the app assembles its components (inspect cmd/webhooker/main.go and internal/server/ to find where providers and lifecycle hooks are registered). Do NOT edit routes.go, http.go, engine.go, source_management.go, or middleware.go — other work is in flight there.

Behaviour:

a background job runs on a ticker at a configurable interval — add a RetentionSweepInterval to config.go with an env var and a sane default such as 1h, following the existing config field/env conventions in that file
on each sweep: list all webhooks from the main DB (the Webhook table carries RetentionDays, default 30). For each webhook with a positive RetentionDays, open its per-webhook DB via the existing WebhookDBManager.GetDB(webhookID) and delete every Event row (and its dependent Delivery and DeliveryResult rows) whose CreatedAt is older than RetentionDays days. Read the Event / Delivery / DeliveryResult models first to get the foreign-key relationships, and delete in the correct order (results, then deliveries, then events) so nothing is orphaned.
a RetentionDays of 0 or less means retain forever — skip that webhook
start the job with an fx OnStart lifecycle hook and stop it cleanly on OnStop (context cancellation), matching how other lifecycle components are wired

Definition of done:

expired events (and their deliveries/results) are deleted per webhook according to RetentionDays; recent rows are kept
the sweep interval is configurable via env/config with a sane default
a test seeds old and recent events in a per-webhook DB and asserts only the expired rows (and their dependents) are removed

Gates and process:

make fmt; validate with docker build . (must exit 0)
branch from main named issue-63-retention-reaper; commit subject ends with (closes #63)
open a PR (base main) and comment on it with the diff summary and the docker build . result
no AI-assistant/tooling references anywhere

## Implementation instructions Implement the retention reaper as a cohesive, self-contained unit. Files: a new reaper in one place (e.g. `internal/database/retention.go`, or a small new `internal/retention` package — your choice), `internal/config/config.go` (add the sweep interval), and the fx wiring where the app assembles its components (inspect `cmd/webhooker/main.go` and `internal/server/` to find where providers and lifecycle hooks are registered). Do NOT edit `routes.go`, `http.go`, `engine.go`, `source_management.go`, or `middleware.go` — other work is in flight there. Behaviour: - a background job runs on a ticker at a configurable interval — add a `RetentionSweepInterval` to `config.go` with an env var and a sane default such as 1h, following the existing config field/env conventions in that file - on each sweep: list all webhooks from the main DB (the `Webhook` table carries `RetentionDays`, default 30). For each webhook with a positive `RetentionDays`, open its per-webhook DB via the existing `WebhookDBManager.GetDB(webhookID)` and delete every `Event` row (and its dependent `Delivery` and `DeliveryResult` rows) whose `CreatedAt` is older than `RetentionDays` days. Read the `Event` / `Delivery` / `DeliveryResult` models first to get the foreign-key relationships, and delete in the correct order (results, then deliveries, then events) so nothing is orphaned. - a `RetentionDays` of 0 or less means retain forever — skip that webhook - start the job with an fx `OnStart` lifecycle hook and stop it cleanly on `OnStop` (context cancellation), matching how other lifecycle components are wired Definition of done: - expired events (and their deliveries/results) are deleted per webhook according to `RetentionDays`; recent rows are kept - the sweep interval is configurable via env/config with a sane default - a test seeds old and recent events in a per-webhook DB and asserts only the expired rows (and their dependents) are removed Gates and process: - `make fmt`; validate with `docker build .` (must exit 0) - branch from `main` named `issue-63-retention-reaper`; commit subject ends with ` (closes #63)` - open a PR (base `main`) and comment on it with the diff summary and the `docker build .` result - no AI-assistant/tooling references anywhere

sneak referenced this issue from a commit

2026-08-07 15:19:04 +02:00

Add per-webhook event retention reaper (closes #63)

clawbot referenced a pull request that will close this issue

2026-08-07 15:19:20 +02:00

Add per-webhook event retention reaper (closes #63) #78

clawbot referenced this issue

2026-08-07 15:26:51 +02:00

RetentionDays cannot be set to 0 (retain forever) via the normal create path #79

clawbot referenced this issue

2026-08-07 15:29:22 +02:00

Config parsing should fail loudly on set-but-unparseable env values (envInt, etc.) #80