Add per-webhook event retention reaper (closes #63) (#78)
All checks were successful
check / check (push) Successful in 2m42s

Enforces each webhook's `RetentionDays` so per-webhook SQLite files no longer grow without bound.

## Reaper

New `RetentionReaper` in `internal/database/retention.go`. A background ticker runs each sweep: it lists all webhooks from the main DB and, for each webhook with a positive `RetentionDays`, opens its per-webhook DB via `WebhookDBManager.GetDB` and deletes every `Event` (and its dependent `Delivery` and `DeliveryResult` rows) whose `CreatedAt` is older than `RetentionDays` days.

- Deletions run in foreign-key-safe order: delivery results, then deliveries, then events.
- Deletes are unscoped (hard deletes) so rows are physically removed and disk is reclaimed, rather than GORM soft-deleting them.
- `RetentionDays <= 0` means retain forever; those webhooks are skipped.
- Webhooks whose per-webhook DB does not yet exist are skipped.

## Config

`internal/config/config.go` gains `RetentionSweepInterval` (env `RETENTION_SWEEP_INTERVAL`, parsed as a Go duration, default `1h`) via a new `envDuration` helper, following the existing env-helper conventions.

## Wiring

`cmd/webhooker/main.go` registers `database.NewRetentionReaper` as an fx provider and forces its construction in `fx.Invoke`. The reaper starts its sweep loop on an fx `OnStart` hook and stops cleanly on `OnStop` via context cancellation, matching the existing lifecycle components.

## Test

`internal/database/retention_test.go` seeds an old event chain (event + delivery + result, 40 days old) and a recent one (1 day old) in a real per-webhook DB and asserts a single sweep removes only the expired chain while keeping the recent one. A second test forces a non-positive `RetentionDays` and asserts an ancient event is retained.

Note: the `Webhook.RetentionDays` column carries `gorm:"default:30"`, so a `0` passed to a GORM `Create` is replaced by the default; the test forces the value with an explicit column update to exercise the retain-forever path. No model changes were made.

Validated with `docker build .` (fmt-check, lint, test, build) exit 0.

Closes #63

Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #78
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
This commit was merged in pull request #78.
This commit is contained in:
2026-08-07 16:15:13 +02:00
committed by Jeffrey Paul
parent 8ea7f76540
commit f6b929f2d7
6 changed files with 722 additions and 13 deletions

View File

@@ -8,6 +8,7 @@ import (
"os"
"strconv"
"strings"
"time"
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/globals"
@@ -26,6 +27,10 @@ const (
// defaultPort is the default HTTP listen port.
defaultPort = 8080
// defaultRetentionSweepInterval is how often the retention
// reaper deletes events older than each webhook's RetentionDays.
defaultRetentionSweepInterval = time.Hour
)
// ErrInvalidEnvironment is returned when WEBHOOKER_ENVIRONMENT
@@ -51,8 +56,12 @@ type Config struct {
MetricsUsername string
Port int
SentryDSN string
params *ConfigParams
log *slog.Logger
// RetentionSweepInterval is how often the retention reaper runs.
RetentionSweepInterval time.Duration
params *ConfigParams
log *slog.Logger
}
// IsDev returns true if running in development environment.
@@ -95,6 +104,30 @@ func envInt(key string, defaultValue int) int {
return defaultValue
}
// envDuration returns the value of the named environment variable
// parsed as a Go duration (e.g. "1h", "30m"). Returns defaultValue if
// not set. If the variable is set but cannot be parsed, it returns a
// wrapped error naming the key and the bad value, so startup fails
// loudly rather than silently falling back to the default.
func envDuration(
key string,
defaultValue time.Duration,
) (time.Duration, error) {
v := os.Getenv(key)
if v == "" {
return defaultValue, nil
}
d, err := time.ParseDuration(v)
if err != nil {
return 0, fmt.Errorf(
"invalid duration for %s: %q: %w", key, v, err,
)
}
return d, nil
}
// New creates a Config by reading environment variables.
//
//nolint:revive // lc parameter is required by fx even if unused.
@@ -118,18 +151,30 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
)
}
// Parse the retention sweep interval; a set-but-unparseable value
// is a hard error so fx aborts startup rather than silently using
// the default.
retentionSweepInterval, err := envDuration(
"RETENTION_SWEEP_INTERVAL",
defaultRetentionSweepInterval,
)
if err != nil {
return nil, err
}
// Load configuration values from environment variables
s := &Config{
DataDir: envString("DATA_DIR"),
Debug: envBool("DEBUG", false),
MaintenanceMode: envBool("MAINTENANCE_MODE", false),
Environment: environment,
MetricsUsername: envString("METRICS_USERNAME"),
MetricsPassword: envString("METRICS_PASSWORD"),
Port: envInt("PORT", defaultPort),
SentryDSN: envString("SENTRY_DSN"),
log: log,
params: &params,
DataDir: envString("DATA_DIR"),
Debug: envBool("DEBUG", false),
MaintenanceMode: envBool("MAINTENANCE_MODE", false),
Environment: environment,
MetricsUsername: envString("METRICS_USERNAME"),
MetricsPassword: envString("METRICS_PASSWORD"),
Port: envInt("PORT", defaultPort),
SentryDSN: envString("SENTRY_DSN"),
RetentionSweepInterval: retentionSweepInterval,
log: log,
params: &params,
}
// Set default DataDir. All SQLite databases (main application
@@ -151,6 +196,7 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
"debug", s.Debug,
"maintenanceMode", s.MaintenanceMode,
"dataDir", s.DataDir,
"retentionSweepInterval", s.RetentionSweepInterval.String(),
"hasSentryDSN", s.SentryDSN != "",
"hasMetricsAuth",
s.MetricsUsername != "" && s.MetricsPassword != "",