5 Commits

Author SHA1 Message Date
36a1bacf11 Update golangci-lint to v2.12.2 with canonical config
All checks were successful
check / check (push) Successful in 2m48s
Bump the golangci-lint Docker image pin in Dockerfile and the
release-archive sha256 pins in script/bootstrap from 2.11.3 to
2.12.2, and replace .golangci.yml with the canonical config. The
canonical config moves lll/funlen/cyclop/dupl settings from the
top-level linters-settings key (ignored by the v2 schema) to
linters.settings, so those thresholds now actually apply.

Fix all findings the newly applied thresholds surfaced:

- lll: wrap or shorten seven over-length lines (struct tag
  comments, test logger construction, a func signature, and a
  nosec comment)
- goconst: use http.MethodPost/http.MethodPut and new shared
  constants for repeated test strings; add tmplKeyError and
  tmplKeyWebhook constants for template data keys in handlers
- dupl: merge buildHTTPTargetConfig and buildSlackTargetConfig
  into a parameterized buildURLTargetConfig; drop the duplicate
  iWebhookDB test helper in favor of testWebhookDB; extract
  shared helpers in middleware and session tests
2026-08-07 20:55:31 +00:00
ee7c626071 Implement the database archiving target (closes #43) (#84)
All checks were successful
check / check (push) Successful in 4s
Implements the `databaseTarget` as a real archiving target, replacing the always-successful stub. Delivering to a `database` target now writes the full event into a per-webhook archive SQLite file for long-term storage.

## Archive-writer semantics

- **Separate file:** each webhook's full events are written as rows into `archive-{webhookID}.db` under the data dir, distinct from the per-webhook event DB (`events-{webhookID}.db`). The file and its schema are created on first write if missing. Each row carries the full event: body, headers, method, content type, webhook id, entrypoint id, event id, and an archived-at timestamp.
- **Close/reopen with debounce:** after each write the archive handle is closed and reopened, unless the last (re)open was less than one second ago. This lets an operator move the archive file away for offline archiving while bounding file churn under load. A per-webhook `archiveWriter` owns this debounce state and serialises writes.
- **Auto-recreate:** the file is opened create-if-missing (`mode=rwc`) and its schema re-migrated on every open, so if the archive was moved or removed since the last open, the next write recreates it. The writer also detects a missing file before writing and reopens first, so a moved-away file is recreated rather than lost.
- **Optional expiry, validated at creation:** an optional `expiry` in the target's config JSON (e.g. `{"expiry":"720h"}`) is validated when the target is created (`ValidateArchiveExpiry`; bad values are rejected with a 400 at the add-target form, the Slack URL precedent). The default (missing, empty, or `"never"`) keeps rows forever with no pruning. When a positive duration is set, rows older than it (measured from each row's archived-at time) are pruned on every (re)open; because the file is reopened after writes, prune-on-open keeps the archive swept without a separate background sweeper. A set-but-invalid expiry in a stored config (unparseable, zero, or negative) is an error at delivery time too — never a silent default.
- **No-retry, fail-loud:** the target performs a single attempt with no retries. On success it records one successful attempt and marks the delivery delivered. If the archive write fails, the attempt is recorded as failed with the error and the delivery is marked failed — archiving errors never report success.

## Scope

- `internal/delivery/target_database.go` — the `databaseTarget` (no-retry) archives via a per-webhook writer registry; an archive error records a failed attempt and marks the delivery failed.
- `internal/delivery/target_database_archive.go` (new) — the `archiveWriter`, the archived-row model, config/expiry parsing (fail-loud on set-but-invalid values), `ValidateArchiveExpiry`, and prune-on-open.
- `internal/handlers/source_management.go` — database targets get a creation-validated `expiry` config (`buildDatabaseTargetConfig`); the expiry form value is read where the request body is bounded and bad values are rejected with a 400 at target creation.
- `templates/source_detail.html` — the add-target form shows an expiry field for database targets.
- `README.md` — the database-target documentation describes the archiving semantics.
- `internal/delivery/export_test.go`, `internal/delivery/target_database_test.go`, `internal/handlers` tests — tests and their exported shims.

No changes to the `Target` interface or other targets.

## Tests

- a row is archived (both at the writer level and end-to-end through `Deliver`)
- a forced archive failure (bad stored expiry config) yields a `Failed` delivery with a non-success `DeliveryResult` carrying the error and no archive file created
- the file is recreated after removal, with only the post-removal row
- the one-second reopen debounce (rapid writes reopen once; a write after the window reopens again)
- expiry pruning removes rows older than the configured expiry
- expiry config parsing (empty / `never` / duration accepted; unparseable, zero, and negative values error)
- expiry validation at target creation (`TestValidateArchiveExpiry`; valid values build the config, bad values get a 400)

## Validation

`docker build .` exits 0 (fmt-check, lint, test, build all pass).

Closes #43

Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #84
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-08-07 22:50:08 +02:00
81413c56e9 Refactor delivery targets to a Target interface (closes #77) (#81)
All checks were successful
check / check (push) Successful in 2m42s
Refactors the delivery engine so each target TYPE is an implementation of a `Target` interface, dispatched from a registry, with each target owning its full delivery including durable retries. Implements the authoritative design from issue #77 (the corrected "hand the DB + Scheduler to the target" design).

## The new interface

```go
type Scheduler interface {
    ScheduleRetry(task Task, delay time.Duration)
}

type Target interface {
    Deliver(ctx context.Context, webhookDB *gorm.DB,
        d *database.Delivery, task *Task, sched Scheduler)
}
```

`Deliver` receives everything a target needs to be autonomous and durable: the request context, the per-webhook `*gorm.DB`, the `*database.Delivery`, the attempt `*Task`, and a `Scheduler` (the engine) for durable re-enqueue. The target makes one attempt, writes the `DeliveryResult`, updates `DeliveryStatus`, and — for retry targets — decides whether to retry, computes its own backoff, gates with its own circuit breaker, and reschedules via the injected `Scheduler`.

`processDelivery` collapses to a registry lookup (`map[database.TargetType]Target`) and a `Deliver` call; an unknown target type still fails the delivery as before.

## Per-target ownership

- `httpTarget` and `slackTarget` share a retry core (`httpCore`) that owns retry, exponential backoff, and the per-target circuit breaker. The core is fire-and-forget when `MaxRetries == 0` and adds breaker-gated backed-off retries when `MaxRetries > 0`. The per-attempt request differs (HTTP forwards the body + filtered headers; Slack posts a formatted message) and is supplied as a closure, so each keeps its exact recording semantics (e.g. HTTP records no error string for a non-2xx, Slack records `HTTP <code>`).
- `databaseTarget` and `logTarget` are fire-and-forget: they record a single successful attempt.

Moved wholesale into the http/slack targets: `deliverHTTP*`, `handleHTTPRetry`, `circuitBreakerBlock`, `calcBackoff` / `calcRemainingBackoff` / `backoffElapsed`, the circuit-breaker `sync.Map` + `getCircuitBreaker`, `clientForConfig`, `doHTTPRequest`, `applyRequestHeaders`, and the config parsers. The engine keeps `recordResult`, `updateDeliveryStatus`, and `ScheduleRetry`.

## Slack MaxRetries gating

Slack is now on the same shared core as HTTP, with retry + breaker gated on `MaxRetries`. A `MaxRetries` of 0 stays single-attempt fire-and-forget, so **every existing Slack target is unchanged**; a Slack target configured with retries gets backoff + circuit breaker.

## Log-target full content

`logTarget` now logs the ENTIRE inbound webhook — full request body and full request headers, plus method, content type, and the webhook id and entrypoint id — rather than a summary line. This supersedes the smaller log-summary work (#70).

## `Task.EntrypointID`

To carry the entrypoint id to the log target, `Task` gains an `EntrypointID` field, populated in the webhook handler's `buildDeliveryTasks`, the engine's recovery-task builder, and `buildEventFromTask`.

## Durability / recovery

The crash-durable async retry model is preserved unchanged: one attempt per worker turn; on failure the status is set `retrying`, backoff is computed, and the task is re-enqueued via `ScheduleRetry` (a `time.AfterFunc` onto the retry channel). On restart, `recoverRetryingDeliveries` and the 60s sweep hand each orphaned `retrying` delivery back to its target to recompute the remaining backoff and reschedule (targets that own retries implement an internal `rescheduler`; fire-and-forget targets, which never produce `retrying` deliveries, are skipped).

## How behaviour is preserved

No external behaviour changes except the two called out above (log target full content; Slack gaining `MaxRetries`-gated retries). All existing delivery tests pass with only their `export_test.go` wrappers re-pointed at the new structure — `ExportDeliverHTTP/Slack/Database/Log` now call the targets, `ExportGetCircuitBreaker` / `ExportClient` / `ExportClientForConfig` / `ExportDoHTTPRequest` resolve against the HTTP target's shared client and breaker map, and `ExportParseHTTPConfig` / `ExportParseSlackConfig` call the relocated free functions. Added: a `logTarget` test asserting the log line contains the full body, headers, and ids, and a Slack `MaxRetries`-gated retry test.

`docker build .` is green (fmt-check, lint, test, static build all pass).

Closes #77

Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #81
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-08-07 17:07:49 +02:00
f6b929f2d7 Add per-webhook event retention reaper (closes #63) (#78)
All checks were successful
check / check (push) Successful in 2m42s
Enforces each webhook's `RetentionDays` so per-webhook SQLite files no longer grow without bound.

## Reaper

New `RetentionReaper` in `internal/database/retention.go`. A background ticker runs each sweep: it lists all webhooks from the main DB and, for each webhook with a positive `RetentionDays`, opens its per-webhook DB via `WebhookDBManager.GetDB` and deletes every `Event` (and its dependent `Delivery` and `DeliveryResult` rows) whose `CreatedAt` is older than `RetentionDays` days.

- Deletions run in foreign-key-safe order: delivery results, then deliveries, then events.
- Deletes are unscoped (hard deletes) so rows are physically removed and disk is reclaimed, rather than GORM soft-deleting them.
- `RetentionDays <= 0` means retain forever; those webhooks are skipped.
- Webhooks whose per-webhook DB does not yet exist are skipped.

## Config

`internal/config/config.go` gains `RetentionSweepInterval` (env `RETENTION_SWEEP_INTERVAL`, parsed as a Go duration, default `1h`) via a new `envDuration` helper, following the existing env-helper conventions.

## Wiring

`cmd/webhooker/main.go` registers `database.NewRetentionReaper` as an fx provider and forces its construction in `fx.Invoke`. The reaper starts its sweep loop on an fx `OnStart` hook and stops cleanly on `OnStop` via context cancellation, matching the existing lifecycle components.

## Test

`internal/database/retention_test.go` seeds an old event chain (event + delivery + result, 40 days old) and a recent one (1 day old) in a real per-webhook DB and asserts a single sweep removes only the expired chain while keeping the recent one. A second test forces a non-positive `RetentionDays` and asserts an ancient event is retained.

Note: the `Webhook.RetentionDays` column carries `gorm:"default:30"`, so a `0` passed to a GORM `Create` is replaced by the default; the test forces the value with an explicit column update to exercise the retain-forever path. No model changes were made.

Validated with `docker build .` (fmt-check, lint, test, build) exit 0.

Closes #63

Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #78
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-08-07 16:15:13 +02:00
8ea7f76540 Add NoCache middleware for authenticated pages (closes #61) (#75)
All checks were successful
check / check (push) Successful in 6s
Adds a `NoCache()` middleware that sets `Cache-Control: no-store` and `Pragma: no-cache`, and wires it onto the dynamic app route groups (`/pages`, `/user/{username}`, `/sources`, `/source/{sourceID}`) adjacent to their existing `CSRF()` call. The static `/s` mount, `/metrics`, `/webhook/{uuid}`, and `/.well-known/healthcheck` are intentionally left untouched (static assets are safe to cache; the others are not authenticated pages).

A middleware unit test asserts both headers are set.

Closes #61

Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #75
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-08-07 15:33:44 +02:00
40 changed files with 3317 additions and 1189 deletions

View File

@@ -1,5 +1,9 @@
version: "2" version: "2"
# Config schema uses the golangci-lint v2 layout (settings live under
# linters.settings, not top-level linters-settings) so that the
# thresholds below are actually applied by golangci-lint >= v2.
run: run:
timeout: 5m timeout: 5m
modules-download-mode: readonly modules-download-mode: readonly
@@ -14,8 +18,7 @@ linters:
- wsl # Deprecated, replaced by wsl_v5 - wsl # Deprecated, replaced by wsl_v5
- wrapcheck # Too verbose for internal packages - wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go - varnamelen # Short names like db, id are idiomatic Go
settings:
linters-settings:
lll: lll:
line-length: 88 line-length: 88
funlen: funlen:
@@ -27,6 +30,5 @@ linters-settings:
threshold: 100 threshold: 100
issues: issues:
exclude-use-default: false
max-issues-per-linter: 0 max-issues-per-linter: 0
max-same-issues: 0 max-same-issues: 0

View File

@@ -1,8 +1,8 @@
# Lint stage # Lint stage
# golangci/golangci-lint:v2.11.3 (Debian-based), 2026-03-17 # golangci/golangci-lint:v2.12.2 (Debian-based), 2026-08-07
# Using Debian-based image because mattn/go-sqlite3 (CGO) does not # Using Debian-based image because mattn/go-sqlite3 (CGO) does not
# compile on Alpine musl (off64_t is a glibc type). # compile on Alpine musl (off64_t is a glibc type).
FROM golangci/golangci-lint:v2.11.3@sha256:e838e8ab68aaefe83e2408691510867ade9329c0e0b895a3fb35eb93d1c2a4ba AS lint FROM golangci/golangci-lint:v2.12.2@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
RUN apt-get update && apt-get install -y --no-install-recommends make && rm -rf /var/lib/apt/lists/* RUN apt-get update && apt-get install -y --no-install-recommends make && rm -rf /var/lib/apt/lists/*

View File

@@ -363,10 +363,12 @@ events should be forwarded.
greater than 0, failed deliveries are retried with exponential backoff greater than 0, failed deliveries are retried with exponential backoff
up to `max_retries` attempts, protected by a per-target circuit up to `max_retries` attempts, protected by a per-target circuit
breaker. breaker.
- **`database`** — Confirm the event is stored in the webhook's - **`database`** — Archive the full event as a row into a separate
per-webhook database (no external delivery). Since events are always per-webhook archive database (`archive-{webhookID}.db`) for long-term
written to the per-webhook DB on ingestion, this target marks delivery retention, with an optional creation-validated expiry (default: keep
as immediately successful. Useful for ensuring durable event archival. forever). No external delivery and no retries; an archive write
failure fails the delivery. See the database target section under
"Per-Webhook Event Databases" for the full semantics.
- **`log`** — Write the event to the application log (stdout). Useful - **`log`** — Write the event to the application log (stdout). Useful
for debugging. for debugging.
@@ -512,11 +514,22 @@ This separation provides:
page cache, and its own lock, so concurrent event ingestion across page cache, and its own lock, so concurrent event ingestion across
webhooks won't contend. webhooks won't contend.
The **database target type** leverages this architecture: since events The **database target type** builds on this architecture to provide
are already stored in the per-webhook database by design, the database long-term archiving, separate from the per-webhook event database (which
target simply marks the delivery as immediately successful. The may prune events under its own retention). Delivering to a database
per-webhook DB IS the dedicated event database — that's the whole point target writes the full event — body, headers, method, content type, and
of the database target type. webhook/entrypoint/event identifiers — as a row into a dedicated archive
database, `archive-{webhookID}.db`, stored under the data directory
beside the event database. After each write the archive handle is closed
and reopened, debounced to at most once per second, so an operator can
move the archive file away for offline archiving without stopping the
service; a moved or removed archive file is recreated automatically on
the next write. An optional `expiry` in the target's config JSON (e.g.
`{"expiry":"720h"}`) is validated when the target is created — the
default (unset or the literal `never`) keeps rows forever — and rows
older than the expiry are pruned each time the archive is (re)opened. An
archive write failure is never silent success: the delivery records a
failed attempt with the error and is marked failed.
The **Slack target type** sends webhook events as formatted messages to The **Slack target type** sends webhook events as formatted messages to
any Slack-compatible incoming webhook URL (works with Slack, Mattermost, any Slack-compatible incoming webhook URL (works with Slack, Mattermost,

View File

@@ -28,6 +28,11 @@ databases currently grow without bound.
# Completed Steps # Completed Steps
- 2026-08-07 Update golangci-lint to v2.12.2 (Docker image digest in
`Dockerfile`, release-archive sha256 pins in `script/bootstrap`),
adopt the canonical `.golangci.yml` (v2 `linters.settings` layout so
`lll`/`funlen`/`cyclop`/`dupl` thresholds actually apply), and fix
all newly surfaced lint findings
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, - 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
Makefile shims, README Entrypoints section Makefile shims, README Entrypoints section
- 2026-03-25 pin golangci-lint Docker image for linting (#55) - 2026-03-25 pin golangci-lint Docker image for linting (#55)

View File

@@ -34,6 +34,7 @@ func main() {
config.New, config.New,
database.New, database.New,
database.NewWebhookDBManager, database.NewWebhookDBManager,
database.NewRetentionReaper,
healthcheck.New, healthcheck.New,
session.New, session.New,
handlers.New, handlers.New,
@@ -44,6 +45,13 @@ func main() {
func(e *delivery.Engine) delivery.Notifier { return e }, func(e *delivery.Engine) delivery.Notifier { return e },
server.New, server.New,
), ),
fx.Invoke(func(*server.Server, *delivery.Engine) {}), fx.Invoke(
func(
*server.Server,
*delivery.Engine,
*database.RetentionReaper,
) {
},
),
).Run() ).Run()
} }

View File

@@ -8,6 +8,7 @@ import (
"os" "os"
"strconv" "strconv"
"strings" "strings"
"time"
"go.uber.org/fx" "go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/globals" "sneak.berlin/go/webhooker/internal/globals"
@@ -26,6 +27,10 @@ const (
// defaultPort is the default HTTP listen port. // defaultPort is the default HTTP listen port.
defaultPort = 8080 defaultPort = 8080
// defaultRetentionSweepInterval is how often the retention
// reaper deletes events older than each webhook's RetentionDays.
defaultRetentionSweepInterval = time.Hour
) )
// ErrInvalidEnvironment is returned when WEBHOOKER_ENVIRONMENT // ErrInvalidEnvironment is returned when WEBHOOKER_ENVIRONMENT
@@ -51,6 +56,10 @@ type Config struct {
MetricsUsername string MetricsUsername string
Port int Port int
SentryDSN string SentryDSN string
// RetentionSweepInterval is how often the retention reaper runs.
RetentionSweepInterval time.Duration
params *ConfigParams params *ConfigParams
log *slog.Logger log *slog.Logger
} }
@@ -95,6 +104,30 @@ func envInt(key string, defaultValue int) int {
return defaultValue return defaultValue
} }
// envDuration returns the value of the named environment variable
// parsed as a Go duration (e.g. "1h", "30m"). Returns defaultValue if
// not set. If the variable is set but cannot be parsed, it returns a
// wrapped error naming the key and the bad value, so startup fails
// loudly rather than silently falling back to the default.
func envDuration(
key string,
defaultValue time.Duration,
) (time.Duration, error) {
v := os.Getenv(key)
if v == "" {
return defaultValue, nil
}
d, err := time.ParseDuration(v)
if err != nil {
return 0, fmt.Errorf(
"invalid duration for %s: %q: %w", key, v, err,
)
}
return d, nil
}
// New creates a Config by reading environment variables. // New creates a Config by reading environment variables.
// //
//nolint:revive // lc parameter is required by fx even if unused. //nolint:revive // lc parameter is required by fx even if unused.
@@ -118,6 +151,17 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
) )
} }
// Parse the retention sweep interval; a set-but-unparseable value
// is a hard error so fx aborts startup rather than silently using
// the default.
retentionSweepInterval, err := envDuration(
"RETENTION_SWEEP_INTERVAL",
defaultRetentionSweepInterval,
)
if err != nil {
return nil, err
}
// Load configuration values from environment variables // Load configuration values from environment variables
s := &Config{ s := &Config{
DataDir: envString("DATA_DIR"), DataDir: envString("DATA_DIR"),
@@ -128,6 +172,7 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
MetricsPassword: envString("METRICS_PASSWORD"), MetricsPassword: envString("METRICS_PASSWORD"),
Port: envInt("PORT", defaultPort), Port: envInt("PORT", defaultPort),
SentryDSN: envString("SENTRY_DSN"), SentryDSN: envString("SENTRY_DSN"),
RetentionSweepInterval: retentionSweepInterval,
log: log, log: log,
params: &params, params: &params,
} }
@@ -151,6 +196,7 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
"debug", s.Debug, "debug", s.Debug,
"maintenanceMode", s.MaintenanceMode, "maintenanceMode", s.MaintenanceMode,
"dataDir", s.DataDir, "dataDir", s.DataDir,
"retentionSweepInterval", s.RetentionSweepInterval.String(),
"hasSentryDSN", s.SentryDSN != "", "hasSentryDSN", s.SentryDSN != "",
"hasMetricsAuth", "hasMetricsAuth",
s.MetricsUsername != "" && s.MetricsPassword != "", s.MetricsUsername != "" && s.MetricsPassword != "",

View File

@@ -3,6 +3,7 @@ package config_test
import ( import (
"os" "os"
"testing" "testing"
"time"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
@@ -120,6 +121,100 @@ func testEnvironmentConfigSuccess(
assert.Equal(t, isProd, cfg.IsProd()) assert.Equal(t, isProd, cfg.IsProd())
} }
func TestRetentionSweepInterval(t *testing.T) {
tests := []struct {
name string
set bool
value string
expectError bool
expected time.Duration
}{
{
name: "unset uses default",
set: false,
expected: time.Hour,
},
{
name: "valid value is parsed",
set: true,
value: "15m",
expected: 15 * time.Minute,
},
{
name: "unparseable value fails startup",
set: true,
value: "not-a-duration",
expectError: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
if tt.set {
t.Setenv("RETENTION_SWEEP_INTERVAL", tt.value)
} else {
require.NoError(t, os.Unsetenv(
"RETENTION_SWEEP_INTERVAL",
))
}
if tt.expectError {
testRetentionSweepIntervalError(t)
} else {
testRetentionSweepIntervalSuccess(t, tt.expected)
}
})
}
}
func testRetentionSweepIntervalError(t *testing.T) {
t.Helper()
var cfg *config.Config
app := fx.New(
fx.NopLogger,
fx.Provide(
globals.New,
logger.New,
config.New,
),
fx.Populate(&cfg),
)
assert.Error(t, app.Err())
}
func testRetentionSweepIntervalSuccess(
t *testing.T,
expected time.Duration,
) {
t.Helper()
var cfg *config.Config
app := fxtest.New(
t,
fx.Provide(
globals.New,
logger.New,
config.New,
),
fx.Populate(&cfg),
)
require.NoError(t, app.Err())
app.RequireStart()
defer app.RequireStop()
assert.Equal(t, expected, cfg.RetentionSweepInterval)
}
func TestDefaultDataDir(t *testing.T) { func TestDefaultDataDir(t *testing.T) {
for _, env := range []string{"", "dev", "prod"} { for _, env := range []string{"", "dev", "prod"} {
name := env name := env

View File

@@ -11,6 +11,15 @@ import (
"sneak.berlin/go/webhooker/internal/logger" "sneak.berlin/go/webhooker/internal/logger"
) )
const (
// testAppname is the Globals.Appname used in tests.
testAppname = "webhooker-test"
// testVersion is the Globals.Version used in tests.
testVersion = "test"
// testContentType is the event content type used in tests.
testContentType = "application/json"
)
func setupTestDB( func setupTestDB(
t *testing.T, t *testing.T,
) (*database.Database, *fxtest.Lifecycle) { ) (*database.Database, *fxtest.Lifecycle) {
@@ -19,8 +28,8 @@ func setupTestDB(
lc := fxtest.NewLifecycle(t) lc := fxtest.NewLifecycle(t)
g := &globals.Globals{ g := &globals.Globals{
Appname: "webhooker-test", Appname: testAppname,
Version: "test", Version: testVersion,
} }
l, err := logger.New( l, err := logger.New(

View File

@@ -0,0 +1,31 @@
package database
import (
"context"
"log/slog"
"os"
"time"
)
// NewTestRetentionReaper builds a RetentionReaper backed by the given
// main database and per-webhook database manager, without the fx
// lifecycle. Intended for tests.
func NewTestRetentionReaper(
db *Database,
mgr *WebhookDBManager,
) *RetentionReaper {
return &RetentionReaper{
db: db,
dbManager: mgr,
log: slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
interval: time.Hour,
}
}
// ExportSweep runs a single retention sweep synchronously for tests.
func (r *RetentionReaper) ExportSweep(ctx context.Context) {
r.sweep(ctx)
}

View File

@@ -5,7 +5,10 @@ type Entrypoint struct {
BaseModel BaseModel
WebhookID string `gorm:"type:uuid;not null" json:"webhookId"` WebhookID string `gorm:"type:uuid;not null" json:"webhookId"`
Path string `gorm:"uniqueIndex;not null" json:"path"` // URL path for this entrypoint
// Path is the URL path for this entrypoint.
Path string `gorm:"uniqueIndex;not null" json:"path"`
Description string `json:"description"` Description string `json:"description"`
Active bool `gorm:"default:true" json:"active"` Active bool `gorm:"default:true" json:"active"`

View File

@@ -23,7 +23,8 @@ type Target struct {
// Configuration fields (JSON stored based on type) // Configuration fields (JSON stored based on type)
Config string `gorm:"type:text" json:"config"` // JSON configuration Config string `gorm:"type:text" json:"config"` // JSON configuration
// For HTTP targets (max_retries=0 means fire-and-forget, >0 enables retries with backoff) // For HTTP targets (max_retries=0 means fire-and-forget,
// >0 enables retries with backoff)
MaxRetries int `json:"maxRetries,omitempty"` MaxRetries int `json:"maxRetries,omitempty"`
MaxQueueSize int `json:"maxQueueSize,omitempty"` MaxQueueSize int `json:"maxQueueSize,omitempty"`

View File

@@ -7,7 +7,9 @@ type Webhook struct {
UserID string `gorm:"type:uuid;not null" json:"userId"` UserID string `gorm:"type:uuid;not null" json:"userId"`
Name string `gorm:"not null" json:"name"` Name string `gorm:"not null" json:"name"`
Description string `json:"description"` Description string `json:"description"`
RetentionDays int `gorm:"default:30" json:"retentionDays"` // Days to retain events
// RetentionDays is the number of days to retain events.
RetentionDays int `gorm:"default:30" json:"retentionDays"`
// Relations // Relations
User User `json:"user,omitzero"` User User `json:"user,omitzero"`

View File

@@ -0,0 +1,252 @@
package database
import (
"context"
"fmt"
"log/slog"
"sync"
"time"
"go.uber.org/fx"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/logger"
)
// hoursPerDay converts a RetentionDays count into hours for cutoff
// computation.
const hoursPerDay = 24
// RetentionReaperParams holds the fx dependencies for the
// RetentionReaper.
type RetentionReaperParams struct {
fx.In
Config *config.Config
Database *Database
DBManager *WebhookDBManager
Logger *logger.Logger
}
// RetentionReaper periodically deletes expired events (and their
// dependent deliveries and delivery results) from each per-webhook
// database, enforcing every webhook's RetentionDays. Rows are removed
// permanently so that per-webhook SQLite files do not grow without
// bound.
type RetentionReaper struct {
db *Database
dbManager *WebhookDBManager
log *slog.Logger
interval time.Duration
cancel context.CancelFunc
wg sync.WaitGroup
}
// NewRetentionReaper creates the retention reaper and registers its
// fx lifecycle hooks. The background sweep loop starts on OnStart and
// stops cleanly on OnStop via context cancellation.
func NewRetentionReaper(
lc fx.Lifecycle,
params RetentionReaperParams,
) *RetentionReaper {
r := &RetentionReaper{
db: params.Database,
dbManager: params.DBManager,
log: params.Logger.Get(),
interval: params.Config.RetentionSweepInterval,
}
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
r.start(ctx)
return nil
},
OnStop: func(_ context.Context) error {
r.stop()
return nil
},
})
return r
}
func (r *RetentionReaper) start(ctx context.Context) {
ctx, cancel := context.WithCancel(ctx)
r.cancel = cancel
r.wg.Add(1)
go r.run(ctx)
r.log.Info(
"retention reaper started",
"interval", r.interval.String(),
)
}
func (r *RetentionReaper) stop() {
r.log.Info("retention reaper stopping")
if r.cancel != nil {
r.cancel()
}
r.wg.Wait()
r.log.Info("retention reaper stopped")
}
func (r *RetentionReaper) run(ctx context.Context) {
defer r.wg.Done()
ticker := time.NewTicker(r.interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
r.sweep(ctx)
}
}
}
// sweep lists every webhook from the main database and reaps expired
// rows from each per-webhook database whose RetentionDays is positive.
func (r *RetentionReaper) sweep(ctx context.Context) {
var webhooks []Webhook
err := r.db.DB().
Model(&Webhook{}).
Find(&webhooks).Error
if err != nil {
r.log.Error(
"retention sweep: failed to list webhooks",
"error", err,
)
return
}
for i := range webhooks {
select {
case <-ctx.Done():
return
default:
}
wh := webhooks[i]
// RetentionDays of zero or less means retain forever.
if wh.RetentionDays <= 0 {
continue
}
// Nothing to reap if the per-webhook database has never
// been created.
if !r.dbManager.DBExists(wh.ID) {
continue
}
r.reapWebhook(wh.ID, wh.RetentionDays)
}
}
// reapWebhook removes every expired event (and its dependents) from a
// single webhook's database.
func (r *RetentionReaper) reapWebhook(
webhookID string,
retentionDays int,
) {
db, err := r.dbManager.GetDB(webhookID)
if err != nil {
r.log.Error(
"retention sweep: failed to open webhook database",
"webhook_id", webhookID,
"error", err,
)
return
}
cutoff := time.Now().Add(
-time.Duration(retentionDays*hoursPerDay) * time.Hour,
)
deleted, err := reapExpired(db, cutoff)
if err != nil {
r.log.Error(
"retention sweep: failed to reap expired events",
"webhook_id", webhookID,
"error", err,
)
return
}
if deleted > 0 {
r.log.Info(
"retention sweep: reaped expired events",
"webhook_id", webhookID,
"retention_days", retentionDays,
"events_deleted", deleted,
)
}
}
// reapExpired hard-deletes, in foreign-key-safe order, the delivery
// results, deliveries, and events associated with events older than
// cutoff. Deletes are unscoped so rows are physically removed rather
// than soft-deleted, reclaiming disk. It returns the number of events
// deleted.
func reapExpired(db *gorm.DB, cutoff time.Time) (int64, error) {
// Fresh subqueries are built per statement to avoid reusing a
// mutated builder across executions.
expiredEventIDs := func() *gorm.DB {
return db.Model(&Event{}).
Select("id").
Where("created_at < ?", cutoff)
}
expiredDeliveryIDs := func() *gorm.DB {
return db.Model(&Delivery{}).
Select("id").
Where("event_id IN (?)", expiredEventIDs())
}
// 1. Delivery results whose delivery belongs to an expired event.
res := db.Unscoped().
Where("delivery_id IN (?)", expiredDeliveryIDs()).
Delete(&DeliveryResult{})
if res.Error != nil {
return 0, fmt.Errorf(
"deleting expired delivery results: %w",
res.Error,
)
}
// 2. Deliveries belonging to an expired event.
del := db.Unscoped().
Where("event_id IN (?)", expiredEventIDs()).
Delete(&Delivery{})
if del.Error != nil {
return 0, fmt.Errorf(
"deleting expired deliveries: %w",
del.Error,
)
}
// 3. The expired events themselves.
ev := db.Unscoped().
Where("created_at < ?", cutoff).
Delete(&Event{})
if ev.Error != nil {
return 0, fmt.Errorf(
"deleting expired events: %w",
ev.Error,
)
}
return ev.RowsAffected, nil
}

View File

@@ -0,0 +1,278 @@
package database_test
import (
"context"
"net/http"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/fx/fxtest"
"gorm.io/gorm"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/globals"
"sneak.berlin/go/webhooker/internal/logger"
)
// retentionTestEnv bundles the pieces a retention test drives.
type retentionTestEnv struct {
reaper *database.RetentionReaper
mainDB *database.Database
mgr *database.WebhookDBManager
}
func setupRetentionTest(t *testing.T) *retentionTestEnv {
t.Helper()
lc := fxtest.NewLifecycle(t)
g := &globals.Globals{
Appname: testAppname,
Version: testVersion,
}
l, err := logger.New(lc, logger.LoggerParams{Globals: g})
require.NoError(t, err)
cfg := &config.Config{
DataDir: t.TempDir(),
Environment: "dev",
}
mainDB, err := database.New(lc, database.DatabaseParams{
Config: cfg,
Logger: l,
})
require.NoError(t, err)
mgr, err := database.NewWebhookDBManager(
lc,
database.WebhookDBManagerParams{Config: cfg, Logger: l},
)
require.NoError(t, err)
ctx := context.Background()
require.NoError(t, lc.Start(ctx))
t.Cleanup(func() { require.NoError(t, lc.Stop(ctx)) })
return &retentionTestEnv{
reaper: database.NewTestRetentionReaper(mainDB, mgr),
mainDB: mainDB,
mgr: mgr,
}
}
// createWebhook inserts a webhook row into the main database with the
// given retention policy and returns its ID.
func createWebhook(
t *testing.T,
db *gorm.DB,
retentionDays int,
) string {
t.Helper()
wh := &database.Webhook{
UserID: uuid.New().String(),
Name: "test-webhook",
RetentionDays: retentionDays,
}
require.NoError(
t,
db.Omit(clause.Associations).Create(wh).Error,
)
// The RetentionDays column carries a GORM default of 30, so a
// zero (or negative) value passed to Create is replaced by that
// default. Force the requested value explicitly so the
// retain-forever (<= 0) path can be exercised.
require.NoError(
t,
db.Model(wh).
Update("retention_days", retentionDays).Error,
)
return wh.ID
}
// eventChain is the set of row IDs seeded for a single event.
type eventChain struct {
eventID string
deliveryID string
resultID string
}
// seedEventChain creates an event with one delivery and one delivery
// result, all stamped with createdAt, and returns their IDs.
func seedEventChain(
t *testing.T,
db *gorm.DB,
webhookID string,
createdAt time.Time,
) eventChain {
t.Helper()
event := &database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Body: `{"seed": true}`,
ContentType: testContentType,
}
event.CreatedAt = createdAt
require.NoError(t, db.Create(event).Error)
delivery := &database.Delivery{
EventID: event.ID,
TargetID: uuid.New().String(),
Status: database.DeliveryStatusDelivered,
}
delivery.CreatedAt = createdAt
require.NoError(t, db.Create(delivery).Error)
result := &database.DeliveryResult{
DeliveryID: delivery.ID,
AttemptNum: 1,
Success: true,
StatusCode: 200,
Duration: 10,
}
result.CreatedAt = createdAt
require.NoError(t, db.Create(result).Error)
return eventChain{
eventID: event.ID,
deliveryID: delivery.ID,
resultID: result.ID,
}
}
// countByID returns how many rows of model match the given id,
// counting even hard-deletable rows via Unscoped.
func countByID(
t *testing.T,
db *gorm.DB,
model any,
id string,
) int64 {
t.Helper()
var n int64
require.NoError(
t,
db.Unscoped().Model(model).
Where("id = ?", id).Count(&n).Error,
)
return n
}
func assertChainGone(
t *testing.T,
db *gorm.DB,
chain eventChain,
) {
t.Helper()
assert.Zero(
t,
countByID(t, db, &database.Event{}, chain.eventID),
"expired event should be removed",
)
assert.Zero(
t,
countByID(t, db, &database.Delivery{}, chain.deliveryID),
"expired delivery should be removed",
)
assert.Zero(
t,
countByID(
t, db, &database.DeliveryResult{}, chain.resultID,
),
"expired delivery result should be removed",
)
}
func assertChainPresent(
t *testing.T,
db *gorm.DB,
chain eventChain,
) {
t.Helper()
assert.Equal(
t,
int64(1),
countByID(t, db, &database.Event{}, chain.eventID),
"recent event should be retained",
)
assert.Equal(
t,
int64(1),
countByID(t, db, &database.Delivery{}, chain.deliveryID),
"recent delivery should be retained",
)
assert.Equal(
t,
int64(1),
countByID(
t, db, &database.DeliveryResult{}, chain.resultID,
),
"recent delivery result should be retained",
)
}
func TestRetentionReaper_ReapsExpiredKeepsRecent(t *testing.T) {
t.Parallel()
env := setupRetentionTest(t)
const retentionDays = 30
webhookID := createWebhook(
t, env.mainDB.DB(), retentionDays,
)
db, err := env.mgr.GetDB(webhookID)
require.NoError(t, err)
now := time.Now()
old := seedEventChain(
t, db, webhookID,
now.Add(-40*24*time.Hour),
)
recent := seedEventChain(
t, db, webhookID,
now.Add(-1*24*time.Hour),
)
env.reaper.ExportSweep(context.Background())
assertChainGone(t, db, old)
assertChainPresent(t, db, recent)
}
func TestRetentionReaper_RetainsForeverWhenNonPositive(t *testing.T) {
t.Parallel()
env := setupRetentionTest(t)
// RetentionDays of zero means retain forever.
webhookID := createWebhook(t, env.mainDB.DB(), 0)
db, err := env.mgr.GetDB(webhookID)
require.NoError(t, err)
ancient := seedEventChain(
t, db, webhookID,
time.Now().Add(-365*24*time.Hour),
)
env.reaper.ExportSweep(context.Background())
assertChainPresent(t, db, ancient)
}

View File

@@ -14,7 +14,10 @@ import (
func NewTestDatabase(db *gorm.DB) *Database { func NewTestDatabase(db *gorm.DB) *Database {
return &Database{ return &Database{
db: db, db: db,
log: slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelDebug})), log: slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
} }
} }
@@ -23,6 +26,9 @@ func NewTestDatabase(db *gorm.DB) *Database {
func NewTestWebhookDBManager(dataDir string) *WebhookDBManager { func NewTestWebhookDBManager(dataDir string) *WebhookDBManager {
return &WebhookDBManager{ return &WebhookDBManager{
dataDir: dataDir, dataDir: dataDir,
log: slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelDebug})), log: slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
} }
} }

View File

@@ -2,6 +2,7 @@ package database_test
import ( import (
"context" "context"
"net/http"
"os" "os"
"path/filepath" "path/filepath"
"testing" "testing"
@@ -25,8 +26,8 @@ func setupTestWebhookDBManager(
lc := fxtest.NewLifecycle(t) lc := fxtest.NewLifecycle(t)
g := &globals.Globals{ g := &globals.Globals{
Appname: "webhooker-test", Appname: testAppname,
Version: "test", Version: testVersion,
} }
l, err := logger.New( l, err := logger.New(
@@ -83,10 +84,10 @@ func TestWebhookDBManager_CreateAndGetDB(t *testing.T) {
event := &database.Event{ event := &database.Event{
WebhookID: webhookID, WebhookID: webhookID,
EntrypointID: uuid.New().String(), EntrypointID: uuid.New().String(),
Method: "POST", Method: http.MethodPost,
Headers: `{"Content-Type":["application/json"]}`, Headers: `{"Content-Type":["application/json"]}`,
Body: `{"test": true}`, Body: `{"test": true}`,
ContentType: "application/json", ContentType: testContentType,
} }
require.NoError(t, db.Create(event).Error) require.NoError(t, db.Create(event).Error)
assert.NotEmpty(t, event.ID) assert.NotEmpty(t, event.ID)
@@ -99,7 +100,7 @@ func TestWebhookDBManager_CreateAndGetDB(t *testing.T) {
db.First(&readEvent, "id = ?", event.ID).Error, db.First(&readEvent, "id = ?", event.ID).Error,
) )
assert.Equal(t, webhookID, readEvent.WebhookID) assert.Equal(t, webhookID, readEvent.WebhookID)
assert.Equal(t, "POST", readEvent.Method) assert.Equal(t, http.MethodPost, readEvent.Method)
assert.Equal(t, `{"test": true}`, readEvent.Body) assert.Equal(t, `{"test": true}`, readEvent.Body)
} }
@@ -123,9 +124,9 @@ func TestWebhookDBManager_DeleteDB(t *testing.T) {
event := &database.Event{ event := &database.Event{
WebhookID: webhookID, WebhookID: webhookID,
EntrypointID: uuid.New().String(), EntrypointID: uuid.New().String(),
Method: "POST", Method: http.MethodPost,
Body: `{"test": true}`, Body: `{"test": true}`,
ContentType: "application/json", ContentType: testContentType,
} }
require.NoError(t, db.Create(event).Error) require.NoError(t, db.Create(event).Error)
@@ -196,10 +197,10 @@ func seedDeliveryWorkflow(
event := &database.Event{ event := &database.Event{
WebhookID: webhookID, WebhookID: webhookID,
EntrypointID: uuid.New().String(), EntrypointID: uuid.New().String(),
Method: "POST", Method: http.MethodPost,
Headers: `{"Content-Type":["application/json"]}`, Headers: `{"Content-Type":["application/json"]}`,
Body: `{"payload": "test"}`, Body: `{"payload": "test"}`,
ContentType: "application/json", ContentType: testContentType,
} }
require.NoError(t, db.Create(event).Error) require.NoError(t, db.Create(event).Error)
@@ -231,7 +232,7 @@ func verifyPendingDeliveries(
) )
require.Len(t, pending, 1) require.Len(t, pending, 1)
assert.Equal(t, event.ID, pending[0].EventID) assert.Equal(t, event.ID, pending[0].EventID)
assert.Equal(t, "POST", pending[0].Event.Method) assert.Equal(t, http.MethodPost, pending[0].Event.Method)
} }
func completeDelivery( func completeDelivery(
@@ -303,16 +304,16 @@ func TestWebhookDBManager_MultipleWebhooks(t *testing.T) {
event1 := &database.Event{ event1 := &database.Event{
WebhookID: webhook1, WebhookID: webhook1,
EntrypointID: uuid.New().String(), EntrypointID: uuid.New().String(),
Method: "POST", Method: http.MethodPost,
Body: `{"webhook": 1}`, Body: `{"webhook": 1}`,
ContentType: "application/json", ContentType: testContentType,
} }
event2 := &database.Event{ event2 := &database.Event{
WebhookID: webhook2, WebhookID: webhook2,
EntrypointID: uuid.New().String(), EntrypointID: uuid.New().String(),
Method: "PUT", Method: http.MethodPut,
Body: `{"webhook": 2}`, Body: `{"webhook": 2}`,
ContentType: "application/json", ContentType: testContentType,
} }
require.NoError(t, db1.Create(event1).Error) require.NoError(t, db1.Create(event1).Error)

File diff suppressed because it is too large Load Diff

View File

@@ -126,36 +126,6 @@ func iHTTPConfig(url string) string {
return string(data) return string(data)
} }
func iWebhookDB(t *testing.T) *gorm.DB {
t.Helper()
dbPath := filepath.Join(
t.TempDir(), "events-test.db",
)
dsn := fmt.Sprintf(
"file:%s?cache=shared&mode=rwc", dbPath,
)
sqlDB, err := sql.Open("sqlite", dsn)
require.NoError(t, err)
t.Cleanup(func() { _ = sqlDB.Close() })
db, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
require.NoError(t, err)
require.NoError(t, db.AutoMigrate(
&database.Event{},
&database.Delivery{},
&database.DeliveryResult{},
))
return db
}
func iEngine( func iEngine(
t *testing.T, workers int, t *testing.T, workers int,
) *delivery.Engine { ) *delivery.Engine {
@@ -182,10 +152,10 @@ func iSeedEvent(
event := database.Event{ event := database.Event{
WebhookID: webhookID, WebhookID: webhookID,
EntrypointID: uuid.New().String(), EntrypointID: uuid.New().String(),
Method: "POST", Method: http.MethodPost,
Headers: `{}`, Headers: `{}`,
Body: body, Body: body,
ContentType: "application/json", ContentType: testContentType,
} }
require.NoError(t, db.Create(&event).Error) require.NoError(t, db.Create(&event).Error)
@@ -935,7 +905,7 @@ func TestDeliverHTTP_CustomTargetHeaders(t *testing.T) {
func TestDeliverHTTP_TargetTimeout(t *testing.T) { func TestDeliverHTTP_TargetTimeout(t *testing.T) {
t.Parallel() t.Parallel()
db := iWebhookDB(t) db := testWebhookDB(t)
e := iEngine(t, 1) e := iEngine(t, 1)
ts := httptest.NewServer( ts := httptest.NewServer(
@@ -987,10 +957,10 @@ func iSeedEventAndDelivery(
event := database.Event{ event := database.Event{
WebhookID: uuid.New().String(), WebhookID: uuid.New().String(),
EntrypointID: uuid.New().String(), EntrypointID: uuid.New().String(),
Method: "POST", Method: http.MethodPost,
Headers: `{"Content-Type":["application/json"]}`, Headers: `{"Content-Type":["application/json"]}`,
Body: body, Body: body,
ContentType: "application/json", ContentType: testContentType,
} }
require.NoError(t, db.Create(&event).Error) require.NoError(t, db.Create(&event).Error)
@@ -1067,7 +1037,7 @@ func iAssertResultFailed(
func TestDeliverHTTP_InvalidConfig(t *testing.T) { func TestDeliverHTTP_InvalidConfig(t *testing.T) {
t.Parallel() t.Parallel()
db := iWebhookDB(t) db := testWebhookDB(t)
e := iEngine(t, 1) e := iEngine(t, 1)
event, del := iSeedEventAndDelivery( event, del := iSeedEventAndDelivery(

View File

@@ -27,6 +27,9 @@ import (
"sneak.berlin/go/webhooker/internal/delivery" "sneak.berlin/go/webhooker/internal/delivery"
) )
// testContentType is the event content type used in tests.
const testContentType = "application/json"
func testWebhookDB(t *testing.T) *gorm.DB { func testWebhookDB(t *testing.T) *gorm.DB {
t.Helper() t.Helper()
@@ -94,10 +97,10 @@ func seedEvent(
event := database.Event{ event := database.Event{
WebhookID: uuid.New().String(), WebhookID: uuid.New().String(),
EntrypointID: uuid.New().String(), EntrypointID: uuid.New().String(),
Method: "POST", Method: http.MethodPost,
Headers: `{"Content-Type":["application/json"]}`, Headers: `{"Content-Type":["application/json"]}`,
Body: body, Body: body,
ContentType: "application/json", ContentType: testContentType,
} }
require.NoError(t, db.Create(&event).Error) require.NoError(t, db.Create(&event).Error)
@@ -342,33 +345,29 @@ func TestDeliverDatabase_ImmediateSuccess(
t.Parallel() t.Parallel()
db := testWebhookDB(t) db := testWebhookDB(t)
e := testEngine(t, 1)
event := seedEvent(t, db, `{"db":"target"}`) // The database target archives for real now, so the engine
// needs a webhook DB manager to locate the data directory.
dlv := seedDelivery( e := delivery.NewTestEngineWithDB(
t, db, event.ID, uuid.New().String(), nil,
database.DeliveryStatusPending, database.NewTestWebhookDBManager(t.TempDir()),
slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
&http.Client{Timeout: 5 * time.Second},
1,
) )
d := &database.Delivery{ event := seedEvent(t, db, `{"db":"target"}`)
EventID: event.ID, d := seedDatabaseTargetDelivery(t, db, event, "")
TargetID: dlv.TargetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: database.Target{
Name: "test-db",
Type: database.TargetTypeDatabase,
},
}
d.ID = dlv.ID
e.ExportDeliverDatabase(db, d) e.ExportDeliverDatabase(db, d)
var updated database.Delivery var updated database.Delivery
require.NoError(t, db.First( require.NoError(t, db.First(
&updated, "id = ?", dlv.ID, &updated, "id = ?", d.ID,
).Error) ).Error)
assert.Equal(t, assert.Equal(t,
@@ -379,7 +378,7 @@ func TestDeliverDatabase_ImmediateSuccess(
var result database.DeliveryResult var result database.DeliveryResult
require.NoError(t, db.Where( require.NoError(t, db.Where(
"delivery_id = ?", dlv.ID, "delivery_id = ?", d.ID,
).First(&result).Error) ).First(&result).Error)
assert.True(t, result.Success) assert.True(t, result.Success)
@@ -436,91 +435,6 @@ func TestDeliverLog_ImmediateSuccess(t *testing.T) {
assert.True(t, result.Success) assert.True(t, result.Success)
} }
func TestDeliverLog_StructuredLogFields(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
var logBuf bytes.Buffer
e := delivery.NewTestEngine(
slog.New(slog.NewTextHandler(
&logBuf,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
&http.Client{Timeout: 5 * time.Second},
1,
)
event := seedEvent(t, db, `{"log":"structured"}`)
dlv := seedDelivery(
t, db, event.ID, uuid.New().String(),
database.DeliveryStatusPending,
)
d := &database.Delivery{
EventID: event.ID,
TargetID: dlv.TargetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: database.Target{
Name: "structured-log",
Type: database.TargetTypeLog,
},
}
d.ID = dlv.ID
e.ExportDeliverLog(db, d)
// The delivery is marked delivered and a success
// DeliveryResult with no HTTP status is recorded,
// mirroring the other target types' bookkeeping.
var updated database.Delivery
require.NoError(t, db.First(
&updated, "id = ?", dlv.ID,
).Error)
assert.Equal(t,
database.DeliveryStatusDelivered, updated.Status,
"log target should immediately succeed",
)
var result database.DeliveryResult
require.NoError(t, db.Where(
"delivery_id = ?", dlv.ID,
).First(&result).Error)
assert.True(t, result.Success)
assert.Equal(t, 0, result.StatusCode,
"log target should not have an HTTP status",
)
assertLogFields(t, logBuf.String(), event, "structured-log")
}
// assertLogFields checks that a log target's structured
// log line carries the required fields: event id,
// webhook/entrypoint, target name, and outcome.
func assertLogFields(
t *testing.T,
logged string,
event database.Event,
targetName string,
) {
t.Helper()
assert.Contains(t, logged, "event_id="+event.ID)
assert.Contains(t, logged, "webhook_id="+event.WebhookID)
assert.Contains(t,
logged, "entrypoint_id="+event.EntrypointID,
)
assert.Contains(t, logged, "target_name="+targetName)
assert.Contains(t, logged, "outcome=delivered")
}
func TestDeliverHTTP_WithRetries_Success(t *testing.T) { func TestDeliverHTTP_WithRetries_Success(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1202,10 +1116,10 @@ func TestDoHTTPRequest_ForwardsHeaders(t *testing.T) {
} }
event := &database.Event{ event := &database.Event{
Method: "POST", Method: http.MethodPost,
Headers: `{"X-Custom":["value1"],"Content-Type":["application/json"]}`, Headers: `{"X-Custom":["value1"],"Content-Type":["application/json"]}`,
Body: `{"test":true}`, Body: `{"test":true}`,
ContentType: "application/json", ContentType: testContentType,
} }
statusCode, _, _, err := e.ExportDoHTTPRequest( statusCode, _, _, err := e.ExportDoHTTPRequest(
@@ -1227,7 +1141,7 @@ func TestDoHTTPRequest_ForwardsHeaders(t *testing.T) {
) )
assert.Equal(t, assert.Equal(t,
"application/json", testContentType,
receivedHeaders.Get("Content-Type"), receivedHeaders.Get("Content-Type"),
) )
@@ -1243,7 +1157,19 @@ func TestProcessDelivery_RoutesToCorrectHandler(
t.Parallel() t.Parallel()
db := testWebhookDB(t) db := testWebhookDB(t)
e := testEngine(t, 1)
// The database target archives for real now, so the engine
// needs a webhook DB manager to locate the data directory.
e := delivery.NewTestEngineWithDB(
nil,
database.NewTestWebhookDBManager(t.TempDir()),
slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
&http.Client{Timeout: 5 * time.Second},
1,
)
tests := []struct { tests := []struct {
name string name string
@@ -1374,8 +1300,8 @@ func TestFormatSlackMessage_JSONBody(t *testing.T) {
t.Parallel() t.Parallel()
event := &database.Event{ event := &database.Event{
Method: "POST", Method: http.MethodPost,
ContentType: "application/json", ContentType: testContentType,
Body: `{"action":"push",` + Body: `{"action":"push",` +
`"repo":"test/repo",` + `"repo":"test/repo",` +
`"ref":"refs/heads/main"}`, `"ref":"refs/heads/main"}`,
@@ -1400,7 +1326,7 @@ func TestFormatSlackMessage_NonJSONBody(t *testing.T) {
t.Parallel() t.Parallel()
event := &database.Event{ event := &database.Event{
Method: "POST", Method: http.MethodPost,
ContentType: "text/plain", ContentType: "text/plain",
Body: "hello world plain text", Body: "hello world plain text",
} }
@@ -1423,8 +1349,8 @@ func TestFormatSlackMessage_EmptyBody(t *testing.T) {
t.Parallel() t.Parallel()
event := &database.Event{ event := &database.Event{
Method: "POST", Method: http.MethodPost,
ContentType: "application/json", ContentType: testContentType,
Body: "", Body: "",
} }
event.CreatedAt = time.Date( event.CreatedAt = time.Date(
@@ -1452,8 +1378,8 @@ func TestFormatSlackMessage_LargeJSONTruncated(
require.NoError(t, err) require.NoError(t, err)
event := &database.Event{ event := &database.Event{
Method: "POST", Method: http.MethodPost,
ContentType: "application/json", ContentType: testContentType,
Body: string(largeJSON), Body: string(largeJSON),
} }
event.CreatedAt = time.Date( event.CreatedAt = time.Date(
@@ -1738,6 +1664,179 @@ func TestProcessDelivery_RoutesToSlack(t *testing.T) {
) )
} }
// newLogCaptureEngine builds a test engine whose logger
// writes to the returned buffer, for inspecting log output.
func newLogCaptureEngine(
t *testing.T,
) (*delivery.Engine, *bytes.Buffer) {
t.Helper()
var buf bytes.Buffer
log := slog.New(slog.NewTextHandler(
&buf,
&slog.HandlerOptions{Level: slog.LevelDebug},
))
e := delivery.NewTestEngine(
log, &http.Client{Timeout: 5 * time.Second}, 1,
)
return e, &buf
}
// assertLogLineComplete asserts the captured log output
// carries the full inbound webhook content and ids.
func assertLogLineComplete(
t *testing.T, out string, event database.Event,
) {
t.Helper()
assert.Contains(t, out, "log-body-marker",
"log line must contain the full request body",
)
assert.Contains(t, out, "Content-Type",
"log line must contain the full request headers",
)
assert.Contains(t, out, event.EntrypointID,
"log line must contain the entrypoint id",
)
assert.Contains(t, out, event.WebhookID,
"log line must contain the webhook id",
)
assert.Contains(t, out, testContentType,
"log line must contain the content type",
)
}
func TestDeliverLog_LogsFullContent(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
e, buf := newLogCaptureEngine(t)
event := seedEvent(
t, db, `{"log-body-marker":"abc123"}`,
)
dlv := seedDelivery(
t, db, event.ID, uuid.New().String(),
database.DeliveryStatusPending,
)
d := &database.Delivery{
EventID: event.ID,
TargetID: dlv.TargetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: database.Target{
Name: "test-log-full",
Type: database.TargetTypeLog,
},
}
d.ID = dlv.ID
e.ExportDeliverLog(db, d)
assertLogLineComplete(t, buf.String(), event)
assertDeliveryStatus(t, db, dlv.ID,
database.DeliveryStatusDelivered,
)
}
// buildSlackRetryDelivery builds a Slack delivery whose
// target is configured with retries enabled.
func buildSlackRetryDelivery(
dlv database.Delivery,
event database.Event,
targetID, cfg string,
) *database.Delivery {
d := &database.Delivery{
EventID: event.ID,
TargetID: targetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: database.Target{
Name: "test-slack-retry",
Type: database.TargetTypeSlack,
Config: cfg,
MaxRetries: 5,
},
}
d.ID = dlv.ID
return d
}
func TestDeliverSlack_WithRetries_SchedulesRetry(
t *testing.T,
) {
t.Parallel()
db := testWebhookDB(t)
ts := newStatusServer(t, http.StatusServiceUnavailable)
e := testEngine(t, 1)
targetID := uuid.New().String()
slackCfg, err := json.Marshal(
delivery.SlackTargetConfig{WebhookURL: ts.URL},
)
require.NoError(t, err)
event := seedEvent(t, db, `{"slack":"retry"}`)
dlv := seedDelivery(
t, db, event.ID, targetID,
database.DeliveryStatusPending,
)
d := buildSlackRetryDelivery(
dlv, event, targetID, string(slackCfg),
)
task := &delivery.Task{
DeliveryID: dlv.ID,
TargetID: targetID,
TargetType: database.TargetTypeSlack,
MaxRetries: 5,
AttemptNum: 1,
}
e.ExportProcessDelivery(context.TODO(), db, d, task)
assertDeliveryStatus(t, db, dlv.ID,
database.DeliveryStatusRetrying,
)
assertDeliveryResult(
t, db, dlv.ID, false,
http.StatusServiceUnavailable,
)
}
// newStatusServer starts a test server that always responds
// with the given status code.
func newStatusServer(
t *testing.T, code int,
) *httptest.Server {
t.Helper()
ts := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(code)
},
))
t.Cleanup(ts.Close)
return ts
}
// readAll is a small helper to avoid importing io in // readAll is a small helper to avoid importing io in
// a test handler inline. // a test handler inline.
func readAll(r interface { func readAll(r interface {

View File

@@ -39,37 +39,50 @@ func ExportTruncate(s string, maxLen int) string {
return truncate(s, maxLen) return truncate(s, maxLen)
} }
// ExportDeliverHTTP exposes deliverHTTP for testing. // ExportDeliverHTTP delivers via the http target for testing.
func (e *Engine) ExportDeliverHTTP( func (e *Engine) ExportDeliverHTTP(
ctx context.Context, ctx context.Context,
webhookDB *gorm.DB, webhookDB *gorm.DB,
d *database.Delivery, d *database.Delivery,
task *Task, task *Task,
) { ) {
e.deliverHTTP(ctx, webhookDB, d, task) e.httpTarget.Deliver(ctx, webhookDB, d, task, e)
} }
// ExportDeliverDatabase exposes deliverDatabase. // ExportDeliverDatabase delivers via the database target.
func (e *Engine) ExportDeliverDatabase( func (e *Engine) ExportDeliverDatabase(
webhookDB *gorm.DB, d *database.Delivery, webhookDB *gorm.DB, d *database.Delivery,
) { ) {
e.deliverDatabase(webhookDB, d) e.targets[database.TargetTypeDatabase].Deliver(
context.Background(), webhookDB, d, &Task{}, e,
)
} }
// ExportDeliverLog exposes deliverLog for testing. // ExportDeliverLog delivers via the log target for testing.
func (e *Engine) ExportDeliverLog( func (e *Engine) ExportDeliverLog(
webhookDB *gorm.DB, d *database.Delivery, webhookDB *gorm.DB, d *database.Delivery,
) { ) {
e.deliverLog(webhookDB, d) e.targets[database.TargetTypeLog].Deliver(
context.Background(), webhookDB, d, &Task{}, e,
)
} }
// ExportDeliverSlack exposes deliverSlack for testing. // ExportDeliverSlack delivers via the slack target for
// testing.
func (e *Engine) ExportDeliverSlack( func (e *Engine) ExportDeliverSlack(
ctx context.Context, ctx context.Context,
webhookDB *gorm.DB, webhookDB *gorm.DB,
d *database.Delivery, d *database.Delivery,
) { ) {
e.deliverSlack(ctx, webhookDB, d) task := &Task{
DeliveryID: d.ID,
TargetID: d.TargetID,
AttemptNum: 1,
}
e.targets[database.TargetTypeSlack].Deliver(
ctx, webhookDB, d, task, e,
)
} }
// ExportProcessNewTask exposes processNewTask. // ExportProcessNewTask exposes processNewTask.
@@ -96,53 +109,56 @@ func (e *Engine) ExportProcessDelivery(
e.processDelivery(ctx, webhookDB, d, task) e.processDelivery(ctx, webhookDB, d, task)
} }
// ExportGetCircuitBreaker exposes getCircuitBreaker. // ExportGetCircuitBreaker exposes the http target's
// getCircuitBreaker.
func (e *Engine) ExportGetCircuitBreaker( func (e *Engine) ExportGetCircuitBreaker(
targetID string, targetID string,
) *CircuitBreaker { ) *CircuitBreaker {
return e.getCircuitBreaker(targetID) return e.httpTarget.getCircuitBreaker(targetID)
} }
// ExportParseHTTPConfig exposes parseHTTPConfig. // ExportParseHTTPConfig exposes parseHTTPConfig.
func (e *Engine) ExportParseHTTPConfig( func (e *Engine) ExportParseHTTPConfig(
configJSON string, configJSON string,
) (*HTTPTargetConfig, error) { ) (*HTTPTargetConfig, error) {
return e.parseHTTPConfig(configJSON) return parseHTTPConfig(configJSON)
} }
// ExportParseSlackConfig exposes parseSlackConfig. // ExportParseSlackConfig exposes parseSlackConfig.
func (e *Engine) ExportParseSlackConfig( func (e *Engine) ExportParseSlackConfig(
configJSON string, configJSON string,
) (*SlackTargetConfig, error) { ) (*SlackTargetConfig, error) {
return e.parseSlackConfig(configJSON) return parseSlackConfig(configJSON)
} }
// ExportDoHTTPRequest exposes doHTTPRequest. // ExportDoHTTPRequest exposes the http target's
// doHTTPRequest.
func (e *Engine) ExportDoHTTPRequest( func (e *Engine) ExportDoHTTPRequest(
ctx context.Context, ctx context.Context,
cfg *HTTPTargetConfig, cfg *HTTPTargetConfig,
event *database.Event, event *database.Event,
) (int, string, int64, error) { ) (int, string, int64, error) {
return e.doHTTPRequest(ctx, cfg, event) return e.httpTarget.doHTTPRequest(ctx, cfg, event)
} }
// ExportClientForConfig exposes clientForConfig. // ExportClientForConfig exposes the http target's
// clientForConfig.
func (e *Engine) ExportClientForConfig( func (e *Engine) ExportClientForConfig(
cfg *HTTPTargetConfig, cfg *HTTPTargetConfig,
) *http.Client { ) *http.Client {
return e.clientForConfig(cfg) return e.httpTarget.clientForConfig(cfg)
} }
// ExportClient returns the engine's shared HTTP client. // ExportClient returns the http target's shared HTTP client.
func (e *Engine) ExportClient() *http.Client { func (e *Engine) ExportClient() *http.Client {
return e.client return e.httpTarget.client
} }
// ExportScheduleRetry exposes scheduleRetry. // ExportScheduleRetry exposes ScheduleRetry.
func (e *Engine) ExportScheduleRetry( func (e *Engine) ExportScheduleRetry(
task Task, delay time.Duration, task Task, delay time.Duration,
) { ) {
e.scheduleRetry(task, delay) e.ScheduleRetry(task, delay)
} }
// ExportRecoverPendingDeliveries exposes // ExportRecoverPendingDeliveries exposes
@@ -199,13 +215,15 @@ func NewTestEngine(
client *http.Client, client *http.Client,
workers int, workers int,
) *Engine { ) *Engine {
return &Engine{ e := &Engine{
log: log, log: log,
client: client,
deliveryCh: make(chan Task, deliveryChannelSize), deliveryCh: make(chan Task, deliveryChannelSize),
retryCh: make(chan Task, retryChannelSize), retryCh: make(chan Task, retryChannelSize),
workers: workers, workers: workers,
} }
e.initTargets(client)
return e
} }
// NewTestEngineSmallRetry creates an Engine with a tiny // NewTestEngineSmallRetry creates an Engine with a tiny
@@ -213,10 +231,13 @@ func NewTestEngine(
func NewTestEngineSmallRetry( func NewTestEngineSmallRetry(
log *slog.Logger, log *slog.Logger,
) *Engine { ) *Engine {
return &Engine{ e := &Engine{
log: log, log: log,
retryCh: make(chan Task, 1), retryCh: make(chan Task, 1),
} }
e.initTargets(nil)
return e
} }
// NewTestEngineWithDB creates an Engine with a real // NewTestEngineWithDB creates an Engine with a real
@@ -228,15 +249,17 @@ func NewTestEngineWithDB(
client *http.Client, client *http.Client,
workers int, workers int,
) *Engine { ) *Engine {
return &Engine{ e := &Engine{
database: db, database: db,
dbManager: dbMgr, dbManager: dbMgr,
log: log, log: log,
client: client,
deliveryCh: make(chan Task, deliveryChannelSize), deliveryCh: make(chan Task, deliveryChannelSize),
retryCh: make(chan Task, retryChannelSize), retryCh: make(chan Task, retryChannelSize),
workers: workers, workers: workers,
} }
e.initTargets(client)
return e
} }
// NewTestCircuitBreaker creates a CircuitBreaker with // NewTestCircuitBreaker creates a CircuitBreaker with
@@ -250,3 +273,64 @@ func NewTestCircuitBreaker(
cooldown: cooldown, cooldown: cooldown,
} }
} }
// ExportArchivedEvent aliases the archive row type so black-box
// tests can construct and read archive rows.
type ExportArchivedEvent = archivedEvent
// ExportArchiveWriter wraps an archiveWriter so black-box tests
// can exercise the per-webhook archive file mechanics.
type ExportArchiveWriter struct {
w *archiveWriter
}
// NewExportArchiveWriter builds an archive writer for tests,
// optionally overriding the reopen debounce (a non-positive
// debounce keeps the production default).
func NewExportArchiveWriter(
path string, log *slog.Logger, debounce time.Duration,
) *ExportArchiveWriter {
w := newArchiveWriter(path, log)
if debounce > 0 {
w.debounce = debounce
}
return &ExportArchiveWriter{w: w}
}
// Write archives a row through the writer.
func (e *ExportArchiveWriter) Write(
row ExportArchivedEvent, expiry time.Duration,
) error {
return e.w.write(row, expiry)
}
// Open opens the archive file, pruning when expiry is positive.
func (e *ExportArchiveWriter) Open(expiry time.Duration) error {
return e.w.open(expiry)
}
// Reopen closes and reopens the archive file.
func (e *ExportArchiveWriter) Reopen(
expiry time.Duration,
) error {
return e.w.reopen(expiry)
}
// Reopens reports how many times the file has been opened.
func (e *ExportArchiveWriter) Reopens() int {
return e.w.reopens
}
// DB returns the writer's current open handle for row
// inspection in tests.
func (e *ExportArchiveWriter) DB() *gorm.DB {
return e.w.db
}
// ExportParseArchiveExpiry exposes parseArchiveExpiry.
func ExportParseArchiveExpiry(
configJSON string,
) (time.Duration, error) {
return parseArchiveExpiry(configJSON)
}

101
internal/delivery/target.go Normal file
View File

@@ -0,0 +1,101 @@
package delivery
import (
"context"
"net/http"
"time"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// Scheduler re-enqueues a task for a future delivery attempt.
// The engine provides one to each target so a target can own
// its retries durably: it records the attempt, marks the
// delivery retrying, and asks the Scheduler to deliver the
// next attempt after delay — exactly what the engine does for
// its own restart recovery.
type Scheduler interface {
ScheduleRetry(task Task, delay time.Duration)
}
// Target delivers an event to one target type. Each type is
// an implementation. A Target owns its whole delivery: it
// makes the attempt, records the DeliveryResult and updates
// the DeliveryStatus, and — for targets that retry — decides
// whether to retry, computes its own backoff, gates with its
// own circuit breaker, and reschedules via the injected
// Scheduler. Fire-and-forget targets simply record a single
// attempt.
type Target interface {
Deliver(
ctx context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
)
}
// rescheduler is implemented by targets that own durable
// retries. The engine's restart recovery and periodic sweep
// use it to let the target recompute the schedule for an
// orphaned retrying delivery, keeping the retry schedule
// target-owned. Fire-and-forget targets do not implement it
// and their (never-occurring) retrying deliveries are
// skipped.
type rescheduler interface {
// remainingBackoff returns how long to wait before the
// next attempt of a recovered retrying delivery.
remainingBackoff(
webhookDB *gorm.DB,
deliveryID string,
attemptNum int,
) time.Duration
// backoffElapsed reports whether the backoff window for
// the last attempt has already passed, so the periodic
// sweep can re-enqueue the delivery now.
backoffElapsed(
webhookDB *gorm.DB,
deliveryID string,
attemptNum int,
) bool
}
// attemptResult is the outcome of a single delivery attempt,
// as reported by a target's per-attempt function to the
// shared retry core.
type attemptResult struct {
statusCode int
respBody string
duration int64
success bool
errMsg string
}
// initTargets builds the target registry, wiring each target
// to the engine's persistence helpers and giving the HTTP and
// Slack targets the shared SSRF-safe client. It is called by
// both New and the test constructors so the registry is
// always populated.
func (e *Engine) initTargets(client *http.Client) {
httpT := &httpTarget{
httpCore: &httpCore{eng: e},
client: client,
}
slackT := &slackTarget{
httpCore: &httpCore{eng: e},
client: client,
}
e.httpTarget = httpT
e.targets = map[database.TargetType]Target{
database.TargetTypeHTTP: httpT,
database.TargetTypeSlack: slackT,
database.TargetTypeDatabase: &databaseTarget{eng: e},
database.TargetTypeLog: &logTarget{eng: e},
}
}

View File

@@ -0,0 +1,137 @@
package delivery
import (
"context"
"fmt"
"path/filepath"
"sync"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// databaseTarget is a no-retry target that archives the
// full inbound event into a per-webhook archive SQLite file,
// separate from the per-webhook event database. The event is
// already persisted in the per-webhook event DB by the time
// delivery runs; the database target additionally writes a
// durable long-term copy into archive-{webhookID}.db and then
// records a single attempt whose outcome reflects whether the
// archive write succeeded. See archiveWriter for the
// close/reopen, auto-recreate, and expiry semantics.
type databaseTarget struct {
eng *Engine
mu sync.Mutex
writers map[string]*archiveWriter
}
// Deliver implements Target. It archives the event, then
// records one successful attempt and marks the delivery
// delivered. An archiving error fails the delivery: the
// attempt is recorded as failed with the error and the
// delivery is marked failed, so a target that could not do
// its one job (archiving) never reports success. The target
// does not retry; the event remains durably stored in the
// per-webhook event database.
func (t *databaseTarget) Deliver(
_ context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
_ *Task,
_ Scheduler,
) {
err := t.archive(d)
if err != nil {
t.eng.log.Error(
"failed to archive event to database target",
"delivery_id", d.ID,
"event_id", d.EventID,
"error", err,
)
t.eng.recordResult(
webhookDB, d, 1, false, 0, "",
err.Error(), 0,
)
t.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusFailed,
)
return
}
t.eng.recordResult(
webhookDB, d, 1, true, 0, "", "", 0,
)
t.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusDelivered,
)
}
// archive writes the full event as a row into the webhook's
// archive database, honouring the optional per-target expiry
// parsed from the target config JSON.
func (t *databaseTarget) archive(d *database.Delivery) error {
webhookID := d.Event.WebhookID
if webhookID == "" {
return errArchiveMissingWebhookID
}
expiry, err := parseArchiveExpiry(d.Target.Config)
if err != nil {
return err
}
w, err := t.writerFor(webhookID)
if err != nil {
return err
}
row := archivedEvent{
EventID: d.Event.ID,
WebhookID: webhookID,
EntrypointID: d.Event.EntrypointID,
Method: d.Event.Method,
Headers: d.Event.Headers,
Body: d.Event.Body,
ContentType: d.Event.ContentType,
}
return w.write(row, expiry)
}
// writerFor returns the archiveWriter for a webhook, creating
// and caching it on first use. Each webhook has one writer so
// its close/reopen debounce state is shared across concurrent
// deliveries. The archive file lives beside the per-webhook
// event database in the data directory.
func (t *databaseTarget) writerFor(
webhookID string,
) (*archiveWriter, error) {
if t.eng.dbManager == nil {
return nil, errArchiveNoDataDir
}
dir := filepath.Dir(t.eng.dbManager.DBPath(webhookID))
path := filepath.Join(
dir, fmt.Sprintf("archive-%s.db", webhookID),
)
t.mu.Lock()
defer t.mu.Unlock()
if t.writers == nil {
t.writers = make(map[string]*archiveWriter)
}
w, ok := t.writers[webhookID]
if !ok {
w = newArchiveWriter(path, t.eng.log)
t.writers[webhookID] = w
}
return w, nil
}

View File

@@ -0,0 +1,312 @@
package delivery
import (
"database/sql"
"encoding/json"
"errors"
"fmt"
"log/slog"
"os"
"sync"
"time"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
)
// archiveExpiryNever is the expiry sentinel (and default) that
// disables pruning so archived rows are kept forever.
const archiveExpiryNever = "never"
// archiveReopenDebounce bounds how often an archive file is
// closed and reopened. After each write the handle is closed
// and reopened so an operator can move the file away for
// offline archiving, but never more than once per this window.
const archiveReopenDebounce = time.Second
var (
// errArchiveMissingWebhookID is returned when an event to
// archive has no webhook id to key its archive file on.
errArchiveMissingWebhookID = errors.New(
"cannot archive event without a webhook id",
)
// errArchiveNoDataDir is returned when the database target
// has no webhook database manager and so cannot locate the
// data directory for archive files.
errArchiveNoDataDir = errors.New(
"database target has no data directory",
)
// errArchiveExpiryNotPositive is returned when a
// user-supplied archive expiry parses as a duration but is
// zero or negative; "never" is the way to disable pruning.
errArchiveExpiryNotPositive = errors.New(
"expiry must be a positive duration or \"never\"",
)
)
// databaseTargetConfig is the optional per-target JSON config
// for a database (archive) target.
type databaseTargetConfig struct {
// Expiry is a Go duration (e.g. "720h") after which
// archived rows are pruned, or "never" (the default) to
// keep them forever.
Expiry string `json:"expiry"`
}
// archivedEvent is one fully captured webhook event stored in a
// per-webhook archive database for long-term retention. It is a
// self-contained copy — independent of the per-webhook event
// database, which may prune events under its own retention.
type archivedEvent struct {
ID uint `gorm:"primaryKey;autoIncrement"`
EventID string `gorm:"index"`
WebhookID string
EntrypointID string
Method string
Headers string
Body string
ContentType string
// ArchivedAt is when the row was archived and is the age
// basis for expiry pruning.
ArchivedAt time.Time `gorm:"index"`
}
// parseArchiveExpiry reads the optional expiry from a database
// target's config JSON. An empty config, an empty expiry, or
// the literal "never" all mean keep forever, returned as a zero
// duration. Any other value must parse as a positive Go
// duration; a set-but-invalid value (unparseable, zero, or
// negative) is an error rather than a silent default, matching
// ValidateArchiveExpiry at target creation.
func parseArchiveExpiry(
configJSON string,
) (time.Duration, error) {
if configJSON == "" {
return 0, nil
}
var cfg databaseTargetConfig
err := json.Unmarshal([]byte(configJSON), &cfg)
if err != nil {
return 0, fmt.Errorf(
"parsing database target config: %w", err,
)
}
if cfg.Expiry == "" || cfg.Expiry == archiveExpiryNever {
return 0, nil
}
dur, err := time.ParseDuration(cfg.Expiry)
if err != nil {
return 0, fmt.Errorf(
"parsing archive expiry %q: %w", cfg.Expiry, err,
)
}
if dur <= 0 {
return 0, fmt.Errorf(
"%w: %q", errArchiveExpiryNotPositive, cfg.Expiry,
)
}
return dur, nil
}
// ValidateArchiveExpiry checks a user-supplied archive expiry
// for a database target at configuration time. Valid values are
// empty, "never" (both meaning keep forever), or a positive Go
// duration such as "720h". Anything else is an error, so a bad
// expiry is rejected when the target is created rather than
// failing every subsequent delivery.
func ValidateArchiveExpiry(expiry string) error {
if expiry == "" || expiry == archiveExpiryNever {
return nil
}
dur, err := time.ParseDuration(expiry)
if err != nil {
return fmt.Errorf(
"expiry must be %q or a Go duration "+
"such as \"720h\": %w",
archiveExpiryNever, err,
)
}
if dur <= 0 {
return fmt.Errorf(
"%w: %q", errArchiveExpiryNotPositive, expiry,
)
}
return nil
}
// archiveWriter owns one per-webhook archive SQLite file. It
// serialises writes, and after each write closes and reopens
// the file (debounced to at most once per debounce window) so
// an operator can move the file away for offline archiving. The
// next write recreates a moved or removed file, because the
// file is opened create-if-missing and its schema is migrated
// on every open.
type archiveWriter struct {
mu sync.Mutex
path string
log *slog.Logger
debounce time.Duration
db *gorm.DB
lastReopen time.Time
reopens int
}
// newArchiveWriter builds an archiveWriter for a file path with
// the default reopen debounce.
func newArchiveWriter(
path string, log *slog.Logger,
) *archiveWriter {
return &archiveWriter{
path: path,
log: log,
debounce: archiveReopenDebounce,
}
}
// write appends the event as a row, then applies the debounced
// close/reopen. It recreates the archive file if it was moved
// or removed since the last open. A positive expiry prunes rows
// older than it on each (re)open.
func (w *archiveWriter) write(
row archivedEvent, expiry time.Duration,
) error {
w.mu.Lock()
defer w.mu.Unlock()
if w.db == nil || !fileExists(w.path) {
err := w.reopen(expiry)
if err != nil {
return err
}
}
row.ArchivedAt = time.Now()
err := w.db.Create(&row).Error
if err != nil {
return fmt.Errorf(
"archiving event to %s: %w", w.path, err,
)
}
if time.Since(w.lastReopen) >= w.debounce {
return w.reopen(expiry)
}
return nil
}
// open opens (creating if missing) the archive file, migrates
// its schema, records the reopen time, and prunes expired rows
// when expiry is positive.
func (w *archiveWriter) open(expiry time.Duration) error {
dbURL := fmt.Sprintf("file:%s?mode=rwc", w.path)
sqlDB, err := sql.Open("sqlite", dbURL)
if err != nil {
return fmt.Errorf(
"opening archive database %s: %w", w.path, err,
)
}
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
if err != nil {
_ = sqlDB.Close()
return fmt.Errorf(
"connecting to archive database %s: %w",
w.path, err,
)
}
err = gdb.AutoMigrate(&archivedEvent{})
if err != nil {
_ = sqlDB.Close()
return fmt.Errorf(
"migrating archive database %s: %w", w.path, err,
)
}
w.db = gdb
w.lastReopen = time.Now()
w.reopens++
if expiry > 0 {
w.prune(expiry)
}
return nil
}
// reopen closes any open handle and opens the file afresh. The
// fresh open recreates the file if it was moved away.
func (w *archiveWriter) reopen(expiry time.Duration) error {
w.close()
return w.open(expiry)
}
// close closes the underlying handle, if any.
func (w *archiveWriter) close() {
if w.db == nil {
return
}
sqlDB, err := w.db.DB()
if err == nil {
_ = sqlDB.Close()
}
w.db = nil
}
// prune deletes archived rows older than expiry, measured from
// each row's archived time. It runs on every (re)open, and
// because the file is reopened after writes this keeps the
// archive swept without a separate background sweeper. Failures
// are logged, not fatal: a prune error must not stop archiving.
func (w *archiveWriter) prune(expiry time.Duration) {
cutoff := time.Now().Add(-expiry)
res := w.db.Where("archived_at < ?", cutoff).
Delete(&archivedEvent{})
if res.Error != nil {
w.log.Error(
"failed to prune expired archive rows",
"path", w.path,
"error", res.Error,
)
return
}
if res.RowsAffected > 0 {
w.log.Info(
"pruned expired archive rows",
"path", w.path,
"rows_deleted", res.RowsAffected,
)
}
}
// fileExists reports whether a path currently exists.
func fileExists(path string) bool {
_, err := os.Stat(path)
return err == nil
}

View File

@@ -0,0 +1,395 @@
package delivery_test
import (
"database/sql"
"fmt"
"log/slog"
"net/http"
"os"
"path/filepath"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
_ "modernc.org/sqlite" // Pure Go SQLite driver.
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
func archiveTestLogger() *slog.Logger {
return slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
))
}
// openArchiveDBForRead opens an archive file read-only so a
// test can inspect the rows the writer persisted.
func openArchiveDBForRead(
t *testing.T, path string,
) *gorm.DB {
t.Helper()
sqlDB, err := sql.Open(
"sqlite",
fmt.Sprintf("file:%s?mode=ro", path),
)
require.NoError(t, err)
t.Cleanup(func() { _ = sqlDB.Close() })
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
require.NoError(t, err)
return gdb
}
// removeArchiveFiles simulates an operator moving the archive
// away by deleting the SQLite file and its sidecar files.
func removeArchiveFiles(t *testing.T, path string) {
t.Helper()
for _, suffix := range []string{
"", "-wal", "-shm", "-journal",
} {
err := os.Remove(path + suffix)
if err != nil && !os.IsNotExist(err) {
t.Fatalf("removing %s%s: %v", path, suffix, err)
}
}
}
// TestDeliverDatabase_ArchivesEvent verifies that delivering to
// a database target marks the delivery delivered and archives
// the full event into a separate per-webhook archive file.
func TestDeliverDatabase_ArchivesEvent(t *testing.T) {
t.Parallel()
dataDir := t.TempDir()
dbMgr := database.NewTestWebhookDBManager(dataDir)
e := delivery.NewTestEngineWithDB(
nil, dbMgr,
archiveTestLogger(),
&http.Client{Timeout: 5 * time.Second},
1,
)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":true}`)
d := seedDatabaseTargetDelivery(t, webhookDB, event, "")
e.ExportDeliverDatabase(webhookDB, d)
var updated database.Delivery
require.NoError(t, webhookDB.First(
&updated, "id = ?", d.ID,
).Error)
assert.Equal(t,
database.DeliveryStatusDelivered, updated.Status,
"database target should mark the delivery delivered",
)
archivePath := filepath.Join(
dataDir,
fmt.Sprintf("archive-%s.db", event.WebhookID),
)
assert.FileExists(t, archivePath)
rdb := openArchiveDBForRead(t, archivePath)
var rows []delivery.ExportArchivedEvent
require.NoError(t, rdb.Find(&rows).Error)
require.Len(t, rows, 1)
assert.Equal(t, event.ID, rows[0].EventID)
assert.Equal(t, event.WebhookID, rows[0].WebhookID)
assert.Equal(t, event.Method, rows[0].Method)
assert.JSONEq(t, `{"archived":true}`, rows[0].Body)
}
func TestArchiveWriter_WritesRow(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-wh.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
row := delivery.ExportArchivedEvent{
EventID: "ev-1",
WebhookID: "wh-1",
EntrypointID: "ep-1",
Method: "POST",
Headers: `{"X":"Y"}`,
Body: `{"hello":"world"}`,
ContentType: "application/json",
}
require.NoError(t, w.Write(row, 0))
assert.FileExists(t, path)
var got []delivery.ExportArchivedEvent
require.NoError(t, w.DB().Find(&got).Error)
require.Len(t, got, 1)
assert.Equal(t, "ev-1", got[0].EventID)
assert.Equal(t, "wh-1", got[0].WebhookID)
assert.Equal(t, "ep-1", got[0].EntrypointID)
assert.Equal(t, row.Method, got[0].Method)
assert.Equal(t, row.ContentType, got[0].ContentType)
assert.JSONEq(t, `{"hello":"world"}`, got[0].Body)
assert.False(t, got[0].ArchivedAt.IsZero())
}
func TestArchiveWriter_RecreatesAfterRemoval(
t *testing.T,
) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-wh.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "a"}, 0,
))
assert.FileExists(t, path)
// The operator moves the archive away while the handle is
// still open.
removeArchiveFiles(t, path)
require.NoFileExists(t, path)
// The next write recreates the file with a fresh schema and
// only the new row.
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "b"}, 0,
))
assert.FileExists(t, path)
var got []delivery.ExportArchivedEvent
require.NoError(t, w.DB().Find(&got).Error)
require.Len(t, got, 1)
assert.Equal(t, "b", got[0].EventID)
}
func TestArchiveWriter_ReopenDebounce(t *testing.T) {
t.Parallel()
// A generous debounce keeps the two rapid writes inside
// the window even on a heavily loaded test machine.
path := filepath.Join(t.TempDir(), "archive-wh.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 2*time.Second,
)
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "a"}, 0,
))
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "b"}, 0,
))
// Two writes inside the debounce window trigger only the
// initial open — no extra close/reopen.
assert.Equal(t, 1, w.Reopens())
time.Sleep(2100 * time.Millisecond)
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "c"}, 0,
))
// A write after the window elapses closes and reopens once.
assert.Equal(t, 2, w.Reopens())
}
func TestArchiveWriter_ExpiryPrune(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-wh.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.Open(0))
old := delivery.ExportArchivedEvent{
EventID: "old",
ArchivedAt: time.Now().Add(-2 * time.Hour),
}
fresh := delivery.ExportArchivedEvent{
EventID: "fresh",
ArchivedAt: time.Now(),
}
require.NoError(t, w.DB().Create(&old).Error)
require.NoError(t, w.DB().Create(&fresh).Error)
// Reopening with a one-hour expiry prunes the old row.
require.NoError(t, w.Reopen(time.Hour))
var got []delivery.ExportArchivedEvent
require.NoError(t, w.DB().Find(&got).Error)
require.Len(t, got, 1)
assert.Equal(t, "fresh", got[0].EventID)
}
func TestParseArchiveExpiry(t *testing.T) {
t.Parallel()
cases := []struct {
name string
in string
want time.Duration
wantErr bool
}{
{"empty config", "", 0, false},
{"explicit never", `{"expiry":"never"}`, 0, false},
{"empty expiry", `{"expiry":""}`, 0, false},
{"duration", `{"expiry":"1h"}`, time.Hour, false},
{"unparseable", `{"expiry":"nonsense"}`, 0, true},
{"zero duration", `{"expiry":"0s"}`, 0, true},
{"negative duration", `{"expiry":"-5h"}`, 0, true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
got, err := delivery.ExportParseArchiveExpiry(tc.in)
if tc.wantErr {
require.Error(t, err)
return
}
require.NoError(t, err)
assert.Equal(t, tc.want, got)
})
}
}
// seedDatabaseTargetDelivery seeds a pending delivery for a
// database target with the given config JSON and returns the
// in-memory delivery the target handler is invoked with.
func seedDatabaseTargetDelivery(
t *testing.T,
webhookDB *gorm.DB,
event database.Event,
config string,
) *database.Delivery {
t.Helper()
dlv := seedDelivery(
t, webhookDB, event.ID, uuid.New().String(),
database.DeliveryStatusPending,
)
d := &database.Delivery{
EventID: event.ID,
TargetID: dlv.TargetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: database.Target{
Name: "test-db",
Type: database.TargetTypeDatabase,
Config: config,
},
}
d.ID = dlv.ID
return d
}
// TestDeliverDatabase_ArchiveFailureFailsDelivery verifies that
// an archive error (here: an unparseable expiry in the target
// config) fails the delivery loudly: the attempt is recorded as
// failed with the error and the delivery is marked failed, not
// delivered.
func TestDeliverDatabase_ArchiveFailureFailsDelivery(
t *testing.T,
) {
t.Parallel()
dataDir := t.TempDir()
e := delivery.NewTestEngineWithDB(
nil, database.NewTestWebhookDBManager(dataDir),
archiveTestLogger(),
&http.Client{Timeout: 5 * time.Second},
1,
)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":false}`)
d := seedDatabaseTargetDelivery(
t, webhookDB, event, `{"expiry":"nonsense"}`,
)
e.ExportDeliverDatabase(webhookDB, d)
var updated database.Delivery
require.NoError(t, webhookDB.First(
&updated, "id = ?", d.ID,
).Error)
assert.Equal(t,
database.DeliveryStatusFailed, updated.Status,
"archive failure must mark the delivery failed",
)
var results []database.DeliveryResult
require.NoError(t, webhookDB.Where(
"delivery_id = ?", d.ID,
).Find(&results).Error)
require.Len(t, results, 1)
assert.False(t,
results[0].Success,
"the attempt must be recorded as failed",
)
assert.Contains(t,
results[0].Error, "nonsense",
"the archive error must be recorded on the attempt",
)
assert.NoFileExists(t,
filepath.Join(
dataDir,
fmt.Sprintf("archive-%s.db", event.WebhookID),
),
"no archive file should exist for a failed config",
)
}
func TestValidateArchiveExpiry(t *testing.T) {
t.Parallel()
valid := []string{"", "never", "1h", "720h", "30m"}
for _, in := range valid {
require.NoError(t,
delivery.ValidateArchiveExpiry(in),
"expiry %q should be accepted", in,
)
}
invalid := []string{"nonsense", "7d", "-5h", "0s", "0"}
for _, in := range invalid {
require.Error(t,
delivery.ValidateArchiveExpiry(in),
"expiry %q should be rejected", in,
)
}
}

View File

@@ -0,0 +1,499 @@
package delivery
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"sync"
"time"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// Sentinel errors returned by the config parsers.
var (
errEmptyTargetConfig = errors.New(
"empty target config",
)
errMissingTargetURL = errors.New(
"target URL is required",
)
)
// HTTPTargetConfig holds configuration for http target
// types.
type HTTPTargetConfig struct {
URL string `json:"url"`
Headers map[string]string `json:"headers,omitempty"`
Timeout int `json:"timeout,omitempty"`
}
// httpCore holds the retry, backoff, and circuit-breaker
// machinery shared by the HTTP and Slack targets. Each of
// those targets owns its own httpCore instance (and thus its
// own circuit breakers); the per-attempt request differs
// between them and is supplied as a closure.
type httpCore struct {
eng *Engine
// circuitBreakers stores a *CircuitBreaker per target ID.
circuitBreakers sync.Map
}
// deliver runs one delivery attempt through the retry core.
// A maxRetries of 0 is fire-and-forget: a single attempt is
// recorded and no circuit breaker is consulted. A positive
// maxRetries gates the attempt on the circuit breaker and
// schedules a backed-off retry on failure.
func (c *httpCore) deliver(
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
maxRetries int,
attempt func() attemptResult,
) {
if maxRetries == 0 {
c.fireAndForget(webhookDB, d, attempt())
return
}
c.withRetry(
webhookDB, d, task, sched, maxRetries, attempt,
)
}
func (c *httpCore) fireAndForget(
webhookDB *gorm.DB,
d *database.Delivery,
res attemptResult,
) {
c.eng.recordResult(
webhookDB, d, 1, res.success,
res.statusCode, res.respBody, res.errMsg,
res.duration,
)
if res.success {
c.eng.updateDeliveryStatus(
webhookDB, d,
database.DeliveryStatusDelivered,
)
return
}
c.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusFailed,
)
}
func (c *httpCore) withRetry(
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
maxRetries int,
attempt func() attemptResult,
) {
cb := c.getCircuitBreaker(task.TargetID)
if c.circuitBreakerBlock(webhookDB, d, task, sched, cb) {
return
}
attemptNum := task.AttemptNum
res := attempt()
c.eng.recordResult(
webhookDB, d, attemptNum, res.success,
res.statusCode, res.respBody, res.errMsg,
res.duration,
)
if res.success {
cb.RecordSuccess()
c.eng.updateDeliveryStatus(
webhookDB, d,
database.DeliveryStatusDelivered,
)
return
}
cb.RecordFailure()
c.handleRetry(
webhookDB, d, task, sched, maxRetries, attemptNum,
)
}
func (c *httpCore) circuitBreakerBlock(
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
cb *CircuitBreaker,
) bool {
if cb.Allow() {
return false
}
remaining := cb.CooldownRemaining()
c.eng.log.Info(
"circuit breaker open, skipping delivery",
"target_id", task.TargetID,
"target_name", task.TargetName,
"delivery_id", d.ID,
"cooldown_remaining", remaining,
)
c.eng.updateDeliveryStatus(
webhookDB, d,
database.DeliveryStatusRetrying,
)
retryTask := *task
sched.ScheduleRetry(retryTask, remaining)
return true
}
func (c *httpCore) handleRetry(
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
maxRetries int,
attemptNum int,
) {
if attemptNum >= maxRetries {
c.eng.updateDeliveryStatus(
webhookDB, d,
database.DeliveryStatusFailed,
)
return
}
c.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusRetrying,
)
backoff := calcBackoff(attemptNum)
retryTask := *task
retryTask.AttemptNum = attemptNum + 1
sched.ScheduleRetry(retryTask, backoff)
}
func (c *httpCore) getCircuitBreaker(
targetID string,
) *CircuitBreaker {
if val, ok := c.circuitBreakers.Load(targetID); ok {
cb, _ := val.(*CircuitBreaker)
return cb
}
fresh := NewCircuitBreaker()
actual, _ := c.circuitBreakers.LoadOrStore(
targetID, fresh,
)
cb, _ := actual.(*CircuitBreaker)
return cb
}
// remainingBackoff returns how long remains of the backoff
// window for the last attempt of a recovered retrying
// delivery. It implements rescheduler.
func (c *httpCore) remainingBackoff(
webhookDB *gorm.DB,
deliveryID string,
attemptNum int,
) time.Duration {
var lastResult database.DeliveryResult
err := webhookDB.
Where("delivery_id = ?", deliveryID).
Order("created_at DESC").
First(&lastResult).Error
if err != nil {
return 0
}
backoff := calcBackoff(attemptNum)
elapsed := time.Since(lastResult.CreatedAt)
remaining := backoff - elapsed
return max(remaining, 0)
}
// backoffElapsed reports whether the backoff window for the
// last attempt of a retrying delivery has passed. It
// implements rescheduler.
func (c *httpCore) backoffElapsed(
webhookDB *gorm.DB,
deliveryID string,
attemptNum int,
) bool {
var lastResult database.DeliveryResult
err := webhookDB.
Where("delivery_id = ?", deliveryID).
Order("created_at DESC").
First(&lastResult).Error
if err != nil {
return true
}
backoff := calcBackoff(attemptNum)
return time.Since(lastResult.CreatedAt) >= backoff
}
func calcBackoff(attemptNum int) time.Duration {
shift := max(attemptNum-1, 0)
shift = min(shift, maxBackoffShift)
return time.Duration(1<<uint(shift)) * time.Second
}
// httpTarget delivers events to http targets. It forwards the
// event body and (filtered) request headers to the configured
// URL and owns retry, backoff, and circuit breaking through
// the shared httpCore.
type httpTarget struct {
*httpCore
client *http.Client
}
// Deliver implements Target.
func (t *httpTarget) Deliver(
ctx context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
) {
cfg, err := parseHTTPConfig(d.Target.Config)
if err != nil {
t.eng.log.Error(
"invalid HTTP target config",
"target_id", d.TargetID,
"error", err,
)
t.eng.recordResult(
webhookDB, d, task.AttemptNum,
false, 0, "", err.Error(), 0,
)
t.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusFailed,
)
return
}
attempt := func() attemptResult {
return t.attempt(ctx, cfg, &d.Event)
}
t.deliver(
webhookDB, d, task, sched,
d.Target.MaxRetries, attempt,
)
}
// attempt performs a single HTTP delivery attempt and derives
// the success flag and error message the same way the engine
// did: a non-2xx response is a failure but carries no error
// string; only a transport-level error does.
func (t *httpTarget) attempt(
ctx context.Context,
cfg *HTTPTargetConfig,
event *database.Event,
) attemptResult {
statusCode, respBody, duration, reqErr :=
t.doHTTPRequest(ctx, cfg, event)
success := reqErr == nil &&
statusCode >= httpSuccessMin &&
statusCode < httpSuccessMax
errMsg := ""
if reqErr != nil {
errMsg = reqErr.Error()
}
return attemptResult{
statusCode: statusCode,
respBody: respBody,
duration: duration,
success: success,
errMsg: errMsg,
}
}
func (t *httpTarget) doHTTPRequest(
ctx context.Context,
cfg *HTTPTargetConfig,
event *database.Event,
) (int, string, int64, error) {
start := time.Now()
req, reqErr := http.NewRequestWithContext(
ctx,
http.MethodPost,
cfg.URL,
bytes.NewReader([]byte(event.Body)),
)
if reqErr != nil {
return 0, "", 0, fmt.Errorf(
"creating request: %w", reqErr,
)
}
applyRequestHeaders(req, event, cfg)
client := t.clientForConfig(cfg)
resp, doErr := executeHTTPRequest(client, req)
dur := time.Since(start).Milliseconds()
if doErr != nil {
return 0, "", dur, fmt.Errorf(
"sending request: %w", doErr,
)
}
defer func() { _ = resp.Body.Close() }()
body, readErr := io.ReadAll(
io.LimitReader(resp.Body, maxBodyLog),
)
if readErr != nil {
return resp.StatusCode, "", dur,
fmt.Errorf(
"reading response body: %w", readErr,
)
}
return resp.StatusCode, string(body), dur, nil
}
func (t *httpTarget) clientForConfig(
cfg *HTTPTargetConfig,
) *http.Client {
if cfg.Timeout > 0 {
// Reuse the shared client's SSRF-safe transport so
// a per-target timeout does not drop the
// request-time private-IP guard. Only the timeout
// is overridden.
return &http.Client{
Timeout: time.Duration(
cfg.Timeout,
) * time.Second,
Transport: t.client.Transport,
}
}
return t.client
}
func parseHTTPConfig(
configJSON string,
) (*HTTPTargetConfig, error) {
if configJSON == "" {
return nil, errEmptyTargetConfig
}
var cfg HTTPTargetConfig
err := json.Unmarshal(
[]byte(configJSON), &cfg,
)
if err != nil {
return nil, fmt.Errorf(
"parsing config JSON: %w", err,
)
}
if cfg.URL == "" {
return nil, errMissingTargetURL
}
return &cfg, nil
}
// isForwardableHeader returns true if the header should
// be forwarded to targets.
func isForwardableHeader(name string) bool {
switch http.CanonicalHeaderKey(name) {
case "Host", "Connection", "Keep-Alive",
"Transfer-Encoding", "Te", "Trailer",
"Upgrade", "Proxy-Authorization",
"Proxy-Connection", "Content-Length":
return false
default:
return true
}
}
func applyRequestHeaders(
req *http.Request,
event *database.Event,
cfg *HTTPTargetConfig,
) {
if event.ContentType != "" {
req.Header.Set(
"Content-Type", event.ContentType,
)
}
var originalHeaders map[string][]string
if event.Headers != "" {
jsonErr := json.Unmarshal(
[]byte(event.Headers),
&originalHeaders,
)
if jsonErr == nil {
for k, vals := range originalHeaders {
if isForwardableHeader(k) {
for _, v := range vals {
req.Header.Add(k, v)
}
}
}
}
}
for k, v := range cfg.Headers {
req.Header.Set(k, v)
}
req.Header.Set("User-Agent", "webhooker/1.0")
}
// executeHTTPRequest sends an HTTP request using the provided
// client. URLs are validated by the config parsers and the
// SSRF-safe transport before reaching here.
func executeHTTPRequest(
client *http.Client, req *http.Request,
) (*http.Response, error) {
return client.Do(req) //#nosec G704 -- validated URL, SSRF-safe transport
}

View File

@@ -0,0 +1,47 @@
package delivery
import (
"context"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// logTarget is a fire-and-forget target that logs the entire
// inbound webhook — the full request body and headers, plus
// the method, content type, and the webhook and entrypoint
// ids — then records a single successful attempt.
type logTarget struct {
eng *Engine
}
// Deliver implements Target.
func (t *logTarget) Deliver(
_ context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
_ *Task,
_ Scheduler,
) {
t.eng.log.Info(
"webhook event delivered to log target",
"delivery_id", d.ID,
"event_id", d.EventID,
"target_id", d.TargetID,
"target_name", d.Target.Name,
"webhook_id", d.Event.WebhookID,
"entrypoint_id", d.Event.EntrypointID,
"method", d.Event.Method,
"content_type", d.Event.ContentType,
"headers", d.Event.Headers,
"body", d.Event.Body,
)
t.eng.recordResult(
webhookDB, d, 1, true, 0, "", "", 0,
)
t.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusDelivered,
)
}

View File

@@ -0,0 +1,299 @@
package delivery
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strings"
"time"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// errMissingWebhookURL is returned when a Slack target config
// omits its webhook URL.
var errMissingWebhookURL = errors.New(
"webhook_url is required",
)
// SlackTargetConfig holds configuration for slack target
// types.
type SlackTargetConfig struct {
WebhookURL string `json:"webhookUrl"`
}
// slackTarget delivers events to Slack incoming webhooks. It
// formats the event into a Slack message and posts it as
// JSON. It shares the retry core with the HTTP target: a
// MaxRetries of 0 stays single-attempt fire-and-forget
// (preserving existing Slack targets), while a positive
// MaxRetries adds backoff and circuit breaking.
type slackTarget struct {
*httpCore
client *http.Client
}
// Deliver implements Target.
func (t *slackTarget) Deliver(
ctx context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
) {
cfg, err := parseSlackConfig(d.Target.Config)
if err != nil {
t.eng.log.Error(
"invalid Slack target config",
"target_id", d.TargetID,
"error", err,
)
t.failConfig(webhookDB, d, err)
return
}
msg := FormatSlackMessage(&d.Event)
payload, err := json.Marshal(
map[string]string{"text": msg},
)
if err != nil {
t.eng.log.Error(
"failed to marshal Slack payload",
"target_id", d.TargetID,
"error", err,
)
t.failConfig(webhookDB, d, err)
return
}
attempt := func() attemptResult {
return t.attempt(ctx, cfg, payload)
}
t.deliver(
webhookDB, d, task, sched,
d.Target.MaxRetries, attempt,
)
}
// failConfig records a first-attempt failure for a delivery
// that could not be prepared (bad config or unmarshalable
// payload) and marks it failed.
func (t *slackTarget) failConfig(
webhookDB *gorm.DB,
d *database.Delivery,
err error,
) {
t.eng.recordResult(
webhookDB, d, 1,
false, 0, "", err.Error(), 0,
)
t.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusFailed,
)
}
// attempt performs a single Slack POST and derives its
// outcome, preserving the engine's original semantics: a
// non-2xx response records an "HTTP <code>" error string and
// a transport error records a "sending request" error.
func (t *slackTarget) attempt(
ctx context.Context,
cfg *SlackTargetConfig,
payload []byte,
) attemptResult {
start := time.Now()
req, err := http.NewRequestWithContext(
ctx,
http.MethodPost,
cfg.WebhookURL,
bytes.NewReader(payload),
)
if err != nil {
return attemptResult{
success: false,
errMsg: err.Error(),
}
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("User-Agent", "webhooker/1.0")
resp, doErr := executeHTTPRequest(t.client, req)
durationMs := time.Since(start).Milliseconds()
if doErr != nil {
return attemptResult{
success: false,
duration: durationMs,
errMsg: fmt.Errorf(
"sending request: %w", doErr,
).Error(),
}
}
defer func() { _ = resp.Body.Close() }()
return t.readSlackResponse(resp, durationMs)
}
func (t *slackTarget) readSlackResponse(
resp *http.Response,
durationMs int64,
) attemptResult {
body, readErr := io.ReadAll(
io.LimitReader(resp.Body, maxBodyLog),
)
if readErr != nil {
t.eng.log.Error(
"failed to read Slack response body",
"error", readErr,
)
}
success := resp.StatusCode >= httpSuccessMin &&
resp.StatusCode < httpSuccessMax
errMsg := ""
if !success {
errMsg = fmt.Sprintf("HTTP %d", resp.StatusCode)
}
return attemptResult{
statusCode: resp.StatusCode,
respBody: string(body),
duration: durationMs,
success: success,
errMsg: errMsg,
}
}
func parseSlackConfig(
configJSON string,
) (*SlackTargetConfig, error) {
if configJSON == "" {
return nil, errEmptyTargetConfig
}
var cfg SlackTargetConfig
err := json.Unmarshal(
[]byte(configJSON), &cfg,
)
if err != nil {
return nil, fmt.Errorf(
"parsing config JSON: %w", err,
)
}
if cfg.WebhookURL == "" {
return nil, errMissingWebhookURL
}
return &cfg, nil
}
// FormatSlackMessage builds a Slack-compatible message
// string from a webhook event.
func FormatSlackMessage(
event *database.Event,
) string {
var b strings.Builder
b.WriteString("*Webhook Event Received*\n")
fmt.Fprintf(
&b, "*Method:* `%s`\n", event.Method,
)
fmt.Fprintf(
&b,
"*Content-Type:* `%s`\n",
event.ContentType,
)
fmt.Fprintf(
&b,
"*Timestamp:* `%s`\n",
event.CreatedAt.UTC().Format(time.RFC3339),
)
fmt.Fprintf(
&b,
"*Body Size:* %d bytes\n",
len(event.Body),
)
if event.Body == "" {
b.WriteString("\n_(empty body)_\n")
return b.String()
}
if formatted := formatJSONBody(event.Body); formatted != "" {
b.WriteString(formatted)
return b.String()
}
formatRawBody(&b, event.Body)
return b.String()
}
func formatJSONBody(body string) string {
var parsed json.RawMessage
if json.Unmarshal([]byte(body), &parsed) != nil {
return ""
}
var pretty bytes.Buffer
if json.Indent(&pretty, parsed, "", " ") != nil {
return ""
}
var b strings.Builder
b.WriteString("\n```\n")
prettyStr := pretty.String()
const maxPayloadDisplay = 3500
if len(prettyStr) > maxPayloadDisplay {
b.WriteString(prettyStr[:maxPayloadDisplay])
b.WriteString("\n... (truncated)")
} else {
b.WriteString(prettyStr)
}
b.WriteString("\n```\n")
return b.String()
}
func formatRawBody(b *strings.Builder, body string) {
b.WriteString("\n```\n")
const maxRawDisplay = 3500
if len(body) > maxRawDisplay {
b.WriteString(body[:maxRawDisplay])
b.WriteString("\n... (truncated)")
} else {
b.WriteString(body)
}
b.WriteString("\n```\n")
}

View File

@@ -19,7 +19,7 @@ func (h *Handlers) HandleLoginPage() http.HandlerFunc {
// Render login page // Render login page
data := map[string]any{ data := map[string]any{
"Error": "", tmplKeyError: "",
} }
h.renderTemplate(w, r, "login.html", data) h.renderTemplate(w, r, "login.html", data)
@@ -86,7 +86,7 @@ func (h *Handlers) renderLoginError(
status int, status int,
) { ) {
data := map[string]any{ data := map[string]any{
"Error": msg, tmplKeyError: msg,
} }
w.WriteHeader(status) w.WriteHeader(status)

View File

@@ -13,12 +13,26 @@ func (s *Handlers) RenderTemplateForTest(
s.renderTemplate(w, r, pageTemplate, data) s.renderTemplate(w, r, pageTemplate, data)
} }
// BuildSlackTargetConfigForTest exposes buildSlackTargetConfig // BuildSlackTargetConfigForTest exposes buildURLTargetConfig
// for use in the handlers_test package. // with the Slack target parameters for use in the
// handlers_test package.
func (s *Handlers) BuildSlackTargetConfigForTest( func (s *Handlers) BuildSlackTargetConfigForTest(
w http.ResponseWriter, w http.ResponseWriter,
r *http.Request, r *http.Request,
targetURL string, targetURL string,
) (string, error) { ) (string, error) {
return s.buildSlackTargetConfig(w, r, targetURL) return s.buildURLTargetConfig(
w, r, targetURL, "webhookUrl",
"Webhook URL is required for Slack targets",
)
}
// BuildDatabaseTargetConfigForTest exposes
// buildDatabaseTargetConfig for use in the handlers_test
// package.
func (s *Handlers) BuildDatabaseTargetConfigForTest(
w http.ResponseWriter,
expiry string,
) (string, error) {
return s.buildDatabaseTargetConfig(w, expiry)
} }

View File

@@ -30,6 +30,11 @@ const (
defaultRetentionDays = 30 defaultRetentionDays = 30
// paginationPerPage is the number of items per page. // paginationPerPage is the number of items per page.
paginationPerPage = 25 paginationPerPage = 25
// tmplKeyError is the template data key for an error message.
tmplKeyError = "Error"
// tmplKeyWebhook is the template data key for a webhook.
tmplKeyWebhook = "Webhook"
) )
// errInvalidPassword is returned when a password does not match. // errInvalidPassword is returned when a password does not match.

View File

@@ -186,3 +186,57 @@ func TestRenderTemplate(t *testing.T) {
t, http.StatusInternalServerError, w.Code, t, http.StatusInternalServerError, w.Code,
) )
} }
func TestBuildDatabaseTargetConfig_Valid(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
app := newTestApp(t, &h)
app.RequireStart()
t.Cleanup(app.RequireStop)
// Empty expiry: the keep-forever default, empty config.
w := httptest.NewRecorder()
cfg, err := h.BuildDatabaseTargetConfigForTest(w, "")
require.NoError(t, err)
assert.Empty(t, cfg)
// Explicit never is stored as config.
w = httptest.NewRecorder()
cfg, err = h.BuildDatabaseTargetConfigForTest(w, "never")
require.NoError(t, err)
assert.JSONEq(t, `{"expiry":"never"}`, cfg)
// A positive duration is stored as config.
w = httptest.NewRecorder()
cfg, err = h.BuildDatabaseTargetConfigForTest(w, "720h")
require.NoError(t, err)
assert.JSONEq(t, `{"expiry":"720h"}`, cfg)
}
func TestBuildDatabaseTargetConfig_RejectsBadExpiry(
t *testing.T,
) {
t.Parallel()
var h *handlers.Handlers
app := newTestApp(t, &h)
app.RequireStart()
t.Cleanup(app.RequireStop)
for _, bad := range []string{"nonsense", "7d", "-5h"} {
w := httptest.NewRecorder()
cfg, err := h.BuildDatabaseTargetConfigForTest(w, bad)
require.Error(t, err, "expiry %q", bad)
assert.Empty(t, cfg)
assert.Equal(
t, http.StatusBadRequest, w.Code,
"expiry %q should be rejected with 400", bad,
)
}
}

View File

@@ -5,6 +5,7 @@ import (
"errors" "errors"
"net/http" "net/http"
"strconv" "strconv"
"strings"
"github.com/go-chi/chi" "github.com/go-chi/chi"
"github.com/google/uuid" "github.com/google/uuid"
@@ -106,7 +107,7 @@ func (h *Handlers) buildWebhookListItems(
func (h *Handlers) HandleSourceCreate() http.HandlerFunc { func (h *Handlers) HandleSourceCreate() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) {
data := map[string]any{ data := map[string]any{
"Error": "", tmplKeyError: "",
} }
h.renderTemplate(w, r, "sources_new.html", data) h.renderTemplate(w, r, "sources_new.html", data)
@@ -145,7 +146,7 @@ func (h *Handlers) HandleSourceCreateSubmit() http.HandlerFunc {
if name == "" { if name == "" {
data := map[string]any{ data := map[string]any{
"Error": "Name is required", tmplKeyError: "Name is required",
} }
w.WriteHeader(http.StatusBadRequest) w.WriteHeader(http.StatusBadRequest)
@@ -315,7 +316,7 @@ func (h *Handlers) renderSourceDetail(
} }
data := map[string]any{ data := map[string]any{
"Webhook": webhook, tmplKeyWebhook: webhook,
"Entrypoints": entrypoints, "Entrypoints": entrypoints,
"Targets": targets, "Targets": targets,
"Events": events, "Events": events,
@@ -351,8 +352,8 @@ func (h *Handlers) HandleSourceEdit() http.HandlerFunc {
} }
data := map[string]any{ data := map[string]any{
"Webhook": webhook, tmplKeyWebhook: webhook,
"Error": "", tmplKeyError: "",
} }
h.renderTemplate(w, r, "source_edit.html", data) h.renderTemplate(w, r, "source_edit.html", data)
@@ -415,8 +416,8 @@ func (h *Handlers) applyWebhookEdit(
name := r.FormValue("name") name := r.FormValue("name")
if name == "" { if name == "" {
data := map[string]any{ data := map[string]any{
"Webhook": *webhook, tmplKeyWebhook: *webhook,
"Error": "Name is required", tmplKeyError: "Name is required",
} }
w.WriteHeader(http.StatusBadRequest) w.WriteHeader(http.StatusBadRequest)
@@ -589,7 +590,7 @@ func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
} }
data := map[string]any{ data := map[string]any{
"Webhook": webhook, tmplKeyWebhook: webhook,
"Events": evts, "Events": evts,
"Page": page, "Page": page,
"TotalPages": totalPages, "TotalPages": totalPages,
@@ -815,6 +816,7 @@ func (h *Handlers) processTargetCreate(
targetType := database.TargetType(r.FormValue("type")) targetType := database.TargetType(r.FormValue("type"))
targetURL := r.FormValue("url") targetURL := r.FormValue("url")
maxRetriesStr := r.FormValue("max_retries") maxRetriesStr := r.FormValue("max_retries")
expiry := r.FormValue("expiry")
if name == "" { if name == "" {
http.Error( http.Error(
@@ -834,7 +836,7 @@ func (h *Handlers) processTargetCreate(
} }
configJSON, err := h.buildTargetConfig( configJSON, err := h.buildTargetConfig(
w, r, targetType, targetURL, w, r, targetType, targetURL, expiry,
) )
if err != nil { if err != nil {
return return
@@ -892,18 +894,28 @@ func parseNonNegativeInt(s string) int {
} }
// buildTargetConfig builds the JSON config string for a target. // buildTargetConfig builds the JSON config string for a target.
// The expiry form value is read by the caller (which bounds the
// request body) and applies to database targets only.
func (h *Handlers) buildTargetConfig( func (h *Handlers) buildTargetConfig(
w http.ResponseWriter, w http.ResponseWriter,
r *http.Request, r *http.Request,
targetType database.TargetType, targetType database.TargetType,
targetURL string, targetURL, expiry string,
) (string, error) { ) (string, error) {
switch targetType { switch targetType {
case database.TargetTypeHTTP: case database.TargetTypeHTTP:
return h.buildHTTPTargetConfig(w, r, targetURL) return h.buildURLTargetConfig(
w, r, targetURL, "url",
"URL is required for HTTP targets",
)
case database.TargetTypeSlack: case database.TargetTypeSlack:
return h.buildSlackTargetConfig(w, r, targetURL) return h.buildURLTargetConfig(
case database.TargetTypeDatabase, database.TargetTypeLog: w, r, targetURL, "webhookUrl",
"Webhook URL is required for Slack targets",
)
case database.TargetTypeDatabase:
return h.buildDatabaseTargetConfig(w, expiry)
case database.TargetTypeLog:
return "", nil return "", nil
default: default:
http.Error( http.Error(
@@ -915,16 +927,18 @@ func (h *Handlers) buildTargetConfig(
} }
} }
// buildHTTPTargetConfig builds config JSON for an HTTP target. // buildURLTargetConfig builds config JSON for a target whose
func (h *Handlers) buildHTTPTargetConfig( // configuration is a single SSRF-validated URL stored under
// configKey. missingMsg is the error shown when no URL is given.
func (h *Handlers) buildURLTargetConfig(
w http.ResponseWriter, w http.ResponseWriter,
r *http.Request, r *http.Request,
targetURL string, targetURL, configKey, missingMsg string,
) (string, error) { ) (string, error) {
if targetURL == "" { if targetURL == "" {
http.Error( http.Error(
w, w,
"URL is required for HTTP targets", missingMsg,
http.StatusBadRequest, http.StatusBadRequest,
) )
@@ -949,7 +963,7 @@ func (h *Handlers) buildHTTPTargetConfig(
return "", err return "", err
} }
cfg := map[string]any{"url": targetURL} cfg := map[string]any{configKey: targetURL}
configBytes, err := json.Marshal(cfg) configBytes, err := json.Marshal(cfg)
if err != nil { if err != nil {
@@ -964,41 +978,33 @@ func (h *Handlers) buildHTTPTargetConfig(
return string(configBytes), nil return string(configBytes), nil
} }
// buildSlackTargetConfig builds config JSON for a Slack target. // buildDatabaseTargetConfig builds config JSON for a database
func (h *Handlers) buildSlackTargetConfig( // (archive) target. The optional expiry (a form value read by
// the caller, which bounds the request body) is validated here,
// at creation time, so an unparseable value is rejected with a
// 400 instead of failing every subsequent delivery. An empty
// expiry yields an empty config (the keep-forever default).
func (h *Handlers) buildDatabaseTargetConfig(
w http.ResponseWriter, w http.ResponseWriter,
r *http.Request, expiry string,
targetURL string,
) (string, error) { ) (string, error) {
if targetURL == "" { expiry = strings.TrimSpace(expiry)
http.Error( if expiry == "" {
w, return "", nil
"Webhook URL is required for Slack targets",
http.StatusBadRequest,
)
return "", errMissingURL
} }
err := delivery.ValidateTargetURL( err := delivery.ValidateArchiveExpiry(expiry)
r.Context(), targetURL,
)
if err != nil { if err != nil {
h.log.Warn(
"target URL blocked by SSRF protection",
"url", targetURL,
"error", err,
)
http.Error( http.Error(
w, w,
"Invalid target URL: "+err.Error(), "Invalid archive expiry: "+err.Error(),
http.StatusBadRequest, http.StatusBadRequest,
) )
return "", err return "", err
} }
cfg := map[string]any{"webhookUrl": targetURL} cfg := map[string]any{"expiry": expiry}
configBytes, err := json.Marshal(cfg) configBytes, err := json.Marshal(cfg)
if err != nil { if err != nil {

View File

@@ -329,6 +329,7 @@ func (h *Handlers) buildDeliveryTasks(
DeliveryID: dlv.ID, DeliveryID: dlv.ID,
EventID: event.ID, EventID: event.ID,
WebhookID: entrypoint.WebhookID, WebhookID: entrypoint.WebhookID,
EntrypointID: entrypoint.ID,
TargetID: targets[i].ID, TargetID: targets[i].ID,
TargetName: targets[i].Name, TargetName: targets[i].Name,
TargetType: targets[i].Type, TargetType: targets[i].Type,

View File

@@ -265,6 +265,26 @@ func (s *Middleware) SecurityHeaders() func(http.Handler) http.Handler {
} }
} }
// NoCache returns middleware that instructs browsers and
// intermediary proxies not to cache the response. It sets
// Cache-Control: no-store and Pragma: no-cache (the latter for
// older HTTP/1.0 intermediaries). Apply it to authenticated pages
// so webhook configuration and captured event data are not stored
// by caches.
func (s *Middleware) NoCache() func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(
w http.ResponseWriter,
r *http.Request,
) {
w.Header().Set("Cache-Control", "no-store")
w.Header().Set("Pragma", "no-cache")
next.ServeHTTP(w, r)
})
}
}
// MaxBodySize returns middleware that limits the request body size // MaxBodySize returns middleware that limits the request body size
// for POST requests. If the body exceeds the given limit in // for POST requests. If the body exceeds the given limit in
// bytes, the server returns 413 Request Entity Too Large. This // bytes, the server returns 413 Request Entity Too Large. This

View File

@@ -387,6 +387,45 @@ func TestRequireAuth_UnauthenticatedSession_RedirectsToLogin(
assert.Equal(t, "/pages/login", w.Header().Get("Location")) assert.Equal(t, "/pages/login", w.Header().Get("Location"))
} }
// --- NoCache Middleware Tests ---
func TestNoCache_SetsHeaders(t *testing.T) {
t.Parallel()
m, _ := testMiddleware(t, config.EnvironmentDev)
var called bool
handler := m.NoCache()(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
called = true
w.WriteHeader(http.StatusOK)
},
))
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet, "/sources", nil,
)
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
assert.True(
t, called,
"NoCache middleware should call the next handler",
)
assert.Equal(
t, "no-store",
w.Header().Get("Cache-Control"),
)
assert.Equal(
t, "no-cache",
w.Header().Get("Pragma"),
)
}
// --- Helper Tests --- // --- Helper Tests ---
func TestIpFromHostPort(t *testing.T) { func TestIpFromHostPort(t *testing.T) {
@@ -445,8 +484,13 @@ func metricsAuthMiddleware(
return middleware.NewForTest(log, cfg, sessManager) return middleware.NewForTest(log, cfg, sessManager)
} }
func TestMetricsAuth_ValidCredentials(t *testing.T) { // runMetricsAuthRequest sends a GET /metrics request with the
t.Parallel() // given basic-auth password through MetricsAuth and reports
// whether the wrapped handler ran plus the recorded response.
func runMetricsAuthRequest(
t *testing.T, password string,
) (bool, *httptest.ResponseRecorder) {
t.Helper()
m := metricsAuthMiddleware(t) m := metricsAuthMiddleware(t)
@@ -464,12 +508,20 @@ func TestMetricsAuth_ValidCredentials(t *testing.T) {
context.Background(), context.Background(),
http.MethodGet, "/metrics", nil, http.MethodGet, "/metrics", nil,
) )
req.SetBasicAuth("admin", "secret") req.SetBasicAuth("admin", password)
w := httptest.NewRecorder() w := httptest.NewRecorder()
handler.ServeHTTP(w, req) handler.ServeHTTP(w, req)
return called, w
}
func TestMetricsAuth_ValidCredentials(t *testing.T) {
t.Parallel()
called, w := runMetricsAuthRequest(t, "secret")
assert.True( assert.True(
t, called, t, called,
"handler should be called with valid basic auth", "handler should be called with valid basic auth",
@@ -480,27 +532,7 @@ func TestMetricsAuth_ValidCredentials(t *testing.T) {
func TestMetricsAuth_InvalidCredentials(t *testing.T) { func TestMetricsAuth_InvalidCredentials(t *testing.T) {
t.Parallel() t.Parallel()
m := metricsAuthMiddleware(t) called, w := runMetricsAuthRequest(t, "wrong-password")
var called bool
handler := m.MetricsAuth()(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
called = true
w.WriteHeader(http.StatusOK)
},
))
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet, "/metrics", nil,
)
req.SetBasicAuth("admin", "wrong-password")
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
assert.False( assert.False(
t, called, t, called,

View File

@@ -91,6 +91,7 @@ func (s *Server) setupRoutes() {
func (s *Server) setupPageRoutes() { func (s *Server) setupPageRoutes() {
s.router.Route("/pages", func(r chi.Router) { s.router.Route("/pages", func(r chi.Router) {
r.Use(s.mw.CSRF()) r.Use(s.mw.CSRF())
r.Use(s.mw.NoCache())
r.Use(s.mw.MaxBodySize(maxFormBodySize)) r.Use(s.mw.MaxBodySize(maxFormBodySize))
r.Group(func(r chi.Router) { r.Group(func(r chi.Router) {
@@ -106,6 +107,7 @@ func (s *Server) setupPageRoutes() {
func (s *Server) setupUserRoutes() { func (s *Server) setupUserRoutes() {
s.router.Route("/user/{username}", func(r chi.Router) { s.router.Route("/user/{username}", func(r chi.Router) {
r.Use(s.mw.CSRF()) r.Use(s.mw.CSRF())
r.Use(s.mw.NoCache())
r.Use(s.mw.RequireAuth()) r.Use(s.mw.RequireAuth())
r.Get("/", s.h.HandleProfile()) r.Get("/", s.h.HandleProfile())
}) })
@@ -114,6 +116,7 @@ func (s *Server) setupUserRoutes() {
func (s *Server) setupSourceRoutes() { func (s *Server) setupSourceRoutes() {
s.router.Route("/sources", func(r chi.Router) { s.router.Route("/sources", func(r chi.Router) {
r.Use(s.mw.CSRF()) r.Use(s.mw.CSRF())
r.Use(s.mw.NoCache())
r.Use(s.mw.RequireAuth()) r.Use(s.mw.RequireAuth())
r.Use(s.mw.MaxBodySize(maxFormBodySize)) r.Use(s.mw.MaxBodySize(maxFormBodySize))
r.Get("/", s.h.HandleSourceList()) r.Get("/", s.h.HandleSourceList())
@@ -123,6 +126,7 @@ func (s *Server) setupSourceRoutes() {
s.router.Route("/source/{sourceID}", func(r chi.Router) { s.router.Route("/source/{sourceID}", func(r chi.Router) {
r.Use(s.mw.CSRF()) r.Use(s.mw.CSRF())
r.Use(s.mw.NoCache())
r.Use(s.mw.RequireAuth()) r.Use(s.mw.RequireAuth())
r.Use(s.mw.MaxBodySize(maxFormBodySize)) r.Use(s.mw.MaxBodySize(maxFormBodySize))
r.Get("/", s.h.HandleSourceDetail()) r.Get("/", s.h.HandleSourceDetail())

View File

@@ -173,8 +173,18 @@ func TestSetUser_SetsAllFields(t *testing.T) {
) )
} }
func TestGetUserID(t *testing.T) { // testSessionGetter exercises a session string getter before and
t.Parallel() // after SetUser: it must report false with an empty value on a
// fresh session, then true with the expected value once
// SetUser(sess, "user-xyz", "bob") has run.
func testSessionGetter(
t *testing.T,
get func(
*session.Session, *sessions.Session,
) (string, bool),
expected string,
) {
t.Helper()
s := testSession(t) s := testSession(t)
@@ -185,44 +195,46 @@ func TestGetUserID(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
// Before setting user // Before setting user
userID, ok := s.GetUserID(sess) val, ok := get(s, sess)
assert.False( assert.False(
t, ok, "should return false when no user ID is set", t, ok, "should return false before SetUser",
) )
assert.Empty(t, userID) assert.Empty(t, val)
// After setting user // After setting user
s.SetUser(sess, "user-xyz", "bob") s.SetUser(sess, "user-xyz", "bob")
userID, ok = s.GetUserID(sess) val, ok = get(s, sess)
assert.True(t, ok) assert.True(t, ok)
assert.Equal(t, "user-xyz", userID) assert.Equal(t, expected, val)
}
func TestGetUserID(t *testing.T) {
t.Parallel()
testSessionGetter(
t,
func(
s *session.Session, sess *sessions.Session,
) (string, bool) {
return s.GetUserID(sess)
},
"user-xyz",
)
} }
func TestGetUsername(t *testing.T) { func TestGetUsername(t *testing.T) {
t.Parallel() t.Parallel()
s := testSession(t) testSessionGetter(
t,
req := httptest.NewRequestWithContext( func(
context.Background(), http.MethodGet, "/", nil) s *session.Session, sess *sessions.Session,
) (string, bool) {
sess, err := s.Get(req) return s.GetUsername(sess)
require.NoError(t, err) },
"bob",
// Before setting user
username, ok := s.GetUsername(sess)
assert.False(
t, ok, "should return false when no username is set",
) )
assert.Empty(t, username)
// After setting user
s.SetUser(sess, "user-xyz", "bob")
username, ok = s.GetUsername(sess)
assert.True(t, ok)
assert.Equal(t, "bob", username)
} }
// --- IsAuthenticated Tests --- // --- IsAuthenticated Tests ---

View File

@@ -12,7 +12,12 @@ import (
// middleware and handler tests to use real session functionality. The key // middleware and handler tests to use real session functionality. The key
// parameter is the raw 32-byte authentication key used for session encryption // parameter is the raw 32-byte authentication key used for session encryption
// and CSRF cookie signing. // and CSRF cookie signing.
func NewForTest(store *sessions.CookieStore, cfg *config.Config, log *slog.Logger, key []byte) *Session { func NewForTest(
store *sessions.CookieStore,
cfg *config.Config,
log *slog.Logger,
key []byte,
) *Session {
return &Session{ return &Session{
store: store, store: store,
key: key, key: key,

View File

@@ -10,11 +10,11 @@ set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Pinned versions, 2026-07-07. Never "latest"; exact versions only. # Pinned versions, 2026-08-07. Never "latest"; exact versions only.
GOLANGCI_LINT_VERSION="2.11.3" GOLANGCI_LINT_VERSION="2.12.2"
# sha256 of golangci-lint-2.11.3-linux-<arch>.tar.gz release archives # sha256 of golangci-lint-2.12.2-linux-<arch>.tar.gz release archives
GOLANGCI_LINT_SHA256_AMD64="87bb8cddbcc825d5778b64e8a91b46c0526b247f4e2f2904dea74ec7450475d1" GOLANGCI_LINT_SHA256_AMD64="8df580d2670fed8fa984aac0507099af8df275e665215f5c7a2ae3943893a553"
GOLANGCI_LINT_SHA256_ARM64="ee3d95f301359e7d578e6d99c8ad5aeadbabc5a13009a30b2b0df11c8058afe9" GOLANGCI_LINT_SHA256_ARM64="44cd40a8c76c86755375adfeea52cfd3533cb43d7bd647771e0ae065e166df3a"
PKGMGR="" PKGMGR=""
SUDO="" SUDO=""

View File

@@ -113,6 +113,10 @@
<input type="url" name="url" placeholder="https://hooks.slack.com/services/..." :disabled="targetType !== 'slack'" class="input text-sm"> <input type="url" name="url" placeholder="https://hooks.slack.com/services/..." :disabled="targetType !== 'slack'" class="input text-sm">
<p class="text-xs text-gray-500 mt-1">Slack or Mattermost incoming webhook URL. Payloads are pretty-printed in code blocks.</p> <p class="text-xs text-gray-500 mt-1">Slack or Mattermost incoming webhook URL. Payloads are pretty-printed in code blocks.</p>
</div> </div>
<div x-show="targetType === 'database'">
<input type="text" name="expiry" placeholder="never" :disabled="targetType !== 'database'" class="input text-sm">
<p class="text-xs text-gray-500 mt-1">Archive expiry: "never" (default) keeps rows forever, or a duration like "720h" prunes older rows.</p>
</div>
<button type="submit" class="btn-primary text-sm">Add Target</button> <button type="submit" class="btn-primary text-sm">Add Target</button>
</form> </form>
</div> </div>