sneak/webhooker
1
0
Fork 0
Code Issues 24 Pull Requests 8 Actions Packages Projects Releases Wiki Activity

Compare commits

base: sneak:issue-89-archive-lifecycle
Branches Tags
sneak:main sneak:issue-66-session-idle-timeout sneak:issue-82-orphaned-retrying sneak:issue-89-archive-lifecycle sneak:issue-97-lifecycle-context sneak:issue-79-retention-forever sneak:issue-90-body-limit-before-csrf sneak:issue-80-config-fail-loud sneak:feat/receiver-rate-limit sneak:issue-64-receiver-rate-limit sneak:issue-70-log-target
..
compare: sneak:issue-70-log-target
Branches Tags
sneak:issue-66-session-idle-timeout sneak:issue-82-orphaned-retrying sneak:issue-89-archive-lifecycle sneak:issue-97-lifecycle-context sneak:issue-79-retention-forever sneak:issue-90-body-limit-before-csrf sneak:issue-80-config-fail-loud sneak:main sneak:feat/receiver-rate-limit sneak:issue-64-receiver-rate-limit sneak:issue-70-log-target

1 Commits
issue-89-a ... issue-70-l

Author SHA1 Message Date
sneak
ed81db137e Implement the log delivery target (closes #70)
All checks were successful
check / check (push) Successful in 4s
Details
2026-08-07 19:57:16 +07:00
50 changed files with 1256 additions and 6336 deletions
Expand all files Collapse all files

8
.golangci.yml
View File

@@ -1,9 +1,5 @@
version: "2"
# Config schema uses the golangci-lint v2 layout (settings live under
# linters.settings, not top-level linters-settings) so that the
# thresholds below are actually applied by golangci-lint >= v2.
run:
timeout: 5m
modules-download-mode: readonly
@@ -18,7 +14,8 @@ linters:
- wsl # Deprecated, replaced by wsl_v5
- wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go
settings:
linters-settings:
lll:
line-length: 88
funlen:
@@ -30,5 +27,6 @@ linters:
threshold: 100
issues:
exclude-use-default: false
max-issues-per-linter: 0
max-same-issues: 0

4
Dockerfile
View File

@@ -1,8 +1,8 @@
# Lint stage
# golangci/golangci-lint:v2.12.2 (Debian-based), 2026-08-07
# golangci/golangci-lint:v2.11.3 (Debian-based), 2026-03-17
# Using Debian-based image because mattn/go-sqlite3 (CGO) does not
# compile on Alpine musl (off64_t is a glibc type).
FROM golangci/golangci-lint:v2.12.2@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
FROM golangci/golangci-lint:v2.11.3@sha256:e838e8ab68aaefe83e2408691510867ade9329c0e0b895a3fb35eb93d1c2a4ba AS lint
RUN apt-get update && apt-get install -y --no-install-recommends make && rm -rf /var/lib/apt/lists/*

61
README.md
View File

@@ -363,12 +363,10 @@ events should be forwarded.
greater than 0, failed deliveries are retried with exponential backoff
up to `max_retries` attempts, protected by a per-target circuit
breaker.
- **`database`** — Archive the full event as a row into a separate
per-webhook archive database (`archive-{webhookID}.db`) for long-term
retention, with an optional creation-validated expiry (default: keep
forever). No external delivery and no retries; an archive write
failure fails the delivery. See the database target section under
"Per-Webhook Event Databases" for the full semantics.
- **`database`** — Confirm the event is stored in the webhook's
per-webhook database (no external delivery). Since events are always
written to the per-webhook DB on ingestion, this target marks delivery
as immediately successful. Useful for ensuring durable event archival.
- **`log`** — Write the event to the application log (stdout). Useful
for debugging.
@@ -514,52 +512,11 @@ This separation provides:
page cache, and its own lock, so concurrent event ingestion across
webhooks won't contend.
The **database target type** builds on this architecture to provide
long-term archiving, separate from the per-webhook event database (which
may prune events under its own retention). Delivering to a database
target writes the full event — body, headers, method, content type, and
webhook/entrypoint/event identifiers — as a row into a dedicated archive
database, `archive-{webhookID}.db`, stored under the data directory
beside the event database. After each write the archive handle is closed
and reopened, debounced to at most once per second, so an operator can
move the archive file away for offline archiving without stopping the
service; a moved or removed archive file is recreated automatically on
the next write. An optional `expiry` in the target's config JSON (e.g.
`{"expiry":"720h"}`) is validated when the target is created — the
default (unset or the literal `never`) keeps rows forever — and rows
older than the expiry are pruned each time the archive is (re)opened. An
archive write failure is never silent success: the delivery records a
failed attempt with the error and is marked failed.
Because reopens only happen on writes, an archive belonging to a webhook
that has stopped receiving events would never be pruned. A background
**archive sweeper** closes that gap: on the same interval as the event
retention reaper (`RETENTION_SWEEP_INTERVAL`) it prunes every archive
whose database target declares a positive expiry, whether or not the
webhook is still receiving traffic. The sweep never creates an archive —
a webhook whose archive file does not yet exist is skipped, not
initialised — it takes the same per-webhook lock the write path uses, so
it can never interleave with a write, and it leaves the archive closed
afterwards so the move-the-file-away workflow keeps working. Archives
with no expiry, or the expiry `never`, are not touched by the sweep at
all.
Note that a webhook has one archive file but may carry more than one
`database` target, each with its own `expiry`. The shortest expiry
configured on any of them therefore governs the whole archive, and the
sweep applies it whether or not the webhook is still receiving events.
Configure a single `database` target per webhook unless you intend that.
Deleting a webhook releases its archive: the delivery engine's cached
archive writer is dropped and its file handle closed, so nothing lingers
after the webhook is gone. The archive **file itself is deliberately
left on disk**. Unlike the event database — per-webhook working storage
that is hard-deleted with the webhook — an archive is long-term storage
an operator may still want to keep or move away for offline retention,
and destroying it as a side effect of deleting a webhook would be
unrecoverable. Removing `archive-{webhookID}.db` is the operator's call.
Deleting a webhook's last `database` target releases the writer the same
way, and for the same reason leaves the file alone.
The **database target type** leverages this architecture: since events
are already stored in the per-webhook database by design, the database
target simply marks the delivery as immediately successful. The
per-webhook DB IS the dedicated event database — that's the whole point
of the database target type.
The **Slack target type** sends webhook events as formatted messages to
any Slack-compatible incoming webhook URL (works with Slack, Mattermost,

21
TODO.md
View File

@@ -10,13 +10,12 @@
# Status
pre-1.0. No git tags exist. main (4f5ecb1) is a working webhook proxy
pre-1.0. No git tags exist. main (afe88c6) is a working webhook proxy
with auth, CSRF/SSRF protections, login rate limiting, Slack target,
event retention (#63), the database archiving target (#43), the admin
password change flow (#65), policy compliance (#6), and pinned lint
tooling (#55). Note: TODO.md was deliberately deleted from this repo in
f9a9569 (2026-03-01, #6); its content was folded into the README TODO
section, which this draft reconstructs as of 2026-07-06.
policy compliance (#6), and pinned lint tooling (#55). Note: TODO.md was
deliberately deleted from this repo in f9a9569 (2026-03-01, #6); its
content was folded into the README TODO section, which this draft
reconstructs as of 2026-07-06.
# Next Step
@@ -29,16 +28,6 @@ databases currently grow without bound.
# Completed Steps
- 2026-08-09 Archive writer lifecycle (#89): deleting a webhook (or its
last `database` target) evicts the cached archive writer and closes
its handle while deliberately leaving `archive-{webhookID}.db` on
disk, and a new `ArchiveSweeper` prunes idle archives on the existing
`RETENTION_SWEEP_INTERVAL` without ever creating an archive file
- 2026-08-07 Update golangci-lint to v2.12.2 (Docker image digest in
`Dockerfile`, release-archive sha256 pins in `script/bootstrap`),
adopt the canonical `.golangci.yml` (v2 `linters.settings` layout so
`lll`/`funlen`/`cyclop`/`dupl` thresholds actually apply), and fix
all newly surfaced lint findings
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
Makefile shims, README Entrypoints section
- 2026-03-25 pin golangci-lint Docker image for linting (#55)

17
cmd/webhooker/main.go
View File

@@ -34,31 +34,16 @@ func main() {
config.New,
database.New,
database.NewWebhookDBManager,
database.NewRetentionReaper,
healthcheck.New,
session.New,
handlers.New,
middleware.New,
delivery.New,
delivery.NewArchiveSweeper,
// Wire *delivery.Engine as delivery.Notifier so the
// webhook handler can notify the engine of new deliveries.
func(e *delivery.Engine) delivery.Notifier { return e },
// Wire *delivery.Engine as delivery.WebhookEvictor so
// deleting a webhook releases its archive writer.
func(e *delivery.Engine) delivery.WebhookEvictor {
return e
},
server.New,
),
fx.Invoke(
func(
*server.Server,
*delivery.Engine,
*database.RetentionReaper,
*delivery.ArchiveSweeper,
) {
},
),
fx.Invoke(func(*server.Server, *delivery.Engine) {}),
).Run()
}

46
internal/config/config.go
View File

@@ -8,7 +8,6 @@ import (
"os"
"strconv"
"strings"
"time"
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/globals"
@@ -27,10 +26,6 @@ const (
// defaultPort is the default HTTP listen port.
defaultPort = 8080
// defaultRetentionSweepInterval is how often the retention
// reaper deletes events older than each webhook's RetentionDays.
defaultRetentionSweepInterval = time.Hour
)
// ErrInvalidEnvironment is returned when WEBHOOKER_ENVIRONMENT
@@ -56,10 +51,6 @@ type Config struct {
MetricsUsername string
Port int
SentryDSN string
// RetentionSweepInterval is how often the retention reaper runs.
RetentionSweepInterval time.Duration
params *ConfigParams
log *slog.Logger
}
@@ -104,30 +95,6 @@ func envInt(key string, defaultValue int) int {
return defaultValue
}
// envDuration returns the value of the named environment variable
// parsed as a Go duration (e.g. "1h", "30m"). Returns defaultValue if
// not set. If the variable is set but cannot be parsed, it returns a
// wrapped error naming the key and the bad value, so startup fails
// loudly rather than silently falling back to the default.
func envDuration(
key string,
defaultValue time.Duration,
) (time.Duration, error) {
v := os.Getenv(key)
if v == "" {
return defaultValue, nil
}
d, err := time.ParseDuration(v)
if err != nil {
return 0, fmt.Errorf(
"invalid duration for %s: %q: %w", key, v, err,
)
}
return d, nil
}
// New creates a Config by reading environment variables.
//
//nolint:revive // lc parameter is required by fx even if unused.
@@ -151,17 +118,6 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
)
}
// Parse the retention sweep interval; a set-but-unparseable value
// is a hard error so fx aborts startup rather than silently using
// the default.
retentionSweepInterval, err := envDuration(
"RETENTION_SWEEP_INTERVAL",
defaultRetentionSweepInterval,
)
if err != nil {
return nil, err
}
// Load configuration values from environment variables
s := &Config{
DataDir: envString("DATA_DIR"),
@@ -172,7 +128,6 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
MetricsPassword: envString("METRICS_PASSWORD"),
Port: envInt("PORT", defaultPort),
SentryDSN: envString("SENTRY_DSN"),
RetentionSweepInterval: retentionSweepInterval,
log: log,
params: &params,
}
@@ -196,7 +151,6 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
"debug", s.Debug,
"maintenanceMode", s.MaintenanceMode,
"dataDir", s.DataDir,
"retentionSweepInterval", s.RetentionSweepInterval.String(),
"hasSentryDSN", s.SentryDSN != "",
"hasMetricsAuth",
s.MetricsUsername != "" && s.MetricsPassword != "",

95
internal/config/config_test.go
View File

@@ -3,7 +3,6 @@ package config_test
import (
"os"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -121,100 +120,6 @@ func testEnvironmentConfigSuccess(
assert.Equal(t, isProd, cfg.IsProd())
}
func TestRetentionSweepInterval(t *testing.T) {
tests := []struct {
name string
set bool
value string
expectError bool
expected time.Duration
}{
{
name: "unset uses default",
set: false,
expected: time.Hour,
},
{
name: "valid value is parsed",
set: true,
value: "15m",
expected: 15 * time.Minute,
},
{
name: "unparseable value fails startup",
set: true,
value: "not-a-duration",
expectError: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
if tt.set {
t.Setenv("RETENTION_SWEEP_INTERVAL", tt.value)
} else {
require.NoError(t, os.Unsetenv(
"RETENTION_SWEEP_INTERVAL",
))
}
if tt.expectError {
testRetentionSweepIntervalError(t)
} else {
testRetentionSweepIntervalSuccess(t, tt.expected)
}
})
}
}
func testRetentionSweepIntervalError(t *testing.T) {
t.Helper()
var cfg *config.Config
app := fx.New(
fx.NopLogger,
fx.Provide(
globals.New,
logger.New,
config.New,
),
fx.Populate(&cfg),
)
assert.Error(t, app.Err())
}
func testRetentionSweepIntervalSuccess(
t *testing.T,
expected time.Duration,
) {
t.Helper()
var cfg *config.Config
app := fxtest.New(
t,
fx.Provide(
globals.New,
logger.New,
config.New,
),
fx.Populate(&cfg),
)
require.NoError(t, app.Err())
app.RequireStart()
defer app.RequireStop()
assert.Equal(t, expected, cfg.RetentionSweepInterval)
}
func TestDefaultDataDir(t *testing.T) {
for _, env := range []string{"", "dev", "prod"} {
name := env

13
internal/database/database_test.go
View File

@@ -11,15 +11,6 @@ import (
"sneak.berlin/go/webhooker/internal/logger"
)
const (
// testAppname is the Globals.Appname used in tests.
testAppname = "webhooker-test"
// testVersion is the Globals.Version used in tests.
testVersion = "test"
// testContentType is the event content type used in tests.
testContentType = "application/json"
)
func setupTestDB(
t *testing.T,
) (*database.Database, *fxtest.Lifecycle) {
@@ -28,8 +19,8 @@ func setupTestDB(
lc := fxtest.NewLifecycle(t)
g := &globals.Globals{
Appname: testAppname,
Version: testVersion,
Appname: "webhooker-test",
Version: "test",
}
l, err := logger.New(

31
internal/database/export_test.go
View File

@@ -1,31 +0,0 @@
package database
import (
"context"
"log/slog"
"os"
"time"
)
// NewTestRetentionReaper builds a RetentionReaper backed by the given
// main database and per-webhook database manager, without the fx
// lifecycle. Intended for tests.
func NewTestRetentionReaper(
db *Database,
mgr *WebhookDBManager,
) *RetentionReaper {
return &RetentionReaper{
db: db,
dbManager: mgr,
log: slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
interval: time.Hour,
}
}
// ExportSweep runs a single retention sweep synchronously for tests.
func (r *RetentionReaper) ExportSweep(ctx context.Context) {
r.sweep(ctx)
}

5
internal/database/model_entrypoint.go
View File

@@ -5,10 +5,7 @@ type Entrypoint struct {
BaseModel
WebhookID string `gorm:"type:uuid;not null" json:"webhookId"`
// Path is the URL path for this entrypoint.
Path string `gorm:"uniqueIndex;not null" json:"path"`
Path string `gorm:"uniqueIndex;not null" json:"path"` // URL path for this entrypoint
Description string `json:"description"`
Active bool `gorm:"default:true" json:"active"`

3
internal/database/model_target.go
View File

@@ -23,8 +23,7 @@ type Target struct {
// Configuration fields (JSON stored based on type)
Config string `gorm:"type:text" json:"config"` // JSON configuration
// For HTTP targets (max_retries=0 means fire-and-forget,
// >0 enables retries with backoff)
// For HTTP targets (max_retries=0 means fire-and-forget, >0 enables retries with backoff)
MaxRetries int `json:"maxRetries,omitempty"`
MaxQueueSize int `json:"maxQueueSize,omitempty"`

4
internal/database/model_webhook.go
View File

@@ -7,9 +7,7 @@ type Webhook struct {
UserID string `gorm:"type:uuid;not null" json:"userId"`
Name string `gorm:"not null" json:"name"`
Description string `json:"description"`
// RetentionDays is the number of days to retain events.
RetentionDays int `gorm:"default:30" json:"retentionDays"`
RetentionDays int `gorm:"default:30" json:"retentionDays"` // Days to retain events
// Relations
User User `json:"user,omitzero"`

252
internal/database/retention.go
View File

@@ -1,252 +0,0 @@
package database
import (
"context"
"fmt"
"log/slog"
"sync"
"time"
"go.uber.org/fx"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/logger"
)
// hoursPerDay converts a RetentionDays count into hours for cutoff
// computation.
const hoursPerDay = 24
// RetentionReaperParams holds the fx dependencies for the
// RetentionReaper.
type RetentionReaperParams struct {
fx.In
Config *config.Config
Database *Database
DBManager *WebhookDBManager
Logger *logger.Logger
}
// RetentionReaper periodically deletes expired events (and their
// dependent deliveries and delivery results) from each per-webhook
// database, enforcing every webhook's RetentionDays. Rows are removed
// permanently so that per-webhook SQLite files do not grow without
// bound.
type RetentionReaper struct {
db *Database
dbManager *WebhookDBManager
log *slog.Logger
interval time.Duration
cancel context.CancelFunc
wg sync.WaitGroup
}
// NewRetentionReaper creates the retention reaper and registers its
// fx lifecycle hooks. The background sweep loop starts on OnStart and
// stops cleanly on OnStop via context cancellation.
func NewRetentionReaper(
lc fx.Lifecycle,
params RetentionReaperParams,
) *RetentionReaper {
r := &RetentionReaper{
db: params.Database,
dbManager: params.DBManager,
log: params.Logger.Get(),
interval: params.Config.RetentionSweepInterval,
}
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
r.start(ctx)
return nil
},
OnStop: func(_ context.Context) error {
r.stop()
return nil
},
})
return r
}
func (r *RetentionReaper) start(ctx context.Context) {
ctx, cancel := context.WithCancel(ctx)
r.cancel = cancel
r.wg.Add(1)
go r.run(ctx)
r.log.Info(
"retention reaper started",
"interval", r.interval.String(),
)
}
func (r *RetentionReaper) stop() {
r.log.Info("retention reaper stopping")
if r.cancel != nil {
r.cancel()
}
r.wg.Wait()
r.log.Info("retention reaper stopped")
}
func (r *RetentionReaper) run(ctx context.Context) {
defer r.wg.Done()
ticker := time.NewTicker(r.interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
r.sweep(ctx)
}
}
}
// sweep lists every webhook from the main database and reaps expired
// rows from each per-webhook database whose RetentionDays is positive.
func (r *RetentionReaper) sweep(ctx context.Context) {
var webhooks []Webhook
err := r.db.DB().
Model(&Webhook{}).
Find(&webhooks).Error
if err != nil {
r.log.Error(
"retention sweep: failed to list webhooks",
"error", err,
)
return
}
for i := range webhooks {
select {
case <-ctx.Done():
return
default:
}
wh := webhooks[i]
// RetentionDays of zero or less means retain forever.
if wh.RetentionDays <= 0 {
continue
}
// Nothing to reap if the per-webhook database has never
// been created.
if !r.dbManager.DBExists(wh.ID) {
continue
}
r.reapWebhook(wh.ID, wh.RetentionDays)
}
}
// reapWebhook removes every expired event (and its dependents) from a
// single webhook's database.
func (r *RetentionReaper) reapWebhook(
webhookID string,
retentionDays int,
) {
db, err := r.dbManager.GetDB(webhookID)
if err != nil {
r.log.Error(
"retention sweep: failed to open webhook database",
"webhook_id", webhookID,
"error", err,
)
return
}
cutoff := time.Now().Add(
-time.Duration(retentionDays*hoursPerDay) * time.Hour,
)
deleted, err := reapExpired(db, cutoff)
if err != nil {
r.log.Error(
"retention sweep: failed to reap expired events",
"webhook_id", webhookID,
"error", err,
)
return
}
if deleted > 0 {
r.log.Info(
"retention sweep: reaped expired events",
"webhook_id", webhookID,
"retention_days", retentionDays,
"events_deleted", deleted,
)
}
}
// reapExpired hard-deletes, in foreign-key-safe order, the delivery
// results, deliveries, and events associated with events older than
// cutoff. Deletes are unscoped so rows are physically removed rather
// than soft-deleted, reclaiming disk. It returns the number of events
// deleted.
func reapExpired(db *gorm.DB, cutoff time.Time) (int64, error) {
// Fresh subqueries are built per statement to avoid reusing a
// mutated builder across executions.
expiredEventIDs := func() *gorm.DB {
return db.Model(&Event{}).
Select("id").
Where("created_at < ?", cutoff)
}
expiredDeliveryIDs := func() *gorm.DB {
return db.Model(&Delivery{}).
Select("id").
Where("event_id IN (?)", expiredEventIDs())
}
// 1. Delivery results whose delivery belongs to an expired event.
res := db.Unscoped().
Where("delivery_id IN (?)", expiredDeliveryIDs()).
Delete(&DeliveryResult{})
if res.Error != nil {
return 0, fmt.Errorf(
"deleting expired delivery results: %w",
res.Error,
)
}
// 2. Deliveries belonging to an expired event.
del := db.Unscoped().
Where("event_id IN (?)", expiredEventIDs()).
Delete(&Delivery{})
if del.Error != nil {
return 0, fmt.Errorf(
"deleting expired deliveries: %w",
del.Error,
)
}
// 3. The expired events themselves.
ev := db.Unscoped().
Where("created_at < ?", cutoff).
Delete(&Event{})
if ev.Error != nil {
return 0, fmt.Errorf(
"deleting expired events: %w",
ev.Error,
)
}
return ev.RowsAffected, nil
}

278
internal/database/retention_test.go
View File

@@ -1,278 +0,0 @@
package database_test
import (
"context"
"net/http"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/fx/fxtest"
"gorm.io/gorm"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/globals"
"sneak.berlin/go/webhooker/internal/logger"
)
// retentionTestEnv bundles the pieces a retention test drives.
type retentionTestEnv struct {
reaper *database.RetentionReaper
mainDB *database.Database
mgr *database.WebhookDBManager
}
func setupRetentionTest(t *testing.T) *retentionTestEnv {
t.Helper()
lc := fxtest.NewLifecycle(t)
g := &globals.Globals{
Appname: testAppname,
Version: testVersion,
}
l, err := logger.New(lc, logger.LoggerParams{Globals: g})
require.NoError(t, err)
cfg := &config.Config{
DataDir: t.TempDir(),
Environment: "dev",
}
mainDB, err := database.New(lc, database.DatabaseParams{
Config: cfg,
Logger: l,
})
require.NoError(t, err)
mgr, err := database.NewWebhookDBManager(
lc,
database.WebhookDBManagerParams{Config: cfg, Logger: l},
)
require.NoError(t, err)
ctx := context.Background()
require.NoError(t, lc.Start(ctx))
t.Cleanup(func() { require.NoError(t, lc.Stop(ctx)) })
return &retentionTestEnv{
reaper: database.NewTestRetentionReaper(mainDB, mgr),
mainDB: mainDB,
mgr: mgr,
}
}
// createWebhook inserts a webhook row into the main database with the
// given retention policy and returns its ID.
func createWebhook(
t *testing.T,
db *gorm.DB,
retentionDays int,
) string {
t.Helper()
wh := &database.Webhook{
UserID: uuid.New().String(),
Name: "test-webhook",
RetentionDays: retentionDays,
}
require.NoError(
t,
db.Omit(clause.Associations).Create(wh).Error,
)
// The RetentionDays column carries a GORM default of 30, so a
// zero (or negative) value passed to Create is replaced by that
// default. Force the requested value explicitly so the
// retain-forever (<= 0) path can be exercised.
require.NoError(
t,
db.Model(wh).
Update("retention_days", retentionDays).Error,
)
return wh.ID
}
// eventChain is the set of row IDs seeded for a single event.
type eventChain struct {
eventID string
deliveryID string
resultID string
}
// seedEventChain creates an event with one delivery and one delivery
// result, all stamped with createdAt, and returns their IDs.
func seedEventChain(
t *testing.T,
db *gorm.DB,
webhookID string,
createdAt time.Time,
) eventChain {
t.Helper()
event := &database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Body: `{"seed": true}`,
ContentType: testContentType,
}
event.CreatedAt = createdAt
require.NoError(t, db.Create(event).Error)
delivery := &database.Delivery{
EventID: event.ID,
TargetID: uuid.New().String(),
Status: database.DeliveryStatusDelivered,
}
delivery.CreatedAt = createdAt
require.NoError(t, db.Create(delivery).Error)
result := &database.DeliveryResult{
DeliveryID: delivery.ID,
AttemptNum: 1,
Success: true,
StatusCode: 200,
Duration: 10,
}
result.CreatedAt = createdAt
require.NoError(t, db.Create(result).Error)
return eventChain{
eventID: event.ID,
deliveryID: delivery.ID,
resultID: result.ID,
}
}
// countByID returns how many rows of model match the given id,
// counting even hard-deletable rows via Unscoped.
func countByID(
t *testing.T,
db *gorm.DB,
model any,
id string,
) int64 {
t.Helper()
var n int64
require.NoError(
t,
db.Unscoped().Model(model).
Where("id = ?", id).Count(&n).Error,
)
return n
}
func assertChainGone(
t *testing.T,
db *gorm.DB,
chain eventChain,
) {
t.Helper()
assert.Zero(
t,
countByID(t, db, &database.Event{}, chain.eventID),
"expired event should be removed",
)
assert.Zero(
t,
countByID(t, db, &database.Delivery{}, chain.deliveryID),
"expired delivery should be removed",
)
assert.Zero(
t,
countByID(
t, db, &database.DeliveryResult{}, chain.resultID,
),
"expired delivery result should be removed",
)
}
func assertChainPresent(
t *testing.T,
db *gorm.DB,
chain eventChain,
) {
t.Helper()
assert.Equal(
t,
int64(1),
countByID(t, db, &database.Event{}, chain.eventID),
"recent event should be retained",
)
assert.Equal(
t,
int64(1),
countByID(t, db, &database.Delivery{}, chain.deliveryID),
"recent delivery should be retained",
)
assert.Equal(
t,
int64(1),
countByID(
t, db, &database.DeliveryResult{}, chain.resultID,
),
"recent delivery result should be retained",
)
}
func TestRetentionReaper_ReapsExpiredKeepsRecent(t *testing.T) {
t.Parallel()
env := setupRetentionTest(t)
const retentionDays = 30
webhookID := createWebhook(
t, env.mainDB.DB(), retentionDays,
)
db, err := env.mgr.GetDB(webhookID)
require.NoError(t, err)
now := time.Now()
old := seedEventChain(
t, db, webhookID,
now.Add(-40*24*time.Hour),
)
recent := seedEventChain(
t, db, webhookID,
now.Add(-1*24*time.Hour),
)
env.reaper.ExportSweep(context.Background())
assertChainGone(t, db, old)
assertChainPresent(t, db, recent)
}
func TestRetentionReaper_RetainsForeverWhenNonPositive(t *testing.T) {
t.Parallel()
env := setupRetentionTest(t)
// RetentionDays of zero means retain forever.
webhookID := createWebhook(t, env.mainDB.DB(), 0)
db, err := env.mgr.GetDB(webhookID)
require.NoError(t, err)
ancient := seedEventChain(
t, db, webhookID,
time.Now().Add(-365*24*time.Hour),
)
env.reaper.ExportSweep(context.Background())
assertChainPresent(t, db, ancient)
}

10
internal/database/testing.go
View File

@@ -14,10 +14,7 @@ import (
func NewTestDatabase(db *gorm.DB) *Database {
return &Database{
db: db,
log: slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
log: slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelDebug})),
}
}
@@ -26,9 +23,6 @@ func NewTestDatabase(db *gorm.DB) *Database {
func NewTestWebhookDBManager(dataDir string) *WebhookDBManager {
return &WebhookDBManager{
dataDir: dataDir,
log: slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
log: slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelDebug})),
}
}

29
internal/database/webhook_db_manager_test.go
View File

@@ -2,7 +2,6 @@ package database_test
import (
"context"
"net/http"
"os"
"path/filepath"
"testing"
@@ -26,8 +25,8 @@ func setupTestWebhookDBManager(
lc := fxtest.NewLifecycle(t)
g := &globals.Globals{
Appname: testAppname,
Version: testVersion,
Appname: "webhooker-test",
Version: "test",
}
l, err := logger.New(
@@ -84,10 +83,10 @@ func TestWebhookDBManager_CreateAndGetDB(t *testing.T) {
event := &database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Method: "POST",
Headers: `{"Content-Type":["application/json"]}`,
Body: `{"test": true}`,
ContentType: testContentType,
ContentType: "application/json",
}
require.NoError(t, db.Create(event).Error)
assert.NotEmpty(t, event.ID)
@@ -100,7 +99,7 @@ func TestWebhookDBManager_CreateAndGetDB(t *testing.T) {
db.First(&readEvent, "id = ?", event.ID).Error,
)
assert.Equal(t, webhookID, readEvent.WebhookID)
assert.Equal(t, http.MethodPost, readEvent.Method)
assert.Equal(t, "POST", readEvent.Method)
assert.Equal(t, `{"test": true}`, readEvent.Body)
}
@@ -124,9 +123,9 @@ func TestWebhookDBManager_DeleteDB(t *testing.T) {
event := &database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Method: "POST",
Body: `{"test": true}`,
ContentType: testContentType,
ContentType: "application/json",
}
require.NoError(t, db.Create(event).Error)
@@ -197,10 +196,10 @@ func seedDeliveryWorkflow(
event := &database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Method: "POST",
Headers: `{"Content-Type":["application/json"]}`,
Body: `{"payload": "test"}`,
ContentType: testContentType,
ContentType: "application/json",
}
require.NoError(t, db.Create(event).Error)
@@ -232,7 +231,7 @@ func verifyPendingDeliveries(
)
require.Len(t, pending, 1)
assert.Equal(t, event.ID, pending[0].EventID)
assert.Equal(t, http.MethodPost, pending[0].Event.Method)
assert.Equal(t, "POST", pending[0].Event.Method)
}
func completeDelivery(
@@ -304,16 +303,16 @@ func TestWebhookDBManager_MultipleWebhooks(t *testing.T) {
event1 := &database.Event{
WebhookID: webhook1,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Method: "POST",
Body: `{"webhook": 1}`,
ContentType: testContentType,
ContentType: "application/json",
}
event2 := &database.Event{
WebhookID: webhook2,
EntrypointID: uuid.New().String(),
Method: http.MethodPut,
Method: "PUT",
Body: `{"webhook": 2}`,
ContentType: testContentType,
ContentType: "application/json",
}
require.NoError(t, db1.Create(event1).Error)

229
internal/delivery/archive_sweeper.go
View File

@@ -1,229 +0,0 @@
package delivery
import (
"context"
"errors"
"log/slog"
"sync"
"time"
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/logger"
)
// ArchiveSweeperParams holds the fx dependencies for the
// ArchiveSweeper.
type ArchiveSweeperParams struct {
fx.In
Config *config.Config
Database *database.Database
Engine *Engine
Logger *logger.Logger
}
// ArchiveSweeper periodically prunes expired rows from
// per-webhook archive databases whose database target carries a
// positive expiry.
//
// Without it, pruning happens only when an archive is
// (re)opened, and archives are only ever reopened by writes: an
// archive belonging to a webhook that has stopped receiving
// events would keep its expired rows forever. The sweep closes
// that gap without changing anything for archives whose expiry
// is unset or "never".
//
// It reuses Config.RetentionSweepInterval rather than
// introducing a second interval: this is a retention sweep with
// the same semantics as the event retention reaper.
type ArchiveSweeper struct {
db *database.Database
eng *Engine
log *slog.Logger
interval time.Duration
cancel context.CancelFunc
wg sync.WaitGroup
}
// NewArchiveSweeper creates the archive sweeper and registers
// its fx lifecycle hooks. The background sweep loop starts on
// OnStart and stops cleanly on OnStop via context cancellation.
func NewArchiveSweeper(
lc fx.Lifecycle,
params ArchiveSweeperParams,
) *ArchiveSweeper {
s := &ArchiveSweeper{
db: params.Database,
eng: params.Engine,
log: params.Logger.Get(),
interval: params.Config.RetentionSweepInterval,
}
s.registerHooks(lc)
return s
}
// registerHooks wires the sweeper's start and stop into the fx
// lifecycle. Both hook contexts are deliberately ignored: see
// start for why the background loop must not inherit the start
// hook's context, and stop for why shutdown blocks on the loop
// rather than on the stop hook's deadline.
func (s *ArchiveSweeper) registerHooks(lc fx.Lifecycle) {
lc.Append(fx.Hook{
//nolint:contextcheck // Not passing the hook context is
// the point: see start.
OnStart: func(_ context.Context) error {
s.start()
return nil
},
OnStop: func(_ context.Context) error {
s.stop()
return nil
},
})
}
// start launches the background sweep loop.
//
// The loop's context is derived from context.Background(), NOT
// from the fx OnStart hook context. The hook context carries
// fx's start timeout (15s by default), so a loop derived from it
// is cancelled 15 seconds after the application starts — long
// before the first tick under the default one-hour sweep
// interval, leaving a sweeper that never sweeps. A long-lived
// goroutine must outlive the startup phase, so its lifetime is
// bounded by OnStop instead: stop cancels this context and waits
// on the WaitGroup.
func (s *ArchiveSweeper) start() {
ctx, cancel := context.WithCancel(context.Background())
s.cancel = cancel
s.wg.Add(1)
go s.run(ctx)
s.log.Info(
"archive sweeper started",
"interval", s.interval.String(),
)
}
func (s *ArchiveSweeper) stop() {
s.log.Info("archive sweeper stopping")
if s.cancel != nil {
s.cancel()
}
s.wg.Wait()
s.log.Info("archive sweeper stopped")
}
func (s *ArchiveSweeper) run(ctx context.Context) {
defer s.wg.Done()
ticker := time.NewTicker(s.interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
s.sweep(ctx)
}
}
}
// sweep prunes every archive whose database target declares a
// positive expiry. Targets belonging to a deleted webhook are
// soft-deleted along with it, so GORM's default scope already
// excludes them.
//
// A failure for one webhook is logged and the sweep continues,
// matching how the write path already treats a prune error as
// non-fatal.
func (s *ArchiveSweeper) sweep(ctx context.Context) {
var targets []database.Target
err := s.db.DB().
Model(&database.Target{}).
Where("type = ?", database.TargetTypeDatabase).
Find(&targets).Error
if err != nil {
s.log.Error(
"archive sweep: failed to list database targets",
"error", err,
)
return
}
for i := range targets {
select {
case <-ctx.Done():
return
default:
}
s.sweepTarget(&targets[i])
}
}
// sweepTarget prunes the archive of a single database target.
// A missing, empty, or "never" expiry parses as a zero duration
// and is skipped entirely, so those archives keep exactly the
// behaviour they had before the sweep existed.
func (s *ArchiveSweeper) sweepTarget(target *database.Target) {
expiry, err := parseArchiveExpiry(target.Config)
if err != nil {
s.log.Error(
"archive sweep: invalid database target config",
"webhook_id", target.WebhookID,
"target_id", target.ID,
"error", err,
)
return
}
if expiry <= 0 {
return
}
if s.eng == nil || s.eng.dbTarget == nil {
return
}
err = s.eng.dbTarget.sweepWebhook(target.WebhookID, expiry)
if err == nil {
return
}
// A writer evicted underneath the sweep means the operator
// deleted the webhook (or its last database target) while the
// sweep was walking the target list. That is an ordinary
// interleaving, not a failure, so it must not produce an
// error line.
if errors.Is(err, errArchiveWriterEvicted) {
s.log.Debug(
"archive sweep: writer evicted mid-sweep",
"webhook_id", target.WebhookID,
"target_id", target.ID,
)
return
}
s.log.Error(
"archive sweep: failed to prune archive",
"webhook_id", target.WebhookID,
"target_id", target.ID,
"error", err,
)
}

930
internal/delivery/archive_sweeper_test.go
View File

@@ -1,930 +0,0 @@
package delivery_test
import (
"context"
"database/sql"
"fmt"
"net/http"
"os"
"path/filepath"
"sync"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/fx"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
"gorm.io/gorm/clause"
_ "modernc.org/sqlite" // Pure Go SQLite driver.
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
const (
// sweepRowOld and sweepRowNew are the event ids
// seedArchiveRows assigns to the first and second seeded
// rows.
sweepRowOld = "ev-0"
sweepRowNew = "ev-1"
// sweepConcurrentWrites is how many deliveries the
// concurrent write-plus-sweep test races against the sweep.
sweepConcurrentWrites = 20
)
// sweeperEnv bundles the pieces an archive sweep test drives:
// a main configuration database holding webhooks and targets, a
// delivery engine owning the archive writer registry, and the
// data directory the archive files live in.
type sweeperEnv struct {
sweeper *delivery.ArchiveSweeper
eng *delivery.Engine
mainDB *database.Database
dataDir string
}
func setupSweeperTest(t *testing.T) *sweeperEnv {
t.Helper()
dataDir := t.TempDir()
log := archiveTestLogger()
sqlDB, err := sql.Open(
"sqlite",
fmt.Sprintf(
"file:%s?mode=rwc",
filepath.Join(dataDir, "main.db"),
),
)
require.NoError(t, err)
t.Cleanup(func() { _ = sqlDB.Close() })
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
require.NoError(t, err)
mainDB := database.NewTestDatabase(gdb)
require.NoError(t, mainDB.Migrate())
eng := delivery.NewTestEngineWithDB(
mainDB,
database.NewTestWebhookDBManager(dataDir),
log,
&http.Client{Timeout: 5 * time.Second},
1,
)
return &sweeperEnv{
sweeper: delivery.NewTestArchiveSweeper(
mainDB, eng, log,
),
eng: eng,
mainDB: mainDB,
dataDir: dataDir,
}
}
// archivePath returns where the engine keeps a webhook's
// archive file.
func (env *sweeperEnv) archivePath(webhookID string) string {
return filepath.Join(
env.dataDir, fmt.Sprintf("archive-%s.db", webhookID),
)
}
// seedDatabaseTarget creates a webhook with one database target
// carrying the given target config JSON, and returns the
// webhook id.
func (env *sweeperEnv) seedDatabaseTarget(
t *testing.T, configJSON string,
) string {
t.Helper()
wh := &database.Webhook{
UserID: uuid.New().String(),
Name: "sweep-test",
}
require.NoError(
t,
env.mainDB.DB().
Omit(clause.Associations).
Create(wh).Error,
)
tgt := &database.Target{
WebhookID: wh.ID,
Name: "archive",
Type: database.TargetTypeDatabase,
Active: true,
Config: configJSON,
}
require.NoError(
t,
env.mainDB.DB().
Omit(clause.Associations).
Create(tgt).Error,
)
return wh.ID
}
// seedArchiveRows creates the archive file for a webhook and
// inserts one row per supplied archived-at timestamp, returning
// the archive path. The handle is closed before returning, so
// the archive is idle exactly as it would be with no traffic.
func (env *sweeperEnv) seedArchiveRows(
t *testing.T, webhookID string, archivedAt ...time.Time,
) string {
t.Helper()
path := env.archivePath(webhookID)
sqlDB, err := sql.Open(
"sqlite", fmt.Sprintf("file:%s?mode=rwc", path),
)
require.NoError(t, err)
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
require.NoError(t, err)
require.NoError(
t, gdb.AutoMigrate(&delivery.ExportArchivedEvent{}),
)
for i, at := range archivedAt {
row := delivery.ExportArchivedEvent{
EventID: fmt.Sprintf("ev-%d", i),
WebhookID: webhookID,
Method: http.MethodPost,
Body: `{"seeded":true}`,
ArchivedAt: at,
}
require.NoError(t, gdb.Create(&row).Error)
}
require.NoError(t, sqlDB.Close())
return path
}
// archivedEventIDs returns the event ids currently stored in an
// archive file, read through a separate read-only handle.
func archivedEventIDs(
t *testing.T, path string,
) []string {
t.Helper()
var rows []delivery.ExportArchivedEvent
rdb := openArchiveDBForRead(t, path)
require.NoError(t, rdb.Order("event_id").Find(&rows).Error)
ids := make([]string, 0, len(rows))
for i := range rows {
ids = append(ids, rows[i].EventID)
}
return ids
}
// countArchivedRows counts the rows in an archive file without
// asserting anything, so it is safe to poll from an
// assert.Eventually condition (which runs off the test
// goroutine, where testify assertions must not be used).
func countArchivedRows(path string) (int64, error) {
sqlDB, err := sql.Open(
"sqlite", fmt.Sprintf("file:%s?mode=ro", path),
)
if err != nil {
return 0, err
}
defer func() { _ = sqlDB.Close() }()
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
if err != nil {
return 0, err
}
var count int64
err = gdb.Model(&delivery.ExportArchivedEvent{}).
Count(&count).Error
if err != nil {
return 0, err
}
return count, nil
}
// captureLifecycle is a minimal fx.Lifecycle that records the
// hooks a component registers, so a test can invoke the real
// OnStart/OnStop functions with a context of its choosing.
type captureLifecycle struct {
hooks []fx.Hook
}
func (l *captureLifecycle) Append(h fx.Hook) {
l.hooks = append(l.hooks, h)
}
// TestArchiveSweeper_LoopOutlivesStartHookContext is the
// regression test for a sweeper that never swept. fx calls
// OnStart with a context carrying the application's start
// timeout (15 seconds by default), so a background loop whose
// context is derived from it is cancelled 15 seconds into the
// process — three quarters of an hour before the first tick
// under the default one-hour sweep interval.
//
// The hook context here is already cancelled, which is the same
// defect taken to its limit: a loop that inherits it never runs
// a single tick, while a correctly rooted loop keeps sweeping
// for as long as the process lives. Handing the hook a plain
// context.Background() would assert nothing at all.
func TestArchiveSweeper_LoopOutlivesStartHookContext(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
now := time.Now()
path := env.seedArchiveRows(
t, webhookID,
now.Add(-48*time.Hour),
now.Add(-time.Minute),
)
env.sweeper.ExportSetInterval(10 * time.Millisecond)
// Drive the genuine fx hooks the application registers,
// rather than a test-only entry point.
lc := &captureLifecycle{}
env.sweeper.ExportRegisterHooks(lc)
require.Len(t, lc.hooks, 1)
hookCtx, cancel := context.WithCancel(context.Background())
cancel()
require.NoError(t, lc.hooks[0].OnStart(hookCtx))
t.Cleanup(func() {
_ = lc.hooks[0].OnStop(context.Background())
})
assert.Eventually(
t,
func() bool {
count, err := countArchivedRows(path)
return err == nil && count == 1
},
5*time.Second,
10*time.Millisecond,
"the sweep loop must keep running after the start "+
"hook's context is done; it pruned nothing, so it "+
"inherited the hook context and died",
)
}
// TestArchiveSweep_DoesNotResurrectEvictedWriter covers the
// interleaving where a sweep tick has already listed a webhook's
// target when the webhook is deleted and its writer evicted. The
// sweep must not put a writer back into the registry: nothing
// would ever evict it again, which is precisely the leak this
// change exists to close.
func TestArchiveSweep_DoesNotResurrectEvictedWriter(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
// Prime the registry the way a delivery would, then evict as
// the deletion path does. The target row is deliberately left
// in place: this is the tick that listed the webhook before
// the deletion committed.
_, err := env.eng.ExportEnsureArchiveWriter(webhookID)
require.NoError(t, err)
env.eng.EvictWebhook(webhookID)
require.False(t, env.eng.ExportHasArchiveWriter(webhookID))
env.sweeper.ExportSweep(context.Background())
assert.False(
t, env.eng.ExportHasArchiveWriter(webhookID),
"a sweep must never re-register a writer for a webhook "+
"whose registry entry has already been released",
)
}
// TestArchiveSweep_LeavesNoRegistryEntry states the same
// invariant in its general form: sweeping an archive whose
// webhook has no cached writer must not leave one behind, so the
// registry keeps holding only writers a delivery created and an
// eviction can reach.
func TestArchiveSweep_LeavesNoRegistryEntry(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path := env.seedArchiveRows(
t, webhookID,
time.Now().Add(-48*time.Hour),
time.Now().Add(-time.Minute),
)
require.False(t, env.eng.ExportHasArchiveWriter(webhookID))
env.sweeper.ExportSweep(context.Background())
assert.Equal(
t, []string{sweepRowNew}, archivedEventIDs(t, path),
"the sweep must still prune an idle archive",
)
assert.False(
t, env.eng.ExportHasArchiveWriter(webhookID),
"the sweep must release the registry entry it created",
)
}
// TestArchiveSweep_KeepsWriterAdoptedByDelivery is the other
// half of that invariant: an entry the sweep created but a
// delivery then claimed belongs to the registry and must survive
// the sweep, or the delivery would be left holding a detached
// writer with an open handle that no eviction can reach.
func TestArchiveSweep_KeepsWriterAdoptedByDelivery(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"n":1}`)
event.WebhookID = webhookID
d := seedDatabaseTargetDelivery(
t, webhookDB, event, `{"expiry":"1h"}`,
)
env.sweeper.ExportSweep(context.Background())
require.False(t, env.eng.ExportHasArchiveWriter(webhookID))
env.eng.ExportDeliverDatabase(webhookDB, d)
assert.True(
t, env.eng.ExportHasArchiveWriter(webhookID),
"a delivery's writer must stay registered",
)
env.sweeper.ExportSweep(context.Background())
assert.True(
t, env.eng.ExportHasArchiveWriter(webhookID),
"a sweep must not drop a writer a delivery owns",
)
}
// TestArchiveSweep_KeepsWriterAdoptedDuringSweep covers the one
// interleaving the sweepOwned flag exists for, which
// TestArchiveSweep_KeepsWriterAdoptedByDelivery cannot reach: a
// delivery adopting the sweep's own entry WHILE that sweep is
// still running.
//
// The registry operations are driven directly, in the order the
// sweep and a concurrent delivery perform them, so the window is
// exercised deterministically rather than hoped for:
//
// 1. the sweep finds no cached writer and registers one of its
// own, marked sweep-owned;
// 2. a delivery arrives, is handed that very writer, clears the
// flag and opens the archive handle;
// 3. the sweep finishes and releases what it created.
//
// Step 3 must leave the entry alone. Dropping it would detach a
// writer that is holding an open archive handle inside its
// debounce window, and no eviction could ever reach it again —
// exactly the process-lifetime handle leak this change exists to
// close. The eviction at the end proves the entry is still
// reachable.
func TestArchiveSweep_KeepsWriterAdoptedDuringSweep(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
sweepWriter, created, err := env.eng.ExportSweepWriterFor(
webhookID,
)
require.NoError(t, err)
require.True(
t, created,
"the sweep must have created the registry entry itself",
)
// The delivery lands mid-sweep and adopts the entry.
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"n":1}`)
event.WebhookID = webhookID
d := seedDatabaseTargetDelivery(
t, webhookDB, event, `{"expiry":"1h"}`,
)
env.eng.ExportDeliverDatabase(webhookDB, d)
adopted := env.eng.ExportArchiveWriterFor(webhookID)
require.NotNil(t, adopted)
require.True(
t, sweepWriter.Same(adopted),
"the delivery must have adopted the sweep's writer",
)
require.True(
t, env.eng.ExportArchiveHandleOpen(webhookID),
"the delivery leaves the archive handle open",
)
// The sweep finishes.
env.eng.ExportReleaseSweepWriter(webhookID, sweepWriter)
require.True(
t, env.eng.ExportHasArchiveWriter(webhookID),
"a writer adopted by a delivery during a sweep must "+
"stay registered, or its open handle is unreachable",
)
env.eng.EvictWebhook(webhookID)
assert.False(
t, env.eng.ExportHasArchiveWriter(webhookID),
"the adopted writer must still be evictable",
)
assert.False(
t, sweepWriter.HandleOpen(),
"eviction must have closed the adopted writer's handle",
)
}
// TestArchiveSweep_ContinuesAfterPerWebhookFailure proves a
// failure for one webhook does not abort the sweep for the
// others: an unparseable expiry and an unreadable archive both
// have to be logged and stepped over.
func TestArchiveSweep_ContinuesAfterPerWebhookFailure(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
// Seeded first so the sweep reaches them before the healthy
// webhook: targets come back in insertion order.
badConfigID := env.seedDatabaseTarget(t, `{"expiry":"!!!"}`)
env.seedArchiveRows(
t, badConfigID, time.Now().Add(-48*time.Hour),
)
corruptID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
require.NoError(t, os.WriteFile(
env.archivePath(corruptID),
[]byte("this is not a sqlite database"),
0o600,
))
healthyID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
healthyPath := env.seedArchiveRows(
t, healthyID,
time.Now().Add(-48*time.Hour),
time.Now().Add(-time.Minute),
)
env.sweeper.ExportSweep(context.Background())
assert.Equal(
t, []string{sweepRowNew},
archivedEventIDs(t, healthyPath),
"a failure for an earlier webhook must not stop the "+
"sweep from pruning the ones after it",
)
}
// TestArchiveSweep_OpenExistingDoesNotCreateFile pins the second
// of the two no-create guards. The first is the stat in
// sweepWebhook; this one is the SQLite open mode, which is what
// protects the window between that stat and the open. Flipping
// the sweep's mode to create-if-missing makes this fail.
func TestArchiveSweep_OpenExistingDoesNotCreateFile(
t *testing.T,
) {
t.Parallel()
dir := t.TempDir()
path := filepath.Join(dir, "archive-absent.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
err := w.OpenExisting(time.Hour)
require.Error(
t, err,
"opening a missing archive without create permission "+
"must fail rather than conjure the file",
)
for _, suffix := range archiveFileSuffixes() {
assert.NoFileExists(t, path+suffix)
}
}
// TestArchiveSweep_PrunesIdleArchive is the core regression
// test for this issue: an archive that receives no further
// writes must still lose its expired rows. Before the sweeper
// existed, pruning only ever ran on a write-triggered reopen,
// so an idle archive kept expired rows forever.
func TestArchiveSweep_PrunesIdleArchive(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
now := time.Now()
path := env.seedArchiveRows(
t, webhookID,
now.Add(-48*time.Hour),
now.Add(-time.Minute),
)
require.Equal(
t, []string{sweepRowOld, sweepRowNew},
archivedEventIDs(t, path),
)
env.sweeper.ExportSweep(context.Background())
assert.Equal(
t, []string{sweepRowNew}, archivedEventIDs(t, path),
"the sweep should prune rows older than the expiry "+
"from an idle archive and keep the rest",
)
}
// TestArchiveSweep_LeavesArchiveClosed proves the sweep does
// not hold the archive open afterwards, so an operator can
// still move the file away for offline retention.
//
// The assertion is made on a writer the test holds a reference
// to, and the handle is proven OPEN before the sweep runs, so the
// test observes the sweep closing it rather than a writer that
// merely never opened anything. Asking the registry instead would
// be vacuous here: the sweep releases an entry it created, and a
// missing entry reports "not open" whether or not anything was
// closed.
func TestArchiveSweep_LeavesArchiveClosed(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path := env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.OpenExisting(time.Hour))
require.True(
t, w.HandleOpen(),
"the writer must hold an open handle before the sweep",
)
require.NoError(t, w.SweepExpired(time.Hour))
assert.False(
t, w.HandleOpen(),
"an idle archive must end the sweep closed",
)
}
// TestArchiveSweep_ClosesHandleOfRegisteredWriter states the same
// guarantee end to end, through the real sweeper and a writer the
// registry keeps.
//
// The delivery leaves the archive handle open inside its debounce
// window and makes the entry delivery-owned, so the sweep finds a
// cached writer (created is false, nothing is released) and the
// registry query afterwards is answered by a writer that really
// exists. A handle left open here would be doubly wrong: it also
// blocks the operator's move-the-file-away workflow.
func TestArchiveSweep_ClosesHandleOfRegisteredWriter(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"n":1}`)
event.WebhookID = webhookID
d := seedDatabaseTargetDelivery(
t, webhookDB, event, `{"expiry":"1h"}`,
)
env.eng.ExportDeliverDatabase(webhookDB, d)
require.True(
t, env.eng.ExportArchiveHandleOpen(webhookID),
"the delivery must leave the archive handle open",
)
env.sweeper.ExportSweep(context.Background())
require.True(
t, env.eng.ExportHasArchiveWriter(webhookID),
"the delivery's registry entry must survive the sweep",
)
assert.False(
t, env.eng.ExportArchiveHandleOpen(webhookID),
"the sweep must leave the archive closed",
)
}
// TestArchiveSweep_NeverExpiryUntouched proves the sweep is a
// no-op for the default retention policy, so archives with no
// expiry (or the literal "never") behave exactly as before.
func TestArchiveSweep_NeverExpiryUntouched(t *testing.T) {
t.Parallel()
for _, configJSON := range []string{
`{"expiry":"never"}`,
`{"expiry":""}`,
"",
} {
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, configJSON)
path := env.seedArchiveRows(
t, webhookID,
time.Now().Add(-10000*time.Hour),
)
env.sweeper.ExportSweep(context.Background())
assert.Equal(
t, []string{sweepRowOld}, archivedEventIDs(t, path),
"config %q must keep rows forever", configJSON,
)
assert.False(
t, env.eng.ExportHasArchiveWriter(webhookID),
"config %q must leave no registry entry behind",
configJSON,
)
}
}
// TestArchiveSweep_NeverExpirySkipsBeforeOpening pins the
// expiry <= 0 boundary in sweepTarget, which the row assertions
// above cannot reach: pruning is separately gated on a positive
// expiry, so a "never" archive keeps its rows even if the sweep
// does open it.
//
// The spec is stronger than that — a "never" archive is skipped
// before any file is touched — so the archive here exists but has
// never been migrated. Opening it at all would run AutoMigrate
// and create the archive table, which is exactly what must not
// happen.
func TestArchiveSweep_NeverExpirySkipsBeforeOpening(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"never"}`)
path := env.archivePath(webhookID)
seedUnmigratedArchive(t, path)
require.False(t, archiveTableExists(t, path))
env.sweeper.ExportSweep(context.Background())
assert.False(
t, archiveTableExists(t, path),
"a never-expiry archive must not be opened at all",
)
}
// seedUnmigratedArchive creates an archive file that exists but
// carries no archive schema, so any open of it is observable: the
// archive table appears only if something ran AutoMigrate.
func seedUnmigratedArchive(t *testing.T, path string) {
t.Helper()
sqlDB, err := sql.Open(
"sqlite", fmt.Sprintf("file:%s?mode=rwc", path),
)
require.NoError(t, err)
_, err = sqlDB.ExecContext(
t.Context(), "CREATE TABLE placeholder (id INTEGER)",
)
require.NoError(t, err)
require.NoError(t, sqlDB.Close())
}
// archiveTableExists reports whether an archive file has had the
// archive schema migrated into it.
func archiveTableExists(t *testing.T, path string) bool {
t.Helper()
return openArchiveDBForRead(t, path).
Migrator().
HasTable(&delivery.ExportArchivedEvent{})
}
// TestArchiveSweep_DoesNotCreateArchiveFile proves the sweep
// never conjures an archive: a webhook with a database target
// that has never received an event must still have no archive
// file (nor SQLite sidecar) after a sweep.
func TestArchiveSweep_DoesNotCreateArchiveFile(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path := env.archivePath(webhookID)
require.NoFileExists(t, path)
env.sweeper.ExportSweep(context.Background())
for _, suffix := range archiveFileSuffixes() {
assert.NoFileExists(
t, path+suffix,
"the sweep must not create an archive file",
)
}
}
// TestArchiveSweep_DoesNotCreateAfterWriterExists covers the
// same guarantee once a writer is cached in the registry but
// the file itself is still absent (for instance because the
// operator moved the archive away).
func TestArchiveSweep_DoesNotCreateAfterWriterExists(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path, err := env.eng.ExportEnsureArchiveWriter(webhookID)
require.NoError(t, err)
require.NoFileExists(t, path)
env.sweeper.ExportSweep(context.Background())
assert.NoFileExists(t, path)
}
// TestArchiveSweep_SkipsDeletedWebhookTargets proves that the
// sweep ignores targets soft-deleted along with their webhook,
// so a deleted webhook's archive is never reopened.
func TestArchiveSweep_SkipsDeletedWebhookTargets(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path := env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
require.NoError(
t,
env.mainDB.DB().
Where("webhook_id = ?", webhookID).
Delete(&database.Target{}).Error,
)
env.sweeper.ExportSweep(context.Background())
assert.Equal(
t, []string{sweepRowOld}, archivedEventIDs(t, path),
"a deleted target's archive must be left alone",
)
}
// TestArchiveSweep_ConcurrentWrites proves the sweep serialises
// against writes through the per-webhook writer mutex. Run
// under -race, an unsynchronised sweep would be caught here.
func TestArchiveSweep_ConcurrentWrites(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
webhookDB := testWebhookDB(t)
// The deliveries are seeded up front, on the test's own
// goroutine: the seed helpers assert, and testify assertions
// must not run off the test goroutine.
deliveries := make(
[]*database.Delivery, 0, sweepConcurrentWrites,
)
for range sweepConcurrentWrites {
event := seedEvent(t, webhookDB, `{"n":1}`)
event.WebhookID = webhookID
deliveries = append(
deliveries,
seedDatabaseTargetDelivery(
t, webhookDB, event, `{"expiry":"1h"}`,
),
)
}
var wg sync.WaitGroup
wg.Add(2)
go func() {
defer wg.Done()
for _, d := range deliveries {
env.eng.ExportDeliverDatabase(webhookDB, d)
}
}()
go func() {
defer wg.Done()
for range sweepConcurrentWrites {
env.sweeper.ExportSweep(context.Background())
}
}()
wg.Wait()
assert.FileExists(t, env.archivePath(webhookID))
}
// TestArchiveSweeper_StopsCleanly proves the background loop
// exits on OnStop rather than leaking a goroutine.
func TestArchiveSweeper_StopsCleanly(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
env.sweeper.ExportSetInterval(time.Millisecond)
env.sweeper.ExportStart()
// stop blocks on the loop's WaitGroup, so returning at all
// proves the loop observed the cancellation and exited.
env.sweeper.ExportStop()
}

948
internal/delivery/engine.go
View File

File diff suppressed because it is too large Load Diff

42
internal/delivery/engine_integration_test.go
View File

@@ -126,6 +126,36 @@ func iHTTPConfig(url string) string {
return string(data)
}
func iWebhookDB(t *testing.T) *gorm.DB {
t.Helper()
dbPath := filepath.Join(
t.TempDir(), "events-test.db",
)
dsn := fmt.Sprintf(
"file:%s?cache=shared&mode=rwc", dbPath,
)
sqlDB, err := sql.Open("sqlite", dsn)
require.NoError(t, err)
t.Cleanup(func() { _ = sqlDB.Close() })
db, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
require.NoError(t, err)
require.NoError(t, db.AutoMigrate(
&database.Event{},
&database.Delivery{},
&database.DeliveryResult{},
))
return db
}
func iEngine(
t *testing.T, workers int,
) *delivery.Engine {
@@ -152,10 +182,10 @@ func iSeedEvent(
event := database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Method: "POST",
Headers: `{}`,
Body: body,
ContentType: testContentType,
ContentType: "application/json",
}
require.NoError(t, db.Create(&event).Error)
@@ -905,7 +935,7 @@ func TestDeliverHTTP_CustomTargetHeaders(t *testing.T) {
func TestDeliverHTTP_TargetTimeout(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
db := iWebhookDB(t)
e := iEngine(t, 1)
ts := httptest.NewServer(
@@ -957,10 +987,10 @@ func iSeedEventAndDelivery(
event := database.Event{
WebhookID: uuid.New().String(),
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Method: "POST",
Headers: `{"Content-Type":["application/json"]}`,
Body: body,
ContentType: testContentType,
ContentType: "application/json",
}
require.NoError(t, db.Create(&event).Error)
@@ -1037,7 +1067,7 @@ func iAssertResultFailed(
func TestDeliverHTTP_InvalidConfig(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
db := iWebhookDB(t)
e := iEngine(t, 1)
event, del := iSeedEventAndDelivery(

335
internal/delivery/engine_test.go
View File

@@ -27,9 +27,6 @@ import (
"sneak.berlin/go/webhooker/internal/delivery"
)
// testContentType is the event content type used in tests.
const testContentType = "application/json"
func testWebhookDB(t *testing.T) *gorm.DB {
t.Helper()
@@ -97,10 +94,10 @@ func seedEvent(
event := database.Event{
WebhookID: uuid.New().String(),
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Method: "POST",
Headers: `{"Content-Type":["application/json"]}`,
Body: body,
ContentType: testContentType,
ContentType: "application/json",
}
require.NoError(t, db.Create(&event).Error)
@@ -345,29 +342,33 @@ func TestDeliverDatabase_ImmediateSuccess(
t.Parallel()
db := testWebhookDB(t)
// The database target archives for real now, so the engine
// needs a webhook DB manager to locate the data directory.
e := delivery.NewTestEngineWithDB(
nil,
database.NewTestWebhookDBManager(t.TempDir()),
slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
&http.Client{Timeout: 5 * time.Second},
1,
)
e := testEngine(t, 1)
event := seedEvent(t, db, `{"db":"target"}`)
d := seedDatabaseTargetDelivery(t, db, event, "")
dlv := seedDelivery(
t, db, event.ID, uuid.New().String(),
database.DeliveryStatusPending,
)
d := &database.Delivery{
EventID: event.ID,
TargetID: dlv.TargetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: database.Target{
Name: "test-db",
Type: database.TargetTypeDatabase,
},
}
d.ID = dlv.ID
e.ExportDeliverDatabase(db, d)
var updated database.Delivery
require.NoError(t, db.First(
&updated, "id = ?", d.ID,
&updated, "id = ?", dlv.ID,
).Error)
assert.Equal(t,
@@ -378,7 +379,7 @@ func TestDeliverDatabase_ImmediateSuccess(
var result database.DeliveryResult
require.NoError(t, db.Where(
"delivery_id = ?", d.ID,
"delivery_id = ?", dlv.ID,
).First(&result).Error)
assert.True(t, result.Success)
@@ -435,6 +436,91 @@ func TestDeliverLog_ImmediateSuccess(t *testing.T) {
assert.True(t, result.Success)
}
func TestDeliverLog_StructuredLogFields(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
var logBuf bytes.Buffer
e := delivery.NewTestEngine(
slog.New(slog.NewTextHandler(
&logBuf,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
&http.Client{Timeout: 5 * time.Second},
1,
)
event := seedEvent(t, db, `{"log":"structured"}`)
dlv := seedDelivery(
t, db, event.ID, uuid.New().String(),
database.DeliveryStatusPending,
)
d := &database.Delivery{
EventID: event.ID,
TargetID: dlv.TargetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: database.Target{
Name: "structured-log",
Type: database.TargetTypeLog,
},
}
d.ID = dlv.ID
e.ExportDeliverLog(db, d)
// The delivery is marked delivered and a success
// DeliveryResult with no HTTP status is recorded,
// mirroring the other target types' bookkeeping.
var updated database.Delivery
require.NoError(t, db.First(
&updated, "id = ?", dlv.ID,
).Error)
assert.Equal(t,
database.DeliveryStatusDelivered, updated.Status,
"log target should immediately succeed",
)
var result database.DeliveryResult
require.NoError(t, db.Where(
"delivery_id = ?", dlv.ID,
).First(&result).Error)
assert.True(t, result.Success)
assert.Equal(t, 0, result.StatusCode,
"log target should not have an HTTP status",
)
assertLogFields(t, logBuf.String(), event, "structured-log")
}
// assertLogFields checks that a log target's structured
// log line carries the required fields: event id,
// webhook/entrypoint, target name, and outcome.
func assertLogFields(
t *testing.T,
logged string,
event database.Event,
targetName string,
) {
t.Helper()
assert.Contains(t, logged, "event_id="+event.ID)
assert.Contains(t, logged, "webhook_id="+event.WebhookID)
assert.Contains(t,
logged, "entrypoint_id="+event.EntrypointID,
)
assert.Contains(t, logged, "target_name="+targetName)
assert.Contains(t, logged, "outcome=delivered")
}
func TestDeliverHTTP_WithRetries_Success(t *testing.T) {
t.Parallel()
@@ -1116,10 +1202,10 @@ func TestDoHTTPRequest_ForwardsHeaders(t *testing.T) {
}
event := &database.Event{
Method: http.MethodPost,
Method: "POST",
Headers: `{"X-Custom":["value1"],"Content-Type":["application/json"]}`,
Body: `{"test":true}`,
ContentType: testContentType,
ContentType: "application/json",
}
statusCode, _, _, err := e.ExportDoHTTPRequest(
@@ -1141,7 +1227,7 @@ func TestDoHTTPRequest_ForwardsHeaders(t *testing.T) {
)
assert.Equal(t,
testContentType,
"application/json",
receivedHeaders.Get("Content-Type"),
)
@@ -1157,19 +1243,7 @@ func TestProcessDelivery_RoutesToCorrectHandler(
t.Parallel()
db := testWebhookDB(t)
// The database target archives for real now, so the engine
// needs a webhook DB manager to locate the data directory.
e := delivery.NewTestEngineWithDB(
nil,
database.NewTestWebhookDBManager(t.TempDir()),
slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
&http.Client{Timeout: 5 * time.Second},
1,
)
e := testEngine(t, 1)
tests := []struct {
name string
@@ -1300,8 +1374,8 @@ func TestFormatSlackMessage_JSONBody(t *testing.T) {
t.Parallel()
event := &database.Event{
Method: http.MethodPost,
ContentType: testContentType,
Method: "POST",
ContentType: "application/json",
Body: `{"action":"push",` +
`"repo":"test/repo",` +
`"ref":"refs/heads/main"}`,
@@ -1326,7 +1400,7 @@ func TestFormatSlackMessage_NonJSONBody(t *testing.T) {
t.Parallel()
event := &database.Event{
Method: http.MethodPost,
Method: "POST",
ContentType: "text/plain",
Body: "hello world plain text",
}
@@ -1349,8 +1423,8 @@ func TestFormatSlackMessage_EmptyBody(t *testing.T) {
t.Parallel()
event := &database.Event{
Method: http.MethodPost,
ContentType: testContentType,
Method: "POST",
ContentType: "application/json",
Body: "",
}
event.CreatedAt = time.Date(
@@ -1378,8 +1452,8 @@ func TestFormatSlackMessage_LargeJSONTruncated(
require.NoError(t, err)
event := &database.Event{
Method: http.MethodPost,
ContentType: testContentType,
Method: "POST",
ContentType: "application/json",
Body: string(largeJSON),
}
event.CreatedAt = time.Date(
@@ -1664,179 +1738,6 @@ func TestProcessDelivery_RoutesToSlack(t *testing.T) {
)
}
// newLogCaptureEngine builds a test engine whose logger
// writes to the returned buffer, for inspecting log output.
func newLogCaptureEngine(
t *testing.T,
) (*delivery.Engine, *bytes.Buffer) {
t.Helper()
var buf bytes.Buffer
log := slog.New(slog.NewTextHandler(
&buf,
&slog.HandlerOptions{Level: slog.LevelDebug},
))
e := delivery.NewTestEngine(
log, &http.Client{Timeout: 5 * time.Second}, 1,
)
return e, &buf
}
// assertLogLineComplete asserts the captured log output
// carries the full inbound webhook content and ids.
func assertLogLineComplete(
t *testing.T, out string, event database.Event,
) {
t.Helper()
assert.Contains(t, out, "log-body-marker",
"log line must contain the full request body",
)
assert.Contains(t, out, "Content-Type",
"log line must contain the full request headers",
)
assert.Contains(t, out, event.EntrypointID,
"log line must contain the entrypoint id",
)
assert.Contains(t, out, event.WebhookID,
"log line must contain the webhook id",
)
assert.Contains(t, out, testContentType,
"log line must contain the content type",
)
}
func TestDeliverLog_LogsFullContent(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
e, buf := newLogCaptureEngine(t)
event := seedEvent(
t, db, `{"log-body-marker":"abc123"}`,
)
dlv := seedDelivery(
t, db, event.ID, uuid.New().String(),
database.DeliveryStatusPending,
)
d := &database.Delivery{
EventID: event.ID,
TargetID: dlv.TargetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: database.Target{
Name: "test-log-full",
Type: database.TargetTypeLog,
},
}
d.ID = dlv.ID
e.ExportDeliverLog(db, d)
assertLogLineComplete(t, buf.String(), event)
assertDeliveryStatus(t, db, dlv.ID,
database.DeliveryStatusDelivered,
)
}
// buildSlackRetryDelivery builds a Slack delivery whose
// target is configured with retries enabled.
func buildSlackRetryDelivery(
dlv database.Delivery,
event database.Event,
targetID, cfg string,
) *database.Delivery {
d := &database.Delivery{
EventID: event.ID,
TargetID: targetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: database.Target{
Name: "test-slack-retry",
Type: database.TargetTypeSlack,
Config: cfg,
MaxRetries: 5,
},
}
d.ID = dlv.ID
return d
}
func TestDeliverSlack_WithRetries_SchedulesRetry(
t *testing.T,
) {
t.Parallel()
db := testWebhookDB(t)
ts := newStatusServer(t, http.StatusServiceUnavailable)
e := testEngine(t, 1)
targetID := uuid.New().String()
slackCfg, err := json.Marshal(
delivery.SlackTargetConfig{WebhookURL: ts.URL},
)
require.NoError(t, err)
event := seedEvent(t, db, `{"slack":"retry"}`)
dlv := seedDelivery(
t, db, event.ID, targetID,
database.DeliveryStatusPending,
)
d := buildSlackRetryDelivery(
dlv, event, targetID, string(slackCfg),
)
task := &delivery.Task{
DeliveryID: dlv.ID,
TargetID: targetID,
TargetType: database.TargetTypeSlack,
MaxRetries: 5,
AttemptNum: 1,
}
e.ExportProcessDelivery(context.TODO(), db, d, task)
assertDeliveryStatus(t, db, dlv.ID,
database.DeliveryStatusRetrying,
)
assertDeliveryResult(
t, db, dlv.ID, false,
http.StatusServiceUnavailable,
)
}
// newStatusServer starts a test server that always responds
// with the given status code.
func newStatusServer(
t *testing.T, code int,
) *httptest.Server {
t.Helper()
ts := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(code)
},
))
t.Cleanup(ts.Close)
return ts
}
// readAll is a small helper to avoid importing io in
// a test handler inline.
func readAll(r interface {

318
internal/delivery/export_test.go
View File

@@ -7,17 +7,10 @@ import (
"net/http"
"time"
"go.uber.org/fx"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// ErrExportArchiveWriterEvicted exposes the sentinel returned by
// an evicted archive writer. It carries the Err prefix rather
// than this file's usual Export one because it is a sentinel
// error.
var ErrExportArchiveWriterEvicted = errArchiveWriterEvicted
// Exported constants for test access.
const (
ExportDeliveryChannelSize = deliveryChannelSize
@@ -46,50 +39,37 @@ func ExportTruncate(s string, maxLen int) string {
return truncate(s, maxLen)
}
// ExportDeliverHTTP delivers via the http target for testing.
// ExportDeliverHTTP exposes deliverHTTP for testing.
func (e *Engine) ExportDeliverHTTP(
ctx context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
) {
e.httpTarget.Deliver(ctx, webhookDB, d, task, e)
e.deliverHTTP(ctx, webhookDB, d, task)
}
// ExportDeliverDatabase delivers via the database target.
// ExportDeliverDatabase exposes deliverDatabase.
func (e *Engine) ExportDeliverDatabase(
webhookDB *gorm.DB, d *database.Delivery,
) {
e.targets[database.TargetTypeDatabase].Deliver(
context.Background(), webhookDB, d, &Task{}, e,
)
e.deliverDatabase(webhookDB, d)
}
// ExportDeliverLog delivers via the log target for testing.
// ExportDeliverLog exposes deliverLog for testing.
func (e *Engine) ExportDeliverLog(
webhookDB *gorm.DB, d *database.Delivery,
) {
e.targets[database.TargetTypeLog].Deliver(
context.Background(), webhookDB, d, &Task{}, e,
)
e.deliverLog(webhookDB, d)
}
// ExportDeliverSlack delivers via the slack target for
// testing.
// ExportDeliverSlack exposes deliverSlack for testing.
func (e *Engine) ExportDeliverSlack(
ctx context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
) {
task := &Task{
DeliveryID: d.ID,
TargetID: d.TargetID,
AttemptNum: 1,
}
e.targets[database.TargetTypeSlack].Deliver(
ctx, webhookDB, d, task, e,
)
e.deliverSlack(ctx, webhookDB, d)
}
// ExportProcessNewTask exposes processNewTask.
@@ -116,56 +96,53 @@ func (e *Engine) ExportProcessDelivery(
e.processDelivery(ctx, webhookDB, d, task)
}
// ExportGetCircuitBreaker exposes the http target's
// getCircuitBreaker.
// ExportGetCircuitBreaker exposes getCircuitBreaker.
func (e *Engine) ExportGetCircuitBreaker(
targetID string,
) *CircuitBreaker {
return e.httpTarget.getCircuitBreaker(targetID)
return e.getCircuitBreaker(targetID)
}
// ExportParseHTTPConfig exposes parseHTTPConfig.
func (e *Engine) ExportParseHTTPConfig(
configJSON string,
) (*HTTPTargetConfig, error) {
return parseHTTPConfig(configJSON)
return e.parseHTTPConfig(configJSON)
}
// ExportParseSlackConfig exposes parseSlackConfig.
func (e *Engine) ExportParseSlackConfig(
configJSON string,
) (*SlackTargetConfig, error) {
return parseSlackConfig(configJSON)
return e.parseSlackConfig(configJSON)
}
// ExportDoHTTPRequest exposes the http target's
// doHTTPRequest.
// ExportDoHTTPRequest exposes doHTTPRequest.
func (e *Engine) ExportDoHTTPRequest(
ctx context.Context,
cfg *HTTPTargetConfig,
event *database.Event,
) (int, string, int64, error) {
return e.httpTarget.doHTTPRequest(ctx, cfg, event)
return e.doHTTPRequest(ctx, cfg, event)
}
// ExportClientForConfig exposes the http target's
// clientForConfig.
// ExportClientForConfig exposes clientForConfig.
func (e *Engine) ExportClientForConfig(
cfg *HTTPTargetConfig,
) *http.Client {
return e.httpTarget.clientForConfig(cfg)
return e.clientForConfig(cfg)
}
// ExportClient returns the http target's shared HTTP client.
// ExportClient returns the engine's shared HTTP client.
func (e *Engine) ExportClient() *http.Client {
return e.httpTarget.client
return e.client
}
// ExportScheduleRetry exposes ScheduleRetry.
// ExportScheduleRetry exposes scheduleRetry.
func (e *Engine) ExportScheduleRetry(
task Task, delay time.Duration,
) {
e.ScheduleRetry(task, delay)
e.scheduleRetry(task, delay)
}
// ExportRecoverPendingDeliveries exposes
@@ -222,15 +199,13 @@ func NewTestEngine(
client *http.Client,
workers int,
) *Engine {
e := &Engine{
return &Engine{
log: log,
client: client,
deliveryCh: make(chan Task, deliveryChannelSize),
retryCh: make(chan Task, retryChannelSize),
workers: workers,
}
e.initTargets(client)
return e
}
// NewTestEngineSmallRetry creates an Engine with a tiny
@@ -238,13 +213,10 @@ func NewTestEngine(
func NewTestEngineSmallRetry(
log *slog.Logger,
) *Engine {
e := &Engine{
return &Engine{
log: log,
retryCh: make(chan Task, 1),
}
e.initTargets(nil)
return e
}
// NewTestEngineWithDB creates an Engine with a real
@@ -256,17 +228,15 @@ func NewTestEngineWithDB(
client *http.Client,
workers int,
) *Engine {
e := &Engine{
return &Engine{
database: db,
dbManager: dbMgr,
log: log,
client: client,
deliveryCh: make(chan Task, deliveryChannelSize),
retryCh: make(chan Task, retryChannelSize),
workers: workers,
}
e.initTargets(client)
return e
}
// NewTestCircuitBreaker creates a CircuitBreaker with
@@ -280,241 +250,3 @@ func NewTestCircuitBreaker(
cooldown: cooldown,
}
}
// ExportArchivedEvent aliases the archive row type so black-box
// tests can construct and read archive rows.
type ExportArchivedEvent = archivedEvent
// ExportArchiveWriter wraps an archiveWriter so black-box tests
// can exercise the per-webhook archive file mechanics.
type ExportArchiveWriter struct {
w *archiveWriter
}
// NewExportArchiveWriter builds an archive writer for tests,
// optionally overriding the reopen debounce (a non-positive
// debounce keeps the production default).
func NewExportArchiveWriter(
path string, log *slog.Logger, debounce time.Duration,
) *ExportArchiveWriter {
w := newArchiveWriter(path, log)
if debounce > 0 {
w.debounce = debounce
}
return &ExportArchiveWriter{w: w}
}
// Write archives a row through the writer.
func (e *ExportArchiveWriter) Write(
row ExportArchivedEvent, expiry time.Duration,
) error {
return e.w.write(row, expiry)
}
// Open opens the archive file, pruning when expiry is positive.
func (e *ExportArchiveWriter) Open(expiry time.Duration) error {
return e.w.open(expiry)
}
// Reopen closes and reopens the archive file.
func (e *ExportArchiveWriter) Reopen(
expiry time.Duration,
) error {
return e.w.reopen(expiry)
}
// Reopens reports how many times the file has been opened.
func (e *ExportArchiveWriter) Reopens() int {
return e.w.reopens
}
// DB returns the writer's current open handle for row
// inspection in tests.
func (e *ExportArchiveWriter) DB() *gorm.DB {
return e.w.db
}
// Path returns the archive file the writer owns.
func (e *ExportArchiveWriter) Path() string {
return e.w.path
}
// OpenExisting opens the archive without permitting creation,
// the way the idle sweep does.
func (e *ExportArchiveWriter) OpenExisting(
expiry time.Duration,
) error {
return e.w.openMode(archiveModeExisting, expiry)
}
// SweepExpired runs an idle sweep of the archive.
func (e *ExportArchiveWriter) SweepExpired(
expiry time.Duration,
) error {
return e.w.sweepExpired(expiry)
}
// Evict marks the writer evicted and closes its handle, exactly
// as leaving the registry does.
func (e *ExportArchiveWriter) Evict() {
e.w.evict()
}
// HandleOpen reports whether the writer currently holds an open
// archive handle.
func (e *ExportArchiveWriter) HandleOpen() bool {
e.w.mu.Lock()
defer e.w.mu.Unlock()
return e.w.db != nil
}
// Same reports whether both wrappers refer to the very same
// underlying archive writer, so a test can prove a registry entry
// is the writer it was handed rather than a replacement.
func (e *ExportArchiveWriter) Same(
other *ExportArchiveWriter,
) bool {
return other != nil && e.w == other.w
}
// ExportArchiveWriterFor returns the archive writer the registry
// currently caches for a webhook, or nil when none is cached. It
// never creates one, so a test can hold a reference to the very
// writer an eviction is about to detach.
func (e *Engine) ExportArchiveWriterFor(
webhookID string,
) *ExportArchiveWriter {
e.dbTarget.mu.Lock()
defer e.dbTarget.mu.Unlock()
w, ok := e.dbTarget.writers[webhookID]
if !ok {
return nil
}
return &ExportArchiveWriter{w: w}
}
// ExportHasArchiveWriter reports whether the database target
// currently caches an archive writer for a webhook.
func (e *Engine) ExportHasArchiveWriter(
webhookID string,
) bool {
e.dbTarget.mu.Lock()
defer e.dbTarget.mu.Unlock()
_, ok := e.dbTarget.writers[webhookID]
return ok
}
// ExportArchiveHandleOpen reports whether the cached archive
// writer for a webhook holds an open database handle. It
// returns false when no writer is cached.
func (e *Engine) ExportArchiveHandleOpen(
webhookID string,
) bool {
e.dbTarget.mu.Lock()
w, ok := e.dbTarget.writers[webhookID]
e.dbTarget.mu.Unlock()
if !ok {
return false
}
w.mu.Lock()
defer w.mu.Unlock()
return w.db != nil
}
// ExportEnsureArchiveWriter creates (if needed) and returns the
// archive file path of the cached writer for a webhook, so a
// test can prime the registry the way a delivery would.
func (e *Engine) ExportEnsureArchiveWriter(
webhookID string,
) (string, error) {
w, err := e.dbTarget.writerFor(webhookID)
if err != nil {
return "", err
}
return w.path, nil
}
// ExportSweepWriterFor takes a webhook's registry writer exactly
// as the idle sweep does, reporting whether the sweep had to
// create the entry. It lets a test drive the registry through the
// sweep's own entry point instead of choreographing goroutines.
func (e *Engine) ExportSweepWriterFor(
webhookID string,
) (*ExportArchiveWriter, bool, error) {
w, created, err := e.dbTarget.sweepWriterFor(webhookID)
if err != nil {
return nil, false, err
}
return &ExportArchiveWriter{w: w}, created, nil
}
// ExportReleaseSweepWriter releases a sweep-created registry entry
// exactly as a finished sweep does.
func (e *Engine) ExportReleaseSweepWriter(
webhookID string, w *ExportArchiveWriter,
) {
e.dbTarget.releaseSweepWriter(webhookID, w.w)
}
// NewTestArchiveSweeper builds an ArchiveSweeper backed by the
// given main database and engine, without the fx lifecycle.
// Intended for tests.
func NewTestArchiveSweeper(
db *database.Database,
eng *Engine,
log *slog.Logger,
) *ArchiveSweeper {
return &ArchiveSweeper{
db: db,
eng: eng,
log: log,
interval: time.Hour,
}
}
// ExportSweep runs a single archive sweep synchronously for
// tests.
func (s *ArchiveSweeper) ExportSweep(ctx context.Context) {
s.sweep(ctx)
}
// ExportStart starts the sweeper's background loop for tests.
func (s *ArchiveSweeper) ExportStart() {
s.start()
}
// ExportRegisterHooks registers the sweeper's real fx lifecycle
// hooks on a lifecycle supplied by a test, so a test can drive
// the exact OnStart/OnStop functions the application runs and
// hand OnStart the kind of context fx actually supplies.
func (s *ArchiveSweeper) ExportRegisterHooks(lc fx.Lifecycle) {
s.registerHooks(lc)
}
// ExportStop stops the sweeper's background loop for tests.
func (s *ArchiveSweeper) ExportStop() {
s.stop()
}
// ExportSetInterval overrides the sweep interval for tests.
func (s *ArchiveSweeper) ExportSetInterval(d time.Duration) {
s.interval = d
}
// ExportParseArchiveExpiry exposes parseArchiveExpiry.
func ExportParseArchiveExpiry(
configJSON string,
) (time.Duration, error) {
return parseArchiveExpiry(configJSON)
}

104
internal/delivery/target.go
View File

@@ -1,104 +0,0 @@
package delivery
import (
"context"
"net/http"
"time"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// Scheduler re-enqueues a task for a future delivery attempt.
// The engine provides one to each target so a target can own
// its retries durably: it records the attempt, marks the
// delivery retrying, and asks the Scheduler to deliver the
// next attempt after delay — exactly what the engine does for
// its own restart recovery.
type Scheduler interface {
ScheduleRetry(task Task, delay time.Duration)
}
// Target delivers an event to one target type. Each type is
// an implementation. A Target owns its whole delivery: it
// makes the attempt, records the DeliveryResult and updates
// the DeliveryStatus, and — for targets that retry — decides
// whether to retry, computes its own backoff, gates with its
// own circuit breaker, and reschedules via the injected
// Scheduler. Fire-and-forget targets simply record a single
// attempt.
type Target interface {
Deliver(
ctx context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
)
}
// rescheduler is implemented by targets that own durable
// retries. The engine's restart recovery and periodic sweep
// use it to let the target recompute the schedule for an
// orphaned retrying delivery, keeping the retry schedule
// target-owned. Fire-and-forget targets do not implement it
// and their (never-occurring) retrying deliveries are
// skipped.
type rescheduler interface {
// remainingBackoff returns how long to wait before the
// next attempt of a recovered retrying delivery.
remainingBackoff(
webhookDB *gorm.DB,
deliveryID string,
attemptNum int,
) time.Duration
// backoffElapsed reports whether the backoff window for
// the last attempt has already passed, so the periodic
// sweep can re-enqueue the delivery now.
backoffElapsed(
webhookDB *gorm.DB,
deliveryID string,
attemptNum int,
) bool
}
// attemptResult is the outcome of a single delivery attempt,
// as reported by a target's per-attempt function to the
// shared retry core.
type attemptResult struct {
statusCode int
respBody string
duration int64
success bool
errMsg string
}
// initTargets builds the target registry, wiring each target
// to the engine's persistence helpers and giving the HTTP and
// Slack targets the shared SSRF-safe client. It is called by
// both New and the test constructors so the registry is
// always populated.
func (e *Engine) initTargets(client *http.Client) {
httpT := &httpTarget{
httpCore: &httpCore{eng: e},
client: client,
}
slackT := &slackTarget{
httpCore: &httpCore{eng: e},
client: client,
}
dbT := &databaseTarget{eng: e}
e.httpTarget = httpT
e.dbTarget = dbT
e.targets = map[database.TargetType]Target{
database.TargetTypeHTTP: httpT,
database.TargetTypeSlack: slackT,
database.TargetTypeDatabase: dbT,
database.TargetTypeLog: &logTarget{eng: e},
}
}

295
internal/delivery/target_database.go
View File

@@ -1,295 +0,0 @@
package delivery
import (
"context"
"fmt"
"path/filepath"
"sync"
"time"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// databaseTarget is a no-retry target that archives the
// full inbound event into a per-webhook archive SQLite file,
// separate from the per-webhook event database. The event is
// already persisted in the per-webhook event DB by the time
// delivery runs; the database target additionally writes a
// durable long-term copy into archive-{webhookID}.db and then
// records a single attempt whose outcome reflects whether the
// archive write succeeded. See archiveWriter for the
// close/reopen, auto-recreate, and expiry semantics.
type databaseTarget struct {
eng *Engine
mu sync.Mutex
writers map[string]*archiveWriter
}
// Deliver implements Target. It archives the event, then
// records one successful attempt and marks the delivery
// delivered. An archiving error fails the delivery: the
// attempt is recorded as failed with the error and the
// delivery is marked failed, so a target that could not do
// its one job (archiving) never reports success. The target
// does not retry; the event remains durably stored in the
// per-webhook event database.
func (t *databaseTarget) Deliver(
_ context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
_ *Task,
_ Scheduler,
) {
err := t.archive(d)
if err != nil {
t.eng.log.Error(
"failed to archive event to database target",
"delivery_id", d.ID,
"event_id", d.EventID,
"error", err,
)
t.eng.recordResult(
webhookDB, d, 1, false, 0, "",
err.Error(), 0,
)
t.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusFailed,
)
return
}
t.eng.recordResult(
webhookDB, d, 1, true, 0, "", "", 0,
)
t.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusDelivered,
)
}
// archive writes the full event as a row into the webhook's
// archive database, honouring the optional per-target expiry
// parsed from the target config JSON.
func (t *databaseTarget) archive(d *database.Delivery) error {
webhookID := d.Event.WebhookID
if webhookID == "" {
return errArchiveMissingWebhookID
}
expiry, err := parseArchiveExpiry(d.Target.Config)
if err != nil {
return err
}
w, err := t.writerFor(webhookID)
if err != nil {
return err
}
row := archivedEvent{
EventID: d.Event.ID,
WebhookID: webhookID,
EntrypointID: d.Event.EntrypointID,
Method: d.Event.Method,
Headers: d.Event.Headers,
Body: d.Event.Body,
ContentType: d.Event.ContentType,
}
return w.write(row, expiry)
}
// writerFor returns the archiveWriter for a webhook, creating
// and caching it on first use. Each webhook has one writer so
// its close/reopen debounce state is shared across concurrent
// deliveries. The archive file lives beside the per-webhook
// event database in the data directory.
func (t *databaseTarget) writerFor(
webhookID string,
) (*archiveWriter, error) {
path, err := t.archivePath(webhookID)
if err != nil {
return nil, err
}
t.mu.Lock()
defer t.mu.Unlock()
if t.writers == nil {
t.writers = make(map[string]*archiveWriter)
}
w, ok := t.writers[webhookID]
if !ok {
w = newArchiveWriter(path, t.eng.log)
t.writers[webhookID] = w
}
// A delivery claims the entry: even if the idle sweep created
// it moments ago, it now belongs to the registry proper and
// the sweep must leave it in place when it finishes.
w.sweepOwned = false
return w, nil
}
// sweepWriterFor returns the archive writer the idle sweep should
// prune a webhook through, together with whether the sweep itself
// created the registry entry.
//
// The sweep must route its prune through the registered writer so
// the writer's mutex orders it against concurrent writes, but it
// must never leave a registry entry behind: a sweep that ran
// concurrently with the webhook's deletion would otherwise
// re-create an entry that nothing will ever evict again, which is
// exactly the leak eviction exists to prevent. An entry the sweep
// creates is therefore marked sweep-owned and handed back to
// releaseSweepWriter when the sweep is done.
func (t *databaseTarget) sweepWriterFor(
webhookID string,
) (*archiveWriter, bool, error) {
path, err := t.archivePath(webhookID)
if err != nil {
return nil, false, err
}
t.mu.Lock()
defer t.mu.Unlock()
if t.writers == nil {
t.writers = make(map[string]*archiveWriter)
}
w, ok := t.writers[webhookID]
if ok {
return w, false, nil
}
w = newArchiveWriter(path, t.eng.log)
w.sweepOwned = true
t.writers[webhookID] = w
return w, true, nil
}
// releaseSweepWriter drops a registry entry that the idle sweep
// created, so a sweep leaves the registry exactly as it found it.
//
// The entry is removed only if it is still the very writer the
// sweep installed and no delivery has claimed it in the meantime
// (writerFor clears sweepOwned when it hands a writer to the
// write path). Both conditions are evaluated under the registry
// lock, so an eviction that raced the sweep — which removes the
// entry outright — simply finds nothing left to do here, and a
// delivery that adopted the writer keeps a registered, evictable
// one.
func (t *databaseTarget) releaseSweepWriter(
webhookID string, w *archiveWriter,
) {
t.mu.Lock()
defer t.mu.Unlock()
cur, ok := t.writers[webhookID]
if !ok || cur != w || !cur.sweepOwned {
return
}
delete(t.writers, webhookID)
}
// archivePath returns the archive file path for a webhook: it
// lives beside the per-webhook event database in the data
// directory. It does not touch the filesystem.
func (t *databaseTarget) archivePath(
webhookID string,
) (string, error) {
if t.eng.dbManager == nil {
return "", errArchiveNoDataDir
}
dir := filepath.Dir(t.eng.dbManager.DBPath(webhookID))
return filepath.Join(
dir, fmt.Sprintf("archive-%s.db", webhookID),
), nil
}
// evict drops a webhook's archive writer from the registry and
// closes its handle, so a deleted webhook does not leave a
// writer (and an open archive handle within its debounce
// window) alive for the process lifetime.
//
// The map entry is removed under the registry lock, which is
// then released before the handle is closed under the writer's
// own lock: that ordering keeps the registry available to other
// webhooks while an in-flight write on this one drains, and
// closing under the writer's lock means eviction can never race
// a write.
//
// Eviction is idempotent and silent for a webhook with no
// writer, which is the common case: a webhook with no database
// target never creates one. It never deletes the archive file.
func (t *databaseTarget) evict(webhookID string) {
t.mu.Lock()
w, ok := t.writers[webhookID]
if ok {
delete(t.writers, webhookID)
}
t.mu.Unlock()
if !ok {
return
}
w.evict()
t.eng.log.Info(
"evicted archive writer",
"webhook_id", webhookID,
"path", w.path,
)
}
// sweepWebhook prunes one webhook's archive of rows older than
// expiry, without requiring a write. It returns nil (nothing to
// do) when the archive file does not exist, so a sweep never
// creates an archive for a webhook that has a database target
// but has never received an event.
//
// It also never leaves a registry entry behind: an entry it had
// to create to reach the writer's mutex is released again once
// the prune is done, so a sweep racing a webhook deletion cannot
// resurrect the writer the eviction just dropped.
func (t *databaseTarget) sweepWebhook(
webhookID string, expiry time.Duration,
) error {
path, err := t.archivePath(webhookID)
if err != nil {
return err
}
// Check before taking a writer at all: a webhook whose
// archive has never been created gets no writer, no handle,
// and no file.
if !fileExists(path) {
return nil
}
w, created, err := t.sweepWriterFor(webhookID)
if err != nil {
return err
}
if created {
defer t.releaseSweepWriter(webhookID, w)
}
return w.sweepExpired(expiry)
}

431
internal/delivery/target_database_archive.go
View File

@@ -1,431 +0,0 @@
package delivery
import (
"database/sql"
"encoding/json"
"errors"
"fmt"
"log/slog"
"os"
"sync"
"time"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
)
// archiveExpiryNever is the expiry sentinel (and default) that
// disables pruning so archived rows are kept forever.
const archiveExpiryNever = "never"
// archiveReopenDebounce bounds how often an archive file is
// closed and reopened. After each write the handle is closed
// and reopened so an operator can move the file away for
// offline archiving, but never more than once per this window.
const archiveReopenDebounce = time.Second
const (
// archiveModeCreate is the SQLite URI mode used by the write
// path: open the archive file, creating it if missing, so a
// first write (or a write after the operator moved the file
// away) recreates it.
archiveModeCreate = "rwc"
// archiveModeExisting is the SQLite URI mode used by the idle
// sweep: open read-write but never create. A sweep must never
// conjure an empty archive file for a webhook that has a
// database target but has never received an event.
archiveModeExisting = "rw"
)
var (
// errArchiveMissingWebhookID is returned when an event to
// archive has no webhook id to key its archive file on.
errArchiveMissingWebhookID = errors.New(
"cannot archive event without a webhook id",
)
// errArchiveNoDataDir is returned when the database target
// has no webhook database manager and so cannot locate the
// data directory for archive files.
errArchiveNoDataDir = errors.New(
"database target has no data directory",
)
// errArchiveExpiryNotPositive is returned when a
// user-supplied archive expiry parses as a duration but is
// zero or negative; "never" is the way to disable pruning.
errArchiveExpiryNotPositive = errors.New(
"expiry must be a positive duration or \"never\"",
)
// errArchiveWriterEvicted is returned when a writer that has
// been evicted (its webhook was deleted, or its last database
// target was removed) is used again. An evicted writer is no
// longer in the registry, so reopening its file would leak a
// handle nothing owns.
errArchiveWriterEvicted = errors.New(
"archive writer has been evicted",
)
)
// databaseTargetConfig is the optional per-target JSON config
// for a database (archive) target.
type databaseTargetConfig struct {
// Expiry is a Go duration (e.g. "720h") after which
// archived rows are pruned, or "never" (the default) to
// keep them forever.
Expiry string `json:"expiry"`
}
// archivedEvent is one fully captured webhook event stored in a
// per-webhook archive database for long-term retention. It is a
// self-contained copy — independent of the per-webhook event
// database, which may prune events under its own retention.
type archivedEvent struct {
ID uint `gorm:"primaryKey;autoIncrement"`
EventID string `gorm:"index"`
WebhookID string
EntrypointID string
Method string
Headers string
Body string
ContentType string
// ArchivedAt is when the row was archived and is the age
// basis for expiry pruning.
ArchivedAt time.Time `gorm:"index"`
}
// parseArchiveExpiry reads the optional expiry from a database
// target's config JSON. An empty config, an empty expiry, or
// the literal "never" all mean keep forever, returned as a zero
// duration. Any other value must parse as a positive Go
// duration; a set-but-invalid value (unparseable, zero, or
// negative) is an error rather than a silent default, matching
// ValidateArchiveExpiry at target creation.
func parseArchiveExpiry(
configJSON string,
) (time.Duration, error) {
if configJSON == "" {
return 0, nil
}
var cfg databaseTargetConfig
err := json.Unmarshal([]byte(configJSON), &cfg)
if err != nil {
return 0, fmt.Errorf(
"parsing database target config: %w", err,
)
}
if cfg.Expiry == "" || cfg.Expiry == archiveExpiryNever {
return 0, nil
}
dur, err := time.ParseDuration(cfg.Expiry)
if err != nil {
return 0, fmt.Errorf(
"parsing archive expiry %q: %w", cfg.Expiry, err,
)
}
if dur <= 0 {
return 0, fmt.Errorf(
"%w: %q", errArchiveExpiryNotPositive, cfg.Expiry,
)
}
return dur, nil
}
// ValidateArchiveExpiry checks a user-supplied archive expiry
// for a database target at configuration time. Valid values are
// empty, "never" (both meaning keep forever), or a positive Go
// duration such as "720h". Anything else is an error, so a bad
// expiry is rejected when the target is created rather than
// failing every subsequent delivery.
func ValidateArchiveExpiry(expiry string) error {
if expiry == "" || expiry == archiveExpiryNever {
return nil
}
dur, err := time.ParseDuration(expiry)
if err != nil {
return fmt.Errorf(
"expiry must be %q or a Go duration "+
"such as \"720h\": %w",
archiveExpiryNever, err,
)
}
if dur <= 0 {
return fmt.Errorf(
"%w: %q", errArchiveExpiryNotPositive, expiry,
)
}
return nil
}
// archiveWriter owns one per-webhook archive SQLite file. It
// serialises writes, and after each write closes and reopens
// the file (debounced to at most once per debounce window) so
// an operator can move the file away for offline archiving. The
// next write recreates a moved or removed file, because the
// file is opened create-if-missing and its schema is migrated
// on every open.
type archiveWriter struct {
mu sync.Mutex
path string
log *slog.Logger
debounce time.Duration
db *gorm.DB
lastReopen time.Time
reopens int
// evicted marks a writer that has been removed from the
// per-webhook registry. Its handle is closed and it must
// never open the file again: nothing holds it any more, so a
// reopen would leak the handle for the process lifetime.
evicted bool
// sweepOwned marks a registry entry that the idle sweep
// created because no writer was cached for the webhook. The
// sweep removes such an entry again when it is done, so a
// sweep can never leave — or resurrect — a registry entry
// for a webhook that has been deleted. A delivery that adopts
// the writer clears the flag, handing the entry to the
// registry proper.
//
// Unlike every other field here it is guarded by
// databaseTarget.mu, not by this writer's mu: it describes the
// registry entry rather than the file.
sweepOwned bool
}
// newArchiveWriter builds an archiveWriter for a file path with
// the default reopen debounce.
func newArchiveWriter(
path string, log *slog.Logger,
) *archiveWriter {
return &archiveWriter{
path: path,
log: log,
debounce: archiveReopenDebounce,
}
}
// write appends the event as a row, then applies the debounced
// close/reopen. It recreates the archive file if it was moved
// or removed since the last open. A positive expiry prunes rows
// older than it on each (re)open.
func (w *archiveWriter) write(
row archivedEvent, expiry time.Duration,
) error {
w.mu.Lock()
defer w.mu.Unlock()
if w.evicted {
return fmt.Errorf(
"%w: %s", errArchiveWriterEvicted, w.path,
)
}
if w.db == nil || !fileExists(w.path) {
err := w.reopen(expiry)
if err != nil {
return err
}
}
row.ArchivedAt = time.Now()
err := w.db.Create(&row).Error
if err != nil {
return fmt.Errorf(
"archiving event to %s: %w", w.path, err,
)
}
if time.Since(w.lastReopen) >= w.debounce {
return w.reopen(expiry)
}
return nil
}
// open opens (creating if missing) the archive file, migrates
// its schema, records the reopen time, and prunes expired rows
// when expiry is positive.
func (w *archiveWriter) open(expiry time.Duration) error {
return w.openMode(archiveModeCreate, expiry)
}
// openMode opens the archive file with the given SQLite URI
// mode, migrates its schema, records the reopen time, and
// prunes expired rows when expiry is positive. The write path
// passes archiveModeCreate so a missing file is recreated; the
// idle sweep passes archiveModeExisting so a missing file is an
// error rather than a newly conjured empty archive.
func (w *archiveWriter) openMode(
mode string, expiry time.Duration,
) error {
dbURL := fmt.Sprintf("file:%s?mode=%s", w.path, mode)
sqlDB, err := sql.Open("sqlite", dbURL)
if err != nil {
return fmt.Errorf(
"opening archive database %s: %w", w.path, err,
)
}
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
if err != nil {
_ = sqlDB.Close()
return fmt.Errorf(
"connecting to archive database %s: %w",
w.path, err,
)
}
err = gdb.AutoMigrate(&archivedEvent{})
if err != nil {
_ = sqlDB.Close()
return fmt.Errorf(
"migrating archive database %s: %w", w.path, err,
)
}
w.db = gdb
w.lastReopen = time.Now()
w.reopens++
if expiry > 0 {
w.prune(expiry)
}
return nil
}
// reopen closes any open handle and opens the file afresh. The
// fresh open recreates the file if it was moved away.
func (w *archiveWriter) reopen(expiry time.Duration) error {
w.close()
return w.open(expiry)
}
// close closes the underlying handle, if any.
func (w *archiveWriter) close() {
if w.db == nil {
return
}
sqlDB, err := w.db.DB()
if err == nil {
_ = sqlDB.Close()
}
w.db = nil
}
// sweepExpired prunes an archive that may have gone idle, with
// no write to trigger the usual on-reopen prune. It takes the
// writer's own mutex for the whole operation, so a sweep is
// ordered against concurrent writes rather than reaching around
// them to the file.
//
// It never creates the archive file: a missing file is skipped,
// and the reopen uses archiveModeExisting so SQLite itself
// refuses to create one if the file disappears between the
// check and the open.
//
// The archive is left CLOSED afterwards. An idle archive holding
// no handle is what keeps the operator's move-the-file-away
// workflow working; the next write reopens (and recreates) the
// file as it always has.
func (w *archiveWriter) sweepExpired(expiry time.Duration) error {
w.mu.Lock()
defer w.mu.Unlock()
if w.evicted {
return fmt.Errorf(
"%w: %s", errArchiveWriterEvicted, w.path,
)
}
if !fileExists(w.path) {
return nil
}
// Drop any live handle first so the prune runs against a
// freshly opened file, matching the write path's semantics.
w.close()
err := w.openMode(archiveModeExisting, expiry)
if err != nil {
return err
}
w.close()
return nil
}
// evict closes the writer's handle and marks it unusable. It is
// called when the writer leaves the registry, either because the
// webhook was deleted or because its last database target was
// removed. The archive FILE is deliberately left on disk: it is
// long-term storage an operator may still want.
func (w *archiveWriter) evict() {
w.mu.Lock()
defer w.mu.Unlock()
w.evicted = true
w.close()
}
// prune deletes archived rows older than expiry, measured from
// each row's archived time. It runs on every (re)open, so a
// steadily written archive is swept by its own write traffic. An
// archive that goes idle receives no further reopens, which is
// why ArchiveSweeper exists to drive sweepExpired on a timer.
// Failures are logged, not fatal: a prune error must not stop
// archiving.
func (w *archiveWriter) prune(expiry time.Duration) {
cutoff := time.Now().Add(-expiry)
res := w.db.Where("archived_at < ?", cutoff).
Delete(&archivedEvent{})
if res.Error != nil {
w.log.Error(
"failed to prune expired archive rows",
"path", w.path,
"error", res.Error,
)
return
}
if res.RowsAffected > 0 {
w.log.Info(
"pruned expired archive rows",
"path", w.path,
"rows_deleted", res.RowsAffected,
)
}
}
// fileExists reports whether a path currently exists.
func fileExists(path string) bool {
_, err := os.Stat(path)
return err == nil
}

363
internal/delivery/target_database_evict_test.go
View File

@@ -1,363 +0,0 @@
package delivery_test
import (
"errors"
"fmt"
"net/http"
"os"
"path/filepath"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
// evictTestEngine builds an engine backed by a temporary data
// directory and returns it along with that directory.
func evictTestEngine(t *testing.T) (*delivery.Engine, string) {
t.Helper()
dataDir := t.TempDir()
eng := delivery.NewTestEngineWithDB(
nil,
database.NewTestWebhookDBManager(dataDir),
archiveTestLogger(),
&http.Client{Timeout: 5 * time.Second},
1,
)
return eng, dataDir
}
// TestEvictWebhook_ClosesAndRemovesWriter proves that evicting
// a webhook drops its archive writer from the registry and
// closes the open archive handle, rather than leaving both
// alive for the process lifetime.
func TestEvictWebhook_ClosesAndRemovesWriter(t *testing.T) {
t.Parallel()
eng, dataDir := evictTestEngine(t)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":true}`)
d := seedDatabaseTargetDelivery(t, webhookDB, event, "")
eng.ExportDeliverDatabase(webhookDB, d)
webhookID := event.WebhookID
require.True(
t, eng.ExportHasArchiveWriter(webhookID),
"a delivery should have cached an archive writer",
)
require.True(
t, eng.ExportArchiveHandleOpen(webhookID),
"the writer should hold an open handle after a write",
)
eng.EvictWebhook(webhookID)
assert.False(
t, eng.ExportHasArchiveWriter(webhookID),
"eviction should remove the registry entry",
)
assert.False(
t, eng.ExportArchiveHandleOpen(webhookID),
"eviction should close the archive handle",
)
archivePath := filepath.Join(
dataDir, fmt.Sprintf("archive-%s.db", webhookID),
)
assert.FileExists(
t, archivePath,
"eviction must not delete the archive file",
)
}
// TestEvictWebhook_UnknownWebhookIsNoOp proves eviction is safe
// for the common case of a webhook that never had a database
// target, and that repeating it does not panic.
func TestEvictWebhook_UnknownWebhookIsNoOp(t *testing.T) {
t.Parallel()
eng, _ := evictTestEngine(t)
assert.NotPanics(t, func() {
eng.EvictWebhook("no-such-webhook")
eng.EvictWebhook("no-such-webhook")
})
assert.False(
t, eng.ExportHasArchiveWriter("no-such-webhook"),
"eviction must not create a writer",
)
}
// evictTestRow builds an archive row for the eviction tests.
func evictTestRow(eventID string) delivery.ExportArchivedEvent {
return delivery.ExportArchivedEvent{
EventID: eventID,
WebhookID: "wh-evict",
Method: http.MethodPost,
Body: `{"seeded":true}`,
}
}
// TestEvictedWriter_WriteDoesNotReopenFile is the direct test of
// the evicted guard on the write path. A writer that has left
// the registry is held by nobody, so a handle it opened could
// never be closed again: it must refuse the write outright
// rather than recreate the archive behind the registry's back.
//
// The archive file is removed before the eviction, so an
// unguarded write is unmistakable — it recreates the file.
func TestEvictedWriter_WriteDoesNotReopenFile(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-evicted.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.Write(evictTestRow("ev-1"), 0))
require.FileExists(t, path)
// The operator moves the archive away for offline retention,
// which the write path would ordinarily undo on the next
// write by recreating the file.
require.NoError(t, os.Remove(path))
w.Evict()
err := w.Write(evictTestRow("ev-2"), 0)
require.ErrorIs(
t, err, delivery.ErrExportArchiveWriterEvicted,
"an evicted writer must refuse writes",
)
assert.NoFileExists(
t, path,
"an evicted writer must not reopen (or recreate) the "+
"archive file",
)
assert.False(
t, w.HandleOpen(),
"an evicted writer must hold no handle",
)
}
// TestEvictedWriter_SweepDoesNotReopenFile is the same test for
// the sweep path: an idle sweep that reaches a writer already
// evicted underneath it must return the sentinel rather than
// reopen a file nothing owns.
func TestEvictedWriter_SweepDoesNotReopenFile(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-evicted.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.Write(evictTestRow("ev-1"), 0))
require.FileExists(t, path)
w.Evict()
err := w.SweepExpired(time.Hour)
require.ErrorIs(
t, err, delivery.ErrExportArchiveWriterEvicted,
"an evicted writer must refuse an idle sweep",
)
assert.False(
t, w.HandleOpen(),
"a refused sweep must not leave a handle open",
)
}
// racingWrites drives a pack of goroutines writing to one
// archive writer until each is refused, so an eviction on the
// test goroutine has to take the writer's mutex away from writes
// that are already contending for it.
type racingWrites struct {
wg sync.WaitGroup
mu sync.Mutex
sawEvicted bool
otherErr error
started chan struct{}
}
// racingWriteGoroutines is how many goroutines contend for the
// writer's mutex while the eviction lands.
const racingWriteGoroutines = 4
// startRacingWrites launches the writing goroutines. Each writes
// in a loop and stops at its first error, recording whether that
// error was the eviction sentinel. The deadline is a backstop
// against a hang, not a timing assumption: the first write after
// the eviction is refused.
func startRacingWrites(
w *delivery.ExportArchiveWriter,
) *racingWrites {
r := &racingWrites{
started: make(chan struct{}, racingWriteGoroutines),
}
deadline := time.Now().Add(10 * time.Second)
r.wg.Add(racingWriteGoroutines)
for i := range racingWriteGoroutines {
go func() {
defer r.wg.Done()
first := true
for time.Now().Before(deadline) {
err := w.Write(
evictTestRow(fmt.Sprintf("ev-%d", i)), 0,
)
if first {
r.started <- struct{}{}
first = false
}
if err == nil {
continue
}
r.record(err)
return
}
}()
}
return r
}
// record classifies the error that stopped one goroutine.
func (r *racingWrites) record(err error) {
r.mu.Lock()
defer r.mu.Unlock()
if errors.Is(err, delivery.ErrExportArchiveWriterEvicted) {
r.sawEvicted = true
return
}
r.otherErr = err
}
// awaitFirstWrite blocks until at least one write has run, so
// the eviction that follows is a genuine race.
func (r *racingWrites) awaitFirstWrite() {
<-r.started
}
// wait joins the goroutines and reports whether any write was
// refused with the eviction sentinel, plus any unexpected error.
func (r *racingWrites) wait() (bool, error) {
r.wg.Wait()
r.mu.Lock()
defer r.mu.Unlock()
return r.sawEvicted, r.otherErr
}
// TestEvictWebhook_RacingWriteDoesNotReopenHandle exercises the
// interleaving the evicted flag exists for: writes already
// contending for the writer's mutex when the eviction takes it.
// The write that wins the mutex after the eviction must abandon
// its work rather than reopen the archive, leaving the writer
// permanently handle-free. Run under -race.
func TestEvictWebhook_RacingWriteDoesNotReopenHandle(
t *testing.T,
) {
t.Parallel()
eng, _ := evictTestEngine(t)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":true}`)
d := seedDatabaseTargetDelivery(t, webhookDB, event, "")
// Prime the registry so the test can hold the very writer the
// eviction is about to detach.
eng.ExportDeliverDatabase(webhookDB, d)
w := eng.ExportArchiveWriterFor(event.WebhookID)
require.NotNil(t, w)
require.True(t, w.HandleOpen())
race := startRacingWrites(w)
// Evict only once writes are genuinely in flight, so the
// eviction has to contend for the writer's mutex.
race.awaitFirstWrite()
eng.EvictWebhook(event.WebhookID)
sawEvicted, otherErr := race.wait()
require.NoError(t, otherErr)
assert.True(
t, sawEvicted,
"a write after eviction must be refused",
)
assert.False(
t, w.HandleOpen(),
"no write may reopen the archive once the writer has "+
"been evicted",
)
assert.False(
t, eng.ExportHasArchiveWriter(event.WebhookID),
"the registry entry must stay gone",
)
}
// TestEvictWebhook_LaterDeliveryRecreatesWriter proves eviction
// does not break archiving for a webhook that is still alive: a
// subsequent delivery gets a brand new writer from the registry.
// It says nothing about the evicted writer itself — that is what
// TestEvictedWriter_WriteDoesNotReopenFile covers.
func TestEvictWebhook_LaterDeliveryRecreatesWriter(t *testing.T) {
t.Parallel()
eng, _ := evictTestEngine(t)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":true}`)
d := seedDatabaseTargetDelivery(t, webhookDB, event, "")
eng.ExportDeliverDatabase(webhookDB, d)
require.True(
t, eng.ExportHasArchiveWriter(event.WebhookID),
)
eng.EvictWebhook(event.WebhookID)
// A fresh delivery for the same webhook gets a brand new
// writer from the registry, so archiving keeps working.
second := seedDatabaseTargetDelivery(
t, webhookDB, event, "",
)
eng.ExportDeliverDatabase(webhookDB, second)
assert.True(
t, eng.ExportHasArchiveWriter(event.WebhookID),
"a later delivery should recreate the writer",
)
}

402
internal/delivery/target_database_test.go
View File

@@ -1,402 +0,0 @@
package delivery_test
import (
"database/sql"
"fmt"
"log/slog"
"net/http"
"os"
"path/filepath"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
_ "modernc.org/sqlite" // Pure Go SQLite driver.
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
func archiveTestLogger() *slog.Logger {
return slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
))
}
// openArchiveDBForRead opens an archive file read-only so a
// test can inspect the rows the writer persisted.
func openArchiveDBForRead(
t *testing.T, path string,
) *gorm.DB {
t.Helper()
sqlDB, err := sql.Open(
"sqlite",
fmt.Sprintf("file:%s?mode=ro", path),
)
require.NoError(t, err)
t.Cleanup(func() { _ = sqlDB.Close() })
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
require.NoError(t, err)
return gdb
}
// archiveFileSuffixes returns the archive file itself and the
// SQLite sidecars that accompany an open database. A test that
// asserts no archive was created has to check all of them.
func archiveFileSuffixes() []string {
return []string{"", "-wal", "-shm"}
}
// removeArchiveFiles simulates an operator moving the archive
// away by deleting the SQLite file and its sidecar files.
func removeArchiveFiles(t *testing.T, path string) {
t.Helper()
for _, suffix := range []string{
"", "-wal", "-shm", "-journal",
} {
err := os.Remove(path + suffix)
if err != nil && !os.IsNotExist(err) {
t.Fatalf("removing %s%s: %v", path, suffix, err)
}
}
}
// TestDeliverDatabase_ArchivesEvent verifies that delivering to
// a database target marks the delivery delivered and archives
// the full event into a separate per-webhook archive file.
func TestDeliverDatabase_ArchivesEvent(t *testing.T) {
t.Parallel()
dataDir := t.TempDir()
dbMgr := database.NewTestWebhookDBManager(dataDir)
e := delivery.NewTestEngineWithDB(
nil, dbMgr,
archiveTestLogger(),
&http.Client{Timeout: 5 * time.Second},
1,
)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":true}`)
d := seedDatabaseTargetDelivery(t, webhookDB, event, "")
e.ExportDeliverDatabase(webhookDB, d)
var updated database.Delivery
require.NoError(t, webhookDB.First(
&updated, "id = ?", d.ID,
).Error)
assert.Equal(t,
database.DeliveryStatusDelivered, updated.Status,
"database target should mark the delivery delivered",
)
archivePath := filepath.Join(
dataDir,
fmt.Sprintf("archive-%s.db", event.WebhookID),
)
assert.FileExists(t, archivePath)
rdb := openArchiveDBForRead(t, archivePath)
var rows []delivery.ExportArchivedEvent
require.NoError(t, rdb.Find(&rows).Error)
require.Len(t, rows, 1)
assert.Equal(t, event.ID, rows[0].EventID)
assert.Equal(t, event.WebhookID, rows[0].WebhookID)
assert.Equal(t, event.Method, rows[0].Method)
assert.JSONEq(t, `{"archived":true}`, rows[0].Body)
}
func TestArchiveWriter_WritesRow(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-wh.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
row := delivery.ExportArchivedEvent{
EventID: "ev-1",
WebhookID: "wh-1",
EntrypointID: "ep-1",
Method: "POST",
Headers: `{"X":"Y"}`,
Body: `{"hello":"world"}`,
ContentType: "application/json",
}
require.NoError(t, w.Write(row, 0))
assert.FileExists(t, path)
var got []delivery.ExportArchivedEvent
require.NoError(t, w.DB().Find(&got).Error)
require.Len(t, got, 1)
assert.Equal(t, "ev-1", got[0].EventID)
assert.Equal(t, "wh-1", got[0].WebhookID)
assert.Equal(t, "ep-1", got[0].EntrypointID)
assert.Equal(t, row.Method, got[0].Method)
assert.Equal(t, row.ContentType, got[0].ContentType)
assert.JSONEq(t, `{"hello":"world"}`, got[0].Body)
assert.False(t, got[0].ArchivedAt.IsZero())
}
func TestArchiveWriter_RecreatesAfterRemoval(
t *testing.T,
) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-wh.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "a"}, 0,
))
assert.FileExists(t, path)
// The operator moves the archive away while the handle is
// still open.
removeArchiveFiles(t, path)
require.NoFileExists(t, path)
// The next write recreates the file with a fresh schema and
// only the new row.
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "b"}, 0,
))
assert.FileExists(t, path)
var got []delivery.ExportArchivedEvent
require.NoError(t, w.DB().Find(&got).Error)
require.Len(t, got, 1)
assert.Equal(t, "b", got[0].EventID)
}
func TestArchiveWriter_ReopenDebounce(t *testing.T) {
t.Parallel()
// A generous debounce keeps the two rapid writes inside
// the window even on a heavily loaded test machine.
path := filepath.Join(t.TempDir(), "archive-wh.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 2*time.Second,
)
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "a"}, 0,
))
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "b"}, 0,
))
// Two writes inside the debounce window trigger only the
// initial open — no extra close/reopen.
assert.Equal(t, 1, w.Reopens())
time.Sleep(2100 * time.Millisecond)
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "c"}, 0,
))
// A write after the window elapses closes and reopens once.
assert.Equal(t, 2, w.Reopens())
}
func TestArchiveWriter_ExpiryPrune(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-wh.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.Open(0))
old := delivery.ExportArchivedEvent{
EventID: "old",
ArchivedAt: time.Now().Add(-2 * time.Hour),
}
fresh := delivery.ExportArchivedEvent{
EventID: "fresh",
ArchivedAt: time.Now(),
}
require.NoError(t, w.DB().Create(&old).Error)
require.NoError(t, w.DB().Create(&fresh).Error)
// Reopening with a one-hour expiry prunes the old row.
require.NoError(t, w.Reopen(time.Hour))
var got []delivery.ExportArchivedEvent
require.NoError(t, w.DB().Find(&got).Error)
require.Len(t, got, 1)
assert.Equal(t, "fresh", got[0].EventID)
}
func TestParseArchiveExpiry(t *testing.T) {
t.Parallel()
cases := []struct {
name string
in string
want time.Duration
wantErr bool
}{
{"empty config", "", 0, false},
{"explicit never", `{"expiry":"never"}`, 0, false},
{"empty expiry", `{"expiry":""}`, 0, false},
{"duration", `{"expiry":"1h"}`, time.Hour, false},
{"unparseable", `{"expiry":"nonsense"}`, 0, true},
{"zero duration", `{"expiry":"0s"}`, 0, true},
{"negative duration", `{"expiry":"-5h"}`, 0, true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
got, err := delivery.ExportParseArchiveExpiry(tc.in)
if tc.wantErr {
require.Error(t, err)
return
}
require.NoError(t, err)
assert.Equal(t, tc.want, got)
})
}
}
// seedDatabaseTargetDelivery seeds a pending delivery for a
// database target with the given config JSON and returns the
// in-memory delivery the target handler is invoked with.
func seedDatabaseTargetDelivery(
t *testing.T,
webhookDB *gorm.DB,
event database.Event,
config string,
) *database.Delivery {
t.Helper()
dlv := seedDelivery(
t, webhookDB, event.ID, uuid.New().String(),
database.DeliveryStatusPending,
)
d := &database.Delivery{
EventID: event.ID,
TargetID: dlv.TargetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: database.Target{
Name: "test-db",
Type: database.TargetTypeDatabase,
Config: config,
},
}
d.ID = dlv.ID
return d
}
// TestDeliverDatabase_ArchiveFailureFailsDelivery verifies that
// an archive error (here: an unparseable expiry in the target
// config) fails the delivery loudly: the attempt is recorded as
// failed with the error and the delivery is marked failed, not
// delivered.
func TestDeliverDatabase_ArchiveFailureFailsDelivery(
t *testing.T,
) {
t.Parallel()
dataDir := t.TempDir()
e := delivery.NewTestEngineWithDB(
nil, database.NewTestWebhookDBManager(dataDir),
archiveTestLogger(),
&http.Client{Timeout: 5 * time.Second},
1,
)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":false}`)
d := seedDatabaseTargetDelivery(
t, webhookDB, event, `{"expiry":"nonsense"}`,
)
e.ExportDeliverDatabase(webhookDB, d)
var updated database.Delivery
require.NoError(t, webhookDB.First(
&updated, "id = ?", d.ID,
).Error)
assert.Equal(t,
database.DeliveryStatusFailed, updated.Status,
"archive failure must mark the delivery failed",
)
var results []database.DeliveryResult
require.NoError(t, webhookDB.Where(
"delivery_id = ?", d.ID,
).Find(&results).Error)
require.Len(t, results, 1)
assert.False(t,
results[0].Success,
"the attempt must be recorded as failed",
)
assert.Contains(t,
results[0].Error, "nonsense",
"the archive error must be recorded on the attempt",
)
assert.NoFileExists(t,
filepath.Join(
dataDir,
fmt.Sprintf("archive-%s.db", event.WebhookID),
),
"no archive file should exist for a failed config",
)
}
func TestValidateArchiveExpiry(t *testing.T) {
t.Parallel()
valid := []string{"", "never", "1h", "720h", "30m"}
for _, in := range valid {
require.NoError(t,
delivery.ValidateArchiveExpiry(in),
"expiry %q should be accepted", in,
)
}
invalid := []string{"nonsense", "7d", "-5h", "0s", "0"}
for _, in := range invalid {
require.Error(t,
delivery.ValidateArchiveExpiry(in),
"expiry %q should be rejected", in,
)
}
}

499
internal/delivery/target_http.go
View File

@@ -1,499 +0,0 @@
package delivery
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"sync"
"time"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// Sentinel errors returned by the config parsers.
var (
errEmptyTargetConfig = errors.New(
"empty target config",
)
errMissingTargetURL = errors.New(
"target URL is required",
)
)
// HTTPTargetConfig holds configuration for http target
// types.
type HTTPTargetConfig struct {
URL string `json:"url"`
Headers map[string]string `json:"headers,omitempty"`
Timeout int `json:"timeout,omitempty"`
}
// httpCore holds the retry, backoff, and circuit-breaker
// machinery shared by the HTTP and Slack targets. Each of
// those targets owns its own httpCore instance (and thus its
// own circuit breakers); the per-attempt request differs
// between them and is supplied as a closure.
type httpCore struct {
eng *Engine
// circuitBreakers stores a *CircuitBreaker per target ID.
circuitBreakers sync.Map
}
// deliver runs one delivery attempt through the retry core.
// A maxRetries of 0 is fire-and-forget: a single attempt is
// recorded and no circuit breaker is consulted. A positive
// maxRetries gates the attempt on the circuit breaker and
// schedules a backed-off retry on failure.
func (c *httpCore) deliver(
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
maxRetries int,
attempt func() attemptResult,
) {
if maxRetries == 0 {
c.fireAndForget(webhookDB, d, attempt())
return
}
c.withRetry(
webhookDB, d, task, sched, maxRetries, attempt,
)
}
func (c *httpCore) fireAndForget(
webhookDB *gorm.DB,
d *database.Delivery,
res attemptResult,
) {
c.eng.recordResult(
webhookDB, d, 1, res.success,
res.statusCode, res.respBody, res.errMsg,
res.duration,
)
if res.success {
c.eng.updateDeliveryStatus(
webhookDB, d,
database.DeliveryStatusDelivered,
)
return
}
c.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusFailed,
)
}
func (c *httpCore) withRetry(
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
maxRetries int,
attempt func() attemptResult,
) {
cb := c.getCircuitBreaker(task.TargetID)
if c.circuitBreakerBlock(webhookDB, d, task, sched, cb) {
return
}
attemptNum := task.AttemptNum
res := attempt()
c.eng.recordResult(
webhookDB, d, attemptNum, res.success,
res.statusCode, res.respBody, res.errMsg,
res.duration,
)
if res.success {
cb.RecordSuccess()
c.eng.updateDeliveryStatus(
webhookDB, d,
database.DeliveryStatusDelivered,
)
return
}
cb.RecordFailure()
c.handleRetry(
webhookDB, d, task, sched, maxRetries, attemptNum,
)
}
func (c *httpCore) circuitBreakerBlock(
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
cb *CircuitBreaker,
) bool {
if cb.Allow() {
return false
}
remaining := cb.CooldownRemaining()
c.eng.log.Info(
"circuit breaker open, skipping delivery",
"target_id", task.TargetID,
"target_name", task.TargetName,
"delivery_id", d.ID,
"cooldown_remaining", remaining,
)
c.eng.updateDeliveryStatus(
webhookDB, d,
database.DeliveryStatusRetrying,
)
retryTask := *task
sched.ScheduleRetry(retryTask, remaining)
return true
}
func (c *httpCore) handleRetry(
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
maxRetries int,
attemptNum int,
) {
if attemptNum >= maxRetries {
c.eng.updateDeliveryStatus(
webhookDB, d,
database.DeliveryStatusFailed,
)
return
}
c.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusRetrying,
)
backoff := calcBackoff(attemptNum)
retryTask := *task
retryTask.AttemptNum = attemptNum + 1
sched.ScheduleRetry(retryTask, backoff)
}
func (c *httpCore) getCircuitBreaker(
targetID string,
) *CircuitBreaker {
if val, ok := c.circuitBreakers.Load(targetID); ok {
cb, _ := val.(*CircuitBreaker)
return cb
}
fresh := NewCircuitBreaker()
actual, _ := c.circuitBreakers.LoadOrStore(
targetID, fresh,
)
cb, _ := actual.(*CircuitBreaker)
return cb
}
// remainingBackoff returns how long remains of the backoff
// window for the last attempt of a recovered retrying
// delivery. It implements rescheduler.
func (c *httpCore) remainingBackoff(
webhookDB *gorm.DB,
deliveryID string,
attemptNum int,
) time.Duration {
var lastResult database.DeliveryResult
err := webhookDB.
Where("delivery_id = ?", deliveryID).
Order("created_at DESC").
First(&lastResult).Error
if err != nil {
return 0
}
backoff := calcBackoff(attemptNum)
elapsed := time.Since(lastResult.CreatedAt)
remaining := backoff - elapsed
return max(remaining, 0)
}
// backoffElapsed reports whether the backoff window for the
// last attempt of a retrying delivery has passed. It
// implements rescheduler.
func (c *httpCore) backoffElapsed(
webhookDB *gorm.DB,
deliveryID string,
attemptNum int,
) bool {
var lastResult database.DeliveryResult
err := webhookDB.
Where("delivery_id = ?", deliveryID).
Order("created_at DESC").
First(&lastResult).Error
if err != nil {
return true
}
backoff := calcBackoff(attemptNum)
return time.Since(lastResult.CreatedAt) >= backoff
}
func calcBackoff(attemptNum int) time.Duration {
shift := max(attemptNum-1, 0)
shift = min(shift, maxBackoffShift)
return time.Duration(1<<uint(shift)) * time.Second
}
// httpTarget delivers events to http targets. It forwards the
// event body and (filtered) request headers to the configured
// URL and owns retry, backoff, and circuit breaking through
// the shared httpCore.
type httpTarget struct {
*httpCore
client *http.Client
}
// Deliver implements Target.
func (t *httpTarget) Deliver(
ctx context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
) {
cfg, err := parseHTTPConfig(d.Target.Config)
if err != nil {
t.eng.log.Error(
"invalid HTTP target config",
"target_id", d.TargetID,
"error", err,
)
t.eng.recordResult(
webhookDB, d, task.AttemptNum,
false, 0, "", err.Error(), 0,
)
t.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusFailed,
)
return
}
attempt := func() attemptResult {
return t.attempt(ctx, cfg, &d.Event)
}
t.deliver(
webhookDB, d, task, sched,
d.Target.MaxRetries, attempt,
)
}
// attempt performs a single HTTP delivery attempt and derives
// the success flag and error message the same way the engine
// did: a non-2xx response is a failure but carries no error
// string; only a transport-level error does.
func (t *httpTarget) attempt(
ctx context.Context,
cfg *HTTPTargetConfig,
event *database.Event,
) attemptResult {
statusCode, respBody, duration, reqErr :=
t.doHTTPRequest(ctx, cfg, event)
success := reqErr == nil &&
statusCode >= httpSuccessMin &&
statusCode < httpSuccessMax
errMsg := ""
if reqErr != nil {
errMsg = reqErr.Error()
}
return attemptResult{
statusCode: statusCode,
respBody: respBody,
duration: duration,
success: success,
errMsg: errMsg,
}
}
func (t *httpTarget) doHTTPRequest(
ctx context.Context,
cfg *HTTPTargetConfig,
event *database.Event,
) (int, string, int64, error) {
start := time.Now()
req, reqErr := http.NewRequestWithContext(
ctx,
http.MethodPost,
cfg.URL,
bytes.NewReader([]byte(event.Body)),
)
if reqErr != nil {
return 0, "", 0, fmt.Errorf(
"creating request: %w", reqErr,
)
}
applyRequestHeaders(req, event, cfg)
client := t.clientForConfig(cfg)
resp, doErr := executeHTTPRequest(client, req)
dur := time.Since(start).Milliseconds()
if doErr != nil {
return 0, "", dur, fmt.Errorf(
"sending request: %w", doErr,
)
}
defer func() { _ = resp.Body.Close() }()
body, readErr := io.ReadAll(
io.LimitReader(resp.Body, maxBodyLog),
)
if readErr != nil {
return resp.StatusCode, "", dur,
fmt.Errorf(
"reading response body: %w", readErr,
)
}
return resp.StatusCode, string(body), dur, nil
}
func (t *httpTarget) clientForConfig(
cfg *HTTPTargetConfig,
) *http.Client {
if cfg.Timeout > 0 {
// Reuse the shared client's SSRF-safe transport so
// a per-target timeout does not drop the
// request-time private-IP guard. Only the timeout
// is overridden.
return &http.Client{
Timeout: time.Duration(
cfg.Timeout,
) * time.Second,
Transport: t.client.Transport,
}
}
return t.client
}
func parseHTTPConfig(
configJSON string,
) (*HTTPTargetConfig, error) {
if configJSON == "" {
return nil, errEmptyTargetConfig
}
var cfg HTTPTargetConfig
err := json.Unmarshal(
[]byte(configJSON), &cfg,
)
if err != nil {
return nil, fmt.Errorf(
"parsing config JSON: %w", err,
)
}
if cfg.URL == "" {
return nil, errMissingTargetURL
}
return &cfg, nil
}
// isForwardableHeader returns true if the header should
// be forwarded to targets.
func isForwardableHeader(name string) bool {
switch http.CanonicalHeaderKey(name) {
case "Host", "Connection", "Keep-Alive",
"Transfer-Encoding", "Te", "Trailer",
"Upgrade", "Proxy-Authorization",
"Proxy-Connection", "Content-Length":
return false
default:
return true
}
}
func applyRequestHeaders(
req *http.Request,
event *database.Event,
cfg *HTTPTargetConfig,
) {
if event.ContentType != "" {
req.Header.Set(
"Content-Type", event.ContentType,
)
}
var originalHeaders map[string][]string
if event.Headers != "" {
jsonErr := json.Unmarshal(
[]byte(event.Headers),
&originalHeaders,
)
if jsonErr == nil {
for k, vals := range originalHeaders {
if isForwardableHeader(k) {
for _, v := range vals {
req.Header.Add(k, v)
}
}
}
}
}
for k, v := range cfg.Headers {
req.Header.Set(k, v)
}
req.Header.Set("User-Agent", "webhooker/1.0")
}
// executeHTTPRequest sends an HTTP request using the provided
// client. URLs are validated by the config parsers and the
// SSRF-safe transport before reaching here.
func executeHTTPRequest(
client *http.Client, req *http.Request,
) (*http.Response, error) {
return client.Do(req) //#nosec G704 -- validated URL, SSRF-safe transport
}

47
internal/delivery/target_log.go
View File

@@ -1,47 +0,0 @@
package delivery
import (
"context"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// logTarget is a fire-and-forget target that logs the entire
// inbound webhook — the full request body and headers, plus
// the method, content type, and the webhook and entrypoint
// ids — then records a single successful attempt.
type logTarget struct {
eng *Engine
}
// Deliver implements Target.
func (t *logTarget) Deliver(
_ context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
_ *Task,
_ Scheduler,
) {
t.eng.log.Info(
"webhook event delivered to log target",
"delivery_id", d.ID,
"event_id", d.EventID,
"target_id", d.TargetID,
"target_name", d.Target.Name,
"webhook_id", d.Event.WebhookID,
"entrypoint_id", d.Event.EntrypointID,
"method", d.Event.Method,
"content_type", d.Event.ContentType,
"headers", d.Event.Headers,
"body", d.Event.Body,
)
t.eng.recordResult(
webhookDB, d, 1, true, 0, "", "", 0,
)
t.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusDelivered,
)
}

299
internal/delivery/target_slack.go
View File

@@ -1,299 +0,0 @@
package delivery
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strings"
"time"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// errMissingWebhookURL is returned when a Slack target config
// omits its webhook URL.
var errMissingWebhookURL = errors.New(
"webhook_url is required",
)
// SlackTargetConfig holds configuration for slack target
// types.
type SlackTargetConfig struct {
WebhookURL string `json:"webhookUrl"`
}
// slackTarget delivers events to Slack incoming webhooks. It
// formats the event into a Slack message and posts it as
// JSON. It shares the retry core with the HTTP target: a
// MaxRetries of 0 stays single-attempt fire-and-forget
// (preserving existing Slack targets), while a positive
// MaxRetries adds backoff and circuit breaking.
type slackTarget struct {
*httpCore
client *http.Client
}
// Deliver implements Target.
func (t *slackTarget) Deliver(
ctx context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
) {
cfg, err := parseSlackConfig(d.Target.Config)
if err != nil {
t.eng.log.Error(
"invalid Slack target config",
"target_id", d.TargetID,
"error", err,
)
t.failConfig(webhookDB, d, err)
return
}
msg := FormatSlackMessage(&d.Event)
payload, err := json.Marshal(
map[string]string{"text": msg},
)
if err != nil {
t.eng.log.Error(
"failed to marshal Slack payload",
"target_id", d.TargetID,
"error", err,
)
t.failConfig(webhookDB, d, err)
return
}
attempt := func() attemptResult {
return t.attempt(ctx, cfg, payload)
}
t.deliver(
webhookDB, d, task, sched,
d.Target.MaxRetries, attempt,
)
}
// failConfig records a first-attempt failure for a delivery
// that could not be prepared (bad config or unmarshalable
// payload) and marks it failed.
func (t *slackTarget) failConfig(
webhookDB *gorm.DB,
d *database.Delivery,
err error,
) {
t.eng.recordResult(
webhookDB, d, 1,
false, 0, "", err.Error(), 0,
)
t.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusFailed,
)
}
// attempt performs a single Slack POST and derives its
// outcome, preserving the engine's original semantics: a
// non-2xx response records an "HTTP <code>" error string and
// a transport error records a "sending request" error.
func (t *slackTarget) attempt(
ctx context.Context,
cfg *SlackTargetConfig,
payload []byte,
) attemptResult {
start := time.Now()
req, err := http.NewRequestWithContext(
ctx,
http.MethodPost,
cfg.WebhookURL,
bytes.NewReader(payload),
)
if err != nil {
return attemptResult{
success: false,
errMsg: err.Error(),
}
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("User-Agent", "webhooker/1.0")
resp, doErr := executeHTTPRequest(t.client, req)
durationMs := time.Since(start).Milliseconds()
if doErr != nil {
return attemptResult{
success: false,
duration: durationMs,
errMsg: fmt.Errorf(
"sending request: %w", doErr,
).Error(),
}
}
defer func() { _ = resp.Body.Close() }()
return t.readSlackResponse(resp, durationMs)
}
func (t *slackTarget) readSlackResponse(
resp *http.Response,
durationMs int64,
) attemptResult {
body, readErr := io.ReadAll(
io.LimitReader(resp.Body, maxBodyLog),
)
if readErr != nil {
t.eng.log.Error(
"failed to read Slack response body",
"error", readErr,
)
}
success := resp.StatusCode >= httpSuccessMin &&
resp.StatusCode < httpSuccessMax
errMsg := ""
if !success {
errMsg = fmt.Sprintf("HTTP %d", resp.StatusCode)
}
return attemptResult{
statusCode: resp.StatusCode,
respBody: string(body),
duration: durationMs,
success: success,
errMsg: errMsg,
}
}
func parseSlackConfig(
configJSON string,
) (*SlackTargetConfig, error) {
if configJSON == "" {
return nil, errEmptyTargetConfig
}
var cfg SlackTargetConfig
err := json.Unmarshal(
[]byte(configJSON), &cfg,
)
if err != nil {
return nil, fmt.Errorf(
"parsing config JSON: %w", err,
)
}
if cfg.WebhookURL == "" {
return nil, errMissingWebhookURL
}
return &cfg, nil
}
// FormatSlackMessage builds a Slack-compatible message
// string from a webhook event.
func FormatSlackMessage(
event *database.Event,
) string {
var b strings.Builder
b.WriteString("*Webhook Event Received*\n")
fmt.Fprintf(
&b, "*Method:* `%s`\n", event.Method,
)
fmt.Fprintf(
&b,
"*Content-Type:* `%s`\n",
event.ContentType,
)
fmt.Fprintf(
&b,
"*Timestamp:* `%s`\n",
event.CreatedAt.UTC().Format(time.RFC3339),
)
fmt.Fprintf(
&b,
"*Body Size:* %d bytes\n",
len(event.Body),
)
if event.Body == "" {
b.WriteString("\n_(empty body)_\n")
return b.String()
}
if formatted := formatJSONBody(event.Body); formatted != "" {
b.WriteString(formatted)
return b.String()
}
formatRawBody(&b, event.Body)
return b.String()
}
func formatJSONBody(body string) string {
var parsed json.RawMessage
if json.Unmarshal([]byte(body), &parsed) != nil {
return ""
}
var pretty bytes.Buffer
if json.Indent(&pretty, parsed, "", " ") != nil {
return ""
}
var b strings.Builder
b.WriteString("\n```\n")
prettyStr := pretty.String()
const maxPayloadDisplay = 3500
if len(prettyStr) > maxPayloadDisplay {
b.WriteString(prettyStr[:maxPayloadDisplay])
b.WriteString("\n... (truncated)")
} else {
b.WriteString(prettyStr)
}
b.WriteString("\n```\n")
return b.String()
}
func formatRawBody(b *strings.Builder, body string) {
b.WriteString("\n```\n")
const maxRawDisplay = 3500
if len(body) > maxRawDisplay {
b.WriteString(body[:maxRawDisplay])
b.WriteString("\n... (truncated)")
} else {
b.WriteString(body)
}
b.WriteString("\n```\n")
}

4
internal/handlers/auth.go
View File

@@ -19,7 +19,7 @@ func (h *Handlers) HandleLoginPage() http.HandlerFunc {
// Render login page
data := map[string]any{
tmplKeyError: "",
"Error": "",
}
h.renderTemplate(w, r, "login.html", data)
@@ -86,7 +86,7 @@ func (h *Handlers) renderLoginError(
status int,
) {
data := map[string]any{
tmplKeyError: msg,
"Error": msg,
}
w.WriteHeader(status)

20
internal/handlers/export_test.go
View File

@@ -13,26 +13,12 @@ func (s *Handlers) RenderTemplateForTest(
s.renderTemplate(w, r, pageTemplate, data)
}
// BuildSlackTargetConfigForTest exposes buildURLTargetConfig
// with the Slack target parameters for use in the
// handlers_test package.
// BuildSlackTargetConfigForTest exposes buildSlackTargetConfig
// for use in the handlers_test package.
func (s *Handlers) BuildSlackTargetConfigForTest(
w http.ResponseWriter,
r *http.Request,
targetURL string,
) (string, error) {
return s.buildURLTargetConfig(
w, r, targetURL, "webhookUrl",
"Webhook URL is required for Slack targets",
)
}
// BuildDatabaseTargetConfigForTest exposes
// buildDatabaseTargetConfig for use in the handlers_test
// package.
func (s *Handlers) BuildDatabaseTargetConfigForTest(
w http.ResponseWriter,
expiry string,
) (string, error) {
return s.buildDatabaseTargetConfig(w, expiry)
return s.buildSlackTargetConfig(w, r, targetURL)
}

8
internal/handlers/handlers.go
View File

@@ -30,11 +30,6 @@ const (
defaultRetentionDays = 30
// paginationPerPage is the number of items per page.
paginationPerPage = 25
// tmplKeyError is the template data key for an error message.
tmplKeyError = "Error"
// tmplKeyWebhook is the template data key for a webhook.
tmplKeyWebhook = "Webhook"
)
// errInvalidPassword is returned when a password does not match.
@@ -51,7 +46,6 @@ type HandlersParams struct {
Healthcheck *healthcheck.Healthcheck
Session *session.Session
Notifier delivery.Notifier
Evictor delivery.WebhookEvictor
}
// Handlers provides HTTP handler methods for all application
@@ -64,7 +58,6 @@ type Handlers struct {
dbMgr *database.WebhookDBManager
session *session.Session
notifier delivery.Notifier
evictor delivery.WebhookEvictor
templates map[string]*template.Template
}
@@ -99,7 +92,6 @@ func New(
s.dbMgr = params.WebhookDBMgr
s.session = params.Session
s.notifier = params.Notifier
s.evictor = params.Evictor
// Parse all page templates once at startup
s.templates = map[string]*template.Template{

87
internal/handlers/handlers_test.go
View File

@@ -4,7 +4,6 @@ import (
"context"
"net/http"
"net/http/httptest"
"sync"
"testing"
"github.com/stretchr/testify/assert"
@@ -25,32 +24,6 @@ type noopNotifier struct{}
func (n *noopNotifier) Notify([]delivery.Task) {}
// recordingEvictor is a delivery.WebhookEvictor that records
// the webhook ids it was asked to evict, so a test can prove
// that a deletion path reached the delivery engine.
type recordingEvictor struct {
mu sync.Mutex
evicted []string
}
func (r *recordingEvictor) EvictWebhook(webhookID string) {
r.mu.Lock()
defer r.mu.Unlock()
r.evicted = append(r.evicted, webhookID)
}
// Evicted returns a copy of the recorded webhook ids.
func (r *recordingEvictor) Evicted() []string {
r.mu.Lock()
defer r.mu.Unlock()
out := make([]string, len(r.evicted))
copy(out, r.evicted)
return out
}
func newTestApp(
t *testing.T,
targets ...any,
@@ -74,12 +47,6 @@ func newTestApp(
func() delivery.Notifier {
return &noopNotifier{}
},
func() *recordingEvictor {
return &recordingEvictor{}
},
func(r *recordingEvictor) delivery.WebhookEvictor {
return r
},
handlers.New,
),
fx.Populate(targets...),
@@ -219,57 +186,3 @@ func TestRenderTemplate(t *testing.T) {
t, http.StatusInternalServerError, w.Code,
)
}
func TestBuildDatabaseTargetConfig_Valid(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
app := newTestApp(t, &h)
app.RequireStart()
t.Cleanup(app.RequireStop)
// Empty expiry: the keep-forever default, empty config.
w := httptest.NewRecorder()
cfg, err := h.BuildDatabaseTargetConfigForTest(w, "")
require.NoError(t, err)
assert.Empty(t, cfg)
// Explicit never is stored as config.
w = httptest.NewRecorder()
cfg, err = h.BuildDatabaseTargetConfigForTest(w, "never")
require.NoError(t, err)
assert.JSONEq(t, `{"expiry":"never"}`, cfg)
// A positive duration is stored as config.
w = httptest.NewRecorder()
cfg, err = h.BuildDatabaseTargetConfigForTest(w, "720h")
require.NoError(t, err)
assert.JSONEq(t, `{"expiry":"720h"}`, cfg)
}
func TestBuildDatabaseTargetConfig_RejectsBadExpiry(
t *testing.T,
) {
t.Parallel()
var h *handlers.Handlers
app := newTestApp(t, &h)
app.RequireStart()
t.Cleanup(app.RequireStop)
for _, bad := range []string{"nonsense", "7d", "-5h"} {
w := httptest.NewRecorder()
cfg, err := h.BuildDatabaseTargetConfigForTest(w, bad)
require.Error(t, err, "expiry %q", bad)
assert.Empty(t, cfg)
assert.Equal(
t, http.StatusBadRequest, w.Code,
"expiry %q should be rejected with 400", bad,
)
}
}

179
internal/handlers/profile.go
View File

@@ -4,202 +4,63 @@ import (
"net/http"
"github.com/go-chi/chi"
"sneak.berlin/go/webhooker/internal/database"
)
// HandleProfile returns a handler for the user profile page
func (h *Handlers) HandleProfile() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
sessionUserID, sessionUsername, ok :=
h.profileOwnerOrDeny(w, r)
if !ok {
return
}
h.renderProfile(w, r, sessionUserID, sessionUsername, "", "")
}
}
// HandlePasswordChange returns a handler that lets an authenticated
// user change their own password. It is served by the CSRF- and
// auth-protected POST /password route under /user/{username}.
func (h *Handlers) HandlePasswordChange() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
sessionUserID, sessionUsername, ok :=
h.profileOwnerOrDeny(w, r)
if !ok {
return
}
// Limit request body to prevent memory exhaustion.
r.Body = http.MaxBytesReader(w, r.Body, 1<<maxBodyShift)
err := r.ParseForm()
if err != nil {
h.log.Error("failed to parse form", "error", err)
http.Error(w, "Bad request", http.StatusBadRequest)
return
}
successMessage, errorMessage, handled := h.applyPasswordChange(
w,
sessionUsername,
r.FormValue("current_password"),
r.FormValue("new_password"),
r.FormValue("confirm_password"),
)
if !handled {
return
}
h.renderProfile(
w, r, sessionUserID, sessionUsername,
successMessage, errorMessage,
)
}
}
// applyPasswordChange verifies the current password and, on success,
// persists a fresh hash for the user, reusing the same helpers that
// bootstrap the admin user. It returns the success and error messages
// to display on the profile page. On an internal failure it writes a
// 500 response itself and returns handled=false, signalling the caller
// to stop without re-rendering the page.
func (h *Handlers) applyPasswordChange(
w http.ResponseWriter,
username, currentPassword, newPassword, confirmPassword string,
) (string, string, bool) {
// Load the user row so we can verify the current password and
// persist the new hash.
var user database.User
err := h.db.DB().Where(
"username = ?", username,
).First(&user).Error
if err != nil {
h.serverError(
w, "failed to load user for password change", err,
)
return "", "", false
}
valid, err := database.VerifyPassword(
currentPassword, user.Password,
)
if err != nil {
h.serverError(w, "failed to verify password", err)
return "", "", false
}
if !valid {
return "", "Current password is incorrect.", true
}
if newPassword == "" {
return "", "New password must not be empty.", true
}
if newPassword != confirmPassword {
return "", "New password and confirmation do not match.", true
}
hashedPassword, err := database.HashPassword(newPassword)
if err != nil {
h.serverError(w, "failed to hash new password", err)
return "", "", false
}
err = h.db.DB().Model(&user).Update(
"password", hashedPassword,
).Error
if err != nil {
h.serverError(w, "failed to update password", err)
return "", "", false
}
h.log.Info("user changed password", "username", username)
return "Password changed successfully.", "", true
}
// profileOwnerOrDeny resolves the session identity and enforces that a
// user may only act on their own profile (the requested username in the
// URL must equal the session username). On any failure it writes the
// appropriate HTTP response and returns ok=false; callers must stop
// when ok is false.
func (h *Handlers) profileOwnerOrDeny(
w http.ResponseWriter,
r *http.Request,
) (string, string, bool) {
// Get username from URL
requestedUsername := chi.URLParam(r, "username")
if requestedUsername == "" {
http.NotFound(w, r)
return "", "", false
return
}
// RequireAuth middleware guarantees an authenticated session
// before this handler runs, so we only need to guard against an
// unexpected retrieval error.
// Get session. RequireAuth middleware guarantees an
// authenticated session before this handler runs, so we
// only need to guard against an unexpected retrieval error.
sess, err := h.session.Get(r)
if err != nil {
h.serverError(w, "failed to get session", err)
h.log.Error("failed to get session", "error", err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return "", "", false
return
}
// Get user info from session
sessionUsername, ok := h.session.GetUsername(sess)
if !ok {
h.log.Error("authenticated session missing username")
http.Error(
w, "Internal server error",
http.StatusInternalServerError,
)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return "", "", false
return
}
sessionUserID, ok := h.session.GetUserID(sess)
if !ok {
h.log.Error("authenticated session missing user ID")
http.Error(
w, "Internal server error",
http.StatusInternalServerError,
)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return "", "", false
return
}
// Only allow users to act on their own profile.
// For now, only allow users to view their own profile
if requestedUsername != sessionUsername {
http.Error(w, "Forbidden", http.StatusForbidden)
return "", "", false
return
}
return sessionUserID, sessionUsername, true
}
// renderProfile renders the profile page for the given user,
// optionally including a success or error message.
func (h *Handlers) renderProfile(
w http.ResponseWriter,
r *http.Request,
userID, username, successMessage, errorMessage string,
) {
// Prepare data for template
data := map[string]any{
"User": &UserInfo{
ID: userID,
Username: username,
ID: sessionUserID,
Username: sessionUsername,
},
"SuccessMessage": successMessage,
"ErrorMessage": errorMessage,
}
// Render the profile page
h.renderTemplate(w, r, "profile.html", data)
}
}

134
internal/handlers/profile_test.go
View File

@@ -4,15 +4,12 @@ import (
"context"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/logger"
"sneak.berlin/go/webhooker/internal/middleware"
@@ -160,134 +157,3 @@ func TestUserRoute_Unauthenticated_RedirectedByMiddleware(t *testing.T) {
assert.Equal(t, http.StatusSeeOther, w.Code)
assert.Equal(t, "/pages/login", w.Header().Get("Location"))
}
// passwordChangeRequest builds a POST request to the password-change
// endpoint for the given username, attaching the supplied cookies, an
// urlencoded form body, and the chi URL parameter the handler reads.
func passwordChangeRequest(
username string,
cookies []*http.Cookie,
form url.Values,
) *http.Request {
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost,
"/user/"+username+"/password",
strings.NewReader(form.Encode()),
)
req.Header.Set(
"Content-Type", "application/x-www-form-urlencoded",
)
for _, c := range cookies {
req.AddCookie(c)
}
rctx := chi.NewRouteContext()
rctx.URLParams.Add("username", username)
return req.WithContext(
context.WithValue(req.Context(), chi.RouteCtxKey, rctx),
)
}
func TestHandlePasswordChange_Success(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
var sess *session.Session
var db *database.Database
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
oldHash, err := database.HashPassword("oldpassword")
require.NoError(t, err)
user := &database.User{Username: "pwuser", Password: oldHash}
require.NoError(t, db.DB().Create(user).Error)
cookies := authenticatedCookies(t, sess, user.ID, "pwuser")
form := url.Values{}
form.Set("current_password", "oldpassword")
form.Set("new_password", "newpassword")
form.Set("confirm_password", "newpassword")
req := passwordChangeRequest("pwuser", cookies, form)
w := httptest.NewRecorder()
h.HandlePasswordChange().ServeHTTP(w, req)
assert.Equal(t, http.StatusOK, w.Code)
assert.Contains(
t, w.Body.String(), "Password changed successfully.",
)
var updated database.User
require.NoError(t,
db.DB().Where("username = ?", "pwuser").First(&updated).Error,
)
assert.NotEqual(t, oldHash, updated.Password)
valid, err := database.VerifyPassword(
"newpassword", updated.Password,
)
require.NoError(t, err)
assert.True(t, valid, "new password should verify against new hash")
}
func TestHandlePasswordChange_WrongCurrentPassword(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
var sess *session.Session
var db *database.Database
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
oldHash, err := database.HashPassword("oldpassword")
require.NoError(t, err)
user := &database.User{Username: "pwuser2", Password: oldHash}
require.NoError(t, db.DB().Create(user).Error)
cookies := authenticatedCookies(t, sess, user.ID, "pwuser2")
form := url.Values{}
form.Set("current_password", "wrongpassword")
form.Set("new_password", "newpassword")
form.Set("confirm_password", "newpassword")
req := passwordChangeRequest("pwuser2", cookies, form)
w := httptest.NewRecorder()
h.HandlePasswordChange().ServeHTTP(w, req)
assert.Equal(t, http.StatusOK, w.Code)
assert.Contains(
t, w.Body.String(), "Current password is incorrect.",
)
var unchanged database.User
require.NoError(t,
db.DB().Where(
"username = ?", "pwuser2",
).First(&unchanged).Error,
)
assert.Equal(
t, oldHash, unchanged.Password,
"stored hash must be unchanged after a rejected change",
)
}

356
internal/handlers/source_delete_test.go
View File

@@ -1,356 +0,0 @@
package handlers_test
import (
"context"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
const (
deleteTestUserID = "test-user-id"
deleteTestUsername = "testuser"
// paramSourceID and paramTargetID are the chi URL parameter
// names the deletion handlers read.
paramSourceID = "sourceID"
paramTargetID = "targetID"
)
// seedWebhook inserts a webhook owned by the test user and
// returns it.
func seedWebhook(
t *testing.T,
db *database.Database,
) *database.Webhook {
t.Helper()
wh := &database.Webhook{
UserID: deleteTestUserID,
Name: "delete-me",
}
require.NoError(
t,
db.DB().Omit(clause.Associations).Create(wh).Error,
)
return wh
}
// seedTarget inserts a target of the given type for a webhook
// and returns it.
func seedTarget(
t *testing.T,
db *database.Database,
webhookID string,
targetType database.TargetType,
) *database.Target {
t.Helper()
tgt := &database.Target{
WebhookID: webhookID,
Name: "t-" + string(targetType),
Type: targetType,
Active: true,
}
require.NoError(
t,
db.DB().Omit(clause.Associations).Create(tgt).Error,
)
return tgt
}
// archivePathFor returns the archive database path the
// delivery engine would use for a webhook: beside the webhook's
// event database in the data directory.
func archivePathFor(
t *testing.T,
mgr *database.WebhookDBManager,
webhookID string,
) string {
t.Helper()
return filepath.Join(
filepath.Dir(mgr.DBPath(webhookID)),
"archive-"+webhookID+".db",
)
}
// writeArchivePlaceholder creates a stand-in archive file so a
// test can assert the file survives webhook deletion.
func writeArchivePlaceholder(path string) error {
return os.WriteFile(path, []byte("archive"), 0o600)
}
// postRequest builds an authenticated POST request carrying the
// given chi URL parameters.
func postRequest(
path string,
cookies []*http.Cookie,
params map[string]string,
) *http.Request {
req := httptest.NewRequestWithContext(
context.Background(), http.MethodPost, path, nil,
)
for _, c := range cookies {
req.AddCookie(c)
}
rctx := chi.NewRouteContext()
for k, v := range params {
rctx.URLParams.Add(k, v)
}
return req.WithContext(
context.WithValue(req.Context(), chi.RouteCtxKey, rctx),
)
}
// TestHandleSourceDelete_EvictsArchiveWriter proves that
// deleting a webhook reaches the delivery engine and releases
// the webhook's archive writer, exercised through the real
// deletion handler rather than by calling the evictor directly.
func TestHandleSourceDelete_EvictsArchiveWriter(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
ev *recordingEvictor
)
app := newTestApp(t, &h, &sess, &db, &ev)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedTarget(t, db, wh.ID, database.TargetTypeDatabase)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/delete",
cookies,
map[string]string{paramSourceID: wh.ID},
)
w := httptest.NewRecorder()
h.HandleSourceDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Equal(
t, []string{wh.ID}, ev.Evicted(),
"deleting a webhook should evict its archive writer",
)
}
// TestHandleSourceDelete_KeepsArchiveFile proves that deleting
// a webhook does not remove its archive database file: the
// archive is long-term storage the operator owns.
func TestHandleSourceDelete_KeepsArchiveFile(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
mgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &mgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
// Place an archive file where the delivery engine would.
archivePath := archivePathFor(t, mgr, wh.ID)
require.NoError(
t,
writeArchivePlaceholder(archivePath),
)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/delete",
cookies,
map[string]string{paramSourceID: wh.ID},
)
w := httptest.NewRecorder()
h.HandleSourceDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
assert.FileExists(
t, archivePath,
"webhook deletion must not destroy the archive file",
)
}
// TestHandleTargetDelete_EvictsWhenLastDatabaseTargetGone
// proves that removing the last database target releases the
// archive writer.
func TestHandleTargetDelete_EvictsWhenLastDatabaseTargetGone(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
ev *recordingEvictor
)
app := newTestApp(t, &h, &sess, &db, &ev)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
tgt := seedTarget(
t, db, wh.ID, database.TargetTypeDatabase,
)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/targets/"+tgt.ID+"/delete",
cookies,
map[string]string{
paramSourceID: wh.ID,
paramTargetID: tgt.ID,
},
)
w := httptest.NewRecorder()
h.HandleTargetDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Equal(
t, []string{wh.ID}, ev.Evicted(),
"removing the last database target should evict",
)
}
// TestHandleTargetDelete_KeepsWriterWhenDatabaseTargetRemains
// proves that deleting one of several database targets leaves
// the still-needed archive writer alone: the surviving target
// keeps archiving to the same file, so the writer must stay.
func TestHandleTargetDelete_KeepsWriterWhenDatabaseTargetRemains(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
ev *recordingEvictor
)
app := newTestApp(t, &h, &sess, &db, &ev)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
doomed := seedTarget(
t, db, wh.ID, database.TargetTypeDatabase,
)
seedTarget(t, db, wh.ID, database.TargetTypeDatabase)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/targets/"+doomed.ID+"/delete",
cookies,
map[string]string{
paramSourceID: wh.ID,
paramTargetID: doomed.ID,
},
)
w := httptest.NewRecorder()
h.HandleTargetDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Empty(
t, ev.Evicted(),
"a second database target still needs the writer",
)
}
// TestHandleTargetDelete_KeepsWriterWhenOtherTypeDeleted proves
// that deleting a target of an unrelated type leaves a
// still-needed archive writer alone: the webhook's database
// target is untouched, so its writer must stay.
func TestHandleTargetDelete_KeepsWriterWhenOtherTypeDeleted(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
ev *recordingEvictor
)
app := newTestApp(t, &h, &sess, &db, &ev)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedTarget(t, db, wh.ID, database.TargetTypeDatabase)
other := seedTarget(t, db, wh.ID, database.TargetTypeLog)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/targets/"+other.ID+"/delete",
cookies,
map[string]string{
paramSourceID: wh.ID,
paramTargetID: other.ID,
},
)
w := httptest.NewRecorder()
h.HandleTargetDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Empty(
t, ev.Evicted(),
"a surviving database target must keep its writer",
)
}

173
internal/handlers/source_management.go
View File

@@ -5,7 +5,6 @@ import (
"errors"
"net/http"
"strconv"
"strings"
"github.com/go-chi/chi"
"github.com/google/uuid"
@@ -107,7 +106,7 @@ func (h *Handlers) buildWebhookListItems(
func (h *Handlers) HandleSourceCreate() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
data := map[string]any{
tmplKeyError: "",
"Error": "",
}
h.renderTemplate(w, r, "sources_new.html", data)
@@ -146,7 +145,7 @@ func (h *Handlers) HandleSourceCreateSubmit() http.HandlerFunc {
if name == "" {
data := map[string]any{
tmplKeyError: "Name is required",
"Error": "Name is required",
}
w.WriteHeader(http.StatusBadRequest)
@@ -316,7 +315,7 @@ func (h *Handlers) renderSourceDetail(
}
data := map[string]any{
tmplKeyWebhook: webhook,
"Webhook": webhook,
"Entrypoints": entrypoints,
"Targets": targets,
"Events": events,
@@ -352,8 +351,8 @@ func (h *Handlers) HandleSourceEdit() http.HandlerFunc {
}
data := map[string]any{
tmplKeyWebhook: webhook,
tmplKeyError: "",
"Webhook": webhook,
"Error": "",
}
h.renderTemplate(w, r, "source_edit.html", data)
@@ -416,8 +415,8 @@ func (h *Handlers) applyWebhookEdit(
name := r.FormValue("name")
if name == "" {
data := map[string]any{
tmplKeyWebhook: *webhook,
tmplKeyError: "Name is required",
"Webhook": *webhook,
"Error": "Name is required",
}
w.WriteHeader(http.StatusBadRequest)
@@ -533,13 +532,6 @@ func (h *Handlers) deleteWebhookResources(
return
}
// Release the delivery engine's per-webhook archiving state
// so a deleted webhook's archive writer (and any handle open
// within its debounce window) does not linger for the
// process lifetime. The archive file itself is deliberately
// left on disk; see evictArchiveWriter.
h.evictArchiveWriter(webhook.ID)
err = h.dbMgr.DeleteDB(webhook.ID)
if err != nil {
h.log.Error(
@@ -558,64 +550,6 @@ func (h *Handlers) deleteWebhookResources(
http.Redirect(w, r, "/sources", http.StatusSeeOther)
}
// evictArchiveWriter asks the delivery engine to drop its
// cached archive writer for a webhook, closing the archive file
// handle.
//
// The archive database file is NOT deleted. Unlike the event
// database — which is per-webhook working storage and is
// hard-deleted with the webhook — an archive is explicitly
// long-term storage that an operator may want to keep or move
// away for offline retention. Destroying it as a side effect of
// deleting a webhook would be a surprising and unrecoverable
// data loss, so the file is left for the operator to handle.
func (h *Handlers) evictArchiveWriter(webhookID string) {
if h.evictor == nil {
return
}
h.evictor.EvictWebhook(webhookID)
}
// evictArchiveWriterIfUnused releases a webhook's archive
// writer once the webhook has no database target left to feed
// it.
//
// It is called after any child resource of a webhook is
// deleted, and is correct without knowing which kind was: it
// evicts only when no database target remains, so deleting one
// of several database targets — or deleting an unrelated
// target type — leaves a still-needed writer alone. When no
// database target ever existed there is no writer and eviction
// is a no-op. Soft-deleted targets are excluded by GORM's
// default scope, so the row just deleted is not counted.
func (h *Handlers) evictArchiveWriterIfUnused(webhookID string) {
var remaining int64
err := h.db.DB().
Model(&database.Target{}).
Where(
"webhook_id = ? AND type = ?",
webhookID, database.TargetTypeDatabase,
).
Count(&remaining).Error
if err != nil {
h.log.Error(
"failed to count remaining database targets",
"webhook_id", webhookID,
"error", err,
)
return
}
if remaining > 0 {
return
}
h.evictArchiveWriter(webhookID)
}
// HandleSourceLogs shows the request/response logs for a
// webhook.
func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
@@ -655,7 +589,7 @@ func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
}
data := map[string]any{
tmplKeyWebhook: webhook,
"Webhook": webhook,
"Events": evts,
"Page": page,
"TotalPages": totalPages,
@@ -881,7 +815,6 @@ func (h *Handlers) processTargetCreate(
targetType := database.TargetType(r.FormValue("type"))
targetURL := r.FormValue("url")
maxRetriesStr := r.FormValue("max_retries")
expiry := r.FormValue("expiry")
if name == "" {
http.Error(
@@ -901,7 +834,7 @@ func (h *Handlers) processTargetCreate(
}
configJSON, err := h.buildTargetConfig(
w, r, targetType, targetURL, expiry,
w, r, targetType, targetURL,
)
if err != nil {
return
@@ -959,28 +892,18 @@ func parseNonNegativeInt(s string) int {
}
// buildTargetConfig builds the JSON config string for a target.
// The expiry form value is read by the caller (which bounds the
// request body) and applies to database targets only.
func (h *Handlers) buildTargetConfig(
w http.ResponseWriter,
r *http.Request,
targetType database.TargetType,
targetURL, expiry string,
targetURL string,
) (string, error) {
switch targetType {
case database.TargetTypeHTTP:
return h.buildURLTargetConfig(
w, r, targetURL, "url",
"URL is required for HTTP targets",
)
return h.buildHTTPTargetConfig(w, r, targetURL)
case database.TargetTypeSlack:
return h.buildURLTargetConfig(
w, r, targetURL, "webhookUrl",
"Webhook URL is required for Slack targets",
)
case database.TargetTypeDatabase:
return h.buildDatabaseTargetConfig(w, expiry)
case database.TargetTypeLog:
return h.buildSlackTargetConfig(w, r, targetURL)
case database.TargetTypeDatabase, database.TargetTypeLog:
return "", nil
default:
http.Error(
@@ -992,18 +915,16 @@ func (h *Handlers) buildTargetConfig(
}
}
// buildURLTargetConfig builds config JSON for a target whose
// configuration is a single SSRF-validated URL stored under
// configKey. missingMsg is the error shown when no URL is given.
func (h *Handlers) buildURLTargetConfig(
// buildHTTPTargetConfig builds config JSON for an HTTP target.
func (h *Handlers) buildHTTPTargetConfig(
w http.ResponseWriter,
r *http.Request,
targetURL, configKey, missingMsg string,
targetURL string,
) (string, error) {
if targetURL == "" {
http.Error(
w,
missingMsg,
"URL is required for HTTP targets",
http.StatusBadRequest,
)
@@ -1028,7 +949,7 @@ func (h *Handlers) buildURLTargetConfig(
return "", err
}
cfg := map[string]any{configKey: targetURL}
cfg := map[string]any{"url": targetURL}
configBytes, err := json.Marshal(cfg)
if err != nil {
@@ -1043,33 +964,41 @@ func (h *Handlers) buildURLTargetConfig(
return string(configBytes), nil
}
// buildDatabaseTargetConfig builds config JSON for a database
// (archive) target. The optional expiry (a form value read by
// the caller, which bounds the request body) is validated here,
// at creation time, so an unparseable value is rejected with a
// 400 instead of failing every subsequent delivery. An empty
// expiry yields an empty config (the keep-forever default).
func (h *Handlers) buildDatabaseTargetConfig(
// buildSlackTargetConfig builds config JSON for a Slack target.
func (h *Handlers) buildSlackTargetConfig(
w http.ResponseWriter,
expiry string,
r *http.Request,
targetURL string,
) (string, error) {
expiry = strings.TrimSpace(expiry)
if expiry == "" {
return "", nil
}
err := delivery.ValidateArchiveExpiry(expiry)
if err != nil {
if targetURL == "" {
http.Error(
w,
"Invalid archive expiry: "+err.Error(),
"Webhook URL is required for Slack targets",
http.StatusBadRequest,
)
return "", errMissingURL
}
err := delivery.ValidateTargetURL(
r.Context(), targetURL,
)
if err != nil {
h.log.Warn(
"target URL blocked by SSRF protection",
"url", targetURL,
"error", err,
)
http.Error(
w,
"Invalid target URL: "+err.Error(),
http.StatusBadRequest,
)
return "", err
}
cfg := map[string]any{"expiry": expiry}
cfg := map[string]any{"webhookUrl": targetURL}
configBytes, err := json.Marshal(cfg)
if err != nil {
@@ -1089,31 +1018,23 @@ func (h *Handlers) HandleEntrypointDelete() http.HandlerFunc {
return h.deleteChildResource(
"entrypointID", &database.Entrypoint{},
"failed to delete entrypoint",
nil,
)
}
// HandleTargetDelete handles deleting a target. Deleting the
// last database target of a webhook leaves its archive writer
// with nothing to write, so the writer is evicted and its
// handle closed; the archive file is left on disk.
// HandleTargetDelete handles deleting a target.
func (h *Handlers) HandleTargetDelete() http.HandlerFunc {
return h.deleteChildResource(
"targetID", &database.Target{},
"failed to delete target",
h.evictArchiveWriterIfUnused,
)
}
// deleteChildResource returns a handler that deletes a child
// resource (entrypoint or target) belonging to a webhook. The
// optional afterDelete hook runs with the webhook's id once the
// delete has succeeded, before the redirect.
// resource (entrypoint or target) belonging to a webhook.
func (h *Handlers) deleteChildResource(
idParam string,
model any,
errMsg string,
afterDelete func(webhookID string),
) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
userID, ok := h.getUserID(r)
@@ -1153,10 +1074,6 @@ func (h *Handlers) deleteChildResource(
return
}
if afterDelete != nil {
afterDelete(webhook.ID)
}
http.Redirect(
w, r,
"/source/"+webhook.ID,

1
internal/handlers/webhook.go
View File

@@ -329,7 +329,6 @@ func (h *Handlers) buildDeliveryTasks(
DeliveryID: dlv.ID,
EventID: event.ID,
WebhookID: entrypoint.WebhookID,
EntrypointID: entrypoint.ID,
TargetID: targets[i].ID,
TargetName: targets[i].Name,
TargetType: targets[i].Type,

4
internal/middleware/export_test.go
View File

@@ -32,7 +32,3 @@ func IsClientTLS(r *http.Request) bool {
// LoginRateLimitConst exposes the loginRateLimit constant.
const LoginRateLimitConst = loginRateLimit
// PasswordChangeRateLimitConst exposes the
// passwordChangeRateLimit constant.
const PasswordChangeRateLimitConst = passwordChangeRateLimit

20
internal/middleware/middleware.go
View File

@@ -265,26 +265,6 @@ func (s *Middleware) SecurityHeaders() func(http.Handler) http.Handler {
}
}
// NoCache returns middleware that instructs browsers and
// intermediary proxies not to cache the response. It sets
// Cache-Control: no-store and Pragma: no-cache (the latter for
// older HTTP/1.0 intermediaries). Apply it to authenticated pages
// so webhook configuration and captured event data are not stored
// by caches.
func (s *Middleware) NoCache() func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(
w http.ResponseWriter,
r *http.Request,
) {
w.Header().Set("Cache-Control", "no-store")
w.Header().Set("Pragma", "no-cache")
next.ServeHTTP(w, r)
})
}
}
// MaxBodySize returns middleware that limits the request body size
// for POST requests. If the body exceeds the given limit in
// bytes, the server returns 413 Request Entity Too Large. This

80
internal/middleware/middleware_test.go
View File

@@ -387,45 +387,6 @@ func TestRequireAuth_UnauthenticatedSession_RedirectsToLogin(
assert.Equal(t, "/pages/login", w.Header().Get("Location"))
}
// --- NoCache Middleware Tests ---
func TestNoCache_SetsHeaders(t *testing.T) {
t.Parallel()
m, _ := testMiddleware(t, config.EnvironmentDev)
var called bool
handler := m.NoCache()(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
called = true
w.WriteHeader(http.StatusOK)
},
))
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet, "/sources", nil,
)
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
assert.True(
t, called,
"NoCache middleware should call the next handler",
)
assert.Equal(
t, "no-store",
w.Header().Get("Cache-Control"),
)
assert.Equal(
t, "no-cache",
w.Header().Get("Pragma"),
)
}
// --- Helper Tests ---
func TestIpFromHostPort(t *testing.T) {
@@ -484,13 +445,8 @@ func metricsAuthMiddleware(
return middleware.NewForTest(log, cfg, sessManager)
}
// runMetricsAuthRequest sends a GET /metrics request with the
// given basic-auth password through MetricsAuth and reports
// whether the wrapped handler ran plus the recorded response.
func runMetricsAuthRequest(
t *testing.T, password string,
) (bool, *httptest.ResponseRecorder) {
t.Helper()
func TestMetricsAuth_ValidCredentials(t *testing.T) {
t.Parallel()
m := metricsAuthMiddleware(t)
@@ -508,20 +464,12 @@ func runMetricsAuthRequest(
context.Background(),
http.MethodGet, "/metrics", nil,
)
req.SetBasicAuth("admin", password)
req.SetBasicAuth("admin", "secret")
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
return called, w
}
func TestMetricsAuth_ValidCredentials(t *testing.T) {
t.Parallel()
called, w := runMetricsAuthRequest(t, "secret")
assert.True(
t, called,
"handler should be called with valid basic auth",
@@ -532,7 +480,27 @@ func TestMetricsAuth_ValidCredentials(t *testing.T) {
func TestMetricsAuth_InvalidCredentials(t *testing.T) {
t.Parallel()
called, w := runMetricsAuthRequest(t, "wrong-password")
m := metricsAuthMiddleware(t)
var called bool
handler := m.MetricsAuth()(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
called = true
w.WriteHeader(http.StatusOK)
},
))
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet, "/metrics", nil,
)
req.SetBasicAuth("admin", "wrong-password")
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
assert.False(
t, called,

55
internal/middleware/ratelimit.go
View File

@@ -14,16 +14,6 @@ const (
// loginRateInterval is the time window for the rate limit.
loginRateInterval = 1 * time.Minute
// passwordChangeRateLimit is the maximum number of password
// change attempts per interval. Each attempt verifies the
// current password, so the endpoint must be rate-limited
// like any other password-based authentication endpoint.
passwordChangeRateLimit = 5
// passwordChangeRateInterval is the time window for the
// password change rate limit.
passwordChangeRateInterval = 1 * time.Minute
)
// LoginRateLimit returns middleware that enforces per-IP rate
@@ -34,53 +24,19 @@ const (
// honours X-Forwarded-For, X-Real-IP, and True-Client-IP headers
// for reverse-proxy setups.
func (m *Middleware) LoginRateLimit() func(http.Handler) http.Handler {
return m.postRateLimit(
limiter := httprate.Limit(
loginRateLimit,
loginRateInterval,
"login rate limit exceeded",
"Too many login attempts. Please try again later.",
)
}
// PasswordChangeRateLimit returns middleware that enforces
// per-IP rate limiting on password change attempts. The change
// endpoint verifies the current password, so without a limit a
// stolen session could be used to brute-force it; the limit
// matches the login endpoint's.
func (m *Middleware) PasswordChangeRateLimit() func(http.Handler) http.Handler {
return m.postRateLimit(
passwordChangeRateLimit,
passwordChangeRateInterval,
"password change rate limit exceeded",
"Too many password change attempts. "+
"Please try again later.",
)
}
// postRateLimit builds middleware that enforces a per-IP rate
// limit on POST requests only; all other methods pass through
// unaffected. Requests over the limit receive a 429 with the
// given response message, and each rejection is logged with the
// given log message. IP extraction honours X-Forwarded-For,
// X-Real-IP, and True-Client-IP headers for reverse-proxy
// setups.
func (m *Middleware) postRateLimit(
limit int,
interval time.Duration,
logMessage, responseMessage string,
) func(http.Handler) http.Handler {
limiter := httprate.Limit(
limit,
interval,
httprate.WithKeyFuncs(httprate.KeyByRealIP),
httprate.WithLimitHandler(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
m.log.Warn(logMessage,
m.log.Warn("login rate limit exceeded",
"path", r.URL.Path,
)
http.Error(
w,
responseMessage,
"Too many login attempts. "+
"Please try again later.",
http.StatusTooManyRequests,
)
},
@@ -94,7 +50,8 @@ func (m *Middleware) postRateLimit(
w http.ResponseWriter,
r *http.Request,
) {
// Only rate-limit POST requests.
// Only rate-limit POST requests (actual login
// attempts)
if r.Method != http.MethodPost {
next.ServeHTTP(w, r)

58
internal/middleware/ratelimit_test.go
View File

@@ -46,20 +46,14 @@ func TestLoginRateLimit_AllowsGET(t *testing.T) {
assert.Equal(t, 20, callCount)
}
// runPostLimitTest exercises a POST-only rate limit middleware:
// the first limit POSTs to path from ip must pass, and the next
// one must be rejected with 429 without reaching the handler.
func runPostLimitTest(
t *testing.T,
mw func(http.Handler) http.Handler,
limit int,
path, ip string,
) {
t.Helper()
func TestLoginRateLimit_LimitsPOST(t *testing.T) {
t.Parallel()
m, _ := testMiddleware(t, config.EnvironmentDev)
var callCount int
handler := mw(http.HandlerFunc(
handler := m.LoginRateLimit()(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
callCount++
@@ -67,13 +61,13 @@ func runPostLimitTest(
},
))
// The first limit POST requests should succeed
for i := range limit {
// First loginRateLimit POST requests should succeed
for i := range middleware.LoginRateLimitConst {
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost, path, nil,
http.MethodPost, "/pages/login", nil,
)
req.RemoteAddr = ip
req.RemoteAddr = "10.0.0.1:12345"
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
@@ -87,9 +81,9 @@ func runPostLimitTest(
// Next POST should be rate-limited
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost, path, nil,
http.MethodPost, "/pages/login", nil,
)
req.RemoteAddr = ip
req.RemoteAddr = "10.0.0.1:12345"
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
@@ -98,35 +92,7 @@ func runPostLimitTest(
t, http.StatusTooManyRequests, w.Code,
"POST after limit should be 429",
)
assert.Equal(t, limit, callCount)
}
func TestLoginRateLimit_LimitsPOST(t *testing.T) {
t.Parallel()
m, _ := testMiddleware(t, config.EnvironmentDev)
runPostLimitTest(
t,
m.LoginRateLimit(),
middleware.LoginRateLimitConst,
"/pages/login",
"10.0.0.1:12345",
)
}
func TestPasswordChangeRateLimit_LimitsPOST(t *testing.T) {
t.Parallel()
m, _ := testMiddleware(t, config.EnvironmentDev)
runPostLimitTest(
t,
m.PasswordChangeRateLimit(),
middleware.PasswordChangeRateLimitConst,
"/user/admin/password",
"10.0.0.2:12345",
)
assert.Equal(t, middleware.LoginRateLimitConst, callCount)
}
func TestLoginRateLimit_IndependentPerIP(t *testing.T) {

7
internal/server/routes.go
View File

@@ -91,7 +91,6 @@ func (s *Server) setupRoutes() {
func (s *Server) setupPageRoutes() {
s.router.Route("/pages", func(r chi.Router) {
r.Use(s.mw.CSRF())
r.Use(s.mw.NoCache())
r.Use(s.mw.MaxBodySize(maxFormBodySize))
r.Group(func(r chi.Router) {
@@ -107,19 +106,14 @@ func (s *Server) setupPageRoutes() {
func (s *Server) setupUserRoutes() {
s.router.Route("/user/{username}", func(r chi.Router) {
r.Use(s.mw.CSRF())
r.Use(s.mw.NoCache())
r.Use(s.mw.RequireAuth())
r.Get("/", s.h.HandleProfile())
r.With(s.mw.PasswordChangeRateLimit()).Post(
"/password", s.h.HandlePasswordChange(),
)
})
}
func (s *Server) setupSourceRoutes() {
s.router.Route("/sources", func(r chi.Router) {
r.Use(s.mw.CSRF())
r.Use(s.mw.NoCache())
r.Use(s.mw.RequireAuth())
r.Use(s.mw.MaxBodySize(maxFormBodySize))
r.Get("/", s.h.HandleSourceList())
@@ -129,7 +123,6 @@ func (s *Server) setupSourceRoutes() {
s.router.Route("/source/{sourceID}", func(r chi.Router) {
r.Use(s.mw.CSRF())
r.Use(s.mw.NoCache())
r.Use(s.mw.RequireAuth())
r.Use(s.mw.MaxBodySize(maxFormBodySize))
r.Get("/", s.h.HandleSourceDetail())

66
internal/session/session_test.go
View File

@@ -173,18 +173,8 @@ func TestSetUser_SetsAllFields(t *testing.T) {
)
}
// testSessionGetter exercises a session string getter before and
// after SetUser: it must report false with an empty value on a
// fresh session, then true with the expected value once
// SetUser(sess, "user-xyz", "bob") has run.
func testSessionGetter(
t *testing.T,
get func(
*session.Session, *sessions.Session,
) (string, bool),
expected string,
) {
t.Helper()
func TestGetUserID(t *testing.T) {
t.Parallel()
s := testSession(t)
@@ -195,46 +185,44 @@ func testSessionGetter(
require.NoError(t, err)
// Before setting user
val, ok := get(s, sess)
userID, ok := s.GetUserID(sess)
assert.False(
t, ok, "should return false before SetUser",
t, ok, "should return false when no user ID is set",
)
assert.Empty(t, val)
assert.Empty(t, userID)
// After setting user
s.SetUser(sess, "user-xyz", "bob")
val, ok = get(s, sess)
userID, ok = s.GetUserID(sess)
assert.True(t, ok)
assert.Equal(t, expected, val)
}
func TestGetUserID(t *testing.T) {
t.Parallel()
testSessionGetter(
t,
func(
s *session.Session, sess *sessions.Session,
) (string, bool) {
return s.GetUserID(sess)
},
"user-xyz",
)
assert.Equal(t, "user-xyz", userID)
}
func TestGetUsername(t *testing.T) {
t.Parallel()
testSessionGetter(
t,
func(
s *session.Session, sess *sessions.Session,
) (string, bool) {
return s.GetUsername(sess)
},
"bob",
s := testSession(t)
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/", nil)
sess, err := s.Get(req)
require.NoError(t, err)
// Before setting user
username, ok := s.GetUsername(sess)
assert.False(
t, ok, "should return false when no username is set",
)
assert.Empty(t, username)
// After setting user
s.SetUser(sess, "user-xyz", "bob")
username, ok = s.GetUsername(sess)
assert.True(t, ok)
assert.Equal(t, "bob", username)
}
// --- IsAuthenticated Tests ---

7
internal/session/testing.go
View File

@@ -12,12 +12,7 @@ import (
// middleware and handler tests to use real session functionality. The key
// parameter is the raw 32-byte authentication key used for session encryption
// and CSRF cookie signing.
func NewForTest(
store *sessions.CookieStore,
cfg *config.Config,
log *slog.Logger,
key []byte,
) *Session {
func NewForTest(store *sessions.CookieStore, cfg *config.Config, log *slog.Logger, key []byte) *Session {
return &Session{
store: store,
key: key,

10
script/bootstrap
View File

@@ -10,11 +10,11 @@ set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Pinned versions, 2026-08-07. Never "latest"; exact versions only.
GOLANGCI_LINT_VERSION="2.12.2"
# sha256 of golangci-lint-2.12.2-linux-<arch>.tar.gz release archives
GOLANGCI_LINT_SHA256_AMD64="8df580d2670fed8fa984aac0507099af8df275e665215f5c7a2ae3943893a553"
GOLANGCI_LINT_SHA256_ARM64="44cd40a8c76c86755375adfeea52cfd3533cb43d7bd647771e0ae065e166df3a"
# Pinned versions, 2026-07-07. Never "latest"; exact versions only.
GOLANGCI_LINT_VERSION="2.11.3"
# sha256 of golangci-lint-2.11.3-linux-<arch>.tar.gz release archives
GOLANGCI_LINT_SHA256_AMD64="87bb8cddbcc825d5778b64e8a91b46c0526b247f4e2f2904dea74ec7450475d1"
GOLANGCI_LINT_SHA256_ARM64="ee3d95f301359e7d578e6d99c8ad5aeadbabc5a13009a30b2b0df11c8058afe9"
PKGMGR=""
SUDO=""

56
templates/profile.html
View File

@@ -6,18 +6,6 @@
<div class="max-w-4xl mx-auto px-6 py-12">
<h1 class="text-2xl font-medium text-gray-900 mb-6">User Profile</h1>
{{if .SuccessMessage}}
<div class="alert-success">
<span>{{.SuccessMessage}}</span>
</div>
{{end}}
{{if .ErrorMessage}}
<div class="alert-error">
<span>{{.ErrorMessage}}</span>
</div>
{{end}}
<div class="card p-6">
<div class="flex items-center mb-6">
<div class="mr-4">
@@ -55,50 +43,6 @@
</div>
</div>
<div class="card p-6 mt-6">
<h3 class="text-lg font-medium text-gray-900 mb-3">Change Password</h3>
<form method="POST" action="/user/{{.User.Username}}/password" class="space-y-6">
<input type="hidden" name="csrf_token" value="{{.CSRFToken}}">
<div class="form-group">
<label for="current_password" class="label">Current Password</label>
<input
type="password"
id="current_password"
name="current_password"
required
autocomplete="current-password"
class="input"
>
</div>
<div class="form-group">
<label for="new_password" class="label">New Password</label>
<input
type="password"
id="new_password"
name="new_password"
required
autocomplete="new-password"
class="input"
>
</div>
<div class="form-group">
<label for="confirm_password" class="label">Confirm New Password</label>
<input
type="password"
id="confirm_password"
name="confirm_password"
required
autocomplete="new-password"
class="input"
>
</div>
<button type="submit" class="btn-primary">Change Password</button>
</form>
</div>
<div class="mt-6">
<a href="/" class="btn-secondary">Back to Home</a>
</div>

4
templates/source_detail.html
View File

@@ -113,10 +113,6 @@
<input type="url" name="url" placeholder="https://hooks.slack.com/services/..." :disabled="targetType !== 'slack'" class="input text-sm">
<p class="text-xs text-gray-500 mt-1">Slack or Mattermost incoming webhook URL. Payloads are pretty-printed in code blocks.</p>
</div>
<div x-show="targetType === 'database'">
<input type="text" name="expiry" placeholder="never" :disabled="targetType !== 'database'" class="input text-sm">
<p class="text-xs text-gray-500 mt-1">Archive expiry: "never" (default) keeps rows forever, or a duration like "720h" prunes older rows.</p>
</div>
<button type="submit" class="btn-primary text-sm">Add Target</button>
</form>
</div>