sneak/webhooker
1
0
Fork 0
Code Issues 22 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Compare commits

base: sneak:1dd0729ce8b882d495e1612cf4bc1dec2e1a8e81
Branches Tags
sneak:main sneak:issue-88-trusted-proxy-gating sneak:next sneak:feat/receiver-rate-limit sneak:issue-70-log-target
..
compare: sneak:feat/receiver-rate-limit
Branches Tags
sneak:issue-88-trusted-proxy-gating sneak:next sneak:main sneak:feat/receiver-rate-limit sneak:issue-70-log-target

1 Commits
1dd0729ce8 ... feat/recei

Author SHA1 Message Date
sneak
8cf9d0525a feat: add receiver rate limiting (refs #64)
Some checks failed
check / check (push) Failing after 59s
Details
2026-08-07 18:32:00 +00:00
62 changed files with 600 additions and 9774 deletions
Expand all files Collapse all files

8
.golangci.yml
View File

@@ -1,9 +1,5 @@
version: "2"
# Config schema uses the golangci-lint v2 layout (settings live under
# linters.settings, not top-level linters-settings) so that the
# thresholds below are actually applied by golangci-lint >= v2.
run:
timeout: 5m
modules-download-mode: readonly
@@ -18,7 +14,8 @@ linters:
- wsl # Deprecated, replaced by wsl_v5
- wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go
settings:
linters-settings:
lll:
line-length: 88
funlen:
@@ -30,5 +27,6 @@ linters:
threshold: 100
issues:
exclude-use-default: false
max-issues-per-linter: 0
max-same-issues: 0

4
Dockerfile
View File

@@ -1,8 +1,8 @@
# Lint stage
# golangci/golangci-lint:v2.12.2 (Debian-based), 2026-08-07
# golangci/golangci-lint:v2.11.3 (Debian-based), 2026-03-17
# Using Debian-based image because mattn/go-sqlite3 (CGO) does not
# compile on Alpine musl (off64_t is a glibc type).
FROM golangci/golangci-lint:v2.12.2@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
FROM golangci/golangci-lint:v2.11.3@sha256:e838e8ab68aaefe83e2408691510867ade9329c0e0b895a3fb35eb93d1c2a4ba AS lint
RUN apt-get update && apt-get install -y --no-install-recommends make && rm -rf /var/lib/apt/lists/*

182
README.md
View File

@@ -89,72 +89,10 @@ TTY detection, and security headers are always applied.
| `PORT` | HTTP listen port | `8080` |
| `DATA_DIR` | Directory for all SQLite databases | `/var/lib/webhooker` |
| `DEBUG` | Enable debug logging | `false` |
| `MAINTENANCE_MODE` | Serve the maintenance page | `false` |
| `METRICS_USERNAME` | Basic auth username for `/metrics` | `""` |
| `METRICS_PASSWORD` | Basic auth password for `/metrics` | `""` |
| `SENTRY_DSN` | Sentry error reporting DSN | `""` |
| `SESSION_IDLE_TIMEOUT` | Idle session timeout (Go duration) | `24h` |
| `RECEIVER_RATE_LIMIT` | Receiver requests/minute per IP per entrypoint | `120` |
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted | `""` (none) |
#### Trusted proxies
`TRUSTED_PROXIES` is a comma-separated list of CIDR blocks (a bare
address such as `10.0.0.1` is accepted and treated as a single host),
for example `10.0.0.0/8, 192.168.1.7, 2001:db8::/32`. It decides whose
`X-Forwarded-For`, `X-Real-IP`, and `True-Client-IP` headers the rate
limiters believe.
Forwarded headers are honoured **only** when the connecting peer is
inside one of these blocks; for every other peer the client identity is
the connection's own address and the headers are ignored. The default
is the empty list, which trusts nobody — anything else would let any
client pick its own rate limit bucket, minting a fresh one per request
or draining someone else's. Set it to the address of your reverse
proxy, and to nothing wider.
Within a trusted request, `X-Forwarded-For` is read right to left and
the first hop that is not itself a trusted proxy wins, so entries a
client prepended before reaching the proxy cannot be selected. A set
but unparseable value aborts startup.
Sessions are bounded by two independent clocks, and end at whichever
one runs out first:
- **Idle expiry** (`SESSION_IDLE_TIMEOUT`, default `24h`) is a sliding
window. Every authenticated request pushes it forward, so a session
in continuous use never hits it, while an abandoned one expires a day
after its last use. Set it to `0` to disable idle expiry entirely;
the absolute cap below still applies. A set-but-unparseable value
aborts startup rather than silently falling back to the default.
- **Absolute expiry** is a fixed 7 days from login. Activity does
**not** extend it: after a week, every session ends and the user
authenticates again.
Only requests that authenticate with the session count as activity, so
an unauthenticated request carrying the cookie cannot keep a session
alive. The idle timestamp is rewritten at most once per tenth of the
idle window rather than on every request, which means a session may
expire up to 10% early relative to the user's true last request, but
never late.
#### Invalid values abort startup
The defaults above apply **only** to variables that are unset (or set
to an empty string). A variable that is set but cannot be parsed is a
fatal configuration error: webhooker logs the offending variable and
its value and refuses to start, rather than silently running with a
substituted default. `PORT=eighty`, `DEBUG=ture`, and
`RETENTION_SWEEP_INTERVAL=1 hour` all abort startup. `PORT` must
additionally be a number in the range 1–65535,
`RECEIVER_RATE_LIMIT` must be at least 1, and every entry in
`TRUSTED_PROXIES` must be a CIDR block or a bare IP address.
Boolean variables (`DEBUG`, `MAINTENANCE_MODE`) accept exactly the
spellings Go's `strconv.ParseBool` accepts — `1`, `t`, `T`, `TRUE`,
`true`, `True`, `0`, `f`, `F`, `FALSE`, `false`, `False` — and nothing
else. `yes`, `on`, and `off` are rejected rather than quietly treated
as false.
On first startup, webhooker automatically generates a cryptographically
secure session encryption key and stores it in the database. This key
@@ -370,29 +308,13 @@ event routing.
| `user_id` | UUID | Foreign key → User |
| `name` | string | Human-readable name |
| `description` | string | Optional description |
| `retention_days` | integer | Days to retain events (default: 30; 0 means retain forever) |
| `retention_days` | integer | Days to retain events (default: 30) |
**Relations:** Belongs to User. Has many Entrypoints. Has many Targets.
The `retention_days` field controls how long event data is kept in the
webhook's dedicated database before automatic cleanup.
Setting `retention_days` to `0` means "retain events forever". Because
the column carries a default of 30, a literal zero cannot survive an
insert, so a zero is rewritten on save to a sentinel of `365 * 1000`
days (`database.RetentionForeverDays`). The retention reaper recognises
that sentinel and skips the webhook entirely, and the web UI displays
such a webhook's retention as "forever" rather than as a day count.
A *finite* retention is capped at `database.MaxFiniteRetentionDays`
(106751 days, about 292 years), and a larger one is rejected with a
400. The cap is not arbitrary: the reaper computes its cutoff as a
`time.Duration`, an int64 nanosecond count, and a longer period
overflows it. An overflowed cutoff lands in the future, where it
matches every row, so the sweep would delete every event the webhook
has instead of none. The reaper also clamps the value it is given, so a
row written by an older version cannot trigger that either.
#### Entrypoint
A receiver URL where external services POST webhook events. Each
@@ -442,12 +364,10 @@ events should be forwarded.
greater than 0, failed deliveries are retried with exponential backoff
up to `max_retries` attempts, protected by a per-target circuit
breaker.
- **`database`** — Archive the full event as a row into a separate
per-webhook archive database (`archive-{webhookID}.db`) for long-term
retention, with an optional creation-validated expiry (default: keep
forever). No external delivery and no retries; an archive write
failure fails the delivery. See the database target section under
"Per-Webhook Event Databases" for the full semantics.
- **`database`** — Confirm the event is stored in the webhook's
per-webhook database (no external delivery). Since events are always
written to the per-webhook DB on ingestion, this target marks delivery
as immediately successful. Useful for ensuring durable event archival.
- **`log`** — Write the event to the application log (stdout). Useful
for debugging.
@@ -588,57 +508,16 @@ This separation provides:
DB; the event database file is hard-deleted (permanently removed).
- **Per-webhook retention** — the `retention_days` field on each webhook
controls automatic cleanup of old events in that webhook's database
only, or disables cleanup entirely when set to `0` (retain forever).
only.
- **Performance** — each webhook's database has its own WAL, its own
page cache, and its own lock, so concurrent event ingestion across
webhooks won't contend.
The **database target type** builds on this architecture to provide
long-term archiving, separate from the per-webhook event database (which
may prune events under its own retention). Delivering to a database
target writes the full event — body, headers, method, content type, and
webhook/entrypoint/event identifiers — as a row into a dedicated archive
database, `archive-{webhookID}.db`, stored under the data directory
beside the event database. After each write the archive handle is closed
and reopened, debounced to at most once per second, so an operator can
move the archive file away for offline archiving without stopping the
service; a moved or removed archive file is recreated automatically on
the next write. An optional `expiry` in the target's config JSON (e.g.
`{"expiry":"720h"}`) is validated when the target is created — the
default (unset or the literal `never`) keeps rows forever — and rows
older than the expiry are pruned each time the archive is (re)opened. An
archive write failure is never silent success: the delivery records a
failed attempt with the error and is marked failed.
Because reopens only happen on writes, an archive belonging to a webhook
that has stopped receiving events would never be pruned. A background
**archive sweeper** closes that gap: on the same interval as the event
retention reaper (`RETENTION_SWEEP_INTERVAL`) it prunes every archive
whose database target declares a positive expiry, whether or not the
webhook is still receiving traffic. The sweep never creates an archive —
a webhook whose archive file does not yet exist is skipped, not
initialised — it takes the same per-webhook lock the write path uses, so
it can never interleave with a write, and it leaves the archive closed
afterwards so the move-the-file-away workflow keeps working. Archives
with no expiry, or the expiry `never`, are not touched by the sweep at
all.
Note that a webhook has one archive file but may carry more than one
`database` target, each with its own `expiry`. The shortest expiry
configured on any of them therefore governs the whole archive, and the
sweep applies it whether or not the webhook is still receiving events.
Configure a single `database` target per webhook unless you intend that.
Deleting a webhook releases its archive: the delivery engine's cached
archive writer is dropped and its file handle closed, so nothing lingers
after the webhook is gone. The archive **file itself is deliberately
left on disk**. Unlike the event database — per-webhook working storage
that is hard-deleted with the webhook — an archive is long-term storage
an operator may still want to keep or move away for offline retention,
and destroying it as a side effect of deleting a webhook would be
unrecoverable. Removing `archive-{webhookID}.db` is the operator's call.
Deleting a webhook's last `database` target releases the writer the same
way, and for the same reason leaves the file alone.
The **database target type** leverages this architecture: since events
are already stored in the per-webhook database by design, the database
target simply marks the delivery as immediately successful. The
per-webhook DB IS the dedicated event database — that's the whole point
of the database target type.
The **Slack target type** sends webhook events as formatted messages to
any Slack-compatible incoming webhook URL (works with Slack, Mattermost,
@@ -745,18 +624,6 @@ This means:
durable fallback that ensures no retry is permanently lost, even under
extreme backpressure.
**Changing a target's type does not migrate in-flight deliveries.** Only
`http` and `slack` targets own durable retries; `database` and `log`
targets are fire-and-forget and never produce a `retrying` delivery. If a
target's `type` is edited from a retrying type to a non-retrying (or
unknown) one while one of its deliveries is still `retrying`, both
recovery paths above terminally mark that delivery `failed` and record a
`DeliveryResult` naming the current target type as the reason, logging it
at warn level. The delivery is not re-dispatched under the new type — the
operator never asked for that delivery — and the event itself remains
stored in the per-webhook event database, so it can be redelivered
manually.
### Circuit Breaker (HTTP Targets with Retries)
HTTP targets with `max_retries` > 0 are protected by a **per-target circuit breaker** that
@@ -825,16 +692,6 @@ legitimate webhook senders). Requests over the limit receive HTTP 429
with a `Retry-After` header. A set-but-invalid `RECEIVER_RATE_LIMIT`
value aborts startup rather than silently falling back to the default.
Every limiter here — receiver, login, and password change — identifies
the client the same way, through one shared key function: the
connection's own address, unless the peer is listed in
`TRUSTED_PROXIES`, in which case the forwarded client address is used
instead. See [Trusted proxies](#trusted-proxies). Deployed without that
variable set, a client behind a reverse proxy shares one bucket with
every other client behind the same proxy, which is the safe direction
to be wrong in: set `TRUSTED_PROXIES` to the proxy's address to get
per-client limits back.
Finer-grained per-webhook rate limits (configured in the web UI and
enforced in the webhook handler) can layer on top of this env-level
abuse limit later; they are tracked as future work.
@@ -1005,17 +862,9 @@ Applied to all routes in this order:
8. **Sentry** — Error reporting to Sentry (if `SENTRY_DSN` is set;
configured with `Repanic: true` so panics still reach Recoverer)
Additionally, form endpoints (`/pages`, `/user/*`, `/sources`,
`/source/*`) apply a **MaxBodySize** middleware that limits
POST/PUT/PATCH request bodies to 1 MB. It is registered ahead of the
CSRF middleware in every one of those route groups, because
gorilla/csrf parses the form; if the cap were installed after it, form
parsing would run under net/http's 10 MB default and the 1 MB limit
would never apply. A request that declares a `Content-Length` over the
limit is answered with `413 Request Entity Too Large` before any other
middleware or handler runs; a chunked request, or one that lies about
its length, is hard-capped by `http.MaxBytesReader` and fails
downstream at form-parse time.
Additionally, form endpoints (`/pages`, `/sources`, `/source/*`) apply a
**MaxBodySize** middleware that limits POST/PUT/PATCH request bodies to
1 MB using `http.MaxBytesReader`, preventing oversized form submissions.
### Authentication
@@ -1037,8 +886,7 @@ downstream at form-parse time.
- Production security headers on all responses: HSTS, X-Content-Type-Options
(`nosniff`), X-Frame-Options (`DENY`), Content-Security-Policy, Referrer-Policy,
and Permissions-Policy
- Request body size limits (1 MB) on all form POST endpoints, enforced
by middleware that runs before CSRF parses the form
- Request body size limits (1 MB) on all form POST endpoints
- **CSRF protection** via [gorilla/csrf](https://github.com/gorilla/csrf)
on all state-changing forms (cookie-based double-submit tokens with
HMAC authentication). Applied to `/pages`, `/sources`, `/source`, and

56
TODO.md
View File

@@ -10,14 +10,14 @@
# Status
pre-1.0. No git tags exist. main (4f5ecb1) is a working webhook proxy
pre-1.0. No git tags exist. main (81413c5) is a working webhook proxy
with auth, CSRF/SSRF protections, login rate limiting, Slack target,
event retention (#63), the database archiving target (#43), the admin
password change flow (#65), policy compliance (#6), pinned lint tooling
(#55), and fail-loud configuration parsing (#80). Note: TODO.md was
deliberately deleted from this repo in f9a9569 (2026-03-01, #6); its
content was folded into the README TODO section, which this draft
reconstructs as of 2026-07-06.
policy compliance (#6), pinned lint tooling (#55), a per-webhook event
retention reaper (#63), and delivery targets behind a Target interface
(#77). Work is tracked as Gitea issues (the authoritative TODO); this
file is a summary. Note: TODO.md was deliberately deleted from this
repo in f9a9569 (2026-03-01, #6); its content was folded into the
README TODO section, which this draft reconstructs as of 2026-07-06.
# Next Step
@@ -26,37 +26,11 @@ capability in the README rationale).
# Completed Steps
- 2026-08-09 Inactivity-based session timeout: sliding idle expiry
(`SESSION_IDLE_TIMEOUT`, default `24h`) refreshed on authenticated
requests, with the 7-day absolute cap kept as an independent
backstop that activity never extends (#66)
- 2026-08-09 Restart recovery and the 60s retry sweep terminally fail an
orphaned `retrying` delivery whose target type no longer supports
retries, recording a `DeliveryResult` with the reason instead of
leaving the delivery stuck forever (#82)
- 2026-08-09 Root the delivery engine's worker pool and the retention
reaper's sweep loop at `context.Background()` rather than the fx
`OnStart` hook context (#97), which carries fx's 15s start timeout and
killed both roughly fifteen seconds after boot: the proxy silently
stopped delivering webhooks entirely, and the reaper never ran a
single sweep under its default one-hour interval
- 2026-08-09 Archive writer lifecycle (#89): deleting a webhook (or its
last `database` target) evicts the cached archive writer and closes
its handle while deliberately leaving `archive-{webhookID}.db` on
disk, and a new `ArchiveSweeper` prunes idle archives on the existing
`RETENTION_SWEEP_INTERVAL` without ever creating an archive file
- 2026-08-09 Configuration parsing fails loudly on set-but-unparseable
environment values: `envInt` removed in favour of `envPositiveInt`
plus a `PORT` range check, `envBool` now parses with
`strconv.ParseBool`, and defaults apply only to unset variables (#80)
- 2026-08-07 Automatic event retention cleanup based on
`retention_days`, deleting expired events, deliveries, and delivery
results from each per-webhook event database (#63)
- 2026-08-07 Update golangci-lint to v2.12.2 (Docker image digest in
`Dockerfile`, release-archive sha256 pins in `script/bootstrap`),
adopt the canonical `.golangci.yml` (v2 `linters.settings` layout so
`lll`/`funlen`/`cyclop`/`dupl` thresholds actually apply), and fix
all newly surfaced lint findings
- 2026-08-07 Rate-limit the public webhook receiver per client IP per
entrypoint, env-configurable with fail-loud parsing (#64)
- 2026-08-07 Per-webhook event retention reaper (#63); NoCache
middleware for authenticated pages (#61); Target interface refactor
(#77)
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
Makefile shims, README Entrypoints section
- 2026-03-25 pin golangci-lint Docker image for linting (#55)
@@ -82,8 +56,8 @@ capability in the README rationale).
- Delivery status and retry management UI
- Per-webhook rate limiting in the receiver handler (per-webhook config
plus handler enforcement; global limits must not apply to receiver
endpoints)
plus handler enforcement, layered on the env-level receiver limit
from #64; global limits must not apply to receiver endpoints)
- Webhook signature verification for GitHub and Stripe HMAC formats
- API key authentication for programmatic access (APIKey model exists;
Bearer token middleware does not)
@@ -93,7 +67,7 @@ capability in the README rationale).
- event redelivery endpoint
- OpenAPI specification
- Analytics dashboard: success rates, response times, volume
- A remember-me option at login
- Session expiration tuning and a remember-me option
- Password change and reset flow
- Later, nice to have
- email delivery target type

7
cmd/webhooker/main.go
View File

@@ -40,15 +40,9 @@ func main() {
handlers.New,
middleware.New,
delivery.New,
delivery.NewArchiveSweeper,
// Wire *delivery.Engine as delivery.Notifier so the
// webhook handler can notify the engine of new deliveries.
func(e *delivery.Engine) delivery.Notifier { return e },
// Wire *delivery.Engine as delivery.WebhookEvictor so
// deleting a webhook releases its archive writer.
func(e *delivery.Engine) delivery.WebhookEvictor {
return e
},
server.New,
),
fx.Invoke(
@@ -56,7 +50,6 @@ func main() {
*server.Server,
*delivery.Engine,
*database.RetentionReaper,
*delivery.ArchiveSweeper,
) {
},
),

232
internal/config/config.go
View File

@@ -5,7 +5,6 @@ import (
"errors"
"fmt"
"log/slog"
"net/netip"
"os"
"strconv"
"strings"
@@ -33,20 +32,12 @@ const (
// reaper deletes events older than each webhook's RetentionDays.
defaultRetentionSweepInterval = time.Hour
// defaultSessionIdleTimeout is how long a session may go without
// authenticated activity before it expires.
defaultSessionIdleTimeout = 24 * time.Hour
// defaultReceiverRateLimit is the default number of requests
// per minute each client IP may send to a single webhook
// receiver entrypoint. Generous for legitimate webhook
// senders while bounding abuse of the one unauthenticated,
// internet-exposed endpoint.
defaultReceiverRateLimit = 120
// maxPort is the highest valid TCP port number. The lower
// bound (at least 1) is enforced by envPositiveInt.
maxPort = 65535
)
// ErrInvalidEnvironment is returned when WEBHOOKER_ENVIRONMENT
@@ -57,15 +48,6 @@ var ErrInvalidEnvironment = errors.New("invalid environment")
// requires a positive integer is set to zero or a negative number.
var ErrNonPositiveValue = errors.New("value must be positive")
// ErrInvalidPort is returned when an environment variable holding a
// TCP port number is set above the valid port range.
var ErrInvalidPort = errors.New("invalid port")
// ErrInvalidCIDR is returned when an environment variable holding a
// list of CIDR blocks contains an entry that is neither a CIDR block
// nor a bare IP address.
var ErrInvalidCIDR = errors.New("invalid CIDR")
//nolint:revive // ConfigParams is a standard fx naming convention.
type ConfigParams struct {
fx.In
@@ -89,22 +71,10 @@ type Config struct {
// RetentionSweepInterval is how often the retention reaper runs.
RetentionSweepInterval time.Duration
// SessionIdleTimeout is the sliding inactivity window after
// which a session expires. Non-positive disables idle expiry.
SessionIdleTimeout time.Duration
// ReceiverRateLimit is the number of requests per minute each
// client IP may send to a single webhook receiver entrypoint.
ReceiverRateLimit int
// TrustedProxies is the set of networks whose members are
// allowed to speak for the client with forwarded headers
// (X-Forwarded-For, X-Real-IP, True-Client-IP). It is empty
// unless TRUSTED_PROXIES is set, and empty means no peer is
// trusted: forwarded headers are then ignored entirely and
// clients are identified by the connection's own address.
TrustedProxies []netip.Prefix
params *ConfigParams
log *slog.Logger
}
@@ -126,29 +96,27 @@ func envString(key string) string {
}
// envBool returns the value of the named environment variable
// parsed as a boolean. Returns defaultValue if not set. If the
// variable is set but cannot be parsed, it returns a wrapped error
// naming the key and the bad value, so startup fails loudly rather
// than silently falling back to the default.
//
// Parsing is strconv.ParseBool, which accepts 1, t, T, TRUE, true,
// True, 0, f, F, FALSE, false and False. Anything else — "yes",
// "on", or a typo like "ture" — is an error rather than a silent
// false.
func envBool(key string, defaultValue bool) (bool, error) {
v := os.Getenv(key)
if v == "" {
return defaultValue, nil
// parsed as a boolean. Returns defaultValue if not set.
func envBool(key string, defaultValue bool) bool {
if v := os.Getenv(key); v != "" {
return strings.EqualFold(v, "true") || v == "1"
}
b, err := strconv.ParseBool(v)
if err != nil {
return false, fmt.Errorf(
"invalid boolean for %s: %q: %w", key, v, err,
)
return defaultValue
}
// envInt returns the value of the named environment variable
// parsed as an integer. Returns defaultValue if not set or
// unparseable.
func envInt(key string, defaultValue int) int {
if v := os.Getenv(key); v != "" {
i, err := strconv.Atoi(v)
if err == nil {
return i
}
}
return b, nil
return defaultValue
}
// envPositiveInt returns the value of the named environment variable
@@ -183,26 +151,6 @@ func envPositiveInt(
return i, nil
}
// envPort returns the value of the named environment variable parsed
// as a TCP port number. Returns defaultValue if not set. A set value
// that is unparseable, below 1, or above maxPort is a hard error
// naming the key and the bad value.
func envPort(key string, defaultValue int) (int, error) {
port, err := envPositiveInt(key, defaultValue)
if err != nil {
return 0, err
}
if port > maxPort {
return 0, fmt.Errorf(
"%w: %s must be at most %d, got %d",
ErrInvalidPort, key, maxPort, port,
)
}
return port, nil
}
// envDuration returns the value of the named environment variable
// parsed as a Go duration (e.g. "1h", "30m"). Returns defaultValue if
// not set. If the variable is set but cannot be parsed, it returns a
@@ -227,106 +175,32 @@ func envDuration(
return d, nil
}
// parseCIDR parses one trusted-proxy list entry, which may be a
// CIDR block ("10.0.0.0/8") or a bare address ("10.0.0.1", treated
// as a single-host block).
func parseCIDR(entry string) (netip.Prefix, error) {
if strings.Contains(entry, "/") {
prefix, err := netip.ParsePrefix(entry)
if err != nil {
return netip.Prefix{}, err //nolint:wrapcheck // wrapped by caller
}
// New creates a Config by reading environment variables.
//
//nolint:revive // lc parameter is required by fx even if unused.
func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
log := params.Logger.Get()
return prefix.Masked(), nil
}
addr, err := netip.ParseAddr(entry)
if err != nil {
return netip.Prefix{}, err //nolint:wrapcheck // wrapped by caller
}
return netip.PrefixFrom(addr.Unmap(), addr.Unmap().BitLen()), nil
}
// envPrefixList returns the value of the named environment variable
// parsed as a comma-separated list of CIDR blocks (bare addresses
// allowed). An unset, empty, or blank value yields an empty list. A
// set value containing an unparseable entry is a hard error naming
// the key and the bad entry, so startup fails loudly rather than
// silently running with a list the operator did not intend.
func envPrefixList(key string) ([]netip.Prefix, error) {
v := strings.TrimSpace(os.Getenv(key))
if v == "" {
return nil, nil
}
var prefixes []netip.Prefix
for entry := range strings.SplitSeq(v, ",") {
entry = strings.TrimSpace(entry)
if entry == "" {
continue
}
prefix, err := parseCIDR(entry)
if err != nil {
return nil, fmt.Errorf(
"%w: %s: %q: %w", ErrInvalidCIDR, key, entry, err,
)
}
prefixes = append(prefixes, prefix)
}
return prefixes, nil
}
// resolveEnvironment reads WEBHOOKER_ENVIRONMENT, defaulting to
// dev, and rejects unrecognised values.
func resolveEnvironment() (string, error) {
// Determine environment from WEBHOOKER_ENVIRONMENT env var,
// default to dev
environment := os.Getenv("WEBHOOKER_ENVIRONMENT")
if environment == "" {
environment = EnvironmentDev
}
// Validate environment
if environment != EnvironmentDev &&
environment != EnvironmentProd {
return "", fmt.Errorf(
return nil, fmt.Errorf(
"%w: WEBHOOKER_ENVIRONMENT must be '%s' or '%s', got '%s'",
ErrInvalidEnvironment,
EnvironmentDev, EnvironmentProd, environment,
)
}
return environment, nil
}
// loadFromEnv builds a Config from the environment. Every value that
// needs parsing fails loudly when it is set but unparseable: the
// documented defaults apply only to variables that are unset (or
// empty), never as a substitute for a value the operator actually
// provided.
func loadFromEnv() (*Config, error) {
environment, err := resolveEnvironment()
if err != nil {
return nil, err
}
port, err := envPort("PORT", defaultPort)
if err != nil {
return nil, err
}
debug, err := envBool("DEBUG", false)
if err != nil {
return nil, err
}
maintenanceMode, err := envBool("MAINTENANCE_MODE", false)
if err != nil {
return nil, err
}
// Parse the retention sweep interval; a set-but-unparseable value
// is a hard error so fx aborts startup rather than silently using
// the default.
retentionSweepInterval, err := envDuration(
"RETENTION_SWEEP_INTERVAL",
defaultRetentionSweepInterval,
@@ -335,14 +209,9 @@ func loadFromEnv() (*Config, error) {
return nil, err
}
sessionIdleTimeout, err := envDuration(
"SESSION_IDLE_TIMEOUT",
defaultSessionIdleTimeout,
)
if err != nil {
return nil, err
}
// Parse the receiver rate limit; a set-but-unparseable or
// non-positive value is a hard error so fx aborts startup
// rather than silently using the default.
receiverRateLimit, err := envPositiveInt(
"RECEIVER_RATE_LIMIT",
defaultReceiverRateLimit,
@@ -351,44 +220,22 @@ func loadFromEnv() (*Config, error) {
return nil, err
}
trustedProxies, err := envPrefixList("TRUSTED_PROXIES")
if err != nil {
return nil, err
}
return &Config{
// Load configuration values from environment variables
s := &Config{
DataDir: envString("DATA_DIR"),
Debug: debug,
MaintenanceMode: maintenanceMode,
Debug: envBool("DEBUG", false),
MaintenanceMode: envBool("MAINTENANCE_MODE", false),
Environment: environment,
MetricsUsername: envString("METRICS_USERNAME"),
MetricsPassword: envString("METRICS_PASSWORD"),
Port: port,
Port: envInt("PORT", defaultPort),
SentryDSN: envString("SENTRY_DSN"),
RetentionSweepInterval: retentionSweepInterval,
SessionIdleTimeout: sessionIdleTimeout,
ReceiverRateLimit: receiverRateLimit,
TrustedProxies: trustedProxies,
}, nil
}
// New creates a Config by reading environment variables.
//
//nolint:revive // lc parameter is required by fx even if unused.
func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
log := params.Logger.Get()
// A set-but-unparseable value anywhere in the environment is a
// hard error, so fx aborts startup rather than running with a
// silently substituted default.
s, err := loadFromEnv()
if err != nil {
return nil, err
log: log,
params: &params,
}
s.log = log
s.params = &params
// Set default DataDir. All SQLite databases (main application
// DB and per-webhook event DBs) live here. The same default is
// used regardless of environment; override with DATA_DIR if
@@ -410,7 +257,6 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
"dataDir", s.DataDir,
"retentionSweepInterval", s.RetentionSweepInterval.String(),
"receiverRateLimit", s.ReceiverRateLimit,
"trustedProxies", len(s.TrustedProxies),
"hasSentryDSN", s.SentryDSN != "",
"hasMetricsAuth",
s.MetricsUsername != "" && s.MetricsPassword != "",

263
internal/config/config_test.go
View File

@@ -14,14 +14,6 @@ import (
"sneak.berlin/go/webhooker/internal/logger"
)
// Shared subtest names for the env-parsing tables below, which all
// exercise the same three cases against different variables.
const (
caseUnsetUsesDefault = "unset uses default"
caseValidValueParsed = "valid value is parsed"
caseUnparseableFails = "unparseable value fails startup"
)
func TestEnvironmentConfig(t *testing.T) {
tests := []struct {
name string
@@ -138,18 +130,18 @@ func TestRetentionSweepInterval(t *testing.T) {
expected time.Duration
}{
{
name: caseUnsetUsesDefault,
name: "unset uses default",
set: false,
expected: time.Hour,
},
{
name: caseValidValueParsed,
name: "valid value is parsed",
set: true,
value: "15m",
expected: 15 * time.Minute,
},
{
name: caseUnparseableFails,
name: "unparseable value fails startup",
set: true,
value: "not-a-duration",
expectError: true,
@@ -171,7 +163,7 @@ func TestRetentionSweepInterval(t *testing.T) {
}
if tt.expectError {
expectStartupError(t)
testRetentionSweepIntervalError(t)
} else {
testRetentionSweepIntervalSuccess(t, tt.expected)
}
@@ -179,10 +171,7 @@ func TestRetentionSweepInterval(t *testing.T) {
}
}
// startupError builds the app config.New belongs to and returns
// the error fx reports, which is non-nil whenever an environment
// value is set but invalid.
func startupError(t *testing.T) error {
func testRetentionSweepIntervalError(t *testing.T) {
t.Helper()
var cfg *config.Config
@@ -197,33 +186,7 @@ func startupError(t *testing.T) error {
fx.Populate(&cfg),
)
return app.Err()
}
// expectStartupError asserts that fx refuses to build the app,
// which is what a set-but-invalid environment value must cause.
func expectStartupError(t *testing.T) {
t.Helper()
assert.Error(t, startupError(t))
}
// expectStartupErrorFor asserts that startup fails, that the error
// names the offending variable so an operator can find it, and,
// when sentinel is non-nil, that it wraps that sentinel.
func expectStartupErrorFor(
t *testing.T,
key string,
sentinel error,
) {
t.Helper()
err := startupError(t)
require.ErrorContains(t, err, key)
if sentinel != nil {
require.ErrorIs(t, err, sentinel)
}
assert.Error(t, app.Err())
}
func testRetentionSweepIntervalSuccess(
@@ -252,82 +215,6 @@ func testRetentionSweepIntervalSuccess(
assert.Equal(t, expected, cfg.RetentionSweepInterval)
}
func TestSessionIdleTimeout(t *testing.T) {
tests := []struct {
name string
set bool
value string
expectError bool
expected time.Duration
}{
{
name: caseUnsetUsesDefault,
set: false,
expected: 24 * time.Hour,
},
{
name: caseValidValueParsed,
set: true,
value: "30m",
expected: 30 * time.Minute,
},
{
name: caseUnparseableFails,
set: true,
value: "not-a-duration",
expectError: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
if tt.set {
t.Setenv("SESSION_IDLE_TIMEOUT", tt.value)
} else {
require.NoError(t, os.Unsetenv(
"SESSION_IDLE_TIMEOUT",
))
}
if tt.expectError {
expectStartupError(t)
} else {
testSessionIdleTimeoutSuccess(t, tt.expected)
}
})
}
}
func testSessionIdleTimeoutSuccess(
t *testing.T,
expected time.Duration,
) {
t.Helper()
var cfg *config.Config
app := fxtest.New(
t,
fx.Provide(
globals.New,
logger.New,
config.New,
),
fx.Populate(&cfg),
)
require.NoError(t, app.Err())
app.RequireStart()
defer app.RequireStop()
assert.Equal(t, expected, cfg.SessionIdleTimeout)
}
func TestDefaultDataDir(t *testing.T) {
for _, env := range []string{"", "dev", "prod"} {
name := env
@@ -378,25 +265,21 @@ func TestReceiverRateLimit(t *testing.T) {
set bool
value string
expectError bool
// sentinel, when set, must be wrapped by the startup
// error; every error case must additionally name the
// variable in its message.
sentinel error
expected int
}{
{
name: caseUnsetUsesDefault,
name: "unset uses default",
set: false,
expected: 120,
},
{
name: caseValidValueParsed,
name: "valid value is parsed",
set: true,
value: "30",
expected: 30,
},
{
name: caseUnparseableFails,
name: "unparseable value fails startup",
set: true,
value: "not-a-number",
expectError: true,
@@ -406,14 +289,12 @@ func TestReceiverRateLimit(t *testing.T) {
set: true,
value: "0",
expectError: true,
sentinel: config.ErrNonPositiveValue,
},
{
name: "negative fails startup",
set: true,
value: "-5",
expectError: true,
sentinel: config.ErrNonPositiveValue,
},
}
@@ -432,9 +313,7 @@ func TestReceiverRateLimit(t *testing.T) {
}
if tt.expectError {
expectStartupErrorFor(
t, "RECEIVER_RATE_LIMIT", tt.sentinel,
)
testReceiverRateLimitError(t)
} else {
testReceiverRateLimitSuccess(t, tt.expected)
}
@@ -442,6 +321,24 @@ func TestReceiverRateLimit(t *testing.T) {
}
}
func testReceiverRateLimitError(t *testing.T) {
t.Helper()
var cfg *config.Config
app := fx.New(
fx.NopLogger,
fx.Provide(
globals.New,
logger.New,
config.New,
),
fx.Populate(&cfg),
)
assert.Error(t, app.Err())
}
func testReceiverRateLimitSuccess(
t *testing.T,
expected int,
@@ -467,107 +364,3 @@ func testReceiverRateLimitSuccess(
assert.Equal(t, expected, cfg.ReceiverRateLimit)
}
func TestTrustedProxies(t *testing.T) {
tests := []struct {
name string
set bool
value string
expectError bool
expected []string
}{
{
// The default must be "trust nobody": an empty list
// means forwarded headers are ignored, never that
// every peer may speak for the client.
name: caseUnsetUsesDefault,
set: false,
expected: []string{},
},
{
name: "blank value trusts nothing",
set: true,
value: " ",
expected: []string{},
},
{
name: caseValidValueParsed,
set: true,
value: "10.0.0.0/8, 192.168.1.7 ,2001:db8::/32",
expected: []string{
"10.0.0.0/8", "192.168.1.7/32", "2001:db8::/32",
},
},
{
name: "host bits are masked off",
set: true,
value: "10.1.2.3/8",
expected: []string{"10.0.0.0/8"},
},
{
name: caseUnparseableFails,
set: true,
value: "10.0.0.0/8,not-an-address",
expectError: true,
},
{
name: "out-of-range prefix length fails startup",
set: true,
value: "10.0.0.0/33",
expectError: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
if tt.set {
t.Setenv("TRUSTED_PROXIES", tt.value)
} else {
require.NoError(t, os.Unsetenv("TRUSTED_PROXIES"))
}
if tt.expectError {
expectStartupErrorFor(
t, "TRUSTED_PROXIES", config.ErrInvalidCIDR,
)
} else {
testTrustedProxiesSuccess(t, tt.expected)
}
})
}
}
func testTrustedProxiesSuccess(
t *testing.T,
expected []string,
) {
t.Helper()
var cfg *config.Config
app := fxtest.New(
t,
fx.Provide(
globals.New,
logger.New,
config.New,
),
fx.Populate(&cfg),
)
require.NoError(t, app.Err())
app.RequireStart()
defer app.RequireStop()
got := make([]string, 0, len(cfg.TrustedProxies))
for _, prefix := range cfg.TrustedProxies {
got = append(got, prefix.String())
}
assert.Equal(t, expected, got)
}

409
internal/config/env_test.go
View File

@@ -1,409 +0,0 @@
package config_test
import (
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/globals"
"sneak.berlin/go/webhooker/internal/logger"
)
// testEnvKey is a throwaway variable name used only by the helper
// tables below, so they cannot disturb real configuration.
const testEnvKey = "WEBHOOKER_TEST_VALUE"
// Real configuration variables exercised by the config.New tests.
const (
envKeyPort = "PORT"
envKeyDebug = "DEBUG"
envKeyMaintenanceMode = "MAINTENANCE_MODE"
)
// envBoolCase is one row of the envBool table.
type envBoolCase struct {
name string
set bool
value string
defaultValue bool
expectError bool
expected bool
}
// envBoolCases is the envBool table, kept out of the test body so
// the test itself stays readable.
func envBoolCases() []envBoolCase {
return []envBoolCase{
{
name: "unset uses default false",
defaultValue: false,
expected: false,
},
{
name: "unset uses default true",
defaultValue: true,
expected: true,
},
{
name: "empty uses default true",
set: true,
value: "",
defaultValue: true,
expected: true,
},
{
name: "true is parsed",
set: true,
value: "true",
expected: true,
},
{
name: "one is parsed",
set: true,
value: "1",
expected: true,
},
{
name: "False is parsed",
set: true,
value: "False",
defaultValue: true,
expected: false,
},
{
name: "zero is parsed",
set: true,
value: "0",
defaultValue: true,
expected: false,
},
{
name: "yes is rejected",
set: true,
value: "yes",
expectError: true,
},
{
name: "on is rejected",
set: true,
value: "on",
expectError: true,
},
{
name: "typo is rejected",
set: true,
value: "ture",
expectError: true,
},
}
}
func TestEnvBool(t *testing.T) {
for _, tt := range envBoolCases() {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
if tt.set {
t.Setenv(testEnvKey, tt.value)
} else {
require.NoError(t, os.Unsetenv(testEnvKey))
}
got, err := config.EnvBoolForTest(
testEnvKey, tt.defaultValue,
)
if tt.expectError {
require.Error(t, err)
assert.Contains(t, err.Error(), testEnvKey)
assert.Contains(t, err.Error(), tt.value)
return
}
require.NoError(t, err)
assert.Equal(t, tt.expected, got)
})
}
}
func TestEnvPositiveInt(t *testing.T) {
const defaultValue = 7
tests := []struct {
name string
set bool
value string
expectError bool
errIs error
expected int
}{
{
name: "unset returns the default integer",
expected: defaultValue,
},
{
name: "empty returns the default integer",
set: true,
value: "",
expected: defaultValue,
},
{
name: "positive value is parsed",
set: true,
value: "42",
expected: 42,
},
{
name: "unparseable value is rejected",
set: true,
value: "not-a-number",
expectError: true,
},
{
name: "zero is rejected",
set: true,
value: "0",
expectError: true,
errIs: config.ErrNonPositiveValue,
},
{
name: "negative is rejected",
set: true,
value: "-5",
expectError: true,
errIs: config.ErrNonPositiveValue,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
if tt.set {
t.Setenv(testEnvKey, tt.value)
} else {
require.NoError(t, os.Unsetenv(testEnvKey))
}
got, err := config.EnvPositiveIntForTest(
testEnvKey, defaultValue,
)
if tt.expectError {
require.Error(t, err)
assert.Contains(t, err.Error(), testEnvKey)
assert.Contains(t, err.Error(), tt.value)
if tt.errIs != nil {
require.ErrorIs(t, err, tt.errIs)
}
return
}
require.NoError(t, err)
assert.Equal(t, tt.expected, got)
})
}
}
func TestEnvPort(t *testing.T) {
const defaultValue = 8080
tests := []struct {
name string
set bool
value string
expectError bool
errIs error
expected int
}{
{
name: "unset returns the default port",
expected: defaultValue,
},
{
name: "valid port is parsed",
set: true,
value: "9000",
expected: 9000,
},
{
name: "highest port is accepted",
set: true,
value: "65535",
expected: 65535,
},
{
name: "unparseable value is rejected",
set: true,
value: "not-a-port",
expectError: true,
},
{
name: "zero is rejected",
set: true,
value: "0",
expectError: true,
errIs: config.ErrNonPositiveValue,
},
{
name: "above the port range is rejected",
set: true,
value: "65536",
expectError: true,
errIs: config.ErrInvalidPort,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
if tt.set {
t.Setenv(testEnvKey, tt.value)
} else {
require.NoError(t, os.Unsetenv(testEnvKey))
}
got, err := config.EnvPortForTest(
testEnvKey, defaultValue,
)
if tt.expectError {
require.Error(t, err)
assert.Contains(t, err.Error(), testEnvKey)
if tt.errIs != nil {
require.ErrorIs(t, err, tt.errIs)
}
return
}
require.NoError(t, err)
assert.Equal(t, tt.expected, got)
})
}
}
// buildConfig constructs a Config through fx exactly as the
// application does, returning the config and any construction error.
func buildConfig(t *testing.T) (*config.Config, error) {
t.Helper()
var cfg *config.Config
app := fx.New(
fx.NopLogger,
fx.Provide(
globals.New,
logger.New,
config.New,
),
fx.Populate(&cfg),
)
return cfg, app.Err()
}
func TestNewRejectsBadEnvValues(t *testing.T) {
tests := []struct {
name string
key string
value string
expectError bool
check func(t *testing.T, cfg *config.Config)
}{
{
name: "valid PORT is used",
key: envKeyPort,
value: "9001",
check: func(t *testing.T, cfg *config.Config) {
t.Helper()
assert.Equal(t, 9001, cfg.Port)
},
},
{
name: "unparseable PORT aborts startup",
key: envKeyPort,
value: "eighty-eighty",
expectError: true,
},
{
name: "out-of-range PORT aborts startup",
key: envKeyPort,
value: "70000",
expectError: true,
},
{
name: "valid DEBUG is used",
key: envKeyDebug,
value: "true",
check: func(t *testing.T, cfg *config.Config) {
t.Helper()
assert.True(t, cfg.Debug)
},
},
{
name: "unparseable DEBUG aborts startup",
key: envKeyDebug,
value: "ture",
expectError: true,
},
{
name: "unparseable MAINTENANCE_MODE aborts startup",
key: envKeyMaintenanceMode,
value: "sometimes",
expectError: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
t.Setenv(tt.key, tt.value)
cfg, err := buildConfig(t)
if tt.expectError {
require.Error(t, err)
assert.Contains(t, err.Error(), tt.key)
assert.Contains(t, err.Error(), tt.value)
return
}
require.NoError(t, err)
require.NotNil(t, cfg)
tt.check(t, cfg)
})
}
}
// TestNewUsesDefaultsWhenUnset proves the fail-loud behaviour did not
// break the legitimate unset case: absent variables still get their
// documented defaults.
func TestNewUsesDefaultsWhenUnset(t *testing.T) {
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
for _, key := range []string{
envKeyPort, envKeyDebug, envKeyMaintenanceMode,
} {
require.NoError(t, os.Unsetenv(key))
}
cfg, err := buildConfig(t)
require.NoError(t, err)
require.NotNil(t, cfg)
assert.Equal(t, 8080, cfg.Port)
assert.False(t, cfg.Debug)
assert.False(t, cfg.MaintenanceMode)
}

20
internal/config/export_test.go
View File

@@ -1,20 +0,0 @@
package config
// This file exposes the unexported environment parsing helpers to
// the external config_test package so each helper can be covered by
// its own table-driven test without weakening the package API.
// EnvBoolForTest exposes envBool.
func EnvBoolForTest(key string, defaultValue bool) (bool, error) {
return envBool(key, defaultValue)
}
// EnvPositiveIntForTest exposes envPositiveInt.
func EnvPositiveIntForTest(key string, defaultValue int) (int, error) {
return envPositiveInt(key, defaultValue)
}
// EnvPortForTest exposes envPort.
func EnvPortForTest(key string, defaultValue int) (int, error) {
return envPort(key, defaultValue)
}

18
internal/database/database_test.go
View File

@@ -11,20 +11,6 @@ import (
"sneak.berlin/go/webhooker/internal/logger"
)
const (
// testAppname is the Globals.Appname used in tests.
testAppname = "webhooker-test"
// testVersion is the Globals.Version used in tests.
testVersion = "test"
// testContentType is the event content type used in tests.
testContentType = "application/json"
// testWebhookName is the Webhook.Name used in tests.
testWebhookName = "test-webhook"
// testForeverLabel is Webhook.RetentionLabel for a retain-forever
// webhook.
testForeverLabel = "forever"
)
func setupTestDB(
t *testing.T,
) (*database.Database, *fxtest.Lifecycle) {
@@ -33,8 +19,8 @@ func setupTestDB(
lc := fxtest.NewLifecycle(t)
g := &globals.Globals{
Appname: testAppname,
Version: testVersion,
Appname: "webhooker-test",
Version: "test",
}
l, err := logger.New(

25
internal/database/export_test.go
View File

@@ -5,8 +5,6 @@ import (
"log/slog"
"os"
"time"
"go.uber.org/fx"
)
// NewTestRetentionReaper builds a RetentionReaper backed by the given
@@ -31,26 +29,3 @@ func NewTestRetentionReaper(
func (r *RetentionReaper) ExportSweep(ctx context.Context) {
r.sweep(ctx)
}
// ExportRegisterHooks registers the reaper's real fx lifecycle hooks
// on a lifecycle supplied by a test, so a test can drive the exact
// OnStart/OnStop functions the application runs and hand OnStart the
// kind of context fx actually supplies.
func (r *RetentionReaper) ExportRegisterHooks(lc fx.Lifecycle) {
r.registerHooks(lc)
}
// ExportStart starts the reaper's background loop for tests.
func (r *RetentionReaper) ExportStart() {
r.start()
}
// ExportStop stops the reaper's background loop for tests.
func (r *RetentionReaper) ExportStop() {
r.stop()
}
// ExportSetInterval overrides the sweep interval for tests.
func (r *RetentionReaper) ExportSetInterval(d time.Duration) {
r.interval = d
}

5
internal/database/model_entrypoint.go
View File

@@ -5,10 +5,7 @@ type Entrypoint struct {
BaseModel
WebhookID string `gorm:"type:uuid;not null" json:"webhookId"`
// Path is the URL path for this entrypoint.
Path string `gorm:"uniqueIndex;not null" json:"path"`
Path string `gorm:"uniqueIndex;not null" json:"path"` // URL path for this entrypoint
Description string `json:"description"`
Active bool `gorm:"default:true" json:"active"`

3
internal/database/model_target.go
View File

@@ -23,8 +23,7 @@ type Target struct {
// Configuration fields (JSON stored based on type)
Config string `gorm:"type:text" json:"config"` // JSON configuration
// For HTTP targets (max_retries=0 means fire-and-forget,
// >0 enables retries with backoff)
// For HTTP targets (max_retries=0 means fire-and-forget, >0 enables retries with backoff)
MaxRetries int `json:"maxRetries,omitempty"`
MaxQueueSize int `json:"maxQueueSize,omitempty"`

111
internal/database/model_webhook.go
View File

@@ -1,125 +1,16 @@
package database
import (
"math"
"strconv"
"time"
"gorm.io/gorm"
)
const (
// DefaultRetentionDays is the event retention period applied to a
// webhook created without an explicit retention value. It is the
// single source of truth for that policy and must stay in sync
// with the `gorm:"default:30"` column default on
// Webhook.RetentionDays below; a struct tag cannot reference a
// constant, so a test asserts the two agree.
DefaultRetentionDays = 30
// RetentionForeverDays is the sentinel RetentionDays value meaning
// "retain events forever". Users express that intent as 0, which
// Webhook.BeforeSave rewrites to this value: the column default
// substitutes DefaultRetentionDays for a zero value at insert
// time, so a zero can never survive a round trip to the database.
// Nothing outside this file may hardcode the number.
RetentionForeverDays = 365 * 1000
// MaxFiniteRetentionDays is the largest finite retention period the
// reaper's cutoff arithmetic can represent, and therefore the
// largest one a caller may request. It is derived from that
// arithmetic rather than picked: retentionCutoff computes
// retentionDays * hoursPerDay * time.Hour, and a time.Duration is
// an int64 nanosecond count, so math.MaxInt64 nanoseconds divided
// by an hour and then by a day is the exact ceiling — 106751 days,
// a little over 292 years.
//
// One day more overflows int64, wraps the product negative, and
// turns the cutoff into a timestamp in the far future that matches
// every row in the webhook's database. That is why this bound is
// enforced on input and why retentionCutoff saturates underneath
// it. Note that RetentionForeverDays deliberately sits above this
// ceiling: such webhooks are skipped before any cutoff is
// computed, and never reach the arithmetic at all.
MaxFiniteRetentionDays = int(
math.MaxInt64 / int64(time.Hour) / hoursPerDay,
)
)
// Webhook represents a webhook processing unit that groups entrypoints and targets
//
// Every method below takes a pointer receiver. BeforeSave has to,
// because it mutates the record and GORM only invokes hooks declared
// that way; the display helpers follow suit so the receiver kinds do
// not mix. Handlers therefore put a *Webhook into template data:
// html/template cannot call a pointer method on a value held in a map,
// because a map element is not addressable.
type Webhook struct {
BaseModel
UserID string `gorm:"type:uuid;not null" json:"userId"`
Name string `gorm:"not null" json:"name"`
Description string `json:"description"`
// RetentionDays is the number of days to retain events. A value of
// RetentionForeverDays means retain forever. The column default
// must equal DefaultRetentionDays.
RetentionDays int `gorm:"default:30" json:"retentionDays"`
RetentionDays int `gorm:"default:30" json:"retentionDays"` // Days to retain events
// Relations
User User `json:"user,omitzero"`
Entrypoints []Entrypoint `json:"entrypoints,omitempty"`
Targets []Target `json:"targets,omitempty"`
}
// BeforeSave normalises RetentionDays on every insert and update. A
// non-positive value is the user's way of asking for "retain forever",
// which is stored as the RetentionForeverDays sentinel.
//
// This has to happen in a hook rather than at the call sites. GORM
// substitutes the column default (DefaultRetentionDays) for a zero
// value while building the insert statement, which runs after
// BeforeSave; rewriting any later than this loses that race and the
// row lands at 30 days. Living on the model also means a future call
// site — a REST API, a fixture, a migration — cannot bypass it.
func (w *Webhook) BeforeSave(_ *gorm.DB) error {
if w.RetentionDays <= 0 {
w.RetentionDays = RetentionForeverDays
}
return nil
}
// retainsForever reports whether a stored RetentionDays value means
// "keep events indefinitely". It is the single definition of that
// question, shared by Webhook.RetainsForever and by the reaper's
// cutoff computation so the two cannot disagree about which webhooks
// are exempt from reaping.
//
// It accepts the RetentionForeverDays sentinel written by BeforeSave
// and, defensively, the non-positive values that rows written before
// the sentinel existed may still carry.
func retainsForever(retentionDays int) bool {
return retentionDays <= 0 ||
retentionDays >= RetentionForeverDays
}
// RetainsForever reports whether this webhook's events are kept
// indefinitely.
func (w *Webhook) RetainsForever() bool {
return retainsForever(w.RetentionDays)
}
// RetentionLabel returns the webhook's retention policy as display
// text, so that no template has to know about the sentinel value.
func (w *Webhook) RetentionLabel() string {
if w.RetainsForever() {
return "forever"
}
if w.RetentionDays == 1 {
return "1 day"
}
return strconv.Itoa(w.RetentionDays) + " days"
}

222
internal/database/model_webhook_test.go
View File

@@ -1,222 +0,0 @@
package database_test
import (
"context"
"reflect"
"strconv"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
)
// startedTestDB returns a started main database for model-level tests.
func startedTestDB(t *testing.T) *gorm.DB {
t.Helper()
db, lc := setupTestDB(t)
ctx := context.Background()
require.NoError(t, lc.Start(ctx))
t.Cleanup(func() { require.NoError(t, lc.Stop(ctx)) })
return db.DB()
}
// storedRetention reads the retention_days column straight out of the
// row, so the assertion is about what was persisted rather than about
// whatever the in-memory struct happens to hold.
func storedRetention(t *testing.T, db *gorm.DB, id string) int {
t.Helper()
var got int
require.NoError(
t,
db.Model(&database.Webhook{}).
Where("id = ?", id).
Pluck("retention_days", &got).Error,
)
return got
}
// newWebhookWithRetention creates a webhook through the ordinary Create
// path, so the BeforeSave hook and the GORM column default both apply
// exactly as they do in production.
func newWebhookWithRetention(
t *testing.T,
db *gorm.DB,
wh *database.Webhook,
) string {
t.Helper()
wh.UserID = uuid.New().String()
wh.Name = testWebhookName
require.NoError(
t,
db.Omit(clause.Associations).Create(wh).Error,
)
return wh.ID
}
func TestWebhookBeforeSave_ZeroBecomesForeverSentinel(t *testing.T) {
t.Parallel()
db := startedTestDB(t)
wh := &database.Webhook{RetentionDays: 0}
id := newWebhookWithRetention(t, db, wh)
assert.Equal(
t,
database.RetentionForeverDays,
storedRetention(t, db, id),
"a zero retention must be stored as the sentinel, "+
"not replaced by the column default",
)
}
func TestWebhookBeforeSave_NegativeBecomesForeverSentinel(t *testing.T) {
t.Parallel()
db := startedTestDB(t)
wh := &database.Webhook{RetentionDays: -5}
id := newWebhookWithRetention(t, db, wh)
assert.Equal(
t,
database.RetentionForeverDays,
storedRetention(t, db, id),
)
}
func TestWebhookBeforeSave_PositiveIsPreserved(t *testing.T) {
t.Parallel()
db := startedTestDB(t)
wh := &database.Webhook{RetentionDays: 7}
id := newWebhookWithRetention(t, db, wh)
assert.Equal(t, 7, storedRetention(t, db, id))
}
// TestWebhookBeforeSave_UpdateToZeroBecomesSentinel proves the hook
// fires on update as well as insert, via the same Save call the edit
// handler makes.
func TestWebhookBeforeSave_UpdateToZeroBecomesSentinel(t *testing.T) {
t.Parallel()
db := startedTestDB(t)
wh := &database.Webhook{RetentionDays: 30}
id := newWebhookWithRetention(t, db, wh)
require.Equal(t, 30, storedRetention(t, db, id))
wh.RetentionDays = 0
require.NoError(t, db.Omit(clause.Associations).Save(wh).Error)
assert.Equal(
t,
database.RetentionForeverDays,
storedRetention(t, db, id),
)
}
// TestWebhookRetentionColumnDefaultMatchesConstant guards the one place
// the default lives twice: a struct tag cannot reference a constant, so
// this asserts the tag and DefaultRetentionDays agree.
func TestWebhookRetentionColumnDefaultMatchesConstant(t *testing.T) {
t.Parallel()
field, ok := reflect.TypeFor[database.Webhook]().
FieldByName("RetentionDays")
require.True(t, ok, "Webhook.RetentionDays must exist")
assert.Equal(
t,
"default:"+strconv.Itoa(database.DefaultRetentionDays),
field.Tag.Get("gorm"),
)
}
// TestMaxFiniteRetentionDaysIsTheOverflowCeiling asserts that the
// constant is exactly where the cutoff arithmetic stops working, which
// is what makes it a derived bound rather than a round number someone
// liked. One day more wraps the int64 nanosecond count negative, and a
// negative span is precisely what turned a cutoff into a future
// timestamp that matched — and deleted — every row.
//
// The multiplications are done through variables on purpose: as
// constant expressions the overflowing one would not compile.
func TestMaxFiniteRetentionDaysIsTheOverflowCeiling(t *testing.T) {
t.Parallel()
const hoursPerDay = 24
atCeiling := database.MaxFiniteRetentionDays
overCeiling := database.MaxFiniteRetentionDays + 1
assert.Positive(
t,
time.Duration(atCeiling*hoursPerDay)*time.Hour,
"the ceiling itself must still be representable",
)
assert.Negative(
t,
time.Duration(overCeiling*hoursPerDay)*time.Hour,
"one day past the ceiling must overflow",
)
assert.Less(
t,
database.MaxFiniteRetentionDays,
database.RetentionForeverDays,
"the sentinel sits above the ceiling and is only safe "+
"because retain-forever webhooks skip the arithmetic",
)
}
func TestWebhookRetainsForeverAndLabel(t *testing.T) {
t.Parallel()
cases := []struct {
name string
days int
forever bool
label string
}{
{
"sentinel",
database.RetentionForeverDays, true, testForeverLabel,
},
{
"above sentinel",
database.RetentionForeverDays + 1, true, testForeverLabel,
},
{"legacy zero", 0, true, testForeverLabel},
{"legacy negative", -1, true, testForeverLabel},
{"default", database.DefaultRetentionDays, false, "30 days"},
{"one day", 1, false, "1 day"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
wh := database.Webhook{RetentionDays: tc.days}
assert.Equal(t, tc.forever, wh.RetainsForever())
assert.Equal(t, tc.label, wh.RetentionLabel())
})
}
}

81
internal/database/retention.go
View File

@@ -56,20 +56,9 @@ func NewRetentionReaper(
interval: params.Config.RetentionSweepInterval,
}
r.registerHooks(lc)
return r
}
// registerHooks wires the reaper's start and stop into the fx
// lifecycle. The start hook's context is deliberately ignored: see
// start for why the sweep loop must not inherit it.
func (r *RetentionReaper) registerHooks(lc fx.Lifecycle) {
lc.Append(fx.Hook{
//nolint:contextcheck // Not inheriting the hook context is
// the point: see start.
OnStart: func(_ context.Context) error {
r.start()
OnStart: func(ctx context.Context) error {
r.start(ctx)
return nil
},
@@ -79,20 +68,12 @@ func (r *RetentionReaper) registerHooks(lc fx.Lifecycle) {
return nil
},
})
return r
}
// start launches the background sweep loop.
//
// The loop's context is derived from context.Background(), NOT from
// the fx OnStart hook context. The hook context carries fx's start
// timeout (15s by default) and is cancelled once the start phase
// completes, so a loop derived from it dies 45 minutes before its
// first tick under the default one-hour sweep interval, leaving a
// reaper that never reaps. A long-lived goroutine must outlive the
// startup phase, so its lifetime is bounded by OnStop instead: stop
// cancels this context and waits on the WaitGroup.
func (r *RetentionReaper) start() {
ctx, cancel := context.WithCancel(context.Background())
func (r *RetentionReaper) start(ctx context.Context) {
ctx, cancel := context.WithCancel(ctx)
r.cancel = cancel
r.wg.Add(1)
@@ -133,8 +114,7 @@ func (r *RetentionReaper) run(ctx context.Context) {
}
// sweep lists every webhook from the main database and reaps expired
// rows from each per-webhook database that has a finite retention
// policy. Webhooks set to retain forever are skipped entirely.
// rows from each per-webhook database whose RetentionDays is positive.
func (r *RetentionReaper) sweep(ctx context.Context) {
var webhooks []Webhook
@@ -159,13 +139,8 @@ func (r *RetentionReaper) sweep(ctx context.Context) {
wh := webhooks[i]
// Skip retain-forever webhooks before building any query.
// RetainsForever covers both the RetentionForeverDays
// sentinel and the non-positive values that predate it: the
// sentinel is a positive number, so without this the reaper
// would compute a cutoff a thousand years in the past and
// issue a DELETE matching nothing on every single sweep.
if wh.RetainsForever() {
// RetentionDays of zero or less means retain forever.
if wh.RetentionDays <= 0 {
continue
}
@@ -196,10 +171,9 @@ func (r *RetentionReaper) reapWebhook(
return
}
cutoff, ok := retentionCutoff(time.Now(), retentionDays)
if !ok {
return
}
cutoff := time.Now().Add(
-time.Duration(retentionDays*hoursPerDay) * time.Hour,
)
deleted, err := reapExpired(db, cutoff)
if err != nil {
@@ -222,37 +196,6 @@ func (r *RetentionReaper) reapWebhook(
}
}
// retentionCutoff returns the timestamp before which a webhook's
// events have expired, and whether any cutoff applies at all. It
// reports false for a retain-forever policy, so no DELETE is issued.
//
// The day count is clamped to MaxFiniteRetentionDays first. This is
// defense in depth rather than decoration: a time.Duration is an int64
// nanosecond count, so an unclamped multiplication overflows above
// that ceiling and wraps the span negative. Subtracting a negative
// span moves the cutoff into the far future, where it matches every
// row in the database: the sweep then deletes every event, delivery,
// and delivery result, including ones created seconds ago. Rejecting
// out-of-range input at the form is the primary guard; saturating here
// means an old row, a migration, or a future call site cannot turn a
// too-large retention into total data loss.
func retentionCutoff(
now time.Time,
retentionDays int,
) (time.Time, bool) {
if retainsForever(retentionDays) {
return time.Time{}, false
}
if retentionDays > MaxFiniteRetentionDays {
retentionDays = MaxFiniteRetentionDays
}
return now.Add(
-time.Duration(retentionDays*hoursPerDay) * time.Hour,
), true
}
// reapExpired hard-deletes, in foreign-key-safe order, the delivery
// results, deliveries, and events associated with events older than
// cutoff. Deletes are unscoped so rows are physically removed rather

209
internal/database/retention_lifecycle_test.go
View File

@@ -1,209 +0,0 @@
package database_test
import (
"context"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/fx"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
const (
// reaperTestInterval is the sweep interval a lifecycle test
// runs the reaper at, so a loop that survives startup produces
// an observable sweep quickly.
reaperTestInterval = 10 * time.Millisecond
// reaperStopTimeout bounds how long a lifecycle test waits for
// the reaper's OnStop hook to return before declaring the
// shutdown hung.
reaperStopTimeout = 10 * time.Second
// reaperTestRetentionDays is the retention policy the lifecycle
// tests give their webhook.
reaperTestRetentionDays = 30
)
// recordingLifecycle is a minimal fx.Lifecycle that records the
// hooks a component registers, so a test can invoke the real
// OnStart/OnStop functions with a context of its choosing.
type recordingLifecycle struct {
hooks []fx.Hook
}
func (l *recordingLifecycle) Append(h fx.Hook) {
l.hooks = append(l.hooks, h)
}
// startReaperViaHook drives the genuine fx hooks the application
// registers for the reaper, handing OnStart a context that is
// already done. It returns the recorded lifecycle so the caller
// can drive OnStop too.
func startReaperViaHook(
t *testing.T, r *database.RetentionReaper,
) *recordingLifecycle {
t.Helper()
lc := &recordingLifecycle{}
r.ExportRegisterHooks(lc)
require.Len(t, lc.hooks, 1)
// fx hands OnStart a context carrying the application start
// timeout, and cancels it when the start phase ends. An
// already-cancelled context is that same defect taken to its
// limit, and unlike a plain context.Background() it actually
// distinguishes a correctly rooted loop from a broken one.
hookCtx, cancel := context.WithCancel(context.Background())
cancel()
require.NoError(t, lc.hooks[0].OnStart(hookCtx))
return lc
}
// eventGone reports whether an event row has been removed. It
// takes no *testing.T because it is polled from an
// assert.Eventually condition, which runs off the test goroutine
// where testify assertions must not be used.
func eventGone(db *gorm.DB, eventID string) bool {
var n int64
err := db.Unscoped().Model(&database.Event{}).
Where("id = ?", eventID).Count(&n).Error
if err != nil {
return false
}
return n == 0
}
// seedExpiredWebhook creates a webhook with a finite retention
// policy plus one long-expired event chain, and returns the
// webhook's database and the chain's event ID.
func seedExpiredWebhook(
t *testing.T, env *retentionTestEnv,
) (*gorm.DB, string) {
t.Helper()
webhookID := createWebhook(
t, env.mainDB.DB(), reaperTestRetentionDays,
)
db, err := env.mgr.GetDB(webhookID)
require.NoError(t, err)
chain := seedEventChain(
t, db, webhookID,
time.Now().Add(-365*24*time.Hour),
)
return db, chain.eventID
}
// TestRetentionReaper_LoopOutlivesStartHookContext is the
// regression test for a reaper that never reaped. fx calls
// OnStart with a context carrying the application's start timeout
// (15s by default) and cancels it when the start phase ends, so a
// sweep loop rooted in it is dead three quarters of an hour
// before its first tick under the default one-hour interval, and
// per-webhook event databases grow without bound exactly as they
// did before retention existed.
//
// Driving OnStart with an already-cancelled context is that
// defect taken to its limit: a loop that inherits the hook
// context never ticks once, while a correctly rooted loop keeps
// sweeping for as long as the process lives.
func TestRetentionReaper_LoopOutlivesStartHookContext(
t *testing.T,
) {
t.Parallel()
env := setupRetentionTest(t)
db, eventID := seedExpiredWebhook(t, env)
env.reaper.ExportSetInterval(reaperTestInterval)
lc := startReaperViaHook(t, env.reaper)
t.Cleanup(func() {
_ = lc.hooks[0].OnStop(context.Background())
})
assert.Eventually(
t,
func() bool { return eventGone(db, eventID) },
5*time.Second,
reaperTestInterval,
"the sweep loop must keep running after the start "+
"hook's context is done; it reaped nothing, so it "+
"inherited the hook context and died",
)
}
// TestRetentionReaper_StopHookStopsLoop proves the fix did not
// trade a startup bug for a shutdown hang: now that the sweep
// loop no longer observes the start hook's cancellation, OnStop
// is the only thing that can stop it, and it must both return
// promptly and actually leave the loop stopped.
func TestRetentionReaper_StopHookStopsLoop(t *testing.T) {
t.Parallel()
env := setupRetentionTest(t)
db, eventID := seedExpiredWebhook(t, env)
env.reaper.ExportSetInterval(reaperTestInterval)
lc := startReaperViaHook(t, env.reaper)
// Let the loop prove it is running before stopping it, so a
// fast OnStop cannot pass by stopping something already dead.
require.Eventually(
t,
func() bool { return eventGone(db, eventID) },
5*time.Second,
reaperTestInterval,
)
var stopErr error
stopped := make(chan struct{})
go func() {
defer close(stopped)
// stop blocks on the loop's WaitGroup, so returning at all
// proves the goroutine observed the cancellation.
stopErr = lc.hooks[0].OnStop(context.Background())
}()
select {
case <-stopped:
case <-time.After(reaperStopTimeout):
t.Fatal(
"OnStop did not return: the retention reaper's " +
"WaitGroup is still waiting on a loop that never " +
"observed cancellation",
)
}
require.NoError(t, stopErr)
// With the loop gone, a newly expired chain must survive.
survivor := seedEventChain(
t, db, "stopped-webhook",
time.Now().Add(-365*24*time.Hour),
)
time.Sleep(20 * reaperTestInterval)
assert.False(
t,
eventGone(db, survivor.eventID),
"a stopped reaper must not sweep anything",
)
}

145
internal/database/retention_test.go
View File

@@ -2,7 +2,6 @@ package database_test
import (
"context"
"net/http"
"testing"
"time"
@@ -31,8 +30,8 @@ func setupRetentionTest(t *testing.T) *retentionTestEnv {
lc := fxtest.NewLifecycle(t)
g := &globals.Globals{
Appname: testAppname,
Version: testVersion,
Appname: "webhooker-test",
Version: "test",
}
l, err := logger.New(lc, logger.LoggerParams{Globals: g})
@@ -77,7 +76,7 @@ func createWebhook(
wh := &database.Webhook{
UserID: uuid.New().String(),
Name: testWebhookName,
Name: "test-webhook",
RetentionDays: retentionDays,
}
require.NoError(
@@ -85,11 +84,10 @@ func createWebhook(
db.Omit(clause.Associations).Create(wh).Error,
)
// Webhook.BeforeSave rewrites a non-positive RetentionDays to the
// retain-forever sentinel, and the column's GORM default would
// otherwise substitute 30. Force the requested value with a
// column-level update so tests can plant legacy rows that predate
// the sentinel and still carry a literal 0 or negative value.
// The RetentionDays column carries a GORM default of 30, so a
// zero (or negative) value passed to Create is replaced by that
// default. Force the requested value explicitly so the
// retain-forever (<= 0) path can be exercised.
require.NoError(
t,
db.Model(wh).
@@ -99,30 +97,6 @@ func createWebhook(
return wh.ID
}
// createWebhookNormally inserts a webhook through the ordinary Create
// path, with no column-level forcing, so Webhook.BeforeSave applies
// exactly as it does in production. Passing 0 therefore yields a row
// holding the RetentionForeverDays sentinel.
func createWebhookNormally(
t *testing.T,
db *gorm.DB,
retentionDays int,
) string {
t.Helper()
wh := &database.Webhook{
UserID: uuid.New().String(),
Name: testWebhookName,
RetentionDays: retentionDays,
}
require.NoError(
t,
db.Omit(clause.Associations).Create(wh).Error,
)
return wh.ID
}
// eventChain is the set of row IDs seeded for a single event.
type eventChain struct {
eventID string
@@ -143,9 +117,9 @@ func seedEventChain(
event := &database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Method: "POST",
Body: `{"seed": true}`,
ContentType: testContentType,
ContentType: "application/json",
}
event.CreatedAt = createdAt
require.NoError(t, db.Create(event).Error)
@@ -281,111 +255,12 @@ func TestRetentionReaper_ReapsExpiredKeepsRecent(t *testing.T) {
assertChainPresent(t, db, recent)
}
// TestRetentionReaper_SkipsSentinelReapsFiniteInSameSweep covers the
// end-to-end retain-forever path: a webhook created the normal way with
// a requested retention of 0 lands on the RetentionForeverDays
// sentinel, and the reaper leaves its ancient events alone while still
// reaping a finite-retention webhook in the very same sweep.
func TestRetentionReaper_SkipsSentinelReapsFiniteInSameSweep(
t *testing.T,
) {
t.Parallel()
env := setupRetentionTest(t)
foreverID := createWebhookNormally(t, env.mainDB.DB(), 0)
var stored database.Webhook
require.NoError(
t,
env.mainDB.DB().Where("id = ?", foreverID).
First(&stored).Error,
)
require.Equal(
t,
database.RetentionForeverDays,
stored.RetentionDays,
"a requested retention of 0 must persist as the sentinel",
)
finiteID := createWebhookNormally(t, env.mainDB.DB(), 30)
foreverDB, err := env.mgr.GetDB(foreverID)
require.NoError(t, err)
finiteDB, err := env.mgr.GetDB(finiteID)
require.NoError(t, err)
ancient := time.Now().Add(-365 * 24 * time.Hour)
kept := seedEventChain(t, foreverDB, foreverID, ancient)
doomed := seedEventChain(t, finiteDB, finiteID, ancient)
env.reaper.ExportSweep(context.Background())
assertChainPresent(t, foreverDB, kept)
assertChainGone(t, finiteDB, doomed)
}
// TestRetentionReaper_HugeFiniteRetentionRetainsRecentEvents pins the
// overflow that made a large finite retention destroy everything.
//
// The cutoff is a time.Duration, an int64 nanosecond count. A day
// count above MaxFiniteRetentionDays multiplied out unclamped wraps
// negative, so subtracting it moves the cutoff into the far future,
// where "created_at < cutoff" matches every row: an event created a
// moment ago, and its delivery and delivery result, were all deleted
// on the first sweep. 200000 is inside that band and below the
// retain-forever sentinel, so it is treated as a finite policy and
// really does reach the arithmetic.
//
// The row is planted at the column level because such a value can no
// longer be submitted through the form; the point of the test is that
// a row from an older version, or a future call site, still cannot
// trigger the wipe.
func TestRetentionReaper_HugeFiniteRetentionRetainsRecentEvents(
t *testing.T,
) {
t.Parallel()
env := setupRetentionTest(t)
const overflowingRetentionDays = 200000
require.Greater(
t,
overflowingRetentionDays,
database.MaxFiniteRetentionDays,
"the test value must exceed what the cutoff can represent",
)
require.Less(
t,
overflowingRetentionDays,
database.RetentionForeverDays,
"the test value must not be rescued by the forever skip",
)
webhookID := createWebhook(
t, env.mainDB.DB(), overflowingRetentionDays,
)
db, err := env.mgr.GetDB(webhookID)
require.NoError(t, err)
fresh := seedEventChain(t, db, webhookID, time.Now())
env.reaper.ExportSweep(context.Background())
assertChainPresent(t, db, fresh)
}
func TestRetentionReaper_RetainsForeverWhenNonPositive(t *testing.T) {
t.Parallel()
env := setupRetentionTest(t)
// A legacy row written before the sentinel existed still carries a
// literal 0; the <= 0 guard must keep honouring it.
// RetentionDays of zero means retain forever.
webhookID := createWebhook(t, env.mainDB.DB(), 0)
db, err := env.mgr.GetDB(webhookID)

10
internal/database/testing.go
View File

@@ -14,10 +14,7 @@ import (
func NewTestDatabase(db *gorm.DB) *Database {
return &Database{
db: db,
log: slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
log: slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelDebug})),
}
}
@@ -26,9 +23,6 @@ func NewTestDatabase(db *gorm.DB) *Database {
func NewTestWebhookDBManager(dataDir string) *WebhookDBManager {
return &WebhookDBManager{
dataDir: dataDir,
log: slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
log: slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelDebug})),
}
}

29
internal/database/webhook_db_manager_test.go
View File

@@ -2,7 +2,6 @@ package database_test
import (
"context"
"net/http"
"os"
"path/filepath"
"testing"
@@ -26,8 +25,8 @@ func setupTestWebhookDBManager(
lc := fxtest.NewLifecycle(t)
g := &globals.Globals{
Appname: testAppname,
Version: testVersion,
Appname: "webhooker-test",
Version: "test",
}
l, err := logger.New(
@@ -84,10 +83,10 @@ func TestWebhookDBManager_CreateAndGetDB(t *testing.T) {
event := &database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Method: "POST",
Headers: `{"Content-Type":["application/json"]}`,
Body: `{"test": true}`,
ContentType: testContentType,
ContentType: "application/json",
}
require.NoError(t, db.Create(event).Error)
assert.NotEmpty(t, event.ID)
@@ -100,7 +99,7 @@ func TestWebhookDBManager_CreateAndGetDB(t *testing.T) {
db.First(&readEvent, "id = ?", event.ID).Error,
)
assert.Equal(t, webhookID, readEvent.WebhookID)
assert.Equal(t, http.MethodPost, readEvent.Method)
assert.Equal(t, "POST", readEvent.Method)
assert.Equal(t, `{"test": true}`, readEvent.Body)
}
@@ -124,9 +123,9 @@ func TestWebhookDBManager_DeleteDB(t *testing.T) {
event := &database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Method: "POST",
Body: `{"test": true}`,
ContentType: testContentType,
ContentType: "application/json",
}
require.NoError(t, db.Create(event).Error)
@@ -197,10 +196,10 @@ func seedDeliveryWorkflow(
event := &database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Method: "POST",
Headers: `{"Content-Type":["application/json"]}`,
Body: `{"payload": "test"}`,
ContentType: testContentType,
ContentType: "application/json",
}
require.NoError(t, db.Create(event).Error)
@@ -232,7 +231,7 @@ func verifyPendingDeliveries(
)
require.Len(t, pending, 1)
assert.Equal(t, event.ID, pending[0].EventID)
assert.Equal(t, http.MethodPost, pending[0].Event.Method)
assert.Equal(t, "POST", pending[0].Event.Method)
}
func completeDelivery(
@@ -304,16 +303,16 @@ func TestWebhookDBManager_MultipleWebhooks(t *testing.T) {
event1 := &database.Event{
WebhookID: webhook1,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Method: "POST",
Body: `{"webhook": 1}`,
ContentType: testContentType,
ContentType: "application/json",
}
event2 := &database.Event{
WebhookID: webhook2,
EntrypointID: uuid.New().String(),
Method: http.MethodPut,
Method: "PUT",
Body: `{"webhook": 2}`,
ContentType: testContentType,
ContentType: "application/json",
}
require.NoError(t, db1.Create(event1).Error)

229
internal/delivery/archive_sweeper.go
View File

@@ -1,229 +0,0 @@
package delivery
import (
"context"
"errors"
"log/slog"
"sync"
"time"
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/logger"
)
// ArchiveSweeperParams holds the fx dependencies for the
// ArchiveSweeper.
type ArchiveSweeperParams struct {
fx.In
Config *config.Config
Database *database.Database
Engine *Engine
Logger *logger.Logger
}
// ArchiveSweeper periodically prunes expired rows from
// per-webhook archive databases whose database target carries a
// positive expiry.
//
// Without it, pruning happens only when an archive is
// (re)opened, and archives are only ever reopened by writes: an
// archive belonging to a webhook that has stopped receiving
// events would keep its expired rows forever. The sweep closes
// that gap without changing anything for archives whose expiry
// is unset or "never".
//
// It reuses Config.RetentionSweepInterval rather than
// introducing a second interval: this is a retention sweep with
// the same semantics as the event retention reaper.
type ArchiveSweeper struct {
db *database.Database
eng *Engine
log *slog.Logger
interval time.Duration
cancel context.CancelFunc
wg sync.WaitGroup
}
// NewArchiveSweeper creates the archive sweeper and registers
// its fx lifecycle hooks. The background sweep loop starts on
// OnStart and stops cleanly on OnStop via context cancellation.
func NewArchiveSweeper(
lc fx.Lifecycle,
params ArchiveSweeperParams,
) *ArchiveSweeper {
s := &ArchiveSweeper{
db: params.Database,
eng: params.Engine,
log: params.Logger.Get(),
interval: params.Config.RetentionSweepInterval,
}
s.registerHooks(lc)
return s
}
// registerHooks wires the sweeper's start and stop into the fx
// lifecycle. Both hook contexts are deliberately ignored: see
// start for why the background loop must not inherit the start
// hook's context, and stop for why shutdown blocks on the loop
// rather than on the stop hook's deadline.
func (s *ArchiveSweeper) registerHooks(lc fx.Lifecycle) {
lc.Append(fx.Hook{
//nolint:contextcheck // Not passing the hook context is
// the point: see start.
OnStart: func(_ context.Context) error {
s.start()
return nil
},
OnStop: func(_ context.Context) error {
s.stop()
return nil
},
})
}
// start launches the background sweep loop.
//
// The loop's context is derived from context.Background(), NOT
// from the fx OnStart hook context. The hook context carries
// fx's start timeout (15s by default), so a loop derived from it
// is cancelled 15 seconds after the application starts — long
// before the first tick under the default one-hour sweep
// interval, leaving a sweeper that never sweeps. A long-lived
// goroutine must outlive the startup phase, so its lifetime is
// bounded by OnStop instead: stop cancels this context and waits
// on the WaitGroup.
func (s *ArchiveSweeper) start() {
ctx, cancel := context.WithCancel(context.Background())
s.cancel = cancel
s.wg.Add(1)
go s.run(ctx)
s.log.Info(
"archive sweeper started",
"interval", s.interval.String(),
)
}
func (s *ArchiveSweeper) stop() {
s.log.Info("archive sweeper stopping")
if s.cancel != nil {
s.cancel()
}
s.wg.Wait()
s.log.Info("archive sweeper stopped")
}
func (s *ArchiveSweeper) run(ctx context.Context) {
defer s.wg.Done()
ticker := time.NewTicker(s.interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
s.sweep(ctx)
}
}
}
// sweep prunes every archive whose database target declares a
// positive expiry. Targets belonging to a deleted webhook are
// soft-deleted along with it, so GORM's default scope already
// excludes them.
//
// A failure for one webhook is logged and the sweep continues,
// matching how the write path already treats a prune error as
// non-fatal.
func (s *ArchiveSweeper) sweep(ctx context.Context) {
var targets []database.Target
err := s.db.DB().
Model(&database.Target{}).
Where("type = ?", database.TargetTypeDatabase).
Find(&targets).Error
if err != nil {
s.log.Error(
"archive sweep: failed to list database targets",
"error", err,
)
return
}
for i := range targets {
select {
case <-ctx.Done():
return
default:
}
s.sweepTarget(&targets[i])
}
}
// sweepTarget prunes the archive of a single database target.
// A missing, empty, or "never" expiry parses as a zero duration
// and is skipped entirely, so those archives keep exactly the
// behaviour they had before the sweep existed.
func (s *ArchiveSweeper) sweepTarget(target *database.Target) {
expiry, err := parseArchiveExpiry(target.Config)
if err != nil {
s.log.Error(
"archive sweep: invalid database target config",
"webhook_id", target.WebhookID,
"target_id", target.ID,
"error", err,
)
return
}
if expiry <= 0 {
return
}
if s.eng == nil || s.eng.dbTarget == nil {
return
}
err = s.eng.dbTarget.sweepWebhook(target.WebhookID, expiry)
if err == nil {
return
}
// A writer evicted underneath the sweep means the operator
// deleted the webhook (or its last database target) while the
// sweep was walking the target list. That is an ordinary
// interleaving, not a failure, so it must not produce an
// error line.
if errors.Is(err, errArchiveWriterEvicted) {
s.log.Debug(
"archive sweep: writer evicted mid-sweep",
"webhook_id", target.WebhookID,
"target_id", target.ID,
)
return
}
s.log.Error(
"archive sweep: failed to prune archive",
"webhook_id", target.WebhookID,
"target_id", target.ID,
"error", err,
)
}

930
internal/delivery/archive_sweeper_test.go
View File

@@ -1,930 +0,0 @@
package delivery_test
import (
"context"
"database/sql"
"fmt"
"net/http"
"os"
"path/filepath"
"sync"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/fx"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
"gorm.io/gorm/clause"
_ "modernc.org/sqlite" // Pure Go SQLite driver.
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
const (
// sweepRowOld and sweepRowNew are the event ids
// seedArchiveRows assigns to the first and second seeded
// rows.
sweepRowOld = "ev-0"
sweepRowNew = "ev-1"
// sweepConcurrentWrites is how many deliveries the
// concurrent write-plus-sweep test races against the sweep.
sweepConcurrentWrites = 20
)
// sweeperEnv bundles the pieces an archive sweep test drives:
// a main configuration database holding webhooks and targets, a
// delivery engine owning the archive writer registry, and the
// data directory the archive files live in.
type sweeperEnv struct {
sweeper *delivery.ArchiveSweeper
eng *delivery.Engine
mainDB *database.Database
dataDir string
}
func setupSweeperTest(t *testing.T) *sweeperEnv {
t.Helper()
dataDir := t.TempDir()
log := archiveTestLogger()
sqlDB, err := sql.Open(
"sqlite",
fmt.Sprintf(
"file:%s?mode=rwc",
filepath.Join(dataDir, "main.db"),
),
)
require.NoError(t, err)
t.Cleanup(func() { _ = sqlDB.Close() })
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
require.NoError(t, err)
mainDB := database.NewTestDatabase(gdb)
require.NoError(t, mainDB.Migrate())
eng := delivery.NewTestEngineWithDB(
mainDB,
database.NewTestWebhookDBManager(dataDir),
log,
&http.Client{Timeout: 5 * time.Second},
1,
)
return &sweeperEnv{
sweeper: delivery.NewTestArchiveSweeper(
mainDB, eng, log,
),
eng: eng,
mainDB: mainDB,
dataDir: dataDir,
}
}
// archivePath returns where the engine keeps a webhook's
// archive file.
func (env *sweeperEnv) archivePath(webhookID string) string {
return filepath.Join(
env.dataDir, fmt.Sprintf("archive-%s.db", webhookID),
)
}
// seedDatabaseTarget creates a webhook with one database target
// carrying the given target config JSON, and returns the
// webhook id.
func (env *sweeperEnv) seedDatabaseTarget(
t *testing.T, configJSON string,
) string {
t.Helper()
wh := &database.Webhook{
UserID: uuid.New().String(),
Name: "sweep-test",
}
require.NoError(
t,
env.mainDB.DB().
Omit(clause.Associations).
Create(wh).Error,
)
tgt := &database.Target{
WebhookID: wh.ID,
Name: "archive",
Type: database.TargetTypeDatabase,
Active: true,
Config: configJSON,
}
require.NoError(
t,
env.mainDB.DB().
Omit(clause.Associations).
Create(tgt).Error,
)
return wh.ID
}
// seedArchiveRows creates the archive file for a webhook and
// inserts one row per supplied archived-at timestamp, returning
// the archive path. The handle is closed before returning, so
// the archive is idle exactly as it would be with no traffic.
func (env *sweeperEnv) seedArchiveRows(
t *testing.T, webhookID string, archivedAt ...time.Time,
) string {
t.Helper()
path := env.archivePath(webhookID)
sqlDB, err := sql.Open(
"sqlite", fmt.Sprintf("file:%s?mode=rwc", path),
)
require.NoError(t, err)
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
require.NoError(t, err)
require.NoError(
t, gdb.AutoMigrate(&delivery.ExportArchivedEvent{}),
)
for i, at := range archivedAt {
row := delivery.ExportArchivedEvent{
EventID: fmt.Sprintf("ev-%d", i),
WebhookID: webhookID,
Method: http.MethodPost,
Body: `{"seeded":true}`,
ArchivedAt: at,
}
require.NoError(t, gdb.Create(&row).Error)
}
require.NoError(t, sqlDB.Close())
return path
}
// archivedEventIDs returns the event ids currently stored in an
// archive file, read through a separate read-only handle.
func archivedEventIDs(
t *testing.T, path string,
) []string {
t.Helper()
var rows []delivery.ExportArchivedEvent
rdb := openArchiveDBForRead(t, path)
require.NoError(t, rdb.Order("event_id").Find(&rows).Error)
ids := make([]string, 0, len(rows))
for i := range rows {
ids = append(ids, rows[i].EventID)
}
return ids
}
// countArchivedRows counts the rows in an archive file without
// asserting anything, so it is safe to poll from an
// assert.Eventually condition (which runs off the test
// goroutine, where testify assertions must not be used).
func countArchivedRows(path string) (int64, error) {
sqlDB, err := sql.Open(
"sqlite", fmt.Sprintf("file:%s?mode=ro", path),
)
if err != nil {
return 0, err
}
defer func() { _ = sqlDB.Close() }()
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
if err != nil {
return 0, err
}
var count int64
err = gdb.Model(&delivery.ExportArchivedEvent{}).
Count(&count).Error
if err != nil {
return 0, err
}
return count, nil
}
// captureLifecycle is a minimal fx.Lifecycle that records the
// hooks a component registers, so a test can invoke the real
// OnStart/OnStop functions with a context of its choosing.
type captureLifecycle struct {
hooks []fx.Hook
}
func (l *captureLifecycle) Append(h fx.Hook) {
l.hooks = append(l.hooks, h)
}
// TestArchiveSweeper_LoopOutlivesStartHookContext is the
// regression test for a sweeper that never swept. fx calls
// OnStart with a context carrying the application's start
// timeout (15 seconds by default), so a background loop whose
// context is derived from it is cancelled 15 seconds into the
// process — three quarters of an hour before the first tick
// under the default one-hour sweep interval.
//
// The hook context here is already cancelled, which is the same
// defect taken to its limit: a loop that inherits it never runs
// a single tick, while a correctly rooted loop keeps sweeping
// for as long as the process lives. Handing the hook a plain
// context.Background() would assert nothing at all.
func TestArchiveSweeper_LoopOutlivesStartHookContext(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
now := time.Now()
path := env.seedArchiveRows(
t, webhookID,
now.Add(-48*time.Hour),
now.Add(-time.Minute),
)
env.sweeper.ExportSetInterval(10 * time.Millisecond)
// Drive the genuine fx hooks the application registers,
// rather than a test-only entry point.
lc := &captureLifecycle{}
env.sweeper.ExportRegisterHooks(lc)
require.Len(t, lc.hooks, 1)
hookCtx, cancel := context.WithCancel(context.Background())
cancel()
require.NoError(t, lc.hooks[0].OnStart(hookCtx))
t.Cleanup(func() {
_ = lc.hooks[0].OnStop(context.Background())
})
assert.Eventually(
t,
func() bool {
count, err := countArchivedRows(path)
return err == nil && count == 1
},
5*time.Second,
10*time.Millisecond,
"the sweep loop must keep running after the start "+
"hook's context is done; it pruned nothing, so it "+
"inherited the hook context and died",
)
}
// TestArchiveSweep_DoesNotResurrectEvictedWriter covers the
// interleaving where a sweep tick has already listed a webhook's
// target when the webhook is deleted and its writer evicted. The
// sweep must not put a writer back into the registry: nothing
// would ever evict it again, which is precisely the leak this
// change exists to close.
func TestArchiveSweep_DoesNotResurrectEvictedWriter(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
// Prime the registry the way a delivery would, then evict as
// the deletion path does. The target row is deliberately left
// in place: this is the tick that listed the webhook before
// the deletion committed.
_, err := env.eng.ExportEnsureArchiveWriter(webhookID)
require.NoError(t, err)
env.eng.EvictWebhook(webhookID)
require.False(t, env.eng.ExportHasArchiveWriter(webhookID))
env.sweeper.ExportSweep(context.Background())
assert.False(
t, env.eng.ExportHasArchiveWriter(webhookID),
"a sweep must never re-register a writer for a webhook "+
"whose registry entry has already been released",
)
}
// TestArchiveSweep_LeavesNoRegistryEntry states the same
// invariant in its general form: sweeping an archive whose
// webhook has no cached writer must not leave one behind, so the
// registry keeps holding only writers a delivery created and an
// eviction can reach.
func TestArchiveSweep_LeavesNoRegistryEntry(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path := env.seedArchiveRows(
t, webhookID,
time.Now().Add(-48*time.Hour),
time.Now().Add(-time.Minute),
)
require.False(t, env.eng.ExportHasArchiveWriter(webhookID))
env.sweeper.ExportSweep(context.Background())
assert.Equal(
t, []string{sweepRowNew}, archivedEventIDs(t, path),
"the sweep must still prune an idle archive",
)
assert.False(
t, env.eng.ExportHasArchiveWriter(webhookID),
"the sweep must release the registry entry it created",
)
}
// TestArchiveSweep_KeepsWriterAdoptedByDelivery is the other
// half of that invariant: an entry the sweep created but a
// delivery then claimed belongs to the registry and must survive
// the sweep, or the delivery would be left holding a detached
// writer with an open handle that no eviction can reach.
func TestArchiveSweep_KeepsWriterAdoptedByDelivery(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"n":1}`)
event.WebhookID = webhookID
d := seedDatabaseTargetDelivery(
t, webhookDB, event, `{"expiry":"1h"}`,
)
env.sweeper.ExportSweep(context.Background())
require.False(t, env.eng.ExportHasArchiveWriter(webhookID))
env.eng.ExportDeliverDatabase(webhookDB, d)
assert.True(
t, env.eng.ExportHasArchiveWriter(webhookID),
"a delivery's writer must stay registered",
)
env.sweeper.ExportSweep(context.Background())
assert.True(
t, env.eng.ExportHasArchiveWriter(webhookID),
"a sweep must not drop a writer a delivery owns",
)
}
// TestArchiveSweep_KeepsWriterAdoptedDuringSweep covers the one
// interleaving the sweepOwned flag exists for, which
// TestArchiveSweep_KeepsWriterAdoptedByDelivery cannot reach: a
// delivery adopting the sweep's own entry WHILE that sweep is
// still running.
//
// The registry operations are driven directly, in the order the
// sweep and a concurrent delivery perform them, so the window is
// exercised deterministically rather than hoped for:
//
// 1. the sweep finds no cached writer and registers one of its
// own, marked sweep-owned;
// 2. a delivery arrives, is handed that very writer, clears the
// flag and opens the archive handle;
// 3. the sweep finishes and releases what it created.
//
// Step 3 must leave the entry alone. Dropping it would detach a
// writer that is holding an open archive handle inside its
// debounce window, and no eviction could ever reach it again —
// exactly the process-lifetime handle leak this change exists to
// close. The eviction at the end proves the entry is still
// reachable.
func TestArchiveSweep_KeepsWriterAdoptedDuringSweep(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
sweepWriter, created, err := env.eng.ExportSweepWriterFor(
webhookID,
)
require.NoError(t, err)
require.True(
t, created,
"the sweep must have created the registry entry itself",
)
// The delivery lands mid-sweep and adopts the entry.
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"n":1}`)
event.WebhookID = webhookID
d := seedDatabaseTargetDelivery(
t, webhookDB, event, `{"expiry":"1h"}`,
)
env.eng.ExportDeliverDatabase(webhookDB, d)
adopted := env.eng.ExportArchiveWriterFor(webhookID)
require.NotNil(t, adopted)
require.True(
t, sweepWriter.Same(adopted),
"the delivery must have adopted the sweep's writer",
)
require.True(
t, env.eng.ExportArchiveHandleOpen(webhookID),
"the delivery leaves the archive handle open",
)
// The sweep finishes.
env.eng.ExportReleaseSweepWriter(webhookID, sweepWriter)
require.True(
t, env.eng.ExportHasArchiveWriter(webhookID),
"a writer adopted by a delivery during a sweep must "+
"stay registered, or its open handle is unreachable",
)
env.eng.EvictWebhook(webhookID)
assert.False(
t, env.eng.ExportHasArchiveWriter(webhookID),
"the adopted writer must still be evictable",
)
assert.False(
t, sweepWriter.HandleOpen(),
"eviction must have closed the adopted writer's handle",
)
}
// TestArchiveSweep_ContinuesAfterPerWebhookFailure proves a
// failure for one webhook does not abort the sweep for the
// others: an unparseable expiry and an unreadable archive both
// have to be logged and stepped over.
func TestArchiveSweep_ContinuesAfterPerWebhookFailure(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
// Seeded first so the sweep reaches them before the healthy
// webhook: targets come back in insertion order.
badConfigID := env.seedDatabaseTarget(t, `{"expiry":"!!!"}`)
env.seedArchiveRows(
t, badConfigID, time.Now().Add(-48*time.Hour),
)
corruptID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
require.NoError(t, os.WriteFile(
env.archivePath(corruptID),
[]byte("this is not a sqlite database"),
0o600,
))
healthyID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
healthyPath := env.seedArchiveRows(
t, healthyID,
time.Now().Add(-48*time.Hour),
time.Now().Add(-time.Minute),
)
env.sweeper.ExportSweep(context.Background())
assert.Equal(
t, []string{sweepRowNew},
archivedEventIDs(t, healthyPath),
"a failure for an earlier webhook must not stop the "+
"sweep from pruning the ones after it",
)
}
// TestArchiveSweep_OpenExistingDoesNotCreateFile pins the second
// of the two no-create guards. The first is the stat in
// sweepWebhook; this one is the SQLite open mode, which is what
// protects the window between that stat and the open. Flipping
// the sweep's mode to create-if-missing makes this fail.
func TestArchiveSweep_OpenExistingDoesNotCreateFile(
t *testing.T,
) {
t.Parallel()
dir := t.TempDir()
path := filepath.Join(dir, "archive-absent.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
err := w.OpenExisting(time.Hour)
require.Error(
t, err,
"opening a missing archive without create permission "+
"must fail rather than conjure the file",
)
for _, suffix := range archiveFileSuffixes() {
assert.NoFileExists(t, path+suffix)
}
}
// TestArchiveSweep_PrunesIdleArchive is the core regression
// test for this issue: an archive that receives no further
// writes must still lose its expired rows. Before the sweeper
// existed, pruning only ever ran on a write-triggered reopen,
// so an idle archive kept expired rows forever.
func TestArchiveSweep_PrunesIdleArchive(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
now := time.Now()
path := env.seedArchiveRows(
t, webhookID,
now.Add(-48*time.Hour),
now.Add(-time.Minute),
)
require.Equal(
t, []string{sweepRowOld, sweepRowNew},
archivedEventIDs(t, path),
)
env.sweeper.ExportSweep(context.Background())
assert.Equal(
t, []string{sweepRowNew}, archivedEventIDs(t, path),
"the sweep should prune rows older than the expiry "+
"from an idle archive and keep the rest",
)
}
// TestArchiveSweep_LeavesArchiveClosed proves the sweep does
// not hold the archive open afterwards, so an operator can
// still move the file away for offline retention.
//
// The assertion is made on a writer the test holds a reference
// to, and the handle is proven OPEN before the sweep runs, so the
// test observes the sweep closing it rather than a writer that
// merely never opened anything. Asking the registry instead would
// be vacuous here: the sweep releases an entry it created, and a
// missing entry reports "not open" whether or not anything was
// closed.
func TestArchiveSweep_LeavesArchiveClosed(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path := env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.OpenExisting(time.Hour))
require.True(
t, w.HandleOpen(),
"the writer must hold an open handle before the sweep",
)
require.NoError(t, w.SweepExpired(time.Hour))
assert.False(
t, w.HandleOpen(),
"an idle archive must end the sweep closed",
)
}
// TestArchiveSweep_ClosesHandleOfRegisteredWriter states the same
// guarantee end to end, through the real sweeper and a writer the
// registry keeps.
//
// The delivery leaves the archive handle open inside its debounce
// window and makes the entry delivery-owned, so the sweep finds a
// cached writer (created is false, nothing is released) and the
// registry query afterwards is answered by a writer that really
// exists. A handle left open here would be doubly wrong: it also
// blocks the operator's move-the-file-away workflow.
func TestArchiveSweep_ClosesHandleOfRegisteredWriter(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"n":1}`)
event.WebhookID = webhookID
d := seedDatabaseTargetDelivery(
t, webhookDB, event, `{"expiry":"1h"}`,
)
env.eng.ExportDeliverDatabase(webhookDB, d)
require.True(
t, env.eng.ExportArchiveHandleOpen(webhookID),
"the delivery must leave the archive handle open",
)
env.sweeper.ExportSweep(context.Background())
require.True(
t, env.eng.ExportHasArchiveWriter(webhookID),
"the delivery's registry entry must survive the sweep",
)
assert.False(
t, env.eng.ExportArchiveHandleOpen(webhookID),
"the sweep must leave the archive closed",
)
}
// TestArchiveSweep_NeverExpiryUntouched proves the sweep is a
// no-op for the default retention policy, so archives with no
// expiry (or the literal "never") behave exactly as before.
func TestArchiveSweep_NeverExpiryUntouched(t *testing.T) {
t.Parallel()
for _, configJSON := range []string{
`{"expiry":"never"}`,
`{"expiry":""}`,
"",
} {
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, configJSON)
path := env.seedArchiveRows(
t, webhookID,
time.Now().Add(-10000*time.Hour),
)
env.sweeper.ExportSweep(context.Background())
assert.Equal(
t, []string{sweepRowOld}, archivedEventIDs(t, path),
"config %q must keep rows forever", configJSON,
)
assert.False(
t, env.eng.ExportHasArchiveWriter(webhookID),
"config %q must leave no registry entry behind",
configJSON,
)
}
}
// TestArchiveSweep_NeverExpirySkipsBeforeOpening pins the
// expiry <= 0 boundary in sweepTarget, which the row assertions
// above cannot reach: pruning is separately gated on a positive
// expiry, so a "never" archive keeps its rows even if the sweep
// does open it.
//
// The spec is stronger than that — a "never" archive is skipped
// before any file is touched — so the archive here exists but has
// never been migrated. Opening it at all would run AutoMigrate
// and create the archive table, which is exactly what must not
// happen.
func TestArchiveSweep_NeverExpirySkipsBeforeOpening(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"never"}`)
path := env.archivePath(webhookID)
seedUnmigratedArchive(t, path)
require.False(t, archiveTableExists(t, path))
env.sweeper.ExportSweep(context.Background())
assert.False(
t, archiveTableExists(t, path),
"a never-expiry archive must not be opened at all",
)
}
// seedUnmigratedArchive creates an archive file that exists but
// carries no archive schema, so any open of it is observable: the
// archive table appears only if something ran AutoMigrate.
func seedUnmigratedArchive(t *testing.T, path string) {
t.Helper()
sqlDB, err := sql.Open(
"sqlite", fmt.Sprintf("file:%s?mode=rwc", path),
)
require.NoError(t, err)
_, err = sqlDB.ExecContext(
t.Context(), "CREATE TABLE placeholder (id INTEGER)",
)
require.NoError(t, err)
require.NoError(t, sqlDB.Close())
}
// archiveTableExists reports whether an archive file has had the
// archive schema migrated into it.
func archiveTableExists(t *testing.T, path string) bool {
t.Helper()
return openArchiveDBForRead(t, path).
Migrator().
HasTable(&delivery.ExportArchivedEvent{})
}
// TestArchiveSweep_DoesNotCreateArchiveFile proves the sweep
// never conjures an archive: a webhook with a database target
// that has never received an event must still have no archive
// file (nor SQLite sidecar) after a sweep.
func TestArchiveSweep_DoesNotCreateArchiveFile(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path := env.archivePath(webhookID)
require.NoFileExists(t, path)
env.sweeper.ExportSweep(context.Background())
for _, suffix := range archiveFileSuffixes() {
assert.NoFileExists(
t, path+suffix,
"the sweep must not create an archive file",
)
}
}
// TestArchiveSweep_DoesNotCreateAfterWriterExists covers the
// same guarantee once a writer is cached in the registry but
// the file itself is still absent (for instance because the
// operator moved the archive away).
func TestArchiveSweep_DoesNotCreateAfterWriterExists(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path, err := env.eng.ExportEnsureArchiveWriter(webhookID)
require.NoError(t, err)
require.NoFileExists(t, path)
env.sweeper.ExportSweep(context.Background())
assert.NoFileExists(t, path)
}
// TestArchiveSweep_SkipsDeletedWebhookTargets proves that the
// sweep ignores targets soft-deleted along with their webhook,
// so a deleted webhook's archive is never reopened.
func TestArchiveSweep_SkipsDeletedWebhookTargets(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path := env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
require.NoError(
t,
env.mainDB.DB().
Where("webhook_id = ?", webhookID).
Delete(&database.Target{}).Error,
)
env.sweeper.ExportSweep(context.Background())
assert.Equal(
t, []string{sweepRowOld}, archivedEventIDs(t, path),
"a deleted target's archive must be left alone",
)
}
// TestArchiveSweep_ConcurrentWrites proves the sweep serialises
// against writes through the per-webhook writer mutex. Run
// under -race, an unsynchronised sweep would be caught here.
func TestArchiveSweep_ConcurrentWrites(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
webhookDB := testWebhookDB(t)
// The deliveries are seeded up front, on the test's own
// goroutine: the seed helpers assert, and testify assertions
// must not run off the test goroutine.
deliveries := make(
[]*database.Delivery, 0, sweepConcurrentWrites,
)
for range sweepConcurrentWrites {
event := seedEvent(t, webhookDB, `{"n":1}`)
event.WebhookID = webhookID
deliveries = append(
deliveries,
seedDatabaseTargetDelivery(
t, webhookDB, event, `{"expiry":"1h"}`,
),
)
}
var wg sync.WaitGroup
wg.Add(2)
go func() {
defer wg.Done()
for _, d := range deliveries {
env.eng.ExportDeliverDatabase(webhookDB, d)
}
}()
go func() {
defer wg.Done()
for range sweepConcurrentWrites {
env.sweeper.ExportSweep(context.Background())
}
}()
wg.Wait()
assert.FileExists(t, env.archivePath(webhookID))
}
// TestArchiveSweeper_StopsCleanly proves the background loop
// exits on OnStop rather than leaking a goroutine.
func TestArchiveSweeper_StopsCleanly(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
env.sweeper.ExportSetInterval(time.Millisecond)
env.sweeper.ExportStart()
// stop blocks on the loop's WaitGroup, so returning at all
// proves the loop observed the cancellation and exited.
env.sweeper.ExportStop()
}

153
internal/delivery/engine.go
View File

@@ -94,23 +94,6 @@ type Notifier interface {
Notify(tasks []Task)
}
// WebhookEvictor releases the delivery engine's per-webhook
// state for a webhook that no longer needs it — currently the
// cached archive writer of the database target, whose open
// file handle would otherwise outlive the webhook.
//
// It is deliberately separate from Notifier and deliberately
// one method wide: archiving lifecycle is not notification, and
// a single-method interface keeps the handlers package free of
// any dependency on the engine's internals while staying
// trivially fakeable in tests.
//
// EvictWebhook never deletes an archive file. It is idempotent
// and is a no-op for a webhook with no engine state.
type WebhookEvictor interface {
EvictWebhook(webhookID string)
}
// EngineParams are the fx dependencies for the delivery
// engine.
type EngineParams struct {
@@ -144,10 +127,6 @@ type Engine struct {
// httpTarget is retained so tests can reach the HTTP
// target's shared client and circuit breakers.
httpTarget *httpTarget
// dbTarget is retained so the engine can reach the archive
// writer registry for webhook eviction and the idle sweep.
dbTarget *databaseTarget
}
// New creates and registers the delivery engine with the
@@ -170,7 +149,18 @@ func New(
Transport: NewSSRFSafeTransport(),
})
e.registerHooks(lc)
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
e.start(ctx)
return nil
},
OnStop: func(_ context.Context) error {
e.stop()
return nil
},
})
return e
}
@@ -192,19 +182,6 @@ func (e *Engine) Notify(tasks []Task) {
}
}
// EvictWebhook implements WebhookEvictor. It releases the
// engine's per-webhook archiving state: the database target's
// cached archive writer is dropped from the registry and its
// file handle closed. The archive file itself is left on disk
// — it is long-term storage the operator owns.
func (e *Engine) EvictWebhook(webhookID string) {
if e.dbTarget == nil {
return
}
e.dbTarget.evict(webhookID)
}
// ScheduleRetry schedules a task to be re-enqueued onto the
// retry channel after delay. It implements the Scheduler
// interface the targets use to own their durable retries.
@@ -233,40 +210,8 @@ func (e *Engine) ScheduleRetry(
})
}
// registerHooks wires the engine's start and stop into the fx
// lifecycle. The start hook's context is deliberately ignored:
// see start for why the worker pool must not inherit it.
func (e *Engine) registerHooks(lc fx.Lifecycle) {
lc.Append(fx.Hook{
//nolint:contextcheck // Not inheriting the hook context
// is the point: see start.
OnStart: func(_ context.Context) error {
e.start()
return nil
},
OnStop: func(_ context.Context) error {
e.stop()
return nil
},
})
}
// start launches the worker pool, restart recovery, and the
// periodic retry sweep.
//
// Their context is derived from context.Background(), NOT from
// the fx OnStart hook context. The hook context carries fx's
// start timeout (15s by default) and is cancelled once the start
// phase completes, so goroutines derived from it stop a few
// seconds into the process: every worker would return and the
// engine would silently stop delivering webhooks entirely. A
// long-lived goroutine must outlive the startup phase, so its
// lifetime is bounded by OnStop instead: stop cancels this
// context and waits on the WaitGroup.
func (e *Engine) start() {
ctx, cancel := context.WithCancel(context.Background())
func (e *Engine) start(ctx context.Context) {
ctx, cancel := context.WithCancel(ctx)
e.cancel = cancel
for range e.workers {
@@ -508,9 +453,8 @@ func (e *Engine) recoverRetryingDeliveries(
// recoverSingleRetry hands an orphaned retrying delivery back
// to its target to recompute the remaining backoff, then
// reschedules it. Targets that do not own durable retries
// (fire-and-forget) never produce retrying deliveries, so a
// delivery found in that state has had its target's type
// changed underneath it and is terminally failed.
// (fire-and-forget) never produce retrying deliveries, so
// they are skipped.
func (e *Engine) recoverSingleRetry(
webhookDB *gorm.DB,
webhookID string,
@@ -531,10 +475,6 @@ func (e *Engine) recoverSingleRetry(
rs, ok := e.targets[target.Type].(rescheduler)
if !ok {
e.failUnretryableRetry(
webhookDB, webhookID, d, &target,
)
return
}
@@ -709,8 +649,8 @@ func (e *Engine) sweepWebhookRetries(
// sweepSingleRetry re-enqueues an orphaned retrying delivery
// whose backoff window has elapsed, delegating the backoff
// decision to the delivery's target. A delivery whose target
// no longer owns durable retries is terminally failed.
// decision to the delivery's target. Targets that do not own
// durable retries are skipped.
func (e *Engine) sweepSingleRetry(
webhookDB *gorm.DB,
webhookID string,
@@ -730,10 +670,6 @@ func (e *Engine) sweepSingleRetry(
rs, ok := e.targets[target.Type].(rescheduler)
if !ok {
e.failUnretryableRetry(
webhookDB, webhookID, d, &target,
)
return
}
@@ -774,59 +710,6 @@ func (e *Engine) sweepSingleRetry(
}
}
// failUnretryableRetry terminally fails an orphaned retrying
// delivery whose target type no longer supports retries. Both
// restart recovery and the periodic sweep call it, so the
// terminal transition exists once.
//
// This is only reachable when a target's type has been changed
// out from under an in-flight retrying delivery (or the type is
// unknown to the registry): fire-and-forget targets never set
// status retrying themselves. Re-dispatching under the new type
// would be a delivery the operator never asked for, and leaving
// the row retrying strands it forever, so the delivery is
// failed with a recorded reason and can be redelivered
// manually. Logged at warn, not error: this is operator-caused
// state, not a system fault.
func (e *Engine) failUnretryableRetry(
webhookDB *gorm.DB,
webhookID string,
d *database.Delivery,
target *database.Target,
) {
e.log.Warn(
"failing orphaned retrying delivery: target "+
"type no longer supports retries",
"webhook_id", webhookID,
"delivery_id", d.ID,
"target_id", target.ID,
"target_name", target.Name,
"target_type", target.Type,
)
reason := fmt.Sprintf(
"target type %q does not support retries; "+
"delivery was left retrying by a previous "+
"target type and has been failed terminally",
target.Type,
)
e.recordResult(
webhookDB,
d,
e.countAttempts(webhookDB, d.ID)+1,
false,
0,
"",
reason,
0,
)
e.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusFailed,
)
}
// processDelivery dispatches a delivery to the target that
// owns its type. Unknown target types fail the delivery.
func (e *Engine) processDelivery(

252
internal/delivery/engine_integration_test.go
View File

@@ -126,6 +126,36 @@ func iHTTPConfig(url string) string {
return string(data)
}
func iWebhookDB(t *testing.T) *gorm.DB {
t.Helper()
dbPath := filepath.Join(
t.TempDir(), "events-test.db",
)
dsn := fmt.Sprintf(
"file:%s?cache=shared&mode=rwc", dbPath,
)
sqlDB, err := sql.Open("sqlite", dsn)
require.NoError(t, err)
t.Cleanup(func() { _ = sqlDB.Close() })
db, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
require.NoError(t, err)
require.NoError(t, db.AutoMigrate(
&database.Event{},
&database.Delivery{},
&database.DeliveryResult{},
))
return db
}
func iEngine(
t *testing.T, workers int,
) *delivery.Engine {
@@ -152,10 +182,10 @@ func iSeedEvent(
event := database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Method: "POST",
Headers: `{}`,
Body: body,
ContentType: testContentType,
ContentType: "application/json",
}
require.NoError(t, db.Create(&event).Error)
@@ -476,7 +506,7 @@ func TestWorkerLifecycle_StartStop(t *testing.T) {
t.Parallel()
s := newISetup(t)
s.Engine.ExportStart()
s.Engine.ExportStart(context.Background())
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID,
@@ -499,17 +529,21 @@ func TestWorkerLifecycle_StartStop(t *testing.T) {
s.Engine.Notify([]delivery.Task{task})
iWaitForDelivered(t, s.WebhookDB, d.ID)
iWaitForStatus(
t, s.WebhookDB, d.ID,
database.DeliveryStatusDelivered,
)
s.Engine.ExportStop()
}
// iWaitForDelivered polls until the delivery reaches the
// delivered status.
func iWaitForDelivered(
// iWaitForStatus polls until the delivery reaches the
// expected status.
func iWaitForStatus(
t *testing.T,
db *gorm.DB,
deliveryID string,
expected database.DeliveryStatus,
) {
t.Helper()
@@ -523,7 +557,7 @@ func iWaitForDelivered(
return false
}
return d.Status == database.DeliveryStatusDelivered
return d.Status == expected
}, 5*time.Second, 50*time.Millisecond)
}
@@ -554,7 +588,7 @@ func TestWorkerLifecycle_ProcessesRetryChannel(
database.DeliveryStatusRetrying,
)
s.Engine.ExportStart()
s.Engine.ExportStart(context.Background())
bodyStr := event.Body
cfg := iHTTPConfig(ts.URL)
@@ -565,7 +599,10 @@ func TestWorkerLifecycle_ProcessesRetryChannel(
s.Engine.ExportRetryCh() <- task
iWaitForDelivered(t, s.WebhookDB, d.ID)
iWaitForStatus(
t, s.WebhookDB, d.ID,
database.DeliveryStatusDelivered,
)
s.Engine.ExportStop()
}
@@ -741,193 +778,6 @@ func TestRecoverWebhookDeliveries_RetryingDeliveries(
case <-time.After(5 * time.Second):
t.Fatal("expected retry task from recovery")
}
// Regression guard: a target that still supports retries
// must be rescheduled, never terminally failed, and must
// not gain a synthetic result row.
iAssertStatus(
t, s.WebhookDB, d.ID,
database.DeliveryStatusRetrying,
)
assert.Len(t, iResults(t, s.WebhookDB, d.ID), 1)
}
// --- Retrying deliveries whose target type changed ---
// iSeedRetryingWithType seeds a retrying delivery with one
// recorded failed attempt against a target of the given type,
// standing in for a target whose type was edited in the main
// database while the delivery was still retrying.
func iSeedRetryingWithType(
t *testing.T,
s iSetup,
targetType database.TargetType,
) string {
t.Helper()
targetID := uuid.New().String()
iCreateTarget(t, s.MainDB, targetID,
s.WebhookID, "mutated-target", targetType,
iHTTPConfig("http://example.com/hook"), 5,
)
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID,
`{"orphaned":"retry"}`,
)
d := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusRetrying,
)
iSeedFailedResult(t, s.WebhookDB, d.ID)
return d.ID
}
// iResults loads a delivery's results in attempt order.
func iResults(
t *testing.T, db *gorm.DB, deliveryID string,
) []database.DeliveryResult {
t.Helper()
var results []database.DeliveryResult
require.NoError(t, db.
Where("delivery_id = ?", deliveryID).
Order("attempt_num").
Find(&results).Error)
return results
}
// iAssertTerminallyFailed asserts the delivery ended failed
// with a result row recording why, and was not rescheduled.
func iAssertTerminallyFailed(
t *testing.T,
s iSetup,
deliveryID string,
targetType database.TargetType,
) {
t.Helper()
iAssertStatus(
t, s.WebhookDB, deliveryID,
database.DeliveryStatusFailed,
)
results := iResults(t, s.WebhookDB, deliveryID)
require.Len(t, results, 2)
last := results[1]
assert.False(t, last.Success)
assert.Equal(t, 2, last.AttemptNum)
assert.Contains(
t, last.Error, string(targetType),
)
assert.Contains(
t, last.Error, "does not support retries",
)
assert.Empty(t, s.Engine.ExportRetryCh())
}
func TestRecoverSingleRetry_TypeNoLongerRetries(
t *testing.T,
) {
t.Parallel()
s := newISetup(t)
iCreateWebhook(
t, s.MainDB, s.WebhookID, "mutated-type",
)
deliveryID := iSeedRetryingWithType(
t, s, database.TargetTypeLog,
)
s.Engine.ExportRecoverWebhookDeliveries(
context.Background(), s.WebhookID,
)
iAssertTerminallyFailed(
t, s, deliveryID, database.TargetTypeLog,
)
}
func TestSweepSingleRetry_TypeNoLongerRetries(
t *testing.T,
) {
t.Parallel()
s := newISetup(t)
iCreateWebhook(
t, s.MainDB, s.WebhookID, "mutated-type-sweep",
)
deliveryID := iSeedRetryingWithType(
t, s, database.TargetTypeDatabase,
)
s.Engine.ExportSweepWebhookRetries(
context.Background(), s.WebhookID,
)
iAssertTerminallyFailed(
t, s, deliveryID, database.TargetTypeDatabase,
)
}
func TestRecoverSingleRetry_UnknownTargetType(
t *testing.T,
) {
t.Parallel()
s := newISetup(t)
iCreateWebhook(
t, s.MainDB, s.WebhookID, "unknown-type",
)
unknown := database.TargetType("not-a-target-type")
deliveryID := iSeedRetryingWithType(t, s, unknown)
s.Engine.ExportRecoverWebhookDeliveries(
context.Background(), s.WebhookID,
)
iAssertTerminallyFailed(t, s, deliveryID, unknown)
}
func TestSweepSingleRetry_UnknownTargetType(
t *testing.T,
) {
t.Parallel()
s := newISetup(t)
iCreateWebhook(
t, s.MainDB, s.WebhookID, "unknown-type-sweep",
)
unknown := database.TargetType("not-a-target-type")
deliveryID := iSeedRetryingWithType(t, s, unknown)
s.Engine.ExportSweepWebhookRetries(
context.Background(), s.WebhookID,
)
iAssertTerminallyFailed(t, s, deliveryID, unknown)
}
// iSeedFailedResult creates a failed delivery result.
@@ -1085,7 +935,7 @@ func TestDeliverHTTP_CustomTargetHeaders(t *testing.T) {
func TestDeliverHTTP_TargetTimeout(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
db := iWebhookDB(t)
e := iEngine(t, 1)
ts := httptest.NewServer(
@@ -1137,10 +987,10 @@ func iSeedEventAndDelivery(
event := database.Event{
WebhookID: uuid.New().String(),
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Method: "POST",
Headers: `{"Content-Type":["application/json"]}`,
Body: body,
ContentType: testContentType,
ContentType: "application/json",
}
require.NoError(t, db.Create(&event).Error)
@@ -1217,7 +1067,7 @@ func iAssertResultFailed(
func TestDeliverHTTP_InvalidConfig(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
db := iWebhookDB(t)
e := iEngine(t, 1)
event, del := iSeedEventAndDelivery(

199
internal/delivery/engine_lifecycle_test.go
View File

@@ -1,199 +0,0 @@
package delivery_test
import (
"context"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/require"
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
const (
// hookStopTimeout bounds how long a lifecycle test waits for
// the engine's OnStop hook to return before declaring the
// shutdown hung.
hookStopTimeout = 10 * time.Second
// hookSettleDelay is how long startEngineViaHook waits after
// OnStart before the caller may enqueue work. A worker pool
// wrongly rooted in the already-done hook context has nothing
// but ctx.Done() ready in its select, so it is deterministically
// gone by the end of this window. Without the wait, Notify would
// race the pool's very first select, in which a ready ctx.Done()
// and a ready deliveryCh are chosen between at random and a
// doomed pool still delivers.
hookSettleDelay = 250 * time.Millisecond
)
// recordingLifecycle is a minimal fx.Lifecycle that records the
// hooks a component registers, so a test can invoke the real
// OnStart/OnStop functions with a context of its choosing.
type recordingLifecycle struct {
hooks []fx.Hook
}
func (l *recordingLifecycle) Append(h fx.Hook) {
l.hooks = append(l.hooks, h)
}
// startEngineViaHook drives the genuine fx hooks the application
// registers for the engine, handing OnStart a context that is
// already done, and returns only once a pool that inherited that
// context would have exited. It returns the recorded lifecycle so
// the caller can drive OnStop too.
//
// Callers must not seed pending or retrying deliveries before
// calling this: restart recovery enqueues those during startup,
// which would put work in the queue while the pool is still
// racing its first select.
func startEngineViaHook(
t *testing.T, eng *delivery.Engine,
) *recordingLifecycle {
t.Helper()
lc := &recordingLifecycle{}
eng.ExportRegisterHooks(lc)
require.Len(t, lc.hooks, 1)
// fx hands OnStart a context carrying the application start
// timeout, and cancels it when the start phase ends. An
// already-cancelled context is that same defect taken to its
// limit, and unlike a plain context.Background() it actually
// distinguishes a correctly rooted loop from a broken one.
hookCtx, cancel := context.WithCancel(context.Background())
cancel()
require.NoError(t, lc.hooks[0].OnStart(hookCtx))
time.Sleep(hookSettleDelay)
return lc
}
// seedLogTask seeds a pending delivery for a log target and
// returns its ID together with the task that drives it. The log
// target needs no network, so a delivery completing proves only
// that a worker picked the task up.
func seedLogTask(
t *testing.T, s iSetup,
) (string, delivery.Task) {
t.Helper()
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID,
`{"lifecycle":"hook-context"}`,
)
targetID := uuid.New().String()
d := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusPending,
)
bodyStr := event.Body
task := iTask(
d, event, s.WebhookID, targetID,
"hook-context-test", "", 0, 1, &bodyStr,
)
task.TargetType = database.TargetTypeLog
return d.ID, task
}
// TestEngine_WorkersOutliveStartHookContext is the regression
// test for a delivery engine that stopped delivering roughly
// fifteen seconds after boot. fx calls OnStart with a context
// carrying the application's start timeout (15s by default) and
// cancels it when the start phase ends, so a worker pool rooted
// in it exits shortly after startup: the process keeps accepting
// and persisting events while nothing at all forwards them.
//
// Driving OnStart with an already-cancelled context is that
// defect taken to its limit. A pool that inherits the hook
// context is gone before the task is even enqueued; a correctly
// rooted pool keeps working for as long as the process lives.
func TestEngine_WorkersOutliveStartHookContext(t *testing.T) {
t.Parallel()
s := newISetup(t)
lc := startEngineViaHook(t, s.Engine)
t.Cleanup(func() {
_ = lc.hooks[0].OnStop(context.Background())
})
// Seeded only after the pool has settled, so restart recovery
// cannot enqueue it during startup.
deliveryID, task := seedLogTask(t, s)
s.Engine.Notify([]delivery.Task{task})
iWaitForDelivered(t, s.WebhookDB, deliveryID)
}
// TestEngine_StopHookStopsWorkers proves the fix did not trade a
// startup bug for a shutdown hang: now that the worker pool no
// longer observes the start hook's cancellation, OnStop is the
// only thing that can stop it, and it must both return promptly
// and actually leave the pool drained.
func TestEngine_StopHookStopsWorkers(t *testing.T) {
t.Parallel()
s := newISetup(t)
lc := startEngineViaHook(t, s.Engine)
// Let the pool prove it is running before stopping it, so a
// fast OnStop cannot pass by stopping something already dead.
firstID, firstTask := seedLogTask(t, s)
s.Engine.Notify([]delivery.Task{firstTask})
iWaitForDelivered(t, s.WebhookDB, firstID)
var stopErr error
stopped := make(chan struct{})
go func() {
defer close(stopped)
// stop blocks on the workers' WaitGroup, so returning at
// all proves every goroutine observed the cancellation.
stopErr = lc.hooks[0].OnStop(context.Background())
}()
select {
case <-stopped:
case <-time.After(hookStopTimeout):
t.Fatal(
"OnStop did not return: the delivery engine's " +
"WaitGroup is still waiting on a goroutine that " +
"never observed cancellation",
)
}
require.NoError(t, stopErr)
// With every worker gone, a freshly notified task must sit
// untouched in the queue rather than being delivered.
secondID, secondTask := seedLogTask(t, s)
s.Engine.Notify([]delivery.Task{secondTask})
time.Sleep(200 * time.Millisecond)
var after database.Delivery
require.NoError(
t,
s.WebhookDB.First(&after, "id = ?", secondID).Error,
)
require.Equal(
t,
database.DeliveryStatusPending,
after.Status,
"a stopped engine must not deliver anything",
)
}

79
internal/delivery/engine_test.go
View File

@@ -27,9 +27,6 @@ import (
"sneak.berlin/go/webhooker/internal/delivery"
)
// testContentType is the event content type used in tests.
const testContentType = "application/json"
func testWebhookDB(t *testing.T) *gorm.DB {
t.Helper()
@@ -97,10 +94,10 @@ func seedEvent(
event := database.Event{
WebhookID: uuid.New().String(),
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Method: "POST",
Headers: `{"Content-Type":["application/json"]}`,
Body: body,
ContentType: testContentType,
ContentType: "application/json",
}
require.NoError(t, db.Create(&event).Error)
@@ -345,29 +342,33 @@ func TestDeliverDatabase_ImmediateSuccess(
t.Parallel()
db := testWebhookDB(t)
// The database target archives for real now, so the engine
// needs a webhook DB manager to locate the data directory.
e := delivery.NewTestEngineWithDB(
nil,
database.NewTestWebhookDBManager(t.TempDir()),
slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
&http.Client{Timeout: 5 * time.Second},
1,
)
e := testEngine(t, 1)
event := seedEvent(t, db, `{"db":"target"}`)
d := seedDatabaseTargetDelivery(t, db, event, "")
dlv := seedDelivery(
t, db, event.ID, uuid.New().String(),
database.DeliveryStatusPending,
)
d := &database.Delivery{
EventID: event.ID,
TargetID: dlv.TargetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: database.Target{
Name: "test-db",
Type: database.TargetTypeDatabase,
},
}
d.ID = dlv.ID
e.ExportDeliverDatabase(db, d)
var updated database.Delivery
require.NoError(t, db.First(
&updated, "id = ?", d.ID,
&updated, "id = ?", dlv.ID,
).Error)
assert.Equal(t,
@@ -378,7 +379,7 @@ func TestDeliverDatabase_ImmediateSuccess(
var result database.DeliveryResult
require.NoError(t, db.Where(
"delivery_id = ?", d.ID,
"delivery_id = ?", dlv.ID,
).First(&result).Error)
assert.True(t, result.Success)
@@ -1116,10 +1117,10 @@ func TestDoHTTPRequest_ForwardsHeaders(t *testing.T) {
}
event := &database.Event{
Method: http.MethodPost,
Method: "POST",
Headers: `{"X-Custom":["value1"],"Content-Type":["application/json"]}`,
Body: `{"test":true}`,
ContentType: testContentType,
ContentType: "application/json",
}
statusCode, _, _, err := e.ExportDoHTTPRequest(
@@ -1141,7 +1142,7 @@ func TestDoHTTPRequest_ForwardsHeaders(t *testing.T) {
)
assert.Equal(t,
testContentType,
"application/json",
receivedHeaders.Get("Content-Type"),
)
@@ -1157,19 +1158,7 @@ func TestProcessDelivery_RoutesToCorrectHandler(
t.Parallel()
db := testWebhookDB(t)
// The database target archives for real now, so the engine
// needs a webhook DB manager to locate the data directory.
e := delivery.NewTestEngineWithDB(
nil,
database.NewTestWebhookDBManager(t.TempDir()),
slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
&http.Client{Timeout: 5 * time.Second},
1,
)
e := testEngine(t, 1)
tests := []struct {
name string
@@ -1300,8 +1289,8 @@ func TestFormatSlackMessage_JSONBody(t *testing.T) {
t.Parallel()
event := &database.Event{
Method: http.MethodPost,
ContentType: testContentType,
Method: "POST",
ContentType: "application/json",
Body: `{"action":"push",` +
`"repo":"test/repo",` +
`"ref":"refs/heads/main"}`,
@@ -1326,7 +1315,7 @@ func TestFormatSlackMessage_NonJSONBody(t *testing.T) {
t.Parallel()
event := &database.Event{
Method: http.MethodPost,
Method: "POST",
ContentType: "text/plain",
Body: "hello world plain text",
}
@@ -1349,8 +1338,8 @@ func TestFormatSlackMessage_EmptyBody(t *testing.T) {
t.Parallel()
event := &database.Event{
Method: http.MethodPost,
ContentType: testContentType,
Method: "POST",
ContentType: "application/json",
Body: "",
}
event.CreatedAt = time.Date(
@@ -1378,8 +1367,8 @@ func TestFormatSlackMessage_LargeJSONTruncated(
require.NoError(t, err)
event := &database.Event{
Method: http.MethodPost,
ContentType: testContentType,
Method: "POST",
ContentType: "application/json",
Body: string(largeJSON),
}
event.CreatedAt = time.Date(
@@ -1708,7 +1697,7 @@ func assertLogLineComplete(
"log line must contain the webhook id",
)
assert.Contains(t, out, testContentType,
assert.Contains(t, out, "application/json",
"log line must contain the content type",
)
}

264
internal/delivery/export_test.go
View File

@@ -7,17 +7,10 @@ import (
"net/http"
"time"
"go.uber.org/fx"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// ErrExportArchiveWriterEvicted exposes the sentinel returned by
// an evicted archive writer. It carries the Err prefix rather
// than this file's usual Export one because it is a sentinel
// error.
var ErrExportArchiveWriterEvicted = errArchiveWriterEvicted
// Exported constants for test access.
const (
ExportDeliveryChannelSize = deliveryChannelSize
@@ -195,24 +188,9 @@ func (e *Engine) ExportRecoverInFlight(
e.recoverInFlight(ctx)
}
// ExportSweepWebhookRetries exposes sweepWebhookRetries.
func (e *Engine) ExportSweepWebhookRetries(
ctx context.Context, webhookID string,
) {
e.sweepWebhookRetries(ctx, webhookID)
}
// ExportStart exposes start for testing.
func (e *Engine) ExportStart() {
e.start()
}
// ExportRegisterHooks registers the engine's real fx lifecycle
// hooks on a lifecycle supplied by a test, so a test can drive
// the exact OnStart/OnStop functions the application runs and
// hand OnStart the kind of context fx actually supplies.
func (e *Engine) ExportRegisterHooks(lc fx.Lifecycle) {
e.registerHooks(lc)
func (e *Engine) ExportStart(ctx context.Context) {
e.start(ctx)
}
// ExportStop exposes stop for testing.
@@ -295,241 +273,3 @@ func NewTestCircuitBreaker(
cooldown: cooldown,
}
}
// ExportArchivedEvent aliases the archive row type so black-box
// tests can construct and read archive rows.
type ExportArchivedEvent = archivedEvent
// ExportArchiveWriter wraps an archiveWriter so black-box tests
// can exercise the per-webhook archive file mechanics.
type ExportArchiveWriter struct {
w *archiveWriter
}
// NewExportArchiveWriter builds an archive writer for tests,
// optionally overriding the reopen debounce (a non-positive
// debounce keeps the production default).
func NewExportArchiveWriter(
path string, log *slog.Logger, debounce time.Duration,
) *ExportArchiveWriter {
w := newArchiveWriter(path, log)
if debounce > 0 {
w.debounce = debounce
}
return &ExportArchiveWriter{w: w}
}
// Write archives a row through the writer.
func (e *ExportArchiveWriter) Write(
row ExportArchivedEvent, expiry time.Duration,
) error {
return e.w.write(row, expiry)
}
// Open opens the archive file, pruning when expiry is positive.
func (e *ExportArchiveWriter) Open(expiry time.Duration) error {
return e.w.open(expiry)
}
// Reopen closes and reopens the archive file.
func (e *ExportArchiveWriter) Reopen(
expiry time.Duration,
) error {
return e.w.reopen(expiry)
}
// Reopens reports how many times the file has been opened.
func (e *ExportArchiveWriter) Reopens() int {
return e.w.reopens
}
// DB returns the writer's current open handle for row
// inspection in tests.
func (e *ExportArchiveWriter) DB() *gorm.DB {
return e.w.db
}
// Path returns the archive file the writer owns.
func (e *ExportArchiveWriter) Path() string {
return e.w.path
}
// OpenExisting opens the archive without permitting creation,
// the way the idle sweep does.
func (e *ExportArchiveWriter) OpenExisting(
expiry time.Duration,
) error {
return e.w.openMode(archiveModeExisting, expiry)
}
// SweepExpired runs an idle sweep of the archive.
func (e *ExportArchiveWriter) SweepExpired(
expiry time.Duration,
) error {
return e.w.sweepExpired(expiry)
}
// Evict marks the writer evicted and closes its handle, exactly
// as leaving the registry does.
func (e *ExportArchiveWriter) Evict() {
e.w.evict()
}
// HandleOpen reports whether the writer currently holds an open
// archive handle.
func (e *ExportArchiveWriter) HandleOpen() bool {
e.w.mu.Lock()
defer e.w.mu.Unlock()
return e.w.db != nil
}
// Same reports whether both wrappers refer to the very same
// underlying archive writer, so a test can prove a registry entry
// is the writer it was handed rather than a replacement.
func (e *ExportArchiveWriter) Same(
other *ExportArchiveWriter,
) bool {
return other != nil && e.w == other.w
}
// ExportArchiveWriterFor returns the archive writer the registry
// currently caches for a webhook, or nil when none is cached. It
// never creates one, so a test can hold a reference to the very
// writer an eviction is about to detach.
func (e *Engine) ExportArchiveWriterFor(
webhookID string,
) *ExportArchiveWriter {
e.dbTarget.mu.Lock()
defer e.dbTarget.mu.Unlock()
w, ok := e.dbTarget.writers[webhookID]
if !ok {
return nil
}
return &ExportArchiveWriter{w: w}
}
// ExportHasArchiveWriter reports whether the database target
// currently caches an archive writer for a webhook.
func (e *Engine) ExportHasArchiveWriter(
webhookID string,
) bool {
e.dbTarget.mu.Lock()
defer e.dbTarget.mu.Unlock()
_, ok := e.dbTarget.writers[webhookID]
return ok
}
// ExportArchiveHandleOpen reports whether the cached archive
// writer for a webhook holds an open database handle. It
// returns false when no writer is cached.
func (e *Engine) ExportArchiveHandleOpen(
webhookID string,
) bool {
e.dbTarget.mu.Lock()
w, ok := e.dbTarget.writers[webhookID]
e.dbTarget.mu.Unlock()
if !ok {
return false
}
w.mu.Lock()
defer w.mu.Unlock()
return w.db != nil
}
// ExportEnsureArchiveWriter creates (if needed) and returns the
// archive file path of the cached writer for a webhook, so a
// test can prime the registry the way a delivery would.
func (e *Engine) ExportEnsureArchiveWriter(
webhookID string,
) (string, error) {
w, err := e.dbTarget.writerFor(webhookID)
if err != nil {
return "", err
}
return w.path, nil
}
// ExportSweepWriterFor takes a webhook's registry writer exactly
// as the idle sweep does, reporting whether the sweep had to
// create the entry. It lets a test drive the registry through the
// sweep's own entry point instead of choreographing goroutines.
func (e *Engine) ExportSweepWriterFor(
webhookID string,
) (*ExportArchiveWriter, bool, error) {
w, created, err := e.dbTarget.sweepWriterFor(webhookID)
if err != nil {
return nil, false, err
}
return &ExportArchiveWriter{w: w}, created, nil
}
// ExportReleaseSweepWriter releases a sweep-created registry entry
// exactly as a finished sweep does.
func (e *Engine) ExportReleaseSweepWriter(
webhookID string, w *ExportArchiveWriter,
) {
e.dbTarget.releaseSweepWriter(webhookID, w.w)
}
// NewTestArchiveSweeper builds an ArchiveSweeper backed by the
// given main database and engine, without the fx lifecycle.
// Intended for tests.
func NewTestArchiveSweeper(
db *database.Database,
eng *Engine,
log *slog.Logger,
) *ArchiveSweeper {
return &ArchiveSweeper{
db: db,
eng: eng,
log: log,
interval: time.Hour,
}
}
// ExportSweep runs a single archive sweep synchronously for
// tests.
func (s *ArchiveSweeper) ExportSweep(ctx context.Context) {
s.sweep(ctx)
}
// ExportStart starts the sweeper's background loop for tests.
func (s *ArchiveSweeper) ExportStart() {
s.start()
}
// ExportRegisterHooks registers the sweeper's real fx lifecycle
// hooks on a lifecycle supplied by a test, so a test can drive
// the exact OnStart/OnStop functions the application runs and
// hand OnStart the kind of context fx actually supplies.
func (s *ArchiveSweeper) ExportRegisterHooks(lc fx.Lifecycle) {
s.registerHooks(lc)
}
// ExportStop stops the sweeper's background loop for tests.
func (s *ArchiveSweeper) ExportStop() {
s.stop()
}
// ExportSetInterval overrides the sweep interval for tests.
func (s *ArchiveSweeper) ExportSetInterval(d time.Duration) {
s.interval = d
}
// ExportParseArchiveExpiry exposes parseArchiveExpiry.
func ExportParseArchiveExpiry(
configJSON string,
) (time.Duration, error) {
return parseArchiveExpiry(configJSON)
}

5
internal/delivery/target.go
View File

@@ -90,15 +90,12 @@ func (e *Engine) initTargets(client *http.Client) {
client: client,
}
dbT := &databaseTarget{eng: e}
e.httpTarget = httpT
e.dbTarget = dbT
e.targets = map[database.TargetType]Target{
database.TargetTypeHTTP: httpT,
database.TargetTypeSlack: slackT,
database.TargetTypeDatabase: dbT,
database.TargetTypeDatabase: &databaseTarget{eng: e},
database.TargetTypeLog: &logTarget{eng: e},
}
}

224
internal/delivery/target_config_view.go
View File

@@ -1,224 +0,0 @@
package delivery
import (
"encoding/json"
"fmt"
"net/url"
"strconv"
"sneak.berlin/go/webhooker/internal/database"
)
// configUnavailable is what a target's configuration renders
// as when it is absent, of an unknown type, or does not
// parse. The stored blob is never shown as a fallback: it can
// hold a credential (a Slack incoming webhook URL is a bearer
// token) and a UI that prints it leaks that credential into
// browser history, screenshots and screen shares.
const configUnavailable = "(unavailable)"
// urlPathElision stands in for a URL's elided path.
const urlPathElision = "/..."
// ConfigField is one labelled, display-safe value derived
// from a target's stored configuration.
type ConfigField struct {
Label string
Value string
}
// TargetView is the display-safe projection of a target for
// the UI. It deliberately has no raw configuration field, so
// no template — present or future — can render the stored
// blob.
type TargetView struct {
ID string
Name string
Type database.TargetType
Active bool
Config []ConfigField
}
// NewTargetViews projects targets for rendering, replacing
// each stored configuration blob with named, display-safe
// fields.
func NewTargetViews(
targets []database.Target,
) []TargetView {
views := make([]TargetView, 0, len(targets))
for i := range targets {
t := &targets[i]
views = append(views, TargetView{
ID: t.ID,
Name: t.Name,
Type: t.Type,
Active: t.Active,
Config: targetConfigFields(t),
})
}
return views
}
// targetConfigFields returns the display-safe fields for a
// target's configuration. Anything it cannot parse becomes
// the neutral placeholder.
func targetConfigFields(
t *database.Target,
) []ConfigField {
switch t.Type {
case database.TargetTypeSlack:
return slackConfigFields(t.Config)
case database.TargetTypeHTTP:
return httpConfigFields(t)
case database.TargetTypeDatabase:
return databaseConfigFields(t.Config)
case database.TargetTypeLog:
// The log target takes no configuration.
return nil
default:
return unavailableConfigFields()
}
}
// unavailableConfigFields is the neutral placeholder shown
// for a configuration that could not be presented.
func unavailableConfigFields() []ConfigField {
return []ConfigField{{
Label: "Configuration",
Value: configUnavailable,
}}
}
// slackConfigFields describes a Slack target. Only the masked
// webhook URL is shown; the full URL is the credential.
func slackConfigFields(configJSON string) []ConfigField {
cfg, err := parseSlackConfig(configJSON)
if err != nil {
return unavailableConfigFields()
}
return []ConfigField{{
Label: "Webhook URL",
Value: cfg.MaskedWebhookURL(),
}}
}
// httpConfigFields describes an HTTP target: its destination
// and its retry settings. Header values are not shown — they
// routinely carry authorization tokens — only how many are
// configured.
func httpConfigFields(t *database.Target) []ConfigField {
cfg, err := parseHTTPConfig(t.Config)
if err != nil {
return unavailableConfigFields()
}
fields := []ConfigField{{
Label: "Destination URL",
Value: cfg.URL,
}}
if cfg.Timeout > 0 {
fields = append(fields, ConfigField{
Label: "Timeout",
Value: strconv.Itoa(cfg.Timeout) + "s",
})
}
if len(cfg.Headers) > 0 {
fields = append(fields, ConfigField{
Label: "Headers",
Value: fmt.Sprintf(
"%d configured", len(cfg.Headers),
),
})
}
return append(fields, retryFields(t)...)
}
// retryFields describes a target's retry settings, which live
// on the target row rather than in its configuration blob.
func retryFields(t *database.Target) []ConfigField {
retries := strconv.Itoa(t.MaxRetries)
if t.MaxRetries == 0 {
retries += " (fire-and-forget)"
}
fields := []ConfigField{{
Label: "Max Retries",
Value: retries,
}}
if t.MaxQueueSize > 0 {
fields = append(fields, ConfigField{
Label: "Max Queue Size",
Value: strconv.Itoa(t.MaxQueueSize),
})
}
return fields
}
// databaseConfigFields describes an archive target. Its
// configuration is optional, and an absent or empty expiry
// means the archive is kept forever. An expiry that is set
// but not a valid duration is reported as unavailable rather
// than echoed back.
func databaseConfigFields(configJSON string) []ConfigField {
expiry := archiveExpiryNever
if configJSON != "" {
var cfg databaseTargetConfig
err := json.Unmarshal([]byte(configJSON), &cfg)
if err != nil {
return unavailableConfigFields()
}
if cfg.Expiry != "" {
if ValidateArchiveExpiry(cfg.Expiry) != nil {
return unavailableConfigFields()
}
expiry = cfg.Expiry
}
}
return []ConfigField{{
Label: "Archive Expiry",
Value: expiry,
}}
}
// MaskedWebhookURL returns the Slack webhook URL reduced to
// its scheme and host, with the path, query and any userinfo
// elided. The path segments are the credential, so none of
// them is shown: the field accepts an arbitrary URL, so no
// segment can be assumed non-secret. A URL that does not
// parse into a scheme and host yields the neutral
// placeholder, never the raw string.
func (c *SlackTargetConfig) MaskedWebhookURL() string {
return maskURL(c.WebhookURL)
}
// maskURL renders a URL as scheme plus host with everything
// that can carry a secret removed.
func maskURL(raw string) string {
parsed, err := url.Parse(raw)
if err != nil || parsed.Scheme == "" ||
parsed.Host == "" {
return configUnavailable
}
masked := parsed.Scheme + "://" + parsed.Host
if parsed.Path != "" && parsed.Path != "/" {
masked += urlPathElision
}
return masked
}

299
internal/delivery/target_config_view_test.go
View File

@@ -1,299 +0,0 @@
package delivery_test
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
const (
// slackSecretPath is the credential-bearing part of a
// Slack incoming webhook URL: everything after the host.
slackSecretPath = "/services/T00000000/B00000000/" +
"XXXXXXXXXXXXXXXXXXXXXXXX"
slackWebhookURL = "https://hooks.slack.com" +
slackSecretPath
viewExampleOrigin = "https://example.com"
viewExampleHook = viewExampleOrigin + "/hook"
viewUnavailable = "(unavailable)"
viewExpiryNever = "never"
)
func TestMaskedWebhookURL(t *testing.T) {
t.Parallel()
tests := map[string]struct {
url string
want string
}{
"slack webhook": {
url: slackWebhookURL,
want: "https://hooks.slack.com/...",
},
"query string dropped": {
url: viewExampleOrigin + "/a?token=secret",
want: viewExampleOrigin + "/...",
},
// Fabricated userinfo in a test URL, not a real
// credential.
//nolint:gosec // G101
"userinfo dropped": {
url: "https://user:pw@example.com/a/b",
want: viewExampleOrigin + "/...",
},
"no path": {
url: viewExampleOrigin,
want: viewExampleOrigin,
},
"root path": {
url: viewExampleOrigin + "/",
want: viewExampleOrigin,
},
"not a url": {
url: "definitely not a url",
want: viewUnavailable,
},
"empty": {
url: "",
want: viewUnavailable,
},
}
for name, tc := range tests {
t.Run(name, func(t *testing.T) {
t.Parallel()
cfg := &delivery.SlackTargetConfig{
WebhookURL: tc.url,
}
assert.Equal(
t, tc.want, cfg.MaskedWebhookURL(),
)
})
}
}
// TestMaskedWebhookURL_NeverLeaksPath is the direct
// expression of the rule: whatever the input, the masked
// value never contains a path segment of it.
func TestMaskedWebhookURL_NeverLeaksPath(t *testing.T) {
t.Parallel()
cfg := &delivery.SlackTargetConfig{
WebhookURL: slackWebhookURL,
}
masked := cfg.MaskedWebhookURL()
assert.NotContains(t, masked, "T00000000")
assert.NotContains(t, masked, "B00000000")
assert.NotContains(
t, masked, "XXXXXXXXXXXXXXXXXXXXXXXX",
)
assert.NotContains(t, masked, slackSecretPath)
}
// fieldMap turns a view's config fields into a lookup so
// assertions read by label.
func fieldMap(fields []delivery.ConfigField) map[string]string {
out := make(map[string]string, len(fields))
for _, f := range fields {
out[f.Label] = f.Value
}
return out
}
// viewFor projects a single target and returns its view.
func viewFor(
t *testing.T,
target database.Target,
) delivery.TargetView {
t.Helper()
views := delivery.NewTargetViews(
[]database.Target{target},
)
require.Len(t, views, 1)
return views[0]
}
func TestNewTargetViews_Slack(t *testing.T) {
t.Parallel()
view := viewFor(t, database.Target{
Name: "slack-target",
Type: database.TargetTypeSlack,
Active: true,
Config: `{"webhookUrl":"` +
slackWebhookURL + `"}`,
})
assert.Equal(t, "slack-target", view.Name)
assert.Equal(
t,
map[string]string{
"Webhook URL": "https://hooks.slack.com/...",
},
fieldMap(view.Config),
)
}
func TestNewTargetViews_HTTP(t *testing.T) {
t.Parallel()
view := viewFor(t, database.Target{
Type: database.TargetTypeHTTP,
Config: `{"url":"` + viewExampleHook + `",` +
`"timeout":30,` +
`"headers":{"Authorization":"Bearer sekrit"}}`,
MaxRetries: 5,
MaxQueueSize: 100,
})
fields := fieldMap(view.Config)
assert.Equal(
t,
map[string]string{
"Destination URL": viewExampleHook,
"Timeout": "30s",
"Headers": "1 configured",
"Max Retries": "5",
"Max Queue Size": "100",
},
fields,
)
// Header values can be credentials and are never shown.
for _, v := range fields {
assert.NotContains(t, v, "sekrit")
}
}
func TestNewTargetViews_HTTPFireAndForget(t *testing.T) {
t.Parallel()
view := viewFor(t, database.Target{
Type: database.TargetTypeHTTP,
Config: `{"url":"` + viewExampleHook + `"}`,
})
assert.Equal(
t,
map[string]string{
"Destination URL": viewExampleHook,
"Max Retries": "0 (fire-and-forget)",
},
fieldMap(view.Config),
)
}
func TestNewTargetViews_Database(t *testing.T) {
t.Parallel()
tests := map[string]struct {
config string
want string
}{
"empty config": {config: "", want: viewExpiryNever},
"empty expiry": {config: `{}`, want: viewExpiryNever},
"explicit": {
config: `{"expiry":"720h"}`,
want: "720h",
},
"never literal": {
config: `{"expiry":"` + viewExpiryNever + `"}`,
want: viewExpiryNever,
},
}
for name, tc := range tests {
t.Run(name, func(t *testing.T) {
t.Parallel()
view := viewFor(t, database.Target{
Type: database.TargetTypeDatabase,
Config: tc.config,
})
assert.Equal(
t,
map[string]string{"Archive Expiry": tc.want},
fieldMap(view.Config),
)
})
}
}
func TestNewTargetViews_Log(t *testing.T) {
t.Parallel()
view := viewFor(t, database.Target{
Type: database.TargetTypeLog,
Config: "",
})
assert.Empty(t, view.Config)
}
// TestNewTargetViews_Unpresentable proves that no config the
// view cannot present falls back to the stored blob.
func TestNewTargetViews_Unpresentable(t *testing.T) {
t.Parallel()
const blob = `{"webhookUrl":"https://hooks.slack.com` +
slackSecretPath + `"`
tests := map[string]database.Target{
"unknown target type": {
Type: database.TargetType("carrier-pigeon"),
Config: blob,
},
"unparseable json": {
Type: database.TargetTypeSlack,
Config: blob,
},
"empty slack config": {
Type: database.TargetTypeSlack,
},
"slack config without url": {
Type: database.TargetTypeSlack,
Config: `{}`,
},
"unparseable http json": {
Type: database.TargetTypeHTTP,
Config: `{"url":`,
},
"unparseable archive json": {
Type: database.TargetTypeDatabase,
Config: `{"expiry":`,
},
"invalid archive expiry": {
Type: database.TargetTypeDatabase,
Config: `{"expiry":"a fortnight"}`,
},
}
for name, target := range tests {
t.Run(name, func(t *testing.T) {
t.Parallel()
view := viewFor(t, target)
assert.Equal(
t,
map[string]string{
"Configuration": viewUnavailable,
},
fieldMap(view.Config),
)
})
}
}

273
internal/delivery/target_database.go
View File

@@ -2,39 +2,21 @@ package delivery
import (
"context"
"fmt"
"path/filepath"
"sync"
"time"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// databaseTarget is a no-retry target that archives the
// full inbound event into a per-webhook archive SQLite file,
// separate from the per-webhook event database. The event is
// already persisted in the per-webhook event DB by the time
// delivery runs; the database target additionally writes a
// durable long-term copy into archive-{webhookID}.db and then
// records a single attempt whose outcome reflects whether the
// archive write succeeded. See archiveWriter for the
// close/reopen, auto-recreate, and expiry semantics.
// databaseTarget is a fire-and-forget target: the event is
// already persisted in the per-webhook database by the time
// delivery runs, so the target records a single successful
// attempt. (Durable archiving to a separate store is tracked
// as its own work.)
type databaseTarget struct {
eng *Engine
mu sync.Mutex
writers map[string]*archiveWriter
}
// Deliver implements Target. It archives the event, then
// records one successful attempt and marks the delivery
// delivered. An archiving error fails the delivery: the
// attempt is recorded as failed with the error and the
// delivery is marked failed, so a target that could not do
// its one job (archiving) never reports success. The target
// does not retry; the event remains durably stored in the
// per-webhook event database.
// Deliver implements Target.
func (t *databaseTarget) Deliver(
_ context.Context,
webhookDB *gorm.DB,
@@ -42,27 +24,6 @@ func (t *databaseTarget) Deliver(
_ *Task,
_ Scheduler,
) {
err := t.archive(d)
if err != nil {
t.eng.log.Error(
"failed to archive event to database target",
"delivery_id", d.ID,
"event_id", d.EventID,
"error", err,
)
t.eng.recordResult(
webhookDB, d, 1, false, 0, "",
err.Error(), 0,
)
t.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusFailed,
)
return
}
t.eng.recordResult(
webhookDB, d, 1, true, 0, "", "", 0,
)
@@ -71,225 +32,3 @@ func (t *databaseTarget) Deliver(
webhookDB, d, database.DeliveryStatusDelivered,
)
}
// archive writes the full event as a row into the webhook's
// archive database, honouring the optional per-target expiry
// parsed from the target config JSON.
func (t *databaseTarget) archive(d *database.Delivery) error {
webhookID := d.Event.WebhookID
if webhookID == "" {
return errArchiveMissingWebhookID
}
expiry, err := parseArchiveExpiry(d.Target.Config)
if err != nil {
return err
}
w, err := t.writerFor(webhookID)
if err != nil {
return err
}
row := archivedEvent{
EventID: d.Event.ID,
WebhookID: webhookID,
EntrypointID: d.Event.EntrypointID,
Method: d.Event.Method,
Headers: d.Event.Headers,
Body: d.Event.Body,
ContentType: d.Event.ContentType,
}
return w.write(row, expiry)
}
// writerFor returns the archiveWriter for a webhook, creating
// and caching it on first use. Each webhook has one writer so
// its close/reopen debounce state is shared across concurrent
// deliveries. The archive file lives beside the per-webhook
// event database in the data directory.
func (t *databaseTarget) writerFor(
webhookID string,
) (*archiveWriter, error) {
path, err := t.archivePath(webhookID)
if err != nil {
return nil, err
}
t.mu.Lock()
defer t.mu.Unlock()
if t.writers == nil {
t.writers = make(map[string]*archiveWriter)
}
w, ok := t.writers[webhookID]
if !ok {
w = newArchiveWriter(path, t.eng.log)
t.writers[webhookID] = w
}
// A delivery claims the entry: even if the idle sweep created
// it moments ago, it now belongs to the registry proper and
// the sweep must leave it in place when it finishes.
w.sweepOwned = false
return w, nil
}
// sweepWriterFor returns the archive writer the idle sweep should
// prune a webhook through, together with whether the sweep itself
// created the registry entry.
//
// The sweep must route its prune through the registered writer so
// the writer's mutex orders it against concurrent writes, but it
// must never leave a registry entry behind: a sweep that ran
// concurrently with the webhook's deletion would otherwise
// re-create an entry that nothing will ever evict again, which is
// exactly the leak eviction exists to prevent. An entry the sweep
// creates is therefore marked sweep-owned and handed back to
// releaseSweepWriter when the sweep is done.
func (t *databaseTarget) sweepWriterFor(
webhookID string,
) (*archiveWriter, bool, error) {
path, err := t.archivePath(webhookID)
if err != nil {
return nil, false, err
}
t.mu.Lock()
defer t.mu.Unlock()
if t.writers == nil {
t.writers = make(map[string]*archiveWriter)
}
w, ok := t.writers[webhookID]
if ok {
return w, false, nil
}
w = newArchiveWriter(path, t.eng.log)
w.sweepOwned = true
t.writers[webhookID] = w
return w, true, nil
}
// releaseSweepWriter drops a registry entry that the idle sweep
// created, so a sweep leaves the registry exactly as it found it.
//
// The entry is removed only if it is still the very writer the
// sweep installed and no delivery has claimed it in the meantime
// (writerFor clears sweepOwned when it hands a writer to the
// write path). Both conditions are evaluated under the registry
// lock, so an eviction that raced the sweep — which removes the
// entry outright — simply finds nothing left to do here, and a
// delivery that adopted the writer keeps a registered, evictable
// one.
func (t *databaseTarget) releaseSweepWriter(
webhookID string, w *archiveWriter,
) {
t.mu.Lock()
defer t.mu.Unlock()
cur, ok := t.writers[webhookID]
if !ok || cur != w || !cur.sweepOwned {
return
}
delete(t.writers, webhookID)
}
// archivePath returns the archive file path for a webhook: it
// lives beside the per-webhook event database in the data
// directory. It does not touch the filesystem.
func (t *databaseTarget) archivePath(
webhookID string,
) (string, error) {
if t.eng.dbManager == nil {
return "", errArchiveNoDataDir
}
dir := filepath.Dir(t.eng.dbManager.DBPath(webhookID))
return filepath.Join(
dir, fmt.Sprintf("archive-%s.db", webhookID),
), nil
}
// evict drops a webhook's archive writer from the registry and
// closes its handle, so a deleted webhook does not leave a
// writer (and an open archive handle within its debounce
// window) alive for the process lifetime.
//
// The map entry is removed under the registry lock, which is
// then released before the handle is closed under the writer's
// own lock: that ordering keeps the registry available to other
// webhooks while an in-flight write on this one drains, and
// closing under the writer's lock means eviction can never race
// a write.
//
// Eviction is idempotent and silent for a webhook with no
// writer, which is the common case: a webhook with no database
// target never creates one. It never deletes the archive file.
func (t *databaseTarget) evict(webhookID string) {
t.mu.Lock()
w, ok := t.writers[webhookID]
if ok {
delete(t.writers, webhookID)
}
t.mu.Unlock()
if !ok {
return
}
w.evict()
t.eng.log.Info(
"evicted archive writer",
"webhook_id", webhookID,
"path", w.path,
)
}
// sweepWebhook prunes one webhook's archive of rows older than
// expiry, without requiring a write. It returns nil (nothing to
// do) when the archive file does not exist, so a sweep never
// creates an archive for a webhook that has a database target
// but has never received an event.
//
// It also never leaves a registry entry behind: an entry it had
// to create to reach the writer's mutex is released again once
// the prune is done, so a sweep racing a webhook deletion cannot
// resurrect the writer the eviction just dropped.
func (t *databaseTarget) sweepWebhook(
webhookID string, expiry time.Duration,
) error {
path, err := t.archivePath(webhookID)
if err != nil {
return err
}
// Check before taking a writer at all: a webhook whose
// archive has never been created gets no writer, no handle,
// and no file.
if !fileExists(path) {
return nil
}
w, created, err := t.sweepWriterFor(webhookID)
if err != nil {
return err
}
if created {
defer t.releaseSweepWriter(webhookID, w)
}
return w.sweepExpired(expiry)
}

431
internal/delivery/target_database_archive.go
View File

@@ -1,431 +0,0 @@
package delivery
import (
"database/sql"
"encoding/json"
"errors"
"fmt"
"log/slog"
"os"
"sync"
"time"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
)
// archiveExpiryNever is the expiry sentinel (and default) that
// disables pruning so archived rows are kept forever.
const archiveExpiryNever = "never"
// archiveReopenDebounce bounds how often an archive file is
// closed and reopened. After each write the handle is closed
// and reopened so an operator can move the file away for
// offline archiving, but never more than once per this window.
const archiveReopenDebounce = time.Second
const (
// archiveModeCreate is the SQLite URI mode used by the write
// path: open the archive file, creating it if missing, so a
// first write (or a write after the operator moved the file
// away) recreates it.
archiveModeCreate = "rwc"
// archiveModeExisting is the SQLite URI mode used by the idle
// sweep: open read-write but never create. A sweep must never
// conjure an empty archive file for a webhook that has a
// database target but has never received an event.
archiveModeExisting = "rw"
)
var (
// errArchiveMissingWebhookID is returned when an event to
// archive has no webhook id to key its archive file on.
errArchiveMissingWebhookID = errors.New(
"cannot archive event without a webhook id",
)
// errArchiveNoDataDir is returned when the database target
// has no webhook database manager and so cannot locate the
// data directory for archive files.
errArchiveNoDataDir = errors.New(
"database target has no data directory",
)
// errArchiveExpiryNotPositive is returned when a
// user-supplied archive expiry parses as a duration but is
// zero or negative; "never" is the way to disable pruning.
errArchiveExpiryNotPositive = errors.New(
"expiry must be a positive duration or \"never\"",
)
// errArchiveWriterEvicted is returned when a writer that has
// been evicted (its webhook was deleted, or its last database
// target was removed) is used again. An evicted writer is no
// longer in the registry, so reopening its file would leak a
// handle nothing owns.
errArchiveWriterEvicted = errors.New(
"archive writer has been evicted",
)
)
// databaseTargetConfig is the optional per-target JSON config
// for a database (archive) target.
type databaseTargetConfig struct {
// Expiry is a Go duration (e.g. "720h") after which
// archived rows are pruned, or "never" (the default) to
// keep them forever.
Expiry string `json:"expiry"`
}
// archivedEvent is one fully captured webhook event stored in a
// per-webhook archive database for long-term retention. It is a
// self-contained copy — independent of the per-webhook event
// database, which may prune events under its own retention.
type archivedEvent struct {
ID uint `gorm:"primaryKey;autoIncrement"`
EventID string `gorm:"index"`
WebhookID string
EntrypointID string
Method string
Headers string
Body string
ContentType string
// ArchivedAt is when the row was archived and is the age
// basis for expiry pruning.
ArchivedAt time.Time `gorm:"index"`
}
// parseArchiveExpiry reads the optional expiry from a database
// target's config JSON. An empty config, an empty expiry, or
// the literal "never" all mean keep forever, returned as a zero
// duration. Any other value must parse as a positive Go
// duration; a set-but-invalid value (unparseable, zero, or
// negative) is an error rather than a silent default, matching
// ValidateArchiveExpiry at target creation.
func parseArchiveExpiry(
configJSON string,
) (time.Duration, error) {
if configJSON == "" {
return 0, nil
}
var cfg databaseTargetConfig
err := json.Unmarshal([]byte(configJSON), &cfg)
if err != nil {
return 0, fmt.Errorf(
"parsing database target config: %w", err,
)
}
if cfg.Expiry == "" || cfg.Expiry == archiveExpiryNever {
return 0, nil
}
dur, err := time.ParseDuration(cfg.Expiry)
if err != nil {
return 0, fmt.Errorf(
"parsing archive expiry %q: %w", cfg.Expiry, err,
)
}
if dur <= 0 {
return 0, fmt.Errorf(
"%w: %q", errArchiveExpiryNotPositive, cfg.Expiry,
)
}
return dur, nil
}
// ValidateArchiveExpiry checks a user-supplied archive expiry
// for a database target at configuration time. Valid values are
// empty, "never" (both meaning keep forever), or a positive Go
// duration such as "720h". Anything else is an error, so a bad
// expiry is rejected when the target is created rather than
// failing every subsequent delivery.
func ValidateArchiveExpiry(expiry string) error {
if expiry == "" || expiry == archiveExpiryNever {
return nil
}
dur, err := time.ParseDuration(expiry)
if err != nil {
return fmt.Errorf(
"expiry must be %q or a Go duration "+
"such as \"720h\": %w",
archiveExpiryNever, err,
)
}
if dur <= 0 {
return fmt.Errorf(
"%w: %q", errArchiveExpiryNotPositive, expiry,
)
}
return nil
}
// archiveWriter owns one per-webhook archive SQLite file. It
// serialises writes, and after each write closes and reopens
// the file (debounced to at most once per debounce window) so
// an operator can move the file away for offline archiving. The
// next write recreates a moved or removed file, because the
// file is opened create-if-missing and its schema is migrated
// on every open.
type archiveWriter struct {
mu sync.Mutex
path string
log *slog.Logger
debounce time.Duration
db *gorm.DB
lastReopen time.Time
reopens int
// evicted marks a writer that has been removed from the
// per-webhook registry. Its handle is closed and it must
// never open the file again: nothing holds it any more, so a
// reopen would leak the handle for the process lifetime.
evicted bool
// sweepOwned marks a registry entry that the idle sweep
// created because no writer was cached for the webhook. The
// sweep removes such an entry again when it is done, so a
// sweep can never leave — or resurrect — a registry entry
// for a webhook that has been deleted. A delivery that adopts
// the writer clears the flag, handing the entry to the
// registry proper.
//
// Unlike every other field here it is guarded by
// databaseTarget.mu, not by this writer's mu: it describes the
// registry entry rather than the file.
sweepOwned bool
}
// newArchiveWriter builds an archiveWriter for a file path with
// the default reopen debounce.
func newArchiveWriter(
path string, log *slog.Logger,
) *archiveWriter {
return &archiveWriter{
path: path,
log: log,
debounce: archiveReopenDebounce,
}
}
// write appends the event as a row, then applies the debounced
// close/reopen. It recreates the archive file if it was moved
// or removed since the last open. A positive expiry prunes rows
// older than it on each (re)open.
func (w *archiveWriter) write(
row archivedEvent, expiry time.Duration,
) error {
w.mu.Lock()
defer w.mu.Unlock()
if w.evicted {
return fmt.Errorf(
"%w: %s", errArchiveWriterEvicted, w.path,
)
}
if w.db == nil || !fileExists(w.path) {
err := w.reopen(expiry)
if err != nil {
return err
}
}
row.ArchivedAt = time.Now()
err := w.db.Create(&row).Error
if err != nil {
return fmt.Errorf(
"archiving event to %s: %w", w.path, err,
)
}
if time.Since(w.lastReopen) >= w.debounce {
return w.reopen(expiry)
}
return nil
}
// open opens (creating if missing) the archive file, migrates
// its schema, records the reopen time, and prunes expired rows
// when expiry is positive.
func (w *archiveWriter) open(expiry time.Duration) error {
return w.openMode(archiveModeCreate, expiry)
}
// openMode opens the archive file with the given SQLite URI
// mode, migrates its schema, records the reopen time, and
// prunes expired rows when expiry is positive. The write path
// passes archiveModeCreate so a missing file is recreated; the
// idle sweep passes archiveModeExisting so a missing file is an
// error rather than a newly conjured empty archive.
func (w *archiveWriter) openMode(
mode string, expiry time.Duration,
) error {
dbURL := fmt.Sprintf("file:%s?mode=%s", w.path, mode)
sqlDB, err := sql.Open("sqlite", dbURL)
if err != nil {
return fmt.Errorf(
"opening archive database %s: %w", w.path, err,
)
}
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
if err != nil {
_ = sqlDB.Close()
return fmt.Errorf(
"connecting to archive database %s: %w",
w.path, err,
)
}
err = gdb.AutoMigrate(&archivedEvent{})
if err != nil {
_ = sqlDB.Close()
return fmt.Errorf(
"migrating archive database %s: %w", w.path, err,
)
}
w.db = gdb
w.lastReopen = time.Now()
w.reopens++
if expiry > 0 {
w.prune(expiry)
}
return nil
}
// reopen closes any open handle and opens the file afresh. The
// fresh open recreates the file if it was moved away.
func (w *archiveWriter) reopen(expiry time.Duration) error {
w.close()
return w.open(expiry)
}
// close closes the underlying handle, if any.
func (w *archiveWriter) close() {
if w.db == nil {
return
}
sqlDB, err := w.db.DB()
if err == nil {
_ = sqlDB.Close()
}
w.db = nil
}
// sweepExpired prunes an archive that may have gone idle, with
// no write to trigger the usual on-reopen prune. It takes the
// writer's own mutex for the whole operation, so a sweep is
// ordered against concurrent writes rather than reaching around
// them to the file.
//
// It never creates the archive file: a missing file is skipped,
// and the reopen uses archiveModeExisting so SQLite itself
// refuses to create one if the file disappears between the
// check and the open.
//
// The archive is left CLOSED afterwards. An idle archive holding
// no handle is what keeps the operator's move-the-file-away
// workflow working; the next write reopens (and recreates) the
// file as it always has.
func (w *archiveWriter) sweepExpired(expiry time.Duration) error {
w.mu.Lock()
defer w.mu.Unlock()
if w.evicted {
return fmt.Errorf(
"%w: %s", errArchiveWriterEvicted, w.path,
)
}
if !fileExists(w.path) {
return nil
}
// Drop any live handle first so the prune runs against a
// freshly opened file, matching the write path's semantics.
w.close()
err := w.openMode(archiveModeExisting, expiry)
if err != nil {
return err
}
w.close()
return nil
}
// evict closes the writer's handle and marks it unusable. It is
// called when the writer leaves the registry, either because the
// webhook was deleted or because its last database target was
// removed. The archive FILE is deliberately left on disk: it is
// long-term storage an operator may still want.
func (w *archiveWriter) evict() {
w.mu.Lock()
defer w.mu.Unlock()
w.evicted = true
w.close()
}
// prune deletes archived rows older than expiry, measured from
// each row's archived time. It runs on every (re)open, so a
// steadily written archive is swept by its own write traffic. An
// archive that goes idle receives no further reopens, which is
// why ArchiveSweeper exists to drive sweepExpired on a timer.
// Failures are logged, not fatal: a prune error must not stop
// archiving.
func (w *archiveWriter) prune(expiry time.Duration) {
cutoff := time.Now().Add(-expiry)
res := w.db.Where("archived_at < ?", cutoff).
Delete(&archivedEvent{})
if res.Error != nil {
w.log.Error(
"failed to prune expired archive rows",
"path", w.path,
"error", res.Error,
)
return
}
if res.RowsAffected > 0 {
w.log.Info(
"pruned expired archive rows",
"path", w.path,
"rows_deleted", res.RowsAffected,
)
}
}
// fileExists reports whether a path currently exists.
func fileExists(path string) bool {
_, err := os.Stat(path)
return err == nil
}

363
internal/delivery/target_database_evict_test.go
View File

@@ -1,363 +0,0 @@
package delivery_test
import (
"errors"
"fmt"
"net/http"
"os"
"path/filepath"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
// evictTestEngine builds an engine backed by a temporary data
// directory and returns it along with that directory.
func evictTestEngine(t *testing.T) (*delivery.Engine, string) {
t.Helper()
dataDir := t.TempDir()
eng := delivery.NewTestEngineWithDB(
nil,
database.NewTestWebhookDBManager(dataDir),
archiveTestLogger(),
&http.Client{Timeout: 5 * time.Second},
1,
)
return eng, dataDir
}
// TestEvictWebhook_ClosesAndRemovesWriter proves that evicting
// a webhook drops its archive writer from the registry and
// closes the open archive handle, rather than leaving both
// alive for the process lifetime.
func TestEvictWebhook_ClosesAndRemovesWriter(t *testing.T) {
t.Parallel()
eng, dataDir := evictTestEngine(t)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":true}`)
d := seedDatabaseTargetDelivery(t, webhookDB, event, "")
eng.ExportDeliverDatabase(webhookDB, d)
webhookID := event.WebhookID
require.True(
t, eng.ExportHasArchiveWriter(webhookID),
"a delivery should have cached an archive writer",
)
require.True(
t, eng.ExportArchiveHandleOpen(webhookID),
"the writer should hold an open handle after a write",
)
eng.EvictWebhook(webhookID)
assert.False(
t, eng.ExportHasArchiveWriter(webhookID),
"eviction should remove the registry entry",
)
assert.False(
t, eng.ExportArchiveHandleOpen(webhookID),
"eviction should close the archive handle",
)
archivePath := filepath.Join(
dataDir, fmt.Sprintf("archive-%s.db", webhookID),
)
assert.FileExists(
t, archivePath,
"eviction must not delete the archive file",
)
}
// TestEvictWebhook_UnknownWebhookIsNoOp proves eviction is safe
// for the common case of a webhook that never had a database
// target, and that repeating it does not panic.
func TestEvictWebhook_UnknownWebhookIsNoOp(t *testing.T) {
t.Parallel()
eng, _ := evictTestEngine(t)
assert.NotPanics(t, func() {
eng.EvictWebhook("no-such-webhook")
eng.EvictWebhook("no-such-webhook")
})
assert.False(
t, eng.ExportHasArchiveWriter("no-such-webhook"),
"eviction must not create a writer",
)
}
// evictTestRow builds an archive row for the eviction tests.
func evictTestRow(eventID string) delivery.ExportArchivedEvent {
return delivery.ExportArchivedEvent{
EventID: eventID,
WebhookID: "wh-evict",
Method: http.MethodPost,
Body: `{"seeded":true}`,
}
}
// TestEvictedWriter_WriteDoesNotReopenFile is the direct test of
// the evicted guard on the write path. A writer that has left
// the registry is held by nobody, so a handle it opened could
// never be closed again: it must refuse the write outright
// rather than recreate the archive behind the registry's back.
//
// The archive file is removed before the eviction, so an
// unguarded write is unmistakable — it recreates the file.
func TestEvictedWriter_WriteDoesNotReopenFile(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-evicted.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.Write(evictTestRow("ev-1"), 0))
require.FileExists(t, path)
// The operator moves the archive away for offline retention,
// which the write path would ordinarily undo on the next
// write by recreating the file.
require.NoError(t, os.Remove(path))
w.Evict()
err := w.Write(evictTestRow("ev-2"), 0)
require.ErrorIs(
t, err, delivery.ErrExportArchiveWriterEvicted,
"an evicted writer must refuse writes",
)
assert.NoFileExists(
t, path,
"an evicted writer must not reopen (or recreate) the "+
"archive file",
)
assert.False(
t, w.HandleOpen(),
"an evicted writer must hold no handle",
)
}
// TestEvictedWriter_SweepDoesNotReopenFile is the same test for
// the sweep path: an idle sweep that reaches a writer already
// evicted underneath it must return the sentinel rather than
// reopen a file nothing owns.
func TestEvictedWriter_SweepDoesNotReopenFile(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-evicted.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.Write(evictTestRow("ev-1"), 0))
require.FileExists(t, path)
w.Evict()
err := w.SweepExpired(time.Hour)
require.ErrorIs(
t, err, delivery.ErrExportArchiveWriterEvicted,
"an evicted writer must refuse an idle sweep",
)
assert.False(
t, w.HandleOpen(),
"a refused sweep must not leave a handle open",
)
}
// racingWrites drives a pack of goroutines writing to one
// archive writer until each is refused, so an eviction on the
// test goroutine has to take the writer's mutex away from writes
// that are already contending for it.
type racingWrites struct {
wg sync.WaitGroup
mu sync.Mutex
sawEvicted bool
otherErr error
started chan struct{}
}
// racingWriteGoroutines is how many goroutines contend for the
// writer's mutex while the eviction lands.
const racingWriteGoroutines = 4
// startRacingWrites launches the writing goroutines. Each writes
// in a loop and stops at its first error, recording whether that
// error was the eviction sentinel. The deadline is a backstop
// against a hang, not a timing assumption: the first write after
// the eviction is refused.
func startRacingWrites(
w *delivery.ExportArchiveWriter,
) *racingWrites {
r := &racingWrites{
started: make(chan struct{}, racingWriteGoroutines),
}
deadline := time.Now().Add(10 * time.Second)
r.wg.Add(racingWriteGoroutines)
for i := range racingWriteGoroutines {
go func() {
defer r.wg.Done()
first := true
for time.Now().Before(deadline) {
err := w.Write(
evictTestRow(fmt.Sprintf("ev-%d", i)), 0,
)
if first {
r.started <- struct{}{}
first = false
}
if err == nil {
continue
}
r.record(err)
return
}
}()
}
return r
}
// record classifies the error that stopped one goroutine.
func (r *racingWrites) record(err error) {
r.mu.Lock()
defer r.mu.Unlock()
if errors.Is(err, delivery.ErrExportArchiveWriterEvicted) {
r.sawEvicted = true
return
}
r.otherErr = err
}
// awaitFirstWrite blocks until at least one write has run, so
// the eviction that follows is a genuine race.
func (r *racingWrites) awaitFirstWrite() {
<-r.started
}
// wait joins the goroutines and reports whether any write was
// refused with the eviction sentinel, plus any unexpected error.
func (r *racingWrites) wait() (bool, error) {
r.wg.Wait()
r.mu.Lock()
defer r.mu.Unlock()
return r.sawEvicted, r.otherErr
}
// TestEvictWebhook_RacingWriteDoesNotReopenHandle exercises the
// interleaving the evicted flag exists for: writes already
// contending for the writer's mutex when the eviction takes it.
// The write that wins the mutex after the eviction must abandon
// its work rather than reopen the archive, leaving the writer
// permanently handle-free. Run under -race.
func TestEvictWebhook_RacingWriteDoesNotReopenHandle(
t *testing.T,
) {
t.Parallel()
eng, _ := evictTestEngine(t)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":true}`)
d := seedDatabaseTargetDelivery(t, webhookDB, event, "")
// Prime the registry so the test can hold the very writer the
// eviction is about to detach.
eng.ExportDeliverDatabase(webhookDB, d)
w := eng.ExportArchiveWriterFor(event.WebhookID)
require.NotNil(t, w)
require.True(t, w.HandleOpen())
race := startRacingWrites(w)
// Evict only once writes are genuinely in flight, so the
// eviction has to contend for the writer's mutex.
race.awaitFirstWrite()
eng.EvictWebhook(event.WebhookID)
sawEvicted, otherErr := race.wait()
require.NoError(t, otherErr)
assert.True(
t, sawEvicted,
"a write after eviction must be refused",
)
assert.False(
t, w.HandleOpen(),
"no write may reopen the archive once the writer has "+
"been evicted",
)
assert.False(
t, eng.ExportHasArchiveWriter(event.WebhookID),
"the registry entry must stay gone",
)
}
// TestEvictWebhook_LaterDeliveryRecreatesWriter proves eviction
// does not break archiving for a webhook that is still alive: a
// subsequent delivery gets a brand new writer from the registry.
// It says nothing about the evicted writer itself — that is what
// TestEvictedWriter_WriteDoesNotReopenFile covers.
func TestEvictWebhook_LaterDeliveryRecreatesWriter(t *testing.T) {
t.Parallel()
eng, _ := evictTestEngine(t)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":true}`)
d := seedDatabaseTargetDelivery(t, webhookDB, event, "")
eng.ExportDeliverDatabase(webhookDB, d)
require.True(
t, eng.ExportHasArchiveWriter(event.WebhookID),
)
eng.EvictWebhook(event.WebhookID)
// A fresh delivery for the same webhook gets a brand new
// writer from the registry, so archiving keeps working.
second := seedDatabaseTargetDelivery(
t, webhookDB, event, "",
)
eng.ExportDeliverDatabase(webhookDB, second)
assert.True(
t, eng.ExportHasArchiveWriter(event.WebhookID),
"a later delivery should recreate the writer",
)
}

402
internal/delivery/target_database_test.go
View File

@@ -1,402 +0,0 @@
package delivery_test
import (
"database/sql"
"fmt"
"log/slog"
"net/http"
"os"
"path/filepath"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
_ "modernc.org/sqlite" // Pure Go SQLite driver.
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
func archiveTestLogger() *slog.Logger {
return slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
))
}
// openArchiveDBForRead opens an archive file read-only so a
// test can inspect the rows the writer persisted.
func openArchiveDBForRead(
t *testing.T, path string,
) *gorm.DB {
t.Helper()
sqlDB, err := sql.Open(
"sqlite",
fmt.Sprintf("file:%s?mode=ro", path),
)
require.NoError(t, err)
t.Cleanup(func() { _ = sqlDB.Close() })
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
require.NoError(t, err)
return gdb
}
// archiveFileSuffixes returns the archive file itself and the
// SQLite sidecars that accompany an open database. A test that
// asserts no archive was created has to check all of them.
func archiveFileSuffixes() []string {
return []string{"", "-wal", "-shm"}
}
// removeArchiveFiles simulates an operator moving the archive
// away by deleting the SQLite file and its sidecar files.
func removeArchiveFiles(t *testing.T, path string) {
t.Helper()
for _, suffix := range []string{
"", "-wal", "-shm", "-journal",
} {
err := os.Remove(path + suffix)
if err != nil && !os.IsNotExist(err) {
t.Fatalf("removing %s%s: %v", path, suffix, err)
}
}
}
// TestDeliverDatabase_ArchivesEvent verifies that delivering to
// a database target marks the delivery delivered and archives
// the full event into a separate per-webhook archive file.
func TestDeliverDatabase_ArchivesEvent(t *testing.T) {
t.Parallel()
dataDir := t.TempDir()
dbMgr := database.NewTestWebhookDBManager(dataDir)
e := delivery.NewTestEngineWithDB(
nil, dbMgr,
archiveTestLogger(),
&http.Client{Timeout: 5 * time.Second},
1,
)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":true}`)
d := seedDatabaseTargetDelivery(t, webhookDB, event, "")
e.ExportDeliverDatabase(webhookDB, d)
var updated database.Delivery
require.NoError(t, webhookDB.First(
&updated, "id = ?", d.ID,
).Error)
assert.Equal(t,
database.DeliveryStatusDelivered, updated.Status,
"database target should mark the delivery delivered",
)
archivePath := filepath.Join(
dataDir,
fmt.Sprintf("archive-%s.db", event.WebhookID),
)
assert.FileExists(t, archivePath)
rdb := openArchiveDBForRead(t, archivePath)
var rows []delivery.ExportArchivedEvent
require.NoError(t, rdb.Find(&rows).Error)
require.Len(t, rows, 1)
assert.Equal(t, event.ID, rows[0].EventID)
assert.Equal(t, event.WebhookID, rows[0].WebhookID)
assert.Equal(t, event.Method, rows[0].Method)
assert.JSONEq(t, `{"archived":true}`, rows[0].Body)
}
func TestArchiveWriter_WritesRow(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-wh.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
row := delivery.ExportArchivedEvent{
EventID: "ev-1",
WebhookID: "wh-1",
EntrypointID: "ep-1",
Method: "POST",
Headers: `{"X":"Y"}`,
Body: `{"hello":"world"}`,
ContentType: "application/json",
}
require.NoError(t, w.Write(row, 0))
assert.FileExists(t, path)
var got []delivery.ExportArchivedEvent
require.NoError(t, w.DB().Find(&got).Error)
require.Len(t, got, 1)
assert.Equal(t, "ev-1", got[0].EventID)
assert.Equal(t, "wh-1", got[0].WebhookID)
assert.Equal(t, "ep-1", got[0].EntrypointID)
assert.Equal(t, row.Method, got[0].Method)
assert.Equal(t, row.ContentType, got[0].ContentType)
assert.JSONEq(t, `{"hello":"world"}`, got[0].Body)
assert.False(t, got[0].ArchivedAt.IsZero())
}
func TestArchiveWriter_RecreatesAfterRemoval(
t *testing.T,
) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-wh.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "a"}, 0,
))
assert.FileExists(t, path)
// The operator moves the archive away while the handle is
// still open.
removeArchiveFiles(t, path)
require.NoFileExists(t, path)
// The next write recreates the file with a fresh schema and
// only the new row.
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "b"}, 0,
))
assert.FileExists(t, path)
var got []delivery.ExportArchivedEvent
require.NoError(t, w.DB().Find(&got).Error)
require.Len(t, got, 1)
assert.Equal(t, "b", got[0].EventID)
}
func TestArchiveWriter_ReopenDebounce(t *testing.T) {
t.Parallel()
// A generous debounce keeps the two rapid writes inside
// the window even on a heavily loaded test machine.
path := filepath.Join(t.TempDir(), "archive-wh.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 2*time.Second,
)
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "a"}, 0,
))
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "b"}, 0,
))
// Two writes inside the debounce window trigger only the
// initial open — no extra close/reopen.
assert.Equal(t, 1, w.Reopens())
time.Sleep(2100 * time.Millisecond)
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "c"}, 0,
))
// A write after the window elapses closes and reopens once.
assert.Equal(t, 2, w.Reopens())
}
func TestArchiveWriter_ExpiryPrune(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-wh.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.Open(0))
old := delivery.ExportArchivedEvent{
EventID: "old",
ArchivedAt: time.Now().Add(-2 * time.Hour),
}
fresh := delivery.ExportArchivedEvent{
EventID: "fresh",
ArchivedAt: time.Now(),
}
require.NoError(t, w.DB().Create(&old).Error)
require.NoError(t, w.DB().Create(&fresh).Error)
// Reopening with a one-hour expiry prunes the old row.
require.NoError(t, w.Reopen(time.Hour))
var got []delivery.ExportArchivedEvent
require.NoError(t, w.DB().Find(&got).Error)
require.Len(t, got, 1)
assert.Equal(t, "fresh", got[0].EventID)
}
func TestParseArchiveExpiry(t *testing.T) {
t.Parallel()
cases := []struct {
name string
in string
want time.Duration
wantErr bool
}{
{"empty config", "", 0, false},
{"explicit never", `{"expiry":"never"}`, 0, false},
{"empty expiry", `{"expiry":""}`, 0, false},
{"duration", `{"expiry":"1h"}`, time.Hour, false},
{"unparseable", `{"expiry":"nonsense"}`, 0, true},
{"zero duration", `{"expiry":"0s"}`, 0, true},
{"negative duration", `{"expiry":"-5h"}`, 0, true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
got, err := delivery.ExportParseArchiveExpiry(tc.in)
if tc.wantErr {
require.Error(t, err)
return
}
require.NoError(t, err)
assert.Equal(t, tc.want, got)
})
}
}
// seedDatabaseTargetDelivery seeds a pending delivery for a
// database target with the given config JSON and returns the
// in-memory delivery the target handler is invoked with.
func seedDatabaseTargetDelivery(
t *testing.T,
webhookDB *gorm.DB,
event database.Event,
config string,
) *database.Delivery {
t.Helper()
dlv := seedDelivery(
t, webhookDB, event.ID, uuid.New().String(),
database.DeliveryStatusPending,
)
d := &database.Delivery{
EventID: event.ID,
TargetID: dlv.TargetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: database.Target{
Name: "test-db",
Type: database.TargetTypeDatabase,
Config: config,
},
}
d.ID = dlv.ID
return d
}
// TestDeliverDatabase_ArchiveFailureFailsDelivery verifies that
// an archive error (here: an unparseable expiry in the target
// config) fails the delivery loudly: the attempt is recorded as
// failed with the error and the delivery is marked failed, not
// delivered.
func TestDeliverDatabase_ArchiveFailureFailsDelivery(
t *testing.T,
) {
t.Parallel()
dataDir := t.TempDir()
e := delivery.NewTestEngineWithDB(
nil, database.NewTestWebhookDBManager(dataDir),
archiveTestLogger(),
&http.Client{Timeout: 5 * time.Second},
1,
)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":false}`)
d := seedDatabaseTargetDelivery(
t, webhookDB, event, `{"expiry":"nonsense"}`,
)
e.ExportDeliverDatabase(webhookDB, d)
var updated database.Delivery
require.NoError(t, webhookDB.First(
&updated, "id = ?", d.ID,
).Error)
assert.Equal(t,
database.DeliveryStatusFailed, updated.Status,
"archive failure must mark the delivery failed",
)
var results []database.DeliveryResult
require.NoError(t, webhookDB.Where(
"delivery_id = ?", d.ID,
).Find(&results).Error)
require.Len(t, results, 1)
assert.False(t,
results[0].Success,
"the attempt must be recorded as failed",
)
assert.Contains(t,
results[0].Error, "nonsense",
"the archive error must be recorded on the attempt",
)
assert.NoFileExists(t,
filepath.Join(
dataDir,
fmt.Sprintf("archive-%s.db", event.WebhookID),
),
"no archive file should exist for a failed config",
)
}
func TestValidateArchiveExpiry(t *testing.T) {
t.Parallel()
valid := []string{"", "never", "1h", "720h", "30m"}
for _, in := range valid {
require.NoError(t,
delivery.ValidateArchiveExpiry(in),
"expiry %q should be accepted", in,
)
}
invalid := []string{"nonsense", "7d", "-5h", "0s", "0"}
for _, in := range invalid {
require.Error(t,
delivery.ValidateArchiveExpiry(in),
"expiry %q should be rejected", in,
)
}
}

2
internal/delivery/target_http.go
View File

@@ -495,5 +495,5 @@ func applyRequestHeaders(
func executeHTTPRequest(
client *http.Client, req *http.Request,
) (*http.Response, error) {
return client.Do(req) //#nosec G704 -- validated URL, SSRF-safe transport
return client.Do(req) //#nosec G704 -- URL validated by parseHTTPConfig/parseSlackConfig and SSRF-safe transport
}

10
internal/handlers/auth.go
View File

@@ -19,7 +19,7 @@ func (h *Handlers) HandleLoginPage() http.HandlerFunc {
// Render login page
data := map[string]any{
tmplKeyError: "",
"Error": "",
}
h.renderTemplate(w, r, "login.html", data)
@@ -29,8 +29,10 @@ func (h *Handlers) HandleLoginPage() http.HandlerFunc {
// HandleLoginSubmit handles the login form submission (POST)
func (h *Handlers) HandleLoginSubmit() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
// The body size cap is enforced by the MaxBodySize
// middleware, which runs before CSRF parses the form.
// Limit request body to prevent memory exhaustion
r.Body = http.MaxBytesReader(w, r.Body, 1<<maxBodyShift)
// Parse form data
err := r.ParseForm()
if err != nil {
h.log.Error("failed to parse form", "error", err)
@@ -84,7 +86,7 @@ func (h *Handlers) renderLoginError(
status int,
) {
data := map[string]any{
tmplKeyError: msg,
"Error": msg,
}
w.WriteHeader(status)

20
internal/handlers/export_test.go
View File

@@ -13,26 +13,12 @@ func (s *Handlers) RenderTemplateForTest(
s.renderTemplate(w, r, pageTemplate, data)
}
// BuildSlackTargetConfigForTest exposes buildURLTargetConfig
// with the Slack target parameters for use in the
// handlers_test package.
// BuildSlackTargetConfigForTest exposes buildSlackTargetConfig
// for use in the handlers_test package.
func (s *Handlers) BuildSlackTargetConfigForTest(
w http.ResponseWriter,
r *http.Request,
targetURL string,
) (string, error) {
return s.buildURLTargetConfig(
w, r, targetURL, "webhookUrl",
"Webhook URL is required for Slack targets",
)
}
// BuildDatabaseTargetConfigForTest exposes
// buildDatabaseTargetConfig for use in the handlers_test
// package.
func (s *Handlers) BuildDatabaseTargetConfigForTest(
w http.ResponseWriter,
expiry string,
) (string, error) {
return s.buildDatabaseTargetConfig(w, expiry)
return s.buildSlackTargetConfig(w, r, targetURL)
}

10
internal/handlers/handlers.go
View File

@@ -26,13 +26,10 @@ const (
maxBodyShift = 20
// recentEventLimit is the number of recent events to show.
recentEventLimit = 20
// defaultRetentionDays is the default event retention period.
defaultRetentionDays = 30
// paginationPerPage is the number of items per page.
paginationPerPage = 25
// tmplKeyError is the template data key for an error message.
tmplKeyError = "Error"
// tmplKeyWebhook is the template data key for a webhook.
tmplKeyWebhook = "Webhook"
)
// errInvalidPassword is returned when a password does not match.
@@ -49,7 +46,6 @@ type HandlersParams struct {
Healthcheck *healthcheck.Healthcheck
Session *session.Session
Notifier delivery.Notifier
Evictor delivery.WebhookEvictor
}
// Handlers provides HTTP handler methods for all application
@@ -62,7 +58,6 @@ type Handlers struct {
dbMgr *database.WebhookDBManager
session *session.Session
notifier delivery.Notifier
evictor delivery.WebhookEvictor
templates map[string]*template.Template
}
@@ -97,7 +92,6 @@ func New(
s.dbMgr = params.WebhookDBMgr
s.session = params.Session
s.notifier = params.Notifier
s.evictor = params.Evictor
// Parse all page templates once at startup
s.templates = map[string]*template.Template{

87
internal/handlers/handlers_test.go
View File

@@ -4,7 +4,6 @@ import (
"context"
"net/http"
"net/http/httptest"
"sync"
"testing"
"github.com/stretchr/testify/assert"
@@ -25,32 +24,6 @@ type noopNotifier struct{}
func (n *noopNotifier) Notify([]delivery.Task) {}
// recordingEvictor is a delivery.WebhookEvictor that records
// the webhook ids it was asked to evict, so a test can prove
// that a deletion path reached the delivery engine.
type recordingEvictor struct {
mu sync.Mutex
evicted []string
}
func (r *recordingEvictor) EvictWebhook(webhookID string) {
r.mu.Lock()
defer r.mu.Unlock()
r.evicted = append(r.evicted, webhookID)
}
// Evicted returns a copy of the recorded webhook ids.
func (r *recordingEvictor) Evicted() []string {
r.mu.Lock()
defer r.mu.Unlock()
out := make([]string, len(r.evicted))
copy(out, r.evicted)
return out
}
func newTestApp(
t *testing.T,
targets ...any,
@@ -74,12 +47,6 @@ func newTestApp(
func() delivery.Notifier {
return &noopNotifier{}
},
func() *recordingEvictor {
return &recordingEvictor{}
},
func(r *recordingEvictor) delivery.WebhookEvictor {
return r
},
handlers.New,
),
fx.Populate(targets...),
@@ -219,57 +186,3 @@ func TestRenderTemplate(t *testing.T) {
t, http.StatusInternalServerError, w.Code,
)
}
func TestBuildDatabaseTargetConfig_Valid(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
app := newTestApp(t, &h)
app.RequireStart()
t.Cleanup(app.RequireStop)
// Empty expiry: the keep-forever default, empty config.
w := httptest.NewRecorder()
cfg, err := h.BuildDatabaseTargetConfigForTest(w, "")
require.NoError(t, err)
assert.Empty(t, cfg)
// Explicit never is stored as config.
w = httptest.NewRecorder()
cfg, err = h.BuildDatabaseTargetConfigForTest(w, "never")
require.NoError(t, err)
assert.JSONEq(t, `{"expiry":"never"}`, cfg)
// A positive duration is stored as config.
w = httptest.NewRecorder()
cfg, err = h.BuildDatabaseTargetConfigForTest(w, "720h")
require.NoError(t, err)
assert.JSONEq(t, `{"expiry":"720h"}`, cfg)
}
func TestBuildDatabaseTargetConfig_RejectsBadExpiry(
t *testing.T,
) {
t.Parallel()
var h *handlers.Handlers
app := newTestApp(t, &h)
app.RequireStart()
t.Cleanup(app.RequireStop)
for _, bad := range []string{"nonsense", "7d", "-5h"} {
w := httptest.NewRecorder()
cfg, err := h.BuildDatabaseTargetConfigForTest(w, bad)
require.Error(t, err, "expiry %q", bad)
assert.Empty(t, cfg)
assert.Equal(
t, http.StatusBadRequest, w.Code,
"expiry %q should be rejected with 400", bad,
)
}
}

178
internal/handlers/profile.go
View File

@@ -4,201 +4,63 @@ import (
"net/http"
"github.com/go-chi/chi"
"sneak.berlin/go/webhooker/internal/database"
)
// HandleProfile returns a handler for the user profile page
func (h *Handlers) HandleProfile() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
sessionUserID, sessionUsername, ok :=
h.profileOwnerOrDeny(w, r)
if !ok {
return
}
h.renderProfile(w, r, sessionUserID, sessionUsername, "", "")
}
}
// HandlePasswordChange returns a handler that lets an authenticated
// user change their own password. It is served by the CSRF- and
// auth-protected POST /password route under /user/{username}.
func (h *Handlers) HandlePasswordChange() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
sessionUserID, sessionUsername, ok :=
h.profileOwnerOrDeny(w, r)
if !ok {
return
}
// The body size cap is enforced by the MaxBodySize
// middleware, which runs before CSRF parses the form.
err := r.ParseForm()
if err != nil {
h.log.Error("failed to parse form", "error", err)
http.Error(w, "Bad request", http.StatusBadRequest)
return
}
successMessage, errorMessage, handled := h.applyPasswordChange(
w,
sessionUsername,
r.FormValue("current_password"),
r.FormValue("new_password"),
r.FormValue("confirm_password"),
)
if !handled {
return
}
h.renderProfile(
w, r, sessionUserID, sessionUsername,
successMessage, errorMessage,
)
}
}
// applyPasswordChange verifies the current password and, on success,
// persists a fresh hash for the user, reusing the same helpers that
// bootstrap the admin user. It returns the success and error messages
// to display on the profile page. On an internal failure it writes a
// 500 response itself and returns handled=false, signalling the caller
// to stop without re-rendering the page.
func (h *Handlers) applyPasswordChange(
w http.ResponseWriter,
username, currentPassword, newPassword, confirmPassword string,
) (string, string, bool) {
// Load the user row so we can verify the current password and
// persist the new hash.
var user database.User
err := h.db.DB().Where(
"username = ?", username,
).First(&user).Error
if err != nil {
h.serverError(
w, "failed to load user for password change", err,
)
return "", "", false
}
valid, err := database.VerifyPassword(
currentPassword, user.Password,
)
if err != nil {
h.serverError(w, "failed to verify password", err)
return "", "", false
}
if !valid {
return "", "Current password is incorrect.", true
}
if newPassword == "" {
return "", "New password must not be empty.", true
}
if newPassword != confirmPassword {
return "", "New password and confirmation do not match.", true
}
hashedPassword, err := database.HashPassword(newPassword)
if err != nil {
h.serverError(w, "failed to hash new password", err)
return "", "", false
}
err = h.db.DB().Model(&user).Update(
"password", hashedPassword,
).Error
if err != nil {
h.serverError(w, "failed to update password", err)
return "", "", false
}
h.log.Info("user changed password", "username", username)
return "Password changed successfully.", "", true
}
// profileOwnerOrDeny resolves the session identity and enforces that a
// user may only act on their own profile (the requested username in the
// URL must equal the session username). On any failure it writes the
// appropriate HTTP response and returns ok=false; callers must stop
// when ok is false.
func (h *Handlers) profileOwnerOrDeny(
w http.ResponseWriter,
r *http.Request,
) (string, string, bool) {
// Get username from URL
requestedUsername := chi.URLParam(r, "username")
if requestedUsername == "" {
http.NotFound(w, r)
return "", "", false
return
}
// RequireAuth middleware guarantees an authenticated session
// before this handler runs, so we only need to guard against an
// unexpected retrieval error.
// Get session. RequireAuth middleware guarantees an
// authenticated session before this handler runs, so we
// only need to guard against an unexpected retrieval error.
sess, err := h.session.Get(r)
if err != nil {
h.serverError(w, "failed to get session", err)
h.log.Error("failed to get session", "error", err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return "", "", false
return
}
// Get user info from session
sessionUsername, ok := h.session.GetUsername(sess)
if !ok {
h.log.Error("authenticated session missing username")
http.Error(
w, "Internal server error",
http.StatusInternalServerError,
)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return "", "", false
return
}
sessionUserID, ok := h.session.GetUserID(sess)
if !ok {
h.log.Error("authenticated session missing user ID")
http.Error(
w, "Internal server error",
http.StatusInternalServerError,
)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return "", "", false
return
}
// Only allow users to act on their own profile.
// For now, only allow users to view their own profile
if requestedUsername != sessionUsername {
http.Error(w, "Forbidden", http.StatusForbidden)
return "", "", false
return
}
return sessionUserID, sessionUsername, true
}
// renderProfile renders the profile page for the given user,
// optionally including a success or error message.
func (h *Handlers) renderProfile(
w http.ResponseWriter,
r *http.Request,
userID, username, successMessage, errorMessage string,
) {
// Prepare data for template
data := map[string]any{
"User": &UserInfo{
ID: userID,
Username: username,
ID: sessionUserID,
Username: sessionUsername,
},
"SuccessMessage": successMessage,
"ErrorMessage": errorMessage,
}
// Render the profile page
h.renderTemplate(w, r, "profile.html", data)
}
}

134
internal/handlers/profile_test.go
View File

@@ -4,15 +4,12 @@ import (
"context"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/logger"
"sneak.berlin/go/webhooker/internal/middleware"
@@ -160,134 +157,3 @@ func TestUserRoute_Unauthenticated_RedirectedByMiddleware(t *testing.T) {
assert.Equal(t, http.StatusSeeOther, w.Code)
assert.Equal(t, "/pages/login", w.Header().Get("Location"))
}
// passwordChangeRequest builds a POST request to the password-change
// endpoint for the given username, attaching the supplied cookies, an
// urlencoded form body, and the chi URL parameter the handler reads.
func passwordChangeRequest(
username string,
cookies []*http.Cookie,
form url.Values,
) *http.Request {
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost,
"/user/"+username+"/password",
strings.NewReader(form.Encode()),
)
req.Header.Set(
"Content-Type", "application/x-www-form-urlencoded",
)
for _, c := range cookies {
req.AddCookie(c)
}
rctx := chi.NewRouteContext()
rctx.URLParams.Add("username", username)
return req.WithContext(
context.WithValue(req.Context(), chi.RouteCtxKey, rctx),
)
}
func TestHandlePasswordChange_Success(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
var sess *session.Session
var db *database.Database
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
oldHash, err := database.HashPassword("oldpassword")
require.NoError(t, err)
user := &database.User{Username: "pwuser", Password: oldHash}
require.NoError(t, db.DB().Create(user).Error)
cookies := authenticatedCookies(t, sess, user.ID, "pwuser")
form := url.Values{}
form.Set("current_password", "oldpassword")
form.Set("new_password", "newpassword")
form.Set("confirm_password", "newpassword")
req := passwordChangeRequest("pwuser", cookies, form)
w := httptest.NewRecorder()
h.HandlePasswordChange().ServeHTTP(w, req)
assert.Equal(t, http.StatusOK, w.Code)
assert.Contains(
t, w.Body.String(), "Password changed successfully.",
)
var updated database.User
require.NoError(t,
db.DB().Where("username = ?", "pwuser").First(&updated).Error,
)
assert.NotEqual(t, oldHash, updated.Password)
valid, err := database.VerifyPassword(
"newpassword", updated.Password,
)
require.NoError(t, err)
assert.True(t, valid, "new password should verify against new hash")
}
func TestHandlePasswordChange_WrongCurrentPassword(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
var sess *session.Session
var db *database.Database
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
oldHash, err := database.HashPassword("oldpassword")
require.NoError(t, err)
user := &database.User{Username: "pwuser2", Password: oldHash}
require.NoError(t, db.DB().Create(user).Error)
cookies := authenticatedCookies(t, sess, user.ID, "pwuser2")
form := url.Values{}
form.Set("current_password", "wrongpassword")
form.Set("new_password", "newpassword")
form.Set("confirm_password", "newpassword")
req := passwordChangeRequest("pwuser2", cookies, form)
w := httptest.NewRecorder()
h.HandlePasswordChange().ServeHTTP(w, req)
assert.Equal(t, http.StatusOK, w.Code)
assert.Contains(
t, w.Body.String(), "Current password is incorrect.",
)
var unchanged database.User
require.NoError(t,
db.DB().Where(
"username = ?", "pwuser2",
).First(&unchanged).Error,
)
assert.Equal(
t, oldHash, unchanged.Password,
"stored hash must be unchanged after a rejected change",
)
}

356
internal/handlers/source_delete_test.go
View File

@@ -1,356 +0,0 @@
package handlers_test
import (
"context"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
const (
deleteTestUserID = "test-user-id"
deleteTestUsername = "testuser"
// paramSourceID and paramTargetID are the chi URL parameter
// names the deletion handlers read.
paramSourceID = "sourceID"
paramTargetID = "targetID"
)
// seedWebhook inserts a webhook owned by the test user and
// returns it.
func seedWebhook(
t *testing.T,
db *database.Database,
) *database.Webhook {
t.Helper()
wh := &database.Webhook{
UserID: deleteTestUserID,
Name: "delete-me",
}
require.NoError(
t,
db.DB().Omit(clause.Associations).Create(wh).Error,
)
return wh
}
// seedTarget inserts a target of the given type for a webhook
// and returns it.
func seedTarget(
t *testing.T,
db *database.Database,
webhookID string,
targetType database.TargetType,
) *database.Target {
t.Helper()
tgt := &database.Target{
WebhookID: webhookID,
Name: "t-" + string(targetType),
Type: targetType,
Active: true,
}
require.NoError(
t,
db.DB().Omit(clause.Associations).Create(tgt).Error,
)
return tgt
}
// archivePathFor returns the archive database path the
// delivery engine would use for a webhook: beside the webhook's
// event database in the data directory.
func archivePathFor(
t *testing.T,
mgr *database.WebhookDBManager,
webhookID string,
) string {
t.Helper()
return filepath.Join(
filepath.Dir(mgr.DBPath(webhookID)),
"archive-"+webhookID+".db",
)
}
// writeArchivePlaceholder creates a stand-in archive file so a
// test can assert the file survives webhook deletion.
func writeArchivePlaceholder(path string) error {
return os.WriteFile(path, []byte("archive"), 0o600)
}
// postRequest builds an authenticated POST request carrying the
// given chi URL parameters.
func postRequest(
path string,
cookies []*http.Cookie,
params map[string]string,
) *http.Request {
req := httptest.NewRequestWithContext(
context.Background(), http.MethodPost, path, nil,
)
for _, c := range cookies {
req.AddCookie(c)
}
rctx := chi.NewRouteContext()
for k, v := range params {
rctx.URLParams.Add(k, v)
}
return req.WithContext(
context.WithValue(req.Context(), chi.RouteCtxKey, rctx),
)
}
// TestHandleSourceDelete_EvictsArchiveWriter proves that
// deleting a webhook reaches the delivery engine and releases
// the webhook's archive writer, exercised through the real
// deletion handler rather than by calling the evictor directly.
func TestHandleSourceDelete_EvictsArchiveWriter(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
ev *recordingEvictor
)
app := newTestApp(t, &h, &sess, &db, &ev)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedTarget(t, db, wh.ID, database.TargetTypeDatabase)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/delete",
cookies,
map[string]string{paramSourceID: wh.ID},
)
w := httptest.NewRecorder()
h.HandleSourceDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Equal(
t, []string{wh.ID}, ev.Evicted(),
"deleting a webhook should evict its archive writer",
)
}
// TestHandleSourceDelete_KeepsArchiveFile proves that deleting
// a webhook does not remove its archive database file: the
// archive is long-term storage the operator owns.
func TestHandleSourceDelete_KeepsArchiveFile(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
mgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &mgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
// Place an archive file where the delivery engine would.
archivePath := archivePathFor(t, mgr, wh.ID)
require.NoError(
t,
writeArchivePlaceholder(archivePath),
)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/delete",
cookies,
map[string]string{paramSourceID: wh.ID},
)
w := httptest.NewRecorder()
h.HandleSourceDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
assert.FileExists(
t, archivePath,
"webhook deletion must not destroy the archive file",
)
}
// TestHandleTargetDelete_EvictsWhenLastDatabaseTargetGone
// proves that removing the last database target releases the
// archive writer.
func TestHandleTargetDelete_EvictsWhenLastDatabaseTargetGone(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
ev *recordingEvictor
)
app := newTestApp(t, &h, &sess, &db, &ev)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
tgt := seedTarget(
t, db, wh.ID, database.TargetTypeDatabase,
)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/targets/"+tgt.ID+"/delete",
cookies,
map[string]string{
paramSourceID: wh.ID,
paramTargetID: tgt.ID,
},
)
w := httptest.NewRecorder()
h.HandleTargetDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Equal(
t, []string{wh.ID}, ev.Evicted(),
"removing the last database target should evict",
)
}
// TestHandleTargetDelete_KeepsWriterWhenDatabaseTargetRemains
// proves that deleting one of several database targets leaves
// the still-needed archive writer alone: the surviving target
// keeps archiving to the same file, so the writer must stay.
func TestHandleTargetDelete_KeepsWriterWhenDatabaseTargetRemains(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
ev *recordingEvictor
)
app := newTestApp(t, &h, &sess, &db, &ev)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
doomed := seedTarget(
t, db, wh.ID, database.TargetTypeDatabase,
)
seedTarget(t, db, wh.ID, database.TargetTypeDatabase)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/targets/"+doomed.ID+"/delete",
cookies,
map[string]string{
paramSourceID: wh.ID,
paramTargetID: doomed.ID,
},
)
w := httptest.NewRecorder()
h.HandleTargetDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Empty(
t, ev.Evicted(),
"a second database target still needs the writer",
)
}
// TestHandleTargetDelete_KeepsWriterWhenOtherTypeDeleted proves
// that deleting a target of an unrelated type leaves a
// still-needed archive writer alone: the webhook's database
// target is untouched, so its writer must stay.
func TestHandleTargetDelete_KeepsWriterWhenOtherTypeDeleted(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
ev *recordingEvictor
)
app := newTestApp(t, &h, &sess, &db, &ev)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedTarget(t, db, wh.ID, database.TargetTypeDatabase)
other := seedTarget(t, db, wh.ID, database.TargetTypeLog)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/targets/"+other.ID+"/delete",
cookies,
map[string]string{
paramSourceID: wh.ID,
paramTargetID: other.ID,
},
)
w := httptest.NewRecorder()
h.HandleTargetDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Empty(
t, ev.Evicted(),
"a surviving database target must keep its writer",
)
}

185
internal/handlers/source_detail_test.go
View File

@@ -1,185 +0,0 @@
package handlers_test
import (
"context"
"net/http"
"net/http/httptest"
"testing"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
// The secret path segments of a Slack incoming webhook URL.
// Holding them is enough to post to the channel forever, so
// they must never reach the rendered page.
const (
slackSecretPath = "/services/T00000000/B00000000/" +
"XXXXXXXXXXXXXXXXXXXXXXXX"
slackWebhookURL = "https://hooks.slack.com" +
slackSecretPath
)
// seedConfiguredTarget inserts a target with a stored config
// blob.
func seedConfiguredTarget(
t *testing.T,
db *database.Database,
webhookID string,
targetType database.TargetType,
config string,
) {
t.Helper()
tgt := &database.Target{
WebhookID: webhookID,
Name: "t-" + string(targetType),
Type: targetType,
Active: true,
Config: config,
}
require.NoError(
t,
db.DB().Omit(clause.Associations).Create(tgt).Error,
)
}
// renderSourceDetailPage runs the real source detail handler
// for a webhook and returns the rendered HTML.
func renderSourceDetailPage(
t *testing.T,
h *handlers.Handlers,
sess *session.Session,
webhookID string,
) string {
t.Helper()
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet,
"/source/"+webhookID,
nil,
)
for _, c := range authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
) {
req.AddCookie(c)
}
rctx := chi.NewRouteContext()
rctx.URLParams.Add(paramSourceID, webhookID)
req = req.WithContext(
context.WithValue(
req.Context(), chi.RouteCtxKey, rctx,
),
)
w := httptest.NewRecorder()
h.HandleSourceDetail().ServeHTTP(w, req)
require.Equal(t, http.StatusOK, w.Code)
return w.Body.String()
}
// TestHandleSourceDetail_MasksSlackWebhookURL is the
// load-bearing regression test for the credential leak: the
// rendered page must show the Slack target without any of the
// secret path segments of its webhook URL.
func TestHandleSourceDetail_MasksSlackWebhookURL(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
)
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedConfiguredTarget(
t, db, wh.ID,
database.TargetTypeSlack,
`{"webhookUrl":"`+slackWebhookURL+`"}`,
)
body := renderSourceDetailPage(t, h, sess, wh.ID)
assert.NotContains(t, body, slackSecretPath)
assert.NotContains(t, body, "T00000000")
assert.NotContains(t, body, "B00000000")
assert.NotContains(
t, body, "XXXXXXXXXXXXXXXXXXXXXXXX",
)
assert.NotContains(t, body, "webhookUrl")
assert.Contains(t, body, "Webhook URL")
assert.Contains(t, body, "https://hooks.slack.com/...")
}
// TestHandleSourceDetail_RendersNamedTargetFields proves the
// other target types render labelled fields rather than the
// stored blob.
func TestHandleSourceDetail_RendersNamedTargetFields(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
)
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedConfiguredTarget(
t, db, wh.ID,
database.TargetTypeHTTP,
`{"url":"https://example.com/hook","timeout":30,`+
`"headers":{"Authorization":"Bearer sekrit"}}`,
)
seedConfiguredTarget(
t, db, wh.ID,
database.TargetTypeDatabase,
`{"expiry":"720h"}`,
)
seedConfiguredTarget(
t, db, wh.ID,
database.TargetType("carrier-pigeon"),
`{"beak":"sharp"}`,
)
body := renderSourceDetailPage(t, h, sess, wh.ID)
assert.Contains(t, body, "Destination URL")
assert.Contains(t, body, "https://example.com/hook")
assert.Contains(t, body, "Timeout")
assert.Contains(t, body, "1 configured")
assert.NotContains(t, body, "sekrit")
assert.Contains(t, body, "Archive Expiry")
assert.Contains(t, body, "720h")
// An unknown type gets the neutral placeholder, never the
// stored blob.
assert.Contains(t, body, "(unavailable)")
assert.NotContains(t, body, "beak")
}

371
internal/handlers/source_management.go
View File

@@ -5,7 +5,6 @@ import (
"errors"
"net/http"
"strconv"
"strings"
"github.com/go-chi/chi"
"github.com/google/uuid"
@@ -25,73 +24,6 @@ type WebhookListItem struct {
// errMissingURL signals that a required URL was not provided.
var errMissingURL = errors.New("missing URL")
// errInvalidRetention signals a retention_days form value that is not
// a non-negative whole number.
var errInvalidRetention = errors.New("invalid retention days")
// errRetentionTooLarge signals a retention_days form value that is a
// whole number but larger than the reaper's cutoff arithmetic can
// represent. It is distinguished from errInvalidRetention so the form
// can tell the user the actual ceiling instead of implying their input
// was not a number.
var errRetentionTooLarge = errors.New("retention days out of range")
// retentionErrorMessage returns the message the create and edit forms
// show the user for a rejected retention_days value. Any error other
// than errRetentionTooLarge falls back to the generic wording, so an
// unrecognised parse failure still produces a sensible 400 rather than
// an empty alert.
func retentionErrorMessage(err error) string {
if errors.Is(err, errRetentionTooLarge) {
return "Retention must be at most " +
strconv.Itoa(database.MaxFiniteRetentionDays) +
" days, or 0 to retain events forever."
}
return "Retention must be a whole number of days, or 0 to " +
"retain events forever."
}
// parseRetentionDays interprets a retention_days form value.
//
// An empty value yields fallback, which lets the create path apply the
// default and the edit path leave the stored value unchanged. A value
// of 0 is returned as 0 and is rewritten to the retain-forever
// sentinel by database.Webhook's BeforeSave hook. Anything unparseable
// or negative is an error rather than a silently substituted default.
//
// The upper bound is not cosmetic. The reaper computes its cutoff as a
// time.Duration, an int64 nanosecond count, so a day count above
// database.MaxFiniteRetentionDays overflows, puts the cutoff in the
// future, and deletes every event the webhook has. A finite value
// above that ceiling is therefore a 400.
//
// A value at or above the retain-forever sentinel is not out of range:
// it is what the edit form pre-fills for a retain-forever webhook, so
// submitting the form back unchanged has to keep meaning "forever"
// rather than being rejected.
func parseRetentionDays(raw string, fallback int) (int, error) {
raw = strings.TrimSpace(raw)
if raw == "" {
return fallback, nil
}
v, err := strconv.Atoi(raw)
if err != nil || v < 0 {
return 0, errInvalidRetention
}
if v >= database.RetentionForeverDays {
return database.RetentionForeverDays, nil
}
if v > database.MaxFiniteRetentionDays {
return 0, errRetentionTooLarge
}
return v, nil
}
// EventWithDeliveries holds an event and its deliveries.
type EventWithDeliveries struct {
database.Event
@@ -173,30 +105,11 @@ func (h *Handlers) buildWebhookListItems(
// HandleSourceCreate shows the form to create a new webhook.
func (h *Handlers) HandleSourceCreate() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
h.renderTemplate(
w, r, "sources_new.html",
newSourceFormData("", "", ""),
)
data := map[string]any{
"Error": "",
}
}
// newSourceFormData builds the template data for the webhook creation
// form.
//
// It carries the retention default so the pre-filled value comes from
// database.DefaultRetentionDays rather than being a third hardcoded
// copy of the same policy, and it carries the submitted name and
// description so that re-rendering the form after a validation failure
// gives the user their input back instead of a blank form. The edit
// form already behaves that way; create now matches it.
func newSourceFormData(
errMsg, name, description string,
) map[string]any {
return map[string]any{
tmplKeyError: errMsg,
"Name": name,
"Description": description,
"DefaultRetentionDays": database.DefaultRetentionDays,
h.renderTemplate(w, r, "sources_new.html", data)
}
}
@@ -213,8 +126,10 @@ func (h *Handlers) HandleSourceCreateSubmit() http.HandlerFunc {
return
}
// The body size cap is enforced by the MaxBodySize
// middleware, which runs before CSRF parses the form.
r.Body = http.MaxBytesReader(
w, r.Body, 1<<maxBodyShift,
)
err := r.ParseForm()
if err != nil {
http.Error(
@@ -229,31 +144,23 @@ func (h *Handlers) HandleSourceCreateSubmit() http.HandlerFunc {
retentionStr := r.FormValue("retention_days")
if name == "" {
data := map[string]any{
"Error": "Name is required",
}
w.WriteHeader(http.StatusBadRequest)
h.renderTemplate(
w, r, "sources_new.html",
newSourceFormData(
"Name is required", name, description,
),
)
h.renderTemplate(w, r, "sources_new.html", data)
return
}
retentionDays, retErr := parseRetentionDays(
retentionStr, database.DefaultRetentionDays,
)
if retErr != nil {
w.WriteHeader(http.StatusBadRequest)
h.renderTemplate(
w, r, "sources_new.html",
newSourceFormData(
retentionErrorMessage(retErr),
name, description,
),
)
retentionDays := defaultRetentionDays
return
if retentionStr != "" {
v, convErr := strconv.Atoi(retentionStr)
if convErr == nil && v > 0 {
retentionDays = v
}
}
h.createWebhookWithEntrypoint(
@@ -407,15 +314,10 @@ func (h *Handlers) renderSourceDetail(
scheme = fwdProto
}
// The template calls Webhook methods, which take pointer
// receivers; html/template cannot address a value stored in a map.
data := map[string]any{
tmplKeyWebhook: &webhook,
"Webhook": webhook,
"Entrypoints": entrypoints,
// Targets are projected to a display-safe view: the
// stored config blob holds credentials and must never
// reach a template.
"Targets": delivery.NewTargetViews(targets),
"Targets": targets,
"Events": events,
"BaseURL": scheme + "://" + host,
}
@@ -449,8 +351,8 @@ func (h *Handlers) HandleSourceEdit() http.HandlerFunc {
}
data := map[string]any{
tmplKeyWebhook: &webhook,
tmplKeyError: "",
"Webhook": webhook,
"Error": "",
}
h.renderTemplate(w, r, "source_edit.html", data)
@@ -483,8 +385,10 @@ func (h *Handlers) HandleSourceEditSubmit() http.HandlerFunc {
return
}
// The body size cap is enforced by the MaxBodySize
// middleware, which runs before CSRF parses the form.
r.Body = http.MaxBytesReader(
w, r.Body, 1<<maxBodyShift,
)
err = r.ParseForm()
if err != nil {
http.Error(
@@ -504,13 +408,15 @@ func (h *Handlers) applyWebhookEdit(
r *http.Request,
webhook *database.Webhook,
) {
// The body size cap is enforced by the MaxBodySize middleware,
// which runs before CSRF parses the form.
r.Body = http.MaxBytesReader(
w, r.Body, 1<<maxBodyShift,
)
name := r.FormValue("name")
if name == "" {
data := map[string]any{
tmplKeyWebhook: webhook,
tmplKeyError: "Name is required",
"Webhook": *webhook,
"Error": "Name is required",
}
w.WriteHeader(http.StatusBadRequest)
@@ -521,25 +427,7 @@ func (h *Handlers) applyWebhookEdit(
webhook.Name = name
webhook.Description = r.FormValue("description")
// An empty field falls back to the stored value, so submitting the
// form without touching retention leaves the policy alone.
retentionDays, retErr := parseRetentionDays(
r.FormValue("retention_days"), webhook.RetentionDays,
)
if retErr != nil {
data := map[string]any{
tmplKeyWebhook: webhook,
tmplKeyError: retentionErrorMessage(retErr),
}
w.WriteHeader(http.StatusBadRequest)
h.renderTemplate(w, r, "source_edit.html", data)
return
}
webhook.RetentionDays = retentionDays
h.parseRetention(r, webhook)
err := h.db.DB().Save(webhook).Error
if err != nil {
@@ -553,6 +441,23 @@ func (h *Handlers) applyWebhookEdit(
)
}
// parseRetention parses and applies retention_days from the
// form.
func (h *Handlers) parseRetention(
r *http.Request,
webhook *database.Webhook,
) {
retStr := r.FormValue("retention_days")
if retStr == "" {
return
}
v, err := strconv.Atoi(retStr)
if err == nil && v > 0 {
webhook.RetentionDays = v
}
}
// HandleSourceDelete handles webhook deletion.
func (h *Handlers) HandleSourceDelete() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
@@ -627,13 +532,6 @@ func (h *Handlers) deleteWebhookResources(
return
}
// Release the delivery engine's per-webhook archiving state
// so a deleted webhook's archive writer (and any handle open
// within its debounce window) does not linger for the
// process lifetime. The archive file itself is deliberately
// left on disk; see evictArchiveWriter.
h.evictArchiveWriter(webhook.ID)
err = h.dbMgr.DeleteDB(webhook.ID)
if err != nil {
h.log.Error(
@@ -652,64 +550,6 @@ func (h *Handlers) deleteWebhookResources(
http.Redirect(w, r, "/sources", http.StatusSeeOther)
}
// evictArchiveWriter asks the delivery engine to drop its
// cached archive writer for a webhook, closing the archive file
// handle.
//
// The archive database file is NOT deleted. Unlike the event
// database — which is per-webhook working storage and is
// hard-deleted with the webhook — an archive is explicitly
// long-term storage that an operator may want to keep or move
// away for offline retention. Destroying it as a side effect of
// deleting a webhook would be a surprising and unrecoverable
// data loss, so the file is left for the operator to handle.
func (h *Handlers) evictArchiveWriter(webhookID string) {
if h.evictor == nil {
return
}
h.evictor.EvictWebhook(webhookID)
}
// evictArchiveWriterIfUnused releases a webhook's archive
// writer once the webhook has no database target left to feed
// it.
//
// It is called after any child resource of a webhook is
// deleted, and is correct without knowing which kind was: it
// evicts only when no database target remains, so deleting one
// of several database targets — or deleting an unrelated
// target type — leaves a still-needed writer alone. When no
// database target ever existed there is no writer and eviction
// is a no-op. Soft-deleted targets are excluded by GORM's
// default scope, so the row just deleted is not counted.
func (h *Handlers) evictArchiveWriterIfUnused(webhookID string) {
var remaining int64
err := h.db.DB().
Model(&database.Target{}).
Where(
"webhook_id = ? AND type = ?",
webhookID, database.TargetTypeDatabase,
).
Count(&remaining).Error
if err != nil {
h.log.Error(
"failed to count remaining database targets",
"webhook_id", webhookID,
"error", err,
)
return
}
if remaining > 0 {
return
}
h.evictArchiveWriter(webhookID)
}
// HandleSourceLogs shows the request/response logs for a
// webhook.
func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
@@ -749,7 +589,7 @@ func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
}
data := map[string]any{
tmplKeyWebhook: &webhook,
"Webhook": webhook,
"Events": evts,
"Page": page,
"TotalPages": totalPages,
@@ -884,8 +724,10 @@ func (h *Handlers) HandleEntrypointCreate() http.HandlerFunc {
return
}
// The body size cap is enforced by the MaxBodySize
// middleware, which runs before CSRF parses the form.
r.Body = http.MaxBytesReader(
w, r.Body, 1<<maxBodyShift,
)
err = r.ParseForm()
if err != nil {
http.Error(
@@ -942,8 +784,10 @@ func (h *Handlers) HandleTargetCreate() http.HandlerFunc {
return
}
// The body size cap is enforced by the MaxBodySize
// middleware, which runs before CSRF parses the form.
r.Body = http.MaxBytesReader(
w, r.Body, 1<<maxBodyShift,
)
err = r.ParseForm()
if err != nil {
http.Error(
@@ -963,13 +807,14 @@ func (h *Handlers) processTargetCreate(
r *http.Request,
webhook database.Webhook,
) {
// The body size cap is enforced by the MaxBodySize middleware,
// which runs before CSRF parses the form.
r.Body = http.MaxBytesReader(
w, r.Body, 1<<maxBodyShift,
)
name := r.FormValue("name")
targetType := database.TargetType(r.FormValue("type"))
targetURL := r.FormValue("url")
maxRetriesStr := r.FormValue("max_retries")
expiry := r.FormValue("expiry")
if name == "" {
http.Error(
@@ -989,7 +834,7 @@ func (h *Handlers) processTargetCreate(
}
configJSON, err := h.buildTargetConfig(
w, r, targetType, targetURL, expiry,
w, r, targetType, targetURL,
)
if err != nil {
return
@@ -1047,28 +892,18 @@ func parseNonNegativeInt(s string) int {
}
// buildTargetConfig builds the JSON config string for a target.
// The expiry form value is read by the caller (which bounds the
// request body) and applies to database targets only.
func (h *Handlers) buildTargetConfig(
w http.ResponseWriter,
r *http.Request,
targetType database.TargetType,
targetURL, expiry string,
targetURL string,
) (string, error) {
switch targetType {
case database.TargetTypeHTTP:
return h.buildURLTargetConfig(
w, r, targetURL, "url",
"URL is required for HTTP targets",
)
return h.buildHTTPTargetConfig(w, r, targetURL)
case database.TargetTypeSlack:
return h.buildURLTargetConfig(
w, r, targetURL, "webhookUrl",
"Webhook URL is required for Slack targets",
)
case database.TargetTypeDatabase:
return h.buildDatabaseTargetConfig(w, expiry)
case database.TargetTypeLog:
return h.buildSlackTargetConfig(w, r, targetURL)
case database.TargetTypeDatabase, database.TargetTypeLog:
return "", nil
default:
http.Error(
@@ -1080,18 +915,16 @@ func (h *Handlers) buildTargetConfig(
}
}
// buildURLTargetConfig builds config JSON for a target whose
// configuration is a single SSRF-validated URL stored under
// configKey. missingMsg is the error shown when no URL is given.
func (h *Handlers) buildURLTargetConfig(
// buildHTTPTargetConfig builds config JSON for an HTTP target.
func (h *Handlers) buildHTTPTargetConfig(
w http.ResponseWriter,
r *http.Request,
targetURL, configKey, missingMsg string,
targetURL string,
) (string, error) {
if targetURL == "" {
http.Error(
w,
missingMsg,
"URL is required for HTTP targets",
http.StatusBadRequest,
)
@@ -1116,7 +949,7 @@ func (h *Handlers) buildURLTargetConfig(
return "", err
}
cfg := map[string]any{configKey: targetURL}
cfg := map[string]any{"url": targetURL}
configBytes, err := json.Marshal(cfg)
if err != nil {
@@ -1131,33 +964,41 @@ func (h *Handlers) buildURLTargetConfig(
return string(configBytes), nil
}
// buildDatabaseTargetConfig builds config JSON for a database
// (archive) target. The optional expiry (a form value read by
// the caller, which bounds the request body) is validated here,
// at creation time, so an unparseable value is rejected with a
// 400 instead of failing every subsequent delivery. An empty
// expiry yields an empty config (the keep-forever default).
func (h *Handlers) buildDatabaseTargetConfig(
// buildSlackTargetConfig builds config JSON for a Slack target.
func (h *Handlers) buildSlackTargetConfig(
w http.ResponseWriter,
expiry string,
r *http.Request,
targetURL string,
) (string, error) {
expiry = strings.TrimSpace(expiry)
if expiry == "" {
return "", nil
}
err := delivery.ValidateArchiveExpiry(expiry)
if err != nil {
if targetURL == "" {
http.Error(
w,
"Invalid archive expiry: "+err.Error(),
"Webhook URL is required for Slack targets",
http.StatusBadRequest,
)
return "", errMissingURL
}
err := delivery.ValidateTargetURL(
r.Context(), targetURL,
)
if err != nil {
h.log.Warn(
"target URL blocked by SSRF protection",
"url", targetURL,
"error", err,
)
http.Error(
w,
"Invalid target URL: "+err.Error(),
http.StatusBadRequest,
)
return "", err
}
cfg := map[string]any{"expiry": expiry}
cfg := map[string]any{"webhookUrl": targetURL}
configBytes, err := json.Marshal(cfg)
if err != nil {
@@ -1177,31 +1018,23 @@ func (h *Handlers) HandleEntrypointDelete() http.HandlerFunc {
return h.deleteChildResource(
"entrypointID", &database.Entrypoint{},
"failed to delete entrypoint",
nil,
)
}
// HandleTargetDelete handles deleting a target. Deleting the
// last database target of a webhook leaves its archive writer
// with nothing to write, so the writer is evicted and its
// handle closed; the archive file is left on disk.
// HandleTargetDelete handles deleting a target.
func (h *Handlers) HandleTargetDelete() http.HandlerFunc {
return h.deleteChildResource(
"targetID", &database.Target{},
"failed to delete target",
h.evictArchiveWriterIfUnused,
)
}
// deleteChildResource returns a handler that deletes a child
// resource (entrypoint or target) belonging to a webhook. The
// optional afterDelete hook runs with the webhook's id once the
// delete has succeeded, before the redirect.
// resource (entrypoint or target) belonging to a webhook.
func (h *Handlers) deleteChildResource(
idParam string,
model any,
errMsg string,
afterDelete func(webhookID string),
) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
userID, ok := h.getUserID(r)
@@ -1241,10 +1074,6 @@ func (h *Handlers) deleteChildResource(
return
}
if afterDelete != nil {
afterDelete(webhook.ID)
}
http.Redirect(
w, r,
"/source/"+webhook.ID,

589
internal/handlers/source_management_test.go
View File

@@ -1,589 +0,0 @@
package handlers_test
import (
"context"
"net/http"
"net/http/httptest"
"net/url"
"strconv"
"strings"
"testing"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
const (
// sourceTestUserID is the session user id used by the webhook
// management tests.
sourceTestUserID = "source-test-user"
// sourceIDParam is the chi URL parameter naming a webhook.
sourceIDParam = "sourceID"
)
// formRequest builds an urlencoded POST to path carrying the given
// cookies, plus any chi URL parameters the handler reads.
func formRequest(
path string,
cookies []*http.Cookie,
form url.Values,
urlParams map[string]string,
) *http.Request {
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost,
path,
strings.NewReader(form.Encode()),
)
req.Header.Set(
"Content-Type", "application/x-www-form-urlencoded",
)
for _, c := range cookies {
req.AddCookie(c)
}
rctx := chi.NewRouteContext()
for k, v := range urlParams {
rctx.URLParams.Add(k, v)
}
return req.WithContext(
context.WithValue(req.Context(), chi.RouteCtxKey, rctx),
)
}
// getRequest builds a GET to path carrying the given cookies, plus any
// chi URL parameters the handler reads.
func getRequest(
t *testing.T,
path string,
cookies []*http.Cookie,
urlParams map[string]string,
) *http.Request {
t.Helper()
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, path, nil,
)
for _, c := range cookies {
req.AddCookie(c)
}
rctx := chi.NewRouteContext()
for k, v := range urlParams {
rctx.URLParams.Add(k, v)
}
return req.WithContext(
context.WithValue(req.Context(), chi.RouteCtxKey, rctx),
)
}
// submitCreate posts the webhook creation form with the given
// retention_days value (omitted entirely when retention is nil) and
// returns the recorder.
func submitCreate(
t *testing.T,
h *handlers.Handlers,
cookies []*http.Cookie,
name string,
retention *string,
) *httptest.ResponseRecorder {
t.Helper()
form := url.Values{}
form.Set("name", name)
if retention != nil {
form.Set("retention_days", *retention)
}
req := formRequest("/sources/new", cookies, form, nil)
w := httptest.NewRecorder()
h.HandleSourceCreateSubmit().ServeHTTP(w, req)
return w
}
// onlyWebhook loads the single webhook belonging to the test user.
func onlyWebhook(
t *testing.T,
db *database.Database,
) database.Webhook {
t.Helper()
var webhooks []database.Webhook
require.NoError(
t,
db.DB().Where("user_id = ?", sourceTestUserID).
Find(&webhooks).Error,
)
require.Len(t, webhooks, 1)
return webhooks[0]
}
// seedWebhookWithRetention inserts a webhook owned by the test user
// with an exact stored retention value, bypassing Webhook.BeforeSave
// via a column-level update so that legacy rows can be planted too.
func seedWebhookWithRetention(
t *testing.T,
db *database.Database,
retentionDays int,
) database.Webhook {
t.Helper()
wh := &database.Webhook{
UserID: sourceTestUserID,
Name: "seeded",
RetentionDays: retentionDays,
}
require.NoError(
t,
db.DB().Omit(clause.Associations).Create(wh).Error,
)
require.NoError(
t,
db.DB().Model(wh).
Update("retention_days", retentionDays).Error,
)
wh.RetentionDays = retentionDays
return *wh
}
// storedRetentionDays reads the retention_days column for a webhook.
func storedRetentionDays(
t *testing.T,
db *database.Database,
id string,
) int {
t.Helper()
var got int
require.NoError(
t,
db.DB().Model(&database.Webhook{}).
Where("id = ?", id).
Pluck("retention_days", &got).Error,
)
return got
}
// sourceTestEnv bundles the handler, session, and database a webhook
// management test drives.
type sourceTestEnv struct {
handlers *handlers.Handlers
db *database.Database
cookies []*http.Cookie
}
func setupSourceTest(t *testing.T) *sourceTestEnv {
t.Helper()
var h *handlers.Handlers
var sess *session.Session
var db *database.Database
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
return &sourceTestEnv{
handlers: h,
db: db,
cookies: authenticatedCookies(
t, sess, sourceTestUserID, "sourceuser",
),
}
}
// TestHandleSourceCreateSubmit_ZeroRetentionPersistsForever is the core
// regression test for the bug: the create form's 0 must reach the
// database as the retain-forever sentinel rather than being replaced by
// the column's default of 30.
func TestHandleSourceCreateSubmit_ZeroRetentionPersistsForever(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
zero := "0"
w := submitCreate(t, env.handlers, env.cookies, "forever", &zero)
require.Equal(t, http.StatusSeeOther, w.Code)
wh := onlyWebhook(t, env.db)
assert.Equal(
t,
database.RetentionForeverDays,
storedRetentionDays(t, env.db, wh.ID),
)
assert.True(t, wh.RetainsForever())
}
func TestHandleSourceCreateSubmit_OmittedRetentionUsesDefault(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
w := submitCreate(t, env.handlers, env.cookies, "defaulted", nil)
require.Equal(t, http.StatusSeeOther, w.Code)
wh := onlyWebhook(t, env.db)
assert.Equal(
t,
database.DefaultRetentionDays,
storedRetentionDays(t, env.db, wh.ID),
)
}
// TestHandleSourceCreate_PrefillsDefaultFromConstant keeps the create
// form's pre-filled retention from becoming a third hardcoded copy of
// the 30-day policy.
func TestHandleSourceCreate_PrefillsDefaultFromConstant(t *testing.T) {
t.Parallel()
env := setupSourceTest(t)
w := httptest.NewRecorder()
env.handlers.HandleSourceCreate().ServeHTTP(
w, getRequest(t, "/sources/new", env.cookies, nil),
)
require.Equal(t, http.StatusOK, w.Code)
body := w.Body.String()
assert.Contains(
t, body,
`value="`+strconv.Itoa(database.DefaultRetentionDays)+`"`,
)
assert.NotContains(
t, body, `max="365"`,
"a max below the sentinel would block retain-forever",
)
assert.Contains(t, body, `min="0"`)
}
func TestHandleSourceCreateSubmit_InvalidRetentionIsRejected(
t *testing.T,
) {
t.Parallel()
for _, raw := range []string{"abc", "-1", "3.5"} {
t.Run(raw, func(t *testing.T) {
t.Parallel()
env := setupSourceTest(t)
w := submitCreate(
t, env.handlers, env.cookies, "bad", &raw,
)
assert.Equal(t, http.StatusBadRequest, w.Code)
assert.Contains(
t, w.Body.String(), "Retention must be",
)
var count int64
require.NoError(
t,
env.db.DB().Model(&database.Webhook{}).
Where("user_id = ?", sourceTestUserID).
Count(&count).Error,
)
assert.Zero(
t, count,
"no webhook may be created from a rejected form",
)
})
}
}
// TestHandleSourceCreateSubmit_OverflowingRetentionIsRejected covers
// the data-loss path directly: a finite retention above the largest one
// the reaper's cutoff arithmetic can represent must never reach the
// database, because the sweep would compute a future cutoff and delete
// every event the webhook has.
func TestHandleSourceCreateSubmit_OverflowingRetentionIsRejected(
t *testing.T,
) {
t.Parallel()
tooBig := strconv.Itoa(database.MaxFiniteRetentionDays + 1)
env := setupSourceTest(t)
w := submitCreate(t, env.handlers, env.cookies, "huge", &tooBig)
assert.Equal(t, http.StatusBadRequest, w.Code)
assert.Contains(
t, w.Body.String(),
strconv.Itoa(database.MaxFiniteRetentionDays),
"the form tells the user the actual ceiling",
)
var count int64
require.NoError(
t,
env.db.DB().Model(&database.Webhook{}).
Where("user_id = ?", sourceTestUserID).
Count(&count).Error,
)
assert.Zero(
t, count,
"no webhook may be created from a rejected form",
)
}
// TestHandleSourceCreateSubmit_SentinelIsAcceptedAsForever guards the
// boundary between "too large to represent" and "retain forever": the
// sentinel is above MaxFiniteRetentionDays, but it is the value the
// edit form pre-fills, so it must be accepted rather than rejected as
// out of range.
func TestHandleSourceCreateSubmit_SentinelIsAcceptedAsForever(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
sentinel := strconv.Itoa(database.RetentionForeverDays)
w := submitCreate(t, env.handlers, env.cookies, "forever", &sentinel)
require.Equal(t, http.StatusSeeOther, w.Code)
wh := onlyWebhook(t, env.db)
assert.Equal(
t,
database.RetentionForeverDays,
storedRetentionDays(t, env.db, wh.ID),
)
}
// TestHandleSourceCreateSubmit_RejectedFormKeepsUserInput checks that a
// validation failure hands the user's typing back, matching what the
// edit form already does. Losing a long description to a mistyped
// retention value is the kind of thing that makes people give up on a
// form.
func TestHandleSourceCreateSubmit_RejectedFormKeepsUserInput(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
const (
name = "kept-name"
description = "a description worth not losing"
)
form := url.Values{}
form.Set("name", name)
form.Set("description", description)
form.Set("retention_days", "nonsense")
req := formRequest("/sources/new", env.cookies, form, nil)
w := httptest.NewRecorder()
env.handlers.HandleSourceCreateSubmit().ServeHTTP(w, req)
require.Equal(t, http.StatusBadRequest, w.Code)
body := w.Body.String()
assert.Contains(t, body, `value="`+name+`"`)
assert.Contains(t, body, description)
}
// submitEdit posts the webhook edit form for the given webhook.
func submitEdit(
t *testing.T,
env *sourceTestEnv,
wh database.Webhook,
retention string,
) *httptest.ResponseRecorder {
t.Helper()
form := url.Values{}
form.Set("name", wh.Name)
form.Set("description", wh.Description)
form.Set("retention_days", retention)
req := formRequest(
"/source/"+wh.ID+"/edit",
env.cookies,
form,
map[string]string{sourceIDParam: wh.ID},
)
w := httptest.NewRecorder()
env.handlers.HandleSourceEditSubmit().ServeHTTP(w, req)
return w
}
func TestHandleSourceEditSubmit_ZeroRetentionPersistsForever(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
wh := seedWebhookWithRetention(
t, env.db, database.DefaultRetentionDays,
)
w := submitEdit(t, env, wh, "0")
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Equal(
t,
database.RetentionForeverDays,
storedRetentionDays(t, env.db, wh.ID),
)
}
func TestHandleSourceEditSubmit_InvalidRetentionIsRejected(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
wh := seedWebhookWithRetention(
t, env.db, database.DefaultRetentionDays,
)
w := submitEdit(t, env, wh, "not-a-number")
assert.Equal(t, http.StatusBadRequest, w.Code)
assert.Contains(t, w.Body.String(), "Retention must be")
assert.Equal(
t,
database.DefaultRetentionDays,
storedRetentionDays(t, env.db, wh.ID),
"a rejected form must not change the stored retention",
)
}
func TestHandleSourceEditSubmit_EmptyRetentionLeavesValueUnchanged(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
wh := seedWebhookWithRetention(t, env.db, 7)
w := submitEdit(t, env, wh, "")
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Equal(t, 7, storedRetentionDays(t, env.db, wh.ID))
}
// TestSourceEditForm_ForeverWebhookRoundTrips walks the exact path that
// the removed max="365" cap used to break: render the edit form for a
// retain-forever webhook, confirm the pre-filled sentinel is not capped
// by browser validation, then submit that pre-filled value straight
// back and confirm the retention policy survives untouched.
func TestSourceEditForm_ForeverWebhookRoundTrips(t *testing.T) {
t.Parallel()
env := setupSourceTest(t)
wh := seedWebhookWithRetention(
t, env.db, database.RetentionForeverDays,
)
req := getRequest(
t, "/source/"+wh.ID+"/edit", env.cookies,
map[string]string{sourceIDParam: wh.ID},
)
w := httptest.NewRecorder()
env.handlers.HandleSourceEdit().ServeHTTP(w, req)
require.Equal(t, http.StatusOK, w.Code)
sentinel := strconv.Itoa(database.RetentionForeverDays)
body := w.Body.String()
assert.Contains(
t, body, `value="`+sentinel+`"`,
"the edit form pre-fills the stored retention",
)
assert.NotContains(
t, body, `max="365"`,
"a max below the sentinel would block saving any edit",
)
// "Currently forever." is the rendered RetentionLabel, not the
// static hint below the input, which says "Enter 0 to retain events
// forever." A bare Contains of "forever" would pass for any
// webhook and would assert nothing about this one.
assert.Contains(
t, body, "Currently forever.",
"the form reports this webhook's policy as forever",
)
// Submit the pre-filled value back, exactly as a browser would.
post := submitEdit(t, env, wh, sentinel)
require.Equal(t, http.StatusSeeOther, post.Code)
assert.Equal(
t,
database.RetentionForeverDays,
storedRetentionDays(t, env.db, wh.ID),
)
}
// TestSourceListAndDetail_ShowForeverNotTheSentinelNumber checks that
// the retain-forever value is never rendered to the user as a raw day
// count on either read-only view.
func TestSourceListAndDetail_ShowForeverNotTheSentinelNumber(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
wh := seedWebhookWithRetention(
t, env.db, database.RetentionForeverDays,
)
sentinel := strconv.Itoa(database.RetentionForeverDays)
listW := httptest.NewRecorder()
env.handlers.HandleSourceList().ServeHTTP(
listW, getRequest(t, "/sources", env.cookies, nil),
)
require.Equal(t, http.StatusOK, listW.Code)
assert.Contains(t, listW.Body.String(), "Retention: forever")
assert.NotContains(t, listW.Body.String(), sentinel)
detailW := httptest.NewRecorder()
env.handlers.HandleSourceDetail().ServeHTTP(
detailW,
getRequest(
t, "/source/"+wh.ID, env.cookies,
map[string]string{sourceIDParam: wh.ID},
),
)
require.Equal(t, http.StatusOK, detailW.Code)
assert.Contains(t, detailW.Body.String(), "Retention: forever")
assert.NotContains(t, detailW.Body.String(), sentinel)
}

4
internal/middleware/export_test.go
View File

@@ -32,7 +32,3 @@ func IsClientTLS(r *http.Request) bool {
// LoginRateLimitConst exposes the loginRateLimit constant.
const LoginRateLimitConst = loginRateLimit
// PasswordChangeRateLimitConst exposes the
// passwordChangeRateLimit constant.
const PasswordChangeRateLimitConst = passwordChangeRateLimit

85
internal/middleware/middleware.go
View File

@@ -186,10 +186,6 @@ func (s *Middleware) RequireAuth() func(http.Handler) http.Handler {
return
}
// IsAuthenticated also enforces both session expiry
// deadlines, so an idle-expired or absolutely-expired
// session lands here and is sent back to the login
// page.
if !s.session.IsAuthenticated(sess) {
s.log.Debug(
"auth middleware: unauthenticated request",
@@ -203,26 +199,6 @@ func (s *Middleware) RequireAuth() func(http.Handler) http.Handler {
return
}
// This request authenticated with the session, so it
// counts as activity: push the idle deadline forward.
// This is the only place sessions are refreshed, which
// is what keeps an unauthenticated request from
// extending someone else's session. Touch advances the
// idle clock only -- the absolute cap is untouched --
// and reports false when nothing changed, so most
// requests do not re-issue the cookie. Save before the
// handler runs, while the headers are still ours to
// write.
if s.session.Touch(sess) {
err = s.session.Save(r, w, sess)
if err != nil {
s.log.Error(
"auth middleware: failed to refresh session",
"error", err,
)
}
}
next.ServeHTTP(w, r)
})
}
@@ -309,36 +285,10 @@ func (s *Middleware) NoCache() func(http.Handler) http.Handler {
}
}
// bodyLimitedMethod reports whether the request method carries a
// body that the MaxBodySize middleware should cap.
func bodyLimitedMethod(method string) bool {
return method == http.MethodPost ||
method == http.MethodPut ||
method == http.MethodPatch
}
// MaxBodySize returns middleware that limits the size of
// POST/PUT/PATCH request bodies to maxBytes. It must be registered
// before any middleware that parses the body — notably CSRF, which
// calls r.PostFormValue — so that form parsing happens under this
// cap rather than net/http's 10 MB default.
//
// Two enforcement paths exist, because http.MaxBytesReader alone
// cannot produce a 413: it reports the overflow as an error from
// Read, by which point the body parser downstream has already
// converted that error into its own response.
//
// - Declared oversize: the request announces a Content-Length
// greater than maxBytes. The middleware answers 413 Request
// Entity Too Large immediately and does not call the next
// handler, so neither CSRF nor the endpoint handler runs.
// - Undeclared oversize: the request is chunked (Content-Length
// of -1) or lies about its Content-Length. There is nothing to
// check up front, so http.MaxBytesReader hard-caps the body at
// maxBytes and the request fails downstream — the form parse
// errors out and CSRF rejects it with 403. The response is less
// precise than a 413, but the body is still never buffered
// beyond the cap, which is the property that matters.
// MaxBodySize returns middleware that limits the request body size
// for POST requests. If the body exceeds the given limit in
// bytes, the server returns 413 Request Entity Too Large. This
// prevents clients from sending arbitrarily large form bodies.
func (s *Middleware) MaxBodySize(
maxBytes int64,
) func(http.Handler) http.Handler {
@@ -347,31 +297,14 @@ func (s *Middleware) MaxBodySize(
w http.ResponseWriter,
r *http.Request,
) {
if !bodyLimitedMethod(r.Method) {
next.ServeHTTP(w, r)
return
}
if r.ContentLength > maxBytes {
s.log.Warn(
"request body exceeds limit",
"method", r.Method,
"path", r.URL.Path,
"content_length", r.ContentLength,
"limit", maxBytes,
if r.Method == http.MethodPost ||
r.Method == http.MethodPut ||
r.Method == http.MethodPatch {
r.Body = http.MaxBytesReader(
w, r.Body, maxBytes,
)
http.Error(
w,
"Request Entity Too Large",
http.StatusRequestEntityTooLarge,
)
return
}
r.Body = http.MaxBytesReader(w, r.Body, maxBytes)
next.ServeHTTP(w, r)
})
}

418
internal/middleware/middleware_test.go
View File

@@ -3,14 +3,11 @@ package middleware_test
import (
"context"
"encoding/base64"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"time"
"github.com/gorilla/sessions"
"github.com/stretchr/testify/assert"
@@ -31,22 +28,6 @@ func testMiddleware(
) (*middleware.Middleware, *session.Session) {
t.Helper()
m, s, _ := testMiddlewareWithSessionClock(t, env, 0, nil)
return m, s
}
// testMiddlewareWithSessionClock is testMiddleware with a
// configurable session idle timeout and a manually advanced clock,
// for the session-expiry tests. A nil clock uses the real one.
func testMiddlewareWithSessionClock(
t *testing.T,
env string,
idleTimeout time.Duration,
clock *fakeClock,
) (*middleware.Middleware, *session.Session, *fakeClock) {
t.Helper()
log := slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
@@ -54,7 +35,6 @@ func testMiddlewareWithSessionClock(
cfg := &config.Config{
Environment: env,
SessionIdleTimeout: idleTimeout,
}
// Create a real session manager with a known key
@@ -73,40 +53,11 @@ func testMiddlewareWithSessionClock(
SameSite: http.SameSiteLaxMode,
}
var now func() time.Time
if clock != nil {
now = clock.Now
}
sessManager := session.NewForTest(store, cfg, log, key, now)
sessManager := session.NewForTest(store, cfg, log, key)
m := middleware.NewForTest(log, cfg, sessManager)
return m, sessManager, clock
}
// fakeClock is a manually advanced clock, so session expiry can be
// tested without sleeping.
type fakeClock struct {
t time.Time
}
func (c *fakeClock) Now() time.Time {
return c.t
}
func (c *fakeClock) Advance(d time.Duration) {
c.t = c.t.Add(d)
}
// newFakeClock returns a clock started at a fixed instant.
func newFakeClock() *fakeClock {
return &fakeClock{
t: time.Date(
2026, time.January, 2, 3, 4, 5, 0, time.UTC,
),
}
return m, sessManager
}
// --- Logging Middleware Tests ---
@@ -436,181 +387,6 @@ func TestRequireAuth_UnauthenticatedSession_RedirectsToLogin(
assert.Equal(t, "/pages/login", w.Header().Get("Location"))
}
// --- RequireAuth Session Expiry Tests ---
// loginCookies authenticates a new session and returns the cookies
// a browser would then send back.
func loginCookies(
t *testing.T,
sessManager *session.Session,
) []*http.Cookie {
t.Helper()
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/login", nil)
w := httptest.NewRecorder()
sess, err := sessManager.Get(req)
require.NoError(t, err)
sessManager.SetUser(sess, "user-123", "testuser")
require.NoError(t, sessManager.Save(req, w, sess))
cookies := w.Result().Cookies()
require.NotEmpty(t, cookies, "session cookie should be set")
return cookies
}
// runAuthed sends a request carrying cookies through RequireAuth
// and reports whether the protected handler ran, plus the response.
func runAuthed(
t *testing.T,
m *middleware.Middleware,
cookies []*http.Cookie,
) (bool, *httptest.ResponseRecorder) {
t.Helper()
var called bool
handler := m.RequireAuth()(http.HandlerFunc(
func(_ http.ResponseWriter, _ *http.Request) {
called = true
},
))
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet, "/dashboard", nil,
)
for _, c := range cookies {
req.AddCookie(c)
}
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
return called, w
}
// sessionCookies filters a response's cookies down to the session
// cookie, so tests can tell whether the session was re-issued.
func sessionCookies(
w *httptest.ResponseRecorder,
) []*http.Cookie {
var out []*http.Cookie
for _, c := range w.Result().Cookies() {
if c.Name == session.SessionName {
out = append(out, c)
}
}
return out
}
func TestRequireAuth_IdleExpiredSession_RedirectsToLogin(
t *testing.T,
) {
t.Parallel()
idle := time.Hour
m, sessManager, clock := testMiddlewareWithSessionClock(
t, config.EnvironmentDev, idle, newFakeClock(),
)
cookies := loginCookies(t, sessManager)
clock.Advance(idle)
called, w := runAuthed(t, m, cookies)
assert.False(
t, called,
"handler should not run for an idle-expired session",
)
assert.Equal(t, http.StatusSeeOther, w.Code)
assert.Equal(t, "/pages/login", w.Header().Get("Location"))
assert.Empty(
t, sessionCookies(w),
"an expired session must not be refreshed",
)
}
func TestRequireAuth_RefreshesIdleDeadlineOnActivity(
t *testing.T,
) {
t.Parallel()
idle := time.Hour
m, sessManager, clock := testMiddlewareWithSessionClock(
t, config.EnvironmentDev, idle, newFakeClock(),
)
cookies := loginCookies(t, sessManager)
// Activity halfway through the idle window.
clock.Advance(idle / 2)
called, w := runAuthed(t, m, cookies)
require.True(t, called, "handler should run while valid")
refreshed := sessionCookies(w)
require.NotEmpty(
t, refreshed,
"activity should re-issue the session cookie",
)
// Past the original deadline. The refreshed cookie is still
// good; the original one is not.
clock.Advance(idle - time.Second)
calledRefreshed, _ := runAuthed(t, m, refreshed)
assert.True(
t, calledRefreshed,
"refreshed session should outlive the original deadline",
)
calledStale, staleW := runAuthed(t, m, cookies)
assert.False(
t, calledStale,
"the pre-refresh cookie carries the old idle deadline",
)
assert.Equal(t, http.StatusSeeOther, staleW.Code)
}
func TestRequireAuth_UnauthenticatedRequestDoesNotRefresh(
t *testing.T,
) {
t.Parallel()
m, sessManager, _ := testMiddlewareWithSessionClock(
t, config.EnvironmentDev, time.Hour, newFakeClock(),
)
// A session cookie that exists but was never authenticated.
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/setup", nil)
setupW := httptest.NewRecorder()
sess, err := sessManager.Get(req)
require.NoError(t, err)
require.NoError(t, sessManager.Save(req, setupW, sess))
cookies := setupW.Result().Cookies()
require.NotEmpty(t, cookies)
called, w := runAuthed(t, m, cookies)
assert.False(t, called)
assert.Empty(
t, sessionCookies(w),
"an unauthenticated request must not stamp the session",
)
}
// --- NoCache Middleware Tests ---
func TestNoCache_SetsHeaders(t *testing.T) {
@@ -650,153 +426,6 @@ func TestNoCache_SetsHeaders(t *testing.T) {
)
}
// --- MaxBodySize Middleware Tests ---
const testBodyLimit int64 = 64
// maxBodySizeHandler wraps a sentinel handler in MaxBodySize with
// testBodyLimit. The sentinel records whether it ran and how much of
// the body it managed to read, so tests can distinguish "never
// reached" from "reached but truncated".
type maxBodySizeResult struct {
called bool
read int
readErr error
response *httptest.ResponseRecorder
}
func runMaxBodySize(
t *testing.T,
req *http.Request,
) *maxBodySizeResult {
t.Helper()
m, _ := testMiddleware(t, config.EnvironmentDev)
res := &maxBodySizeResult{response: httptest.NewRecorder()}
handler := m.MaxBodySize(testBodyLimit)(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
res.called = true
body, err := io.ReadAll(r.Body)
res.read = len(body)
res.readErr = err
w.WriteHeader(http.StatusOK)
},
))
handler.ServeHTTP(res.response, req)
return res
}
// postWithBody builds a POST request whose Content-Length is
// accurate for the given payload size.
func postWithBody(size int) *http.Request {
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost, "/pages/login",
strings.NewReader(strings.Repeat("a", size)),
)
req.Header.Set(
"Content-Type", "application/x-www-form-urlencoded",
)
return req
}
func TestMaxBodySize_DeclaredOversize_413AndHandlerNotReached(
t *testing.T,
) {
t.Parallel()
res := runMaxBodySize(t, postWithBody(int(testBodyLimit)+1))
assert.False(
t, res.called,
"handler must not be reached for an oversized body",
)
assert.Equal(
t, http.StatusRequestEntityTooLarge, res.response.Code,
)
}
func TestMaxBodySize_AtLimit_PassesThrough(t *testing.T) {
t.Parallel()
res := runMaxBodySize(t, postWithBody(int(testBodyLimit)))
assert.True(
t, res.called,
"handler should be reached for a body at the limit",
)
require.NoError(t, res.readErr)
assert.Equal(t, int(testBodyLimit), res.read)
assert.Equal(t, http.StatusOK, res.response.Code)
}
func TestMaxBodySize_UnderLimit_PassesThrough(t *testing.T) {
t.Parallel()
res := runMaxBodySize(t, postWithBody(1))
assert.True(t, res.called)
require.NoError(t, res.readErr)
assert.Equal(t, 1, res.read)
assert.Equal(t, http.StatusOK, res.response.Code)
}
func TestMaxBodySize_GetWithOversizeBody_NotCapped(t *testing.T) {
t.Parallel()
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet, "/pages/login",
strings.NewReader(
strings.Repeat("a", int(testBodyLimit)+1),
),
)
res := runMaxBodySize(t, req)
assert.True(
t, res.called,
"GET requests are not subject to the POST body cap",
)
require.NoError(t, res.readErr)
assert.Equal(t, int(testBodyLimit)+1, res.read)
}
// TestMaxBodySize_UndeclaredOversize_TruncatedAtCap covers the
// chunked / lying-Content-Length case: there is nothing to check up
// front, so the request reaches the handler but MaxBytesReader
// hard-caps the body and the read fails at the limit.
func TestMaxBodySize_UndeclaredOversize_TruncatedAtCap(
t *testing.T,
) {
t.Parallel()
req := postWithBody(int(testBodyLimit) + 1)
// Simulate a chunked request: no declared length.
req.ContentLength = -1
res := runMaxBodySize(t, req)
assert.True(
t, res.called,
"an undeclared oversize body cannot be rejected up front",
)
require.Error(
t, res.readErr,
"reading past the cap must fail",
)
assert.Equal(
t, int(testBodyLimit), res.read,
"the handler must not see more than the cap",
)
}
// --- Helper Tests ---
func TestIpFromHostPort(t *testing.T) {
@@ -850,18 +479,13 @@ func metricsAuthMiddleware(
store := sessions.NewCookieStore(key)
store.Options = &sessions.Options{Path: "/", MaxAge: 86400}
sessManager := session.NewForTest(store, cfg, log, key, nil)
sessManager := session.NewForTest(store, cfg, log, key)
return middleware.NewForTest(log, cfg, sessManager)
}
// runMetricsAuthRequest sends a GET /metrics request with the
// given basic-auth password through MetricsAuth and reports
// whether the wrapped handler ran plus the recorded response.
func runMetricsAuthRequest(
t *testing.T, password string,
) (bool, *httptest.ResponseRecorder) {
t.Helper()
func TestMetricsAuth_ValidCredentials(t *testing.T) {
t.Parallel()
m := metricsAuthMiddleware(t)
@@ -879,20 +503,12 @@ func runMetricsAuthRequest(
context.Background(),
http.MethodGet, "/metrics", nil,
)
req.SetBasicAuth("admin", password)
req.SetBasicAuth("admin", "secret")
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
return called, w
}
func TestMetricsAuth_ValidCredentials(t *testing.T) {
t.Parallel()
called, w := runMetricsAuthRequest(t, "secret")
assert.True(
t, called,
"handler should be called with valid basic auth",
@@ -903,7 +519,27 @@ func TestMetricsAuth_ValidCredentials(t *testing.T) {
func TestMetricsAuth_InvalidCredentials(t *testing.T) {
t.Parallel()
called, w := runMetricsAuthRequest(t, "wrong-password")
m := metricsAuthMiddleware(t)
var called bool
handler := m.MetricsAuth()(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
called = true
w.WriteHeader(http.StatusOK)
},
))
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet, "/metrics", nil,
)
req.SetBasicAuth("admin", "wrong-password")
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
assert.False(
t, called,

196
internal/middleware/ratelimit.go
View File

@@ -2,9 +2,6 @@ package middleware
import (
"net/http"
"net/netip"
"slices"
"strings"
"time"
"github.com/go-chi/httprate"
@@ -18,175 +15,37 @@ const (
// loginRateInterval is the time window for the rate limit.
loginRateInterval = 1 * time.Minute
// passwordChangeRateLimit is the maximum number of password
// change attempts per interval. Each attempt verifies the
// current password, so the endpoint must be rate-limited
// like any other password-based authentication endpoint.
passwordChangeRateLimit = 5
// passwordChangeRateInterval is the time window for the
// password change rate limit.
passwordChangeRateInterval = 1 * time.Minute
// receiverRateInterval is the time window for the webhook
// receiver rate limit. The configured limit is expressed in
// requests per minute.
receiverRateInterval = 1 * time.Minute
)
// normalizeAddr strips the IPv4-in-IPv6 wrapper and any zone from
// addr so that comparisons and bucket keys are canonical.
func normalizeAddr(addr netip.Addr) netip.Addr {
return addr.Unmap().WithZone("")
}
// isTrustedProxy reports whether addr belongs to a network the
// operator listed in TRUSTED_PROXIES. The list is empty by default,
// so by default nothing is trusted.
func (m *Middleware) isTrustedProxy(addr netip.Addr) bool {
for _, prefix := range m.params.Config.TrustedProxies {
if prefix.Contains(addr) {
return true
}
}
return false
}
// forwardedClientAddr returns the client address named by this
// request's forwarded headers. It is consulted only for requests
// whose direct peer is a trusted proxy.
//
// True-Client-IP and X-Real-IP are single-valued, and a trusted
// proxy is expected to overwrite whatever the client sent, so they
// are taken as given. X-Forwarded-For is a chain the client can
// prepend to, so it is walked right to left and the first hop that
// is not itself a trusted proxy wins: entries the client inserted
// sit to the left of the proxies' own appends and cannot be picked
// while the chain is intact.
func (m *Middleware) forwardedClientAddr(
r *http.Request,
) (netip.Addr, bool) {
for _, header := range []string{"True-Client-IP", "X-Real-IP"} {
addr, err := netip.ParseAddr(
strings.TrimSpace(r.Header.Get(header)),
)
if err == nil {
return normalizeAddr(addr), true
}
}
hops := strings.Split(
strings.Join(r.Header.Values("X-Forwarded-For"), ","), ",",
)
for _, hop := range slices.Backward(hops) {
addr, err := netip.ParseAddr(strings.TrimSpace(hop))
if err != nil {
continue
}
if addr = normalizeAddr(addr); !m.isTrustedProxy(addr) {
return addr, true
}
}
return netip.Addr{}, false
}
// rateLimitKey is the client identity every rate limiter in this
// package buckets on. Forwarded headers are honoured only when the
// direct peer (RemoteAddr) is inside the configured trusted-proxy
// set; otherwise the peer address itself is the key. Without that
// gate any client could mint a fresh bucket per request, or starve
// another client's bucket, by picking an X-Forwarded-For value —
// which makes every limit here decorative against a deliberate
// attacker.
func (m *Middleware) rateLimitKey(r *http.Request) (string, error) {
return m.clientKey(r), nil
}
// clientKey computes the bucket key described on rateLimitKey.
func (m *Middleware) clientKey(r *http.Request) string {
peer, err := netip.ParseAddr(ipFromHostPort(r.RemoteAddr))
if err != nil {
// Not an address we can reason about; key on the raw
// value rather than collapsing such peers into one
// shared bucket.
return r.RemoteAddr
}
peer = normalizeAddr(peer)
if !m.isTrustedProxy(peer) {
return peer.String()
}
if addr, ok := m.forwardedClientAddr(r); ok {
return addr.String()
}
return peer.String()
}
// tooManyRequests returns the 429 handler shared by every limiter:
// it logs the rejection with logMessage and answers with
// responseMessage. httprate adds the Retry-After header (RFC 6585).
func (m *Middleware) tooManyRequests(
logMessage, responseMessage string,
) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
m.log.Warn(logMessage, "path", r.URL.Path)
http.Error(w, responseMessage, http.StatusTooManyRequests)
}
}
// LoginRateLimit returns middleware that enforces per-IP rate
// limiting on login attempts using go-chi/httprate. Only POST
// requests are rate-limited; GET requests (rendering the login
// form) pass through unaffected. When the rate limit is exceeded,
// a 429 Too Many Requests response is returned. Clients are
// identified by rateLimitKey.
// a 429 Too Many Requests response is returned. IP extraction
// honours X-Forwarded-For, X-Real-IP, and True-Client-IP headers
// for reverse-proxy setups.
func (m *Middleware) LoginRateLimit() func(http.Handler) http.Handler {
return m.postRateLimit(
limiter := httprate.Limit(
loginRateLimit,
loginRateInterval,
"login rate limit exceeded",
"Too many login attempts. Please try again later.",
httprate.WithKeyFuncs(httprate.KeyByRealIP),
httprate.WithLimitHandler(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
m.log.Warn("login rate limit exceeded",
"path", r.URL.Path,
)
}
// PasswordChangeRateLimit returns middleware that enforces
// per-IP rate limiting on password change attempts. The change
// endpoint verifies the current password, so without a limit a
// stolen session could be used to brute-force it; the limit
// matches the login endpoint's.
func (m *Middleware) PasswordChangeRateLimit() func(http.Handler) http.Handler {
return m.postRateLimit(
passwordChangeRateLimit,
passwordChangeRateInterval,
"password change rate limit exceeded",
"Too many password change attempts. "+
http.Error(
w,
"Too many login attempts. "+
"Please try again later.",
http.StatusTooManyRequests,
)
}
// postRateLimit builds middleware that enforces a per-IP rate
// limit on POST requests only; all other methods pass through
// unaffected. Requests over the limit receive a 429 with the
// given response message, and each rejection is logged with the
// given log message. Clients are identified by rateLimitKey.
func (m *Middleware) postRateLimit(
limit int,
interval time.Duration,
logMessage, responseMessage string,
) func(http.Handler) http.Handler {
limiter := httprate.Limit(
limit,
interval,
httprate.WithKeyFuncs(m.rateLimitKey),
httprate.WithLimitHandler(
m.tooManyRequests(logMessage, responseMessage),
),
},
)),
)
return func(next http.Handler) http.Handler {
@@ -196,7 +55,8 @@ func (m *Middleware) postRateLimit(
w http.ResponseWriter,
r *http.Request,
) {
// Only rate-limit POST requests.
// Only rate-limit POST requests (actual login
// attempts)
if r.Method != http.MethodPost {
next.ServeHTTP(w, r)
@@ -213,19 +73,31 @@ func (m *Middleware) postRateLimit(
// path (the path contains the entrypoint UUID, so each sender
// is limited per entrypoint without affecting other senders or
// other entrypoints). The limit is Config.ReceiverRateLimit
// requests per minute. Requests over the limit receive a 429.
// Clients are identified by rateLimitKey.
// requests per minute. Requests over the limit receive a 429;
// httprate adds the Retry-After header (RFC 6585). IP
// extraction honours X-Forwarded-For, X-Real-IP, and
// True-Client-IP headers for reverse-proxy setups.
func (m *Middleware) ReceiverRateLimit() func(http.Handler) http.Handler {
return httprate.Limit(
m.params.Config.ReceiverRateLimit,
receiverRateInterval,
httprate.WithKeyFuncs(
m.rateLimitKey,
httprate.KeyByRealIP,
httprate.KeyByEndpoint,
),
httprate.WithLimitHandler(m.tooManyRequests(
httprate.WithLimitHandler(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
m.log.Warn(
"webhook receiver rate limit exceeded",
"Too many requests. Please slow down.",
"path", r.URL.Path,
)
http.Error(
w,
"Too many requests. "+
"Please slow down.",
http.StatusTooManyRequests,
)
},
)),
)
}

353
internal/middleware/ratelimit_test.go
View File

@@ -2,11 +2,9 @@ package middleware_test
import (
"context"
"fmt"
"log/slog"
"net/http"
"net/http/httptest"
"net/netip"
"os"
"testing"
@@ -50,20 +48,14 @@ func TestLoginRateLimit_AllowsGET(t *testing.T) {
assert.Equal(t, 20, callCount)
}
// runPostLimitTest exercises a POST-only rate limit middleware:
// the first limit POSTs to path from ip must pass, and the next
// one must be rejected with 429 without reaching the handler.
func runPostLimitTest(
t *testing.T,
mw func(http.Handler) http.Handler,
limit int,
path, ip string,
) {
t.Helper()
func TestLoginRateLimit_LimitsPOST(t *testing.T) {
t.Parallel()
m, _ := testMiddleware(t, config.EnvironmentDev)
var callCount int
handler := mw(http.HandlerFunc(
handler := m.LoginRateLimit()(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
callCount++
@@ -71,13 +63,13 @@ func runPostLimitTest(
},
))
// The first limit POST requests should succeed
for i := range limit {
// First loginRateLimit POST requests should succeed
for i := range middleware.LoginRateLimitConst {
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost, path, nil,
http.MethodPost, "/pages/login", nil,
)
req.RemoteAddr = ip
req.RemoteAddr = "10.0.0.1:12345"
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
@@ -91,9 +83,9 @@ func runPostLimitTest(
// Next POST should be rate-limited
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost, path, nil,
http.MethodPost, "/pages/login", nil,
)
req.RemoteAddr = ip
req.RemoteAddr = "10.0.0.1:12345"
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
@@ -102,35 +94,7 @@ func runPostLimitTest(
t, http.StatusTooManyRequests, w.Code,
"POST after limit should be 429",
)
assert.Equal(t, limit, callCount)
}
func TestLoginRateLimit_LimitsPOST(t *testing.T) {
t.Parallel()
m, _ := testMiddleware(t, config.EnvironmentDev)
runPostLimitTest(
t,
m.LoginRateLimit(),
middleware.LoginRateLimitConst,
"/pages/login",
"10.0.0.1:12345",
)
}
func TestPasswordChangeRateLimit_LimitsPOST(t *testing.T) {
t.Parallel()
m, _ := testMiddleware(t, config.EnvironmentDev)
runPostLimitTest(
t,
m.PasswordChangeRateLimit(),
middleware.PasswordChangeRateLimitConst,
"/user/admin/password",
"10.0.0.2:12345",
)
assert.Equal(t, middleware.LoginRateLimitConst, callCount)
}
func TestLoginRateLimit_IndependentPerIP(t *testing.T) {
@@ -184,22 +148,11 @@ func TestLoginRateLimit_IndependentPerIP(t *testing.T) {
)
}
// okHandler is the terminal handler the limiter middleware wraps
// in these tests: it answers 200 to anything that reaches it.
func okHandler() http.Handler {
return http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
},
)
}
// rateLimitMiddleware builds a Middleware around cfg, whose
// TrustedProxies field is what the rate limit key function gates
// forwarded-header trust on.
func rateLimitMiddleware(
t *testing.T, cfg *config.Config,
) *middleware.Middleware {
// receiverLimitedHandler builds a ReceiverRateLimit-wrapped
// handler with the given per-minute limit.
func receiverLimitedHandler(
t *testing.T, limit int,
) http.Handler {
t.Helper()
log := slog.New(slog.NewTextHandler(
@@ -207,53 +160,17 @@ func rateLimitMiddleware(
&slog.HandlerOptions{Level: slog.LevelDebug},
))
return middleware.NewForTest(log, cfg, nil)
}
// trustedProxies parses CIDR strings for a test Config.
func trustedProxies(cidrs ...string) []netip.Prefix {
prefixes := make([]netip.Prefix, 0, len(cidrs))
for _, cidr := range cidrs {
prefixes = append(prefixes, netip.MustParsePrefix(cidr))
}
return prefixes
}
// postWithHeaders sends one POST to the handler from peer with the
// given headers set and returns the recorder.
func postWithHeaders(
handler http.Handler,
peer, path string,
headers map[string]string,
) *httptest.ResponseRecorder {
req := httptest.NewRequestWithContext(
context.Background(), http.MethodPost, path, nil,
)
req.RemoteAddr = peer
for name, value := range headers {
req.Header.Set(name, value)
}
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
return w
}
// receiverLimitedHandler builds a ReceiverRateLimit-wrapped
// handler with the given per-minute limit and no trusted proxies.
func receiverLimitedHandler(
t *testing.T, limit int,
) http.Handler {
t.Helper()
m := rateLimitMiddleware(
t, &config.Config{ReceiverRateLimit: limit},
m := middleware.NewForTest(
log,
&config.Config{ReceiverRateLimit: limit},
nil,
)
return m.ReceiverRateLimit()(okHandler())
return m.ReceiverRateLimit()(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
},
))
}
// receiverPost sends one POST to the handler from the given IP
@@ -321,221 +238,3 @@ func TestReceiverRateLimit_LimitsPerIPAndPath(t *testing.T) {
"a different client IP must not be affected",
)
}
// TestReceiverRateLimit_CountsEveryMethod proves the receiver
// limit counts non-POST requests too: a GET shares the bucket
// with a POST and is itself rejected once over the limit.
func TestReceiverRateLimit_CountsEveryMethod(t *testing.T) {
t.Parallel()
const (
limit = 2
ip = "7.7.7.7:1234"
path = "/webhook/uuid-c"
)
handler := receiverLimitedHandler(t, limit)
get := func() *httptest.ResponseRecorder {
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, path, nil,
)
req.RemoteAddr = ip
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
return w
}
// One POST plus one GET fill the bucket, so the GET must
// have been counted.
assert.Equal(
t, http.StatusOK, receiverPost(handler, ip, path).Code,
)
assert.Equal(t, http.StatusOK, get().Code)
assert.Equal(
t, http.StatusTooManyRequests, get().Code,
"a GET over the limit must be rate-limited",
)
}
const (
loginPath = "/pages/login"
headerXFF = "X-Forwarded-For"
headerReal = "X-Real-IP"
headerTrue = "True-Client-IP"
)
// TestRateLimitKey_SpoofedForwardedFromUntrustedPeer is the test
// this gating exists for: with no trusted proxies configured (the
// default), a client that rotates a forwarded header on every
// request must stay in one bucket. If forwarded headers were
// trusted unconditionally, each spoofed value would mint a fresh
// bucket and the limit would stop no one.
func TestRateLimitKey_SpoofedForwardedFromUntrustedPeer(
t *testing.T,
) {
t.Parallel()
for _, header := range []string{
headerXFF, headerReal, headerTrue,
} {
t.Run(header, func(t *testing.T) {
t.Parallel()
m := rateLimitMiddleware(t, &config.Config{})
handler := m.LoginRateLimit()(okHandler())
const peer = "203.0.113.9:44444"
for i := range middleware.LoginRateLimitConst {
w := postWithHeaders(
handler, peer, loginPath,
map[string]string{
header: fmt.Sprintf(
"198.51.100.%d", i+1,
),
},
)
assert.Equal(
t, http.StatusOK, w.Code,
"request %d should pass", i,
)
}
w := postWithHeaders(
handler, peer, loginPath,
map[string]string{header: "198.51.100.200"},
)
assert.Equal(
t, http.StatusTooManyRequests, w.Code,
"a spoofed %s from an untrusted peer must "+
"not mint a fresh bucket", header,
)
})
}
}
// TestRateLimitKey_ForwardedHonouredFromTrustedPeer checks the
// other half: when the direct peer is a configured trusted proxy,
// the forwarded client address is what buckets are keyed on, so
// one sender behind the proxy cannot exhaust another's limit.
func TestRateLimitKey_ForwardedHonouredFromTrustedPeer(
t *testing.T,
) {
t.Parallel()
m := rateLimitMiddleware(t, &config.Config{
TrustedProxies: trustedProxies("10.0.0.0/8"),
})
handler := m.LoginRateLimit()(okHandler())
const peer = "10.0.0.1:44444"
first := map[string]string{headerXFF: "198.51.100.7"}
for range middleware.LoginRateLimitConst {
postWithHeaders(handler, peer, loginPath, first)
}
w := postWithHeaders(handler, peer, loginPath, first)
assert.Equal(
t, http.StatusTooManyRequests, w.Code,
"the forwarded client's own bucket must fill up",
)
w = postWithHeaders(
handler, peer, loginPath,
map[string]string{headerXFF: "198.51.100.8"},
)
assert.Equal(
t, http.StatusOK, w.Code,
"a forwarded header from a trusted peer must be honoured",
)
}
// TestRateLimitKey_ChainWalkSkipsClientPrepended covers the
// residual spoofing route behind a trusted proxy: the client
// controls the leftmost X-Forwarded-For entries, so the key is the
// rightmost hop that is not itself trusted. Rotating the prepended
// entry must not create new buckets.
func TestRateLimitKey_ChainWalkSkipsClientPrepended(t *testing.T) {
t.Parallel()
m := rateLimitMiddleware(t, &config.Config{
TrustedProxies: trustedProxies("10.0.0.0/8"),
})
handler := m.LoginRateLimit()(okHandler())
const peer = "10.0.0.1:44444"
chain := func(spoof string) map[string]string {
return map[string]string{
headerXFF: spoof + ", 198.51.100.7, 10.0.0.2",
}
}
for i := range middleware.LoginRateLimitConst {
w := postWithHeaders(
handler, peer, loginPath,
chain(fmt.Sprintf("9.9.9.%d", i+1)),
)
assert.Equal(
t, http.StatusOK, w.Code,
"request %d should pass", i,
)
}
w := postWithHeaders(
handler, peer, loginPath, chain("9.9.9.200"),
)
assert.Equal(
t, http.StatusTooManyRequests, w.Code,
"a client-prepended X-Forwarded-For entry must not "+
"mint a fresh bucket",
)
}
// TestReceiverRateLimit_IgnoresForwardedFromUntrustedPeer proves
// the receiver limiter uses the same gated key function as the
// POST limiters.
func TestReceiverRateLimit_IgnoresForwardedFromUntrustedPeer(
t *testing.T,
) {
t.Parallel()
const (
limit = 3
peer = "203.0.113.10:44444"
path = "/webhook/uuid-d"
)
handler := receiverLimitedHandler(t, limit)
for i := range limit {
w := postWithHeaders(
handler, peer, path,
map[string]string{
headerXFF: fmt.Sprintf(
"198.51.100.%d", i+1,
),
},
)
assert.Equal(
t, http.StatusOK, w.Code,
"request %d should pass", i,
)
}
w := postWithHeaders(
handler, peer, path,
map[string]string{headerXFF: "198.51.100.200"},
)
assert.Equal(
t, http.StatusTooManyRequests, w.Code,
"a spoofed X-Forwarded-For from an untrusted peer must "+
"not mint a fresh receiver bucket",
)
}

36
internal/server/export_test.go
View File

@@ -1,36 +0,0 @@
package server
import (
"log/slog"
"net/http"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/middleware"
)
// MaxFormBodySizeForTest exposes the form body cap so tests can
// build requests that sit exactly at, below, and above it.
const MaxFormBodySizeForTest = maxFormBodySize
// NewRouterForTest builds the real route tree via SetupRoutes with
// the supplied middleware and handlers, bypassing the fx lifecycle
// and the HTTP listener. Tests use it so that route-group middleware
// registration order is exercised exactly as it ships, rather than
// against a hand-rebuilt chain that could drift from routes.go.
func NewRouterForTest(
log *slog.Logger,
cfg *config.Config,
mw *middleware.Middleware,
h *handlers.Handlers,
) http.Handler {
s := &Server{
log: log,
mw: mw,
h: h,
params: ServerParams{Config: cfg},
}
s.SetupRoutes()
return s.router
}

18
internal/server/routes.go
View File

@@ -90,11 +90,9 @@ func (s *Server) setupRoutes() {
func (s *Server) setupPageRoutes() {
s.router.Route("/pages", func(r chi.Router) {
// MaxBodySize must precede CSRF: gorilla/csrf parses the
// form, so the cap has to be installed before it runs.
r.Use(s.mw.MaxBodySize(maxFormBodySize))
r.Use(s.mw.CSRF())
r.Use(s.mw.NoCache())
r.Use(s.mw.MaxBodySize(maxFormBodySize))
r.Group(func(r chi.Router) {
r.Use(s.mw.LoginRateLimit())
@@ -108,39 +106,29 @@ func (s *Server) setupPageRoutes() {
func (s *Server) setupUserRoutes() {
s.router.Route("/user/{username}", func(r chi.Router) {
// MaxBodySize must precede CSRF: gorilla/csrf parses the
// form, so the cap has to be installed before it runs.
r.Use(s.mw.MaxBodySize(maxFormBodySize))
r.Use(s.mw.CSRF())
r.Use(s.mw.NoCache())
r.Use(s.mw.RequireAuth())
r.Get("/", s.h.HandleProfile())
r.With(s.mw.PasswordChangeRateLimit()).Post(
"/password", s.h.HandlePasswordChange(),
)
})
}
func (s *Server) setupSourceRoutes() {
s.router.Route("/sources", func(r chi.Router) {
// MaxBodySize must precede CSRF: gorilla/csrf parses the
// form, so the cap has to be installed before it runs.
r.Use(s.mw.MaxBodySize(maxFormBodySize))
r.Use(s.mw.CSRF())
r.Use(s.mw.NoCache())
r.Use(s.mw.RequireAuth())
r.Use(s.mw.MaxBodySize(maxFormBodySize))
r.Get("/", s.h.HandleSourceList())
r.Get("/new", s.h.HandleSourceCreate())
r.Post("/new", s.h.HandleSourceCreateSubmit())
})
s.router.Route("/source/{sourceID}", func(r chi.Router) {
// MaxBodySize must precede CSRF: gorilla/csrf parses the
// form, so the cap has to be installed before it runs.
r.Use(s.mw.MaxBodySize(maxFormBodySize))
r.Use(s.mw.CSRF())
r.Use(s.mw.NoCache())
r.Use(s.mw.RequireAuth())
r.Use(s.mw.MaxBodySize(maxFormBodySize))
r.Get("/", s.h.HandleSourceDetail())
r.Get("/edit", s.h.HandleSourceEdit())
r.Post("/edit", s.h.HandleSourceEditSubmit())

383
internal/server/routes_test.go
View File

@@ -1,383 +0,0 @@
package server_test
import (
"context"
"html"
"net/http"
"net/http/httptest"
"net/url"
"regexp"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/fx"
"go.uber.org/fx/fxtest"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
"sneak.berlin/go/webhooker/internal/globals"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/healthcheck"
"sneak.berlin/go/webhooker/internal/logger"
"sneak.berlin/go/webhooker/internal/middleware"
"sneak.berlin/go/webhooker/internal/server"
"sneak.berlin/go/webhooker/internal/session"
)
// csrfCookieName is the cookie gorilla/csrf issues when it runs. Its
// presence or absence on a response is how these tests tell whether
// the CSRF middleware executed.
const csrfCookieName = "_gorilla_csrf"
type noopNotifier struct{}
func (n *noopNotifier) Notify([]delivery.Task) {}
// noopEvictor satisfies handlers.New's delivery.WebhookEvictor
// dependency. These tests never delete a webhook, so there is
// nothing to record.
type noopEvictor struct{}
func (e *noopEvictor) EvictWebhook(string) {}
// testEnv is the real router from routes.go plus the collaborators
// tests need to seed users and forge sessions.
type testEnv struct {
router http.Handler
sess *session.Session
db *database.Database
}
// newTestEnv wires the dependency graph with fx and builds the
// production route tree, so middleware registration order is
// exercised exactly as it ships.
func newTestEnv(t *testing.T) *testEnv {
t.Helper()
var (
log *logger.Logger
cfg *config.Config
mw *middleware.Middleware
hnd *handlers.Handlers
sess *session.Session
db *database.Database
)
app := fxtest.New(
t,
fx.Provide(
globals.New,
logger.New,
func() *config.Config {
return &config.Config{
DataDir: t.TempDir(),
Environment: config.EnvironmentDev,
}
},
database.New,
database.NewWebhookDBManager,
healthcheck.New,
session.New,
func() delivery.Notifier { return &noopNotifier{} },
func() delivery.WebhookEvictor { return &noopEvictor{} },
middleware.New,
handlers.New,
),
fx.Populate(&log, &cfg, &mw, &hnd, &sess, &db),
)
app.RequireStart()
t.Cleanup(app.RequireStop)
return &testEnv{
router: server.NewRouterForTest(log.Get(), cfg, mw, hnd),
sess: sess,
db: db,
}
}
// oversizeValue returns a form value one byte past the route-group
// body cap, so an encoded form containing it is guaranteed oversize.
func oversizeValue() string {
return strings.Repeat("a", int(server.MaxFormBodySizeForTest)+1)
}
// csrfCookieSet reports whether the response issued a gorilla/csrf
// cookie, which only happens if the CSRF middleware ran.
func csrfCookieSet(w *httptest.ResponseRecorder) bool {
for _, c := range w.Result().Cookies() {
if c.Name == csrfCookieName {
return true
}
}
return false
}
// get issues a GET through the router with the supplied cookies.
func (e *testEnv) get(
path string,
cookies []*http.Cookie,
) *httptest.ResponseRecorder {
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, path, nil,
)
for _, c := range cookies {
req.AddCookie(c)
}
w := httptest.NewRecorder()
e.router.ServeHTTP(w, req)
return w
}
// post issues a urlencoded form POST through the router. The body is
// a strings.Reader, so the request carries an accurate
// Content-Length — the signal MaxBodySize checks up front.
func (e *testEnv) post(
path string,
form url.Values,
cookies []*http.Cookie,
) *httptest.ResponseRecorder {
req := httptest.NewRequestWithContext(
context.Background(), http.MethodPost, path,
strings.NewReader(form.Encode()),
)
req.Header.Set(
"Content-Type", "application/x-www-form-urlencoded",
)
for _, c := range cookies {
req.AddCookie(c)
}
w := httptest.NewRecorder()
e.router.ServeHTTP(w, req)
return w
}
// csrfFrom renders the page at path and returns the CSRF token from
// its form together with every cookie needed for the follow-up POST.
func (e *testEnv) csrfFrom(
t *testing.T,
path string,
cookies []*http.Cookie,
) (string, []*http.Cookie) {
t.Helper()
w := e.get(path, cookies)
require.Equal(t, http.StatusOK, w.Code)
pattern := regexp.MustCompile(
`name="csrf_token" value="([^"]+)"`,
)
match := pattern.FindStringSubmatch(w.Body.String())
require.Len(t, match, 2, "form must embed a CSRF token")
// html/template escapes "+" and "=" in attribute values, and
// gorilla/csrf tokens are standard base64, so the value read
// out of the markup has to be unescaped before it is submitted.
token := html.UnescapeString(match[1])
combined := make([]*http.Cookie, 0, len(cookies))
combined = append(combined, cookies...)
combined = append(combined, w.Result().Cookies()...)
return token, combined
}
// authCookies forges an authenticated session for the given user.
func (e *testEnv) authCookies(
t *testing.T,
userID, username string,
) []*http.Cookie {
t.Helper()
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/setup", nil,
)
w := httptest.NewRecorder()
s, err := e.sess.Get(req)
require.NoError(t, err)
e.sess.SetUser(s, userID, username)
require.NoError(t, e.sess.Save(req, w, s))
cookies := w.Result().Cookies()
require.NotEmpty(t, cookies, "session cookie should be set")
return cookies
}
// seedUser creates a user with the given password and returns the
// stored hash so tests can assert whether it later changed.
func (e *testEnv) seedUser(
t *testing.T,
username, password string,
) (string, string) {
t.Helper()
hash, err := database.HashPassword(password)
require.NoError(t, err)
user := &database.User{Username: username, Password: hash}
require.NoError(t, e.db.DB().Create(user).Error)
return user.ID, hash
}
// storedHash reads the current password hash for a username.
func (e *testEnv) storedHash(t *testing.T, username string) string {
t.Helper()
var user database.User
require.NoError(t,
e.db.DB().Where("username = ?", username).
First(&user).Error,
)
return user.Password
}
// --- /pages group ---
// TestPagesLogin_OversizeBody_RejectedBeforeCSRF proves the cap runs
// ahead of gorilla/csrf: the response is a clean 413 and no CSRF
// cookie was issued, so neither the CSRF middleware nor the login
// handler ran.
func TestPagesLogin_OversizeBody_RejectedBeforeCSRF(t *testing.T) {
t.Parallel()
env := newTestEnv(t)
form := url.Values{}
form.Set("username", oversizeValue())
form.Set("password", "irrelevant")
w := env.post("/pages/login", form, nil)
assert.Equal(
t, http.StatusRequestEntityTooLarge, w.Code,
)
assert.False(
t, csrfCookieSet(w),
"CSRF middleware must not run for an oversized body",
)
}
// TestPagesLogin_UnderLimit_NoToken_CSRFRejects is the control for
// the test above: an identically shaped but under-limit POST does
// reach gorilla/csrf, which rejects it and issues its cookie. Without
// this, the missing-cookie assertion above would prove nothing.
func TestPagesLogin_UnderLimit_NoToken_CSRFRejects(t *testing.T) {
t.Parallel()
env := newTestEnv(t)
form := url.Values{}
form.Set("username", "someone")
form.Set("password", "irrelevant")
w := env.post("/pages/login", form, nil)
assert.Equal(t, http.StatusForbidden, w.Code)
assert.True(
t, csrfCookieSet(w),
"CSRF middleware should run for an under-limit body",
)
}
// TestPagesLogin_UnderLimit_ValidToken_ReachesHandler proves the
// reorder did not break CSRF token handling: a token harvested from
// the rendered login form is still accepted and the request lands in
// the handler.
func TestPagesLogin_UnderLimit_ValidToken_ReachesHandler(
t *testing.T,
) {
t.Parallel()
env := newTestEnv(t)
token, cookies := env.csrfFrom(t, "/pages/login", nil)
form := url.Values{}
form.Set("csrf_token", token)
form.Set("username", "nosuchuser")
form.Set("password", "wrongpassword")
w := env.post("/pages/login", form, cookies)
assert.Equal(t, http.StatusUnauthorized, w.Code)
assert.Contains(
t, w.Body.String(), "Invalid username or password",
"request should reach the login handler",
)
}
// --- /user/{username} group ---
// TestPasswordChange_OversizeBody_RejectedAndPasswordUnchanged
// covers the route that previously had no middleware body cap at
// all. The request carries a valid session and a valid CSRF token,
// so the only thing that can stop it is the size cap; the unchanged
// password hash is the observable proof the handler never ran.
func TestPasswordChange_OversizeBody_RejectedAndPasswordUnchanged(
t *testing.T,
) {
t.Parallel()
env := newTestEnv(t)
userID, originalHash := env.seedUser(t, "pwuser", "oldpassword")
cookies := env.authCookies(t, userID, "pwuser")
token, cookies := env.csrfFrom(t, "/user/pwuser/", cookies)
form := url.Values{}
form.Set("csrf_token", token)
form.Set("current_password", "oldpassword")
form.Set("new_password", oversizeValue())
form.Set("confirm_password", oversizeValue())
w := env.post("/user/pwuser/password", form, cookies)
assert.Equal(
t, http.StatusRequestEntityTooLarge, w.Code,
)
assert.Equal(
t, originalHash, env.storedHash(t, "pwuser"),
"handler must not run, so the password must be unchanged",
)
}
// TestPasswordChange_UnderLimit_Succeeds proves that adding the cap
// to the /user/{username} group did not break the route it guards.
func TestPasswordChange_UnderLimit_Succeeds(t *testing.T) {
t.Parallel()
env := newTestEnv(t)
userID, originalHash := env.seedUser(t, "okuser", "oldpassword")
cookies := env.authCookies(t, userID, "okuser")
token, cookies := env.csrfFrom(t, "/user/okuser/", cookies)
form := url.Values{}
form.Set("csrf_token", token)
form.Set("current_password", "oldpassword")
form.Set("new_password", "brandnewpassword")
form.Set("confirm_password", "brandnewpassword")
w := env.post("/user/okuser/password", form, cookies)
assert.Equal(t, http.StatusOK, w.Code)
assert.NotEqual(
t, originalHash, env.storedHash(t, "okuser"),
"an under-limit password change should still apply",
)
}

155
internal/session/session.go
View File

@@ -10,7 +10,6 @@ import (
"log/slog"
"maps"
"net/http"
"time"
"github.com/gorilla/sessions"
"go.uber.org/fx"
@@ -33,18 +32,6 @@ const (
// status.
AuthenticatedKey = "authenticated"
// CreatedAtKey is the session key holding the Unix timestamp at
// which the session was authenticated. It anchors the ABSOLUTE
// expiry clock and is written exactly once, by SetUser. Nothing
// refreshes it: an absolute deadline that moved with activity
// would not be a cap at all.
CreatedAtKey = "created_at"
// LastSeenKey is the session key holding the Unix timestamp of
// the most recent authenticated request. It anchors the IDLE
// expiry clock and is pushed forward by Touch.
LastSeenKey = "last_seen"
// sessionKeyLength is the required length in bytes for the
// session authentication key.
sessionKeyLength = 32
@@ -54,19 +41,6 @@ const (
// secondsPerDay is the number of seconds in a day.
secondsPerDay = 86400
// sessionAbsoluteMaxAge is the hard upper bound on how long a
// session may live, measured from CreatedAtKey. Activity never
// extends it, so even a continuously used session ends here and
// the user has to authenticate again.
sessionAbsoluteMaxAge = sessionMaxAgeDays * secondsPerDay * time.Second
// idleRefreshDivisor rate-limits idle-deadline refreshes. Touch
// only rewrites LastSeenKey once the stored value is older than
// idleTimeout/idleRefreshDivisor, so an active session is
// re-saved at most this many times per idle window instead of
// once per request. See Touch for the tradeoff this buys.
idleRefreshDivisor = 10
)
// ErrSessionKeyLength is returned when the decoded session key
@@ -88,16 +62,6 @@ type Session struct {
key []byte // raw 32-byte auth key, also used for CSRF cookie signing
log *slog.Logger
config *config.Config
// idleTimeout is the sliding inactivity window. A session that
// sees no authenticated request within this window expires,
// independently of the absolute cap. Non-positive disables idle
// expiry and leaves sessionAbsoluteMaxAge as the only bound.
idleTimeout time.Duration
// now reads the current time. Injected so expiry can be tested
// without sleeping.
now func() time.Time
}
// New creates a new session manager. The cookie store is
@@ -111,8 +75,6 @@ func New(
s := &Session{
log: params.Logger.Get(),
config: params.Config,
idleTimeout: params.Config.SessionIdleTimeout,
now: time.Now,
}
lc.Append(fx.Hook{
@@ -187,98 +149,29 @@ func (s *Session) Save(
return sess.Save(r, w)
}
// SetUser sets the user information in the session. It starts both
// expiry clocks: CreatedAtKey (absolute, never refreshed again) and
// LastSeenKey (idle, refreshed by Touch).
// SetUser sets the user information in the session.
func (s *Session) SetUser(
sess *sessions.Session,
userID, username string,
) {
now := s.now().Unix()
sess.Values[UserIDKey] = userID
sess.Values[UsernameKey] = username
sess.Values[AuthenticatedKey] = true
sess.Values[CreatedAtKey] = now
sess.Values[LastSeenKey] = now
}
// ClearUser removes user information from the session, including
// both expiry timestamps.
// ClearUser removes user information from the session.
func (s *Session) ClearUser(sess *sessions.Session) {
delete(sess.Values, UserIDKey)
delete(sess.Values, UsernameKey)
delete(sess.Values, AuthenticatedKey)
delete(sess.Values, CreatedAtKey)
delete(sess.Values, LastSeenKey)
}
// sessionTime reads a Unix-second timestamp stored under key.
func sessionTime(
sess *sessions.Session,
key string,
) (time.Time, bool) {
secs, ok := sess.Values[key].(int64)
if !ok {
return time.Time{}, false
}
return time.Unix(secs, 0), true
}
// IsAuthenticated checks if the session has an authenticated user
// whose session has not passed either expiry deadline. Every
// authentication decision goes through here, so neither clock can
// be bypassed by a caller that forgets to check it.
// IsAuthenticated checks if the session has an authenticated
// user.
func (s *Session) IsAuthenticated(sess *sessions.Session) bool {
auth, ok := sess.Values[AuthenticatedKey].(bool)
if !ok || !auth {
return false
}
return !s.expired(sess)
}
// Touch records authenticated activity by pushing the IDLE deadline
// forward. It writes LastSeenKey only; CreatedAtKey is left alone so
// the absolute cap keeps counting down even for a user who never
// stops clicking.
//
// Callers must only invoke Touch for a request that authenticated
// with this session. Refreshing on an unauthenticated request would
// let anyone holding a stolen or abandoned cookie keep the session
// alive by polling a public endpoint. Touch enforces that itself by
// returning false for any session that is not currently
// authenticated and unexpired.
//
// To avoid re-encrypting and re-emitting the session cookie on every
// single request, the timestamp is advanced only once it is older
// than idleTimeout/idleRefreshDivisor. The tradeoff is that
// LastSeenKey lags real activity by up to that much, so a session
// can expire slightly early relative to the user's true last
// request -- never late.
//
// Touch reports whether it changed the session; only then does the
// caller need to save it.
func (s *Session) Touch(sess *sessions.Session) bool {
if s.idleTimeout <= 0 {
return false
}
if !s.IsAuthenticated(sess) {
return false
}
now := s.now()
lastSeen, ok := sessionTime(sess, LastSeenKey)
if ok && now.Sub(lastSeen) < s.idleTimeout/idleRefreshDivisor {
return false
}
sess.Values[LastSeenKey] = now.Unix()
return true
return ok && auth
}
// GetUserID retrieves the user ID from the session.
@@ -360,41 +253,3 @@ func (s *Session) Regenerate(
return newSess, nil
}
// expired reports whether the session has passed either of its two
// independent deadlines. They are deliberately kept apart:
//
// - the ABSOLUTE deadline is CreatedAtKey + sessionAbsoluteMaxAge.
// It is fixed at login and no amount of activity moves it.
// - the IDLE deadline is LastSeenKey + idleTimeout. Activity moves
// it forward via Touch.
//
// Whichever comes first ends the session.
//
// A session that claims to be authenticated but carries no
// timestamps predates this check; it is treated as expired so the
// user re-authenticates rather than being granted an unbounded
// session.
func (s *Session) expired(sess *sessions.Session) bool {
now := s.now()
createdAt, ok := sessionTime(sess, CreatedAtKey)
if !ok {
return true
}
if !now.Before(createdAt.Add(sessionAbsoluteMaxAge)) {
return true
}
if s.idleTimeout <= 0 {
return false
}
lastSeen, ok := sessionTime(sess, LastSeenKey)
if !ok {
return true
}
return !now.Before(lastSeen.Add(s.idleTimeout))
}

406
internal/session/session_test.go
View File

@@ -7,7 +7,6 @@ import (
"net/http/httptest"
"os"
"testing"
"time"
"github.com/gorilla/sessions"
"github.com/stretchr/testify/assert"
@@ -18,47 +17,11 @@ import (
const testKeySize = 32
// testIdleTimeout is the idle window used by the expiry tests.
const testIdleTimeout = time.Hour
// testAbsoluteMaxAge restates the documented absolute session cap
// independently of the implementation constant.
const testAbsoluteMaxAge = 7 * 24 * time.Hour
// fakeClock is a manually advanced clock, so expiry can be tested
// without sleeping.
type fakeClock struct {
t time.Time
}
func (c *fakeClock) Now() time.Time {
return c.t
}
func (c *fakeClock) Advance(d time.Duration) {
c.t = c.t.Add(d)
}
// testSession creates a Session with a real cookie store and the
// real clock.
// testSession creates a Session with a real cookie store for
// testing.
func testSession(t *testing.T) *session.Session {
t.Helper()
s, _ := testSessionWithClock(t, testIdleTimeout, nil)
return s
}
// testSessionWithClock creates a Session with a real cookie store,
// the given idle timeout, and a manually advanced clock. Passing a
// nil clock uses the real one.
func testSessionWithClock(
t *testing.T,
idleTimeout time.Duration,
clock *fakeClock,
) (*session.Session, *fakeClock) {
t.Helper()
key := make([]byte, testKeySize)
for i := range key {
@@ -76,7 +39,6 @@ func testSessionWithClock(
cfg := &config.Config{
Environment: config.EnvironmentDev,
SessionIdleTimeout: idleTimeout,
}
log := slog.New(slog.NewTextHandler(
@@ -84,46 +46,7 @@ func testSessionWithClock(
&slog.HandlerOptions{Level: slog.LevelDebug},
))
var now func() time.Time
if clock != nil {
now = clock.Now
}
return session.NewForTest(store, cfg, log, key, now), clock
}
// newFakeClock returns a clock started at a fixed instant.
func newFakeClock() *fakeClock {
return &fakeClock{
t: time.Date(
2026, time.January, 2, 3, 4, 5, 0, time.UTC,
),
}
}
// authenticatedSession returns a fresh session that has just been
// logged in, along with its manager and clock.
func authenticatedSession(
t *testing.T,
idleTimeout time.Duration,
) (*session.Session, *sessions.Session, *fakeClock) {
t.Helper()
s, clock := testSessionWithClock(
t, idleTimeout, newFakeClock(),
)
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/", nil)
sess, err := s.Get(req)
require.NoError(t, err)
s.SetUser(sess, "user-123", "alice")
require.True(t, s.IsAuthenticated(sess))
return s, sess, clock
return session.NewForTest(store, cfg, log, key)
}
// --- Get and Save Tests ---
@@ -250,18 +173,8 @@ func TestSetUser_SetsAllFields(t *testing.T) {
)
}
// testSessionGetter exercises a session string getter before and
// after SetUser: it must report false with an empty value on a
// fresh session, then true with the expected value once
// SetUser(sess, "user-xyz", "bob") has run.
func testSessionGetter(
t *testing.T,
get func(
*session.Session, *sessions.Session,
) (string, bool),
expected string,
) {
t.Helper()
func TestGetUserID(t *testing.T) {
t.Parallel()
s := testSession(t)
@@ -272,46 +185,44 @@ func testSessionGetter(
require.NoError(t, err)
// Before setting user
val, ok := get(s, sess)
userID, ok := s.GetUserID(sess)
assert.False(
t, ok, "should return false before SetUser",
t, ok, "should return false when no user ID is set",
)
assert.Empty(t, val)
assert.Empty(t, userID)
// After setting user
s.SetUser(sess, "user-xyz", "bob")
val, ok = get(s, sess)
userID, ok = s.GetUserID(sess)
assert.True(t, ok)
assert.Equal(t, expected, val)
}
func TestGetUserID(t *testing.T) {
t.Parallel()
testSessionGetter(
t,
func(
s *session.Session, sess *sessions.Session,
) (string, bool) {
return s.GetUserID(sess)
},
"user-xyz",
)
assert.Equal(t, "user-xyz", userID)
}
func TestGetUsername(t *testing.T) {
t.Parallel()
testSessionGetter(
t,
func(
s *session.Session, sess *sessions.Session,
) (string, bool) {
return s.GetUsername(sess)
},
"bob",
s := testSession(t)
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/", nil)
sess, err := s.Get(req)
require.NoError(t, err)
// Before setting user
username, ok := s.GetUsername(sess)
assert.False(
t, ok, "should return false when no username is set",
)
assert.Empty(t, username)
// After setting user
s.SetUser(sess, "user-xyz", "bob")
username, ok = s.GetUsername(sess)
assert.True(t, ok)
assert.Equal(t, "bob", username)
}
// --- IsAuthenticated Tests ---
@@ -507,263 +418,6 @@ func TestSessionConstants(t *testing.T) {
assert.Equal(t, "user_id", session.UserIDKey)
assert.Equal(t, "username", session.UsernameKey)
assert.Equal(t, "authenticated", session.AuthenticatedKey)
assert.Equal(t, "created_at", session.CreatedAtKey)
assert.Equal(t, "last_seen", session.LastSeenKey)
}
// --- Expiry Tests ---
func TestSetUser_StartsBothClocks(t *testing.T) {
t.Parallel()
_, sess, clock := authenticatedSession(t, testIdleTimeout)
assert.Equal(
t, clock.Now().Unix(), sess.Values[session.CreatedAtKey],
"SetUser should anchor the absolute clock",
)
assert.Equal(
t, clock.Now().Unix(), sess.Values[session.LastSeenKey],
"SetUser should anchor the idle clock",
)
}
func TestIsAuthenticated_WithinIdleWindow(t *testing.T) {
t.Parallel()
s, sess, clock := authenticatedSession(t, testIdleTimeout)
clock.Advance(testIdleTimeout - time.Second)
assert.True(
t, s.IsAuthenticated(sess),
"session should still be valid just inside the idle window",
)
}
func TestIsAuthenticated_IdleExpired(t *testing.T) {
t.Parallel()
s, sess, clock := authenticatedSession(t, testIdleTimeout)
clock.Advance(testIdleTimeout)
assert.False(
t, s.IsAuthenticated(sess),
"session should expire once the idle window lapses",
)
}
// TestTouch_DoesNotExtendAbsoluteCap is the regression test for the
// refresh-the-wrong-clock bug: a session that is used continuously
// must survive well past the idle window and still die at the
// absolute cap.
func TestTouch_DoesNotExtendAbsoluteCap(t *testing.T) {
t.Parallel()
s, sess, clock := authenticatedSession(t, testIdleTimeout)
createdAt := sess.Values[session.CreatedAtKey]
// Stay active: a request every half idle window, right up to
// the absolute cap.
step := testIdleTimeout / 2
steps := int(testAbsoluteMaxAge/step) - 1
for i := range steps {
clock.Advance(step)
s.Touch(sess)
require.True(
t, s.IsAuthenticated(sess),
"active session should survive the idle window "+
"(step %d of %d)", i+1, steps,
)
}
// One more step of activity takes the session to exactly the
// absolute cap, measured from login. Nothing that happened in
// the loop may have moved that deadline.
clock.Advance(step)
s.Touch(sess)
assert.False(
t, s.IsAuthenticated(sess),
"activity must not extend the absolute cap",
)
assert.Equal(
t, createdAt, sess.Values[session.CreatedAtKey],
"Touch must never rewrite the absolute-clock anchor",
)
}
func TestTouch_RefreshesIdleDeadline(t *testing.T) {
t.Parallel()
s, sess, clock := authenticatedSession(t, testIdleTimeout)
// Halfway through the window, activity happens.
clock.Advance(testIdleTimeout / 2)
assert.True(
t, s.Touch(sess),
"Touch should refresh once past the lazy-refresh threshold",
)
// Past the original deadline, but inside the refreshed one.
clock.Advance(testIdleTimeout - time.Second)
assert.True(
t, s.IsAuthenticated(sess),
"refreshed session should outlive the original deadline",
)
// And it still expires an idle window after that activity.
clock.Advance(time.Second)
assert.False(
t, s.IsAuthenticated(sess),
"refreshed session should expire one window after activity",
)
}
func TestTouch_LazyBelowRefreshThreshold(t *testing.T) {
t.Parallel()
s, sess, clock := authenticatedSession(t, testIdleTimeout)
before := sess.Values[session.LastSeenKey]
// A request arriving almost immediately is not worth a cookie
// rewrite.
clock.Advance(time.Second)
assert.False(
t, s.Touch(sess),
"Touch should not rewrite the session below the threshold",
)
assert.Equal(
t, before, sess.Values[session.LastSeenKey],
"last-seen should be unchanged below the threshold",
)
}
func TestTouch_UnauthenticatedSessionIsNotRefreshed(t *testing.T) {
t.Parallel()
s, clock := testSessionWithClock(
t, testIdleTimeout, newFakeClock(),
)
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/", nil)
sess, err := s.Get(req)
require.NoError(t, err)
clock.Advance(testIdleTimeout / 2)
assert.False(
t, s.Touch(sess),
"an unauthenticated session must not be refreshed",
)
_, hasLastSeen := sess.Values[session.LastSeenKey]
assert.False(
t, hasLastSeen,
"Touch must not stamp an unauthenticated session",
)
}
func TestTouch_IdleExpiredSessionIsNotRevived(t *testing.T) {
t.Parallel()
s, sess, clock := authenticatedSession(t, testIdleTimeout)
clock.Advance(testIdleTimeout)
require.False(t, s.IsAuthenticated(sess))
assert.False(
t, s.Touch(sess),
"an already expired session must not be refreshed",
)
assert.False(
t, s.IsAuthenticated(sess),
"Touch must not revive an expired session",
)
}
func TestIsAuthenticated_MissingTimestamps(t *testing.T) {
t.Parallel()
s, _ := testSessionWithClock(
t, testIdleTimeout, newFakeClock(),
)
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/", nil)
sess, err := s.Get(req)
require.NoError(t, err)
// A session from before idle expiry existed: authenticated,
// but with no timestamps. Fail closed.
sess.Values[session.AuthenticatedKey] = true
assert.False(
t, s.IsAuthenticated(sess),
"a session with no timestamps should be rejected",
)
}
func TestIsAuthenticated_MissingLastSeen(t *testing.T) {
t.Parallel()
s, sess, _ := authenticatedSession(t, testIdleTimeout)
delete(sess.Values, session.LastSeenKey)
assert.False(
t, s.IsAuthenticated(sess),
"a session with no idle anchor should be rejected",
)
}
func TestIdleTimeoutDisabled_AbsoluteCapStillApplies(t *testing.T) {
t.Parallel()
s, sess, clock := authenticatedSession(t, 0)
// Idle expiry is off, so an untouched session survives an
// arbitrary idle stretch.
clock.Advance(testAbsoluteMaxAge - time.Second)
assert.True(
t, s.IsAuthenticated(sess),
"idle expiry should be disabled by a non-positive timeout",
)
assert.False(
t, s.Touch(sess),
"Touch should be a no-op when idle expiry is disabled",
)
// The absolute cap still ends it.
clock.Advance(time.Second)
assert.False(
t, s.IsAuthenticated(sess),
"the absolute cap must still apply with idle expiry off",
)
}
func TestClearUser_RemovesTimestamps(t *testing.T) {
t.Parallel()
s, sess, _ := authenticatedSession(t, testIdleTimeout)
s.ClearUser(sess)
_, hasCreatedAt := sess.Values[session.CreatedAtKey]
assert.False(t, hasCreatedAt, "CreatedAtKey should be removed")
_, hasLastSeen := sess.Values[session.LastSeenKey]
assert.False(t, hasLastSeen, "LastSeenKey should be removed")
}
// --- Edge Cases ---

20
internal/session/testing.go
View File

@@ -2,7 +2,6 @@ package session
import (
"log/slog"
"time"
"github.com/gorilla/sessions"
"sneak.berlin/go/webhooker/internal/config"
@@ -13,28 +12,11 @@ import (
// middleware and handler tests to use real session functionality. The key
// parameter is the raw 32-byte authentication key used for session encryption
// and CSRF cookie signing.
//
// The idle timeout is taken from cfg.SessionIdleTimeout, exactly as in
// production. The now parameter supplies the clock used for expiry
// checks so tests can advance time without sleeping; pass nil for the
// real clock.
func NewForTest(
store *sessions.CookieStore,
cfg *config.Config,
log *slog.Logger,
key []byte,
now func() time.Time,
) *Session {
if now == nil {
now = time.Now
}
func NewForTest(store *sessions.CookieStore, cfg *config.Config, log *slog.Logger, key []byte) *Session {
return &Session{
store: store,
key: key,
config: cfg,
log: log,
idleTimeout: cfg.SessionIdleTimeout,
now: now,
}
}

10
script/bootstrap
View File

@@ -10,11 +10,11 @@ set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Pinned versions, 2026-08-07. Never "latest"; exact versions only.
GOLANGCI_LINT_VERSION="2.12.2"
# sha256 of golangci-lint-2.12.2-linux-<arch>.tar.gz release archives
GOLANGCI_LINT_SHA256_AMD64="8df580d2670fed8fa984aac0507099af8df275e665215f5c7a2ae3943893a553"
GOLANGCI_LINT_SHA256_ARM64="44cd40a8c76c86755375adfeea52cfd3533cb43d7bd647771e0ae065e166df3a"
# Pinned versions, 2026-07-07. Never "latest"; exact versions only.
GOLANGCI_LINT_VERSION="2.11.3"
# sha256 of golangci-lint-2.11.3-linux-<arch>.tar.gz release archives
GOLANGCI_LINT_SHA256_AMD64="87bb8cddbcc825d5778b64e8a91b46c0526b247f4e2f2904dea74ec7450475d1"
GOLANGCI_LINT_SHA256_ARM64="ee3d95f301359e7d578e6d99c8ad5aeadbabc5a13009a30b2b0df11c8058afe9"
PKGMGR=""
SUDO=""

56
templates/profile.html
View File

@@ -6,18 +6,6 @@
<div class="max-w-4xl mx-auto px-6 py-12">
<h1 class="text-2xl font-medium text-gray-900 mb-6">User Profile</h1>
{{if .SuccessMessage}}
<div class="alert-success">
<span>{{.SuccessMessage}}</span>
</div>
{{end}}
{{if .ErrorMessage}}
<div class="alert-error">
<span>{{.ErrorMessage}}</span>
</div>
{{end}}
<div class="card p-6">
<div class="flex items-center mb-6">
<div class="mr-4">
@@ -55,50 +43,6 @@
</div>
</div>
<div class="card p-6 mt-6">
<h3 class="text-lg font-medium text-gray-900 mb-3">Change Password</h3>
<form method="POST" action="/user/{{.User.Username}}/password" class="space-y-6">
<input type="hidden" name="csrf_token" value="{{.CSRFToken}}">
<div class="form-group">
<label for="current_password" class="label">Current Password</label>
<input
type="password"
id="current_password"
name="current_password"
required
autocomplete="current-password"
class="input"
>
</div>
<div class="form-group">
<label for="new_password" class="label">New Password</label>
<input
type="password"
id="new_password"
name="new_password"
required
autocomplete="new-password"
class="input"
>
</div>
<div class="form-group">
<label for="confirm_password" class="label">Confirm New Password</label>
<input
type="password"
id="confirm_password"
name="confirm_password"
required
autocomplete="new-password"
class="input"
>
</div>
<button type="submit" class="btn-primary">Change Password</button>
</form>
</div>
<div class="mt-6">
<a href="/" class="btn-secondary">Back to Home</a>
</div>

13
templates/source_detail.html
View File

@@ -113,10 +113,6 @@
<input type="url" name="url" placeholder="https://hooks.slack.com/services/..." :disabled="targetType !== 'slack'" class="input text-sm">
<p class="text-xs text-gray-500 mt-1">Slack or Mattermost incoming webhook URL. Payloads are pretty-printed in code blocks.</p>
</div>
<div x-show="targetType === 'database'">
<input type="text" name="expiry" placeholder="never" :disabled="targetType !== 'database'" class="input text-sm">
<p class="text-xs text-gray-500 mt-1">Archive expiry: "never" (default) keeps rows forever, or a duration like "720h" prunes older rows.</p>
</div>
<button type="submit" class="btn-primary text-sm">Add Target</button>
</form>
</div>
@@ -145,11 +141,8 @@
</form>
</div>
</div>
{{range .Config}}
<div class="text-xs text-gray-500 break-all mt-1">
<span class="font-medium text-gray-700">{{.Label}}:</span>
<span>{{.Value}}</span>
</div>
{{if .Config}}
<code class="text-xs text-gray-500 break-all block mt-1">{{.Config}}</code>
{{end}}
</div>
{{else}}
@@ -184,7 +177,7 @@
<!-- Info -->
<div class="mt-4 text-sm text-gray-400">
<p>Retention: {{.Webhook.RetentionLabel}} &middot; Created: {{.Webhook.CreatedAt.Format "2006-01-02 15:04:05 UTC"}}</p>
<p>Retention: {{.Webhook.RetentionDays}} days &middot; Created: {{.Webhook.CreatedAt.Format "2006-01-02 15:04:05 UTC"}}</p>
</div>
</div>
{{end}}

3
templates/source_edit.html
View File

@@ -28,8 +28,7 @@
<div class="form-group">
<label for="retention_days" class="label">Retention (days)</label>
<input type="number" id="retention_days" name="retention_days" value="{{.Webhook.RetentionDays}}" min="0" class="input">
<p class="text-xs text-gray-500 mt-1">Currently {{.Webhook.RetentionLabel}}. Enter 0 to retain events forever.</p>
<input type="number" id="retention_days" name="retention_days" value="{{.Webhook.RetentionDays}}" min="1" max="365" class="input">
</div>
<div class="flex gap-3">

2
templates/sources_list.html
View File

@@ -25,7 +25,7 @@
<p class="text-sm text-gray-500 mt-1">{{.Description}}</p>
{{end}}
</div>
<span class="badge-info">Retention: {{.RetentionLabel}}</span>
<span class="badge-info">{{.RetentionDays}}d retention</span>
</div>
<div class="flex gap-6 mt-4 text-sm text-gray-500">
<span>{{.EntrypointCount}} entrypoint{{if ne .EntrypointCount 1}}s{{end}}</span>

8
templates/sources_new.html
View File

@@ -18,18 +18,18 @@
<input type="hidden" name="csrf_token" value="{{.CSRFToken}}">
<div class="form-group">
<label for="name" class="label">Name</label>
<input type="text" id="name" name="name" value="{{.Name}}" required autofocus placeholder="My Webhook" class="input">
<input type="text" id="name" name="name" required autofocus placeholder="My Webhook" class="input">
</div>
<div class="form-group">
<label for="description" class="label">Description</label>
<textarea id="description" name="description" rows="3" placeholder="Optional description" class="input">{{.Description}}</textarea>
<textarea id="description" name="description" rows="3" placeholder="Optional description" class="input"></textarea>
</div>
<div class="form-group">
<label for="retention_days" class="label">Retention (days)</label>
<input type="number" id="retention_days" name="retention_days" value="{{.DefaultRetentionDays}}" min="0" class="input">
<p class="text-xs text-gray-500 mt-1">How long to keep event data. Enter 0 to retain events forever.</p>
<input type="number" id="retention_days" name="retention_days" value="30" min="1" max="365" class="input">
<p class="text-xs text-gray-500 mt-1">How long to keep event data.</p>
</div>
<div class="flex gap-3">