sneak/webhooker
1
0
Fork 0
Code Issues 34 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Compare commits

base: sneak:issue-70-log-target
Branches Tags
sneak:main sneak:next sneak:feat/receiver-rate-limit sneak:issue-70-log-target
...
compare: sneak:7f27362d425b502dddfbb774442549ef82b64586
Branches Tags
sneak:next sneak:main sneak:feat/receiver-rate-limit sneak:issue-70-log-target

50 Commits
issue-70-l ... 7f27362d42

Author SHA1 Message Date
sneak
7f27362d42 Raise script/test's per-package timeout to 180s (closes #194)
All checks were successful
check / check (push) Successful in 2m57s
Details 
`go test -timeout` is per package, so the budget has to clear the slowest
single package: internal/handlers. Measured in a cache-defeated builder
stage on the 48-core shared build host, 2026-08-18:

  16.9s                  host load 5-20, GOMAXPROCS 48
  45.9s / 47.3s / 49.0s  three runs at deliberate host load 31-73
  30.6s / 39.7s          host load 5-20, GOMAXPROCS 6 / 4
  67.3s / 97.5s          host load 5-20, GOMAXPROCS 2 / 1
  67.3s                  GOMAXPROCS 4 at deliberate host load 52-68

The old 30s budget was breached by every loaded run and by every
GOMAXPROCS at or below 6. At GOMAXPROCS 4 it failed outright on plain
33e4fa4 with no other change ("panic: test timed out after 30s"); with
180s the same run passes at 41.262s. 180s is 1.85x the worst figure
above and still bounds a hung package to a tolerable wait.

No test is changed, skipped, shortened or desampled.

#194
 2026-08-18 07:24:21 +00:00
clawbot
33e4fa4faa Report handler panics through the logger and answer 500 (closes #187)
All checks were successful
check / check (push) Successful in 2m50s
Details
2026-08-18 08:33:12 +02:00
clawbot
0c64c411cc Route GORM's logger through slog and bound it (closes #178)
All checks were successful
check / check (push) Successful in 2m57s
Details
2026-08-18 07:17:43 +02:00
clawbot
563e834cf2 Bound every slog line against client-chosen text (closes #176)
All checks were successful
check / check (push) Successful in 2m55s
Details
2026-08-18 06:03:10 +02:00
clawbot
f6ec78e2c8 Stop a slow host turning a login-guard test into a segfault (closes #186)
All checks were successful
check / check (push) Successful in 2m49s
Details
2026-08-18 05:01:13 +02:00
clawbot
9313b0fb41 Merge pull request 'Correct TODO.md milestone state and record seventeen landed units' (#192) from todo-md-milestone-state into next
All checks were successful
check / check (push) Successful in 6s
Details
2026-08-18 04:07:39 +02:00
clawbot
d2cebb5783 Correct TODO.md milestone state and record seventeen landed units
All checks were successful
check / check (push) Successful in 6s
Details 
The Status section claimed next held the completed 1.0.0 milestone with
every issue closed. Four are open (#176, #178, #186, #187), so a merge
of next to main would have shipped that claim to main.

Next Step still named #115 and #125 as open owner decisions; both
landed. It now names the real open items, #150 and #112, and the forced
merge order for the remaining four.

Completed Steps was seventeen units behind, back to 2026-08-12.
 2026-08-18 02:06:43 +00:00
clawbot
b573959a26 Send the chi route pattern to Sentry, not the concrete path (closes #179)
All checks were successful
check / check (push) Successful in 2m51s
Details 
#160 scrubbed the Sentry body, query, cookies, env and headers but kept
Request.URL, which the SDK builds from the concrete path. On the
receiver that path is /webhook/<uuid> in full — a write capability, not
an identifier: anyone holding it can inject events the operator's
targets then deliver. #146's "2xx and 5xx keep the concrete path" ruling
was reasoned about a log the operator owns and does not transfer to a
tracker with its own retention and access control.

The chi route pattern now replaces the path on every route, reached via
the request the SDK carries on hint.Context. Unconditional, because a
route-conditional rule leaks on any route someone forgets to add, and on
a static route the pattern is the path anyway. The fallback is never the
concrete path.

Also rewrites event.Transaction, which carries the same UUID on the
sibling dispatch and which the issue did not name. Tracing is off today,
so that half is a floor rather than a live fix — and it is why enabling
tracing later needs #185 first, or every transaction collapses into one
bucket.

Independently reviewed. The reviewer ran fourteen adversarial probes —
404 and 405 panics, panics in middleware before and after routing,
direct CaptureException, mounted subrouters, wildcards, tracing on and
off — and found no path where the concrete URL survives, and no third
field carrying it.

Merge note: the final round was a two-comment documentation fix on an
already-passed review, correcting a rationale that called the host
operator configuration when it is the client's Host header. I verified
that amend is comment-only myself rather than spending a fifth review
round on it.
 2026-08-18 02:42:58 +02:00
clawbot
76725cffc4 Read form fields from the POST body only (closes #160)
All checks were successful
check / check (push) Successful in 2m53s
Details 
r.FormValue falls back to the query string, so
POST /source/{id}/targets?url=<secret> created a working target from a
value carried on the request line — where proxy logs, browser history
and Referer all record it. Every form read is now r.PostFormValue,
including the login password and both password-change fields, which had
the same defect in a more acute form.

The Sentry leg needed more than the query string. sentryhttp attaches
the whole request to the scope, and ApplyToEvent copies the teed body
into Request.Data with no SendDefaultPII guard — so reading every field
from the body only pointed every credential this change protects at the
one field the first revision did not scrub. Body and query are now
redacted, Cookies and Env cleared, and Headers reduced to an allowlist,
because the SDK's own filter removes four names and would otherwise ship
X-Csrf-Token and the shared secrets senders put on the receiver route.

Also adds json:"-" to Target.Config, APIKey.Key and Setting.Value —
TargetView is the masking barrier for the HTML path only, and the first
handler to marshal a model would serialise a bearer token or the session
encryption key.

Independently reviewed three times. The second review found the Data
leak and proved it with a scratch module; the third disproved the
PR's own claim that BeforeSend gets no request, so the README now
records that redacting unconditionally is a deliberate choice rather
than a limitation — which is what makes #179 cheap to fix.
 2026-08-18 02:04:09 +02:00
clawbot
977fe87588 Verify login credentials before spending rate-limit budget (closes #150)
All checks were successful
check / check (push) Successful in 2m46s
Details 
In the shipped default, any stranger denied the operator the only
administrative path at 5 requests per minute: TRUSTED_PROXIES is empty,
the README requires a reverse proxy, so every login POST shared one
bucket keyed on the proxy.

Credentials are now verified first and only a FAILED attempt spends
budget, so a correct password is never throttled. Failures are counted
per (client bucket, submitted username), bounded. Concurrent Argon2id
verifications are capped at two, and the queue for them at 16 — because
verifying first lets an attacker force a 64 MB hash per request, and
bounding the wait alone bounds nothing.

The issue's own recommendation was insufficient and is rejected here:
keying by username stops an attacker locking out a DIFFERENT account,
but this is a single-admin product with a predictable bootstrap
username, so flooding the operator's own name still locks them out.

This is speculative — it implements a corrected recommendation ahead of
the owner's ruling so the decision can be made by merging or reverting.
Three things are disclosed rather than glossed: online guessing rises
from 5/min to roughly 27/s, because the 429 is a label on the response
and not a gate in front of the hash; the residual exposure is a loss of
login AVAILABILITY, not latency, and a determined flood still denies
login while it runs, at ~400x the cost and clearing the moment it
stops; and the endpoint should be provisioned for ~400 MB resident, not
the 203 MB of live commitment it itemises.

Independently reviewed four times. Reviewers disproved the suspected
FIFO starvation by measurement, then caught two successive memory
bounds the code did not have — the second by parking waiters and
reading the heap rather than checking the arithmetic.
 2026-08-18 01:55:41 +02:00
clawbot
992b3c68f5 Run all linting in Docker via Dockerfile.lint (closes #109)
All checks were successful
check / check (push) Successful in 2m45s
Details 
golangci-lint no longer runs on the host. script/lint builds
Dockerfile.lint, which copies the repo into the digest-pinned linter
image, so the container holds only this repo and the cross-worktree
cache contamination of #106 becomes structurally impossible rather than
filtered after the fact. Five workers hit that contamination in one
evening, in both directions.

Three properties are load-bearing. --no-cache-filter=lint forces the
lint stage to re-execute while deps keeps its cache, because a cached
build lints nothing in 0.27s and exits 0. script/lint does not trust
that flag, since Docker silently ignores an unmatched stage name: it
asserts golangci-lint's own summary line appears, so no summary means no
lint whatever the exit code says. And both lint steps run
--network=none, which enforces rather than assumes that config verify
does not fetch its schema — verify is kept, because golangci-lint run
silently ignores unrecognized config keys and it is the only thing that
catches a typo that disables a setting.

Independently reviewed four times. Three passed the behaviour; the
remaining rounds were a README merge against #151, whose premise was
that the file contains no false statements. Six statements this change
falsified were found and corrected across those rounds — the last
reviewer re-derived every countable claim against the tree rather than
reading for plausibility, and found no seventh.

Supersedes #106.
 2026-08-18 01:07:16 +02:00
clawbot
41ff16a817 Serve an event's full stored body over HTTP (closes #157)
All checks were successful
check / check (push) Successful in 2m44s
Details 
The 8 KB render cap from #135 left storage untouched but no route served
the rest, so a body over the cap was reachable only with filesystem
access to the SQLite files — in a product whose purpose is storing
webhooks so they can be inspected.

GET /source/{sourceID}/logs/{eventID}/body serves the whole body to the
webhook's owner, as application/octet-stream with an attachment
disposition and nosniff. Those are a security control, not formatting:
the bytes come from the public receiver and are handed back inside the
operator's authenticated origin, and the existing CSP would not stop a
stored HTML payload executing there. The truncation marker links to it
only when a body was actually cut.

Accepted deviation, documented rather than glossed: #157's definition of
done asks the route to stream from the row. It buffers whole instead,
because database/sql exposes no incremental handle on a SQLite BLOB and
substr range reads re-materialise the entire column per call — an
earlier revision chunked at 64 KiB and was 11-15x slower for a worse
bound. Three independent reviewers confirmed no streaming path exists.

Independently reviewed three times. Two earlier revisions each asserted
a memory bound the code did not have; the final reviewer measured
2.057x at the ingest cap and pinned the two overlapping allocations from
source — the driver's column buffer and database/sql's convertAssign
clone — confirming the stated "roughly two bodies, and 2x is a floor
not a ceiling" is now accurate, since SQLite's own materialisation sits
outside the Go heap.
 2026-08-18 00:41:31 +02:00
clawbot
5888d14438 Bound the access log line against client-chosen text (closes #146)
All checks were successful
check / check (push) Successful in 2m45s
Details 
The access log wrote one INFO line per request carrying the full
attacker-controlled URL, on the unauthenticated public receiver, so a
client inventing paths wrote unbounded arbitrary text into the
operator's logs.

Rejected requests now log the chi route pattern instead of the concrete
URL — extended to 3xx as well as 4xx, because RequireAuth answers 303
and so /user/<anything> was an unauthenticated path-varying vector. The
query is redacted on the branches that keep a concrete path, and every
client-supplied field is capped: url, useragent and referer at 512
bytes, request_id at 128, method at 32. The caps are spent in ENCODED
bytes, so escaping cannot multiply them.

One INFO line per request, at most 2,560 bytes — a figure derived
arithmetically rather than observed, with the fixed portion measured at
336 (JSON) and 286 (text).

Independently reviewed four times, and broken three of those times on
the same class of defect: a stated bound the code did not have. Round 1
left the 2xx query and the headers unbounded; round 2 counted raw bytes
against an encoded ceiling and broke at 2,611; round 3 charged 6 bytes
for every non-printable when strconv.Quote spells astral ones as
\UXXXXXXXX, and broke at 2,676. Two independent exhaustive audits over
all 1,112,064 code points, built by different methods, now both report
zero undercharged runes on either handler. Measured worst case over a
real TCP socket is 1,972 bytes, 77% of the ceiling.

Follow-up filed to assert that charge against every code point in the
suite, so the ceiling defends itself rather than resting on one
hand-picked rune.
 2026-08-18 00:32:29 +02:00
clawbot
7702f38168 Mark superseded commits honestly instead of skipped (closes #152)
Some checks failed
check / check (push) Superseded by a newer commit; never tested
Gitea records a cancelled run as failure, and the #119 repair rewrote
that to skipped. Gitea's Combine() folds skipped into success, so a
commit nothing ever tested reported a combined green — observed on three
commits on next, including the very change a prior integration review
had failed a PR for.

Superseded commits are now marked failure with an honest description, so
never-tested no longer reads as passed and git bisect archaeology can
tell "passed", "failed" and "never ran" apart. Option 1, re-running the
superseded commit, was verified unreachable for automation on Gitea
1.25.4: no rerun endpoint, dispatches takes a ref not a SHA and lands
under a different context, and CancelPreviousJobs is unconditional.

The rewrite moves out of the workflow into script/ci-mark-superseded so
the tested artifact is the shipped one, and every failure path in it is
loud: an unparseable or empty ANCESTOR_LIMIT, an unreadable ancestor
status, and a shallow clone all abort rather than exiting 0 having
marked nothing. Each has a regression test. The status context is
derived rather than hardcoded, which also closes #147 item 2; item 1
remains open.

Independently reviewed four times. The final reviewer confirmed the
shallow-clone test is genuinely shallow — a file:// URL is load-bearing,
since git silently ignores --depth on a local path — and that deleting
the guard fails that one test out of 331 and cannot pass for the wrong
reason. They also reproduced deterministically that go test's cache
serves a stale PASS after a script-only edit, which internal/ciscript's
doc.go now records.
 2026-08-18 00:31:55 +02:00
clawbot
bef9986542 Set fx.StopTimeout inside the container stop grace (closes #134)
All checks were successful
check / check (push) Successful in 3m3s
Details 
fx defaults to a 15s stop timeout and the Dockerfile sets no grace
override, so Docker SIGKILLed at 10s and the bounded shutdown #130 built
— including the log line that tells an operator a component is wedged —
was unreachable in the image this repo produces.

Sets fx.StopTimeout to 5s, and lowers the HTTP drain to 3s so a
full-length drain no longer exhausts the whole sequence budget and skip
every later hook, database close included. The Sentry flush, which runs
in the same hook and honours no context, is clamped to the remaining
stop budget less a 2s tail reserve, so a stalled flush drops Sentry
events rather than the database close.

Also fixes a latent coin flip in the shared stop-hook waiter, which
reported "shutdown timed out" about half the time for a component that
drained cleanly against an already-expired context.

Independently reviewed three times. The final reviewer derived a
stronger invariant than the implementation claims — the server hook's
absolute end is bounded at stopTimeout minus the reserve regardless of
drain length or of time consumed by preceding hooks — and confirmed the
guard's 10ms sweep cannot step over the maximum, since both breakpoints
land on its grid. Both Sentry probe arms, the docker stop demo and every
mutation were reproduced independently.

Known residual, filed separately: the HTTP drain itself is not clamped
by the reserve, so slow preceding hooks can still jointly exhaust the
budget. Demonstrated with a 2.2s sweeper delay.
 2026-08-18 00:12:51 +02:00
clawbot
c3b6623be1 Bucket IPv6 rate-limit keys by /64 (closes #125)
All checks were successful
check / check (push) Successful in 3m5s
Details 
Rate-limit keys were per-address, i.e. per /128 for IPv6. A routed /64
is the normal residential and mobile allocation, so a client rotated
source addresses inside its own prefix and minted a fresh bucket per
request — evading every limiter at the network layer, with no spoofing
and nothing to detect. #88 closed the header half of this control; this
is the network half.

IPv6 now keys on the /64, IPv4 on the full address, via stdlib
net/netip. IPv4-mapped form is unmapped rather than masked, so clients
behind a mapping proxy do not collapse into one bucket.

Independently reviewed twice. The first round found the trusted-proxy
forwarded path — the one carrying production traffic — had no coverage
at all, so a silent revert there was undetectable; that is now pinned.
The reviewer confirmed both branches are independently mutation-tested:
reverting either the direct-peer return or the forwarded return alone
fails only that branch's tests. The 18-site test-constant refactor was
verified byte-identical against next, with no pre-existing assertion
changed.

Known remaining coverage gap, judged not a defect: the fallback when the
peer is trusted but the forwarded address does not parse has no test.
Only operator-controlled addresses inside TRUSTED_PROXIES reach it, they
already share the proxy's single bucket, and masking there can only
merge operator proxies — fail-closed, nothing attacker-controlled.
 2026-08-17 23:52:15 +02:00
clawbot
39064a3d6c Correct release-blocking README and startup-warning inaccuracies (closes #151)
All checks were successful
check / check (push) Successful in 3m0s
Details 
Publishing this README would have shipped false statements about the
product. Corrects the eight items on the issue plus everything a full
sweep turned up: the Slack circuit-breaker scope, a nonexistent WAL, the
wrong config key for slack targets, six undocumented routes, the
conditional /metrics registration, wrong retention bands, wrong shutdown
mechanism, and a Quick Start that led a new contributor into a red
build.

The lockout warning now fires whenever TRUSTED_PROXIES is empty rather
than only in production, since the variable it was gated on defaults to
dev. Rate-limit keying, the limits and the TRUSTED_PROXIES default are
untouched — those belong to #150.

What /s/* actually serves was settled empirically rather than by
reading: all five of GET/HEAD/POST/PUT/DELETE return 200, pinned by
TestStaticServesEveryMethod. Restricting it is filed separately.

Independently reviewed after three prior rounds. The reviewer
re-derived all fifteen claim-table rows against the code, including
every row a previous revision had marked "correct, left alone" and got
wrong, and found zero false; then verified every route method-by-method,
all twelve environment variables, all nine entity tables, and the
package tree against git ls-files. The Quick Start was confirmed by
running it in a fresh clone.
 2026-08-17 23:44:59 +02:00
clawbot
c378690977 Fetch and verify Alpine at build time instead of committing it (closes #145)
All checks were successful
check / check (push) Successful in 2m58s
Details 
static/js/alpine.min.js was a committed minified bundle, which
REPO_POLICIES forbids, referenced by no content hash at all. A minified
blob is unreviewable, which is the shape a supply-chain compromise
takes.

script/fetch-assets now downloads Alpine 3.14.9 from the npm registry
and verifies sha256 on both the tarball and the extracted file, and
static/vendor_test.go re-hashes the bytes go:embed actually placed in
the binary. The shipped bytes are byte-identical to the blob that was
committed, so the served asset does not change.

Independently reviewed. Five negative controls reproduced by the
reviewer: flipped expected hash, repointed URL, post-fetch tampering,
asset absent, and manifest inconsistencies — each fails closed with
static/js/ left clean. Registry hashes confirmed against the pins, and
the runtime image was built, run and curled to confirm the asset is
still served and the login page still loads it.

Known gap, filed separately: static/static.go embeds the js directory
rather than named files, so a missing fetched asset is not a compile
error on ungated local build paths. Every gated path fails loudly, so
the release artifact is unaffected.
 2026-08-17 23:12:17 +02:00
clawbot
279effb4c2 Bound the event log's rendered bodies in the query (closes #135)
All checks were successful
check / check (push) Successful in 3m0s
Details 
The event log rendered stored bodies untruncated. Since buffered
rendering landed (#123) that became resident memory per concurrent
viewer, up to tens of MB, driven by payloads unauthenticated clients
supply to the public receiver.

Bound in the query rather than the template, via
substr(cast(body as blob), 1, ?) plus length(cast(body as blob)), so an
oversized body never becomes a Go string at all. Adds an EventLogView
projection carrying the true byte count, and trims a partial UTF-8 tail
without rewriting bodies that are merely invalid UTF-8.

Independently reviewed. The generated SQL was dumped under GORM DryRun
to confirm the cap is a bound parameter, both casts are present, and no
other path selects the full column; soft-delete scope, ordering and
pagination are unchanged.

Correction to the PR body: its quoted mutation output was produced by
removing the bound from eventLogColumns, not by raising the cap to
1&lt;&lt;30 as the text claimed. The reviewer reproduced the real
mutation and confirmed the tests do catch removal of the bound.

Follow-up #157 restores in-app retrieval of bodies above the cap.
 2026-08-17 22:57:08 +02:00
clawbot
9ae19159a3 Mask the http target's destination URL in the UI (closes #115)
Some checks failed
check / check (push) Superseded by a newer commit; never tested
The http target's destination URL can itself be a bearer credential, and
the source detail page rendered it in full. Render it through the
existing MaskURL instead, matching the rule already applied to slack
targets.

Independently reviewed: mutation-verified (reverting to the raw value
fails the absence assertions, not merely the masked-form ones), MaskURL
probed against userinfo, query, fragment, port, IPv6 literal and
non-http schemes, and every sibling path that surfaces target data
re-walked and found clean.
 2026-08-17 22:50:26 +02:00
clawbot
2ee720a9af Bound shutdown hooks by their stop context (closes #102)
All checks were successful
check / check (push) Successful in 3m49s
Details
2026-08-14 06:18:33 +02:00
clawbot
0b457ea713 Render templates via a buffer, not the ResponseWriter (closes #123)
Some checks failed
check / check (push) Superseded by a newer commit; never tested
 2026-08-14 06:18:22 +02:00
clawbot
5f18bc3eae Align session codec max-age with the 7-day cap (closes #108)
Some checks failed
check / check (push) Superseded by a newer commit; never tested
 2026-08-14 06:17:43 +02:00
clawbot
d8f9d149b5 Warn when TRUSTED_PROXIES is empty in production (closes #149)
All checks were successful
check / check (push) Successful in 2m40s
Details 
With no trusted proxies configured, every client behind the reverse proxy production requires shares one rate-limit bucket per limit, so five POSTs per minute from anywhere holds the login limit full and denies the admin login until restart. The default is still correct; it was the consequence that was invisible. Startup now warns, and the README no longer claims the login limit is per-IP unconditionally.
 2026-08-12 13:49:39 +02:00
clawbot
339548d794 Record the last four milestone units in TODO.md
All checks were successful
check / check (push) Successful in 6s
Details 
Adds Completed Steps for the receiver aggregate rate limit, the documentation accuracy pass, the CI gate repair and the RETENTION_SWEEP_INTERVAL bound. Drops the commit hash that pinned the Status paragraph to a specific next head, and rewrites Next Step now that the gate repair it named has landed.
 2026-08-12 13:21:38 +02:00
clawbot
95161c7768 Bound the receiver rate limit per client IP across /webhook/* (closes #139)
Some checks failed
check / check (push) Superseded by a newer commit; never tested
The receiver limiter keyed on the request path, and /webhook/{uuid} matches any single segment, so a client minted a fresh bucket per invented path and had unlimited aggregate rate against the only unauthenticated endpoint. An outer limiter keyed on the client address alone now bounds that, chained in front of the unchanged per-entrypoint limiter. Its rejections log at DEBUG without the path, and the README states what each limit does and does not bound.
 2026-08-12 13:19:43 +02:00
clawbot
0e397b3174 Correct release-blocking documentation inaccuracies (closes #141)
Some checks failed
check / check (push) Superseded by a newer commit; never tested
The README env table was missing RETENTION_SWEEP_INTERVAL, TODO.md omitted five landed units, and three passages sold manual redelivery in the present tense when nothing implements it. The same false claim was corrected in the doc comment on failUnretryableRetry, which was its source text. Also removes a console.log from the shipped static asset.
 2026-08-12 13:15:04 +02:00
clawbot
be576096aa Make the CI gate execute the checks it reports on (closes #119)
All checks were successful
check / check (push) Successful in 2m52s
Details 
The workflow writes a build-context fingerprint before calling script/cibuild, so a code commit invalidates the COPY layer of the lint and builder stages and the checks really run, while a docs-only commit still replays from cache. A superseding run also rewrites the exact failure/Has been cancelled status left on commits that were never tested to skipped, so cancellation no longer reads as red. script/cibuild itself is untouched.
 2026-08-12 13:00:51 +02:00
clawbot
3941f0b0ff Require a positive RETENTION_SWEEP_INTERVAL (closes #140)
All checks were successful
check / check (push) Successful in 6s
Details 
A non-positive value reached time.NewTicker in the retention reaper and the archive sweeper, panicking both goroutines after startup had already reported success. envPositiveDuration now rejects it in loadFromEnv, matching how PORT and RECEIVER_RATE_LIMIT fail. SESSION_IDLE_TIMEOUT keeps treating non-positive as disabled, which is guarded at every use site.
 2026-08-12 12:46:39 +02:00
clawbot
543005c0c2 Update TODO.md for the completed 1.0.0 milestone
All checks were successful
check / check (push) Successful in 5s
Details 
Records the trusted-proxy gating, hop cap and bounded scan, and corrects the Workflow section, which still described branching from main and committing TODO.md alongside the work.
 2026-08-12 12:20:34 +02:00
clawbot
9bfd033a29 Bound X-Forwarded-For scanning allocation to the hop cap (closes #133)
Some checks failed
check / check (push) Superseded by a newer commit; never tested
forwardedClientAddr now walks the header values in reverse with strings.LastIndexByte instead of joining and splitting, so allocation is bounded by the 64-hop cap rather than by header length: 1.6 MB per call becomes 16 bytes for a 1 MB chain. Semantics are unchanged, verified by differential testing against the previous implementation.
 2026-08-12 12:19:14 +02:00
clawbot
fd6397154a Cap the X-Forwarded-For hop walk at 64 entries (closes #124)
All checks were successful
check / check (push) Successful in 7s
Details 
The walk now keeps only the rightmost 64 hops, so an attacker-supplied chain cannot burn unbounded CPU in the rate-limit key function. Running off the end of the truncated slice falls back to the peer address, the same fail-closed direction the rest of the function takes. Also corrects the unparseable-RemoteAddr comment, which overclaimed about Unix-socket peers.
 2026-08-12 11:53:48 +02:00
clawbot
d19e33671c Gate forwarded-header trust behind trusted-proxy config (closes #88)
All checks were successful
check / check (push) Successful in 6s
Details 
All three rate limiters (receiver, login, password change) now key on the connection's own address unless the direct peer is inside the new TRUSTED_PROXIES CIDR list, in which case X-Forwarded-For is walked right to left for the first non-proxy hop. Default is the empty list, which trusts nothing. A set-but-unparseable value aborts startup.
 2026-08-12 11:36:10 +02:00
clawbot
aab448b076 Clarify web UI terminology, copy, and the entrypoint URL (closes #57)
All checks were successful
check / check (push) Successful in 4s
Details 
Unifies user-visible copy on "Webhook" (routes and URLs unchanged), drops
the placeholder Profile settings section, and adds a copy-to-clipboard
affordance for the entrypoint URL as progressive enhancement — the button
stays hidden unless both the target element and the Clipboard API resolve,
so no dead control appears without JavaScript and the URL stays selectable.

Retention copy now matches what the code does: deletion is permanent, 0
retains forever, and a blank field means the default on create or the
current value on edit. The permanent-deletion sentence is suppressed for a
retain-forever webhook, which the reaper exempts before computing a cutoff.

Template tests gained a render-completed assertion. Without it, a page that
aborted mid-render still satisfied assertions matching the already-flushed
prefix, because renderTemplate streams to the ResponseWriter (#123).
 2026-08-11 15:42:08 +02:00
clawbot
7c43e095a6 Mask the webhook credential in delivery errors and logs (closes #118)
All checks were successful
check / check (push) Successful in 9s
Details 
Go embeds the request URL in *url.Error, so any transport failure — DNS,
TLS, refused, timeout, SSRF dial block — persisted the full Slack webhook
URL into the per-webhook SQLite database via DeliveryResult.Error. That
field is tagged json:"error,omitempty", so a future REST API would have
served it.

maskURLError rebuilds the error preserving Op and the wrapped cause, so DNS
vs TLS vs timeout still read differently and errors.Is/As and Timeout()
keep working; only path, query and userinfo are dropped. Applied where the
errors are born, which covers both the Slack and HTTP targets. url.Parse
embeds the URL too, so ValidateTargetURL's parse branch gets the same
treatment.

The SSRF rejection log now logs the masked URL, and source_logs.html
receives view types rather than raw rows, so no config blob is reachable
from that template.

MaskURL is now the single masker for the whole tree.
 2026-08-11 15:11:57 +02:00
clawbot
84b758b785 Rate-limit the public webhook receiver endpoint (closes #64)
All checks were successful
check / check (push) Successful in 5s
Details 
The receiver was the one unauthenticated, internet-facing endpoint with no
rate limit, so a misbehaving or hostile sender could flood a webhook
without bound. RECEIVER_RATE_LIMIT (default 120/min) now caps it, keyed on
client IP plus entrypoint path so one entrypoint cannot exhaust another's
budget. Over-limit requests get 429 with Retry-After.

The limiter deliberately does not reuse postRateLimit: that helper is
POST-only and keys on IP alone, whereas the receiver must count every
method. A test locks that property in.

Config parsing follows the fail-loudly idiom: a set-but-unparseable or
non-positive value aborts startup rather than falling back to the default.

Known limitation, tracked in #88: the key still trusts forwarded headers
unconditionally, so the limit is evadable by rotating X-Forwarded-For until
trusted-proxy gating lands.
 2026-08-11 14:47:21 +02:00
clawbot
d51cd0fd29 Enforce the body size limit before CSRF parses the form (closes #90)
Some checks failed
check / check (push) Superseded by a newer commit; never tested
CSRF ran before MaxBodySize, so the CSRF middleware parsed the form body
before any cap applied and an oversized request was read in full before
being rejected. MaxBodySize is now the first middleware in all four route
groups that parse forms, ahead of CSRF and RequireAuth.

An oversize request therefore gets 413 without the handler running and
without state changing, including the password-change route.

Note the ordering trade: an unauthenticated client now receives 413 rather
than an auth redirect on /user/{username}/password.
 2026-08-11 14:37:38 +02:00
clawbot
15a61173fc Mask target config on the source detail page (closes #113)
Some checks failed
check / check (push) Superseded by a newer commit; never tested
The page rendered the stored target config verbatim, exposing the Slack
incoming-webhook URL, which is a bearer credential: anyone holding it can
post to the channel indefinitely, and it cannot be scoped or revoked
per-holder.

Target config now reaches the template only as a TargetView carrying
labelled fields, so no code path can render the raw blob. maskURL keeps
scheme and host and elides the path, and drops query, fragment and
userinfo; every parse failure yields a neutral placeholder rather than
falling back to the stored string. HTTP header values are never rendered,
only a count.

Rendering change only: the stored config format and the delivery path are
unchanged.
 2026-08-11 14:37:09 +02:00
clawbot
e50a79ced9 Allow retention_days of 0 to mean retain forever (closes #79)
Some checks failed
check / check (push) Superseded by a newer commit; never tested
Rewrites retention_days=0 to the RetentionForeverDays sentinel (365 * 1000)
in Webhook.BeforeSave, so the GORM column default cannot win the race. The
reaper skips retain-forever webhooks before building any query.

Also bounds the reaper's cutoff arithmetic: a time.Duration is int64
nanoseconds, so day counts above MaxFiniteRetentionDays (106751) overflowed
and wrapped the cutoff into the future, where created_at &lt; cutoff matched
every row and the sweep deleted everything. parseRetentionDays now rejects
finite values above the ceiling, and retentionCutoff saturates so rows
written by older versions cannot reach it either.

Views render RetentionLabel() rather than the raw sentinel.
 2026-08-11 14:35:34 +02:00
clawbot
c2cd2c440b Add inactivity-based session timeout (closes #66) (#105)
All checks were successful
check / check (push) Successful in 4s
Details 
Sessions now carry a server-enforced idle deadline (SESSION_IDLE_TIMEOUT,
default 24h) alongside the 7-day absolute cap, refreshed on authenticated
activity. Activity never extends the absolute cap.
 2026-08-10 16:12:40 +02:00
clawbot
45890d4f82 Fail loudly on set-but-unparseable env config values (closes #80) (#92)
All checks were successful
check / check (push) Superseded by a newer commit; never tested
Defaults now apply only to unset or empty environment variables; a set-but-
unparseable value aborts startup with an error naming the key and the value.
envInt is gone, envBool parses with strconv.ParseBool, and PORT is bounded.
 2026-08-10 16:06:12 +02:00
clawbot
0ce8565f51 Terminally fail retrying deliveries with a non-retry target type (closes #82) (#104)
All checks were successful
check / check (push) Superseded by a newer commit; never tested
A delivery left in `retrying` whose target type was edited to a fire-and-forget
or unknown type was skipped forever by both restart recovery and the retry
sweep. Both paths now record a result row and mark it `failed`.
 2026-08-10 16:00:03 +02:00
clawbot
3e261d2f01 Evict archive writers on deletion and sweep idle archives (closes #89) (#95)
All checks were successful
check / check (push) Superseded by a newer commit; never tested
Per-webhook archive writers are now evicted when the webhook or its last
database target is deleted, and a background sweeper prunes expired rows from
idle archives that no longer receive writes. Archive files themselves are never
deleted.
 2026-08-10 15:52:20 +02:00
clawbot
62481a6f1a Root background loops at context.Background() (closes #97) (#100)
All checks were successful
check / check (push) Successful in 5s
Details 
The delivery engine worker pool and the retention reaper both rooted their
goroutines in the fx OnStart hook context, which fx cancels 15s into startup.
Both now use context.WithCancel(context.Background()), bounded by OnStop.
 2026-08-10 15:44:56 +02:00
clawbot
4f5ecb18e5 Add admin password change flow (closes #65) (#83)
All checks were successful
check / check (push) Superseded by a newer commit; never tested
Adds an authenticated, CSRF-protected flow that lets a user change their own password from the profile page.

## Route

- New `POST /password` under the `/user/{username}` group in `setupUserRoutes` (`internal/server/routes.go`). That group already applies `CSRF`, `NoCache`, and `RequireAuth`, so the new endpoint inherits all three.

## Handler (`internal/handlers/profile.go`)

- `HandlePasswordChange` enforces own-user access: the `{username}` path parameter must equal the session username (same 403 rule `HandleProfile` uses). This check plus the session lookup is factored into a shared `profileOwnerOrDeny` helper now used by both handlers.
- Parses `current_password`, `new_password`, and `confirm_password` (body size limited via `http.MaxBytesReader`).
- Verifies the current password with `database.VerifyPassword` against the stored hash.
- Requires the new password to be non-empty and equal to the confirmation.
- Hashes the new password with `database.HashPassword` — the same Argon2id helper used to bootstrap the admin user — and persists it on the user row. No new crypto.
- Re-renders the profile page with a clear success or error message. Wrong current password, empty new password, and mismatched confirmation are each rejected with their own message and leave the stored hash unchanged.

## Template (`templates/profile.html`)

- Adds a "Change Password" card with current / new / confirm password fields plus the hidden `csrf_token` (matching the login form's CSRF embedding).
- Renders success/error alerts using the existing `alert-success` / `alert-error` styles. No new CSS classes, so no Tailwind rebuild is required.

## Tests (`internal/handlers/profile_test.go`)

- `TestHandlePasswordChange_Success`: seeds a user, posts a valid change, asserts success message and that the stored hash changed and verifies against the new password.
- `TestHandlePasswordChange_WrongCurrentPassword`: posts a wrong current password, asserts the rejection message and that the stored hash is unchanged.

Validated with `docker build .` (fmt-check, lint, test, build) — exit 0.

Closes #65

Co-authored-by: sneak <sneak@sneak.berlin>
Co-authored-by: Jeffrey Paul <sneak@noreply.example.org>
Reviewed-on: #83
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
 2026-08-07 23:23:05 +02:00
clawbot
734606b7af Update golangci-lint to v2.12.2 with canonical config (#86)
All checks were successful
check / check (push) Successful in 3s
Details 
Bumps golangci-lint from v2.11.3 to v2.12.2 and adopts the canonical lint config.

## Version pins

- `Dockerfile`: `golangci/golangci-lint:v2.12.2` Debian image, pinned by digest, dated `2026-08-07`
- `script/bootstrap`: `GOLANGCI_LINT_VERSION=2.12.2` with updated sha256 pins for the `linux-amd64` and `linux-arm64` release archives

## Config

`.golangci.yml` replaced with the canonical config. The previous file kept `lll`/`funlen`/`cyclop`/`dupl` settings under the top-level `linters-settings` key, which the v2 schema ignores; the canonical config nests them under `linters.settings`, so those thresholds now actually apply. The unsupported `issues.exclude-use-default` key was dropped.

## Lint fixes (32 findings)

- `lll` (7): wrapped or shortened over-length lines (struct tag comments moved above fields, test logger construction split, `session.NewForTest` signature wrapped, shortened a `#nosec` comment)
- `goconst` (17): replaced repeated `"POST"`/`"PUT"` literals with `http.MethodPost`/`http.MethodPut`, added shared test constants for `webhooker-test`/`test`/`application/json`, and added `tmplKeyError`/`tmplKeyWebhook` constants for template data keys in `internal/handlers`
- `dupl` (8): merged `buildHTTPTargetConfig` and `buildSlackTargetConfig` into a parameterized `buildURLTargetConfig`; removed the duplicate `iWebhookDB` test helper in favor of `testWebhookDB`; extracted shared helpers in middleware and session tests

No `//nolint` directives were added and behavior is unchanged. `make check` (fmt-check, tests, lint) passes.

Note: golangci-lint v2.12 deprecates the `gomodguard` linter in favor of `gomodguard_v2`; the canonical config change for that is left for a future coordinated update.
Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #86
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
 2026-08-07 23:18:49 +02:00
clawbot
ee7c626071 Implement the database archiving target (closes #43) (#84)
All checks were successful
check / check (push) Successful in 4s
Details 
Implements the `databaseTarget` as a real archiving target, replacing the always-successful stub. Delivering to a `database` target now writes the full event into a per-webhook archive SQLite file for long-term storage.

## Archive-writer semantics

- **Separate file:** each webhook's full events are written as rows into `archive-{webhookID}.db` under the data dir, distinct from the per-webhook event DB (`events-{webhookID}.db`). The file and its schema are created on first write if missing. Each row carries the full event: body, headers, method, content type, webhook id, entrypoint id, event id, and an archived-at timestamp.
- **Close/reopen with debounce:** after each write the archive handle is closed and reopened, unless the last (re)open was less than one second ago. This lets an operator move the archive file away for offline archiving while bounding file churn under load. A per-webhook `archiveWriter` owns this debounce state and serialises writes.
- **Auto-recreate:** the file is opened create-if-missing (`mode=rwc`) and its schema re-migrated on every open, so if the archive was moved or removed since the last open, the next write recreates it. The writer also detects a missing file before writing and reopens first, so a moved-away file is recreated rather than lost.
- **Optional expiry, validated at creation:** an optional `expiry` in the target's config JSON (e.g. `{"expiry":"720h"}`) is validated when the target is created (`ValidateArchiveExpiry`; bad values are rejected with a 400 at the add-target form, the Slack URL precedent). The default (missing, empty, or `"never"`) keeps rows forever with no pruning. When a positive duration is set, rows older than it (measured from each row's archived-at time) are pruned on every (re)open; because the file is reopened after writes, prune-on-open keeps the archive swept without a separate background sweeper. A set-but-invalid expiry in a stored config (unparseable, zero, or negative) is an error at delivery time too — never a silent default.
- **No-retry, fail-loud:** the target performs a single attempt with no retries. On success it records one successful attempt and marks the delivery delivered. If the archive write fails, the attempt is recorded as failed with the error and the delivery is marked failed — archiving errors never report success.

## Scope

- `internal/delivery/target_database.go` — the `databaseTarget` (no-retry) archives via a per-webhook writer registry; an archive error records a failed attempt and marks the delivery failed.
- `internal/delivery/target_database_archive.go` (new) — the `archiveWriter`, the archived-row model, config/expiry parsing (fail-loud on set-but-invalid values), `ValidateArchiveExpiry`, and prune-on-open.
- `internal/handlers/source_management.go` — database targets get a creation-validated `expiry` config (`buildDatabaseTargetConfig`); the expiry form value is read where the request body is bounded and bad values are rejected with a 400 at target creation.
- `templates/source_detail.html` — the add-target form shows an expiry field for database targets.
- `README.md` — the database-target documentation describes the archiving semantics.
- `internal/delivery/export_test.go`, `internal/delivery/target_database_test.go`, `internal/handlers` tests — tests and their exported shims.

No changes to the `Target` interface or other targets.

## Tests

- a row is archived (both at the writer level and end-to-end through `Deliver`)
- a forced archive failure (bad stored expiry config) yields a `Failed` delivery with a non-success `DeliveryResult` carrying the error and no archive file created
- the file is recreated after removal, with only the post-removal row
- the one-second reopen debounce (rapid writes reopen once; a write after the window reopens again)
- expiry pruning removes rows older than the configured expiry
- expiry config parsing (empty / `never` / duration accepted; unparseable, zero, and negative values error)
- expiry validation at target creation (`TestValidateArchiveExpiry`; valid values build the config, bad values get a 400)

## Validation

`docker build .` exits 0 (fmt-check, lint, test, build all pass).

Closes #43

Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #84
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
 2026-08-07 22:50:08 +02:00
clawbot
81413c56e9 Refactor delivery targets to a Target interface (closes #77) (#81)
All checks were successful
check / check (push) Successful in 2m42s
Details 
Refactors the delivery engine so each target TYPE is an implementation of a `Target` interface, dispatched from a registry, with each target owning its full delivery including durable retries. Implements the authoritative design from issue #77 (the corrected "hand the DB + Scheduler to the target" design).

## The new interface

```go
type Scheduler interface {
    ScheduleRetry(task Task, delay time.Duration)
}

type Target interface {
    Deliver(ctx context.Context, webhookDB *gorm.DB,
        d *database.Delivery, task *Task, sched Scheduler)
}
```

`Deliver` receives everything a target needs to be autonomous and durable: the request context, the per-webhook `*gorm.DB`, the `*database.Delivery`, the attempt `*Task`, and a `Scheduler` (the engine) for durable re-enqueue. The target makes one attempt, writes the `DeliveryResult`, updates `DeliveryStatus`, and — for retry targets — decides whether to retry, computes its own backoff, gates with its own circuit breaker, and reschedules via the injected `Scheduler`.

`processDelivery` collapses to a registry lookup (`map[database.TargetType]Target`) and a `Deliver` call; an unknown target type still fails the delivery as before.

## Per-target ownership

- `httpTarget` and `slackTarget` share a retry core (`httpCore`) that owns retry, exponential backoff, and the per-target circuit breaker. The core is fire-and-forget when `MaxRetries == 0` and adds breaker-gated backed-off retries when `MaxRetries > 0`. The per-attempt request differs (HTTP forwards the body + filtered headers; Slack posts a formatted message) and is supplied as a closure, so each keeps its exact recording semantics (e.g. HTTP records no error string for a non-2xx, Slack records `HTTP <code>`).
- `databaseTarget` and `logTarget` are fire-and-forget: they record a single successful attempt.

Moved wholesale into the http/slack targets: `deliverHTTP*`, `handleHTTPRetry`, `circuitBreakerBlock`, `calcBackoff` / `calcRemainingBackoff` / `backoffElapsed`, the circuit-breaker `sync.Map` + `getCircuitBreaker`, `clientForConfig`, `doHTTPRequest`, `applyRequestHeaders`, and the config parsers. The engine keeps `recordResult`, `updateDeliveryStatus`, and `ScheduleRetry`.

## Slack MaxRetries gating

Slack is now on the same shared core as HTTP, with retry + breaker gated on `MaxRetries`. A `MaxRetries` of 0 stays single-attempt fire-and-forget, so **every existing Slack target is unchanged**; a Slack target configured with retries gets backoff + circuit breaker.

## Log-target full content

`logTarget` now logs the ENTIRE inbound webhook — full request body and full request headers, plus method, content type, and the webhook id and entrypoint id — rather than a summary line. This supersedes the smaller log-summary work (#70).

## `Task.EntrypointID`

To carry the entrypoint id to the log target, `Task` gains an `EntrypointID` field, populated in the webhook handler's `buildDeliveryTasks`, the engine's recovery-task builder, and `buildEventFromTask`.

## Durability / recovery

The crash-durable async retry model is preserved unchanged: one attempt per worker turn; on failure the status is set `retrying`, backoff is computed, and the task is re-enqueued via `ScheduleRetry` (a `time.AfterFunc` onto the retry channel). On restart, `recoverRetryingDeliveries` and the 60s sweep hand each orphaned `retrying` delivery back to its target to recompute the remaining backoff and reschedule (targets that own retries implement an internal `rescheduler`; fire-and-forget targets, which never produce `retrying` deliveries, are skipped).

## How behaviour is preserved

No external behaviour changes except the two called out above (log target full content; Slack gaining `MaxRetries`-gated retries). All existing delivery tests pass with only their `export_test.go` wrappers re-pointed at the new structure — `ExportDeliverHTTP/Slack/Database/Log` now call the targets, `ExportGetCircuitBreaker` / `ExportClient` / `ExportClientForConfig` / `ExportDoHTTPRequest` resolve against the HTTP target's shared client and breaker map, and `ExportParseHTTPConfig` / `ExportParseSlackConfig` call the relocated free functions. Added: a `logTarget` test asserting the log line contains the full body, headers, and ids, and a Slack `MaxRetries`-gated retry test.

`docker build .` is green (fmt-check, lint, test, static build all pass).

Closes #77

Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #81
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
 2026-08-07 17:07:49 +02:00
clawbot
f6b929f2d7 Add per-webhook event retention reaper (closes #63) (#78)
All checks were successful
check / check (push) Successful in 2m42s
Details 
Enforces each webhook's `RetentionDays` so per-webhook SQLite files no longer grow without bound.

## Reaper

New `RetentionReaper` in `internal/database/retention.go`. A background ticker runs each sweep: it lists all webhooks from the main DB and, for each webhook with a positive `RetentionDays`, opens its per-webhook DB via `WebhookDBManager.GetDB` and deletes every `Event` (and its dependent `Delivery` and `DeliveryResult` rows) whose `CreatedAt` is older than `RetentionDays` days.

- Deletions run in foreign-key-safe order: delivery results, then deliveries, then events.
- Deletes are unscoped (hard deletes) so rows are physically removed and disk is reclaimed, rather than GORM soft-deleting them.
- `RetentionDays <= 0` means retain forever; those webhooks are skipped.
- Webhooks whose per-webhook DB does not yet exist are skipped.

## Config

`internal/config/config.go` gains `RetentionSweepInterval` (env `RETENTION_SWEEP_INTERVAL`, parsed as a Go duration, default `1h`) via a new `envDuration` helper, following the existing env-helper conventions.

## Wiring

`cmd/webhooker/main.go` registers `database.NewRetentionReaper` as an fx provider and forces its construction in `fx.Invoke`. The reaper starts its sweep loop on an fx `OnStart` hook and stops cleanly on `OnStop` via context cancellation, matching the existing lifecycle components.

## Test

`internal/database/retention_test.go` seeds an old event chain (event + delivery + result, 40 days old) and a recent one (1 day old) in a real per-webhook DB and asserts a single sweep removes only the expired chain while keeping the recent one. A second test forces a non-positive `RetentionDays` and asserts an ancient event is retained.

Note: the `Webhook.RetentionDays` column carries `gorm:"default:30"`, so a `0` passed to a GORM `Create` is replaced by the default; the test forces the value with an explicit column update to exercise the retain-forever path. No model changes were made.

Validated with `docker build .` (fmt-check, lint, test, build) exit 0.

Closes #63

Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #78
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
 2026-08-07 16:15:13 +02:00
clawbot
8ea7f76540 Add NoCache middleware for authenticated pages (closes #61) (#75)
All checks were successful
check / check (push) Successful in 6s
Details 
Adds a `NoCache()` middleware that sets `Cache-Control: no-store` and `Pragma: no-cache`, and wires it onto the dynamic app route groups (`/pages`, `/user/{username}`, `/sources`, `/source/{sourceID}`) adjacent to their existing `CSRF()` call. The static `/s` mount, `/metrics`, `/webhook/{uuid}`, and `/.well-known/healthcheck` are intentionally left untouched (static assets are safe to cache; the others are not authenticated pages).

A middleware unit test asserts both headers are set.

Closes #61

Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #75
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
 2026-08-07 15:33:44 +02:00
129 changed files with 26857 additions and 1748 deletions
Expand all files Collapse all files

8
.dockerignore
View File

@@ -1,5 +1,13 @@
# .ci-fingerprint is deliberately NOT excluded: it is the CI cache barrier
# that keeps the check stages from replaying a cached pass. See the lint
# stage of the Dockerfile.
.git/
bin/
# Third-party browser assets are fetched and hash-verified inside the build by
# script/fetch-assets. Excluding any host copy keeps a developer's working tree
# from supplying the bytes that get shipped. The script and its
# static/vendor.sha256 manifest stay in the context.
static/js/alpine.min.js
*.md
LICENSE
.editorconfig

28
.gitea/workflows/check.yml
View File

@@ -11,5 +11,33 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 2024-10-23
with:
# The fingerprint step below needs history to find the last commit
# that touched the Docker build context, and the superseded-status
# step needs it to walk ancestors (it aborts on a shallow clone).
fetch-depth: 0
- name: Mark superseded run statuses
# Gitea cancels the in-flight run when another commit is pushed to the
# same branch and records the cancellation as `failure`, so a commit
# that was never tested reads as a test result. The script rewrites
# those statuses to say what happened. See its header for why the
# state stays `failure` and not `skipped`.
env:
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
run: script/ci-mark-superseded
- name: Fingerprint the build context
# `.dockerignore` keeps docs out of the build context, so a docs-only
# commit legitimately replays the whole image from cache and stays
# cheap. Every other commit writes a new fingerprint into the context,
# which invalidates the `COPY . .` layer of both check stages: a
# commit that was never linted, formatted-checked, tested and built
# cannot report success from cache.
run: |
set -eu
fp="$(git log -1 --format=%H -- . ':!*.md' ':!LICENSE' ':!.editorconfig')"
printf '%s\n' "${fp:-$GITHUB_SHA}" > .ci-fingerprint
- name: Build Docker image (runs make check)
run: script/cibuild

8
.gitignore vendored
View File

@@ -42,3 +42,11 @@ data/
# Temporary files
tmp/
temp/
# CI cache barrier, written into the build context by the check workflow
.ci-fingerprint
# Third-party browser assets, fetched and hash-verified by
# script/fetch-assets against static/vendor.sha256. Not committed:
# REPO_POLICIES.md forbids minified bundles in version control.
/static/js/alpine.min.js

8
.golangci.yml
View File

@@ -1,5 +1,9 @@
version: "2"
# Config schema uses the golangci-lint v2 layout (settings live under
# linters.settings, not top-level linters-settings) so that the
# thresholds below are actually applied by golangci-lint >= v2.
run:
timeout: 5m
modules-download-mode: readonly
@@ -14,8 +18,7 @@ linters:
- wsl # Deprecated, replaced by wsl_v5
- wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go
linters-settings:
settings:
lll:
line-length: 88
funlen:
@@ -27,6 +30,5 @@ linters-settings:
threshold: 100
issues:
exclude-use-default: false
max-issues-per-linter: 0
max-same-issues: 0

34
Dockerfile
View File

@@ -1,8 +1,8 @@
# Lint stage
# golangci/golangci-lint:v2.11.3 (Debian-based), 2026-03-17
# golangci/golangci-lint:v2.12.2 (Debian-based), 2026-08-07
# Using Debian-based image because mattn/go-sqlite3 (CGO) does not
# compile on Alpine musl (off64_t is a glibc type).
FROM golangci/golangci-lint:v2.11.3@sha256:e838e8ab68aaefe83e2408691510867ade9329c0e0b895a3fb35eb93d1c2a4ba AS lint
FROM golangci/golangci-lint:v2.12.2@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
RUN apt-get update && apt-get install -y --no-install-recommends make && rm -rf /var/lib/apt/lists/*
@@ -12,12 +12,21 @@ WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
# Copy source code
# Copy source code. In CI the context also carries .ci-fingerprint, whose
# value changes with every commit that touches the build context (see
# .gitea/workflows/check.yml). That invalidates this layer, so the checks
# below cannot report success by replaying a cached pass. Do not add it to
# .dockerignore.
COPY . .
# Run formatting check and linter
# Run formatting check and linter. golangci-lint is invoked directly rather
# than through `make lint`: this stage is already the pinned linter image, and
# script/lint is a wrapper that builds Dockerfile.lint, so calling it here
# would need a docker daemon inside the build. Keep these steps in step with
# Dockerfile.lint, including --network=none (see its header for why).
RUN make fmt-check
RUN make lint
RUN --network=none golangci-lint config verify --config .golangci.yml
RUN --network=none golangci-lint run --config .golangci.yml ./...
# Build stage
# golang:1.26.1-bookworm (Debian-based), 2026-03-17
@@ -28,7 +37,9 @@ FROM golang:1.26.1-bookworm@sha256:4465644228bc2857a954b092167e12aa59c006a349228
# Depend on lint stage passing
COPY --from=lint /src/go.sum /dev/null
RUN apt-get update && apt-get install -y --no-install-recommends make && rm -rf /var/lib/apt/lists/*
# jq is a runtime dependency of script/ci-mark-superseded, which the test
# suite executes.
RUN apt-get update && apt-get install -y --no-install-recommends make curl ca-certificates jq && rm -rf /var/lib/apt/lists/*
WORKDIR /build
@@ -36,9 +47,18 @@ WORKDIR /build
COPY go.mod go.sum ./
RUN go mod download
# Copy source code
# Copy source code, including the .ci-fingerprint cache barrier described in
# the lint stage above.
COPY . .
# Fetch the third-party browser assets the UI serves. They are not committed
# (REPO_POLICIES.md forbids minified bundles in version control) and
# .dockerignore keeps any host copy out of the build context, so this step is
# the only way they enter the image. Each download is checked against a
# hardcoded sha256 and the build fails on mismatch; make test re-checks the
# hashes against the bytes go:embed actually put in the binary.
RUN script/fetch-assets
# Run tests and build
RUN make test
RUN make build

37
Dockerfile.lint Normal file
View File

@@ -0,0 +1,37 @@
# Lint-only image, built by script/lint. golangci-lint is never installed on
# the host: the repo is COPYed into the pinned image and linted as a build
# step, so a successful build IS a clean lint. This works even when the docker
# daemon is remote and bind mounts are impossible.
#
# script/lint passes --no-cache-filter=lint. Without it an unchanged tree
# replays the lint stage from cache and the build succeeds in under a second
# having run no linter at all. Do not drop that flag.
#
# The lint steps run with --network=none. `golangci-lint config verify` is
# documented as fetching its JSON schema over HTTPS, which would make linting
# depend on an unpinned remote artifact; this pinned image resolves the schema
# without any network, and --network=none enforces that rather than trusting
# it. It also proves no linter reaches out at analysis time. If a future image
# bump makes either step need the network, this build fails loudly instead of
# quietly acquiring an unpinned dependency.
# golangci/golangci-lint:v2.12.2 (Debian-based), 2026-08-07
# Using Debian-based image because mattn/go-sqlite3 (CGO) does not
# compile on Alpine musl (off64_t is a glibc type).
FROM golangci/golangci-lint:v2.12.2@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS deps
WORKDIR /src
# Copy go mod files first for better layer caching. This stage is cacheable;
# only the lint stage below is forced to re-execute.
COPY go.mod go.sum ./
RUN go mod download
FROM deps AS lint
COPY . .
# `run` silently ignores config keys it does not recognize, so a typo would
# disable a setting without a word. `config verify` is what catches that.
RUN --network=none golangci-lint config verify --config .golangci.yml
RUN --network=none golangci-lint run --config .golangci.yml ./...

5
Makefile
View File

@@ -1,4 +1,4 @@
.PHONY: bootstrap setup test lint fmt fmt-check check build run dev deps docker clean hooks css
.PHONY: bootstrap setup assets test lint fmt fmt-check check build run dev deps docker clean hooks css
# Default target
.DEFAULT_GOAL := check
@@ -9,6 +9,9 @@ bootstrap:
setup:
@script/setup
assets:
@script/fetch-assets
test:
@script/test

1307
README.md
View File

File diff suppressed because it is too large Load Diff

237
TODO.md
View File

@@ -1,33 +1,223 @@
# Workflow
* branch (from `main`)
* do the work in Next Step
* move Next Step to the top of Completed Steps
* move the top item of Future Steps into Next Step
* commit (`TODO.md` changes in the same commit as the work)
* merge to `main` if the branch is not protected, otherwise open a PR
* push
One issue per unit of work, one branch and one PR per issue:
* ensure a tracked issue exists with a definition of done
* branch from `next` (never from `main`)
* do the work; open a PR based on `next` (never on `main`)
* pass an independent review, then the manager squash-merges into `next`
* push; nothing stays local-only
`next` is the branch for the next milestone and must stay green and
mergeable to `main` without notice. One `next` -> `main` PR accumulates
the milestone; releases are cut from `main` separately.
Issue branches do NOT touch this file — the manager maintains it on
`next`. Every branch editing `TODO.md` conflicts with every other
(#112).
# Status
pre-1.0. No git tags exist. main (afe88c6) is a working webhook proxy
pre-1.0. No git tags exist. `main` (4f5ecb1) is a working webhook proxy
with auth, CSRF/SSRF protections, login rate limiting, Slack target,
policy compliance (#6), and pinned lint tooling (#55). Note: TODO.md was
deliberately deleted from this repo in f9a9569 (2026-03-01, #6); its
content was folded into the README TODO section, which this draft
reconstructs as of 2026-07-06.
event retention (#63), the database archiving target (#43), the admin
password change flow (#65), policy compliance (#6), pinned lint tooling
(#55), and fail-loud configuration parsing (#80).
`next` holds the 1.0.0 milestone less its final four issues (#176, #178,
#186, #187 — all in review or held on merge order), and is verified
green by cache-defeated container runs
(`docker build --no-cache-filter=lint --no-cache-filter=builder`). The
CI status is not independently claimed here: a superseded run is
recorded as `skipped` and still rolls up green, so a commit status on
`next` does not by itself evidence an executed check (#152). Before
#119, a warm layer cache also let the gate report success without
executing anything, and replayed the previous build's console log so
the lie looked like a real run. Note: `TODO.md` was deliberately
deleted from this repo in f9a9569 (2026-03-01, #6); its content was
folded into the README TODO section, which this draft reconstructs as
of 2026-07-06.
# Next Step
Implement automatic event retention cleanup based on retention_days: a
periodic maintenance job that deletes Events, Deliveries, and
DeliveryResults older than the parent webhook's retention_days from each
per-webhook event database. The field exists on the Webhook model and
the README promises the behavior, but nothing enforces it, so event
databases currently grow without bound.
Land the last four 1.0.0 issues, then merge the milestone PR to `main`
and tag 1.0.0 from it. Merge order is forced by a real conflict on
`README.md` and `internal/middleware/middleware.go`: #186, then #176,
then #178, then #187.
Two items belong to the owner, neither blocking the tag. #150 was
decided by the manager rather than left to stall the queue and is
flagged on the issue for reversal if that call was wrong. #112 (whether
`Completed Steps` should exist at all, given it once conflicted on every
unit) is unanswered; the provisional ruling in force is that issue
branches do not touch this file.
# Completed Steps
- 2026-08-18 Send the chi route pattern to Sentry rather than the
concrete path. The receiver's path carries the entrypoint capability
token, so every Sentry event from `/webhook/{uuid}` shipped a live
credential to a third party. Request `Data`, `QueryString`, `Cookies`
and `Env` are dropped and headers reduced to an allowlist (#179)
- 2026-08-18 Read form fields from the POST body only. `r.FormValue`
merges the query string, so a login could be driven by URL parameters
— putting the password somewhere that lands in access logs, proxy
logs and browser history (#160)
- 2026-08-18 Verify login credentials before spending rate-limit
budget, so a flood of wrong passwords cannot lock out the account it
is guessing at. The manager took this decision rather than stall the
queue; it is flagged on the issue for reversal (#150)
- 2026-08-18 Run all linting in Docker via `Dockerfile.lint`. Host lint
was wrong in both directions from version skew and shared caches.
`script/lint` asserts the summary line, because `--no-cache-filter`
silently ignores a stage name it does not match — the flag that makes
the gate meaningful fails open (#109)
- 2026-08-18 Serve an event's full stored body over HTTP. The list
query truncates for rendering, and that truncated value was the only
way to read a body, so the full payload was unreachable (#157)
- 2026-08-18 Bound the access log line against client-chosen text.
`internal/logfield` budgets by *encoded* bytes, not runes, so a
handler's JSON escaping cannot multiply a field past its allowance
(#146)
- 2026-08-18 Mark superseded CI commits `failure` rather than
`skipped`. A skipped run rolls up green, so a commit that was never
tested reported success (#152)
- 2026-08-18 Set `fx.StopTimeout` inside the container stop grace, so
shutdown hooks are bounded by a deadline the orchestrator will
actually honour rather than being killed mid-flush (#134)
- 2026-08-17 Bucket IPv6 rate-limit keys by `/64`. A single allocation
hands out 2^64 addresses, so per-address keying let one client mint
unlimited buckets. Manager decision, recorded on the issue (#125)
- 2026-08-17 Correct release-blocking README and startup-warning
inaccuracies, including claims about behaviour the code does not have
(#151)
- 2026-08-17 Fetch and verify Alpine.js at build time against
`static/vendor.sha256` instead of committing the minified blob, so
the dependency is pinned by hash rather than by trust (#145)
- 2026-08-17 Bound the event log's rendered bodies in the query itself,
so a large stored payload cannot be read into memory just to be
truncated for display (#135)
- 2026-08-17 Mask the `http` target's destination URL in the UI: it can
carry a bearer credential in its path or query, and was rendered
verbatim. Manager decision to mask unconditionally (#115)
- 2026-08-14 Bound shutdown hooks by their stop context, so a hook that
hangs cannot hold the process past its grace period (#102)
- 2026-08-14 Render templates via a buffer rather than the
`ResponseWriter`, so a template error part-way through cannot commit
a 200 and then fail — the response is written only once it is whole
(#123)
- 2026-08-14 Align the session codec's max-age with the 7-day absolute
cap. The codec accepted cookies the session layer considered expired,
so the cap was enforced in one place and not the other (#108)
- 2026-08-12 Warn when `TRUSTED_PROXIES` is empty in production, where
the safe default silently discards forwarded headers and every client
rate-limits as the proxy's address (#149)
- 2026-08-12 Bound the receiver rate limit per client IP across the
whole `/webhook/*` route. The existing limiter keyed on the request
path and `/webhook/{uuid}` matches any single segment, so a client
that invented a fresh path per request minted a fresh bucket per
request: the limit on the only unauthenticated endpoint bounded
nothing in aggregate, and every request still cost an entrypoint
lookup before it 404ed. An outer limiter keyed on the client address
alone now bounds that, chained in front of the unchanged
per-entrypoint limiter (#139)
- 2026-08-12 Correct release-blocking documentation inaccuracies: the
README promised manual redelivery in the present tense in three
places when nothing implements it (the same false claim also sat in
the doc comment that was its source text), the env table omitted
`RETENTION_SWEEP_INTERVAL`, and `TODO.md` itself omitted five landed
units (#141)
- 2026-08-12 Make the CI gate execute the checks it reports on. The
workflow now writes a build-context fingerprint before calling
`script/cibuild`, so a code commit invalidates the `COPY` layer of
the lint and builder stages while a docs-only commit still replays
from cache; a superseding run also rewrites the `failure` status
Gitea leaves on commits it cancelled and never tested. Verified by
pushing a deliberately broken test and watching CI go red (#119)
- 2026-08-12 Require a positive `RETENTION_SWEEP_INTERVAL`: a
non-positive value reached `time.NewTicker` in both the retention
reaper and the archive sweeper, panicking two goroutines with no
recover after startup had already reported success (#140)
- 2026-08-12 Bound the `X-Forwarded-For` scan's allocation to the hop
cap: the reverse walk cuts entries with `strings.LastIndexByte`
instead of joining and splitting, so a 1 MB header allocates 16 bytes
rather than 1.6 MB per request on the unauthenticated receiver.
Semantics proven unchanged by differential testing against the
previous implementation (#133)
- 2026-08-12 Cap the `X-Forwarded-For` hop walk at 64 entries, so an
attacker-supplied chain cannot burn unbounded CPU in the rate-limit
key function; running off the end falls back to the peer address
(#124)
- 2026-08-12 Gate forwarded-header trust behind a `TRUSTED_PROXIES` CIDR
list: all three rate limiters key on the connection's own address
unless the direct peer is a configured proxy, in which case
`X-Forwarded-For` is walked right to left for the first non-proxy hop.
Default trusts nothing, and a set-but-unparseable value aborts
startup. Before this, any client could mint a fresh bucket or drain
another's by rotating a spoofed header (#88)
- 2026-08-11 Web UI cleanup: nav terminology unified on Webhooks, the
Profile settings placeholder removed, a progressive-enhancement copy
button for the entrypoint URL, and retention form copy that states the
actual policy (deletion by the reaper, 0 retains forever) (#57)
- 2026-08-11 Mask the webhook credential in delivery errors and logs:
Go embeds the request URL in `*url.Error`, so every transport failure
persisted the full Slack webhook URL into the per-webhook event
database via `DeliveryResult.Error`, a field a future REST API would
have served. `maskURLError` drops path, query and userinfo while
preserving the wrapped cause, so `errors.Is`/`As` and `Timeout()`
still work and DNS, TLS and timeout failures still read differently
(#118)
- 2026-08-11 Rate-limit the public webhook receiver endpoint
(`RECEIVER_RATE_LIMIT`, default 120/min), keyed on client IP plus
entrypoint path so one entrypoint cannot exhaust another's budget;
over-limit requests get 429 with `Retry-After`. It was the one
unauthenticated, internet-facing endpoint with no limit at all (#64)
- 2026-08-11 Enforce the body size limit before CSRF parses the form:
`MaxBodySize` is now first in all four form-parsing route groups, so
an oversized request is rejected with 413 instead of being read in
full by the CSRF middleware before any cap applied (#90)
- 2026-08-11 Mask target config on the source detail page, which
rendered the stored blob verbatim and so exposed the Slack
incoming-webhook URL — a bearer credential that cannot be revoked
per-holder. Config reaches the template only as a `TargetView` of
labelled fields, and header values are rendered as a count (#113)
- 2026-08-11 Allow `retention_days` of 0 to mean retain forever, via a
sentinel written in `BeforeSave` so the GORM column default cannot
win the race. Also bounds the reaper's cutoff arithmetic: day counts
above 106751 overflowed `time.Duration` and wrapped the cutoff into
the future, where every row matched and the sweep deleted everything
(#79)
- 2026-08-09 Inactivity-based session timeout: sliding idle expiry
(`SESSION_IDLE_TIMEOUT`, default `24h`) refreshed on authenticated
requests, with the 7-day absolute cap kept as an independent
backstop that activity never extends (#66)
- 2026-08-09 Restart recovery and the 60s retry sweep terminally fail an
orphaned `retrying` delivery whose target type no longer supports
retries, recording a `DeliveryResult` with the reason instead of
leaving the delivery stuck forever (#82)
- 2026-08-09 Root the delivery engine's worker pool and the retention
reaper's sweep loop at `context.Background()` rather than the fx
`OnStart` hook context (#97), which carries fx's 15s start timeout and
killed both roughly fifteen seconds after boot: the proxy silently
stopped delivering webhooks entirely, and the reaper never ran a
single sweep under its default one-hour interval
- 2026-08-09 Archive writer lifecycle (#89): deleting a webhook (or its
last `database` target) evicts the cached archive writer and closes
its handle while deliberately leaving `archive-{webhookID}.db` on
disk, and a new `ArchiveSweeper` prunes idle archives on the existing
`RETENTION_SWEEP_INTERVAL` without ever creating an archive file
- 2026-08-09 Configuration parsing fails loudly on set-but-unparseable
environment values: `envInt` removed in favour of `envPositiveInt`
plus a `PORT` range check, `envBool` now parses with
`strconv.ParseBool`, and defaults apply only to unset variables (#80)
- 2026-08-07 Automatic event retention cleanup based on
`retention_days`, deleting expired events, deliveries, and delivery
results from each per-webhook event database (#63)
- 2026-08-07 Update golangci-lint to v2.12.2 (Docker image digest in
`Dockerfile`, release-archive sha256 pins in `script/bootstrap`),
adopt the canonical `.golangci.yml` (v2 `linters.settings` layout so
`lll`/`funlen`/`cyclop`/`dupl` thresholds actually apply), and fix
all newly surfaced lint findings
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
Makefile shims, README Entrypoints section
- 2026-03-25 pin golangci-lint Docker image for linting (#55)
@@ -51,8 +241,9 @@ databases currently grow without bound.
# Future Steps
- Manual event redelivery from the web UI (replay is a core promised
capability in the README rationale)
- Manual event redelivery from the web UI — the "Replay" capability the
README describes as planned. No redelivery code exists anywhere in the
tree; events are stored in full, which is all it would be built on
- Delivery status and retry management UI
- Per-webhook rate limiting in the receiver handler (per-webhook config
plus handler enforcement; global limits must not apply to receiver
@@ -66,8 +257,10 @@ databases currently grow without bound.
- event redelivery endpoint
- OpenAPI specification
- Analytics dashboard: success rates, response times, volume
- Session expiration tuning and a remember-me option
- Password change and reset flow
- A remember-me option at login
- Password reset flow for a forgotten password. The authenticated
password *change* flow already landed on `main` (#65); reset does not
exist
- Later, nice to have
- email delivery target type
- SNS and S3 delivery targets

57
cmd/webhooker/main.go
View File

@@ -2,6 +2,8 @@
package main
import (
"time"
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database"
@@ -15,6 +17,33 @@ import (
"sneak.berlin/go/webhooker/internal/session"
)
// stopTimeout bounds the whole fx stop sequence, not each hook.
//
// fx defaults to 15s, which is longer than Docker's 10s default
// stop grace: the container would be SIGKILLed before the bound
// could fire, so nothing bounded by it would ever be observed.
// 5s leaves headroom inside that grace for signal delivery and
// process exit; the observed wedge case already exits at ~5.3s,
// so a larger bound would trade a rare skipped database close for
// a more common hard kill.
//
// The server's stop hook must fit inside it with room to spare: a
// hook that used the whole budget would exhaust it at that instant,
// and fx would skip every hook after the server — the delivery
// engine, the healthcheck, the webhook DB manager and the database
// close. That hook is the 3s HTTP drain plus the Sentry flush that
// follows it in the same hook, so the flush is clamped to the stop
// context's remaining time less server.TailHookReserve rather than
// running for its own fixed 2s; the reserve is what the tail hooks
// live on, and they are microsecond-scale in normal operation.
// TestStopTimeout_LeavesHeadroomForTailHooks pins the arithmetic
// across every drain length.
//
// This does not make the database close unconditional: the
// ArchiveSweeper and RetentionReaper hooks run before the server
// and can still consume the whole budget on their own.
const stopTimeout = 5 * time.Second
// Build-time variables set via -ldflags.
//
//nolint:gochecknoglobals // Build-time variables injected by the linker.
@@ -27,23 +56,45 @@ func main() {
globals.Appname = appname
globals.Version = version
fx.New(
newApp().Run()
}
// newApp builds the application graph. It is separate from main so
// a test can assert the options it carries.
func newApp() *fx.App {
return fx.New(
fx.StopTimeout(stopTimeout),
fx.Provide(
globals.New,
logger.New,
config.New,
database.New,
database.NewWebhookDBManager,
database.NewRetentionReaper,
healthcheck.New,
session.New,
handlers.New,
middleware.New,
delivery.New,
delivery.NewArchiveSweeper,
// Wire *delivery.Engine as delivery.Notifier so the
// webhook handler can notify the engine of new deliveries.
func(e *delivery.Engine) delivery.Notifier { return e },
// Wire *delivery.Engine as delivery.WebhookEvictor so
// deleting a webhook releases its archive writer.
func(e *delivery.Engine) delivery.WebhookEvictor {
return e
},
server.New,
),
fx.Invoke(func(*server.Server, *delivery.Engine) {}),
).Run()
fx.Invoke(
func(
*server.Server,
*delivery.Engine,
*database.RetentionReaper,
*delivery.ArchiveSweeper,
) {
},
),
)
}

75
cmd/webhooker/main_test.go Normal file
View File

@@ -0,0 +1,75 @@
package main
import (
"testing"
"time"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/server"
)
// dockerStopGrace is Docker's default `docker stop` grace period.
// The Dockerfile sets no STOPSIGNAL or grace override, so this is
// the deadline the container is actually held to, and the fx stop
// timeout has to fit inside it with room for signal delivery and
// process exit.
const dockerStopGrace = 10 * time.Second
// TestNewApp_StopTimeout pins the fx stop timeout. Without the
// explicit fx.StopTimeout option the app reads fx's 15s
// DefaultTimeout, which exceeds dockerStopGrace: the container is
// SIGKILLed before the bound fires and every shutdown hook bounded
// by it — including the operator-facing timeout log — becomes
// unreachable in the image this repo produces.
//
// fx.New applies options before it executes invokes, so the timeout
// is set whether or not the graph itself can be constructed here.
func TestNewApp_StopTimeout(t *testing.T) {
t.Setenv("DATA_DIR", t.TempDir())
got := newApp().StopTimeout()
require.Equal(t, stopTimeout, got)
require.Less(t, got, dockerStopGrace)
}
// tailHeadroom is the slack the fx stop budget must keep beyond the
// server stop hook. The hooks that run after the server — the
// delivery engine, the healthcheck, the webhook DB manager and the
// database close — are microsecond-scale in normal operation, so
// this is generous for them.
const tailHeadroom = 2 * time.Second
// TestStopTimeout_LeavesHeadroomForTailHooks pins the relationship
// between the server's stop hook and the fx stop budget. fx bounds
// the whole stop sequence, and returns without running its
// remaining hooks once the stop context has expired. If the hook
// could use the entire budget, every later hook — the database close
// included — would be skipped in exactly the case where the drain
// mattered.
//
// The hook is not just the HTTP drain: a Sentry flush follows it in
// the same hook, and sentry.Flush honours no context, so both halves
// have to be counted. The sweep walks every drain length the hook
// can produce, since a shorter drain leaves the flush more room and
// the worst case is not necessarily at either extreme.
//
// Shrinking either budget, or unbounding the flush again, must fail
// here rather than silently recreating a hook that swallows the
// whole sequence.
func TestStopTimeout_LeavesHeadroomForTailHooks(t *testing.T) {
t.Parallel()
require.Less(t, server.ShutdownTimeout, stopTimeout)
const step = 10 * time.Millisecond
for drain := time.Duration(0); drain <= server.ShutdownTimeout; drain += step {
hook := drain + server.SentryFlushBudget(stopTimeout-drain)
require.LessOrEqual(
t, hook+tailHeadroom, stopTimeout,
"a %s drain leaves the tail hooks short", drain,
)
}
}

2
go.mod
View File

@@ -17,6 +17,7 @@ require (
github.com/stretchr/testify v1.8.4
go.uber.org/fx v1.20.1
golang.org/x/crypto v0.38.0
gopkg.in/yaml.v3 v3.0.1
gorm.io/driver/sqlite v1.5.4
gorm.io/gorm v1.25.5
modernc.org/sqlite v1.28.0
@@ -52,7 +53,6 @@ require (
golang.org/x/text v0.25.0 // indirect
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
google.golang.org/protobuf v1.31.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
lukechampine.com/uint128 v1.2.0 // indirect
modernc.org/cc/v3 v3.40.0 // indirect
modernc.org/ccgo/v3 v3.16.13 // indirect

387
internal/ciscript/ci_mark_superseded_test.go Normal file
View File

@@ -0,0 +1,387 @@
package ciscript_test
import (
"maps"
"os"
"os/exec"
"path/filepath"
"slices"
"strings"
"testing"
"github.com/stretchr/testify/require"
"gopkg.in/yaml.v3"
)
const (
// supersededDesc is the description script/ci-mark-superseded
// writes, and the one an earlier revision of it wrote alongside a
// `skipped` state.
supersededDesc = "Superseded by a newer commit; never tested"
// liveContext is the commit-status context Gitea uses for this
// repository's runs, as seen in its API. The script derives it from
// the workflow and job names rather than hardcoding it; the
// derivation is checked against this value below.
liveContext = "check / check (push)"
scriptPath = "../../script/ci-mark-superseded"
workflow = "../../.gitea/workflows/check.yml"
// failure is the only state that neither folds into a combined
// `success` (as `skipped` does) nor blocks the commit forever (as
// `pending` does).
failure = "failure"
)
// repo is a throwaway git history: parent is the commit a run would be
// cancelled on, head the commit that superseded it.
type repo struct {
dir string
head string
parent string
}
// scriptEnv is the run identity the Gitea runner exports and the script
// builds its context string from.
type scriptEnv struct {
workflow string
job string
event string
}
func defaultEnv() scriptEnv {
return scriptEnv{workflow: "check", job: "check", event: "push"}
}
func cancelled() commitStatus {
return commitStatus{
Context: liveContext,
Status: failure,
Description: "Has been cancelled",
}
}
func running() commitStatus {
return commitStatus{
Context: liveContext,
Status: "pending",
Description: "Has started running",
}
}
func TestMarkSuperseded(t *testing.T) {
t.Parallel()
cases := map[string]struct {
parent commitStatus
wantMark bool
}{
"a cancelled run is marked": {
parent: cancelled(),
wantMark: true,
},
"a laundered skipped status is marked": {
parent: commitStatus{
Context: liveContext,
Status: "skipped",
Description: supersededDesc,
},
wantMark: true,
},
"a genuine failure is left alone": {
parent: commitStatus{
Context: liveContext,
Status: failure,
Description: "Failing after 3m1s",
},
wantMark: false,
},
"a passing run is left alone": {
parent: commitStatus{
Context: liveContext,
Status: "success",
Description: "Successful in 2m52s",
},
wantMark: false,
},
"another context is left alone": {
parent: commitStatus{
Context: "other / other (push)",
Status: failure,
Description: "Has been cancelled",
},
wantMark: false,
},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
t.Parallel()
requireTools(t)
history := newRepo(t)
fake, api := newFakeGitea(t)
fake.setStatus(history.head, running())
fake.setStatus(history.parent, tc.parent)
out, err := runScript(t, history, api, defaultEnv())
require.NoError(t, err, out)
posted := fake.postedFor(history.parent)
if !tc.wantMark {
require.Empty(t, posted)
return
}
require.Equal(t, []postedStatus{{
Context: liveContext,
// Not `skipped`: Gitea's combined status folds
// that into `success`, which is what made a
// never-tested commit read green.
State: failure,
Description: supersededDesc,
}}, posted)
})
}
}
// A second run must not rewrite what the first one wrote, or every
// later push would post a duplicate status.
func TestMarkSupersededIsIdempotent(t *testing.T) {
t.Parallel()
requireTools(t)
history := newRepo(t)
fake, api := newFakeGitea(t)
fake.setStatus(history.head, running())
fake.setStatus(history.parent, cancelled())
for range 2 {
out, err := runScript(t, history, api, defaultEnv())
require.NoError(t, err, out)
}
require.Len(t, fake.postedFor(history.parent), 1)
}
// Renaming the workflow or the job changes the context string Gitea
// uses. The script must say so instead of quietly matching nothing.
func TestMarkSupersededRejectsAnUnknownContext(t *testing.T) {
t.Parallel()
requireTools(t)
history := newRepo(t)
fake, api := newFakeGitea(t)
fake.setStatus(history.head, running())
fake.setStatus(history.parent, cancelled())
env := defaultEnv()
env.job = "renamed"
out, err := runScript(t, history, api, env)
require.Error(t, err)
require.Contains(t, out, "renamed")
require.Contains(t, out, liveContext)
require.Empty(t, fake.postedFor(history.parent))
}
// ANCESTOR_LIMIT is a documented knob. A value that is set but unusable
// must abort: handing it to git and discarding the exit status left the
// walk empty and the step green, marking nothing.
func TestMarkSupersededRejectsAnUnparseableAncestorLimit(t *testing.T) {
t.Parallel()
requireTools(t)
history := newRepo(t)
fake, api := newFakeGitea(t)
fake.setStatus(history.head, running())
fake.setStatus(history.parent, cancelled())
out, err := runScript(
t, history, api, defaultEnv(), "ANCESTOR_LIMIT=twenty",
)
require.Error(t, err)
require.Contains(t, out, "ANCESTOR_LIMIT")
require.Contains(t, out, "twenty")
require.Empty(t, fake.postedFor(history.parent))
}
// A status read that fails is not the same as a commit with nothing to
// do. Losing curl's exit status through a pipe made the two identical
// and left a laundered commit laundered with no signal.
func TestMarkSupersededFailsOnAnUnreadableAncestorStatus(t *testing.T) {
t.Parallel()
requireTools(t)
history := newRepo(t)
fake, api := newFakeGitea(t)
fake.setStatus(history.head, running())
fake.setStatus(history.parent, cancelled())
fake.failStatusRead(history.parent)
out, err := runScript(t, history, api, defaultEnv())
require.Error(t, err)
require.Contains(t, out, history.parent)
require.Contains(t, out, "cannot read commit statuses")
require.Empty(t, fake.postedFor(history.parent))
}
// A shallow clone cannot resolve the parent, so it is indistinguishable
// from a root commit to rev-parse and the walk would exit 0 having
// marked nothing. It must abort instead: dropping `fetch-depth: 0` from
// the checkout step is one edit, and a silent no-op there restores the
// false-green bug this script exists to prevent.
func TestMarkSupersededRejectsAShallowRepository(t *testing.T) {
t.Parallel()
requireTools(t)
history := shallowClone(t, newRepo(t))
fake, api := newFakeGitea(t)
fake.setStatus(history.head, running())
fake.setStatus(history.parent, cancelled())
out, err := runScript(t, history, api, defaultEnv())
require.Error(t, err)
require.Contains(t, out, "shallow repository")
require.Empty(t, fake.postedFor(history.parent))
require.Empty(t, fake.postedFor(history.head))
}
// shallowClone returns the same history as a depth-1 clone. The `file://`
// URL is required: git ignores --depth for a plain local path.
func shallowClone(t *testing.T, history repo) repo {
t.Helper()
dir := t.TempDir()
//nolint:gosec // fixed argv, arguments are test-local paths
cmd := exec.CommandContext(t.Context(), "git", "clone", "-q",
"--depth=1", "file://"+history.dir, dir)
out, err := cmd.CombinedOutput()
require.NoError(t, err, string(out))
return repo{dir: dir, head: history.head, parent: history.parent}
}
// The derived context must equal the one Gitea actually uses, which is
// built from the same workflow and job names.
func TestDerivedContextMatchesGitea(t *testing.T) {
t.Parallel()
requireTools(t)
name, job := workflowIdentity(t)
history := newRepo(t)
fake, api := newFakeGitea(t)
fake.setStatus(history.head, running())
fake.setStatus(history.parent, cancelled())
out, err := runScript(t, history, api, scriptEnv{
workflow: name,
job: job,
event: "push",
})
require.NoError(t, err, out)
posted := fake.postedFor(history.parent)
require.Len(t, posted, 1)
require.Equal(t, liveContext, posted[0].Context)
}
// workflowIdentity reads the workflow name and its single job id out of
// the checked-in workflow file.
func workflowIdentity(t *testing.T) (string, string) {
t.Helper()
raw, err := os.ReadFile(workflow)
require.NoError(t, err)
var parsed struct {
Name string `yaml:"name"`
Jobs map[string]any `yaml:"jobs"`
}
require.NoError(t, yaml.Unmarshal(raw, &parsed))
jobs := slices.Collect(maps.Keys(parsed.Jobs))
require.Len(t, jobs, 1)
return parsed.Name, jobs[0]
}
func runScript(
t *testing.T, history repo, api string, env scriptEnv,
extra ...string,
) (string, error) {
t.Helper()
script, err := filepath.Abs(scriptPath)
require.NoError(t, err)
//nolint:gosec // fixed argv, repo-local script under test
cmd := exec.CommandContext(t.Context(), "sh", script)
cmd.Dir = history.dir
cmd.Env = append(os.Environ(),
"GITHUB_API_URL="+api,
"GITHUB_REPOSITORY=sneak/webhooker",
"GITHUB_SHA="+history.head,
"GITHUB_WORKFLOW="+env.workflow,
"GITHUB_JOB="+env.job,
"GITHUB_EVENT_NAME="+env.event,
"GITEA_TOKEN=test-token",
)
cmd.Env = append(cmd.Env, extra...)
out, err := cmd.CombinedOutput()
return string(out), err
}
func newRepo(t *testing.T) repo {
t.Helper()
dir := t.TempDir()
git := func(args ...string) string {
//nolint:gosec // fixed argv, arguments are test constants
cmd := exec.CommandContext(t.Context(), "git", args...)
cmd.Dir = dir
out, err := cmd.CombinedOutput()
require.NoError(t, err, string(out))
return strings.TrimSpace(string(out))
}
commit := func(message string) string {
git(
"-c", "user.email=ci@example.invalid",
"-c", "user.name=ci",
"-c", "commit.gpgsign=false",
"commit", "-q", "--allow-empty", "-m", message,
)
return git("rev-parse", "HEAD")
}
git("init", "-q", "-b", "main")
parent := commit("parent")
head := commit("head")
return repo{dir: dir, head: head, parent: parent}
}
func requireTools(t *testing.T) {
t.Helper()
for _, tool := range []string{"sh", "git", "curl", "jq"} {
_, err := exec.LookPath(tool)
if err != nil {
t.Skipf("%s is not installed: %v", tool, err)
}
}
}

10
internal/ciscript/doc.go Normal file
View File

@@ -0,0 +1,10 @@
// Package ciscript holds the tests for the repository's CI shell
// scripts in script/. It carries no runtime code: the scripts run on
// the CI runner, not inside the binary, but their behaviour still has
// to be verified by the test suite.
//
// The scripts under test are outside the Go build graph, so `go test`'s
// result cache serves a stale PASS when only a script changed: run the
// container build, or GOFLAGS=-count=1, to trust a result here after
// editing script/.
package ciscript

162
internal/ciscript/fakegitea_test.go Normal file
View File

@@ -0,0 +1,162 @@
package ciscript_test
import (
"encoding/json"
"net/http"
"net/http/httptest"
"sync"
"testing"
)
// commitStatus is the part of an entry in Gitea's combined-status
// response that script/ci-mark-superseded reads.
type commitStatus struct {
Context string `json:"context"`
Status string `json:"status"`
Description string `json:"description"`
}
// postedStatus is the part of a create-status request body the script
// writes.
type postedStatus struct {
Context string `json:"context"`
State string `json:"state"`
Description string `json:"description"`
}
// fakeGitea serves the two endpoints the script talks to. Like Gitea,
// the newest status for a context replaces the previous one, so a
// second run of the script sees what the first one wrote.
type fakeGitea struct {
mu sync.Mutex
statuses map[string][]commitStatus
posted map[string][]postedStatus
// failRead is a commit whose combined-status read answers HTTP
// 500, standing in for a status API that is down.
failRead string
}
// newFakeGitea returns the fake and the base URL to hand the script as
// GITHUB_API_URL.
func newFakeGitea(t *testing.T) (*fakeGitea, string) {
t.Helper()
fake := &fakeGitea{
mu: sync.Mutex{},
statuses: map[string][]commitStatus{},
posted: map[string][]postedStatus{},
failRead: "",
}
srv := httptest.NewServer(fake.routes())
t.Cleanup(srv.Close)
return fake, srv.URL
}
func (f *fakeGitea) routes() http.Handler {
mux := http.NewServeMux()
mux.HandleFunc(
"GET /repos/{owner}/{repo}/commits/{sha}/status",
f.handleCombined,
)
mux.HandleFunc(
"POST /repos/{owner}/{repo}/statuses/{sha}",
f.handleCreate,
)
return mux
}
func (f *fakeGitea) handleCombined(
w http.ResponseWriter, r *http.Request,
) {
f.mu.Lock()
defer f.mu.Unlock()
sha := r.PathValue("sha")
if f.failRead != "" && f.failRead == sha {
http.Error(w, "boom", http.StatusInternalServerError)
return
}
body := struct {
Statuses []commitStatus `json:"statuses"`
}{Statuses: f.statuses[sha]}
payload, err := json.Marshal(body)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write(payload)
}
func (f *fakeGitea) handleCreate(w http.ResponseWriter, r *http.Request) {
var got postedStatus
err := json.NewDecoder(r.Body).Decode(&got)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
sha := r.PathValue("sha")
f.mu.Lock()
defer f.mu.Unlock()
f.posted[sha] = append(f.posted[sha], got)
f.replaceLocked(sha, commitStatus{
Context: got.Context,
Status: got.State,
Description: got.Description,
})
w.WriteHeader(http.StatusCreated)
}
// failStatusRead makes the combined-status read for one commit answer
// HTTP 500.
func (f *fakeGitea) failStatusRead(sha string) {
f.mu.Lock()
defer f.mu.Unlock()
f.failRead = sha
}
// setStatus gives a commit its latest status for a context.
func (f *fakeGitea) setStatus(sha string, status commitStatus) {
f.mu.Lock()
defer f.mu.Unlock()
f.replaceLocked(sha, status)
}
// postedFor returns the statuses the script created for a commit.
func (f *fakeGitea) postedFor(sha string) []postedStatus {
f.mu.Lock()
defer f.mu.Unlock()
return append([]postedStatus(nil), f.posted[sha]...)
}
// replaceLocked requires f.mu.
func (f *fakeGitea) replaceLocked(sha string, status commitStatus) {
for i, existing := range f.statuses[sha] {
if existing.Context == status.Context {
f.statuses[sha][i] = status
return
}
}
f.statuses[sha] = append(f.statuses[sha], status)
}

448
internal/config/config.go
View File

@@ -5,9 +5,11 @@ import (
"errors"
"fmt"
"log/slog"
"net/netip"
"os"
"strconv"
"strings"
"time"
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/globals"
@@ -26,12 +28,49 @@ const (
// defaultPort is the default HTTP listen port.
defaultPort = 8080
// defaultRetentionSweepInterval is how often the retention
// reaper deletes events older than each webhook's RetentionDays.
defaultRetentionSweepInterval = time.Hour
// defaultSessionIdleTimeout is how long a session may go without
// authenticated activity before it expires.
defaultSessionIdleTimeout = 24 * time.Hour
// defaultReceiverRateLimit is the default number of requests
// per minute each client IP may send to a single webhook
// receiver entrypoint. Generous for legitimate webhook
// senders while bounding abuse of the one unauthenticated,
// internet-exposed endpoint.
defaultReceiverRateLimit = 120
// maxPort is the highest valid TCP port number. The lower
// bound (at least 1) is enforced by envPositiveInt.
maxPort = 65535
// mappedV4Offset is the number of leading bits an IPv4-mapped
// IPv6 prefix spends on the ::ffff:0:0/96 wrapper, so a /104
// covers the same addresses as an IPv4 /8.
mappedV4Offset = 96
)
// ErrInvalidEnvironment is returned when WEBHOOKER_ENVIRONMENT
// contains an unrecognised value.
var ErrInvalidEnvironment = errors.New("invalid environment")
// ErrNonPositiveValue is returned when an environment variable that
// requires a positive integer is set to zero or a negative number.
var ErrNonPositiveValue = errors.New("value must be positive")
// ErrInvalidPort is returned when an environment variable holding a
// TCP port number is set above the valid port range.
var ErrInvalidPort = errors.New("invalid port")
// ErrInvalidCIDR is returned when an environment variable holding a
// list of CIDR blocks contains an entry that is neither a CIDR block
// nor a bare IP address.
var ErrInvalidCIDR = errors.New("invalid CIDR")
//nolint:revive // ConfigParams is a standard fx naming convention.
type ConfigParams struct {
fx.In
@@ -51,6 +90,30 @@ type Config struct {
MetricsUsername string
Port int
SentryDSN string
// RetentionSweepInterval is how often the retention reaper runs.
// Always positive: it becomes a time.NewTicker period.
RetentionSweepInterval time.Duration
// SessionIdleTimeout is the sliding inactivity window after
// which a session expires. Non-positive disables idle expiry.
SessionIdleTimeout time.Duration
// ReceiverRateLimit is the number of requests per minute each
// client IP may send to a single webhook receiver entrypoint.
ReceiverRateLimit int
// TrustedProxies is the set of networks whose members are
// allowed to speak for the client with X-Forwarded-For, the
// only forwarded header read. It is empty unless
// TRUSTED_PROXIES is set, and empty means no peer is
// trusted: forwarded headers are then ignored entirely and
// clients are identified by the connection's own address.
// Members can choose their own rate-limit key, so this must
// name proxy hosts only, never a block that also covers
// clients.
TrustedProxies []netip.Prefix
params *ConfigParams
log *slog.Logger
}
@@ -72,27 +135,337 @@ func envString(key string) string {
}
// envBool returns the value of the named environment variable
// parsed as a boolean. Returns defaultValue if not set.
func envBool(key string, defaultValue bool) bool {
if v := os.Getenv(key); v != "" {
return strings.EqualFold(v, "true") || v == "1"
// parsed as a boolean. Returns defaultValue if not set. If the
// variable is set but cannot be parsed, it returns a wrapped error
// naming the key and the bad value, so startup fails loudly rather
// than silently falling back to the default.
//
// Parsing is strconv.ParseBool, which accepts 1, t, T, TRUE, true,
// True, 0, f, F, FALSE, false and False. Anything else — "yes",
// "on", or a typo like "ture" — is an error rather than a silent
// false.
func envBool(key string, defaultValue bool) (bool, error) {
v := os.Getenv(key)
if v == "" {
return defaultValue, nil
}
return defaultValue
b, err := strconv.ParseBool(v)
if err != nil {
return false, fmt.Errorf(
"invalid boolean for %s: %q: %w", key, v, err,
)
}
return b, nil
}
// envInt returns the value of the named environment variable
// parsed as an integer. Returns defaultValue if not set or
// unparseable.
func envInt(key string, defaultValue int) int {
if v := os.Getenv(key); v != "" {
i, err := strconv.Atoi(v)
if err == nil {
return i
}
// envPositiveInt returns the value of the named environment variable
// parsed as a positive integer. Returns defaultValue if not set. If
// the variable is set but cannot be parsed, or parses to less than
// one, it returns a wrapped error naming the key and the bad value,
// so startup fails loudly rather than silently falling back to the
// default.
func envPositiveInt(
key string,
defaultValue int,
) (int, error) {
v := os.Getenv(key)
if v == "" {
return defaultValue, nil
}
return defaultValue
i, err := strconv.Atoi(v)
if err != nil {
return 0, fmt.Errorf(
"invalid integer for %s: %q: %w", key, v, err,
)
}
if i < 1 {
return 0, fmt.Errorf(
"%w: %s must be at least 1, got %q",
ErrNonPositiveValue, key, v,
)
}
return i, nil
}
// envPort returns the value of the named environment variable parsed
// as a TCP port number. Returns defaultValue if not set. A set value
// that is unparseable, below 1, or above maxPort is a hard error
// naming the key and the bad value.
func envPort(key string, defaultValue int) (int, error) {
port, err := envPositiveInt(key, defaultValue)
if err != nil {
return 0, err
}
if port > maxPort {
return 0, fmt.Errorf(
"%w: %s must be at most %d, got %d",
ErrInvalidPort, key, maxPort, port,
)
}
return port, nil
}
// envDuration returns the value of the named environment variable
// parsed as a Go duration (e.g. "1h", "30m"). Returns defaultValue if
// not set. If the variable is set but cannot be parsed, it returns a
// wrapped error naming the key and the bad value, so startup fails
// loudly rather than silently falling back to the default.
func envDuration(
key string,
defaultValue time.Duration,
) (time.Duration, error) {
v := os.Getenv(key)
if v == "" {
return defaultValue, nil
}
d, err := time.ParseDuration(v)
if err != nil {
return 0, fmt.Errorf(
"invalid duration for %s: %q: %w", key, v, err,
)
}
return d, nil
}
// envPositiveDuration returns the value of the named environment
// variable parsed as a Go duration that must be greater than zero.
// Returns defaultValue if not set. A set value that is unparseable or
// non-positive is a hard error naming the key and the bad value.
//
// This is for durations that reach time.NewTicker, which panics on a
// non-positive period, in a goroutine started after startup has
// already reported success. It is deliberately not used for durations
// where non-positive means "disabled" (SESSION_IDLE_TIMEOUT).
func envPositiveDuration(
key string,
defaultValue time.Duration,
) (time.Duration, error) {
d, err := envDuration(key, defaultValue)
if err != nil {
return 0, err
}
if d <= 0 {
return 0, fmt.Errorf(
"%w: %s must be greater than zero, got %s",
ErrNonPositiveValue, key, d,
)
}
return d, nil
}
// parseCIDR parses one trusted-proxy list entry, which may be a
// CIDR block ("10.0.0.0/8") or a bare address ("10.0.0.1", treated
// as a single-host block).
//
// Both forms are unmapped, because peer addresses are unmapped
// before they are matched against the list: an IPv4-mapped prefix
// left in that form would silently never match.
func parseCIDR(entry string) (netip.Prefix, error) {
if strings.Contains(entry, "/") {
prefix, err := netip.ParsePrefix(entry)
if err != nil {
return netip.Prefix{}, err //nolint:wrapcheck // wrapped by caller
}
if addr := prefix.Addr(); addr.Is4In6() &&
prefix.Bits() >= mappedV4Offset {
prefix = netip.PrefixFrom(
addr.Unmap(), prefix.Bits()-mappedV4Offset,
)
}
return prefix.Masked(), nil
}
addr, err := netip.ParseAddr(entry)
if err != nil {
return netip.Prefix{}, err //nolint:wrapcheck // wrapped by caller
}
return netip.PrefixFrom(addr.Unmap(), addr.Unmap().BitLen()), nil
}
// envPrefixList returns the value of the named environment variable
// parsed as a comma-separated list of CIDR blocks (bare addresses
// allowed). An unset, empty, or blank value yields an empty list. A
// set value containing an unparseable entry is a hard error naming
// the key and the bad entry, so startup fails loudly rather than
// silently running with a list the operator did not intend.
func envPrefixList(key string) ([]netip.Prefix, error) {
v := strings.TrimSpace(os.Getenv(key))
if v == "" {
return nil, nil
}
var prefixes []netip.Prefix
for entry := range strings.SplitSeq(v, ",") {
entry = strings.TrimSpace(entry)
if entry == "" {
continue
}
prefix, err := parseCIDR(entry)
if err != nil {
return nil, fmt.Errorf(
"%w: %s: %q: %w", ErrInvalidCIDR, key, entry, err,
)
}
prefixes = append(prefixes, prefix)
}
return prefixes, nil
}
// resolveEnvironment reads WEBHOOKER_ENVIRONMENT, defaulting to
// dev, and rejects unrecognised values.
func resolveEnvironment() (string, error) {
environment := os.Getenv("WEBHOOKER_ENVIRONMENT")
if environment == "" {
environment = EnvironmentDev
}
if environment != EnvironmentDev &&
environment != EnvironmentProd {
return "", fmt.Errorf(
"%w: WEBHOOKER_ENVIRONMENT must be '%s' or '%s', got '%s'",
ErrInvalidEnvironment,
EnvironmentDev, EnvironmentProd, environment,
)
}
return environment, nil
}
// loadFromEnv builds a Config from the environment. Every value that
// needs parsing fails loudly when it is set but unparseable: the
// documented defaults apply only to variables that are unset (or
// empty), never as a substitute for a value the operator actually
// provided.
func loadFromEnv() (*Config, error) {
environment, err := resolveEnvironment()
if err != nil {
return nil, err
}
port, err := envPort("PORT", defaultPort)
if err != nil {
return nil, err
}
debug, err := envBool("DEBUG", false)
if err != nil {
return nil, err
}
maintenanceMode, err := envBool("MAINTENANCE_MODE", false)
if err != nil {
return nil, err
}
retentionSweepInterval, err := envPositiveDuration(
"RETENTION_SWEEP_INTERVAL",
defaultRetentionSweepInterval,
)
if err != nil {
return nil, err
}
// Non-positive is "disabled" here, not invalid, so this stays on
// envDuration.
sessionIdleTimeout, err := envDuration(
"SESSION_IDLE_TIMEOUT",
defaultSessionIdleTimeout,
)
if err != nil {
return nil, err
}
receiverRateLimit, err := envPositiveInt(
"RECEIVER_RATE_LIMIT",
defaultReceiverRateLimit,
)
if err != nil {
return nil, err
}
trustedProxies, err := envPrefixList("TRUSTED_PROXIES")
if err != nil {
return nil, err
}
return &Config{
DataDir: envString("DATA_DIR"),
Debug: debug,
MaintenanceMode: maintenanceMode,
Environment: environment,
MetricsUsername: envString("METRICS_USERNAME"),
MetricsPassword: envString("METRICS_PASSWORD"),
Port: port,
SentryDSN: envString("SENTRY_DSN"),
RetentionSweepInterval: retentionSweepInterval,
SessionIdleTimeout: sessionIdleTimeout,
ReceiverRateLimit: receiverRateLimit,
TrustedProxies: trustedProxies,
}, nil
}
// warnSharedRateLimitBucket logs a startup warning whenever
// TRUSTED_PROXIES is empty, in any environment.
//
// With no trusted proxies every rate limiter keys on the connecting
// peer's address. Whether that is harmless or dangerous depends on
// what is in front of the process, which this code cannot observe:
// with nothing in front, the peer is the client and the limits are
// per-client as intended; behind a reverse proxy the peer is the proxy
// for every request, so all clients share one bucket per limiter.
//
// The login endpoint no longer spends budget on arrival — it verifies
// credentials first and charges only failures — so a shared bucket
// cannot deny the operator a correct password. What it does collapse
// is the failure counting: one client's wrong passwords throttle
// everyone else's wrong passwords, and the receiver's limits become
// service-wide ceilings.
//
// The warning is deliberately not gated on WEBHOOKER_ENVIRONMENT. That
// variable defaults to dev, so gating on it would silence the warning
// for exactly the operator who forgot to configure the deployment —
// the case it exists to catch.
//
// The default of trusting nobody is deliberate — trusting forwarded
// headers from arbitrary peers lets any client choose its own bucket —
// so this warns rather than failing startup or changing the key.
func (c *Config) warnSharedRateLimitBucket(log *slog.Logger) {
if len(c.TrustedProxies) > 0 {
return
}
log.Warn(
"TRUSTED_PROXIES is empty: every rate limit keys on the "+
"connecting peer's address. With nothing proxying to "+
"this process that is the client itself and the limits "+
"are per-client as intended. Behind a reverse proxy the "+
"peer is the proxy on every request, so all clients "+
"share one bucket per limit: the receiver limits become "+
"service-wide ceilings, and one client's failed logins "+
"throttle every other client's failed logins — a "+
"correct password still gets in. If anything proxies to "+
"this process, set TRUSTED_PROXIES to its address.",
"environment", c.Environment,
"trustedProxies", len(c.TrustedProxies),
)
}
// New creates a Config by reading environment variables.
@@ -101,36 +474,16 @@ func envInt(key string, defaultValue int) int {
func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
log := params.Logger.Get()
// Determine environment from WEBHOOKER_ENVIRONMENT env var,
// default to dev
environment := os.Getenv("WEBHOOKER_ENVIRONMENT")
if environment == "" {
environment = EnvironmentDev
// A set-but-unparseable value anywhere in the environment is a
// hard error, so fx aborts startup rather than running with a
// silently substituted default.
s, err := loadFromEnv()
if err != nil {
return nil, err
}
// Validate environment
if environment != EnvironmentDev &&
environment != EnvironmentProd {
return nil, fmt.Errorf(
"%w: WEBHOOKER_ENVIRONMENT must be '%s' or '%s', got '%s'",
ErrInvalidEnvironment,
EnvironmentDev, EnvironmentProd, environment,
)
}
// Load configuration values from environment variables
s := &Config{
DataDir: envString("DATA_DIR"),
Debug: envBool("DEBUG", false),
MaintenanceMode: envBool("MAINTENANCE_MODE", false),
Environment: environment,
MetricsUsername: envString("METRICS_USERNAME"),
MetricsPassword: envString("METRICS_PASSWORD"),
Port: envInt("PORT", defaultPort),
SentryDSN: envString("SENTRY_DSN"),
log: log,
params: &params,
}
s.log = log
s.params = &params
// Set default DataDir. All SQLite databases (main application
// DB and per-webhook event DBs) live here. The same default is
@@ -151,10 +504,19 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
"debug", s.Debug,
"maintenanceMode", s.MaintenanceMode,
"dataDir", s.DataDir,
"retentionSweepInterval", s.RetentionSweepInterval.String(),
// Logged because a perfectly valid non-positive value here
// disables idle expiry entirely, and that is worth showing
// back to the operator.
"sessionIdleTimeout", s.SessionIdleTimeout.String(),
"receiverRateLimit", s.ReceiverRateLimit,
"trustedProxies", len(s.TrustedProxies),
"hasSentryDSN", s.SentryDSN != "",
"hasMetricsAuth",
s.MetricsUsername != "" && s.MetricsPassword != "",
)
s.warnSharedRateLimitBucket(log)
return s, nil
}

563
internal/config/config_test.go
View File

@@ -1,8 +1,11 @@
package config_test
import (
"bytes"
"log/slog"
"os"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -13,6 +16,18 @@ import (
"sneak.berlin/go/webhooker/internal/logger"
)
// Shared subtest names for the env-parsing tables below, which all
// exercise the same three cases against different variables.
const (
caseUnsetUsesDefault = "unset uses default"
caseValidValueParsed = "valid value is parsed"
caseUnparseableFails = "unparseable value fails startup"
// cidrPrivateV4 is the sample trusted-proxy block the
// TRUSTED_PROXIES cases are built from.
cidrPrivateV4 = "10.0.0.0/8"
)
func TestEnvironmentConfig(t *testing.T) {
tests := []struct {
name string
@@ -120,6 +135,245 @@ func testEnvironmentConfigSuccess(
assert.Equal(t, isProd, cfg.IsProd())
}
func TestRetentionSweepInterval(t *testing.T) {
tests := []struct {
name string
set bool
value string
expectError bool
// sentinel, when set, must be wrapped by the startup
// error; every error case must additionally name the
// variable in its message.
sentinel error
expected time.Duration
}{
{
name: caseUnsetUsesDefault,
set: false,
expected: time.Hour,
},
{
name: caseValidValueParsed,
set: true,
value: "15m",
expected: 15 * time.Minute,
},
{
name: caseUnparseableFails,
set: true,
value: "not-a-duration",
expectError: true,
},
{
// A non-positive period panics the ticker in the
// reaper and archive-sweeper goroutines, long after
// startup has reported success, so it has to fail
// here instead.
name: "zero fails startup",
set: true,
value: "0s",
expectError: true,
sentinel: config.ErrNonPositiveValue,
},
{
name: "negative fails startup",
set: true,
value: "-1h",
expectError: true,
sentinel: config.ErrNonPositiveValue,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
if tt.set {
t.Setenv("RETENTION_SWEEP_INTERVAL", tt.value)
} else {
require.NoError(t, os.Unsetenv(
"RETENTION_SWEEP_INTERVAL",
))
}
if tt.expectError {
expectStartupErrorFor(
t, "RETENTION_SWEEP_INTERVAL", tt.sentinel,
)
} else {
testRetentionSweepIntervalSuccess(t, tt.expected)
}
})
}
}
// startupError builds the app config.New belongs to and returns
// the error fx reports, which is non-nil whenever an environment
// value is set but invalid.
func startupError(t *testing.T) error {
t.Helper()
var cfg *config.Config
app := fx.New(
fx.NopLogger,
fx.Provide(
globals.New,
logger.New,
config.New,
),
fx.Populate(&cfg),
)
return app.Err()
}
// expectStartupError asserts that fx refuses to build the app,
// which is what a set-but-invalid environment value must cause.
func expectStartupError(t *testing.T) {
t.Helper()
assert.Error(t, startupError(t))
}
// expectStartupErrorFor asserts that startup fails, that the error
// names the offending variable so an operator can find it, and,
// when sentinel is non-nil, that it wraps that sentinel.
func expectStartupErrorFor(
t *testing.T,
key string,
sentinel error,
) {
t.Helper()
err := startupError(t)
require.ErrorContains(t, err, key)
if sentinel != nil {
require.ErrorIs(t, err, sentinel)
}
}
func testRetentionSweepIntervalSuccess(
t *testing.T,
expected time.Duration,
) {
t.Helper()
var cfg *config.Config
app := fxtest.New(
t,
fx.Provide(
globals.New,
logger.New,
config.New,
),
fx.Populate(&cfg),
)
require.NoError(t, app.Err())
app.RequireStart()
defer app.RequireStop()
assert.Equal(t, expected, cfg.RetentionSweepInterval)
}
func TestSessionIdleTimeout(t *testing.T) {
tests := []struct {
name string
set bool
value string
expectError bool
expected time.Duration
}{
{
name: caseUnsetUsesDefault,
set: false,
expected: 24 * time.Hour,
},
{
name: caseValidValueParsed,
set: true,
value: "30m",
expected: 30 * time.Minute,
},
{
name: caseUnparseableFails,
set: true,
value: "not-a-duration",
expectError: true,
},
{
// Non-positive is "idle expiry disabled" for this
// variable, not a configuration error: unlike
// RETENTION_SWEEP_INTERVAL it never becomes a ticker
// period.
name: "zero disables idle expiry",
set: true,
value: "0s",
expected: 0,
},
{
name: "negative disables idle expiry",
set: true,
value: "-1h",
expected: -time.Hour,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
if tt.set {
t.Setenv("SESSION_IDLE_TIMEOUT", tt.value)
} else {
require.NoError(t, os.Unsetenv(
"SESSION_IDLE_TIMEOUT",
))
}
if tt.expectError {
expectStartupError(t)
} else {
testSessionIdleTimeoutSuccess(t, tt.expected)
}
})
}
}
func testSessionIdleTimeoutSuccess(
t *testing.T,
expected time.Duration,
) {
t.Helper()
var cfg *config.Config
app := fxtest.New(
t,
fx.Provide(
globals.New,
logger.New,
config.New,
),
fx.Populate(&cfg),
)
require.NoError(t, app.Err())
app.RequireStart()
defer app.RequireStop()
assert.Equal(t, expected, cfg.SessionIdleTimeout)
}
func TestDefaultDataDir(t *testing.T) {
for _, env := range []string{"", "dev", "prod"} {
name := env
@@ -163,3 +417,312 @@ func TestDefaultDataDir(t *testing.T) {
})
}
}
func TestReceiverRateLimit(t *testing.T) {
tests := []struct {
name string
set bool
value string
expectError bool
// sentinel, when set, must be wrapped by the startup
// error; every error case must additionally name the
// variable in its message.
sentinel error
expected int
}{
{
name: caseUnsetUsesDefault,
set: false,
expected: 120,
},
{
name: caseValidValueParsed,
set: true,
value: "30",
expected: 30,
},
{
name: caseUnparseableFails,
set: true,
value: "not-a-number",
expectError: true,
},
{
name: "zero fails startup",
set: true,
value: "0",
expectError: true,
sentinel: config.ErrNonPositiveValue,
},
{
name: "negative fails startup",
set: true,
value: "-5",
expectError: true,
sentinel: config.ErrNonPositiveValue,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
if tt.set {
t.Setenv("RECEIVER_RATE_LIMIT", tt.value)
} else {
require.NoError(t, os.Unsetenv(
"RECEIVER_RATE_LIMIT",
))
}
if tt.expectError {
expectStartupErrorFor(
t, "RECEIVER_RATE_LIMIT", tt.sentinel,
)
} else {
testReceiverRateLimitSuccess(t, tt.expected)
}
})
}
}
func testReceiverRateLimitSuccess(
t *testing.T,
expected int,
) {
t.Helper()
var cfg *config.Config
app := fxtest.New(
t,
fx.Provide(
globals.New,
logger.New,
config.New,
),
fx.Populate(&cfg),
)
require.NoError(t, app.Err())
app.RequireStart()
defer app.RequireStop()
assert.Equal(t, expected, cfg.ReceiverRateLimit)
}
func TestTrustedProxies(t *testing.T) {
tests := []struct {
name string
set bool
value string
expectError bool
expected []string
}{
{
// The default must be "trust nobody": an empty list
// means forwarded headers are ignored, never that
// every peer may speak for the client.
name: caseUnsetUsesDefault,
set: false,
expected: []string{},
},
{
name: "blank value trusts nothing",
set: true,
value: " ",
expected: []string{},
},
{
name: caseValidValueParsed,
set: true,
value: cidrPrivateV4 + ", 192.168.1.7 ,2001:db8::/32",
expected: []string{
cidrPrivateV4, "192.168.1.7/32", "2001:db8::/32",
},
},
{
name: "host bits are masked off",
set: true,
value: "10.1.2.3/8",
expected: []string{cidrPrivateV4},
},
{
// Peer addresses are unmapped before they are
// matched, so an IPv4-mapped prefix kept in that
// form could never match anything.
name: "IPv4-mapped prefix is unmapped",
set: true,
value: "::ffff:10.0.0.0/104",
expected: []string{cidrPrivateV4},
},
{
name: caseUnparseableFails,
set: true,
value: cidrPrivateV4 + ",not-an-address",
expectError: true,
},
{
name: "out-of-range prefix length fails startup",
set: true,
value: "10.0.0.0/33",
expectError: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
if tt.set {
t.Setenv("TRUSTED_PROXIES", tt.value)
} else {
require.NoError(t, os.Unsetenv("TRUSTED_PROXIES"))
}
if tt.expectError {
expectStartupErrorFor(
t, "TRUSTED_PROXIES", config.ErrInvalidCIDR,
)
} else {
testTrustedProxiesSuccess(t, tt.expected)
}
})
}
}
func testTrustedProxiesSuccess(
t *testing.T,
expected []string,
) {
t.Helper()
var cfg *config.Config
app := fxtest.New(
t,
fx.Provide(
globals.New,
logger.New,
config.New,
),
fx.Populate(&cfg),
)
require.NoError(t, app.Err())
app.RequireStart()
defer app.RequireStop()
got := make([]string, 0, len(cfg.TrustedProxies))
for _, prefix := range cfg.TrustedProxies {
got = append(got, prefix.String())
}
assert.Equal(t, expected, got)
}
// TestSharedRateLimitBucketWarning covers the startup warning that
// tells an operator a deployment behind a reverse proxy shares one
// rate-limit bucket between every client, which turns the receiver
// limits into service-wide ceilings and collapses login failure
// counting. It must fire whenever TRUSTED_PROXIES is empty,
// in any environment: WEBHOOKER_ENVIRONMENT defaults to dev, so gating
// on it would silence the warning for exactly the operator who never
// configured the deployment. It stays quiet once proxies are named.
func TestSharedRateLimitBucketWarning(t *testing.T) {
tests := []struct {
name string
environment string
trustedProxies string
expectWarning bool
}{
{
name: "prod without trusted proxies warns",
environment: config.EnvironmentProd,
expectWarning: true,
},
{
name: "prod with trusted proxies is quiet",
environment: config.EnvironmentProd,
trustedProxies: cidrPrivateV4,
expectWarning: false,
},
{
// The default environment. An internet-exposed
// deployment whose operator never set
// WEBHOOKER_ENVIRONMENT lands here and has exactly
// the exposure the warning announces.
name: "dev without trusted proxies warns",
environment: config.EnvironmentDev,
expectWarning: true,
},
{
name: "dev with trusted proxies is quiet",
environment: config.EnvironmentDev,
trustedProxies: cidrPrivateV4,
expectWarning: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
t.Setenv("WEBHOOKER_ENVIRONMENT", tt.environment)
if tt.trustedProxies == "" {
require.NoError(
t, os.Unsetenv("TRUSTED_PROXIES"),
)
} else {
t.Setenv("TRUSTED_PROXIES", tt.trustedProxies)
}
var buf bytes.Buffer
log := slog.New(slog.NewJSONHandler(
&buf, &slog.HandlerOptions{
Level: slog.LevelDebug,
},
))
require.NoError(
t,
config.WarnSharedRateLimitBucketForTest(log),
)
if !tt.expectWarning {
assert.Empty(t, buf.String())
return
}
logged := buf.String()
assert.Contains(t, logged, `"level":"WARN"`)
assert.Contains(t, logged, "TRUSTED_PROXIES")
assert.Contains(t, logged, "share one bucket")
assert.Contains(
t, logged, "throttle every other client's failed logins",
)
// The warning must not claim a lockout the login
// endpoint no longer permits: credentials are verified
// before any budget is spent.
assert.Contains(
t, logged, "a correct password still gets in",
)
// The text must stay accurate for a developer with
// nothing in front of the process, where an empty
// list costs nothing.
assert.Contains(
t, logged, "nothing proxying to this process",
)
})
}
}

409
internal/config/env_test.go Normal file
View File

@@ -0,0 +1,409 @@
package config_test
import (
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/globals"
"sneak.berlin/go/webhooker/internal/logger"
)
// testEnvKey is a throwaway variable name used only by the helper
// tables below, so they cannot disturb real configuration.
const testEnvKey = "WEBHOOKER_TEST_VALUE"
// Real configuration variables exercised by the config.New tests.
const (
envKeyPort = "PORT"
envKeyDebug = "DEBUG"
envKeyMaintenanceMode = "MAINTENANCE_MODE"
)
// envBoolCase is one row of the envBool table.
type envBoolCase struct {
name string
set bool
value string
defaultValue bool
expectError bool
expected bool
}
// envBoolCases is the envBool table, kept out of the test body so
// the test itself stays readable.
func envBoolCases() []envBoolCase {
return []envBoolCase{
{
name: "unset uses default false",
defaultValue: false,
expected: false,
},
{
name: "unset uses default true",
defaultValue: true,
expected: true,
},
{
name: "empty uses default true",
set: true,
value: "",
defaultValue: true,
expected: true,
},
{
name: "true is parsed",
set: true,
value: "true",
expected: true,
},
{
name: "one is parsed",
set: true,
value: "1",
expected: true,
},
{
name: "False is parsed",
set: true,
value: "False",
defaultValue: true,
expected: false,
},
{
name: "zero is parsed",
set: true,
value: "0",
defaultValue: true,
expected: false,
},
{
name: "yes is rejected",
set: true,
value: "yes",
expectError: true,
},
{
name: "on is rejected",
set: true,
value: "on",
expectError: true,
},
{
name: "typo is rejected",
set: true,
value: "ture",
expectError: true,
},
}
}
func TestEnvBool(t *testing.T) {
for _, tt := range envBoolCases() {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
if tt.set {
t.Setenv(testEnvKey, tt.value)
} else {
require.NoError(t, os.Unsetenv(testEnvKey))
}
got, err := config.EnvBoolForTest(
testEnvKey, tt.defaultValue,
)
if tt.expectError {
require.Error(t, err)
assert.Contains(t, err.Error(), testEnvKey)
assert.Contains(t, err.Error(), tt.value)
return
}
require.NoError(t, err)
assert.Equal(t, tt.expected, got)
})
}
}
func TestEnvPositiveInt(t *testing.T) {
const defaultValue = 7
tests := []struct {
name string
set bool
value string
expectError bool
errIs error
expected int
}{
{
name: "unset returns the default integer",
expected: defaultValue,
},
{
name: "empty returns the default integer",
set: true,
value: "",
expected: defaultValue,
},
{
name: "positive value is parsed",
set: true,
value: "42",
expected: 42,
},
{
name: "unparseable value is rejected",
set: true,
value: "not-a-number",
expectError: true,
},
{
name: "zero is rejected",
set: true,
value: "0",
expectError: true,
errIs: config.ErrNonPositiveValue,
},
{
name: "negative is rejected",
set: true,
value: "-5",
expectError: true,
errIs: config.ErrNonPositiveValue,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
if tt.set {
t.Setenv(testEnvKey, tt.value)
} else {
require.NoError(t, os.Unsetenv(testEnvKey))
}
got, err := config.EnvPositiveIntForTest(
testEnvKey, defaultValue,
)
if tt.expectError {
require.Error(t, err)
assert.Contains(t, err.Error(), testEnvKey)
assert.Contains(t, err.Error(), tt.value)
if tt.errIs != nil {
require.ErrorIs(t, err, tt.errIs)
}
return
}
require.NoError(t, err)
assert.Equal(t, tt.expected, got)
})
}
}
func TestEnvPort(t *testing.T) {
const defaultValue = 8080
tests := []struct {
name string
set bool
value string
expectError bool
errIs error
expected int
}{
{
name: "unset returns the default port",
expected: defaultValue,
},
{
name: "valid port is parsed",
set: true,
value: "9000",
expected: 9000,
},
{
name: "highest port is accepted",
set: true,
value: "65535",
expected: 65535,
},
{
name: "unparseable value is rejected",
set: true,
value: "not-a-port",
expectError: true,
},
{
name: "zero is rejected",
set: true,
value: "0",
expectError: true,
errIs: config.ErrNonPositiveValue,
},
{
name: "above the port range is rejected",
set: true,
value: "65536",
expectError: true,
errIs: config.ErrInvalidPort,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
if tt.set {
t.Setenv(testEnvKey, tt.value)
} else {
require.NoError(t, os.Unsetenv(testEnvKey))
}
got, err := config.EnvPortForTest(
testEnvKey, defaultValue,
)
if tt.expectError {
require.Error(t, err)
assert.Contains(t, err.Error(), testEnvKey)
if tt.errIs != nil {
require.ErrorIs(t, err, tt.errIs)
}
return
}
require.NoError(t, err)
assert.Equal(t, tt.expected, got)
})
}
}
// buildConfig constructs a Config through fx exactly as the
// application does, returning the config and any construction error.
func buildConfig(t *testing.T) (*config.Config, error) {
t.Helper()
var cfg *config.Config
app := fx.New(
fx.NopLogger,
fx.Provide(
globals.New,
logger.New,
config.New,
),
fx.Populate(&cfg),
)
return cfg, app.Err()
}
func TestNewRejectsBadEnvValues(t *testing.T) {
tests := []struct {
name string
key string
value string
expectError bool
check func(t *testing.T, cfg *config.Config)
}{
{
name: "valid PORT is used",
key: envKeyPort,
value: "9001",
check: func(t *testing.T, cfg *config.Config) {
t.Helper()
assert.Equal(t, 9001, cfg.Port)
},
},
{
name: "unparseable PORT aborts startup",
key: envKeyPort,
value: "eighty-eighty",
expectError: true,
},
{
name: "out-of-range PORT aborts startup",
key: envKeyPort,
value: "70000",
expectError: true,
},
{
name: "valid DEBUG is used",
key: envKeyDebug,
value: "true",
check: func(t *testing.T, cfg *config.Config) {
t.Helper()
assert.True(t, cfg.Debug)
},
},
{
name: "unparseable DEBUG aborts startup",
key: envKeyDebug,
value: "ture",
expectError: true,
},
{
name: "unparseable MAINTENANCE_MODE aborts startup",
key: envKeyMaintenanceMode,
value: "sometimes",
expectError: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
t.Setenv(tt.key, tt.value)
cfg, err := buildConfig(t)
if tt.expectError {
require.Error(t, err)
assert.Contains(t, err.Error(), tt.key)
assert.Contains(t, err.Error(), tt.value)
return
}
require.NoError(t, err)
require.NotNil(t, cfg)
tt.check(t, cfg)
})
}
}
// TestNewUsesDefaultsWhenUnset proves the fail-loud behaviour did not
// break the legitimate unset case: absent variables still get their
// documented defaults.
func TestNewUsesDefaultsWhenUnset(t *testing.T) {
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
for _, key := range []string{
envKeyPort, envKeyDebug, envKeyMaintenanceMode,
} {
require.NoError(t, os.Unsetenv(key))
}
cfg, err := buildConfig(t)
require.NoError(t, err)
require.NotNil(t, cfg)
assert.Equal(t, 8080, cfg.Port)
assert.False(t, cfg.Debug)
assert.False(t, cfg.MaintenanceMode)
}

37
internal/config/export_test.go Normal file
View File

@@ -0,0 +1,37 @@
package config
import "log/slog"
// This file exposes the unexported environment parsing helpers to
// the external config_test package so each helper can be covered by
// its own table-driven test without weakening the package API.
// WarnSharedRateLimitBucketForTest loads a Config from the current
// environment and emits its startup warnings to log. The real logger
// writes to stdout, so this lets the warning's firing condition be
// asserted against a handler the test controls.
func WarnSharedRateLimitBucketForTest(log *slog.Logger) error {
c, err := loadFromEnv()
if err != nil {
return err
}
c.warnSharedRateLimitBucket(log)
return nil
}
// EnvBoolForTest exposes envBool.
func EnvBoolForTest(key string, defaultValue bool) (bool, error) {
return envBool(key, defaultValue)
}
// EnvPositiveIntForTest exposes envPositiveInt.
func EnvPositiveIntForTest(key string, defaultValue int) (int, error) {
return envPositiveInt(key, defaultValue)
}
// EnvPortForTest exposes envPort.
func EnvPortForTest(key string, defaultValue int) (int, error) {
return envPort(key, defaultValue)
}

6
internal/database/database.go
View File

@@ -17,6 +17,7 @@ import (
"gorm.io/gorm"
_ "modernc.org/sqlite" // Pure Go SQLite driver
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/gormlog"
"sneak.berlin/go/webhooker/internal/logger"
)
@@ -155,7 +156,10 @@ func (d *Database) connect() error {
// Then use it with GORM
db, err := gorm.Open(sqlite.Dialector{
Conn: sqlDB,
}, &gorm.Config{})
}, &gorm.Config{
// Never leave this at GORM's default. See internal/gormlog.
Logger: gormlog.New(d.log),
})
if err != nil {
d.log.Error(
"failed to connect to database",

18
internal/database/database_test.go
View File

@@ -11,6 +11,20 @@ import (
"sneak.berlin/go/webhooker/internal/logger"
)
const (
// testAppname is the Globals.Appname used in tests.
testAppname = "webhooker-test"
// testVersion is the Globals.Version used in tests.
testVersion = "test"
// testContentType is the event content type used in tests.
testContentType = "application/json"
// testWebhookName is the Webhook.Name used in tests.
testWebhookName = "test-webhook"
// testForeverLabel is Webhook.RetentionLabel for a retain-forever
// webhook.
testForeverLabel = "forever"
)
func setupTestDB(
t *testing.T,
) (*database.Database, *fxtest.Lifecycle) {
@@ -19,8 +33,8 @@ func setupTestDB(
lc := fxtest.NewLifecycle(t)
g := &globals.Globals{
Appname: "webhooker-test",
Version: "test",
Appname: testAppname,
Version: testVersion,
}
l, err := logger.New(

73
internal/database/export_test.go Normal file
View File

@@ -0,0 +1,73 @@
package database
import (
"context"
"log/slog"
"os"
"time"
"go.uber.org/fx"
)
// NewTestRetentionReaper builds a RetentionReaper backed by the given
// main database and per-webhook database manager, without the fx
// lifecycle. Intended for tests.
func NewTestRetentionReaper(
db *Database,
mgr *WebhookDBManager,
) *RetentionReaper {
return &RetentionReaper{
db: db,
dbManager: mgr,
log: slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
interval: time.Hour,
}
}
// ExportSweep runs a single retention sweep synchronously for tests.
func (r *RetentionReaper) ExportSweep(ctx context.Context) {
r.sweep(ctx)
}
// ExportRegisterHooks registers the reaper's real fx lifecycle hooks
// on a lifecycle supplied by a test, so a test can drive the exact
// OnStart/OnStop functions the application runs and hand OnStart the
// kind of context fx actually supplies.
func (r *RetentionReaper) ExportRegisterHooks(lc fx.Lifecycle) {
r.registerHooks(lc)
}
// ExportStart starts the reaper's background loop for tests.
func (r *RetentionReaper) ExportStart() {
r.start()
}
// ExportStop stops the reaper's background loop for tests.
func (r *RetentionReaper) ExportStop(ctx context.Context) error {
return r.stop(ctx)
}
// ExportWedgeLoop adds a goroutine to the reaper's WaitGroup that
// never observes cancellation and returns only when release is
// closed. It stands in for a sweep stuck on a locked database.
func (r *RetentionReaper) ExportWedgeLoop(
release <-chan struct{},
) {
r.wg.Go(func() {
<-release
})
}
// ExportSetInterval overrides the sweep interval for tests.
func (r *RetentionReaper) ExportSetInterval(d time.Duration) {
r.interval = d
}
// DummyPasswordHashForTest exposes the encoded hash that unknown
// usernames are verified against.
func DummyPasswordHashForTest() string {
return dummyPasswordHash()
}

8
internal/database/model_apikey.go
View File

@@ -2,12 +2,16 @@ package database
import "time"
// APIKey represents an API key for a user
// APIKey represents an API key for a user.
//
// Key is a bearer credential, so it is never marshalled with the
// model. A creation handler that has to show it once returns it in its
// own response type.
type APIKey struct {
BaseModel
UserID string `gorm:"type:uuid;not null" json:"userId"`
Key string `gorm:"uniqueIndex;not null" json:"key"`
Key string `gorm:"uniqueIndex;not null" json:"-"`
Description string `json:"description"`
LastUsedAt *time.Time `json:"lastUsedAt,omitempty"`

5
internal/database/model_entrypoint.go
View File

@@ -5,7 +5,10 @@ type Entrypoint struct {
BaseModel
WebhookID string `gorm:"type:uuid;not null" json:"webhookId"`
Path string `gorm:"uniqueIndex;not null" json:"path"` // URL path for this entrypoint
// Path is the URL path for this entrypoint.
Path string `gorm:"uniqueIndex;not null" json:"path"`
Description string `json:"description"`
Active bool `gorm:"default:true" json:"active"`

107
internal/database/model_secrets_test.go Normal file
View File

@@ -0,0 +1,107 @@
package database_test
import (
"encoding/json"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
)
// keptField is a non-secret value planted alongside each secret, so
// the assertions below cannot pass by the model marshalling to nothing.
const keptField = "keepme"
// marshalModel encodes a model the way a future JSON handler would.
func marshalModel(t *testing.T, v any) string {
t.Helper()
encoded, err := json.Marshal(v)
require.NoError(t, err)
return string(encoded)
}
// TestModelsDoNotMarshalTheirSecrets pins the barrier for the JSON
// path. The /api/v1 route group exists and is empty; delivery's
// TargetView masks the credential for the HTML path only, so without
// these tags the first handler that marshals a model serialises the
// secret with it. Each field below is a live credential:
//
// - Target.Config holds an incoming-webhook URL whose path segments
// are the bearer token.
// - APIKey.Key is a bearer token outright.
// - Setting.Value holds the session encryption key.
// - User.Password holds the Argon2 hash, and was already tagged.
func TestModelsDoNotMarshalTheirSecrets(t *testing.T) {
t.Parallel()
const marker = "QQMODELMARKERQQ"
cases := []struct {
name string
model any
}{
{
name: "target config",
model: database.Target{
Name: keptField,
Type: database.TargetTypeSlack,
Config: `{"webhookUrl":"https://h/s/` + marker + `"}`,
},
},
{
name: "api key",
model: database.APIKey{
Description: keptField,
Key: marker,
},
},
{
name: "setting value",
model: database.Setting{
Key: keptField,
Value: marker,
},
},
{
name: "user password hash",
model: database.User{
Username: keptField,
Password: marker,
},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
encoded := marshalModel(t, tc.model)
assert.NotContains(t, encoded, marker)
assert.Contains(t, encoded, keptField)
})
}
}
// TestWebhookMarshalsNoTargetConfig covers the nested case: a webhook
// marshalled with its targets preloaded must not carry the credential
// through the association either.
func TestWebhookMarshalsNoTargetConfig(t *testing.T) {
t.Parallel()
const marker = "QQNESTEDMARKERQQ"
encoded := marshalModel(t, database.Webhook{
Name: keptField,
Targets: []database.Target{{
Name: "slack",
Config: `{"webhookUrl":"https://h/s/` + marker + `"}`,
}},
})
assert.NotContains(t, encoded, marker)
assert.Contains(t, encoded, keptField)
}

5
internal/database/model_setting.go
View File

@@ -4,5 +4,8 @@ package database
// Used for auto-generated values like the session encryption key.
type Setting struct {
Key string `gorm:"primaryKey" json:"key"`
Value string `gorm:"type:text;not null" json:"value"`
// Value holds the session encryption key, so it is never
// marshalled with the model.
Value string `gorm:"type:text;not null" json:"-"`
}

13
internal/database/model_target.go
View File

@@ -20,10 +20,17 @@ type Target struct {
Type TargetType `gorm:"not null" json:"type"`
Active bool `gorm:"default:true" json:"active"`
// Configuration fields (JSON stored based on type)
Config string `gorm:"type:text" json:"config"` // JSON configuration
// Configuration fields (JSON stored based on type).
//
// json:"-" because the blob holds the target's credential — a
// Slack incoming-webhook URL, or an http destination whose path
// segments are the secret. delivery.TargetView is the masking
// barrier for the HTML path; this tag is the barrier for any
// handler that marshals the model itself.
Config string `gorm:"type:text" json:"-"` // JSON configuration
// For HTTP targets (max_retries=0 means fire-and-forget, >0 enables retries with backoff)
// For HTTP targets (max_retries=0 means fire-and-forget,
// >0 enables retries with backoff)
MaxRetries int `json:"maxRetries,omitempty"`
MaxQueueSize int `json:"maxQueueSize,omitempty"`

111
internal/database/model_webhook.go
View File

@@ -1,16 +1,125 @@
package database
import (
"math"
"strconv"
"time"
"gorm.io/gorm"
)
const (
// DefaultRetentionDays is the event retention period applied to a
// webhook created without an explicit retention value. It is the
// single source of truth for that policy and must stay in sync
// with the `gorm:"default:30"` column default on
// Webhook.RetentionDays below; a struct tag cannot reference a
// constant, so a test asserts the two agree.
DefaultRetentionDays = 30
// RetentionForeverDays is the sentinel RetentionDays value meaning
// "retain events forever". Users express that intent as 0, which
// Webhook.BeforeSave rewrites to this value: the column default
// substitutes DefaultRetentionDays for a zero value at insert
// time, so a zero can never survive a round trip to the database.
// Nothing outside this file may hardcode the number.
RetentionForeverDays = 365 * 1000
// MaxFiniteRetentionDays is the largest finite retention period the
// reaper's cutoff arithmetic can represent, and therefore the
// largest one a caller may request. It is derived from that
// arithmetic rather than picked: retentionCutoff computes
// retentionDays * hoursPerDay * time.Hour, and a time.Duration is
// an int64 nanosecond count, so math.MaxInt64 nanoseconds divided
// by an hour and then by a day is the exact ceiling — 106751 days,
// a little over 292 years.
//
// One day more overflows int64, wraps the product negative, and
// turns the cutoff into a timestamp in the far future that matches
// every row in the webhook's database. That is why this bound is
// enforced on input and why retentionCutoff saturates underneath
// it. Note that RetentionForeverDays deliberately sits above this
// ceiling: such webhooks are skipped before any cutoff is
// computed, and never reach the arithmetic at all.
MaxFiniteRetentionDays = int(
math.MaxInt64 / int64(time.Hour) / hoursPerDay,
)
)
// Webhook represents a webhook processing unit that groups entrypoints and targets
//
// Every method below takes a pointer receiver. BeforeSave has to,
// because it mutates the record and GORM only invokes hooks declared
// that way; the display helpers follow suit so the receiver kinds do
// not mix. Handlers therefore put a *Webhook into template data:
// html/template cannot call a pointer method on a value held in a map,
// because a map element is not addressable.
type Webhook struct {
BaseModel
UserID string `gorm:"type:uuid;not null" json:"userId"`
Name string `gorm:"not null" json:"name"`
Description string `json:"description"`
RetentionDays int `gorm:"default:30" json:"retentionDays"` // Days to retain events
// RetentionDays is the number of days to retain events. A value of
// RetentionForeverDays means retain forever. The column default
// must equal DefaultRetentionDays.
RetentionDays int `gorm:"default:30" json:"retentionDays"`
// Relations
User User `json:"user,omitzero"`
Entrypoints []Entrypoint `json:"entrypoints,omitempty"`
Targets []Target `json:"targets,omitempty"`
}
// BeforeSave normalises RetentionDays on every insert and update. A
// non-positive value is the user's way of asking for "retain forever",
// which is stored as the RetentionForeverDays sentinel.
//
// This has to happen in a hook rather than at the call sites. GORM
// substitutes the column default (DefaultRetentionDays) for a zero
// value while building the insert statement, which runs after
// BeforeSave; rewriting any later than this loses that race and the
// row lands at 30 days. Living on the model also means a future call
// site — a REST API, a fixture, a migration — cannot bypass it.
func (w *Webhook) BeforeSave(_ *gorm.DB) error {
if w.RetentionDays <= 0 {
w.RetentionDays = RetentionForeverDays
}
return nil
}
// retainsForever reports whether a stored RetentionDays value means
// "keep events indefinitely". It is the single definition of that
// question, shared by Webhook.RetainsForever and by the reaper's
// cutoff computation so the two cannot disagree about which webhooks
// are exempt from reaping.
//
// It accepts the RetentionForeverDays sentinel written by BeforeSave
// and, defensively, the non-positive values that rows written before
// the sentinel existed may still carry.
func retainsForever(retentionDays int) bool {
return retentionDays <= 0 ||
retentionDays >= RetentionForeverDays
}
// RetainsForever reports whether this webhook's events are kept
// indefinitely.
func (w *Webhook) RetainsForever() bool {
return retainsForever(w.RetentionDays)
}
// RetentionLabel returns the webhook's retention policy as display
// text, so that no template has to know about the sentinel value.
func (w *Webhook) RetentionLabel() string {
if w.RetainsForever() {
return "forever"
}
if w.RetentionDays == 1 {
return "1 day"
}
return strconv.Itoa(w.RetentionDays) + " days"
}

222
internal/database/model_webhook_test.go Normal file
View File

@@ -0,0 +1,222 @@
package database_test
import (
"context"
"reflect"
"strconv"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
)
// startedTestDB returns a started main database for model-level tests.
func startedTestDB(t *testing.T) *gorm.DB {
t.Helper()
db, lc := setupTestDB(t)
ctx := context.Background()
require.NoError(t, lc.Start(ctx))
t.Cleanup(func() { require.NoError(t, lc.Stop(ctx)) })
return db.DB()
}
// storedRetention reads the retention_days column straight out of the
// row, so the assertion is about what was persisted rather than about
// whatever the in-memory struct happens to hold.
func storedRetention(t *testing.T, db *gorm.DB, id string) int {
t.Helper()
var got int
require.NoError(
t,
db.Model(&database.Webhook{}).
Where("id = ?", id).
Pluck("retention_days", &got).Error,
)
return got
}
// newWebhookWithRetention creates a webhook through the ordinary Create
// path, so the BeforeSave hook and the GORM column default both apply
// exactly as they do in production.
func newWebhookWithRetention(
t *testing.T,
db *gorm.DB,
wh *database.Webhook,
) string {
t.Helper()
wh.UserID = uuid.New().String()
wh.Name = testWebhookName
require.NoError(
t,
db.Omit(clause.Associations).Create(wh).Error,
)
return wh.ID
}
func TestWebhookBeforeSave_ZeroBecomesForeverSentinel(t *testing.T) {
t.Parallel()
db := startedTestDB(t)
wh := &database.Webhook{RetentionDays: 0}
id := newWebhookWithRetention(t, db, wh)
assert.Equal(
t,
database.RetentionForeverDays,
storedRetention(t, db, id),
"a zero retention must be stored as the sentinel, "+
"not replaced by the column default",
)
}
func TestWebhookBeforeSave_NegativeBecomesForeverSentinel(t *testing.T) {
t.Parallel()
db := startedTestDB(t)
wh := &database.Webhook{RetentionDays: -5}
id := newWebhookWithRetention(t, db, wh)
assert.Equal(
t,
database.RetentionForeverDays,
storedRetention(t, db, id),
)
}
func TestWebhookBeforeSave_PositiveIsPreserved(t *testing.T) {
t.Parallel()
db := startedTestDB(t)
wh := &database.Webhook{RetentionDays: 7}
id := newWebhookWithRetention(t, db, wh)
assert.Equal(t, 7, storedRetention(t, db, id))
}
// TestWebhookBeforeSave_UpdateToZeroBecomesSentinel proves the hook
// fires on update as well as insert, via the same Save call the edit
// handler makes.
func TestWebhookBeforeSave_UpdateToZeroBecomesSentinel(t *testing.T) {
t.Parallel()
db := startedTestDB(t)
wh := &database.Webhook{RetentionDays: 30}
id := newWebhookWithRetention(t, db, wh)
require.Equal(t, 30, storedRetention(t, db, id))
wh.RetentionDays = 0
require.NoError(t, db.Omit(clause.Associations).Save(wh).Error)
assert.Equal(
t,
database.RetentionForeverDays,
storedRetention(t, db, id),
)
}
// TestWebhookRetentionColumnDefaultMatchesConstant guards the one place
// the default lives twice: a struct tag cannot reference a constant, so
// this asserts the tag and DefaultRetentionDays agree.
func TestWebhookRetentionColumnDefaultMatchesConstant(t *testing.T) {
t.Parallel()
field, ok := reflect.TypeFor[database.Webhook]().
FieldByName("RetentionDays")
require.True(t, ok, "Webhook.RetentionDays must exist")
assert.Equal(
t,
"default:"+strconv.Itoa(database.DefaultRetentionDays),
field.Tag.Get("gorm"),
)
}
// TestMaxFiniteRetentionDaysIsTheOverflowCeiling asserts that the
// constant is exactly where the cutoff arithmetic stops working, which
// is what makes it a derived bound rather than a round number someone
// liked. One day more wraps the int64 nanosecond count negative, and a
// negative span is precisely what turned a cutoff into a future
// timestamp that matched — and deleted — every row.
//
// The multiplications are done through variables on purpose: as
// constant expressions the overflowing one would not compile.
func TestMaxFiniteRetentionDaysIsTheOverflowCeiling(t *testing.T) {
t.Parallel()
const hoursPerDay = 24
atCeiling := database.MaxFiniteRetentionDays
overCeiling := database.MaxFiniteRetentionDays + 1
assert.Positive(
t,
time.Duration(atCeiling*hoursPerDay)*time.Hour,
"the ceiling itself must still be representable",
)
assert.Negative(
t,
time.Duration(overCeiling*hoursPerDay)*time.Hour,
"one day past the ceiling must overflow",
)
assert.Less(
t,
database.MaxFiniteRetentionDays,
database.RetentionForeverDays,
"the sentinel sits above the ceiling and is only safe "+
"because retain-forever webhooks skip the arithmetic",
)
}
func TestWebhookRetainsForeverAndLabel(t *testing.T) {
t.Parallel()
cases := []struct {
name string
days int
forever bool
label string
}{
{
"sentinel",
database.RetentionForeverDays, true, testForeverLabel,
},
{
"above sentinel",
database.RetentionForeverDays + 1, true, testForeverLabel,
},
{"legacy zero", 0, true, testForeverLabel},
{"legacy negative", -1, true, testForeverLabel},
{"default", database.DefaultRetentionDays, false, "30 days"},
{"one day", 1, false, "1 day"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
wh := database.Webhook{RetentionDays: tc.days}
assert.Equal(t, tc.forever, wh.RetainsForever())
assert.Equal(t, tc.label, wh.RetentionLabel())
})
}
}

37
internal/database/password.go
View File

@@ -8,6 +8,7 @@ import (
"fmt"
"math/big"
"strings"
"sync"
"golang.org/x/crypto/argon2"
)
@@ -29,6 +30,10 @@ const hashParts = 6
// triggers per-character-class complexity enforcement.
const minPasswordComplexityLen = 4
// dummyPasswordLen is the length of the throwaway password behind
// dummyPasswordHash.
const dummyPasswordLen = 32
// Sentinel errors returned by decodeHash.
var (
errInvalidHashFormat = errors.New("invalid hash format")
@@ -122,6 +127,38 @@ func VerifyPassword(
return subtle.ConstantTimeCompare(hash, otherHash) == 1, nil
}
// dummyPasswordHash is an encoded Argon2id hash of a random
// password, computed once on first use. Nothing can match it: the
// password it encodes is discarded as soon as it is hashed. It is
// process-wide because building it per request would add a second
// 64 MB Argon2id pass to every login for an unknown username.
//
//nolint:gochecknoglobals // computed once, see above
var dummyPasswordHash = sync.OnceValue(func() string {
password, err := GenerateRandomPassword(dummyPasswordLen)
if err != nil {
panic(fmt.Sprintf("generating the dummy password: %v", err))
}
hash, err := HashPassword(password)
if err != nil {
panic(fmt.Sprintf("hashing the dummy password: %v", err))
}
return hash
})
// VerifyDummyPassword performs a credential verification that cannot
// succeed, at the same cost as a real one.
//
// Login must charge an unknown username the same work as a known
// one. Returning early for an account that does not exist answers in
// microseconds where a real account takes tens of milliseconds, which
// is a username oracle any client can read off the response time.
func VerifyDummyPassword(password string) {
_, _ = VerifyPassword(password, dummyPasswordHash())
}
// decodeHash extracts parameters, salt, and hash from an
// encoded hash string.
func decodeHash(

38
internal/database/password_test.go
View File

@@ -191,3 +191,41 @@ func TestHashPasswordUniqueness(t *testing.T) {
)
}
}
// TestVerifyDummyPassword_DoesRealWork covers the anti-enumeration
// path. Login charges an unknown username a verification against a
// dummy hash so that a nonexistent account is not answered in
// microseconds where a real one takes tens of milliseconds. That only
// works if the dummy hash is a real, decodable Argon2id hash: a
// malformed one would make VerifyPassword fail on the decode and
// return before hashing anything.
func TestVerifyDummyPassword_DoesRealWork(t *testing.T) {
t.Parallel()
// Runs the OnceValue that builds the dummy hash, so a panic in
// it surfaces here rather than on a live login.
database.VerifyDummyPassword("whatever was submitted")
dummy := database.DummyPasswordHashForTest()
// A hash the verifier cannot decode would make VerifyPassword
// return on the decode error, before hashing anything — the
// timing oracle this path exists to close.
valid, err := database.VerifyPassword("whatever", dummy)
if err != nil {
t.Fatalf(
"the dummy hash must decode like a real one: %v", err,
)
}
if valid {
t.Error("nothing may authenticate against the dummy hash")
}
if !strings.HasPrefix(dummy, "$argon2id$") {
t.Errorf(
"the dummy hash must use the same algorithm as real "+
"hashes, got %q", dummy,
)
}
}

321
internal/database/retention.go Normal file
View File

@@ -0,0 +1,321 @@
package database
import (
"context"
"fmt"
"log/slog"
"sync"
"time"
"go.uber.org/fx"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/lifecycle"
"sneak.berlin/go/webhooker/internal/logger"
)
// hoursPerDay converts a RetentionDays count into hours for cutoff
// computation.
const hoursPerDay = 24
// RetentionReaperParams holds the fx dependencies for the
// RetentionReaper.
type RetentionReaperParams struct {
fx.In
Config *config.Config
Database *Database
DBManager *WebhookDBManager
Logger *logger.Logger
}
// RetentionReaper periodically deletes expired events (and their
// dependent deliveries and delivery results) from each per-webhook
// database, enforcing every webhook's RetentionDays. Rows are removed
// permanently so that per-webhook SQLite files do not grow without
// bound.
type RetentionReaper struct {
db *Database
dbManager *WebhookDBManager
log *slog.Logger
interval time.Duration
cancel context.CancelFunc
wg sync.WaitGroup
}
// NewRetentionReaper creates the retention reaper and registers its
// fx lifecycle hooks. The background sweep loop starts on OnStart and
// stops cleanly on OnStop via context cancellation.
func NewRetentionReaper(
lc fx.Lifecycle,
params RetentionReaperParams,
) *RetentionReaper {
r := &RetentionReaper{
db: params.Database,
dbManager: params.DBManager,
log: params.Logger.Get(),
interval: params.Config.RetentionSweepInterval,
}
r.registerHooks(lc)
return r
}
// registerHooks wires the reaper's start and stop into the fx
// lifecycle. The start hook's context is deliberately ignored (see
// start for why the sweep loop must not inherit it); the stop hook's
// context is honoured (see stop).
func (r *RetentionReaper) registerHooks(lc fx.Lifecycle) {
lc.Append(fx.Hook{
//nolint:contextcheck // Not inheriting the hook context is
// the point: see start.
OnStart: func(_ context.Context) error {
r.start()
return nil
},
OnStop: func(ctx context.Context) error {
return r.stop(ctx)
},
})
}
// start launches the background sweep loop.
//
// The loop's context is derived from context.Background(), NOT from
// the fx OnStart hook context. The hook context carries fx's start
// timeout (15s by default) and is cancelled once the start phase
// completes, so a loop derived from it dies 45 minutes before its
// first tick under the default one-hour sweep interval, leaving a
// reaper that never reaps. A long-lived goroutine must outlive the
// startup phase, so its lifetime is bounded by OnStop instead: stop
// cancels this context and waits on the WaitGroup.
func (r *RetentionReaper) start() {
ctx, cancel := context.WithCancel(context.Background())
r.cancel = cancel
r.wg.Add(1)
go r.run(ctx)
r.log.Info(
"retention reaper started",
"interval", r.interval.String(),
)
}
// stop cancels the sweep loop's context and waits for it to
// exit, bounded by the stop hook's context: a sweep wedged on a
// locked database must not hang the process past fx's stop
// timeout.
func (r *RetentionReaper) stop(ctx context.Context) error {
r.log.Info("retention reaper stopping")
if r.cancel != nil {
r.cancel()
}
err := lifecycle.WaitForShutdown(
ctx, r.log, "retention reaper", &r.wg,
)
if err != nil {
return err
}
r.log.Info("retention reaper stopped")
return nil
}
func (r *RetentionReaper) run(ctx context.Context) {
defer r.wg.Done()
ticker := time.NewTicker(r.interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
r.sweep(ctx)
}
}
}
// sweep lists every webhook from the main database and reaps expired
// rows from each per-webhook database that has a finite retention
// policy. Webhooks set to retain forever are skipped entirely.
func (r *RetentionReaper) sweep(ctx context.Context) {
var webhooks []Webhook
err := r.db.DB().
Model(&Webhook{}).
Find(&webhooks).Error
if err != nil {
r.log.Error(
"retention sweep: failed to list webhooks",
"error", err,
)
return
}
for i := range webhooks {
select {
case <-ctx.Done():
return
default:
}
wh := webhooks[i]
// Skip retain-forever webhooks before building any query.
// RetainsForever covers both the RetentionForeverDays
// sentinel and the non-positive values that predate it: the
// sentinel is a positive number, so without this the reaper
// would compute a cutoff a thousand years in the past and
// issue a DELETE matching nothing on every single sweep.
if wh.RetainsForever() {
continue
}
// Nothing to reap if the per-webhook database has never
// been created.
if !r.dbManager.DBExists(wh.ID) {
continue
}
r.reapWebhook(wh.ID, wh.RetentionDays)
}
}
// reapWebhook removes every expired event (and its dependents) from a
// single webhook's database.
func (r *RetentionReaper) reapWebhook(
webhookID string,
retentionDays int,
) {
db, err := r.dbManager.GetDB(webhookID)
if err != nil {
r.log.Error(
"retention sweep: failed to open webhook database",
"webhook_id", webhookID,
"error", err,
)
return
}
cutoff, ok := retentionCutoff(time.Now(), retentionDays)
if !ok {
return
}
deleted, err := reapExpired(db, cutoff)
if err != nil {
r.log.Error(
"retention sweep: failed to reap expired events",
"webhook_id", webhookID,
"error", err,
)
return
}
if deleted > 0 {
r.log.Info(
"retention sweep: reaped expired events",
"webhook_id", webhookID,
"retention_days", retentionDays,
"events_deleted", deleted,
)
}
}
// retentionCutoff returns the timestamp before which a webhook's
// events have expired, and whether any cutoff applies at all. It
// reports false for a retain-forever policy, so no DELETE is issued.
//
// The day count is clamped to MaxFiniteRetentionDays first. This is
// defense in depth rather than decoration: a time.Duration is an int64
// nanosecond count, so an unclamped multiplication overflows above
// that ceiling and wraps the span negative. Subtracting a negative
// span moves the cutoff into the far future, where it matches every
// row in the database: the sweep then deletes every event, delivery,
// and delivery result, including ones created seconds ago. Rejecting
// out-of-range input at the form is the primary guard; saturating here
// means an old row, a migration, or a future call site cannot turn a
// too-large retention into total data loss.
func retentionCutoff(
now time.Time,
retentionDays int,
) (time.Time, bool) {
if retainsForever(retentionDays) {
return time.Time{}, false
}
if retentionDays > MaxFiniteRetentionDays {
retentionDays = MaxFiniteRetentionDays
}
return now.Add(
-time.Duration(retentionDays*hoursPerDay) * time.Hour,
), true
}
// reapExpired hard-deletes, in foreign-key-safe order, the delivery
// results, deliveries, and events associated with events older than
// cutoff. Deletes are unscoped so rows are physically removed rather
// than soft-deleted, reclaiming disk. It returns the number of events
// deleted.
func reapExpired(db *gorm.DB, cutoff time.Time) (int64, error) {
// Fresh subqueries are built per statement to avoid reusing a
// mutated builder across executions.
expiredEventIDs := func() *gorm.DB {
return db.Model(&Event{}).
Select("id").
Where("created_at < ?", cutoff)
}
expiredDeliveryIDs := func() *gorm.DB {
return db.Model(&Delivery{}).
Select("id").
Where("event_id IN (?)", expiredEventIDs())
}
// 1. Delivery results whose delivery belongs to an expired event.
res := db.Unscoped().
Where("delivery_id IN (?)", expiredDeliveryIDs()).
Delete(&DeliveryResult{})
if res.Error != nil {
return 0, fmt.Errorf(
"deleting expired delivery results: %w",
res.Error,
)
}
// 2. Deliveries belonging to an expired event.
del := db.Unscoped().
Where("event_id IN (?)", expiredEventIDs()).
Delete(&Delivery{})
if del.Error != nil {
return 0, fmt.Errorf(
"deleting expired deliveries: %w",
del.Error,
)
}
// 3. The expired events themselves.
ev := db.Unscoped().
Where("created_at < ?", cutoff).
Delete(&Event{})
if ev.Error != nil {
return 0, fmt.Errorf(
"deleting expired events: %w",
ev.Error,
)
}
return ev.RowsAffected, nil
}

272
internal/database/retention_lifecycle_test.go Normal file
View File

@@ -0,0 +1,272 @@
package database_test
import (
"context"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/fx"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
const (
// reaperTestInterval is the sweep interval a lifecycle test
// runs the reaper at, so a loop that survives startup produces
// an observable sweep quickly.
reaperTestInterval = 10 * time.Millisecond
// reaperStopTimeout bounds how long a lifecycle test waits for
// the reaper's OnStop hook to return before declaring the
// shutdown hung.
reaperStopTimeout = 10 * time.Second
// reaperTestRetentionDays is the retention policy the lifecycle
// tests give their webhook.
reaperTestRetentionDays = 30
// reaperWedgeStopTimeout is the stop timeout the wedged-shutdown
// test hands OnStop, standing in for fx's StopTimeout. The test
// asserts only that the hook returns at all, and allows it
// reaperStopTimeout — forty times this budget — to do so, so no
// assertion races the wall clock.
reaperWedgeStopTimeout = 250 * time.Millisecond
)
// recordingLifecycle is a minimal fx.Lifecycle that records the
// hooks a component registers, so a test can invoke the real
// OnStart/OnStop functions with a context of its choosing.
type recordingLifecycle struct {
hooks []fx.Hook
}
func (l *recordingLifecycle) Append(h fx.Hook) {
l.hooks = append(l.hooks, h)
}
// startReaperViaHook drives the genuine fx hooks the application
// registers for the reaper, handing OnStart a context that is
// already done. It returns the recorded lifecycle so the caller
// can drive OnStop too.
func startReaperViaHook(
t *testing.T, r *database.RetentionReaper,
) *recordingLifecycle {
t.Helper()
lc := &recordingLifecycle{}
r.ExportRegisterHooks(lc)
require.Len(t, lc.hooks, 1)
// fx hands OnStart a context carrying the application start
// timeout, and cancels it when the start phase ends. An
// already-cancelled context is that same defect taken to its
// limit, and unlike a plain context.Background() it actually
// distinguishes a correctly rooted loop from a broken one.
hookCtx, cancel := context.WithCancel(context.Background())
cancel()
require.NoError(t, lc.hooks[0].OnStart(hookCtx))
return lc
}
// eventGone reports whether an event row has been removed. It
// takes no *testing.T because it is polled from an
// assert.Eventually condition, which runs off the test goroutine
// where testify assertions must not be used.
func eventGone(db *gorm.DB, eventID string) bool {
var n int64
err := db.Unscoped().Model(&database.Event{}).
Where("id = ?", eventID).Count(&n).Error
if err != nil {
return false
}
return n == 0
}
// seedExpiredWebhook creates a webhook with a finite retention
// policy plus one long-expired event chain, and returns the
// webhook's database and the chain's event ID.
func seedExpiredWebhook(
t *testing.T, env *retentionTestEnv,
) (*gorm.DB, string) {
t.Helper()
webhookID := createWebhook(
t, env.mainDB.DB(), reaperTestRetentionDays,
)
db, err := env.mgr.GetDB(webhookID)
require.NoError(t, err)
chain := seedEventChain(
t, db, webhookID,
time.Now().Add(-365*24*time.Hour),
)
return db, chain.eventID
}
// TestRetentionReaper_LoopOutlivesStartHookContext is the
// regression test for a reaper that never reaped. fx calls
// OnStart with a context carrying the application's start timeout
// (15s by default) and cancels it when the start phase ends, so a
// sweep loop rooted in it is dead three quarters of an hour
// before its first tick under the default one-hour interval, and
// per-webhook event databases grow without bound exactly as they
// did before retention existed.
//
// Driving OnStart with an already-cancelled context is that
// defect taken to its limit: a loop that inherits the hook
// context never ticks once, while a correctly rooted loop keeps
// sweeping for as long as the process lives.
func TestRetentionReaper_LoopOutlivesStartHookContext(
t *testing.T,
) {
t.Parallel()
env := setupRetentionTest(t)
db, eventID := seedExpiredWebhook(t, env)
env.reaper.ExportSetInterval(reaperTestInterval)
lc := startReaperViaHook(t, env.reaper)
t.Cleanup(func() {
_ = lc.hooks[0].OnStop(context.Background())
})
assert.Eventually(
t,
func() bool { return eventGone(db, eventID) },
5*time.Second,
reaperTestInterval,
"the sweep loop must keep running after the start "+
"hook's context is done; it reaped nothing, so it "+
"inherited the hook context and died",
)
}
// TestRetentionReaper_StopHookStopsLoop proves the fix did not
// trade a startup bug for a shutdown hang: now that the sweep
// loop no longer observes the start hook's cancellation, OnStop
// is the only thing that can stop it, and it must both return
// promptly and actually leave the loop stopped.
func TestRetentionReaper_StopHookStopsLoop(t *testing.T) {
t.Parallel()
env := setupRetentionTest(t)
db, eventID := seedExpiredWebhook(t, env)
env.reaper.ExportSetInterval(reaperTestInterval)
lc := startReaperViaHook(t, env.reaper)
// Let the loop prove it is running before stopping it, so a
// fast OnStop cannot pass by stopping something already dead.
require.Eventually(
t,
func() bool { return eventGone(db, eventID) },
5*time.Second,
reaperTestInterval,
)
var stopErr error
stopped := make(chan struct{})
go func() {
defer close(stopped)
// stop blocks on the loop's WaitGroup, so returning at all
// proves the goroutine observed the cancellation.
stopErr = lc.hooks[0].OnStop(context.Background())
}()
select {
case <-stopped:
case <-time.After(reaperStopTimeout):
t.Fatal(
"OnStop did not return: the retention reaper's " +
"WaitGroup is still waiting on a loop that never " +
"observed cancellation",
)
}
require.NoError(t, stopErr)
// With the loop gone, a newly expired chain must survive.
survivor := seedEventChain(
t, db, "stopped-webhook",
time.Now().Add(-365*24*time.Hour),
)
time.Sleep(20 * reaperTestInterval)
assert.False(
t,
eventGone(db, survivor.eventID),
"a stopped reaper must not sweep anything",
)
}
// TestRetentionReaper_StopHookHonoursStopTimeout is the
// regression test for a shutdown that could never complete. fx
// hands OnStop a context carrying the application's stop timeout;
// an OnStop that discards it and calls wg.Wait() bare hangs the
// process forever on a sweep blocked on a locked SQLite database
// — precisely when a bounded shutdown matters most.
//
// The wedged goroutine here never observes cancellation, so the
// hook can only return by honouring its context, and it must say
// so rather than reporting a clean stop.
func TestRetentionReaper_StopHookHonoursStopTimeout(
t *testing.T,
) {
t.Parallel()
env := setupRetentionTest(t)
env.reaper.ExportSetInterval(reaperTestInterval)
lc := startReaperViaHook(t, env.reaper)
release := make(chan struct{})
t.Cleanup(func() { close(release) })
env.reaper.ExportWedgeLoop(release)
stopCtx, cancel := context.WithTimeout(
context.Background(), reaperWedgeStopTimeout,
)
defer cancel()
var stopErr error
stopped := make(chan struct{})
go func() {
defer close(stopped)
stopErr = lc.hooks[0].OnStop(stopCtx)
}()
select {
case <-stopped:
case <-time.After(reaperStopTimeout):
t.Fatal(
"OnStop did not return: it discarded the stop " +
"context and is waiting on a wedged goroutine " +
"that will never observe cancellation",
)
}
require.ErrorIs(t, stopErr, context.DeadlineExceeded)
require.ErrorContains(t, stopErr, "retention reaper")
}

402
internal/database/retention_test.go Normal file
View File

@@ -0,0 +1,402 @@
package database_test
import (
"context"
"net/http"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/fx/fxtest"
"gorm.io/gorm"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/globals"
"sneak.berlin/go/webhooker/internal/logger"
)
// retentionTestEnv bundles the pieces a retention test drives.
type retentionTestEnv struct {
reaper *database.RetentionReaper
mainDB *database.Database
mgr *database.WebhookDBManager
}
func setupRetentionTest(t *testing.T) *retentionTestEnv {
t.Helper()
lc := fxtest.NewLifecycle(t)
g := &globals.Globals{
Appname: testAppname,
Version: testVersion,
}
l, err := logger.New(lc, logger.LoggerParams{Globals: g})
require.NoError(t, err)
cfg := &config.Config{
DataDir: t.TempDir(),
Environment: "dev",
}
mainDB, err := database.New(lc, database.DatabaseParams{
Config: cfg,
Logger: l,
})
require.NoError(t, err)
mgr, err := database.NewWebhookDBManager(
lc,
database.WebhookDBManagerParams{Config: cfg, Logger: l},
)
require.NoError(t, err)
ctx := context.Background()
require.NoError(t, lc.Start(ctx))
t.Cleanup(func() { require.NoError(t, lc.Stop(ctx)) })
return &retentionTestEnv{
reaper: database.NewTestRetentionReaper(mainDB, mgr),
mainDB: mainDB,
mgr: mgr,
}
}
// createWebhook inserts a webhook row into the main database with the
// given retention policy and returns its ID.
func createWebhook(
t *testing.T,
db *gorm.DB,
retentionDays int,
) string {
t.Helper()
wh := &database.Webhook{
UserID: uuid.New().String(),
Name: testWebhookName,
RetentionDays: retentionDays,
}
require.NoError(
t,
db.Omit(clause.Associations).Create(wh).Error,
)
// Webhook.BeforeSave rewrites a non-positive RetentionDays to the
// retain-forever sentinel, and the column's GORM default would
// otherwise substitute 30. Force the requested value with a
// column-level update so tests can plant legacy rows that predate
// the sentinel and still carry a literal 0 or negative value.
require.NoError(
t,
db.Model(wh).
Update("retention_days", retentionDays).Error,
)
return wh.ID
}
// createWebhookNormally inserts a webhook through the ordinary Create
// path, with no column-level forcing, so Webhook.BeforeSave applies
// exactly as it does in production. Passing 0 therefore yields a row
// holding the RetentionForeverDays sentinel.
func createWebhookNormally(
t *testing.T,
db *gorm.DB,
retentionDays int,
) string {
t.Helper()
wh := &database.Webhook{
UserID: uuid.New().String(),
Name: testWebhookName,
RetentionDays: retentionDays,
}
require.NoError(
t,
db.Omit(clause.Associations).Create(wh).Error,
)
return wh.ID
}
// eventChain is the set of row IDs seeded for a single event.
type eventChain struct {
eventID string
deliveryID string
resultID string
}
// seedEventChain creates an event with one delivery and one delivery
// result, all stamped with createdAt, and returns their IDs.
func seedEventChain(
t *testing.T,
db *gorm.DB,
webhookID string,
createdAt time.Time,
) eventChain {
t.Helper()
event := &database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Body: `{"seed": true}`,
ContentType: testContentType,
}
event.CreatedAt = createdAt
require.NoError(t, db.Create(event).Error)
delivery := &database.Delivery{
EventID: event.ID,
TargetID: uuid.New().String(),
Status: database.DeliveryStatusDelivered,
}
delivery.CreatedAt = createdAt
require.NoError(t, db.Create(delivery).Error)
result := &database.DeliveryResult{
DeliveryID: delivery.ID,
AttemptNum: 1,
Success: true,
StatusCode: 200,
Duration: 10,
}
result.CreatedAt = createdAt
require.NoError(t, db.Create(result).Error)
return eventChain{
eventID: event.ID,
deliveryID: delivery.ID,
resultID: result.ID,
}
}
// countByID returns how many rows of model match the given id,
// counting even hard-deletable rows via Unscoped.
func countByID(
t *testing.T,
db *gorm.DB,
model any,
id string,
) int64 {
t.Helper()
var n int64
require.NoError(
t,
db.Unscoped().Model(model).
Where("id = ?", id).Count(&n).Error,
)
return n
}
func assertChainGone(
t *testing.T,
db *gorm.DB,
chain eventChain,
) {
t.Helper()
assert.Zero(
t,
countByID(t, db, &database.Event{}, chain.eventID),
"expired event should be removed",
)
assert.Zero(
t,
countByID(t, db, &database.Delivery{}, chain.deliveryID),
"expired delivery should be removed",
)
assert.Zero(
t,
countByID(
t, db, &database.DeliveryResult{}, chain.resultID,
),
"expired delivery result should be removed",
)
}
func assertChainPresent(
t *testing.T,
db *gorm.DB,
chain eventChain,
) {
t.Helper()
assert.Equal(
t,
int64(1),
countByID(t, db, &database.Event{}, chain.eventID),
"recent event should be retained",
)
assert.Equal(
t,
int64(1),
countByID(t, db, &database.Delivery{}, chain.deliveryID),
"recent delivery should be retained",
)
assert.Equal(
t,
int64(1),
countByID(
t, db, &database.DeliveryResult{}, chain.resultID,
),
"recent delivery result should be retained",
)
}
func TestRetentionReaper_ReapsExpiredKeepsRecent(t *testing.T) {
t.Parallel()
env := setupRetentionTest(t)
const retentionDays = 30
webhookID := createWebhook(
t, env.mainDB.DB(), retentionDays,
)
db, err := env.mgr.GetDB(webhookID)
require.NoError(t, err)
now := time.Now()
old := seedEventChain(
t, db, webhookID,
now.Add(-40*24*time.Hour),
)
recent := seedEventChain(
t, db, webhookID,
now.Add(-1*24*time.Hour),
)
env.reaper.ExportSweep(context.Background())
assertChainGone(t, db, old)
assertChainPresent(t, db, recent)
}
// TestRetentionReaper_SkipsSentinelReapsFiniteInSameSweep covers the
// end-to-end retain-forever path: a webhook created the normal way with
// a requested retention of 0 lands on the RetentionForeverDays
// sentinel, and the reaper leaves its ancient events alone while still
// reaping a finite-retention webhook in the very same sweep.
func TestRetentionReaper_SkipsSentinelReapsFiniteInSameSweep(
t *testing.T,
) {
t.Parallel()
env := setupRetentionTest(t)
foreverID := createWebhookNormally(t, env.mainDB.DB(), 0)
var stored database.Webhook
require.NoError(
t,
env.mainDB.DB().Where("id = ?", foreverID).
First(&stored).Error,
)
require.Equal(
t,
database.RetentionForeverDays,
stored.RetentionDays,
"a requested retention of 0 must persist as the sentinel",
)
finiteID := createWebhookNormally(t, env.mainDB.DB(), 30)
foreverDB, err := env.mgr.GetDB(foreverID)
require.NoError(t, err)
finiteDB, err := env.mgr.GetDB(finiteID)
require.NoError(t, err)
ancient := time.Now().Add(-365 * 24 * time.Hour)
kept := seedEventChain(t, foreverDB, foreverID, ancient)
doomed := seedEventChain(t, finiteDB, finiteID, ancient)
env.reaper.ExportSweep(context.Background())
assertChainPresent(t, foreverDB, kept)
assertChainGone(t, finiteDB, doomed)
}
// TestRetentionReaper_HugeFiniteRetentionRetainsRecentEvents pins the
// overflow that made a large finite retention destroy everything.
//
// The cutoff is a time.Duration, an int64 nanosecond count. A day
// count above MaxFiniteRetentionDays multiplied out unclamped wraps
// negative, so subtracting it moves the cutoff into the far future,
// where "created_at < cutoff" matches every row: an event created a
// moment ago, and its delivery and delivery result, were all deleted
// on the first sweep. 200000 is inside that band and below the
// retain-forever sentinel, so it is treated as a finite policy and
// really does reach the arithmetic.
//
// The row is planted at the column level because such a value can no
// longer be submitted through the form; the point of the test is that
// a row from an older version, or a future call site, still cannot
// trigger the wipe.
func TestRetentionReaper_HugeFiniteRetentionRetainsRecentEvents(
t *testing.T,
) {
t.Parallel()
env := setupRetentionTest(t)
const overflowingRetentionDays = 200000
require.Greater(
t,
overflowingRetentionDays,
database.MaxFiniteRetentionDays,
"the test value must exceed what the cutoff can represent",
)
require.Less(
t,
overflowingRetentionDays,
database.RetentionForeverDays,
"the test value must not be rescued by the forever skip",
)
webhookID := createWebhook(
t, env.mainDB.DB(), overflowingRetentionDays,
)
db, err := env.mgr.GetDB(webhookID)
require.NoError(t, err)
fresh := seedEventChain(t, db, webhookID, time.Now())
env.reaper.ExportSweep(context.Background())
assertChainPresent(t, db, fresh)
}
func TestRetentionReaper_RetainsForeverWhenNonPositive(t *testing.T) {
t.Parallel()
env := setupRetentionTest(t)
// A legacy row written before the sentinel existed still carries a
// literal 0; the <= 0 guard must keep honouring it.
webhookID := createWebhook(t, env.mainDB.DB(), 0)
db, err := env.mgr.GetDB(webhookID)
require.NoError(t, err)
ancient := seedEventChain(
t, db, webhookID,
time.Now().Add(-365*24*time.Hour),
)
env.reaper.ExportSweep(context.Background())
assertChainPresent(t, db, ancient)
}

10
internal/database/testing.go
View File

@@ -14,7 +14,10 @@ import (
func NewTestDatabase(db *gorm.DB) *Database {
return &Database{
db: db,
log: slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelDebug})),
log: slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
}
}
@@ -23,6 +26,9 @@ func NewTestDatabase(db *gorm.DB) *Database {
func NewTestWebhookDBManager(dataDir string) *WebhookDBManager {
return &WebhookDBManager{
dataDir: dataDir,
log: slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelDebug})),
log: slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
}
}

6
internal/database/webhook_db_manager.go
View File

@@ -14,6 +14,7 @@ import (
"gorm.io/driver/sqlite"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/gormlog"
"sneak.berlin/go/webhooker/internal/logger"
)
@@ -248,7 +249,10 @@ func (m *WebhookDBManager) openDB(
db, err := gorm.Open(sqlite.Dialector{
Conn: sqlDB,
}, &gorm.Config{})
}, &gorm.Config{
// Never leave this at GORM's default. See internal/gormlog.
Logger: gormlog.New(m.log),
})
if err != nil {
_ = sqlDB.Close()

36
internal/database/webhook_db_manager_test.go
View File

@@ -2,6 +2,7 @@ package database_test
import (
"context"
"net/http"
"os"
"path/filepath"
"testing"
@@ -25,8 +26,8 @@ func setupTestWebhookDBManager(
lc := fxtest.NewLifecycle(t)
g := &globals.Globals{
Appname: "webhooker-test",
Version: "test",
Appname: testAppname,
Version: testVersion,
}
l, err := logger.New(
@@ -83,10 +84,10 @@ func TestWebhookDBManager_CreateAndGetDB(t *testing.T) {
event := &database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: "POST",
Method: http.MethodPost,
Headers: `{"Content-Type":["application/json"]}`,
Body: `{"test": true}`,
ContentType: "application/json",
ContentType: testContentType,
}
require.NoError(t, db.Create(event).Error)
assert.NotEmpty(t, event.ID)
@@ -99,7 +100,7 @@ func TestWebhookDBManager_CreateAndGetDB(t *testing.T) {
db.First(&readEvent, "id = ?", event.ID).Error,
)
assert.Equal(t, webhookID, readEvent.WebhookID)
assert.Equal(t, "POST", readEvent.Method)
assert.Equal(t, http.MethodPost, readEvent.Method)
assert.Equal(t, `{"test": true}`, readEvent.Body)
}
@@ -123,9 +124,9 @@ func TestWebhookDBManager_DeleteDB(t *testing.T) {
event := &database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: "POST",
Method: http.MethodPost,
Body: `{"test": true}`,
ContentType: "application/json",
ContentType: testContentType,
}
require.NoError(t, db.Create(event).Error)
@@ -196,10 +197,10 @@ func seedDeliveryWorkflow(
event := &database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: "POST",
Method: http.MethodPost,
Headers: `{"Content-Type":["application/json"]}`,
Body: `{"payload": "test"}`,
ContentType: "application/json",
ContentType: testContentType,
}
require.NoError(t, db.Create(event).Error)
@@ -231,7 +232,7 @@ func verifyPendingDeliveries(
)
require.Len(t, pending, 1)
assert.Equal(t, event.ID, pending[0].EventID)
assert.Equal(t, "POST", pending[0].Event.Method)
assert.Equal(t, http.MethodPost, pending[0].Event.Method)
}
func completeDelivery(
@@ -303,16 +304,16 @@ func TestWebhookDBManager_MultipleWebhooks(t *testing.T) {
event1 := &database.Event{
WebhookID: webhook1,
EntrypointID: uuid.New().String(),
Method: "POST",
Method: http.MethodPost,
Body: `{"webhook": 1}`,
ContentType: "application/json",
ContentType: testContentType,
}
event2 := &database.Event{
WebhookID: webhook2,
EntrypointID: uuid.New().String(),
Method: "PUT",
Method: http.MethodPut,
Body: `{"webhook": 2}`,
ContentType: "application/json",
ContentType: testContentType,
}
require.NoError(t, db1.Create(event1).Error)
@@ -338,7 +339,12 @@ func TestWebhookDBManager_MultipleWebhooks(t *testing.T) {
var events []database.Event
require.NoError(t, db2.Find(&events).Error)
assert.Len(t, events, 1)
// require, not assert: this is exactly the regression the test
// guards, so the empty slice is the expected failure, and a
// non-fatal length check would index into it on the next line and
// panic the whole package test binary instead of failing here.
require.Len(t, events, 1)
assert.Equal(t, "PUT", events[0].Method)
}

239
internal/delivery/archive_sweeper.go Normal file
View File

@@ -0,0 +1,239 @@
package delivery
import (
"context"
"errors"
"log/slog"
"sync"
"time"
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/lifecycle"
"sneak.berlin/go/webhooker/internal/logger"
)
// ArchiveSweeperParams holds the fx dependencies for the
// ArchiveSweeper.
type ArchiveSweeperParams struct {
fx.In
Config *config.Config
Database *database.Database
Engine *Engine
Logger *logger.Logger
}
// ArchiveSweeper periodically prunes expired rows from
// per-webhook archive databases whose database target carries a
// positive expiry.
//
// Without it, pruning happens only when an archive is
// (re)opened, and archives are only ever reopened by writes: an
// archive belonging to a webhook that has stopped receiving
// events would keep its expired rows forever. The sweep closes
// that gap without changing anything for archives whose expiry
// is unset or "never".
//
// It reuses Config.RetentionSweepInterval rather than
// introducing a second interval: this is a retention sweep with
// the same semantics as the event retention reaper.
type ArchiveSweeper struct {
db *database.Database
eng *Engine
log *slog.Logger
interval time.Duration
cancel context.CancelFunc
wg sync.WaitGroup
}
// NewArchiveSweeper creates the archive sweeper and registers
// its fx lifecycle hooks. The background sweep loop starts on
// OnStart and stops cleanly on OnStop via context cancellation.
func NewArchiveSweeper(
lc fx.Lifecycle,
params ArchiveSweeperParams,
) *ArchiveSweeper {
s := &ArchiveSweeper{
db: params.Database,
eng: params.Engine,
log: params.Logger.Get(),
interval: params.Config.RetentionSweepInterval,
}
s.registerHooks(lc)
return s
}
// registerHooks wires the sweeper's start and stop into the fx
// lifecycle. The start hook's context is deliberately ignored
// (see start for why the background loop must not inherit it);
// the stop hook's context is honoured (see stop).
func (s *ArchiveSweeper) registerHooks(lc fx.Lifecycle) {
lc.Append(fx.Hook{
//nolint:contextcheck // Not passing the hook context is
// the point: see start.
OnStart: func(_ context.Context) error {
s.start()
return nil
},
OnStop: func(ctx context.Context) error {
return s.stop(ctx)
},
})
}
// start launches the background sweep loop.
//
// The loop's context is derived from context.Background(), NOT
// from the fx OnStart hook context. The hook context carries
// fx's start timeout (15s by default), so a loop derived from it
// is cancelled 15 seconds after the application starts — long
// before the first tick under the default one-hour sweep
// interval, leaving a sweeper that never sweeps. A long-lived
// goroutine must outlive the startup phase, so its lifetime is
// bounded by OnStop instead: stop cancels this context and waits
// on the WaitGroup.
func (s *ArchiveSweeper) start() {
ctx, cancel := context.WithCancel(context.Background())
s.cancel = cancel
s.wg.Add(1)
go s.run(ctx)
s.log.Info(
"archive sweeper started",
"interval", s.interval.String(),
)
}
// stop cancels the sweep loop's context and waits for it to
// exit, bounded by the stop hook's context: a prune wedged on a
// locked archive must not hang the process past fx's stop
// timeout.
func (s *ArchiveSweeper) stop(ctx context.Context) error {
s.log.Info("archive sweeper stopping")
if s.cancel != nil {
s.cancel()
}
err := lifecycle.WaitForShutdown(
ctx, s.log, "archive sweeper", &s.wg,
)
if err != nil {
return err
}
s.log.Info("archive sweeper stopped")
return nil
}
func (s *ArchiveSweeper) run(ctx context.Context) {
defer s.wg.Done()
ticker := time.NewTicker(s.interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
s.sweep(ctx)
}
}
}
// sweep prunes every archive whose database target declares a
// positive expiry. Targets belonging to a deleted webhook are
// soft-deleted along with it, so GORM's default scope already
// excludes them.
//
// A failure for one webhook is logged and the sweep continues,
// matching how the write path already treats a prune error as
// non-fatal.
func (s *ArchiveSweeper) sweep(ctx context.Context) {
var targets []database.Target
err := s.db.DB().
Model(&database.Target{}).
Where("type = ?", database.TargetTypeDatabase).
Find(&targets).Error
if err != nil {
s.log.Error(
"archive sweep: failed to list database targets",
"error", err,
)
return
}
for i := range targets {
select {
case <-ctx.Done():
return
default:
}
s.sweepTarget(&targets[i])
}
}
// sweepTarget prunes the archive of a single database target.
// A missing, empty, or "never" expiry parses as a zero duration
// and is skipped entirely, so those archives keep exactly the
// behaviour they had before the sweep existed.
func (s *ArchiveSweeper) sweepTarget(target *database.Target) {
expiry, err := parseArchiveExpiry(target.Config)
if err != nil {
s.log.Error(
"archive sweep: invalid database target config",
"webhook_id", target.WebhookID,
"target_id", target.ID,
"error", err,
)
return
}
if expiry <= 0 {
return
}
if s.eng == nil || s.eng.dbTarget == nil {
return
}
err = s.eng.dbTarget.sweepWebhook(target.WebhookID, expiry)
if err == nil {
return
}
// A writer evicted underneath the sweep means the operator
// deleted the webhook (or its last database target) while the
// sweep was walking the target list. That is an ordinary
// interleaving, not a failure, so it must not produce an
// error line.
if errors.Is(err, errArchiveWriterEvicted) {
s.log.Debug(
"archive sweep: writer evicted mid-sweep",
"webhook_id", target.WebhookID,
"target_id", target.ID,
)
return
}
s.log.Error(
"archive sweep: failed to prune archive",
"webhook_id", target.WebhookID,
"target_id", target.ID,
"error", err,
)
}

947
internal/delivery/archive_sweeper_test.go Normal file
View File

@@ -0,0 +1,947 @@
package delivery_test
import (
"context"
"database/sql"
"fmt"
"net/http"
"os"
"path/filepath"
"sync"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
"gorm.io/gorm/clause"
_ "modernc.org/sqlite" // Pure Go SQLite driver.
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
const (
// sweepRowOld and sweepRowNew are the event ids
// seedArchiveRows assigns to the first and second seeded
// rows.
sweepRowOld = "ev-0"
sweepRowNew = "ev-1"
// sweepConcurrentWrites is how many deliveries the
// concurrent write-plus-sweep test races against the sweep.
sweepConcurrentWrites = 20
)
// sweeperEnv bundles the pieces an archive sweep test drives:
// a main configuration database holding webhooks and targets, a
// delivery engine owning the archive writer registry, and the
// data directory the archive files live in.
type sweeperEnv struct {
sweeper *delivery.ArchiveSweeper
eng *delivery.Engine
mainDB *database.Database
dataDir string
}
func setupSweeperTest(t *testing.T) *sweeperEnv {
t.Helper()
dataDir := t.TempDir()
log := archiveTestLogger()
sqlDB, err := sql.Open(
"sqlite",
fmt.Sprintf(
"file:%s?mode=rwc",
filepath.Join(dataDir, "main.db"),
),
)
require.NoError(t, err)
t.Cleanup(func() { _ = sqlDB.Close() })
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
require.NoError(t, err)
mainDB := database.NewTestDatabase(gdb)
require.NoError(t, mainDB.Migrate())
eng := delivery.NewTestEngineWithDB(
mainDB,
database.NewTestWebhookDBManager(dataDir),
log,
&http.Client{Timeout: 5 * time.Second},
1,
)
return &sweeperEnv{
sweeper: delivery.NewTestArchiveSweeper(
mainDB, eng, log,
),
eng: eng,
mainDB: mainDB,
dataDir: dataDir,
}
}
// archivePath returns where the engine keeps a webhook's
// archive file.
func (env *sweeperEnv) archivePath(webhookID string) string {
return filepath.Join(
env.dataDir, fmt.Sprintf("archive-%s.db", webhookID),
)
}
// seedDatabaseTarget creates a webhook with one database target
// carrying the given target config JSON, and returns the
// webhook id.
func (env *sweeperEnv) seedDatabaseTarget(
t *testing.T, configJSON string,
) string {
t.Helper()
wh := &database.Webhook{
UserID: uuid.New().String(),
Name: "sweep-test",
}
require.NoError(
t,
env.mainDB.DB().
Omit(clause.Associations).
Create(wh).Error,
)
tgt := &database.Target{
WebhookID: wh.ID,
Name: "archive",
Type: database.TargetTypeDatabase,
Active: true,
Config: configJSON,
}
require.NoError(
t,
env.mainDB.DB().
Omit(clause.Associations).
Create(tgt).Error,
)
return wh.ID
}
// seedArchiveRows creates the archive file for a webhook and
// inserts one row per supplied archived-at timestamp, returning
// the archive path. The handle is closed before returning, so
// the archive is idle exactly as it would be with no traffic.
func (env *sweeperEnv) seedArchiveRows(
t *testing.T, webhookID string, archivedAt ...time.Time,
) string {
t.Helper()
path := env.archivePath(webhookID)
sqlDB, err := sql.Open(
"sqlite", fmt.Sprintf("file:%s?mode=rwc", path),
)
require.NoError(t, err)
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
require.NoError(t, err)
require.NoError(
t, gdb.AutoMigrate(&delivery.ExportArchivedEvent{}),
)
for i, at := range archivedAt {
row := delivery.ExportArchivedEvent{
EventID: fmt.Sprintf("ev-%d", i),
WebhookID: webhookID,
Method: http.MethodPost,
Body: `{"seeded":true}`,
ArchivedAt: at,
}
require.NoError(t, gdb.Create(&row).Error)
}
require.NoError(t, sqlDB.Close())
return path
}
// archivedEventIDs returns the event ids currently stored in an
// archive file, read through a separate read-only handle.
func archivedEventIDs(
t *testing.T, path string,
) []string {
t.Helper()
var rows []delivery.ExportArchivedEvent
rdb := openArchiveDBForRead(t, path)
require.NoError(t, rdb.Order("event_id").Find(&rows).Error)
ids := make([]string, 0, len(rows))
for i := range rows {
ids = append(ids, rows[i].EventID)
}
return ids
}
// countArchivedRows counts the rows in an archive file without
// asserting anything, so it is safe to poll from an
// assert.Eventually condition (which runs off the test
// goroutine, where testify assertions must not be used).
func countArchivedRows(path string) (int64, error) {
sqlDB, err := sql.Open(
"sqlite", fmt.Sprintf("file:%s?mode=ro", path),
)
if err != nil {
return 0, err
}
defer func() { _ = sqlDB.Close() }()
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
if err != nil {
return 0, err
}
var count int64
err = gdb.Model(&delivery.ExportArchivedEvent{}).
Count(&count).Error
if err != nil {
return 0, err
}
return count, nil
}
// TestArchiveSweeper_LoopOutlivesStartHookContext is the
// regression test for a sweeper that never swept. fx calls
// OnStart with a context carrying the application's start
// timeout (15 seconds by default), so a background loop whose
// context is derived from it is cancelled 15 seconds into the
// process — three quarters of an hour before the first tick
// under the default one-hour sweep interval.
//
// The hook context here is already cancelled, which is the same
// defect taken to its limit: a loop that inherits it never runs
// a single tick, while a correctly rooted loop keeps sweeping
// for as long as the process lives. Handing the hook a plain
// context.Background() would assert nothing at all.
func TestArchiveSweeper_LoopOutlivesStartHookContext(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
now := time.Now()
path := env.seedArchiveRows(
t, webhookID,
now.Add(-48*time.Hour),
now.Add(-time.Minute),
)
env.sweeper.ExportSetInterval(10 * time.Millisecond)
// Drive the genuine fx hooks the application registers,
// rather than a test-only entry point.
lc := &recordingLifecycle{}
env.sweeper.ExportRegisterHooks(lc)
require.Len(t, lc.hooks, 1)
hookCtx, cancel := context.WithCancel(context.Background())
cancel()
require.NoError(t, lc.hooks[0].OnStart(hookCtx))
t.Cleanup(func() {
_ = lc.hooks[0].OnStop(context.Background())
})
assert.Eventually(
t,
func() bool {
count, err := countArchivedRows(path)
return err == nil && count == 1
},
5*time.Second,
10*time.Millisecond,
"the sweep loop must keep running after the start "+
"hook's context is done; it pruned nothing, so it "+
"inherited the hook context and died",
)
}
// TestArchiveSweep_DoesNotResurrectEvictedWriter covers the
// interleaving where a sweep tick has already listed a webhook's
// target when the webhook is deleted and its writer evicted. The
// sweep must not put a writer back into the registry: nothing
// would ever evict it again, which is precisely the leak this
// change exists to close.
func TestArchiveSweep_DoesNotResurrectEvictedWriter(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
// Prime the registry the way a delivery would, then evict as
// the deletion path does. The target row is deliberately left
// in place: this is the tick that listed the webhook before
// the deletion committed.
_, err := env.eng.ExportEnsureArchiveWriter(webhookID)
require.NoError(t, err)
env.eng.EvictWebhook(webhookID)
require.False(t, env.eng.ExportHasArchiveWriter(webhookID))
env.sweeper.ExportSweep(context.Background())
assert.False(
t, env.eng.ExportHasArchiveWriter(webhookID),
"a sweep must never re-register a writer for a webhook "+
"whose registry entry has already been released",
)
}
// TestArchiveSweep_LeavesNoRegistryEntry states the same
// invariant in its general form: sweeping an archive whose
// webhook has no cached writer must not leave one behind, so the
// registry keeps holding only writers a delivery created and an
// eviction can reach.
func TestArchiveSweep_LeavesNoRegistryEntry(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path := env.seedArchiveRows(
t, webhookID,
time.Now().Add(-48*time.Hour),
time.Now().Add(-time.Minute),
)
require.False(t, env.eng.ExportHasArchiveWriter(webhookID))
env.sweeper.ExportSweep(context.Background())
assert.Equal(
t, []string{sweepRowNew}, archivedEventIDs(t, path),
"the sweep must still prune an idle archive",
)
assert.False(
t, env.eng.ExportHasArchiveWriter(webhookID),
"the sweep must release the registry entry it created",
)
}
// TestArchiveSweep_KeepsWriterAdoptedByDelivery is the other
// half of that invariant: an entry the sweep created but a
// delivery then claimed belongs to the registry and must survive
// the sweep, or the delivery would be left holding a detached
// writer with an open handle that no eviction can reach.
func TestArchiveSweep_KeepsWriterAdoptedByDelivery(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"n":1}`)
event.WebhookID = webhookID
d := seedDatabaseTargetDelivery(
t, webhookDB, event, `{"expiry":"1h"}`,
)
env.sweeper.ExportSweep(context.Background())
require.False(t, env.eng.ExportHasArchiveWriter(webhookID))
env.eng.ExportDeliverDatabase(webhookDB, d)
assert.True(
t, env.eng.ExportHasArchiveWriter(webhookID),
"a delivery's writer must stay registered",
)
env.sweeper.ExportSweep(context.Background())
assert.True(
t, env.eng.ExportHasArchiveWriter(webhookID),
"a sweep must not drop a writer a delivery owns",
)
}
// TestArchiveSweep_KeepsWriterAdoptedDuringSweep covers the one
// interleaving the sweepOwned flag exists for, which
// TestArchiveSweep_KeepsWriterAdoptedByDelivery cannot reach: a
// delivery adopting the sweep's own entry WHILE that sweep is
// still running.
//
// The registry operations are driven directly, in the order the
// sweep and a concurrent delivery perform them, so the window is
// exercised deterministically rather than hoped for:
//
// 1. the sweep finds no cached writer and registers one of its
// own, marked sweep-owned;
// 2. a delivery arrives, is handed that very writer, clears the
// flag and opens the archive handle;
// 3. the sweep finishes and releases what it created.
//
// Step 3 must leave the entry alone. Dropping it would detach a
// writer that is holding an open archive handle inside its
// debounce window, and no eviction could ever reach it again —
// exactly the process-lifetime handle leak this change exists to
// close. The eviction at the end proves the entry is still
// reachable.
func TestArchiveSweep_KeepsWriterAdoptedDuringSweep(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
sweepWriter, created, err := env.eng.ExportSweepWriterFor(
webhookID,
)
require.NoError(t, err)
require.True(
t, created,
"the sweep must have created the registry entry itself",
)
// The delivery lands mid-sweep and adopts the entry.
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"n":1}`)
event.WebhookID = webhookID
d := seedDatabaseTargetDelivery(
t, webhookDB, event, `{"expiry":"1h"}`,
)
env.eng.ExportDeliverDatabase(webhookDB, d)
adopted := env.eng.ExportArchiveWriterFor(webhookID)
require.NotNil(t, adopted)
require.True(
t, sweepWriter.Same(adopted),
"the delivery must have adopted the sweep's writer",
)
require.True(
t, env.eng.ExportArchiveHandleOpen(webhookID),
"the delivery leaves the archive handle open",
)
// The sweep finishes.
env.eng.ExportReleaseSweepWriter(webhookID, sweepWriter)
require.True(
t, env.eng.ExportHasArchiveWriter(webhookID),
"a writer adopted by a delivery during a sweep must "+
"stay registered, or its open handle is unreachable",
)
env.eng.EvictWebhook(webhookID)
assert.False(
t, env.eng.ExportHasArchiveWriter(webhookID),
"the adopted writer must still be evictable",
)
assert.False(
t, sweepWriter.HandleOpen(),
"eviction must have closed the adopted writer's handle",
)
}
// TestArchiveSweep_ContinuesAfterPerWebhookFailure proves a
// failure for one webhook does not abort the sweep for the
// others: an unparseable expiry and an unreadable archive both
// have to be logged and stepped over.
func TestArchiveSweep_ContinuesAfterPerWebhookFailure(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
// Seeded first so the sweep reaches them before the healthy
// webhook: targets come back in insertion order.
badConfigID := env.seedDatabaseTarget(t, `{"expiry":"!!!"}`)
env.seedArchiveRows(
t, badConfigID, time.Now().Add(-48*time.Hour),
)
corruptID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
require.NoError(t, os.WriteFile(
env.archivePath(corruptID),
[]byte("this is not a sqlite database"),
0o600,
))
healthyID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
healthyPath := env.seedArchiveRows(
t, healthyID,
time.Now().Add(-48*time.Hour),
time.Now().Add(-time.Minute),
)
env.sweeper.ExportSweep(context.Background())
assert.Equal(
t, []string{sweepRowNew},
archivedEventIDs(t, healthyPath),
"a failure for an earlier webhook must not stop the "+
"sweep from pruning the ones after it",
)
}
// TestArchiveSweep_OpenExistingDoesNotCreateFile pins the second
// of the two no-create guards. The first is the stat in
// sweepWebhook; this one is the SQLite open mode, which is what
// protects the window between that stat and the open. Flipping
// the sweep's mode to create-if-missing makes this fail.
func TestArchiveSweep_OpenExistingDoesNotCreateFile(
t *testing.T,
) {
t.Parallel()
dir := t.TempDir()
path := filepath.Join(dir, "archive-absent.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
err := w.OpenExisting(time.Hour)
require.Error(
t, err,
"opening a missing archive without create permission "+
"must fail rather than conjure the file",
)
for _, suffix := range archiveFileSuffixes() {
assert.NoFileExists(t, path+suffix)
}
}
// TestArchiveSweep_PrunesIdleArchive is the core regression
// test for this issue: an archive that receives no further
// writes must still lose its expired rows. Before the sweeper
// existed, pruning only ever ran on a write-triggered reopen,
// so an idle archive kept expired rows forever.
func TestArchiveSweep_PrunesIdleArchive(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
now := time.Now()
path := env.seedArchiveRows(
t, webhookID,
now.Add(-48*time.Hour),
now.Add(-time.Minute),
)
require.Equal(
t, []string{sweepRowOld, sweepRowNew},
archivedEventIDs(t, path),
)
env.sweeper.ExportSweep(context.Background())
assert.Equal(
t, []string{sweepRowNew}, archivedEventIDs(t, path),
"the sweep should prune rows older than the expiry "+
"from an idle archive and keep the rest",
)
}
// TestArchiveSweep_LeavesArchiveClosed proves the sweep does
// not hold the archive open afterwards, so an operator can
// still move the file away for offline retention.
//
// The assertion is made on a writer the test holds a reference
// to, and the handle is proven OPEN before the sweep runs, so the
// test observes the sweep closing it rather than a writer that
// merely never opened anything. Asking the registry instead would
// be vacuous here: the sweep releases an entry it created, and a
// missing entry reports "not open" whether or not anything was
// closed.
func TestArchiveSweep_LeavesArchiveClosed(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path := env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.OpenExisting(time.Hour))
require.True(
t, w.HandleOpen(),
"the writer must hold an open handle before the sweep",
)
require.NoError(t, w.SweepExpired(time.Hour))
assert.False(
t, w.HandleOpen(),
"an idle archive must end the sweep closed",
)
}
// TestArchiveSweep_ClosesHandleOfRegisteredWriter states the same
// guarantee end to end, through the real sweeper and a writer the
// registry keeps.
//
// The delivery leaves the archive handle open inside its debounce
// window and makes the entry delivery-owned, so the sweep finds a
// cached writer (created is false, nothing is released) and the
// registry query afterwards is answered by a writer that really
// exists. A handle left open here would be doubly wrong: it also
// blocks the operator's move-the-file-away workflow.
func TestArchiveSweep_ClosesHandleOfRegisteredWriter(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"n":1}`)
event.WebhookID = webhookID
d := seedDatabaseTargetDelivery(
t, webhookDB, event, `{"expiry":"1h"}`,
)
env.eng.ExportDeliverDatabase(webhookDB, d)
require.True(
t, env.eng.ExportArchiveHandleOpen(webhookID),
"the delivery must leave the archive handle open",
)
env.sweeper.ExportSweep(context.Background())
require.True(
t, env.eng.ExportHasArchiveWriter(webhookID),
"the delivery's registry entry must survive the sweep",
)
assert.False(
t, env.eng.ExportArchiveHandleOpen(webhookID),
"the sweep must leave the archive closed",
)
}
// TestArchiveSweep_NeverExpiryUntouched proves the sweep is a
// no-op for the default retention policy, so archives with no
// expiry (or the literal "never") behave exactly as before.
func TestArchiveSweep_NeverExpiryUntouched(t *testing.T) {
t.Parallel()
for _, configJSON := range []string{
`{"expiry":"never"}`,
`{"expiry":""}`,
"",
} {
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, configJSON)
path := env.seedArchiveRows(
t, webhookID,
time.Now().Add(-10000*time.Hour),
)
env.sweeper.ExportSweep(context.Background())
assert.Equal(
t, []string{sweepRowOld}, archivedEventIDs(t, path),
"config %q must keep rows forever", configJSON,
)
assert.False(
t, env.eng.ExportHasArchiveWriter(webhookID),
"config %q must leave no registry entry behind",
configJSON,
)
}
}
// TestArchiveSweep_NeverExpirySkipsBeforeOpening pins the
// expiry <= 0 boundary in sweepTarget, which the row assertions
// above cannot reach: pruning is separately gated on a positive
// expiry, so a "never" archive keeps its rows even if the sweep
// does open it.
//
// The spec is stronger than that — a "never" archive is skipped
// before any file is touched — so the archive here exists but has
// never been migrated. Opening it at all would run AutoMigrate
// and create the archive table, which is exactly what must not
// happen.
func TestArchiveSweep_NeverExpirySkipsBeforeOpening(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"never"}`)
path := env.archivePath(webhookID)
seedUnmigratedArchive(t, path)
require.False(t, archiveTableExists(t, path))
env.sweeper.ExportSweep(context.Background())
assert.False(
t, archiveTableExists(t, path),
"a never-expiry archive must not be opened at all",
)
}
// seedUnmigratedArchive creates an archive file that exists but
// carries no archive schema, so any open of it is observable: the
// archive table appears only if something ran AutoMigrate.
func seedUnmigratedArchive(t *testing.T, path string) {
t.Helper()
sqlDB, err := sql.Open(
"sqlite", fmt.Sprintf("file:%s?mode=rwc", path),
)
require.NoError(t, err)
_, err = sqlDB.ExecContext(
t.Context(), "CREATE TABLE placeholder (id INTEGER)",
)
require.NoError(t, err)
require.NoError(t, sqlDB.Close())
}
// archiveTableExists reports whether an archive file has had the
// archive schema migrated into it.
func archiveTableExists(t *testing.T, path string) bool {
t.Helper()
return openArchiveDBForRead(t, path).
Migrator().
HasTable(&delivery.ExportArchivedEvent{})
}
// TestArchiveSweep_DoesNotCreateArchiveFile proves the sweep
// never conjures an archive: a webhook with a database target
// that has never received an event must still have no archive
// file (nor SQLite sidecar) after a sweep.
func TestArchiveSweep_DoesNotCreateArchiveFile(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path := env.archivePath(webhookID)
require.NoFileExists(t, path)
env.sweeper.ExportSweep(context.Background())
for _, suffix := range archiveFileSuffixes() {
assert.NoFileExists(
t, path+suffix,
"the sweep must not create an archive file",
)
}
}
// TestArchiveSweep_DoesNotCreateAfterWriterExists covers the
// same guarantee once a writer is cached in the registry but
// the file itself is still absent (for instance because the
// operator moved the archive away).
func TestArchiveSweep_DoesNotCreateAfterWriterExists(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path, err := env.eng.ExportEnsureArchiveWriter(webhookID)
require.NoError(t, err)
require.NoFileExists(t, path)
env.sweeper.ExportSweep(context.Background())
assert.NoFileExists(t, path)
}
// TestArchiveSweep_SkipsDeletedWebhookTargets proves that the
// sweep ignores targets soft-deleted along with their webhook,
// so a deleted webhook's archive is never reopened.
func TestArchiveSweep_SkipsDeletedWebhookTargets(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path := env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
require.NoError(
t,
env.mainDB.DB().
Where("webhook_id = ?", webhookID).
Delete(&database.Target{}).Error,
)
env.sweeper.ExportSweep(context.Background())
assert.Equal(
t, []string{sweepRowOld}, archivedEventIDs(t, path),
"a deleted target's archive must be left alone",
)
}
// TestArchiveSweep_ConcurrentWrites proves the sweep serialises
// against writes through the per-webhook writer mutex. Run
// under -race, an unsynchronised sweep would be caught here.
func TestArchiveSweep_ConcurrentWrites(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
webhookDB := testWebhookDB(t)
// The deliveries are seeded up front, on the test's own
// goroutine: the seed helpers assert, and testify assertions
// must not run off the test goroutine.
deliveries := make(
[]*database.Delivery, 0, sweepConcurrentWrites,
)
for range sweepConcurrentWrites {
event := seedEvent(t, webhookDB, `{"n":1}`)
event.WebhookID = webhookID
deliveries = append(
deliveries,
seedDatabaseTargetDelivery(
t, webhookDB, event, `{"expiry":"1h"}`,
),
)
}
var wg sync.WaitGroup
wg.Add(2)
go func() {
defer wg.Done()
for _, d := range deliveries {
env.eng.ExportDeliverDatabase(webhookDB, d)
}
}()
go func() {
defer wg.Done()
for range sweepConcurrentWrites {
env.sweeper.ExportSweep(context.Background())
}
}()
wg.Wait()
assert.FileExists(t, env.archivePath(webhookID))
}
// TestArchiveSweeper_StopsCleanly proves the background loop
// exits on OnStop rather than leaking a goroutine.
func TestArchiveSweeper_StopsCleanly(t *testing.T) {
t.Parallel()
env := setupSweeperTest(t)
webhookID := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, webhookID, time.Now().Add(-48*time.Hour),
)
env.sweeper.ExportSetInterval(time.Millisecond)
env.sweeper.ExportStart()
// stop blocks on the loop's WaitGroup, so returning without
// error proves the loop observed the cancellation and exited
// well inside the stop context.
require.NoError(
t, env.sweeper.ExportStop(context.Background()),
)
}
// TestArchiveSweeper_StopHookHonoursStopTimeout is the sweeper's
// half of the same shutdown defect the engine and the retention
// reaper carried: an OnStop that discards its context and waits
// on the WaitGroup bare hangs the process forever on a prune
// wedged inside a locked archive.
func TestArchiveSweeper_StopHookHonoursStopTimeout(
t *testing.T,
) {
t.Parallel()
env := setupSweeperTest(t)
lc := &recordingLifecycle{}
env.sweeper.ExportRegisterHooks(lc)
require.Len(t, lc.hooks, 1)
require.NoError(t, lc.hooks[0].OnStart(context.Background()))
release := make(chan struct{})
t.Cleanup(func() { close(release) })
env.sweeper.ExportWedgeLoop(release)
requireStopHookExpires(t, lc.hooks[0], "archive sweeper")
}

1081
internal/delivery/engine.go
View File

File diff suppressed because it is too large Load Diff

256
internal/delivery/engine_integration_test.go
View File

@@ -126,36 +126,6 @@ func iHTTPConfig(url string) string {
return string(data)
}
func iWebhookDB(t *testing.T) *gorm.DB {
t.Helper()
dbPath := filepath.Join(
t.TempDir(), "events-test.db",
)
dsn := fmt.Sprintf(
"file:%s?cache=shared&mode=rwc", dbPath,
)
sqlDB, err := sql.Open("sqlite", dsn)
require.NoError(t, err)
t.Cleanup(func() { _ = sqlDB.Close() })
db, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
require.NoError(t, err)
require.NoError(t, db.AutoMigrate(
&database.Event{},
&database.Delivery{},
&database.DeliveryResult{},
))
return db
}
func iEngine(
t *testing.T, workers int,
) *delivery.Engine {
@@ -182,10 +152,10 @@ func iSeedEvent(
event := database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: "POST",
Method: http.MethodPost,
Headers: `{}`,
Body: body,
ContentType: "application/json",
ContentType: testContentType,
}
require.NoError(t, db.Create(&event).Error)
@@ -506,7 +476,7 @@ func TestWorkerLifecycle_StartStop(t *testing.T) {
t.Parallel()
s := newISetup(t)
s.Engine.ExportStart(context.Background())
s.Engine.ExportStart()
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID,
@@ -529,21 +499,17 @@ func TestWorkerLifecycle_StartStop(t *testing.T) {
s.Engine.Notify([]delivery.Task{task})
iWaitForStatus(
t, s.WebhookDB, d.ID,
database.DeliveryStatusDelivered,
)
iWaitForDelivered(t, s.WebhookDB, d.ID)
s.Engine.ExportStop()
require.NoError(t, s.Engine.ExportStop(context.Background()))
}
// iWaitForStatus polls until the delivery reaches the
// expected status.
func iWaitForStatus(
// iWaitForDelivered polls until the delivery reaches the
// delivered status.
func iWaitForDelivered(
t *testing.T,
db *gorm.DB,
deliveryID string,
expected database.DeliveryStatus,
) {
t.Helper()
@@ -557,7 +523,7 @@ func iWaitForStatus(
return false
}
return d.Status == expected
return d.Status == database.DeliveryStatusDelivered
}, 5*time.Second, 50*time.Millisecond)
}
@@ -588,7 +554,7 @@ func TestWorkerLifecycle_ProcessesRetryChannel(
database.DeliveryStatusRetrying,
)
s.Engine.ExportStart(context.Background())
s.Engine.ExportStart()
bodyStr := event.Body
cfg := iHTTPConfig(ts.URL)
@@ -599,12 +565,9 @@ func TestWorkerLifecycle_ProcessesRetryChannel(
s.Engine.ExportRetryCh() <- task
iWaitForStatus(
t, s.WebhookDB, d.ID,
database.DeliveryStatusDelivered,
)
iWaitForDelivered(t, s.WebhookDB, d.ID)
s.Engine.ExportStop()
require.NoError(t, s.Engine.ExportStop(context.Background()))
}
// --- processDelivery: unknown target type ---
@@ -778,6 +741,193 @@ func TestRecoverWebhookDeliveries_RetryingDeliveries(
case <-time.After(5 * time.Second):
t.Fatal("expected retry task from recovery")
}
// Regression guard: a target that still supports retries
// must be rescheduled, never terminally failed, and must
// not gain a synthetic result row.
iAssertStatus(
t, s.WebhookDB, d.ID,
database.DeliveryStatusRetrying,
)
assert.Len(t, iResults(t, s.WebhookDB, d.ID), 1)
}
// --- Retrying deliveries whose target type changed ---
// iSeedRetryingWithType seeds a retrying delivery with one
// recorded failed attempt against a target of the given type,
// standing in for a target whose type was edited in the main
// database while the delivery was still retrying.
func iSeedRetryingWithType(
t *testing.T,
s iSetup,
targetType database.TargetType,
) string {
t.Helper()
targetID := uuid.New().String()
iCreateTarget(t, s.MainDB, targetID,
s.WebhookID, "mutated-target", targetType,
iHTTPConfig("http://example.com/hook"), 5,
)
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID,
`{"orphaned":"retry"}`,
)
d := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusRetrying,
)
iSeedFailedResult(t, s.WebhookDB, d.ID)
return d.ID
}
// iResults loads a delivery's results in attempt order.
func iResults(
t *testing.T, db *gorm.DB, deliveryID string,
) []database.DeliveryResult {
t.Helper()
var results []database.DeliveryResult
require.NoError(t, db.
Where("delivery_id = ?", deliveryID).
Order("attempt_num").
Find(&results).Error)
return results
}
// iAssertTerminallyFailed asserts the delivery ended failed
// with a result row recording why, and was not rescheduled.
func iAssertTerminallyFailed(
t *testing.T,
s iSetup,
deliveryID string,
targetType database.TargetType,
) {
t.Helper()
iAssertStatus(
t, s.WebhookDB, deliveryID,
database.DeliveryStatusFailed,
)
results := iResults(t, s.WebhookDB, deliveryID)
require.Len(t, results, 2)
last := results[1]
assert.False(t, last.Success)
assert.Equal(t, 2, last.AttemptNum)
assert.Contains(
t, last.Error, string(targetType),
)
assert.Contains(
t, last.Error, "does not support retries",
)
assert.Empty(t, s.Engine.ExportRetryCh())
}
func TestRecoverSingleRetry_TypeNoLongerRetries(
t *testing.T,
) {
t.Parallel()
s := newISetup(t)
iCreateWebhook(
t, s.MainDB, s.WebhookID, "mutated-type",
)
deliveryID := iSeedRetryingWithType(
t, s, database.TargetTypeLog,
)
s.Engine.ExportRecoverWebhookDeliveries(
context.Background(), s.WebhookID,
)
iAssertTerminallyFailed(
t, s, deliveryID, database.TargetTypeLog,
)
}
func TestSweepSingleRetry_TypeNoLongerRetries(
t *testing.T,
) {
t.Parallel()
s := newISetup(t)
iCreateWebhook(
t, s.MainDB, s.WebhookID, "mutated-type-sweep",
)
deliveryID := iSeedRetryingWithType(
t, s, database.TargetTypeDatabase,
)
s.Engine.ExportSweepWebhookRetries(
context.Background(), s.WebhookID,
)
iAssertTerminallyFailed(
t, s, deliveryID, database.TargetTypeDatabase,
)
}
func TestRecoverSingleRetry_UnknownTargetType(
t *testing.T,
) {
t.Parallel()
s := newISetup(t)
iCreateWebhook(
t, s.MainDB, s.WebhookID, "unknown-type",
)
unknown := database.TargetType("not-a-target-type")
deliveryID := iSeedRetryingWithType(t, s, unknown)
s.Engine.ExportRecoverWebhookDeliveries(
context.Background(), s.WebhookID,
)
iAssertTerminallyFailed(t, s, deliveryID, unknown)
}
func TestSweepSingleRetry_UnknownTargetType(
t *testing.T,
) {
t.Parallel()
s := newISetup(t)
iCreateWebhook(
t, s.MainDB, s.WebhookID, "unknown-type-sweep",
)
unknown := database.TargetType("not-a-target-type")
deliveryID := iSeedRetryingWithType(t, s, unknown)
s.Engine.ExportSweepWebhookRetries(
context.Background(), s.WebhookID,
)
iAssertTerminallyFailed(t, s, deliveryID, unknown)
}
// iSeedFailedResult creates a failed delivery result.
@@ -935,7 +1085,7 @@ func TestDeliverHTTP_CustomTargetHeaders(t *testing.T) {
func TestDeliverHTTP_TargetTimeout(t *testing.T) {
t.Parallel()
db := iWebhookDB(t)
db := testWebhookDB(t)
e := iEngine(t, 1)
ts := httptest.NewServer(
@@ -987,10 +1137,10 @@ func iSeedEventAndDelivery(
event := database.Event{
WebhookID: uuid.New().String(),
EntrypointID: uuid.New().String(),
Method: "POST",
Method: http.MethodPost,
Headers: `{"Content-Type":["application/json"]}`,
Body: body,
ContentType: "application/json",
ContentType: testContentType,
}
require.NoError(t, db.Create(&event).Error)
@@ -1067,7 +1217,7 @@ func iAssertResultFailed(
func TestDeliverHTTP_InvalidConfig(t *testing.T) {
t.Parallel()
db := iWebhookDB(t)
db := testWebhookDB(t)
e := iEngine(t, 1)
event, del := iSeedEventAndDelivery(

271
internal/delivery/engine_lifecycle_test.go Normal file
View File

@@ -0,0 +1,271 @@
package delivery_test
import (
"context"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/require"
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
const (
// hookStopTimeout bounds how long a lifecycle test waits for
// the engine's OnStop hook to return before declaring the
// shutdown hung.
hookStopTimeout = 10 * time.Second
// hookSettleDelay is how long startEngineViaHook waits after
// OnStart before the caller may enqueue work. A worker pool
// wrongly rooted in the already-done hook context has nothing
// but ctx.Done() ready in its select, so it is deterministically
// gone by the end of this window. Without the wait, Notify would
// race the pool's very first select, in which a ready ctx.Done()
// and a ready deliveryCh are chosen between at random and a
// doomed pool still delivers.
hookSettleDelay = 250 * time.Millisecond
// wedgeStopTimeout is the stop timeout a wedged-shutdown test
// hands OnStop, standing in for fx's StopTimeout. The test
// asserts only that the hook returns at all, and allows it
// hookStopTimeout — forty times this budget — to do so, so no
// assertion here races the wall clock.
wedgeStopTimeout = 250 * time.Millisecond
)
// recordingLifecycle is a minimal fx.Lifecycle that records the
// hooks a component registers, so a test can invoke the real
// OnStart/OnStop functions with a context of its choosing.
type recordingLifecycle struct {
hooks []fx.Hook
}
func (l *recordingLifecycle) Append(h fx.Hook) {
l.hooks = append(l.hooks, h)
}
// requireStopHookExpires drives hook.OnStop with a stop context
// that expires while a wedged goroutine is still running, and
// requires the hook to return the deadline error naming
// component instead of blocking on the WaitGroup forever.
func requireStopHookExpires(
t *testing.T, hook fx.Hook, component string,
) {
t.Helper()
stopCtx, cancel := context.WithTimeout(
context.Background(), wedgeStopTimeout,
)
defer cancel()
var stopErr error
stopped := make(chan struct{})
go func() {
defer close(stopped)
stopErr = hook.OnStop(stopCtx)
}()
select {
case <-stopped:
case <-time.After(hookStopTimeout):
t.Fatal(
"OnStop did not return: it discarded the stop " +
"context and is waiting on a wedged goroutine " +
"that will never observe cancellation",
)
}
require.ErrorIs(t, stopErr, context.DeadlineExceeded)
require.ErrorContains(t, stopErr, component)
}
// startEngineViaHook drives the genuine fx hooks the application
// registers for the engine, handing OnStart a context that is
// already done, and returns only once a pool that inherited that
// context would have exited. It returns the recorded lifecycle so
// the caller can drive OnStop too.
//
// Callers must not seed pending or retrying deliveries before
// calling this: restart recovery enqueues those during startup,
// which would put work in the queue while the pool is still
// racing its first select.
func startEngineViaHook(
t *testing.T, eng *delivery.Engine,
) *recordingLifecycle {
t.Helper()
lc := &recordingLifecycle{}
eng.ExportRegisterHooks(lc)
require.Len(t, lc.hooks, 1)
// fx hands OnStart a context carrying the application start
// timeout, and cancels it when the start phase ends. An
// already-cancelled context is that same defect taken to its
// limit, and unlike a plain context.Background() it actually
// distinguishes a correctly rooted loop from a broken one.
hookCtx, cancel := context.WithCancel(context.Background())
cancel()
require.NoError(t, lc.hooks[0].OnStart(hookCtx))
time.Sleep(hookSettleDelay)
return lc
}
// seedLogTask seeds a pending delivery for a log target and
// returns its ID together with the task that drives it. The log
// target needs no network, so a delivery completing proves only
// that a worker picked the task up.
func seedLogTask(
t *testing.T, s iSetup,
) (string, delivery.Task) {
t.Helper()
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID,
`{"lifecycle":"hook-context"}`,
)
targetID := uuid.New().String()
d := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusPending,
)
bodyStr := event.Body
task := iTask(
d, event, s.WebhookID, targetID,
"hook-context-test", "", 0, 1, &bodyStr,
)
task.TargetType = database.TargetTypeLog
return d.ID, task
}
// TestEngine_WorkersOutliveStartHookContext is the regression
// test for a delivery engine that stopped delivering roughly
// fifteen seconds after boot. fx calls OnStart with a context
// carrying the application's start timeout (15s by default) and
// cancels it when the start phase ends, so a worker pool rooted
// in it exits shortly after startup: the process keeps accepting
// and persisting events while nothing at all forwards them.
//
// Driving OnStart with an already-cancelled context is that
// defect taken to its limit. A pool that inherits the hook
// context is gone before the task is even enqueued; a correctly
// rooted pool keeps working for as long as the process lives.
func TestEngine_WorkersOutliveStartHookContext(t *testing.T) {
t.Parallel()
s := newISetup(t)
lc := startEngineViaHook(t, s.Engine)
t.Cleanup(func() {
_ = lc.hooks[0].OnStop(context.Background())
})
// Seeded only after the pool has settled, so restart recovery
// cannot enqueue it during startup.
deliveryID, task := seedLogTask(t, s)
s.Engine.Notify([]delivery.Task{task})
iWaitForDelivered(t, s.WebhookDB, deliveryID)
}
// TestEngine_StopHookStopsWorkers proves the fix did not trade a
// startup bug for a shutdown hang: now that the worker pool no
// longer observes the start hook's cancellation, OnStop is the
// only thing that can stop it, and it must both return promptly
// and actually leave the pool drained.
func TestEngine_StopHookStopsWorkers(t *testing.T) {
t.Parallel()
s := newISetup(t)
lc := startEngineViaHook(t, s.Engine)
// Let the pool prove it is running before stopping it, so a
// fast OnStop cannot pass by stopping something already dead.
firstID, firstTask := seedLogTask(t, s)
s.Engine.Notify([]delivery.Task{firstTask})
iWaitForDelivered(t, s.WebhookDB, firstID)
var stopErr error
stopped := make(chan struct{})
go func() {
defer close(stopped)
// stop blocks on the workers' WaitGroup, so returning at
// all proves every goroutine observed the cancellation.
stopErr = lc.hooks[0].OnStop(context.Background())
}()
select {
case <-stopped:
case <-time.After(hookStopTimeout):
t.Fatal(
"OnStop did not return: the delivery engine's " +
"WaitGroup is still waiting on a goroutine that " +
"never observed cancellation",
)
}
require.NoError(t, stopErr)
// With every worker gone, a freshly notified task must sit
// untouched in the queue rather than being delivered.
secondID, secondTask := seedLogTask(t, s)
s.Engine.Notify([]delivery.Task{secondTask})
time.Sleep(200 * time.Millisecond)
var after database.Delivery
require.NoError(
t,
s.WebhookDB.First(&after, "id = ?", secondID).Error,
)
require.Equal(
t,
database.DeliveryStatusPending,
after.Status,
"a stopped engine must not deliver anything",
)
}
// TestEngine_StopHookHonoursStopTimeout is the regression test
// for a shutdown that could never complete. fx hands OnStop a
// context carrying the application's stop timeout; an OnStop
// that discards it and calls wg.Wait() bare hangs the process
// forever on a single worker stuck inside a delivery target that
// never returns — precisely when a bounded shutdown matters
// most.
//
// The wedged goroutine here never observes cancellation, so the
// hook can only return by honouring its context, and it must say
// so rather than reporting a clean stop.
func TestEngine_StopHookHonoursStopTimeout(t *testing.T) {
t.Parallel()
s := newISetup(t)
lc := startEngineViaHook(t, s.Engine)
release := make(chan struct{})
t.Cleanup(func() { close(release) })
s.Engine.ExportWedgeWorker(release)
requireStopHookExpires(t, lc.hooks[0], "delivery engine")
}

249
internal/delivery/engine_test.go
View File

@@ -1,6 +1,7 @@
package delivery_test
import (
"bytes"
"context"
"database/sql"
"encoding/json"
@@ -26,6 +27,9 @@ import (
"sneak.berlin/go/webhooker/internal/delivery"
)
// testContentType is the event content type used in tests.
const testContentType = "application/json"
func testWebhookDB(t *testing.T) *gorm.DB {
t.Helper()
@@ -93,10 +97,10 @@ func seedEvent(
event := database.Event{
WebhookID: uuid.New().String(),
EntrypointID: uuid.New().String(),
Method: "POST",
Method: http.MethodPost,
Headers: `{"Content-Type":["application/json"]}`,
Body: body,
ContentType: "application/json",
ContentType: testContentType,
}
require.NoError(t, db.Create(&event).Error)
@@ -341,33 +345,29 @@ func TestDeliverDatabase_ImmediateSuccess(
t.Parallel()
db := testWebhookDB(t)
e := testEngine(t, 1)
event := seedEvent(t, db, `{"db":"target"}`)
dlv := seedDelivery(
t, db, event.ID, uuid.New().String(),
database.DeliveryStatusPending,
// The database target archives for real now, so the engine
// needs a webhook DB manager to locate the data directory.
e := delivery.NewTestEngineWithDB(
nil,
database.NewTestWebhookDBManager(t.TempDir()),
slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
&http.Client{Timeout: 5 * time.Second},
1,
)
d := &database.Delivery{
EventID: event.ID,
TargetID: dlv.TargetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: database.Target{
Name: "test-db",
Type: database.TargetTypeDatabase,
},
}
d.ID = dlv.ID
event := seedEvent(t, db, `{"db":"target"}`)
d := seedDatabaseTargetDelivery(t, db, event, "")
e.ExportDeliverDatabase(db, d)
var updated database.Delivery
require.NoError(t, db.First(
&updated, "id = ?", dlv.ID,
&updated, "id = ?", d.ID,
).Error)
assert.Equal(t,
@@ -378,7 +378,7 @@ func TestDeliverDatabase_ImmediateSuccess(
var result database.DeliveryResult
require.NoError(t, db.Where(
"delivery_id = ?", dlv.ID,
"delivery_id = ?", d.ID,
).First(&result).Error)
assert.True(t, result.Success)
@@ -1116,10 +1116,10 @@ func TestDoHTTPRequest_ForwardsHeaders(t *testing.T) {
}
event := &database.Event{
Method: "POST",
Method: http.MethodPost,
Headers: `{"X-Custom":["value1"],"Content-Type":["application/json"]}`,
Body: `{"test":true}`,
ContentType: "application/json",
ContentType: testContentType,
}
statusCode, _, _, err := e.ExportDoHTTPRequest(
@@ -1141,7 +1141,7 @@ func TestDoHTTPRequest_ForwardsHeaders(t *testing.T) {
)
assert.Equal(t,
"application/json",
testContentType,
receivedHeaders.Get("Content-Type"),
)
@@ -1157,7 +1157,19 @@ func TestProcessDelivery_RoutesToCorrectHandler(
t.Parallel()
db := testWebhookDB(t)
e := testEngine(t, 1)
// The database target archives for real now, so the engine
// needs a webhook DB manager to locate the data directory.
e := delivery.NewTestEngineWithDB(
nil,
database.NewTestWebhookDBManager(t.TempDir()),
slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
)),
&http.Client{Timeout: 5 * time.Second},
1,
)
tests := []struct {
name string
@@ -1288,8 +1300,8 @@ func TestFormatSlackMessage_JSONBody(t *testing.T) {
t.Parallel()
event := &database.Event{
Method: "POST",
ContentType: "application/json",
Method: http.MethodPost,
ContentType: testContentType,
Body: `{"action":"push",` +
`"repo":"test/repo",` +
`"ref":"refs/heads/main"}`,
@@ -1314,7 +1326,7 @@ func TestFormatSlackMessage_NonJSONBody(t *testing.T) {
t.Parallel()
event := &database.Event{
Method: "POST",
Method: http.MethodPost,
ContentType: "text/plain",
Body: "hello world plain text",
}
@@ -1337,8 +1349,8 @@ func TestFormatSlackMessage_EmptyBody(t *testing.T) {
t.Parallel()
event := &database.Event{
Method: "POST",
ContentType: "application/json",
Method: http.MethodPost,
ContentType: testContentType,
Body: "",
}
event.CreatedAt = time.Date(
@@ -1366,8 +1378,8 @@ func TestFormatSlackMessage_LargeJSONTruncated(
require.NoError(t, err)
event := &database.Event{
Method: "POST",
ContentType: "application/json",
Method: http.MethodPost,
ContentType: testContentType,
Body: string(largeJSON),
}
event.CreatedAt = time.Date(
@@ -1652,6 +1664,179 @@ func TestProcessDelivery_RoutesToSlack(t *testing.T) {
)
}
// newLogCaptureEngine builds a test engine whose logger
// writes to the returned buffer, for inspecting log output.
func newLogCaptureEngine(
t *testing.T,
) (*delivery.Engine, *bytes.Buffer) {
t.Helper()
var buf bytes.Buffer
log := slog.New(slog.NewTextHandler(
&buf,
&slog.HandlerOptions{Level: slog.LevelDebug},
))
e := delivery.NewTestEngine(
log, &http.Client{Timeout: 5 * time.Second}, 1,
)
return e, &buf
}
// assertLogLineComplete asserts the captured log output
// carries the full inbound webhook content and ids.
func assertLogLineComplete(
t *testing.T, out string, event database.Event,
) {
t.Helper()
assert.Contains(t, out, "log-body-marker",
"log line must contain the full request body",
)
assert.Contains(t, out, "Content-Type",
"log line must contain the full request headers",
)
assert.Contains(t, out, event.EntrypointID,
"log line must contain the entrypoint id",
)
assert.Contains(t, out, event.WebhookID,
"log line must contain the webhook id",
)
assert.Contains(t, out, testContentType,
"log line must contain the content type",
)
}
func TestDeliverLog_LogsFullContent(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
e, buf := newLogCaptureEngine(t)
event := seedEvent(
t, db, `{"log-body-marker":"abc123"}`,
)
dlv := seedDelivery(
t, db, event.ID, uuid.New().String(),
database.DeliveryStatusPending,
)
d := &database.Delivery{
EventID: event.ID,
TargetID: dlv.TargetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: database.Target{
Name: "test-log-full",
Type: database.TargetTypeLog,
},
}
d.ID = dlv.ID
e.ExportDeliverLog(db, d)
assertLogLineComplete(t, buf.String(), event)
assertDeliveryStatus(t, db, dlv.ID,
database.DeliveryStatusDelivered,
)
}
// buildSlackRetryDelivery builds a Slack delivery whose
// target is configured with retries enabled.
func buildSlackRetryDelivery(
dlv database.Delivery,
event database.Event,
targetID, cfg string,
) *database.Delivery {
d := &database.Delivery{
EventID: event.ID,
TargetID: targetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: database.Target{
Name: "test-slack-retry",
Type: database.TargetTypeSlack,
Config: cfg,
MaxRetries: 5,
},
}
d.ID = dlv.ID
return d
}
func TestDeliverSlack_WithRetries_SchedulesRetry(
t *testing.T,
) {
t.Parallel()
db := testWebhookDB(t)
ts := newStatusServer(t, http.StatusServiceUnavailable)
e := testEngine(t, 1)
targetID := uuid.New().String()
slackCfg, err := json.Marshal(
delivery.SlackTargetConfig{WebhookURL: ts.URL},
)
require.NoError(t, err)
event := seedEvent(t, db, `{"slack":"retry"}`)
dlv := seedDelivery(
t, db, event.ID, targetID,
database.DeliveryStatusPending,
)
d := buildSlackRetryDelivery(
dlv, event, targetID, string(slackCfg),
)
task := &delivery.Task{
DeliveryID: dlv.ID,
TargetID: targetID,
TargetType: database.TargetTypeSlack,
MaxRetries: 5,
AttemptNum: 1,
}
e.ExportProcessDelivery(context.TODO(), db, d, task)
assertDeliveryStatus(t, db, dlv.ID,
database.DeliveryStatusRetrying,
)
assertDeliveryResult(
t, db, dlv.ID, false,
http.StatusServiceUnavailable,
)
}
// newStatusServer starts a test server that always responds
// with the given status code.
func newStatusServer(
t *testing.T, code int,
) *httptest.Server {
t.Helper()
ts := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(code)
},
))
t.Cleanup(ts.Close)
return ts
}
// readAll is a small helper to avoid importing io in
// a test handler inline.
func readAll(r interface {

363
internal/delivery/export_test.go
View File

@@ -7,10 +7,17 @@ import (
"net/http"
"time"
"go.uber.org/fx"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// ErrExportArchiveWriterEvicted exposes the sentinel returned by
// an evicted archive writer. It carries the Err prefix rather
// than this file's usual Export one because it is a sentinel
// error.
var ErrExportArchiveWriterEvicted = errArchiveWriterEvicted
// Exported constants for test access.
const (
ExportDeliveryChannelSize = deliveryChannelSize
@@ -39,37 +46,50 @@ func ExportTruncate(s string, maxLen int) string {
return truncate(s, maxLen)
}
// ExportDeliverHTTP exposes deliverHTTP for testing.
// ExportDeliverHTTP delivers via the http target for testing.
func (e *Engine) ExportDeliverHTTP(
ctx context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
) {
e.deliverHTTP(ctx, webhookDB, d, task)
e.httpTarget.Deliver(ctx, webhookDB, d, task, e)
}
// ExportDeliverDatabase exposes deliverDatabase.
// ExportDeliverDatabase delivers via the database target.
func (e *Engine) ExportDeliverDatabase(
webhookDB *gorm.DB, d *database.Delivery,
) {
e.deliverDatabase(webhookDB, d)
e.targets[database.TargetTypeDatabase].Deliver(
context.Background(), webhookDB, d, &Task{}, e,
)
}
// ExportDeliverLog exposes deliverLog for testing.
// ExportDeliverLog delivers via the log target for testing.
func (e *Engine) ExportDeliverLog(
webhookDB *gorm.DB, d *database.Delivery,
) {
e.deliverLog(webhookDB, d)
e.targets[database.TargetTypeLog].Deliver(
context.Background(), webhookDB, d, &Task{}, e,
)
}
// ExportDeliverSlack exposes deliverSlack for testing.
// ExportDeliverSlack delivers via the slack target for
// testing.
func (e *Engine) ExportDeliverSlack(
ctx context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
) {
e.deliverSlack(ctx, webhookDB, d)
task := &Task{
DeliveryID: d.ID,
TargetID: d.TargetID,
AttemptNum: 1,
}
e.targets[database.TargetTypeSlack].Deliver(
ctx, webhookDB, d, task, e,
)
}
// ExportProcessNewTask exposes processNewTask.
@@ -96,53 +116,56 @@ func (e *Engine) ExportProcessDelivery(
e.processDelivery(ctx, webhookDB, d, task)
}
// ExportGetCircuitBreaker exposes getCircuitBreaker.
// ExportGetCircuitBreaker exposes the http target's
// getCircuitBreaker.
func (e *Engine) ExportGetCircuitBreaker(
targetID string,
) *CircuitBreaker {
return e.getCircuitBreaker(targetID)
return e.httpTarget.getCircuitBreaker(targetID)
}
// ExportParseHTTPConfig exposes parseHTTPConfig.
func (e *Engine) ExportParseHTTPConfig(
configJSON string,
) (*HTTPTargetConfig, error) {
return e.parseHTTPConfig(configJSON)
return parseHTTPConfig(configJSON)
}
// ExportParseSlackConfig exposes parseSlackConfig.
func (e *Engine) ExportParseSlackConfig(
configJSON string,
) (*SlackTargetConfig, error) {
return e.parseSlackConfig(configJSON)
return parseSlackConfig(configJSON)
}
// ExportDoHTTPRequest exposes doHTTPRequest.
// ExportDoHTTPRequest exposes the http target's
// doHTTPRequest.
func (e *Engine) ExportDoHTTPRequest(
ctx context.Context,
cfg *HTTPTargetConfig,
event *database.Event,
) (int, string, int64, error) {
return e.doHTTPRequest(ctx, cfg, event)
return e.httpTarget.doHTTPRequest(ctx, cfg, event)
}
// ExportClientForConfig exposes clientForConfig.
// ExportClientForConfig exposes the http target's
// clientForConfig.
func (e *Engine) ExportClientForConfig(
cfg *HTTPTargetConfig,
) *http.Client {
return e.clientForConfig(cfg)
return e.httpTarget.clientForConfig(cfg)
}
// ExportClient returns the engine's shared HTTP client.
// ExportClient returns the http target's shared HTTP client.
func (e *Engine) ExportClient() *http.Client {
return e.client
return e.httpTarget.client
}
// ExportScheduleRetry exposes scheduleRetry.
// ExportScheduleRetry exposes ScheduleRetry.
func (e *Engine) ExportScheduleRetry(
task Task, delay time.Duration,
) {
e.scheduleRetry(task, delay)
e.ScheduleRetry(task, delay)
}
// ExportRecoverPendingDeliveries exposes
@@ -172,14 +195,40 @@ func (e *Engine) ExportRecoverInFlight(
e.recoverInFlight(ctx)
}
// ExportSweepWebhookRetries exposes sweepWebhookRetries.
func (e *Engine) ExportSweepWebhookRetries(
ctx context.Context, webhookID string,
) {
e.sweepWebhookRetries(ctx, webhookID)
}
// ExportStart exposes start for testing.
func (e *Engine) ExportStart(ctx context.Context) {
e.start(ctx)
func (e *Engine) ExportStart() {
e.start()
}
// ExportRegisterHooks registers the engine's real fx lifecycle
// hooks on a lifecycle supplied by a test, so a test can drive
// the exact OnStart/OnStop functions the application runs and
// hand OnStart the kind of context fx actually supplies.
func (e *Engine) ExportRegisterHooks(lc fx.Lifecycle) {
e.registerHooks(lc)
}
// ExportStop exposes stop for testing.
func (e *Engine) ExportStop() {
e.stop()
func (e *Engine) ExportStop(ctx context.Context) error {
return e.stop(ctx)
}
// ExportWedgeWorker adds a goroutine to the engine's WaitGroup
// that never observes cancellation and returns only when release
// is closed. It stands in for a worker stuck inside a delivery
// target that never returns, which is the only way stop can be
// made to outlast its context.
func (e *Engine) ExportWedgeWorker(release <-chan struct{}) {
e.wg.Go(func() {
<-release
})
}
// ExportDeliveryCh returns the delivery channel.
@@ -199,13 +248,15 @@ func NewTestEngine(
client *http.Client,
workers int,
) *Engine {
return &Engine{
e := &Engine{
log: log,
client: client,
deliveryCh: make(chan Task, deliveryChannelSize),
retryCh: make(chan Task, retryChannelSize),
workers: workers,
}
e.initTargets(client)
return e
}
// NewTestEngineSmallRetry creates an Engine with a tiny
@@ -213,10 +264,13 @@ func NewTestEngine(
func NewTestEngineSmallRetry(
log *slog.Logger,
) *Engine {
return &Engine{
e := &Engine{
log: log,
retryCh: make(chan Task, 1),
}
e.initTargets(nil)
return e
}
// NewTestEngineWithDB creates an Engine with a real
@@ -228,15 +282,17 @@ func NewTestEngineWithDB(
client *http.Client,
workers int,
) *Engine {
return &Engine{
e := &Engine{
database: db,
dbManager: dbMgr,
log: log,
client: client,
deliveryCh: make(chan Task, deliveryChannelSize),
retryCh: make(chan Task, retryChannelSize),
workers: workers,
}
e.initTargets(client)
return e
}
// NewTestCircuitBreaker creates a CircuitBreaker with
@@ -250,3 +306,252 @@ func NewTestCircuitBreaker(
cooldown: cooldown,
}
}
// ExportArchivedEvent aliases the archive row type so black-box
// tests can construct and read archive rows.
type ExportArchivedEvent = archivedEvent
// ExportArchiveWriter wraps an archiveWriter so black-box tests
// can exercise the per-webhook archive file mechanics.
type ExportArchiveWriter struct {
w *archiveWriter
}
// NewExportArchiveWriter builds an archive writer for tests,
// optionally overriding the reopen debounce (a non-positive
// debounce keeps the production default).
func NewExportArchiveWriter(
path string, log *slog.Logger, debounce time.Duration,
) *ExportArchiveWriter {
w := newArchiveWriter(path, log)
if debounce > 0 {
w.debounce = debounce
}
return &ExportArchiveWriter{w: w}
}
// Write archives a row through the writer.
func (e *ExportArchiveWriter) Write(
row ExportArchivedEvent, expiry time.Duration,
) error {
return e.w.write(row, expiry)
}
// Open opens the archive file, pruning when expiry is positive.
func (e *ExportArchiveWriter) Open(expiry time.Duration) error {
return e.w.open(expiry)
}
// Reopen closes and reopens the archive file.
func (e *ExportArchiveWriter) Reopen(
expiry time.Duration,
) error {
return e.w.reopen(expiry)
}
// Reopens reports how many times the file has been opened.
func (e *ExportArchiveWriter) Reopens() int {
return e.w.reopens
}
// DB returns the writer's current open handle for row
// inspection in tests.
func (e *ExportArchiveWriter) DB() *gorm.DB {
return e.w.db
}
// Path returns the archive file the writer owns.
func (e *ExportArchiveWriter) Path() string {
return e.w.path
}
// OpenExisting opens the archive without permitting creation,
// the way the idle sweep does.
func (e *ExportArchiveWriter) OpenExisting(
expiry time.Duration,
) error {
return e.w.openMode(archiveModeExisting, expiry)
}
// SweepExpired runs an idle sweep of the archive.
func (e *ExportArchiveWriter) SweepExpired(
expiry time.Duration,
) error {
return e.w.sweepExpired(expiry)
}
// Evict marks the writer evicted and closes its handle, exactly
// as leaving the registry does.
func (e *ExportArchiveWriter) Evict() {
e.w.evict()
}
// HandleOpen reports whether the writer currently holds an open
// archive handle.
func (e *ExportArchiveWriter) HandleOpen() bool {
e.w.mu.Lock()
defer e.w.mu.Unlock()
return e.w.db != nil
}
// Same reports whether both wrappers refer to the very same
// underlying archive writer, so a test can prove a registry entry
// is the writer it was handed rather than a replacement.
func (e *ExportArchiveWriter) Same(
other *ExportArchiveWriter,
) bool {
return other != nil && e.w == other.w
}
// ExportArchiveWriterFor returns the archive writer the registry
// currently caches for a webhook, or nil when none is cached. It
// never creates one, so a test can hold a reference to the very
// writer an eviction is about to detach.
func (e *Engine) ExportArchiveWriterFor(
webhookID string,
) *ExportArchiveWriter {
e.dbTarget.mu.Lock()
defer e.dbTarget.mu.Unlock()
w, ok := e.dbTarget.writers[webhookID]
if !ok {
return nil
}
return &ExportArchiveWriter{w: w}
}
// ExportHasArchiveWriter reports whether the database target
// currently caches an archive writer for a webhook.
func (e *Engine) ExportHasArchiveWriter(
webhookID string,
) bool {
e.dbTarget.mu.Lock()
defer e.dbTarget.mu.Unlock()
_, ok := e.dbTarget.writers[webhookID]
return ok
}
// ExportArchiveHandleOpen reports whether the cached archive
// writer for a webhook holds an open database handle. It
// returns false when no writer is cached.
func (e *Engine) ExportArchiveHandleOpen(
webhookID string,
) bool {
e.dbTarget.mu.Lock()
w, ok := e.dbTarget.writers[webhookID]
e.dbTarget.mu.Unlock()
if !ok {
return false
}
w.mu.Lock()
defer w.mu.Unlock()
return w.db != nil
}
// ExportEnsureArchiveWriter creates (if needed) and returns the
// archive file path of the cached writer for a webhook, so a
// test can prime the registry the way a delivery would.
func (e *Engine) ExportEnsureArchiveWriter(
webhookID string,
) (string, error) {
w, err := e.dbTarget.writerFor(webhookID)
if err != nil {
return "", err
}
return w.path, nil
}
// ExportSweepWriterFor takes a webhook's registry writer exactly
// as the idle sweep does, reporting whether the sweep had to
// create the entry. It lets a test drive the registry through the
// sweep's own entry point instead of choreographing goroutines.
func (e *Engine) ExportSweepWriterFor(
webhookID string,
) (*ExportArchiveWriter, bool, error) {
w, created, err := e.dbTarget.sweepWriterFor(webhookID)
if err != nil {
return nil, false, err
}
return &ExportArchiveWriter{w: w}, created, nil
}
// ExportReleaseSweepWriter releases a sweep-created registry entry
// exactly as a finished sweep does.
func (e *Engine) ExportReleaseSweepWriter(
webhookID string, w *ExportArchiveWriter,
) {
e.dbTarget.releaseSweepWriter(webhookID, w.w)
}
// NewTestArchiveSweeper builds an ArchiveSweeper backed by the
// given main database and engine, without the fx lifecycle.
// Intended for tests.
func NewTestArchiveSweeper(
db *database.Database,
eng *Engine,
log *slog.Logger,
) *ArchiveSweeper {
return &ArchiveSweeper{
db: db,
eng: eng,
log: log,
interval: time.Hour,
}
}
// ExportSweep runs a single archive sweep synchronously for
// tests.
func (s *ArchiveSweeper) ExportSweep(ctx context.Context) {
s.sweep(ctx)
}
// ExportStart starts the sweeper's background loop for tests.
func (s *ArchiveSweeper) ExportStart() {
s.start()
}
// ExportRegisterHooks registers the sweeper's real fx lifecycle
// hooks on a lifecycle supplied by a test, so a test can drive
// the exact OnStart/OnStop functions the application runs and
// hand OnStart the kind of context fx actually supplies.
func (s *ArchiveSweeper) ExportRegisterHooks(lc fx.Lifecycle) {
s.registerHooks(lc)
}
// ExportStop stops the sweeper's background loop for tests.
func (s *ArchiveSweeper) ExportStop(ctx context.Context) error {
return s.stop(ctx)
}
// ExportWedgeLoop adds a goroutine to the sweeper's WaitGroup
// that never observes cancellation and returns only when release
// is closed. It stands in for a prune stuck on a locked archive.
func (s *ArchiveSweeper) ExportWedgeLoop(
release <-chan struct{},
) {
s.wg.Go(func() {
<-release
})
}
// ExportSetInterval overrides the sweep interval for tests.
func (s *ArchiveSweeper) ExportSetInterval(d time.Duration) {
s.interval = d
}
// ExportParseArchiveExpiry exposes parseArchiveExpiry.
func ExportParseArchiveExpiry(
configJSON string,
) (time.Duration, error) {
return parseArchiveExpiry(configJSON)
}

7
internal/delivery/ssrf.go
View File

@@ -92,7 +92,12 @@ func ValidateTargetURL(
) error {
parsed, err := url.Parse(targetURL)
if err != nil {
return fmt.Errorf("invalid URL: %w", err)
// url.Parse embeds the whole URL in its error, and
// this one is logged and shown; mask it. Every other
// branch below reports only the hostname.
return fmt.Errorf(
"invalid URL: %w", maskURLError(err),
)
}
err = validateScheme(parsed.Scheme)

104
internal/delivery/target.go Normal file
View File

@@ -0,0 +1,104 @@
package delivery
import (
"context"
"net/http"
"time"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// Scheduler re-enqueues a task for a future delivery attempt.
// The engine provides one to each target so a target can own
// its retries durably: it records the attempt, marks the
// delivery retrying, and asks the Scheduler to deliver the
// next attempt after delay — exactly what the engine does for
// its own restart recovery.
type Scheduler interface {
ScheduleRetry(task Task, delay time.Duration)
}
// Target delivers an event to one target type. Each type is
// an implementation. A Target owns its whole delivery: it
// makes the attempt, records the DeliveryResult and updates
// the DeliveryStatus, and — for targets that retry — decides
// whether to retry, computes its own backoff, gates with its
// own circuit breaker, and reschedules via the injected
// Scheduler. Fire-and-forget targets simply record a single
// attempt.
type Target interface {
Deliver(
ctx context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
)
}
// rescheduler is implemented by targets that own durable
// retries. The engine's restart recovery and periodic sweep
// use it to let the target recompute the schedule for an
// orphaned retrying delivery, keeping the retry schedule
// target-owned. Fire-and-forget targets do not implement it
// and their (never-occurring) retrying deliveries are
// skipped.
type rescheduler interface {
// remainingBackoff returns how long to wait before the
// next attempt of a recovered retrying delivery.
remainingBackoff(
webhookDB *gorm.DB,
deliveryID string,
attemptNum int,
) time.Duration
// backoffElapsed reports whether the backoff window for
// the last attempt has already passed, so the periodic
// sweep can re-enqueue the delivery now.
backoffElapsed(
webhookDB *gorm.DB,
deliveryID string,
attemptNum int,
) bool
}
// attemptResult is the outcome of a single delivery attempt,
// as reported by a target's per-attempt function to the
// shared retry core.
type attemptResult struct {
statusCode int
respBody string
duration int64
success bool
errMsg string
}
// initTargets builds the target registry, wiring each target
// to the engine's persistence helpers and giving the HTTP and
// Slack targets the shared SSRF-safe client. It is called by
// both New and the test constructors so the registry is
// always populated.
func (e *Engine) initTargets(client *http.Client) {
httpT := &httpTarget{
httpCore: &httpCore{eng: e},
client: client,
}
slackT := &slackTarget{
httpCore: &httpCore{eng: e},
client: client,
}
dbT := &databaseTarget{eng: e}
e.httpTarget = httpT
e.dbTarget = dbT
e.targets = map[database.TargetType]Target{
database.TargetTypeHTTP: httpT,
database.TargetTypeSlack: slackT,
database.TargetTypeDatabase: dbT,
database.TargetTypeLog: &logTarget{eng: e},
}
}

208
internal/delivery/target_config_view.go Normal file
View File

@@ -0,0 +1,208 @@
package delivery
import (
"encoding/json"
"fmt"
"strconv"
"sneak.berlin/go/webhooker/internal/database"
)
// configUnavailable is what a target's configuration renders
// as when it is absent, of an unknown type, or does not
// parse. The stored blob is never shown as a fallback: it can
// hold a credential (a Slack incoming webhook URL is a bearer
// token) and a UI that prints it leaks that credential into
// browser history, screenshots and screen shares.
const configUnavailable = "(unavailable)"
// ConfigField is one labelled, display-safe value derived
// from a target's stored configuration.
type ConfigField struct {
Label string
Value string
}
// TargetView is the display-safe projection of a target for
// the UI. It deliberately has no raw configuration field, so
// no template — present or future — can render the stored
// blob.
type TargetView struct {
ID string
Name string
Type database.TargetType
Active bool
Config []ConfigField
}
// NewTargetViews projects targets for rendering, replacing
// each stored configuration blob with named, display-safe
// fields.
func NewTargetViews(
targets []database.Target,
) []TargetView {
views := make([]TargetView, 0, len(targets))
for i := range targets {
t := &targets[i]
views = append(views, TargetView{
ID: t.ID,
Name: t.Name,
Type: t.Type,
Active: t.Active,
Config: targetConfigFields(t),
})
}
return views
}
// targetConfigFields returns the display-safe fields for a
// target's configuration. Anything it cannot parse becomes
// the neutral placeholder.
func targetConfigFields(
t *database.Target,
) []ConfigField {
switch t.Type {
case database.TargetTypeSlack:
return slackConfigFields(t.Config)
case database.TargetTypeHTTP:
return httpConfigFields(t)
case database.TargetTypeDatabase:
return databaseConfigFields(t.Config)
case database.TargetTypeLog:
// The log target takes no configuration.
return nil
default:
return unavailableConfigFields()
}
}
// unavailableConfigFields is the neutral placeholder shown
// for a configuration that could not be presented.
func unavailableConfigFields() []ConfigField {
return []ConfigField{{
Label: "Configuration",
Value: configUnavailable,
}}
}
// slackConfigFields describes a Slack target. Only the masked
// webhook URL is shown; the full URL is the credential.
func slackConfigFields(configJSON string) []ConfigField {
cfg, err := parseSlackConfig(configJSON)
if err != nil {
return unavailableConfigFields()
}
return []ConfigField{{
Label: "Webhook URL",
Value: cfg.MaskedWebhookURL(),
}}
}
// httpConfigFields describes an HTTP target: its destination
// and its retry settings. Header values are not shown — they
// routinely carry authorization tokens — only how many are
// configured.
//
// The destination is masked to scheme and host by the same
// rule the Slack target uses. An HTTP target's destination is
// commonly a Slack, Discord or Teams incoming-webhook endpoint
// whose path segments are the credential, and the field takes
// an arbitrary URL, so no segment can be assumed non-secret.
func httpConfigFields(t *database.Target) []ConfigField {
cfg, err := parseHTTPConfig(t.Config)
if err != nil {
return unavailableConfigFields()
}
fields := []ConfigField{{
Label: "Destination URL",
Value: MaskURL(cfg.URL),
}}
if cfg.Timeout > 0 {
fields = append(fields, ConfigField{
Label: "Timeout",
Value: strconv.Itoa(cfg.Timeout) + "s",
})
}
if len(cfg.Headers) > 0 {
fields = append(fields, ConfigField{
Label: "Headers",
Value: fmt.Sprintf(
"%d configured", len(cfg.Headers),
),
})
}
return append(fields, retryFields(t)...)
}
// retryFields describes a target's retry settings, which live
// on the target row rather than in its configuration blob.
func retryFields(t *database.Target) []ConfigField {
retries := strconv.Itoa(t.MaxRetries)
if t.MaxRetries == 0 {
retries += " (fire-and-forget)"
}
fields := []ConfigField{{
Label: "Max Retries",
Value: retries,
}}
if t.MaxQueueSize > 0 {
fields = append(fields, ConfigField{
Label: "Max Queue Size",
Value: strconv.Itoa(t.MaxQueueSize),
})
}
return fields
}
// databaseConfigFields describes an archive target. Its
// configuration is optional, and an absent or empty expiry
// means the archive is kept forever. An expiry that is set
// but not a valid duration is reported as unavailable rather
// than echoed back.
func databaseConfigFields(configJSON string) []ConfigField {
expiry := archiveExpiryNever
if configJSON != "" {
var cfg databaseTargetConfig
err := json.Unmarshal([]byte(configJSON), &cfg)
if err != nil {
return unavailableConfigFields()
}
if cfg.Expiry != "" {
if ValidateArchiveExpiry(cfg.Expiry) != nil {
return unavailableConfigFields()
}
expiry = cfg.Expiry
}
}
return []ConfigField{{
Label: "Archive Expiry",
Value: expiry,
}}
}
// MaskedWebhookURL returns the Slack webhook URL reduced to
// its scheme and host, with the path, query and any userinfo
// elided. The path segments are the credential, so none of
// them is shown: the field accepts an arbitrary URL, so no
// segment can be assumed non-secret. A URL that does not
// parse into a scheme and host yields the neutral
// placeholder, never the raw string.
func (c *SlackTargetConfig) MaskedWebhookURL() string {
return MaskURL(c.WebhookURL)
}

328
internal/delivery/target_config_view_test.go Normal file
View File

@@ -0,0 +1,328 @@
package delivery_test
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
const (
// slackSecretPath is the credential-bearing part of a
// Slack incoming webhook URL: everything after the host.
slackSecretPath = "/services/T00000000/B00000000/" +
"XXXXXXXXXXXXXXXXXXXXXXXX"
slackWebhookURL = "https://hooks.slack.com" +
slackSecretPath
viewExampleOrigin = "https://example.com"
viewExampleHook = viewExampleOrigin + "/hook"
viewMaskedOrigin = viewExampleOrigin + "/..."
viewUnavailable = "(unavailable)"
viewExpiryNever = "never"
)
func TestMaskedWebhookURL(t *testing.T) {
t.Parallel()
tests := map[string]struct {
url string
want string
}{
"slack webhook": {
url: slackWebhookURL,
want: "https://hooks.slack.com/...",
},
"query string dropped": {
url: viewExampleOrigin + "/a?token=secret",
want: viewExampleOrigin + "/...",
},
// Fabricated userinfo in a test URL, not a real
// credential.
//nolint:gosec // G101
"userinfo dropped": {
url: "https://user:pw@example.com/a/b",
want: viewExampleOrigin + "/...",
},
"no path": {
url: viewExampleOrigin,
want: viewExampleOrigin,
},
"root path": {
url: viewExampleOrigin + "/",
want: viewExampleOrigin,
},
"not a url": {
url: "definitely not a url",
want: viewUnavailable,
},
"empty": {
url: "",
want: viewUnavailable,
},
}
for name, tc := range tests {
t.Run(name, func(t *testing.T) {
t.Parallel()
cfg := &delivery.SlackTargetConfig{
WebhookURL: tc.url,
}
assert.Equal(
t, tc.want, cfg.MaskedWebhookURL(),
)
})
}
}
// TestMaskedWebhookURL_NeverLeaksPath is the direct
// expression of the rule: whatever the input, the masked
// value never contains a path segment of it.
func TestMaskedWebhookURL_NeverLeaksPath(t *testing.T) {
t.Parallel()
cfg := &delivery.SlackTargetConfig{
WebhookURL: slackWebhookURL,
}
masked := cfg.MaskedWebhookURL()
assert.NotContains(t, masked, "T00000000")
assert.NotContains(t, masked, "B00000000")
assert.NotContains(
t, masked, "XXXXXXXXXXXXXXXXXXXXXXXX",
)
assert.NotContains(t, masked, slackSecretPath)
}
// fieldMap turns a view's config fields into a lookup so
// assertions read by label.
func fieldMap(fields []delivery.ConfigField) map[string]string {
out := make(map[string]string, len(fields))
for _, f := range fields {
out[f.Label] = f.Value
}
return out
}
// viewFor projects a single target and returns its view.
func viewFor(
t *testing.T,
target database.Target,
) delivery.TargetView {
t.Helper()
views := delivery.NewTargetViews(
[]database.Target{target},
)
require.Len(t, views, 1)
return views[0]
}
func TestNewTargetViews_Slack(t *testing.T) {
t.Parallel()
view := viewFor(t, database.Target{
Name: "slack-target",
Type: database.TargetTypeSlack,
Active: true,
Config: `{"webhookUrl":"` +
slackWebhookURL + `"}`,
})
assert.Equal(t, "slack-target", view.Name)
assert.Equal(
t,
map[string]string{
"Webhook URL": "https://hooks.slack.com/...",
},
fieldMap(view.Config),
)
}
func TestNewTargetViews_HTTP(t *testing.T) {
t.Parallel()
view := viewFor(t, database.Target{
Type: database.TargetTypeHTTP,
Config: `{"url":"` + viewExampleHook + `",` +
`"timeout":30,` +
`"headers":{"Authorization":"Bearer sekrit"}}`,
MaxRetries: 5,
MaxQueueSize: 100,
})
fields := fieldMap(view.Config)
assert.Equal(
t,
map[string]string{
"Destination URL": viewMaskedOrigin,
"Timeout": "30s",
"Headers": "1 configured",
"Max Retries": "5",
"Max Queue Size": "100",
},
fields,
)
// Header values can be credentials and are never shown.
for _, v := range fields {
assert.NotContains(t, v, "sekrit")
}
}
func TestNewTargetViews_HTTPFireAndForget(t *testing.T) {
t.Parallel()
view := viewFor(t, database.Target{
Type: database.TargetTypeHTTP,
Config: `{"url":"` + viewExampleHook + `"}`,
})
assert.Equal(
t,
map[string]string{
"Destination URL": viewMaskedOrigin,
"Max Retries": "0 (fire-and-forget)",
},
fieldMap(view.Config),
)
}
// TestNewTargetViews_HTTPMasksDestinationURL proves the rule
// holds for the http target too: an http destination is
// routinely an incoming-webhook endpoint whose path segments
// are the credential, so none of them is shown.
func TestNewTargetViews_HTTPMasksDestinationURL(t *testing.T) {
t.Parallel()
view := viewFor(t, database.Target{
Type: database.TargetTypeHTTP,
Config: `{"url":"` + slackWebhookURL + `"}`,
})
fields := fieldMap(view.Config)
assert.Equal(
t,
"https://hooks.slack.com/...",
fields["Destination URL"],
)
for _, v := range fields {
assert.NotContains(t, v, slackSecretPath)
assert.NotContains(t, v, "T00000000")
assert.NotContains(t, v, "B00000000")
assert.NotContains(t, v, "XXXXXXXXXXXXXXXXXXXXXXXX")
}
}
func TestNewTargetViews_Database(t *testing.T) {
t.Parallel()
tests := map[string]struct {
config string
want string
}{
"empty config": {config: "", want: viewExpiryNever},
"empty expiry": {config: `{}`, want: viewExpiryNever},
"explicit": {
config: `{"expiry":"720h"}`,
want: "720h",
},
"never literal": {
config: `{"expiry":"` + viewExpiryNever + `"}`,
want: viewExpiryNever,
},
}
for name, tc := range tests {
t.Run(name, func(t *testing.T) {
t.Parallel()
view := viewFor(t, database.Target{
Type: database.TargetTypeDatabase,
Config: tc.config,
})
assert.Equal(
t,
map[string]string{"Archive Expiry": tc.want},
fieldMap(view.Config),
)
})
}
}
func TestNewTargetViews_Log(t *testing.T) {
t.Parallel()
view := viewFor(t, database.Target{
Type: database.TargetTypeLog,
Config: "",
})
assert.Empty(t, view.Config)
}
// TestNewTargetViews_Unpresentable proves that no config the
// view cannot present falls back to the stored blob.
func TestNewTargetViews_Unpresentable(t *testing.T) {
t.Parallel()
const blob = `{"webhookUrl":"https://hooks.slack.com` +
slackSecretPath + `"`
tests := map[string]database.Target{
"unknown target type": {
Type: database.TargetType("carrier-pigeon"),
Config: blob,
},
"unparseable json": {
Type: database.TargetTypeSlack,
Config: blob,
},
"empty slack config": {
Type: database.TargetTypeSlack,
},
"slack config without url": {
Type: database.TargetTypeSlack,
Config: `{}`,
},
"unparseable http json": {
Type: database.TargetTypeHTTP,
Config: `{"url":`,
},
"unparseable archive json": {
Type: database.TargetTypeDatabase,
Config: `{"expiry":`,
},
"invalid archive expiry": {
Type: database.TargetTypeDatabase,
Config: `{"expiry":"a fortnight"}`,
},
}
for name, target := range tests {
t.Run(name, func(t *testing.T) {
t.Parallel()
view := viewFor(t, target)
assert.Equal(
t,
map[string]string{
"Configuration": viewUnavailable,
},
fieldMap(view.Config),
)
})
}
}

295
internal/delivery/target_database.go Normal file
View File

@@ -0,0 +1,295 @@
package delivery
import (
"context"
"fmt"
"path/filepath"
"sync"
"time"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// databaseTarget is a no-retry target that archives the
// full inbound event into a per-webhook archive SQLite file,
// separate from the per-webhook event database. The event is
// already persisted in the per-webhook event DB by the time
// delivery runs; the database target additionally writes a
// durable long-term copy into archive-{webhookID}.db and then
// records a single attempt whose outcome reflects whether the
// archive write succeeded. See archiveWriter for the
// close/reopen, auto-recreate, and expiry semantics.
type databaseTarget struct {
eng *Engine
mu sync.Mutex
writers map[string]*archiveWriter
}
// Deliver implements Target. It archives the event, then
// records one successful attempt and marks the delivery
// delivered. An archiving error fails the delivery: the
// attempt is recorded as failed with the error and the
// delivery is marked failed, so a target that could not do
// its one job (archiving) never reports success. The target
// does not retry; the event remains durably stored in the
// per-webhook event database.
func (t *databaseTarget) Deliver(
_ context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
_ *Task,
_ Scheduler,
) {
err := t.archive(d)
if err != nil {
t.eng.log.Error(
"failed to archive event to database target",
"delivery_id", d.ID,
"event_id", d.EventID,
"error", err,
)
t.eng.recordResult(
webhookDB, d, 1, false, 0, "",
err.Error(), 0,
)
t.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusFailed,
)
return
}
t.eng.recordResult(
webhookDB, d, 1, true, 0, "", "", 0,
)
t.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusDelivered,
)
}
// archive writes the full event as a row into the webhook's
// archive database, honouring the optional per-target expiry
// parsed from the target config JSON.
func (t *databaseTarget) archive(d *database.Delivery) error {
webhookID := d.Event.WebhookID
if webhookID == "" {
return errArchiveMissingWebhookID
}
expiry, err := parseArchiveExpiry(d.Target.Config)
if err != nil {
return err
}
w, err := t.writerFor(webhookID)
if err != nil {
return err
}
row := archivedEvent{
EventID: d.Event.ID,
WebhookID: webhookID,
EntrypointID: d.Event.EntrypointID,
Method: d.Event.Method,
Headers: d.Event.Headers,
Body: d.Event.Body,
ContentType: d.Event.ContentType,
}
return w.write(row, expiry)
}
// writerFor returns the archiveWriter for a webhook, creating
// and caching it on first use. Each webhook has one writer so
// its close/reopen debounce state is shared across concurrent
// deliveries. The archive file lives beside the per-webhook
// event database in the data directory.
func (t *databaseTarget) writerFor(
webhookID string,
) (*archiveWriter, error) {
path, err := t.archivePath(webhookID)
if err != nil {
return nil, err
}
t.mu.Lock()
defer t.mu.Unlock()
if t.writers == nil {
t.writers = make(map[string]*archiveWriter)
}
w, ok := t.writers[webhookID]
if !ok {
w = newArchiveWriter(path, t.eng.log)
t.writers[webhookID] = w
}
// A delivery claims the entry: even if the idle sweep created
// it moments ago, it now belongs to the registry proper and
// the sweep must leave it in place when it finishes.
w.sweepOwned = false
return w, nil
}
// sweepWriterFor returns the archive writer the idle sweep should
// prune a webhook through, together with whether the sweep itself
// created the registry entry.
//
// The sweep must route its prune through the registered writer so
// the writer's mutex orders it against concurrent writes, but it
// must never leave a registry entry behind: a sweep that ran
// concurrently with the webhook's deletion would otherwise
// re-create an entry that nothing will ever evict again, which is
// exactly the leak eviction exists to prevent. An entry the sweep
// creates is therefore marked sweep-owned and handed back to
// releaseSweepWriter when the sweep is done.
func (t *databaseTarget) sweepWriterFor(
webhookID string,
) (*archiveWriter, bool, error) {
path, err := t.archivePath(webhookID)
if err != nil {
return nil, false, err
}
t.mu.Lock()
defer t.mu.Unlock()
if t.writers == nil {
t.writers = make(map[string]*archiveWriter)
}
w, ok := t.writers[webhookID]
if ok {
return w, false, nil
}
w = newArchiveWriter(path, t.eng.log)
w.sweepOwned = true
t.writers[webhookID] = w
return w, true, nil
}
// releaseSweepWriter drops a registry entry that the idle sweep
// created, so a sweep leaves the registry exactly as it found it.
//
// The entry is removed only if it is still the very writer the
// sweep installed and no delivery has claimed it in the meantime
// (writerFor clears sweepOwned when it hands a writer to the
// write path). Both conditions are evaluated under the registry
// lock, so an eviction that raced the sweep — which removes the
// entry outright — simply finds nothing left to do here, and a
// delivery that adopted the writer keeps a registered, evictable
// one.
func (t *databaseTarget) releaseSweepWriter(
webhookID string, w *archiveWriter,
) {
t.mu.Lock()
defer t.mu.Unlock()
cur, ok := t.writers[webhookID]
if !ok || cur != w || !cur.sweepOwned {
return
}
delete(t.writers, webhookID)
}
// archivePath returns the archive file path for a webhook: it
// lives beside the per-webhook event database in the data
// directory. It does not touch the filesystem.
func (t *databaseTarget) archivePath(
webhookID string,
) (string, error) {
if t.eng.dbManager == nil {
return "", errArchiveNoDataDir
}
dir := filepath.Dir(t.eng.dbManager.DBPath(webhookID))
return filepath.Join(
dir, fmt.Sprintf("archive-%s.db", webhookID),
), nil
}
// evict drops a webhook's archive writer from the registry and
// closes its handle, so a deleted webhook does not leave a
// writer (and an open archive handle within its debounce
// window) alive for the process lifetime.
//
// The map entry is removed under the registry lock, which is
// then released before the handle is closed under the writer's
// own lock: that ordering keeps the registry available to other
// webhooks while an in-flight write on this one drains, and
// closing under the writer's lock means eviction can never race
// a write.
//
// Eviction is idempotent and silent for a webhook with no
// writer, which is the common case: a webhook with no database
// target never creates one. It never deletes the archive file.
func (t *databaseTarget) evict(webhookID string) {
t.mu.Lock()
w, ok := t.writers[webhookID]
if ok {
delete(t.writers, webhookID)
}
t.mu.Unlock()
if !ok {
return
}
w.evict()
t.eng.log.Info(
"evicted archive writer",
"webhook_id", webhookID,
"path", w.path,
)
}
// sweepWebhook prunes one webhook's archive of rows older than
// expiry, without requiring a write. It returns nil (nothing to
// do) when the archive file does not exist, so a sweep never
// creates an archive for a webhook that has a database target
// but has never received an event.
//
// It also never leaves a registry entry behind: an entry it had
// to create to reach the writer's mutex is released again once
// the prune is done, so a sweep racing a webhook deletion cannot
// resurrect the writer the eviction just dropped.
func (t *databaseTarget) sweepWebhook(
webhookID string, expiry time.Duration,
) error {
path, err := t.archivePath(webhookID)
if err != nil {
return err
}
// Check before taking a writer at all: a webhook whose
// archive has never been created gets no writer, no handle,
// and no file.
if !fileExists(path) {
return nil
}
w, created, err := t.sweepWriterFor(webhookID)
if err != nil {
return err
}
if created {
defer t.releaseSweepWriter(webhookID, w)
}
return w.sweepExpired(expiry)
}

436
internal/delivery/target_database_archive.go Normal file
View File

@@ -0,0 +1,436 @@
package delivery
import (
"database/sql"
"encoding/json"
"errors"
"fmt"
"log/slog"
"os"
"sync"
"time"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/gormlog"
)
// archiveExpiryNever is the expiry sentinel (and default) that
// disables pruning so archived rows are kept forever.
const archiveExpiryNever = "never"
// archiveReopenDebounce bounds how often an archive file is
// closed and reopened. After each write the handle is closed
// and reopened so an operator can move the file away for
// offline archiving, but never more than once per this window.
const archiveReopenDebounce = time.Second
const (
// archiveModeCreate is the SQLite URI mode used by the write
// path: open the archive file, creating it if missing, so a
// first write (or a write after the operator moved the file
// away) recreates it.
archiveModeCreate = "rwc"
// archiveModeExisting is the SQLite URI mode used by the idle
// sweep: open read-write but never create. A sweep must never
// conjure an empty archive file for a webhook that has a
// database target but has never received an event.
archiveModeExisting = "rw"
)
var (
// errArchiveMissingWebhookID is returned when an event to
// archive has no webhook id to key its archive file on.
errArchiveMissingWebhookID = errors.New(
"cannot archive event without a webhook id",
)
// errArchiveNoDataDir is returned when the database target
// has no webhook database manager and so cannot locate the
// data directory for archive files.
errArchiveNoDataDir = errors.New(
"database target has no data directory",
)
// errArchiveExpiryNotPositive is returned when a
// user-supplied archive expiry parses as a duration but is
// zero or negative; "never" is the way to disable pruning.
errArchiveExpiryNotPositive = errors.New(
"expiry must be a positive duration or \"never\"",
)
// errArchiveWriterEvicted is returned when a writer that has
// been evicted (its webhook was deleted, or its last database
// target was removed) is used again. An evicted writer is no
// longer in the registry, so reopening its file would leak a
// handle nothing owns.
errArchiveWriterEvicted = errors.New(
"archive writer has been evicted",
)
)
// databaseTargetConfig is the optional per-target JSON config
// for a database (archive) target.
type databaseTargetConfig struct {
// Expiry is a Go duration (e.g. "720h") after which
// archived rows are pruned, or "never" (the default) to
// keep them forever.
Expiry string `json:"expiry"`
}
// archivedEvent is one fully captured webhook event stored in a
// per-webhook archive database for long-term retention. It is a
// self-contained copy — independent of the per-webhook event
// database, which may prune events under its own retention.
type archivedEvent struct {
ID uint `gorm:"primaryKey;autoIncrement"`
EventID string `gorm:"index"`
WebhookID string
EntrypointID string
Method string
Headers string
Body string
ContentType string
// ArchivedAt is when the row was archived and is the age
// basis for expiry pruning.
ArchivedAt time.Time `gorm:"index"`
}
// parseArchiveExpiry reads the optional expiry from a database
// target's config JSON. An empty config, an empty expiry, or
// the literal "never" all mean keep forever, returned as a zero
// duration. Any other value must parse as a positive Go
// duration; a set-but-invalid value (unparseable, zero, or
// negative) is an error rather than a silent default, matching
// ValidateArchiveExpiry at target creation.
func parseArchiveExpiry(
configJSON string,
) (time.Duration, error) {
if configJSON == "" {
return 0, nil
}
var cfg databaseTargetConfig
err := json.Unmarshal([]byte(configJSON), &cfg)
if err != nil {
return 0, fmt.Errorf(
"parsing database target config: %w", err,
)
}
if cfg.Expiry == "" || cfg.Expiry == archiveExpiryNever {
return 0, nil
}
dur, err := time.ParseDuration(cfg.Expiry)
if err != nil {
return 0, fmt.Errorf(
"parsing archive expiry %q: %w", cfg.Expiry, err,
)
}
if dur <= 0 {
return 0, fmt.Errorf(
"%w: %q", errArchiveExpiryNotPositive, cfg.Expiry,
)
}
return dur, nil
}
// ValidateArchiveExpiry checks a user-supplied archive expiry
// for a database target at configuration time. Valid values are
// empty, "never" (both meaning keep forever), or a positive Go
// duration such as "720h". Anything else is an error, so a bad
// expiry is rejected when the target is created rather than
// failing every subsequent delivery.
func ValidateArchiveExpiry(expiry string) error {
if expiry == "" || expiry == archiveExpiryNever {
return nil
}
dur, err := time.ParseDuration(expiry)
if err != nil {
return fmt.Errorf(
"expiry must be %q or a Go duration "+
"such as \"720h\": %w",
archiveExpiryNever, err,
)
}
if dur <= 0 {
return fmt.Errorf(
"%w: %q", errArchiveExpiryNotPositive, expiry,
)
}
return nil
}
// archiveWriter owns one per-webhook archive SQLite file. It
// serialises writes, and after each write closes and reopens
// the file (debounced to at most once per debounce window) so
// an operator can move the file away for offline archiving. The
// next write recreates a moved or removed file, because the
// file is opened create-if-missing and its schema is migrated
// on every open.
type archiveWriter struct {
mu sync.Mutex
path string
log *slog.Logger
debounce time.Duration
db *gorm.DB
lastReopen time.Time
reopens int
// evicted marks a writer that has been removed from the
// per-webhook registry. Its handle is closed and it must
// never open the file again: nothing holds it any more, so a
// reopen would leak the handle for the process lifetime.
evicted bool
// sweepOwned marks a registry entry that the idle sweep
// created because no writer was cached for the webhook. The
// sweep removes such an entry again when it is done, so a
// sweep can never leave — or resurrect — a registry entry
// for a webhook that has been deleted. A delivery that adopts
// the writer clears the flag, handing the entry to the
// registry proper.
//
// Unlike every other field here it is guarded by
// databaseTarget.mu, not by this writer's mu: it describes the
// registry entry rather than the file.
sweepOwned bool
}
// newArchiveWriter builds an archiveWriter for a file path with
// the default reopen debounce.
func newArchiveWriter(
path string, log *slog.Logger,
) *archiveWriter {
return &archiveWriter{
path: path,
log: log,
debounce: archiveReopenDebounce,
}
}
// write appends the event as a row, then applies the debounced
// close/reopen. It recreates the archive file if it was moved
// or removed since the last open. A positive expiry prunes rows
// older than it on each (re)open.
func (w *archiveWriter) write(
row archivedEvent, expiry time.Duration,
) error {
w.mu.Lock()
defer w.mu.Unlock()
if w.evicted {
return fmt.Errorf(
"%w: %s", errArchiveWriterEvicted, w.path,
)
}
if w.db == nil || !fileExists(w.path) {
err := w.reopen(expiry)
if err != nil {
return err
}
}
row.ArchivedAt = time.Now()
err := w.db.Create(&row).Error
if err != nil {
return fmt.Errorf(
"archiving event to %s: %w", w.path, err,
)
}
if time.Since(w.lastReopen) >= w.debounce {
return w.reopen(expiry)
}
return nil
}
// open opens (creating if missing) the archive file, migrates
// its schema, records the reopen time, and prunes expired rows
// when expiry is positive.
func (w *archiveWriter) open(expiry time.Duration) error {
return w.openMode(archiveModeCreate, expiry)
}
// openMode opens the archive file with the given SQLite URI
// mode, migrates its schema, records the reopen time, and
// prunes expired rows when expiry is positive. The write path
// passes archiveModeCreate so a missing file is recreated; the
// idle sweep passes archiveModeExisting so a missing file is an
// error rather than a newly conjured empty archive.
func (w *archiveWriter) openMode(
mode string, expiry time.Duration,
) error {
dbURL := fmt.Sprintf("file:%s?mode=%s", w.path, mode)
sqlDB, err := sql.Open("sqlite", dbURL)
if err != nil {
return fmt.Errorf(
"opening archive database %s: %w", w.path, err,
)
}
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{
// Never leave this at GORM's default. See
// internal/gormlog.
Logger: gormlog.New(w.log),
},
)
if err != nil {
_ = sqlDB.Close()
return fmt.Errorf(
"connecting to archive database %s: %w",
w.path, err,
)
}
err = gdb.AutoMigrate(&archivedEvent{})
if err != nil {
_ = sqlDB.Close()
return fmt.Errorf(
"migrating archive database %s: %w", w.path, err,
)
}
w.db = gdb
w.lastReopen = time.Now()
w.reopens++
if expiry > 0 {
w.prune(expiry)
}
return nil
}
// reopen closes any open handle and opens the file afresh. The
// fresh open recreates the file if it was moved away.
func (w *archiveWriter) reopen(expiry time.Duration) error {
w.close()
return w.open(expiry)
}
// close closes the underlying handle, if any.
func (w *archiveWriter) close() {
if w.db == nil {
return
}
sqlDB, err := w.db.DB()
if err == nil {
_ = sqlDB.Close()
}
w.db = nil
}
// sweepExpired prunes an archive that may have gone idle, with
// no write to trigger the usual on-reopen prune. It takes the
// writer's own mutex for the whole operation, so a sweep is
// ordered against concurrent writes rather than reaching around
// them to the file.
//
// It never creates the archive file: a missing file is skipped,
// and the reopen uses archiveModeExisting so SQLite itself
// refuses to create one if the file disappears between the
// check and the open.
//
// The archive is left CLOSED afterwards. An idle archive holding
// no handle is what keeps the operator's move-the-file-away
// workflow working; the next write reopens (and recreates) the
// file as it always has.
func (w *archiveWriter) sweepExpired(expiry time.Duration) error {
w.mu.Lock()
defer w.mu.Unlock()
if w.evicted {
return fmt.Errorf(
"%w: %s", errArchiveWriterEvicted, w.path,
)
}
if !fileExists(w.path) {
return nil
}
// Drop any live handle first so the prune runs against a
// freshly opened file, matching the write path's semantics.
w.close()
err := w.openMode(archiveModeExisting, expiry)
if err != nil {
return err
}
w.close()
return nil
}
// evict closes the writer's handle and marks it unusable. It is
// called when the writer leaves the registry, either because the
// webhook was deleted or because its last database target was
// removed. The archive FILE is deliberately left on disk: it is
// long-term storage an operator may still want.
func (w *archiveWriter) evict() {
w.mu.Lock()
defer w.mu.Unlock()
w.evicted = true
w.close()
}
// prune deletes archived rows older than expiry, measured from
// each row's archived time. It runs on every (re)open, so a
// steadily written archive is swept by its own write traffic. An
// archive that goes idle receives no further reopens, which is
// why ArchiveSweeper exists to drive sweepExpired on a timer.
// Failures are logged, not fatal: a prune error must not stop
// archiving.
func (w *archiveWriter) prune(expiry time.Duration) {
cutoff := time.Now().Add(-expiry)
res := w.db.Where("archived_at < ?", cutoff).
Delete(&archivedEvent{})
if res.Error != nil {
w.log.Error(
"failed to prune expired archive rows",
"path", w.path,
"error", res.Error,
)
return
}
if res.RowsAffected > 0 {
w.log.Info(
"pruned expired archive rows",
"path", w.path,
"rows_deleted", res.RowsAffected,
)
}
}
// fileExists reports whether a path currently exists.
func fileExists(path string) bool {
_, err := os.Stat(path)
return err == nil
}

155
internal/delivery/target_database_archive_gormlog_test.go Normal file
View File

@@ -0,0 +1,155 @@
package delivery_test
import (
"bytes"
"log"
"log/slog"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
gormlogger "gorm.io/gorm/logger"
"sneak.berlin/go/webhooker/internal/delivery"
"sneak.berlin/go/webhooker/internal/middleware"
)
// archiveGORMTailMarker sits at the far end of the value this file
// drives into an archive lookup. Its presence in a log line means the
// whole value reached the log, so nothing truncated it.
const archiveGORMTailMarker = "ENDOFCLIENTVALUE"
// archiveGORMFillBytes is how much text the lookup carries. It is far
// past every budget in play.
const archiveGORMFillBytes = 8 << 10
// gormDefaultBuf collects what GORM's package-level default logger
// writes, if anything reaches it.
type gormDefaultBuf struct {
mu sync.Mutex
b bytes.Buffer
}
func (g *gormDefaultBuf) Write(p []byte) (int, error) {
g.mu.Lock()
defer g.mu.Unlock()
return g.b.Write(p)
}
func (g *gormDefaultBuf) String() string {
g.mu.Lock()
defer g.mu.Unlock()
return g.b.String()
}
// captureArchiveGORMDefault replaces GORM's package-level default
// logger with one configured exactly as GORM configures its own,
// writing to a buffer.
//
// This duplicates the detector in internal/handlers rather than
// sharing it: a test helper cannot cross a package's test boundary
// without exporting production code to carry it, and a logging
// detector is not worth a production symbol. What it detects is the
// third gorm.Open in this service, at
// internal/delivery/target_database_archive.go — the archive writer,
// whose type is unexported, so nothing outside this package can drive
// it.
func captureArchiveGORMDefault(t *testing.T) *gormDefaultBuf {
t.Helper()
buf := &gormDefaultBuf{}
orig := gormlogger.Default
gormlogger.Default = gormlogger.New(
log.New(buf, "", log.LstdFlags),
gormlogger.Config{
SlowThreshold: 200 * time.Millisecond,
LogLevel: gormlogger.Warn,
IgnoreRecordNotFoundError: false,
Colorful: false,
},
)
t.Cleanup(func() { gormlogger.Default = orig })
return buf
}
// TestArchiveWriter_NeverUsesGORMsDefaultLogger pins the archive
// writer's gorm.Open to the adapter.
//
// Restore a bare &gorm.Config{} at
// internal/delivery/target_database_archive.go and this fails: the
// default logger prints the fully interpolated SELECT on every
// ErrRecordNotFound, so the client-chosen event id below arrives whole
// and unbounded on stdout, answering to no level the operator set.
//
// Not parallel: gormlogger.Default is process-global. Go runs every
// non-parallel top-level test to completion before it resumes the
// parallel ones.
//
//nolint:paralleltest // Deliberately sequential; see above.
func TestArchiveWriter_NeverUsesGORMsDefaultLogger(t *testing.T) {
var captured bytes.Buffer
gormDefault := captureArchiveGORMDefault(t)
w := delivery.NewExportArchiveWriter(
filepath.Join(t.TempDir(), "archive.db"),
slog.New(slog.NewTextHandler(
&captured, &slog.HandlerOptions{Level: slog.LevelDebug},
)),
0,
)
require.NoError(t, w.Open(0))
t.Cleanup(w.Evict)
// A lookup that misses, carrying a value the size of an inbound
// event id. Under the default logger this is the line that gets
// interpolated and printed.
value := strings.Repeat("\x01", archiveGORMFillBytes) +
archiveGORMTailMarker
var row delivery.ExportArchivedEvent
err := w.DB().Where("event_id = ?", value).First(&row).Error
require.ErrorIs(t, err, gorm.ErrRecordNotFound)
got := gormDefault.String()
assert.Empty(
t, got,
"GORM's default logger wrote %d bytes, so the archive "+
"writer's gorm.Open is back on a bare &gorm.Config{}; "+
"the first of them: %s",
len(got), got[:min(len(got), 300)],
)
// The adapter drops a miss, so this should be silent too — and
// whatever it does write stays inside the stated ceiling.
out := captured.String()
assert.NotContains(
t, out, archiveGORMTailMarker,
"the far end of the client-chosen value reached the log",
)
for line := range strings.SplitSeq(strings.TrimRight(out, "\n"), "\n") {
if line == "" {
continue
}
assert.LessOrEqual(
t, len(line), middleware.MaxAccessLogLineBytes,
"log line exceeded its bound: %s",
line[:min(len(line), 300)],
)
}
}

363
internal/delivery/target_database_evict_test.go Normal file
View File

@@ -0,0 +1,363 @@
package delivery_test
import (
"errors"
"fmt"
"net/http"
"os"
"path/filepath"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
// evictTestEngine builds an engine backed by a temporary data
// directory and returns it along with that directory.
func evictTestEngine(t *testing.T) (*delivery.Engine, string) {
t.Helper()
dataDir := t.TempDir()
eng := delivery.NewTestEngineWithDB(
nil,
database.NewTestWebhookDBManager(dataDir),
archiveTestLogger(),
&http.Client{Timeout: 5 * time.Second},
1,
)
return eng, dataDir
}
// TestEvictWebhook_ClosesAndRemovesWriter proves that evicting
// a webhook drops its archive writer from the registry and
// closes the open archive handle, rather than leaving both
// alive for the process lifetime.
func TestEvictWebhook_ClosesAndRemovesWriter(t *testing.T) {
t.Parallel()
eng, dataDir := evictTestEngine(t)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":true}`)
d := seedDatabaseTargetDelivery(t, webhookDB, event, "")
eng.ExportDeliverDatabase(webhookDB, d)
webhookID := event.WebhookID
require.True(
t, eng.ExportHasArchiveWriter(webhookID),
"a delivery should have cached an archive writer",
)
require.True(
t, eng.ExportArchiveHandleOpen(webhookID),
"the writer should hold an open handle after a write",
)
eng.EvictWebhook(webhookID)
assert.False(
t, eng.ExportHasArchiveWriter(webhookID),
"eviction should remove the registry entry",
)
assert.False(
t, eng.ExportArchiveHandleOpen(webhookID),
"eviction should close the archive handle",
)
archivePath := filepath.Join(
dataDir, fmt.Sprintf("archive-%s.db", webhookID),
)
assert.FileExists(
t, archivePath,
"eviction must not delete the archive file",
)
}
// TestEvictWebhook_UnknownWebhookIsNoOp proves eviction is safe
// for the common case of a webhook that never had a database
// target, and that repeating it does not panic.
func TestEvictWebhook_UnknownWebhookIsNoOp(t *testing.T) {
t.Parallel()
eng, _ := evictTestEngine(t)
assert.NotPanics(t, func() {
eng.EvictWebhook("no-such-webhook")
eng.EvictWebhook("no-such-webhook")
})
assert.False(
t, eng.ExportHasArchiveWriter("no-such-webhook"),
"eviction must not create a writer",
)
}
// evictTestRow builds an archive row for the eviction tests.
func evictTestRow(eventID string) delivery.ExportArchivedEvent {
return delivery.ExportArchivedEvent{
EventID: eventID,
WebhookID: "wh-evict",
Method: http.MethodPost,
Body: `{"seeded":true}`,
}
}
// TestEvictedWriter_WriteDoesNotReopenFile is the direct test of
// the evicted guard on the write path. A writer that has left
// the registry is held by nobody, so a handle it opened could
// never be closed again: it must refuse the write outright
// rather than recreate the archive behind the registry's back.
//
// The archive file is removed before the eviction, so an
// unguarded write is unmistakable — it recreates the file.
func TestEvictedWriter_WriteDoesNotReopenFile(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-evicted.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.Write(evictTestRow("ev-1"), 0))
require.FileExists(t, path)
// The operator moves the archive away for offline retention,
// which the write path would ordinarily undo on the next
// write by recreating the file.
require.NoError(t, os.Remove(path))
w.Evict()
err := w.Write(evictTestRow("ev-2"), 0)
require.ErrorIs(
t, err, delivery.ErrExportArchiveWriterEvicted,
"an evicted writer must refuse writes",
)
assert.NoFileExists(
t, path,
"an evicted writer must not reopen (or recreate) the "+
"archive file",
)
assert.False(
t, w.HandleOpen(),
"an evicted writer must hold no handle",
)
}
// TestEvictedWriter_SweepDoesNotReopenFile is the same test for
// the sweep path: an idle sweep that reaches a writer already
// evicted underneath it must return the sentinel rather than
// reopen a file nothing owns.
func TestEvictedWriter_SweepDoesNotReopenFile(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-evicted.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.Write(evictTestRow("ev-1"), 0))
require.FileExists(t, path)
w.Evict()
err := w.SweepExpired(time.Hour)
require.ErrorIs(
t, err, delivery.ErrExportArchiveWriterEvicted,
"an evicted writer must refuse an idle sweep",
)
assert.False(
t, w.HandleOpen(),
"a refused sweep must not leave a handle open",
)
}
// racingWrites drives a pack of goroutines writing to one
// archive writer until each is refused, so an eviction on the
// test goroutine has to take the writer's mutex away from writes
// that are already contending for it.
type racingWrites struct {
wg sync.WaitGroup
mu sync.Mutex
sawEvicted bool
otherErr error
started chan struct{}
}
// racingWriteGoroutines is how many goroutines contend for the
// writer's mutex while the eviction lands.
const racingWriteGoroutines = 4
// startRacingWrites launches the writing goroutines. Each writes
// in a loop and stops at its first error, recording whether that
// error was the eviction sentinel. The deadline is a backstop
// against a hang, not a timing assumption: the first write after
// the eviction is refused.
func startRacingWrites(
w *delivery.ExportArchiveWriter,
) *racingWrites {
r := &racingWrites{
started: make(chan struct{}, racingWriteGoroutines),
}
deadline := time.Now().Add(10 * time.Second)
r.wg.Add(racingWriteGoroutines)
for i := range racingWriteGoroutines {
go func() {
defer r.wg.Done()
first := true
for time.Now().Before(deadline) {
err := w.Write(
evictTestRow(fmt.Sprintf("ev-%d", i)), 0,
)
if first {
r.started <- struct{}{}
first = false
}
if err == nil {
continue
}
r.record(err)
return
}
}()
}
return r
}
// record classifies the error that stopped one goroutine.
func (r *racingWrites) record(err error) {
r.mu.Lock()
defer r.mu.Unlock()
if errors.Is(err, delivery.ErrExportArchiveWriterEvicted) {
r.sawEvicted = true
return
}
r.otherErr = err
}
// awaitFirstWrite blocks until at least one write has run, so
// the eviction that follows is a genuine race.
func (r *racingWrites) awaitFirstWrite() {
<-r.started
}
// wait joins the goroutines and reports whether any write was
// refused with the eviction sentinel, plus any unexpected error.
func (r *racingWrites) wait() (bool, error) {
r.wg.Wait()
r.mu.Lock()
defer r.mu.Unlock()
return r.sawEvicted, r.otherErr
}
// TestEvictWebhook_RacingWriteDoesNotReopenHandle exercises the
// interleaving the evicted flag exists for: writes already
// contending for the writer's mutex when the eviction takes it.
// The write that wins the mutex after the eviction must abandon
// its work rather than reopen the archive, leaving the writer
// permanently handle-free. Run under -race.
func TestEvictWebhook_RacingWriteDoesNotReopenHandle(
t *testing.T,
) {
t.Parallel()
eng, _ := evictTestEngine(t)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":true}`)
d := seedDatabaseTargetDelivery(t, webhookDB, event, "")
// Prime the registry so the test can hold the very writer the
// eviction is about to detach.
eng.ExportDeliverDatabase(webhookDB, d)
w := eng.ExportArchiveWriterFor(event.WebhookID)
require.NotNil(t, w)
require.True(t, w.HandleOpen())
race := startRacingWrites(w)
// Evict only once writes are genuinely in flight, so the
// eviction has to contend for the writer's mutex.
race.awaitFirstWrite()
eng.EvictWebhook(event.WebhookID)
sawEvicted, otherErr := race.wait()
require.NoError(t, otherErr)
assert.True(
t, sawEvicted,
"a write after eviction must be refused",
)
assert.False(
t, w.HandleOpen(),
"no write may reopen the archive once the writer has "+
"been evicted",
)
assert.False(
t, eng.ExportHasArchiveWriter(event.WebhookID),
"the registry entry must stay gone",
)
}
// TestEvictWebhook_LaterDeliveryRecreatesWriter proves eviction
// does not break archiving for a webhook that is still alive: a
// subsequent delivery gets a brand new writer from the registry.
// It says nothing about the evicted writer itself — that is what
// TestEvictedWriter_WriteDoesNotReopenFile covers.
func TestEvictWebhook_LaterDeliveryRecreatesWriter(t *testing.T) {
t.Parallel()
eng, _ := evictTestEngine(t)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":true}`)
d := seedDatabaseTargetDelivery(t, webhookDB, event, "")
eng.ExportDeliverDatabase(webhookDB, d)
require.True(
t, eng.ExportHasArchiveWriter(event.WebhookID),
)
eng.EvictWebhook(event.WebhookID)
// A fresh delivery for the same webhook gets a brand new
// writer from the registry, so archiving keeps working.
second := seedDatabaseTargetDelivery(
t, webhookDB, event, "",
)
eng.ExportDeliverDatabase(webhookDB, second)
assert.True(
t, eng.ExportHasArchiveWriter(event.WebhookID),
"a later delivery should recreate the writer",
)
}

402
internal/delivery/target_database_test.go Normal file
View File

@@ -0,0 +1,402 @@
package delivery_test
import (
"database/sql"
"fmt"
"log/slog"
"net/http"
"os"
"path/filepath"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
_ "modernc.org/sqlite" // Pure Go SQLite driver.
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
func archiveTestLogger() *slog.Logger {
return slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
))
}
// openArchiveDBForRead opens an archive file read-only so a
// test can inspect the rows the writer persisted.
func openArchiveDBForRead(
t *testing.T, path string,
) *gorm.DB {
t.Helper()
sqlDB, err := sql.Open(
"sqlite",
fmt.Sprintf("file:%s?mode=ro", path),
)
require.NoError(t, err)
t.Cleanup(func() { _ = sqlDB.Close() })
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
)
require.NoError(t, err)
return gdb
}
// archiveFileSuffixes returns the archive file itself and the
// SQLite sidecars that accompany an open database. A test that
// asserts no archive was created has to check all of them.
func archiveFileSuffixes() []string {
return []string{"", "-wal", "-shm"}
}
// removeArchiveFiles simulates an operator moving the archive
// away by deleting the SQLite file and its sidecar files.
func removeArchiveFiles(t *testing.T, path string) {
t.Helper()
for _, suffix := range []string{
"", "-wal", "-shm", "-journal",
} {
err := os.Remove(path + suffix)
if err != nil && !os.IsNotExist(err) {
t.Fatalf("removing %s%s: %v", path, suffix, err)
}
}
}
// TestDeliverDatabase_ArchivesEvent verifies that delivering to
// a database target marks the delivery delivered and archives
// the full event into a separate per-webhook archive file.
func TestDeliverDatabase_ArchivesEvent(t *testing.T) {
t.Parallel()
dataDir := t.TempDir()
dbMgr := database.NewTestWebhookDBManager(dataDir)
e := delivery.NewTestEngineWithDB(
nil, dbMgr,
archiveTestLogger(),
&http.Client{Timeout: 5 * time.Second},
1,
)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":true}`)
d := seedDatabaseTargetDelivery(t, webhookDB, event, "")
e.ExportDeliverDatabase(webhookDB, d)
var updated database.Delivery
require.NoError(t, webhookDB.First(
&updated, "id = ?", d.ID,
).Error)
assert.Equal(t,
database.DeliveryStatusDelivered, updated.Status,
"database target should mark the delivery delivered",
)
archivePath := filepath.Join(
dataDir,
fmt.Sprintf("archive-%s.db", event.WebhookID),
)
assert.FileExists(t, archivePath)
rdb := openArchiveDBForRead(t, archivePath)
var rows []delivery.ExportArchivedEvent
require.NoError(t, rdb.Find(&rows).Error)
require.Len(t, rows, 1)
assert.Equal(t, event.ID, rows[0].EventID)
assert.Equal(t, event.WebhookID, rows[0].WebhookID)
assert.Equal(t, event.Method, rows[0].Method)
assert.JSONEq(t, `{"archived":true}`, rows[0].Body)
}
func TestArchiveWriter_WritesRow(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-wh.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
row := delivery.ExportArchivedEvent{
EventID: "ev-1",
WebhookID: "wh-1",
EntrypointID: "ep-1",
Method: "POST",
Headers: `{"X":"Y"}`,
Body: `{"hello":"world"}`,
ContentType: "application/json",
}
require.NoError(t, w.Write(row, 0))
assert.FileExists(t, path)
var got []delivery.ExportArchivedEvent
require.NoError(t, w.DB().Find(&got).Error)
require.Len(t, got, 1)
assert.Equal(t, "ev-1", got[0].EventID)
assert.Equal(t, "wh-1", got[0].WebhookID)
assert.Equal(t, "ep-1", got[0].EntrypointID)
assert.Equal(t, row.Method, got[0].Method)
assert.Equal(t, row.ContentType, got[0].ContentType)
assert.JSONEq(t, `{"hello":"world"}`, got[0].Body)
assert.False(t, got[0].ArchivedAt.IsZero())
}
func TestArchiveWriter_RecreatesAfterRemoval(
t *testing.T,
) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-wh.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "a"}, 0,
))
assert.FileExists(t, path)
// The operator moves the archive away while the handle is
// still open.
removeArchiveFiles(t, path)
require.NoFileExists(t, path)
// The next write recreates the file with a fresh schema and
// only the new row.
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "b"}, 0,
))
assert.FileExists(t, path)
var got []delivery.ExportArchivedEvent
require.NoError(t, w.DB().Find(&got).Error)
require.Len(t, got, 1)
assert.Equal(t, "b", got[0].EventID)
}
func TestArchiveWriter_ReopenDebounce(t *testing.T) {
t.Parallel()
// A generous debounce keeps the two rapid writes inside
// the window even on a heavily loaded test machine.
path := filepath.Join(t.TempDir(), "archive-wh.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 2*time.Second,
)
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "a"}, 0,
))
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "b"}, 0,
))
// Two writes inside the debounce window trigger only the
// initial open — no extra close/reopen.
assert.Equal(t, 1, w.Reopens())
time.Sleep(2100 * time.Millisecond)
require.NoError(t, w.Write(
delivery.ExportArchivedEvent{EventID: "c"}, 0,
))
// A write after the window elapses closes and reopens once.
assert.Equal(t, 2, w.Reopens())
}
func TestArchiveWriter_ExpiryPrune(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-wh.db")
w := delivery.NewExportArchiveWriter(
path, archiveTestLogger(), 0,
)
require.NoError(t, w.Open(0))
old := delivery.ExportArchivedEvent{
EventID: "old",
ArchivedAt: time.Now().Add(-2 * time.Hour),
}
fresh := delivery.ExportArchivedEvent{
EventID: "fresh",
ArchivedAt: time.Now(),
}
require.NoError(t, w.DB().Create(&old).Error)
require.NoError(t, w.DB().Create(&fresh).Error)
// Reopening with a one-hour expiry prunes the old row.
require.NoError(t, w.Reopen(time.Hour))
var got []delivery.ExportArchivedEvent
require.NoError(t, w.DB().Find(&got).Error)
require.Len(t, got, 1)
assert.Equal(t, "fresh", got[0].EventID)
}
func TestParseArchiveExpiry(t *testing.T) {
t.Parallel()
cases := []struct {
name string
in string
want time.Duration
wantErr bool
}{
{"empty config", "", 0, false},
{"explicit never", `{"expiry":"never"}`, 0, false},
{"empty expiry", `{"expiry":""}`, 0, false},
{"duration", `{"expiry":"1h"}`, time.Hour, false},
{"unparseable", `{"expiry":"nonsense"}`, 0, true},
{"zero duration", `{"expiry":"0s"}`, 0, true},
{"negative duration", `{"expiry":"-5h"}`, 0, true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
got, err := delivery.ExportParseArchiveExpiry(tc.in)
if tc.wantErr {
require.Error(t, err)
return
}
require.NoError(t, err)
assert.Equal(t, tc.want, got)
})
}
}
// seedDatabaseTargetDelivery seeds a pending delivery for a
// database target with the given config JSON and returns the
// in-memory delivery the target handler is invoked with.
func seedDatabaseTargetDelivery(
t *testing.T,
webhookDB *gorm.DB,
event database.Event,
config string,
) *database.Delivery {
t.Helper()
dlv := seedDelivery(
t, webhookDB, event.ID, uuid.New().String(),
database.DeliveryStatusPending,
)
d := &database.Delivery{
EventID: event.ID,
TargetID: dlv.TargetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: database.Target{
Name: "test-db",
Type: database.TargetTypeDatabase,
Config: config,
},
}
d.ID = dlv.ID
return d
}
// TestDeliverDatabase_ArchiveFailureFailsDelivery verifies that
// an archive error (here: an unparseable expiry in the target
// config) fails the delivery loudly: the attempt is recorded as
// failed with the error and the delivery is marked failed, not
// delivered.
func TestDeliverDatabase_ArchiveFailureFailsDelivery(
t *testing.T,
) {
t.Parallel()
dataDir := t.TempDir()
e := delivery.NewTestEngineWithDB(
nil, database.NewTestWebhookDBManager(dataDir),
archiveTestLogger(),
&http.Client{Timeout: 5 * time.Second},
1,
)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":false}`)
d := seedDatabaseTargetDelivery(
t, webhookDB, event, `{"expiry":"nonsense"}`,
)
e.ExportDeliverDatabase(webhookDB, d)
var updated database.Delivery
require.NoError(t, webhookDB.First(
&updated, "id = ?", d.ID,
).Error)
assert.Equal(t,
database.DeliveryStatusFailed, updated.Status,
"archive failure must mark the delivery failed",
)
var results []database.DeliveryResult
require.NoError(t, webhookDB.Where(
"delivery_id = ?", d.ID,
).Find(&results).Error)
require.Len(t, results, 1)
assert.False(t,
results[0].Success,
"the attempt must be recorded as failed",
)
assert.Contains(t,
results[0].Error, "nonsense",
"the archive error must be recorded on the attempt",
)
assert.NoFileExists(t,
filepath.Join(
dataDir,
fmt.Sprintf("archive-%s.db", event.WebhookID),
),
"no archive file should exist for a failed config",
)
}
func TestValidateArchiveExpiry(t *testing.T) {
t.Parallel()
valid := []string{"", "never", "1h", "720h", "30m"}
for _, in := range valid {
require.NoError(t,
delivery.ValidateArchiveExpiry(in),
"expiry %q should be accepted", in,
)
}
invalid := []string{"nonsense", "7d", "-5h", "0s", "0"}
for _, in := range invalid {
require.Error(t,
delivery.ValidateArchiveExpiry(in),
"expiry %q should be rejected", in,
)
}
}

511
internal/delivery/target_http.go Normal file
View File

@@ -0,0 +1,511 @@
package delivery
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"sync"
"time"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// Sentinel errors returned by the config parsers.
var (
errEmptyTargetConfig = errors.New(
"empty target config",
)
errMissingTargetURL = errors.New(
"target URL is required",
)
)
// HTTPTargetConfig holds configuration for http target
// types.
type HTTPTargetConfig struct {
URL string `json:"url"`
Headers map[string]string `json:"headers,omitempty"`
Timeout int `json:"timeout,omitempty"`
}
// httpCore holds the retry, backoff, and circuit-breaker
// machinery shared by the HTTP and Slack targets. Each of
// those targets owns its own httpCore instance (and thus its
// own circuit breakers); the per-attempt request differs
// between them and is supplied as a closure.
type httpCore struct {
eng *Engine
// circuitBreakers stores a *CircuitBreaker per target ID.
circuitBreakers sync.Map
}
// deliver runs one delivery attempt through the retry core.
// A maxRetries of 0 is fire-and-forget: a single attempt is
// recorded and no circuit breaker is consulted. A positive
// maxRetries gates the attempt on the circuit breaker and
// schedules a backed-off retry on failure.
func (c *httpCore) deliver(
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
maxRetries int,
attempt func() attemptResult,
) {
if maxRetries == 0 {
c.fireAndForget(webhookDB, d, attempt())
return
}
c.withRetry(
webhookDB, d, task, sched, maxRetries, attempt,
)
}
func (c *httpCore) fireAndForget(
webhookDB *gorm.DB,
d *database.Delivery,
res attemptResult,
) {
c.eng.recordResult(
webhookDB, d, 1, res.success,
res.statusCode, res.respBody, res.errMsg,
res.duration,
)
if res.success {
c.eng.updateDeliveryStatus(
webhookDB, d,
database.DeliveryStatusDelivered,
)
return
}
c.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusFailed,
)
}
func (c *httpCore) withRetry(
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
maxRetries int,
attempt func() attemptResult,
) {
cb := c.getCircuitBreaker(task.TargetID)
if c.circuitBreakerBlock(webhookDB, d, task, sched, cb) {
return
}
attemptNum := task.AttemptNum
res := attempt()
c.eng.recordResult(
webhookDB, d, attemptNum, res.success,
res.statusCode, res.respBody, res.errMsg,
res.duration,
)
if res.success {
cb.RecordSuccess()
c.eng.updateDeliveryStatus(
webhookDB, d,
database.DeliveryStatusDelivered,
)
return
}
cb.RecordFailure()
c.handleRetry(
webhookDB, d, task, sched, maxRetries, attemptNum,
)
}
func (c *httpCore) circuitBreakerBlock(
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
cb *CircuitBreaker,
) bool {
if cb.Allow() {
return false
}
remaining := cb.CooldownRemaining()
c.eng.log.Info(
"circuit breaker open, skipping delivery",
"target_id", task.TargetID,
"target_name", task.TargetName,
"delivery_id", d.ID,
"cooldown_remaining", remaining,
)
c.eng.updateDeliveryStatus(
webhookDB, d,
database.DeliveryStatusRetrying,
)
retryTask := *task
sched.ScheduleRetry(retryTask, remaining)
return true
}
func (c *httpCore) handleRetry(
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
maxRetries int,
attemptNum int,
) {
if attemptNum >= maxRetries {
c.eng.updateDeliveryStatus(
webhookDB, d,
database.DeliveryStatusFailed,
)
return
}
c.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusRetrying,
)
backoff := calcBackoff(attemptNum)
retryTask := *task
retryTask.AttemptNum = attemptNum + 1
sched.ScheduleRetry(retryTask, backoff)
}
func (c *httpCore) getCircuitBreaker(
targetID string,
) *CircuitBreaker {
if val, ok := c.circuitBreakers.Load(targetID); ok {
cb, _ := val.(*CircuitBreaker)
return cb
}
fresh := NewCircuitBreaker()
actual, _ := c.circuitBreakers.LoadOrStore(
targetID, fresh,
)
cb, _ := actual.(*CircuitBreaker)
return cb
}
// remainingBackoff returns how long remains of the backoff
// window for the last attempt of a recovered retrying
// delivery. It implements rescheduler.
func (c *httpCore) remainingBackoff(
webhookDB *gorm.DB,
deliveryID string,
attemptNum int,
) time.Duration {
var lastResult database.DeliveryResult
err := webhookDB.
Where("delivery_id = ?", deliveryID).
Order("created_at DESC").
First(&lastResult).Error
if err != nil {
return 0
}
backoff := calcBackoff(attemptNum)
elapsed := time.Since(lastResult.CreatedAt)
remaining := backoff - elapsed
return max(remaining, 0)
}
// backoffElapsed reports whether the backoff window for the
// last attempt of a retrying delivery has passed. It
// implements rescheduler.
func (c *httpCore) backoffElapsed(
webhookDB *gorm.DB,
deliveryID string,
attemptNum int,
) bool {
var lastResult database.DeliveryResult
err := webhookDB.
Where("delivery_id = ?", deliveryID).
Order("created_at DESC").
First(&lastResult).Error
if err != nil {
return true
}
backoff := calcBackoff(attemptNum)
return time.Since(lastResult.CreatedAt) >= backoff
}
func calcBackoff(attemptNum int) time.Duration {
shift := max(attemptNum-1, 0)
shift = min(shift, maxBackoffShift)
return time.Duration(1<<uint(shift)) * time.Second
}
// httpTarget delivers events to http targets. It forwards the
// event body and (filtered) request headers to the configured
// URL and owns retry, backoff, and circuit breaking through
// the shared httpCore.
type httpTarget struct {
*httpCore
client *http.Client
}
// Deliver implements Target.
func (t *httpTarget) Deliver(
ctx context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
) {
cfg, err := parseHTTPConfig(d.Target.Config)
if err != nil {
t.eng.log.Error(
"invalid HTTP target config",
"target_id", d.TargetID,
"error", err,
)
t.eng.recordResult(
webhookDB, d, task.AttemptNum,
false, 0, "", err.Error(), 0,
)
t.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusFailed,
)
return
}
attempt := func() attemptResult {
return t.attempt(ctx, cfg, &d.Event)
}
t.deliver(
webhookDB, d, task, sched,
d.Target.MaxRetries, attempt,
)
}
// attempt performs a single HTTP delivery attempt and derives
// the success flag and error message the same way the engine
// did: a non-2xx response is a failure but carries no error
// string; only a transport-level error does.
func (t *httpTarget) attempt(
ctx context.Context,
cfg *HTTPTargetConfig,
event *database.Event,
) attemptResult {
statusCode, respBody, duration, reqErr :=
t.doHTTPRequest(ctx, cfg, event)
success := reqErr == nil &&
statusCode >= httpSuccessMin &&
statusCode < httpSuccessMax
errMsg := ""
if reqErr != nil {
errMsg = reqErr.Error()
}
return attemptResult{
statusCode: statusCode,
respBody: respBody,
duration: duration,
success: success,
errMsg: errMsg,
}
}
func (t *httpTarget) doHTTPRequest(
ctx context.Context,
cfg *HTTPTargetConfig,
event *database.Event,
) (int, string, int64, error) {
start := time.Now()
req, reqErr := http.NewRequestWithContext(
ctx,
http.MethodPost,
cfg.URL,
bytes.NewReader([]byte(event.Body)),
)
if reqErr != nil {
return 0, "", 0, fmt.Errorf(
"creating request: %w",
maskURLError(reqErr),
)
}
applyRequestHeaders(req, event, cfg)
client := t.clientForConfig(cfg)
resp, doErr := executeHTTPRequest(client, req)
dur := time.Since(start).Milliseconds()
if doErr != nil {
return 0, "", dur, fmt.Errorf(
"sending request: %w", doErr,
)
}
defer func() { _ = resp.Body.Close() }()
body, readErr := io.ReadAll(
io.LimitReader(resp.Body, maxBodyLog),
)
if readErr != nil {
return resp.StatusCode, "", dur,
fmt.Errorf(
"reading response body: %w", readErr,
)
}
return resp.StatusCode, string(body), dur, nil
}
func (t *httpTarget) clientForConfig(
cfg *HTTPTargetConfig,
) *http.Client {
if cfg.Timeout > 0 {
// Reuse the shared client's SSRF-safe transport so
// a per-target timeout does not drop the
// request-time private-IP guard. Only the timeout
// is overridden.
return &http.Client{
Timeout: time.Duration(
cfg.Timeout,
) * time.Second,
Transport: t.client.Transport,
}
}
return t.client
}
func parseHTTPConfig(
configJSON string,
) (*HTTPTargetConfig, error) {
if configJSON == "" {
return nil, errEmptyTargetConfig
}
var cfg HTTPTargetConfig
err := json.Unmarshal(
[]byte(configJSON), &cfg,
)
if err != nil {
return nil, fmt.Errorf(
"parsing config JSON: %w", err,
)
}
if cfg.URL == "" {
return nil, errMissingTargetURL
}
return &cfg, nil
}
// isForwardableHeader returns true if the header should
// be forwarded to targets.
func isForwardableHeader(name string) bool {
switch http.CanonicalHeaderKey(name) {
case "Host", "Connection", "Keep-Alive",
"Transfer-Encoding", "Te", "Trailer",
"Upgrade", "Proxy-Authorization",
"Proxy-Connection", "Content-Length":
return false
default:
return true
}
}
func applyRequestHeaders(
req *http.Request,
event *database.Event,
cfg *HTTPTargetConfig,
) {
if event.ContentType != "" {
req.Header.Set(
"Content-Type", event.ContentType,
)
}
var originalHeaders map[string][]string
if event.Headers != "" {
jsonErr := json.Unmarshal(
[]byte(event.Headers),
&originalHeaders,
)
if jsonErr == nil {
for k, vals := range originalHeaders {
if isForwardableHeader(k) {
for _, v := range vals {
req.Header.Add(k, v)
}
}
}
}
}
for k, v := range cfg.Headers {
req.Header.Set(k, v)
}
req.Header.Set("User-Agent", "webhooker/1.0")
}
// executeHTTPRequest sends an HTTP request using the provided
// client. URLs are validated by the config parsers and the
// SSRF-safe transport before reaching here.
//
// Transport failures are masked here, at the single point
// where every target's request errors are born, because the
// caller stores them in DeliveryResult.Error: an unmasked
// *url.Error would write the target URL — the credential for
// a Slack incoming webhook — into the per-webhook database.
func executeHTTPRequest(
client *http.Client, req *http.Request,
) (*http.Response, error) {
resp, err := client.Do(req) //#nosec G704 -- validated URL, SSRF-safe transport
if err != nil {
return nil, maskURLError(err)
}
return resp, nil
}

58
internal/delivery/target_log.go Normal file
View File

@@ -0,0 +1,58 @@
package delivery
import (
"context"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// logTarget is a fire-and-forget target that logs the entire
// inbound webhook — the full request body and headers, plus
// the method, content type, and the webhook and entrypoint
// ids — then records a single successful attempt.
//
// This is the one log call in the service that deliberately writes
// unbounded client-chosen bytes, so it is the one exception to the
// per-field budgets in internal/logfield and to the ceiling stated on
// middleware.MaxAccessLogLineBytes. Capping here would defeat the
// target: emitting the payload IS the delivery. It costs nothing by
// default — an authenticated operator has to create a target of this
// type on a specific webhook before a single line is written — and the
// bytes it writes are bounded per event by maxWebhookBodySize (1 MB).
// An operator who adds one is choosing to spend log volume on the
// payloads that webhook receives.
type logTarget struct {
eng *Engine
}
// Deliver implements Target.
func (t *logTarget) Deliver(
_ context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
_ *Task,
_ Scheduler,
) {
t.eng.log.Info(
"webhook event delivered to log target",
"delivery_id", d.ID,
"event_id", d.EventID,
"target_id", d.TargetID,
"target_name", d.Target.Name,
"webhook_id", d.Event.WebhookID,
"entrypoint_id", d.Event.EntrypointID,
"method", d.Event.Method,
"content_type", d.Event.ContentType,
"headers", d.Event.Headers,
"body", d.Event.Body,
)
t.eng.recordResult(
webhookDB, d, 1, true, 0, "", "", 0,
)
t.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusDelivered,
)
}

299
internal/delivery/target_slack.go Normal file
View File

@@ -0,0 +1,299 @@
package delivery
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strings"
"time"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// errMissingWebhookURL is returned when a Slack target config
// omits its webhook URL.
var errMissingWebhookURL = errors.New(
"webhook_url is required",
)
// SlackTargetConfig holds configuration for slack target
// types.
type SlackTargetConfig struct {
WebhookURL string `json:"webhookUrl"`
}
// slackTarget delivers events to Slack incoming webhooks. It
// formats the event into a Slack message and posts it as
// JSON. It shares the retry core with the HTTP target: a
// MaxRetries of 0 stays single-attempt fire-and-forget
// (preserving existing Slack targets), while a positive
// MaxRetries adds backoff and circuit breaking.
type slackTarget struct {
*httpCore
client *http.Client
}
// Deliver implements Target.
func (t *slackTarget) Deliver(
ctx context.Context,
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
sched Scheduler,
) {
cfg, err := parseSlackConfig(d.Target.Config)
if err != nil {
t.eng.log.Error(
"invalid Slack target config",
"target_id", d.TargetID,
"error", err,
)
t.failConfig(webhookDB, d, err)
return
}
msg := FormatSlackMessage(&d.Event)
payload, err := json.Marshal(
map[string]string{"text": msg},
)
if err != nil {
t.eng.log.Error(
"failed to marshal Slack payload",
"target_id", d.TargetID,
"error", err,
)
t.failConfig(webhookDB, d, err)
return
}
attempt := func() attemptResult {
return t.attempt(ctx, cfg, payload)
}
t.deliver(
webhookDB, d, task, sched,
d.Target.MaxRetries, attempt,
)
}
// failConfig records a first-attempt failure for a delivery
// that could not be prepared (bad config or unmarshalable
// payload) and marks it failed.
func (t *slackTarget) failConfig(
webhookDB *gorm.DB,
d *database.Delivery,
err error,
) {
t.eng.recordResult(
webhookDB, d, 1,
false, 0, "", err.Error(), 0,
)
t.eng.updateDeliveryStatus(
webhookDB, d, database.DeliveryStatusFailed,
)
}
// attempt performs a single Slack POST and derives its
// outcome, preserving the engine's original semantics: a
// non-2xx response records an "HTTP <code>" error string and
// a transport error records a "sending request" error.
func (t *slackTarget) attempt(
ctx context.Context,
cfg *SlackTargetConfig,
payload []byte,
) attemptResult {
start := time.Now()
req, err := http.NewRequestWithContext(
ctx,
http.MethodPost,
cfg.WebhookURL,
bytes.NewReader(payload),
)
if err != nil {
return attemptResult{
success: false,
errMsg: maskURLError(err).Error(),
}
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("User-Agent", "webhooker/1.0")
resp, doErr := executeHTTPRequest(t.client, req)
durationMs := time.Since(start).Milliseconds()
if doErr != nil {
return attemptResult{
success: false,
duration: durationMs,
errMsg: fmt.Errorf(
"sending request: %w", doErr,
).Error(),
}
}
defer func() { _ = resp.Body.Close() }()
return t.readSlackResponse(resp, durationMs)
}
func (t *slackTarget) readSlackResponse(
resp *http.Response,
durationMs int64,
) attemptResult {
body, readErr := io.ReadAll(
io.LimitReader(resp.Body, maxBodyLog),
)
if readErr != nil {
t.eng.log.Error(
"failed to read Slack response body",
"error", readErr,
)
}
success := resp.StatusCode >= httpSuccessMin &&
resp.StatusCode < httpSuccessMax
errMsg := ""
if !success {
errMsg = fmt.Sprintf("HTTP %d", resp.StatusCode)
}
return attemptResult{
statusCode: resp.StatusCode,
respBody: string(body),
duration: durationMs,
success: success,
errMsg: errMsg,
}
}
func parseSlackConfig(
configJSON string,
) (*SlackTargetConfig, error) {
if configJSON == "" {
return nil, errEmptyTargetConfig
}
var cfg SlackTargetConfig
err := json.Unmarshal(
[]byte(configJSON), &cfg,
)
if err != nil {
return nil, fmt.Errorf(
"parsing config JSON: %w", err,
)
}
if cfg.WebhookURL == "" {
return nil, errMissingWebhookURL
}
return &cfg, nil
}
// FormatSlackMessage builds a Slack-compatible message
// string from a webhook event.
func FormatSlackMessage(
event *database.Event,
) string {
var b strings.Builder
b.WriteString("*Webhook Event Received*\n")
fmt.Fprintf(
&b, "*Method:* `%s`\n", event.Method,
)
fmt.Fprintf(
&b,
"*Content-Type:* `%s`\n",
event.ContentType,
)
fmt.Fprintf(
&b,
"*Timestamp:* `%s`\n",
event.CreatedAt.UTC().Format(time.RFC3339),
)
fmt.Fprintf(
&b,
"*Body Size:* %d bytes\n",
len(event.Body),
)
if event.Body == "" {
b.WriteString("\n_(empty body)_\n")
return b.String()
}
if formatted := formatJSONBody(event.Body); formatted != "" {
b.WriteString(formatted)
return b.String()
}
formatRawBody(&b, event.Body)
return b.String()
}
func formatJSONBody(body string) string {
var parsed json.RawMessage
if json.Unmarshal([]byte(body), &parsed) != nil {
return ""
}
var pretty bytes.Buffer
if json.Indent(&pretty, parsed, "", " ") != nil {
return ""
}
var b strings.Builder
b.WriteString("\n```\n")
prettyStr := pretty.String()
const maxPayloadDisplay = 3500
if len(prettyStr) > maxPayloadDisplay {
b.WriteString(prettyStr[:maxPayloadDisplay])
b.WriteString("\n... (truncated)")
} else {
b.WriteString(prettyStr)
}
b.WriteString("\n```\n")
return b.String()
}
func formatRawBody(b *strings.Builder, body string) {
b.WriteString("\n```\n")
const maxRawDisplay = 3500
if len(body) > maxRawDisplay {
b.WriteString(body[:maxRawDisplay])
b.WriteString("\n... (truncated)")
} else {
b.WriteString(body)
}
b.WriteString("\n```\n")
}

61
internal/delivery/url_mask.go Normal file
View File

@@ -0,0 +1,61 @@
package delivery
import (
"errors"
"net/url"
)
// urlPathElision stands in for a URL's elided path.
const urlPathElision = "/..."
// MaskURL renders a URL as scheme plus host with everything
// that can carry a secret removed. A delivery target URL is
// itself a credential — a Slack incoming webhook URL is a
// bearer token — so the path, query and userinfo are never
// reproduced, in a page, a log line or a stored error. A URL
// that does not parse into a scheme and host yields the
// neutral placeholder, never the raw string.
func MaskURL(raw string) string {
parsed, err := url.Parse(raw)
if err != nil || parsed.Scheme == "" ||
parsed.Host == "" {
return configUnavailable
}
masked := parsed.Scheme + "://" + parsed.Host
if parsed.Path != "" && parsed.Path != "/" {
masked += urlPathElision
}
return masked
}
// maskURLError strips the credential from an error raised
// against a request URL. The net/http and net/url packages
// embed the full request URL in every *url.Error they return,
// so an unmodified transport error persisted into
// DeliveryResult.Error writes the credential to disk.
//
// The masked error keeps the operation and the wrapped cause,
// so a DNS failure still reads differently from a refused
// connection, a TLS handshake failure or a timeout, and Is,
// As, Timeout and Temporary keep working on it. Only the
// path, query and userinfo of the URL are dropped. Errors
// that carry no URL are returned unchanged.
//
// Call it where the error is raised, before any wrapping: it
// replaces the *url.Error itself, so any context wrapped
// around it first would be discarded.
func maskURLError(err error) error {
var urlErr *url.Error
if !errors.As(err, &urlErr) {
return err
}
return &url.Error{
Op: urlErr.Op,
URL: MaskURL(urlErr.URL),
Err: urlErr.Err,
}
}

196
internal/delivery/url_mask_test.go Normal file
View File

@@ -0,0 +1,196 @@
package delivery_test
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
// The path of a Slack incoming webhook URL is the credential:
// whoever holds these segments can post to the channel
// forever. None of them may reach a stored delivery error,
// which lives on disk in the per-webhook database and is
// serialized by the JSON tag on DeliveryResult.Error.
const (
maskSecretPath = "/services/T00000000/B00000000/" +
"XXXXXXXXXXXXXXXXXXXXXXXX"
)
// assertNoCredential fails if the whole path or any single
// segment of it survived into the message, so a partial leak
// fails the test too.
func assertNoCredential(t *testing.T, msg string) {
t.Helper()
segments := []string{
maskSecretPath,
"services",
"T00000000",
"B00000000",
"XXXXXXXXXXXXXXXXXXXXXXXX",
}
for _, segment := range segments {
assert.NotContains(t, msg, segment)
}
}
// storedDeliveryError returns the error string persisted for a
// delivery, which is what an operator and any future API read.
func storedDeliveryError(
t *testing.T, db *gorm.DB, deliveryID string,
) string {
t.Helper()
var result database.DeliveryResult
require.NoError(t, db.Where(
"delivery_id = ?", deliveryID,
).First(&result).Error)
return result.Error
}
// deliverSlackTo runs a Slack delivery against webhookURL and
// returns the error string it persisted.
func deliverSlackTo(
t *testing.T, webhookURL string,
) string {
t.Helper()
db := testWebhookDB(t)
e := testEngine(t, 1)
targetID := uuid.New().String()
slackCfg, err := json.Marshal(
delivery.SlackTargetConfig{
WebhookURL: webhookURL,
},
)
require.NoError(t, err)
event := seedEvent(t, db, `{"test":true}`)
dlv := seedDelivery(
t, db, event.ID, targetID,
database.DeliveryStatusPending,
)
d := buildSlackDelivery(
dlv, event, targetID,
"test-slack-mask", string(slackCfg),
)
e.ExportDeliverSlack(context.TODO(), db, d)
assertDeliveryStatus(t, db, dlv.ID,
database.DeliveryStatusFailed,
)
return storedDeliveryError(t, db, dlv.ID)
}
// TestDeliverSlack_TransportErrorMasksWebhookURL is the
// load-bearing regression test: a transport failure must not
// persist the webhook URL's credential into the database, and
// must still say what went wrong and where.
func TestDeliverSlack_TransportErrorMasksWebhookURL(
t *testing.T,
) {
t.Parallel()
// A server closed before use gives a deterministic
// transport failure against a known host.
ts := httptest.NewServer(http.NewServeMux())
host := ts.URL
ts.Close()
errMsg := deliverSlackTo(t, host+maskSecretPath)
require.NotEmpty(t, errMsg)
assertNoCredential(t, errMsg)
// The diagnostic value survives: the operation, the host
// and the transport failure are all still reported, and
// only the path is elided.
assert.Contains(t, errMsg, "sending request")
assert.Contains(t, errMsg, "Post")
assert.Contains(t, errMsg, host+"/...")
assert.Contains(t, errMsg, "connection refused")
}
// TestDeliverSlack_UnparsableURLMasksWebhookURL covers the
// other error path out of a Slack attempt: url.Parse also
// embeds the whole URL in the error it returns.
func TestDeliverSlack_UnparsableURLMasksWebhookURL(
t *testing.T,
) {
t.Parallel()
errMsg := deliverSlackTo(
t,
"https://hooks.slack.com"+maskSecretPath+"\n",
)
require.NotEmpty(t, errMsg)
assertNoCredential(t, errMsg)
assert.Contains(t, errMsg, "invalid control character")
}
// TestDoHTTPRequest_TransportErrorMasksURL proves the HTTP
// target's transport errors are masked too; its destination
// URL can carry a token in a query string.
func TestDoHTTPRequest_TransportErrorMasksURL(t *testing.T) {
t.Parallel()
ts := httptest.NewServer(http.NewServeMux())
host := ts.URL
ts.Close()
e := testEngine(t, 1)
cfg, err := e.ExportParseHTTPConfig(
newHTTPTargetConfig(host + maskSecretPath),
)
require.NoError(t, err)
statusCode, _, _, reqErr := e.ExportDoHTTPRequest(
context.TODO(), cfg,
&database.Event{Body: `{"test":true}`},
)
require.Error(t, reqErr)
assert.Zero(t, statusCode)
assertNoCredential(t, reqErr.Error())
assert.Contains(t, reqErr.Error(), host+"/...")
assert.Contains(
t, reqErr.Error(), "connection refused",
)
}
// TestValidateTargetURL_UnparsableURLIsMasked proves the SSRF
// validator's error does not carry the submitted URL, which
// the handler both logs and shows.
func TestValidateTargetURL_UnparsableURLIsMasked(t *testing.T) {
t.Parallel()
err := delivery.ValidateTargetURL(
context.TODO(),
"https://hooks.slack.com"+maskSecretPath+"\n",
)
require.Error(t, err)
assertNoCredential(t, err.Error())
assert.Contains(t, err.Error(), "invalid URL")
}

17
internal/gormlog/export_test.go Normal file
View File

@@ -0,0 +1,17 @@
package gormlog
import (
"log/slog"
"time"
)
// ExportNewWithSlowThreshold builds a Logger whose slow-statement
// threshold is d rather than DefaultSlowThreshold, so a test can pin
// which arm of Trace it is exercising instead of racing the clock on a
// loaded machine. The threshold is set at construction, like every
// other field, so the type's concurrency guarantee still holds.
func ExportNewWithSlowThreshold(
log *slog.Logger, d time.Duration,
) *Logger {
return &Logger{log: log, slowThreshold: d}
}

168
internal/gormlog/gormlog.go Normal file
View File

@@ -0,0 +1,168 @@
// Package gormlog adapts GORM's logger onto the service's slog
// logger.
//
// GORM's own default logger is not usable here. It is built at package
// init with log.New(os.Stdout, ...) at LogLevel Warn with
// IgnoreRecordNotFoundError false, so it writes the fully interpolated
// SQL — parameters and all — for every statement that returns an
// error, including gorm.ErrRecordNotFound. Two of this service's
// lookups miss by design on unauthenticated routes: the entrypoint
// lookup on /webhook/{uuid}, whose path segment the client picks
// outright, and the user lookup behind the login form, whose username
// the client picks outright. Under the default logger each of those
// misses printed an unbounded, attacker-chosen string, at no level the
// operator can turn down, past every handler internal/logger installs.
//
// This adapter fixes all three properties at once: the lines get a
// level the operator controls, they are shaped by whichever handler
// internal/logger selected, and every value a client can influence is
// spent through logfield.Truncate.
package gormlog
import (
"context"
"errors"
"fmt"
"log/slog"
"time"
gormlogger "gorm.io/gorm/logger"
"sneak.berlin/go/webhooker/internal/logfield"
)
// DefaultSlowThreshold is the duration at or above which a statement
// is logged as slow. It is GORM's own default, kept deliberately: slow
// SQL is the one thing GORM's logger reports that nothing else in this
// service does, so silencing the logger outright would have cost real
// observability to fix a log-volume defect.
const DefaultSlowThreshold = 200 * time.Millisecond
// Logger implements gormlogger.Interface on top of an *slog.Logger.
//
// It is safe for concurrent use: every field is set at construction
// and never written again.
type Logger struct {
log *slog.Logger
slowThreshold time.Duration
}
// Interface compliance is asserted here rather than discovered at the
// gorm.Open call sites.
var _ gormlogger.Interface = (*Logger)(nil)
// New returns a GORM logger that writes through log.
func New(log *slog.Logger) *Logger {
return &Logger{
log: log,
slowThreshold: DefaultSlowThreshold,
}
}
// LogMode returns the logger unchanged.
//
// GORM's LogLevel is deliberately not honoured. Level is the operator's
// decision and it is expressed once, through LOG_LEVEL and the
// slog.LevelVar internal/logger holds; a second level knob inside the
// database layer could only disagree with it. The mapping from GORM's
// four categories onto slog levels is fixed in Trace below.
//
//nolint:ireturn // The interface return is GORM's signature, not a choice.
func (l *Logger) LogMode(gormlogger.LogLevel) gormlogger.Interface {
return l
}
// Info logs one of GORM's own informational messages.
func (l *Logger) Info(
ctx context.Context, msg string, data ...any,
) {
l.log.InfoContext(ctx, "gorm", "message", format(msg, data...))
}
// Warn logs one of GORM's own warnings.
func (l *Logger) Warn(
ctx context.Context, msg string, data ...any,
) {
l.log.WarnContext(ctx, "gorm", "message", format(msg, data...))
}
// Error logs one of GORM's own errors.
func (l *Logger) Error(
ctx context.Context, msg string, data ...any,
) {
l.log.ErrorContext(ctx, "gorm", "message", format(msg, data...))
}
// Trace reports the outcome of a single statement. GORM calls it for
// every statement it runs, so the cheap paths stay cheap: fc()
// renders the interpolated SQL and is called only on a branch that
// will actually emit.
//
// The arms are ordered exactly as GORM's own Trace orders them —
// non-record-not-found error, then slow, then the routine case — so
// that a statement which both misses and runs slow is still reported
// as slow. A miss is the likeliest statement to be slow, since it is
// the one that scans without finding a row, and ordering the drop
// ahead of the slow arm would have made this adapter less observant
// than the IgnoreRecordNotFoundError option it was chosen over.
func (l *Logger) Trace(
ctx context.Context,
begin time.Time,
fc func() (string, int64),
err error,
) {
elapsed := time.Since(begin)
switch {
case err != nil && !errors.Is(err, gormlogger.ErrRecordNotFound):
sql, rows := fc()
l.log.ErrorContext(ctx, "sql statement failed",
"error", logfield.Truncate(err.Error(), logfield.MaxBytes),
"sql", logfield.Truncate(sql, logfield.MaxBytes),
"rows", rows,
"elapsed_ms", elapsed.Milliseconds(),
)
case l.slowThreshold > 0 && elapsed >= l.slowThreshold:
sql, rows := fc()
l.log.WarnContext(ctx, "slow sql statement",
"sql", logfield.Truncate(sql, logfield.MaxBytes),
"rows", rows,
"elapsed_ms", elapsed.Milliseconds(),
"threshold_ms", l.slowThreshold.Milliseconds(),
)
case err != nil:
// gorm.ErrRecordNotFound is not an error on the paths that
// produce it here: an invented entrypoint UUID and an unknown
// username are the expected outcome of an unauthenticated
// request, not a fault. This is the IgnoreRecordNotFoundError
// behaviour, and it is unconditional rather than configurable
// because no caller in this service wants the other one — the
// two handlers that care already record the miss themselves,
// at DEBUG, without the SQL. A miss that ran slow has already
// been reported by the arm above.
return
case l.log.Enabled(ctx, slog.LevelDebug):
sql, rows := fc()
l.log.DebugContext(ctx, "sql statement",
"sql", logfield.Truncate(sql, logfield.MaxBytes),
"rows", rows,
"elapsed_ms", elapsed.Milliseconds(),
)
}
}
// format renders one of GORM's printf-style internal messages and
// bounds it. GORM builds these itself, but they can quote a value the
// statement carried, so they are spent through the same budget as
// everything else rather than trusted.
func format(msg string, data ...any) string {
if len(data) == 0 {
return logfield.Truncate(msg, logfield.MaxBytes)
}
return logfield.Truncate(
fmt.Sprintf(msg, data...), logfield.MaxBytes,
)
}

436
internal/gormlog/gormlog_test.go Normal file
View File

@@ -0,0 +1,436 @@
package gormlog_test
import (
"bytes"
"context"
"database/sql"
"fmt"
"log/slog"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
_ "modernc.org/sqlite" // Pure Go SQLite driver.
"sneak.berlin/go/webhooker/internal/gormlog"
"sneak.berlin/go/webhooker/internal/middleware"
)
// fillBytes is how much client-chosen text each case drives into the
// statement. It is well past every budget in play, so a value that
// arrives short arrived short because something cut it.
const fillBytes = 8 << 10
// tailMarker sits at the far end of every generated value. A line that
// contains it carried the whole value, which means nothing cut it — so
// a value that merely happened to be short cannot pass for a truncated
// one.
const tailMarker = "ENDOFCLIENTVALUE"
// fills are the characters a client can drive into a SQL parameter,
// chosen for what the log handlers charge for them rather than for
// looking dangerous.
//
// The C0 control is the one that matters. Both handlers spell U+0001
// as a six-byte escape for the single byte it costs a client to send,
// which is the widest multiplier available in the basic multilingual
// plane and the case a raw-byte budget breaks on first. The astral
// non-printable costs ten under the text handler, four more than the
// JSON handler ever spends.
func fills() []struct {
name string
fill string
} {
return []struct {
name string
fill string
}{
{"plain", "x"},
{"quote", `"`},
{"backslash", `\`},
{"tab", "\t"},
{"newline", "\n"},
{"c0_control", "\x01"},
{"astral_nonprintable", "\U0001000C"},
}
}
// clientValue builds a value of at least fillBytes raw bytes out of
// fill, ending in tailMarker.
func clientValue(fill string) string {
var b strings.Builder
for b.Len() < fillBytes {
b.WriteString(fill)
}
b.WriteString(tailMarker)
return b.String()
}
// handlers are the two slog handlers internal/logger can install. The
// ceiling is quoted to operators unqualified, so every case is
// asserted under both.
func handlers() []struct {
name string
make func(*bytes.Buffer) slog.Handler
} {
opts := &slog.HandlerOptions{Level: slog.LevelDebug}
return []struct {
name string
make func(*bytes.Buffer) slog.Handler
}{
{"json", func(b *bytes.Buffer) slog.Handler {
return slog.NewJSONHandler(b, opts)
}},
{"text", func(b *bytes.Buffer) slog.Handler {
return slog.NewTextHandler(b, opts)
}},
}
}
type thing struct {
ID string `gorm:"primaryKey"`
Name string
}
// neverSlow is a slow-statement threshold no statement in this file
// can reach. Cases that are about a non-slow arm of Trace set it, so
// that a machine under load cannot turn a miss into a slow report and
// decide the outcome for them.
const neverSlow = time.Hour
// alwaysSlow makes every statement count as slow, so the slow arm is
// reached without the test waiting for it.
const alwaysSlow = time.Nanosecond
// openDB opens a real SQLite database behind the adapter under test,
// so every assertion below is made against SQL that GORM actually
// rendered rather than against a string a test wrote by hand. slow is
// the adapter's slow-statement threshold.
func openDB(
t *testing.T, buf *bytes.Buffer, h slog.Handler, slow time.Duration,
) *gorm.DB {
t.Helper()
sqlDB, err := sql.Open("sqlite", fmt.Sprintf(
"file:%s?mode=rwc",
filepath.Join(t.TempDir(), "gormlog.db"),
))
require.NoError(t, err)
t.Cleanup(func() { _ = sqlDB.Close() })
gl := gormlog.ExportNewWithSlowThreshold(slog.New(h), slow)
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB},
&gorm.Config{Logger: gl},
)
require.NoError(t, err)
require.NoError(t, gdb.AutoMigrate(&thing{}))
// Migration chatter is not what any of these cases is about.
buf.Reset()
return gdb
}
// assertBounded holds every line the adapter wrote to the stated
// ceiling and proves each was cut rather than merely short.
func assertBounded(t *testing.T, out string) {
t.Helper()
assert.NotContains(
t, out, tailMarker,
"the far end of the client value reached the log, so "+
"nothing truncated it",
)
for line := range strings.SplitSeq(
strings.TrimRight(out, "\n"), "\n",
) {
if line == "" {
continue
}
assert.LessOrEqual(
t, len(line), middleware.MaxAccessLogLineBytes,
"log line exceeded its bound: %s",
line[:min(len(line), 300)],
)
}
}
// TestRecordNotFound_WritesNothing is the defect itself. GORM's own
// default logger prints the fully interpolated SELECT on every
// ErrRecordNotFound, and on this service's two unauthenticated
// lookups the interpolated parameter is whatever the client sent.
func TestRecordNotFound_WritesNothing(t *testing.T) {
t.Parallel()
for _, h := range handlers() {
for _, f := range fills() {
t.Run(h.name+"/"+f.name, func(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
gdb := openDB(t, &buf, h.make(&buf), neverSlow)
var got thing
err := gdb.Where(
"id = ?", clientValue(f.fill),
).First(&got).Error
require.ErrorIs(t, err, gorm.ErrRecordNotFound)
assert.Empty(
t, buf.String(),
"a miss on a client-chosen key must not "+
"write a log line",
)
})
}
}
}
// TestSlowRecordNotFound_IsStillReportedSlow pins the arm ordering in
// Trace against the drop above.
//
// GORM's own Trace orders its cases error-that-is-not-a-miss, then
// slow, then routine, so IgnoreRecordNotFoundError: true — the cheap
// option this adapter was chosen over — still reports a miss that ran
// slow. An adapter that dropped the miss first would be strictly less
// observant than the option it replaced, on exactly the two lookups
// this package exists for. A miss is also the statement most likely to
// be slow, since it is the one that scans without finding a row.
func TestSlowRecordNotFound_IsStillReportedSlow(t *testing.T) {
t.Parallel()
for _, h := range handlers() {
for _, f := range fills() {
t.Run(h.name+"/"+f.name, func(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
gdb := openDB(t, &buf, h.make(&buf), alwaysSlow)
var got thing
err := gdb.Where(
"id = ?", clientValue(f.fill),
).First(&got).Error
require.ErrorIs(t, err, gorm.ErrRecordNotFound)
assert.Contains(
t, buf.String(), "slow sql statement",
"a slow statement that missed was not "+
"reported as slow",
)
assertBounded(t, buf.String())
})
}
}
}
// TestRecordNotFoundFlood_DoesNotGrowWithInput states the definition
// of done directly: a flood of misses at two input sizes 64 times
// apart must cost the same number of bytes of log.
func TestRecordNotFoundFlood_DoesNotGrowWithInput(t *testing.T) {
t.Parallel()
const requests = 50
flood := func(t *testing.T, size int) int {
t.Helper()
var buf bytes.Buffer
gdb := openDB(
t, &buf,
slog.NewJSONHandler(&buf, &slog.HandlerOptions{
Level: slog.LevelDebug,
}),
neverSlow,
)
value := strings.Repeat("\x01", size)
for range requests {
var got thing
_ = gdb.Where("id = ?", value).First(&got).Error
}
return buf.Len()
}
small := flood(t, 128)
big := flood(t, 128*64)
assert.Equal(
t, small, big,
"log volume tracked the size of the client's input",
)
}
// TestStatementError_LineIsBounded covers the branch that does log.
// A driver error is not ErrRecordNotFound, so the interpolated
// statement is written — and on an insert the interpolated value is
// still whatever the client supplied.
func TestStatementError_LineIsBounded(t *testing.T) {
t.Parallel()
for _, h := range handlers() {
for _, f := range fills() {
t.Run(h.name+"/"+f.name, func(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
gdb := openDB(t, &buf, h.make(&buf), neverSlow)
row := thing{ID: clientValue(f.fill), Name: "a"}
require.NoError(t, gdb.Create(&row).Error)
buf.Reset()
// The same primary key a second time: a UNIQUE
// constraint failure, which is an error GORM logs.
err := gdb.Create(&thing{
ID: row.ID, Name: "b",
}).Error
require.Error(t, err)
assert.Contains(
t, buf.String(), "sql statement failed",
)
assertBounded(t, buf.String())
})
}
}
}
// TestSucceedingStatement_LineIsBoundedOnEitherArm covers the two
// arms a statement that returns no error can take, over the same
// query, so neither can be bounded by accident of the other.
//
// - slow. Silencing GORM outright would have been the cheaper fix
// and would have cost this report, which is the one thing GORM's
// logger gave an operator that nothing else in this service does.
// - routine. The branch an operator reaches by turning the level
// down to DEBUG: every statement is reported, so every
// statement's interpolated parameters have to be bounded too.
func TestSucceedingStatement_LineIsBoundedOnEitherArm(t *testing.T) {
t.Parallel()
// "sql statement" is a substring of "slow sql statement", so the
// routine arm carries notWant as well: Contains alone cannot tell
// the two arms apart in that direction.
arms := []struct {
name string
slow time.Duration
want string
notWant string
}{
{"slow", alwaysSlow, "slow sql statement", ""},
{"routine", neverSlow, "sql statement", "slow sql statement"},
}
for _, a := range arms {
for _, h := range handlers() {
for _, f := range fills() {
name := a.name + "/" + h.name + "/" + f.name
t.Run(name, func(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
gdb := openDB(t, &buf, h.make(&buf), a.slow)
var got []thing
require.NoError(t, gdb.Where(
"name = ?", clientValue(f.fill),
).Find(&got).Error)
assert.Contains(t, buf.String(), a.want)
if a.notWant != "" {
assert.NotContains(
t, buf.String(), a.notWant,
)
}
assertBounded(t, buf.String())
})
}
}
}
}
// TestGORMOwnMessages_AreBounded covers the three printf-style
// entry points. GORM builds these itself, but nothing stops one of
// them quoting a value the statement carried.
func TestGORMOwnMessages_AreBounded(t *testing.T) {
t.Parallel()
for _, h := range handlers() {
for _, f := range fills() {
t.Run(h.name+"/"+f.name, func(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
gl := gormlog.New(slog.New(h.make(&buf)))
ctx := context.Background()
value := clientValue(f.fill)
gl.Info(ctx, "%s", value)
gl.Warn(ctx, "%s", value)
gl.Error(ctx, "%s", value)
// The no-argument form, which is how GORM reports
// most of its own conditions. Reached through a
// function value so the vet printf check does not
// read the message as a format string — which is
// also why the adapter does not.
noArgs := func(
f func(context.Context, string, ...any),
msg string,
) {
f(ctx, msg)
}
noArgs(gl.Info, value)
assertBounded(t, buf.String())
})
}
}
}
// TestLogMode_KeepsTheOperatorsLevel records that GORM's own level
// knob is deliberately inert: level belongs to LOG_LEVEL, and a
// second one inside the database layer could only disagree with it.
func TestLogMode_KeepsTheOperatorsLevel(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
gl := gormlog.New(slog.New(slog.NewJSONHandler(
&buf, &slog.HandlerOptions{Level: slog.LevelDebug},
)))
assert.Same(t, gl, gl.LogMode(0))
}

124
internal/handlers/auth.go
View File

@@ -2,8 +2,10 @@ package handlers
import (
"net/http"
"strconv"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/logfield"
)
// HandleLoginPage returns a handler for the login page (GET)
@@ -19,7 +21,7 @@ func (h *Handlers) HandleLoginPage() http.HandlerFunc {
// Render login page
data := map[string]any{
"Error": "",
tmplKeyError: "",
}
h.renderTemplate(w, r, "login.html", data)
@@ -29,10 +31,8 @@ func (h *Handlers) HandleLoginPage() http.HandlerFunc {
// HandleLoginSubmit handles the login form submission (POST)
func (h *Handlers) HandleLoginSubmit() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
// Limit request body to prevent memory exhaustion
r.Body = http.MaxBytesReader(w, r.Body, 1<<maxBodyShift)
// Parse form data
// The body size cap is enforced by the MaxBodySize
// middleware, which runs before CSRF parses the form.
err := r.ParseForm()
if err != nil {
h.log.Error("failed to parse form", "error", err)
@@ -41,8 +41,10 @@ func (h *Handlers) HandleLoginSubmit() http.HandlerFunc {
return
}
username := r.FormValue("username")
password := r.FormValue("password")
// PostFormValue, not FormValue: the credential must come
// from the body, never from the query string.
username := r.PostFormValue("username")
password := r.PostFormValue("password")
// Validate input
if username == "" || password == "" {
@@ -69,7 +71,9 @@ func (h *Handlers) HandleLoginSubmit() http.HandlerFunc {
h.log.Info(
"user logged in",
"username", username,
"username", logfield.Truncate(
username, logfield.MaxBytes,
),
"user_id", user.ID,
)
@@ -86,7 +90,7 @@ func (h *Handlers) renderLoginError(
status int,
) {
data := map[string]any{
"Error": msg,
tmplKeyError: msg,
}
w.WriteHeader(status)
@@ -95,6 +99,16 @@ func (h *Handlers) renderLoginError(
// authenticateUser looks up and verifies a user's credentials.
// On failure it writes an HTTP response and returns an error.
//
// The credential check runs BEFORE any rate-limit budget is
// consulted, and only a failed check spends budget. That is what
// keeps the single administrative path reachable: behind the reverse
// proxy this deployment requires, with TRUSTED_PROXIES unset, every
// client shares one bucket, so a limiter spent on arrival lets any
// stranger deny the operator's own correct password indefinitely.
//
// Verifying first means every login POST costs an Argon2id hash, so
// the work is taken under a bounded number of verification slots.
func (h *Handlers) authenticateUser(
w http.ResponseWriter,
r *http.Request,
@@ -102,16 +116,49 @@ func (h *Handlers) authenticateUser(
) (database.User, error) {
var user database.User
release, ok := h.mw.BeginPasswordVerification(r.Context())
if !ok {
h.log.Warn(
"password verification capacity exhausted",
"path", logfield.Truncate(
r.URL.Path, logfield.MaxBytes,
),
)
h.renderLoginError(
w, r,
"The server is busy verifying credentials. "+
"Please try again.",
http.StatusServiceUnavailable,
)
return user, errVerificationBusy
}
defer release()
err := h.db.DB().Where(
"username = ?", username,
).First(&user).Error
if err != nil {
h.log.Debug("user not found", "username", username)
h.renderLoginError(
w, r,
"Invalid username or password",
http.StatusUnauthorized,
// A username that does not exist is charged the same work
// as one that does. Skipping the hash here would answer in
// microseconds where a real account takes tens of
// milliseconds, handing every client a username oracle.
h.dummyVerifications.Add(1)
database.VerifyDummyPassword(password)
// Login is unauthenticated, and the submitted username is
// a form field the client fills to any length the 1 MB
// body cap allows. On this branch it matched no row, so
// nothing else bounds it. The rate limiter caps how often
// the line is written, not how wide it is.
h.log.Debug(
"user not found",
"username", logfield.Truncate(
username, logfield.MaxBytes,
),
)
h.rejectLogin(w, r, username)
return user, err
}
@@ -128,17 +175,60 @@ func (h *Handlers) authenticateUser(
}
if !valid {
h.log.Debug("invalid password", "username", username)
// Reached only once the username matched a stored row, so
// it is bounded by the operator's own data. Capped anyway,
// so that every username this unauthenticated endpoint
// logs is capped and no reader has to work out which
// branch narrowed it.
h.log.Debug(
"invalid password",
"username", logfield.Truncate(
username, logfield.MaxBytes,
),
)
h.rejectLogin(w, r, username)
return user, errInvalidPassword
}
// The password was correct, so forgive whatever failures this
// client accumulated: an operator who mistypes a few times and
// then gets it right must not stay throttled afterwards.
h.mw.ForgiveLoginFailures(r, username)
return user, nil
}
// rejectLogin counts one failed credential verification and answers
// it: 401 while this client still has failure budget against the
// submitted username, 429 with a Retry-After once it is spent.
//
// The 429 throttles wrong passwords only. A correct one never
// reaches here, so no amount of failure — from this client or any
// other sharing its bucket — can keep the operator out.
func (h *Handlers) rejectLogin(
w http.ResponseWriter,
r *http.Request,
username string,
) {
if !h.mw.RecordLoginFailure(r, username) {
h.renderLoginError(
w, r,
"Invalid username or password",
http.StatusUnauthorized,
)
return user, errInvalidPassword
return
}
return user, nil
w.Header().Set("Retry-After", strconv.Itoa(int(
h.mw.LoginFailureInterval().Seconds(),
)))
h.renderLoginError(
w, r,
"Too many failed login attempts. Please try again later.",
http.StatusTooManyRequests,
)
}
// createAuthenticatedSession regenerates the session and stores

455
internal/handlers/auth_test.go Normal file
View File

@@ -0,0 +1,455 @@
package handlers_test
import (
"context"
"fmt"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"sync"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
const (
// operatorUser and operatorPassword are the single admin account
// these tests defend.
operatorUser = "admin"
operatorPassword = "correct horse battery staple"
// sharedProxyPeer is the whole point of this file. Production is
// required to run behind a TLS-terminating reverse proxy, and
// TRUSTED_PROXIES defaults to empty, so every client — attacker
// and operator alike — reaches the process from the proxy's
// address and shares one rate-limit bucket. Both parties in
// these tests therefore use the same RemoteAddr.
sharedProxyPeer = "10.0.0.1:44444"
// loginFailureLimit is the failure budget one client has against
// one submitted username. Restated here rather than imported
// from the middleware package, so that changing the production
// limit fails these tests instead of silently moving with them.
loginFailureLimit = 5
)
// seedOperator gives the bootstrapped admin account a password these
// tests know. The account itself is created at startup with a random
// password, which is exactly why its username is predictable to an
// attacker and why keying failures by username alone does not fix
// this issue.
func seedOperator(t *testing.T, db *database.Database) {
t.Helper()
hash, err := database.HashPassword(operatorPassword)
require.NoError(t, err)
result := db.DB().Model(&database.User{}).
Where("username = ?", operatorUser).
Update("password", hash)
require.NoError(t, result.Error)
require.EqualValues(
t, 1, result.RowsAffected,
"the bootstrap admin account must exist",
)
}
// loginPost builds a login form POST arriving from peer.
func loginPost(peer, username, password string) *http.Request {
form := url.Values{}
form.Set("username", username)
form.Set("password", password)
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost,
"/pages/login",
strings.NewReader(form.Encode()),
)
req.Header.Set(
"Content-Type", "application/x-www-form-urlencoded",
)
req.RemoteAddr = peer
return req
}
// submitLogin drives one login POST through the handler.
func submitLogin(
h *handlers.Handlers, peer, username, password string,
) *httptest.ResponseRecorder {
w := httptest.NewRecorder()
h.HandleLoginSubmit().ServeHTTP(w, loginPost(
peer, username, password,
))
return w
}
// floodFailures sends attempts wrong-password logins for username
// from peer, which is what an attacker does.
func floodFailures(
t *testing.T,
h *handlers.Handlers,
peer, username string,
attempts int,
) {
t.Helper()
for i := range attempts {
w := submitLogin(h, peer, username, fmt.Sprintf("guess-%d", i))
require.NotEqual(
t, http.StatusSeeOther, w.Code,
"attempt %d must not authenticate", i,
)
}
}
// TestLogin_StrangersFloodCannotLockOutTheOperator is the
// done-criterion of https://git.eeqj.de/sneak/webhooker/issues/150.
//
// The attacker and the operator share one rate-limit bucket, because
// behind the mandated reverse proxy with TRUSTED_PROXIES unset every
// client keys on the proxy's address. The attacker floods the
// operator's own username — a single-admin product has a predictable
// one — far past the failure limit. The operator must still be able
// to log in with the correct password.
//
// This fails if credentials stop being verified ahead of the limiter.
func TestLogin_StrangersFloodCannotLockOutTheOperator(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
db *database.Database
)
app := newTestApp(t, &h, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
seedOperator(t, db)
// Well past the limit, and from the same bucket the operator
// will arrive in.
floodFailures(
t, h, sharedProxyPeer, operatorUser,
loginFailureLimit*2,
)
w := submitLogin(
h, sharedProxyPeer, operatorUser, operatorPassword,
)
assert.Equal(
t, http.StatusSeeOther, w.Code,
"a correct password must never be throttled: the operator "+
"has no second administrative path",
)
assert.Equal(t, "/", w.Header().Get("Location"))
}
// TestLogin_StrangersFloodCannotDenyAnotherAccount is the
// cross-account half: flooding one username must not spend another
// account's budget, even from the same shared bucket.
func TestLogin_StrangersFloodCannotDenyAnotherAccount(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
db *database.Database
)
app := newTestApp(t, &h, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
seedOperator(t, db)
floodFailures(
t, h, sharedProxyPeer, "someone-else",
loginFailureLimit*2,
)
w := submitLogin(h, sharedProxyPeer, operatorUser, "wrong")
assert.Equal(
t, http.StatusUnauthorized, w.Code,
"a flood against one username must not spend another "+
"account's failure budget",
)
}
// TestLogin_RepeatedWrongPasswordsAreThrottled is the brute-force
// half. Verifying before counting must not remove the throttle:
// repeated wrong passwords for one username from one client key run
// out of budget and are answered 429 with a Retry-After.
func TestLogin_RepeatedWrongPasswordsAreThrottled(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
db *database.Database
)
app := newTestApp(t, &h, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
seedOperator(t, db)
for i := range loginFailureLimit - 1 {
w := submitLogin(
h, sharedProxyPeer, operatorUser,
fmt.Sprintf("guess-%d", i),
)
assert.Equal(
t, http.StatusUnauthorized, w.Code,
"attempt %d is still inside the budget", i,
)
}
w := submitLogin(h, sharedProxyPeer, operatorUser, "guess-last")
assert.Equal(
t, http.StatusTooManyRequests, w.Code,
"wrong passwords must still run out of budget",
)
assert.NotEmpty(
t, w.Header().Get("Retry-After"),
"a throttled login must say when to come back",
)
}
// TestLogin_SuccessForgivesEarlierMistakes covers the operator who
// mistypes several times and then gets it right: the successful
// attempt clears the counter, so the next mistake is answered 401
// rather than 429.
func TestLogin_SuccessForgivesEarlierMistakes(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
db *database.Database
)
app := newTestApp(t, &h, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
seedOperator(t, db)
floodFailures(
t, h, sharedProxyPeer, operatorUser,
loginFailureLimit,
)
require.Equal(
t, http.StatusSeeOther,
submitLogin(
h, sharedProxyPeer, operatorUser, operatorPassword,
).Code,
)
w := submitLogin(h, sharedProxyPeer, operatorUser, "typo")
assert.Equal(
t, http.StatusUnauthorized, w.Code,
"a success must forgive the failures before it",
)
}
// TestLogin_UnknownUsernameCostsTheSameVerification is the
// username-enumeration guard. Verifying credentials before the
// limiter means response time is observable per attempt, so an
// unknown username must be charged an equivalent-cost verification
// against a dummy hash rather than returning early.
//
// The assertion is on the code path, not on wall-clock time: timing
// assertions are flaky, and what actually has to hold is that the
// hash is computed.
func TestLogin_UnknownUsernameCostsTheSameVerification(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
db *database.Database
)
app := newTestApp(t, &h, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
seedOperator(t, db)
require.Zero(t, h.DummyVerificationsForTest())
// A username that exists, with the wrong password: a real
// Argon2id verification runs, and no dummy is needed.
require.Equal(
t, http.StatusUnauthorized,
submitLogin(h, sharedProxyPeer, operatorUser, "wrong").Code,
)
assert.Zero(
t, h.DummyVerificationsForTest(),
"a known username verifies against its own hash",
)
// A username that does not exist: indistinguishable response,
// and the equivalent-cost verification must have run.
require.Equal(
t, http.StatusUnauthorized,
submitLogin(h, sharedProxyPeer, "nosuchuser", "wrong").Code,
)
assert.Equal(
t, uint64(1), h.DummyVerificationsForTest(),
"an unknown username must still pay for a hash, or the "+
"response time says whether the account exists",
)
}
// TestLogin_ConcurrentLoginsAreAllAnswered covers the login path
// under the verification bound. The bound itself is pinned in the
// middleware package; what matters here is that funnelling every
// login through two slots does not lose or wedge a request — each one
// is answered, whether it got a slot or was shed with 503.
func TestLogin_ConcurrentLoginsAreAllAnswered(t *testing.T) {
t.Parallel()
const workers = 4
var (
h *handlers.Handlers
db *database.Database
)
app := newTestApp(t, &h, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
seedOperator(t, db)
var (
wg sync.WaitGroup
mu sync.Mutex
answers = map[int]int{}
)
for i := range workers {
wg.Go(func() {
w := submitLogin(
h, fmt.Sprintf("203.0.113.%d:5000", i),
operatorUser, fmt.Sprintf("guess-%d", i),
)
mu.Lock()
answers[w.Code]++
mu.Unlock()
})
}
wg.Wait()
mu.Lock()
defer mu.Unlock()
assert.Zero(
t, answers[http.StatusInternalServerError],
"concurrent logins must not error",
)
assert.Equal(
t, workers,
answers[http.StatusUnauthorized]+
answers[http.StatusTooManyRequests]+
answers[http.StatusServiceUnavailable],
"every concurrent login must be answered, whether it got "+
"a verification slot or was shed with 503",
)
}
// TestLogin_MissingCredentialsRejectedBeforeAnyHash pins that the
// empty-field check still runs ahead of the verification slot, so a
// client sending nothing cannot occupy one.
func TestLogin_MissingCredentialsRejectedBeforeAnyHash(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
db *database.Database
)
app := newTestApp(t, &h, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
seedOperator(t, db)
w := submitLogin(h, sharedProxyPeer, "", "")
assert.Equal(t, http.StatusBadRequest, w.Code)
assert.Zero(
t, h.DummyVerificationsForTest(),
"an empty submission must not cost a hash",
)
}
// TestLogin_SuccessCreatesSession is the control for the tests above:
// the success path they assert on really does authenticate.
func TestLogin_SuccessCreatesSession(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
db *database.Database
sess *session.Session
)
app := newTestApp(t, &h, &db, &sess)
app.RequireStart()
t.Cleanup(app.RequireStop)
seedOperator(t, db)
w := submitLogin(
h, sharedProxyPeer, operatorUser, operatorPassword,
)
require.Equal(t, http.StatusSeeOther, w.Code)
require.NotEmpty(
t, w.Result().Cookies(), "a session cookie must be issued",
)
next := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/", nil,
)
// Login regenerates the session, so the response carries two
// Set-Cookie headers under the same name: one expiring the
// pre-login cookie and one issuing the new one. A browser keeps
// only the second, so replay only the one that is not an
// expiry.
for _, c := range w.Result().Cookies() {
if c.MaxAge >= 0 {
next.AddCookie(c)
}
}
s, err := sess.Get(next)
require.NoError(t, err)
assert.True(
t, sess.IsAuthenticated(s),
"the issued cookie must carry an authenticated session",
)
}

199
internal/handlers/event_body.go Normal file
View File

@@ -0,0 +1,199 @@
package handlers
import (
"database/sql"
"errors"
"net/http"
"strconv"
"github.com/go-chi/chi"
"github.com/google/uuid"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// eventBodyQuery reads one event's stored body as bytes. The cast
// to blob is what makes the driver hand back the stored bytes
// rather than a string conversion, so Content-Length taken from
// the result matches what goes on the wire. The soft-delete
// predicate is spelled out because Raw bypasses GORM's default
// scope, and it is what stops a reaped event still being
// downloadable.
const eventBodyQuery = "SELECT cast(body as blob) " +
"FROM events WHERE id = ? AND webhook_id = ? AND deleted_at IS NULL"
// HandleEventBodyDownload serves one event's stored body in
// full, which the event log page cannot: it caps each rendered
// body at maxRenderedBodyBytes.
//
// The bytes are attacker-supplied — anyone who can reach the
// public receiver chooses them — and this route hands them back
// inside the operator's own authenticated origin, so the
// response is deliberately not renderable. Content-Disposition
// makes the browser download rather than display it, and the
// octet-stream type plus nosniff stop it being interpreted as
// HTML or script. Without those a stored payload would execute
// as the logged-in operator. The application's CSP does not
// help here: script-src allows 'unsafe-inline' from 'self', so
// a document served from this origin could run its own inline
// script.
func (h *Handlers) HandleEventBodyDownload() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
webhook, ok := h.ownedWebhook(w, r)
if !ok {
return
}
// Parsing the id before use serves two purposes: a
// malformed id can never reach the SQL or the response
// header, and the canonical form below is drawn from
// uuid's own fixed alphabet rather than from the
// request, so the Content-Disposition value cannot be
// steered by a client.
eventID, err := uuid.Parse(chi.URLParam(r, "eventID"))
if err != nil {
http.NotFound(w, r)
return
}
h.serveEventBody(w, r, webhook, eventID.String())
}
}
// serveEventBody writes the named event's stored body to w.
//
// The event must belong to webhook, which is what keeps this
// route from reading any event in the system by id alone. Two
// things enforce that and they are not equally strong. The
// operative one is that events live in a per-webhook SQLite
// file, so a sibling webhook's event is not in the database
// being queried at all. The webhook_id predicate on the query
// below is the second guard, and it is currently redundant
// against that isolation; it is there so the scoping survives
// any future change that puts more than one webhook's events in
// one file.
//
// The body is read in one query and held whole in memory while
// it is written. That costs roughly two body-sized allocations
// per concurrent download, not one: the driver's column buffer
// and the copy database/sql makes in convertAssign when a
// []byte column is scanned into a *[]byte are live at the same
// time. Measured allocation is ~2x the body plus ~45 KB, so at
// the 1 MB ingest cap a download costs ~2 MB of Go heap. On
// top of that, SQLite's own materialisation of the column
// value sits in the driver's allocator outside the Go heap, so
// process peak is higher again: 2x is a floor, not a ceiling.
// There is no cheaper bound available — database/sql exposes
// no incremental handle on a SQLite BLOB, and reading byte
// ranges with substr does not avoid the cost either, because
// SQLite materialises the whole column value to evaluate each
// substr call. Range reads only pay for that materialisation
// once per range.
//
// One consequence is worth keeping in view: the read finishes
// before the client is written to, so no read lock is held for
// the length of a slow download. These per-webhook databases
// run in SQLite's default journal mode rather than WAL, so a
// lock held that long would block the receiver from recording
// new events.
func (h *Handlers) serveEventBody(
w http.ResponseWriter,
r *http.Request,
webhook database.Webhook,
eventID string,
) {
if !h.dbMgr.DBExists(webhook.ID) {
http.NotFound(w, r)
return
}
webhookDB, err := h.dbMgr.GetDB(webhook.ID)
if err != nil {
h.serverError(w, "failed to get webhook database", err)
return
}
body, found, err := eventBody(webhookDB, webhook.ID, eventID)
if err != nil {
h.serverError(w, "failed to read event body", err)
return
}
// A miss is a 404 whether the event belongs to another
// webhook or does not exist at all, so the response does
// not report which. Reading the body before any header is
// written is also what keeps an event reaped mid-request
// from producing a torn response: either the read finds the
// row and the whole body is served, or it does not and the
// response is a clean 404.
if !found {
http.NotFound(w, r)
return
}
setEventBodyHeaders(w, eventID, int64(len(body)))
_, err = w.Write(body)
if err != nil {
// The status and Content-Length are already committed,
// so the client sees a short download. There is no way
// to report a 500 from here; the log is the record.
h.log.Error(
"failed to write event body",
"webhook_id", webhook.ID,
"event_id", eventID,
"error", err,
)
}
}
// eventBody returns an event's stored body and whether the event
// exists within the webhook.
func eventBody(
webhookDB *gorm.DB,
webhookID, eventID string,
) ([]byte, bool, error) {
var body []byte
err := webhookDB.Raw(
eventBodyQuery, eventID, webhookID,
).Row().Scan(&body)
if errors.Is(err, sql.ErrNoRows) {
return nil, false, nil
}
if err != nil {
return nil, false, err
}
return body, true, nil
}
// setEventBodyHeaders applies the response headers that make
// this route safe to hand attacker-supplied bytes through. See
// HandleEventBodyDownload for why they are a security control
// and not a formatting choice.
//
// nosniff is also set by the global SecurityHeaders middleware.
// It is repeated here so the guarantee belongs to the route
// that needs it rather than to a middleware someone could
// reorder or scope away.
func setEventBodyHeaders(
w http.ResponseWriter,
eventID string,
size int64,
) {
w.Header().Set("Content-Type", "application/octet-stream")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set(
"Content-Disposition",
`attachment; filename="webhooker-event-`+eventID+`.bin"`,
)
w.Header().Set("Content-Length", strconv.FormatInt(size, 10))
}

506
internal/handlers/event_body_test.go Normal file
View File

@@ -0,0 +1,506 @@
package handlers_test
import (
"context"
"net/http"
"net/http/httptest"
"net/url"
"strconv"
"strings"
"testing"
"github.com/go-chi/chi"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
// paramEventID is the chi URL parameter the body download
// handler reads.
const paramEventID = "eventID"
// otherTestUserID owns webhooks the session user must not be
// able to read.
const otherTestUserID = "other-user-id"
// seedWebhookFor inserts a webhook owned by the given user.
func seedWebhookFor(
t *testing.T,
db *database.Database,
userID string,
) *database.Webhook {
t.Helper()
wh := &database.Webhook{
UserID: userID,
Name: "wh-" + userID,
}
require.NoError(
t,
db.DB().Omit(clause.Associations).Create(wh).Error,
)
return wh
}
// fetchEventBody runs the real download handler as the test user
// for the given source and event ids.
func fetchEventBody(
t *testing.T,
h *handlers.Handlers,
sess *session.Session,
sourceID, eventID string,
) *httptest.ResponseRecorder {
t.Helper()
// The path is escaped and the raw id goes in the route
// context, which is what chi hands a handler: the param is
// already percent-decoded by the time it is read.
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet,
"/source/"+url.PathEscape(sourceID)+
"/logs/"+url.PathEscape(eventID)+"/body",
nil,
)
for _, c := range authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
) {
req.AddCookie(c)
}
rctx := chi.NewRouteContext()
rctx.URLParams.Add(paramSourceID, sourceID)
rctx.URLParams.Add(paramEventID, eventID)
req = req.WithContext(
context.WithValue(
req.Context(), chi.RouteCtxKey, rctx,
),
)
w := httptest.NewRecorder()
h.HandleEventBodyDownload().ServeHTTP(w, req)
return w
}
// TestHandleEventBodyDownload_ServesOversizeBodyInFull is the
// capability the render cap took away: a body far above what the
// event log page will show comes back whole and byte-identical,
// with the headers that keep it from being rendered.
func TestHandleEventBodyDownload_ServesOversizeBodyInFull(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
// Far above the render cap, with multibyte runes and a
// distinctive tail, so a body that the log page can only
// show a slice of comes back whole and in order.
const sentinel = "TAIL-SENTINEL-1f4a9c"
stored := strings.Repeat("A", 200*1024) +
strings.Repeat(snowman, 1000) + sentinel
wh := seedWebhook(t, db)
evt := seedEventWithBody(t, dbMgr, wh.ID, stored)
w := fetchEventBody(t, h, sess, wh.ID, evt.ID)
require.Equal(t, http.StatusOK, w.Code)
assert.Greater(t, len(stored), bodyCap)
assert.Equal(t, stored, w.Body.String())
assert.Equal(
t, strconv.Itoa(len(stored)),
w.Header().Get("Content-Length"),
)
}
// TestHandleEventBodyDownload_BodiesRoundTripByteIdentical
// covers the sizes and byte values a stored body can actually
// take: empty, one byte, either side of the render cap, and
// bytes that are not text at all. Content-Length has to equal
// the bytes written in every case, since it is derived from the
// same read that produces them.
func TestHandleEventBodyDownload_BodiesRoundTripByteIdentical(
t *testing.T,
) {
t.Parallel()
// A NUL, invalid UTF-8 and a multibyte rune, so nothing on
// the path can be treating the body as text.
binary := "\x00\x01\xff\xfe" + snowman + "\x00tail"
cases := map[string]string{
"empty": "",
"single byte": "x",
"one below cap": strings.Repeat("b", bodyCap-1),
"exactly cap": strings.Repeat("c", bodyCap),
"one above cap": strings.Repeat("d", bodyCap+1),
"binary": binary,
}
for name, stored := range cases {
t.Run(name, func(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
evt := seedEventWithBody(t, dbMgr, wh.ID, stored)
w := fetchEventBody(t, h, sess, wh.ID, evt.ID)
require.Equal(t, http.StatusOK, w.Code)
assert.Equal(t, stored, w.Body.String())
assert.Equal(
t, strconv.Itoa(len(stored)),
w.Header().Get("Content-Length"),
)
assert.Equal(
t, len(stored), w.Body.Len(),
"Content-Length must equal bytes written",
)
})
}
}
// TestHandleEventBodyDownload_HeadersAreNotRenderable pins the
// response headers that stop attacker-supplied bytes executing
// in the operator's own origin. They are a security control, not
// presentation.
func TestHandleEventBodyDownload_HeadersAreNotRenderable(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
evt := seedEventWithBody(t, dbMgr, wh.ID, `{"small":true}`)
w := fetchEventBody(t, h, sess, wh.ID, evt.ID)
require.Equal(t, http.StatusOK, w.Code)
assert.Equal(
t, "application/octet-stream",
w.Header().Get("Content-Type"),
)
assert.Equal(
t, "nosniff",
w.Header().Get("X-Content-Type-Options"),
)
disposition := w.Header().Get("Content-Disposition")
assert.Equal(
t,
`attachment; filename="webhooker-event-`+evt.ID+`.bin"`,
disposition,
)
}
// TestHandleEventBodyDownload_ScriptBodyStaysInert proves a
// stored HTML payload is handed back as an attachment of opaque
// bytes rather than as anything a browser will execute. The
// bytes themselves are unaltered: this route reports what was
// delivered.
func TestHandleEventBodyDownload_ScriptBodyStaysInert(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
const payload = `<html><script>alert(document.cookie)` +
`</script></html>`
wh := seedWebhook(t, db)
evt := seedEventWithBody(t, dbMgr, wh.ID, payload)
w := fetchEventBody(t, h, sess, wh.ID, evt.ID)
require.Equal(t, http.StatusOK, w.Code)
assert.Equal(t, payload, w.Body.String())
contentType := w.Header().Get("Content-Type")
assert.Equal(t, "application/octet-stream", contentType)
assert.NotContains(t, contentType, "html")
assert.NotContains(t, contentType, "xml")
assert.NotContains(t, contentType, "javascript")
assert.Contains(
t, w.Header().Get("Content-Disposition"), "attachment",
)
assert.Equal(
t, "nosniff",
w.Header().Get("X-Content-Type-Options"),
)
}
// TestHandleEventBodyDownload_OtherUsersEvent404s is the
// authorization test the definition of done asks for: an event
// stored under a webhook the session user does not own is not
// readable, and the miss does not distinguish itself from a
// nonexistent one.
func TestHandleEventBodyDownload_OtherUsersEvent404s(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
const theirPayload = "OTHER-USERS-PAYLOAD-8b1d"
theirs := seedWebhookFor(t, db, otherTestUserID)
evt := seedEventWithBody(t, dbMgr, theirs.ID, theirPayload)
w := fetchEventBody(t, h, sess, theirs.ID, evt.ID)
assert.Equal(t, http.StatusNotFound, w.Code)
assert.NotContains(t, w.Body.String(), theirPayload)
}
// TestHandleEventBodyDownload_EventOfAnotherWebhook404s pins
// that holding a valid event id is not enough: the event has to
// belong to the webhook in the path. Both webhooks here are the
// session user's and both have event databases, so the
// ownership check cannot be what produces the 404.
//
// What does produce it is the per-webhook database file rather
// than the webhook_id predicate on the query — removing that
// predicate leaves this test green, because the sibling's event
// is in a different file. The test is kept as the behavioural
// guard the route owes; see serveEventBody for which mechanism
// is load-bearing.
func TestHandleEventBodyDownload_EventOfAnotherWebhook404s(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
const other = "BELONGS-TO-THE-OTHER-WEBHOOK-3c7e"
mine := seedWebhook(t, db)
seedEventWithBody(t, dbMgr, mine.ID, `{"mine":true}`)
sibling := seedWebhook(t, db)
evt := seedEventWithBody(t, dbMgr, sibling.ID, other)
w := fetchEventBody(t, h, sess, mine.ID, evt.ID)
assert.Equal(t, http.StatusNotFound, w.Code)
assert.NotContains(t, w.Body.String(), other)
}
// TestHandleEventBodyDownload_UnknownEvent404s covers the plain
// miss, including an id that is not a uuid at all and so never
// reaches the query or the response header.
func TestHandleEventBodyDownload_UnknownEvent404s(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedEventWithBody(t, dbMgr, wh.ID, `{"mine":true}`)
for _, id := range []string{
uuid.New().String(),
`../../etc/passwd`,
"not-a-uuid",
`x"; rm -rf /`,
} {
w := fetchEventBody(t, h, sess, wh.ID, id)
assert.Equal(
t, http.StatusNotFound, w.Code,
"event id %q", id,
)
assert.Empty(
t, w.Header().Get("Content-Disposition"),
"event id %q must not reach a header", id,
)
}
}
// TestHandleEventBodyDownload_ReapedEvent404s pins what happens
// when the retention reaper takes an event out from under this
// route. The body is read in one query before any header is
// written, so a reaped event cannot produce a partial download:
// it is a clean 404 with no Content-Length and no
// Content-Disposition. Both removals the codebase performs are
// covered — the reaper hard-deletes, and a soft-deleted row is
// excluded by the query's own deleted_at predicate rather than
// by GORM's default scope, which Raw bypasses.
func TestHandleEventBodyDownload_ReapedEvent404s(t *testing.T) {
t.Parallel()
for name, hard := range map[string]bool{
"soft deleted": false,
"hard deleted": true,
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
const payload = "REAPED-PAYLOAD-4d2a"
wh := seedWebhook(t, db)
evt := seedEventWithBody(t, dbMgr, wh.ID, payload)
webhookDB, err := dbMgr.GetDB(wh.ID)
require.NoError(t, err)
del := webhookDB
if hard {
del = del.Unscoped()
}
require.NoError(
t,
del.Delete(&database.Event{}, "id = ?", evt.ID).
Error,
)
w := fetchEventBody(t, h, sess, wh.ID, evt.ID)
assert.Equal(t, http.StatusNotFound, w.Code)
assert.NotContains(t, w.Body.String(), payload)
assert.Empty(t, w.Header().Get("Content-Length"))
assert.Empty(
t, w.Header().Get("Content-Disposition"),
)
})
}
}
// TestHandleSourceLogs_TruncationMarkerLinksToDownload proves
// the page tells the reader where the rest of the body is, and
// only when there is a rest to fetch.
func TestHandleSourceLogs_TruncationMarkerLinksToDownload(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
big := seedWebhook(t, db)
bigEvt := seedEventWithBody(
t, dbMgr, big.ID, strings.Repeat("A", 4*bodyCap),
)
page := renderSourceLogsPage(t, h, sess, big.ID)
assert.Contains(
t, page,
"/source/"+big.ID+"/logs/"+bigEvt.ID+"/body",
)
small := seedWebhook(t, db)
smallEvt := seedEventWithBody(
t, dbMgr, small.ID, `{"kept":"whole"}`,
)
page = renderSourceLogsPage(t, h, sess, small.ID)
assert.NotContains(
t, page,
"/source/"+small.ID+"/logs/"+smallEvt.ID+"/body",
)
}

120
internal/handlers/event_log_view.go Normal file
View File

@@ -0,0 +1,120 @@
package handlers
import (
"time"
"unicode/utf8"
)
// maxRenderedBodyBytes caps how many bytes of a stored event
// body reach the event log page. Bodies come from the
// unauthenticated receiver under the 1 MB ingest cap and
// renderTemplate buffers a whole page before writing it, so
// an uncapped page of paginationPerPage events is tens of
// megabytes of resident memory per concurrent viewer.
const maxRenderedBodyBytes = 8192
// eventLogColumns is the event log's projection. The casts to
// blob are load-bearing: they make substr and length count
// bytes rather than characters, so the cap bounds the page in
// bytes whatever the payload's encoding. Cutting in SQLite
// rather than in Go is the point of the projection — an
// oversized body never becomes a Go string at all.
const eventLogColumns = "id, created_at, method, content_type, " +
"substr(cast(body as blob), 1, ?) AS body, " +
"length(cast(body as blob)) AS body_bytes"
// EventLogView is the display-safe projection of an event for
// the event log page, alongside DeliveryView and TargetView.
// It carries a capped body plus the true stored size, so the
// page can mark a body as truncated without ever holding the
// whole thing.
type EventLogView struct {
ID string
CreatedAt time.Time
Method string
ContentType string
// Body holds at most maxRenderedBodyBytes bytes of the
// stored body.
Body string
// BodyBytes is the true size of the stored body.
BodyBytes int64
// BodyTruncated reports that the stored body was larger
// than the cap, so the page owes the reader a marker.
BodyTruncated bool
Deliveries []DeliveryView
}
// BodyShownBytes is how many body bytes the page is actually
// rendering, which the truncation marker reports beside the
// true size.
func (v EventLogView) BodyShownBytes() int {
return len(v.Body)
}
// eventLogRow is one row of the event log projection. Its
// body column arrives already cut to the cap by SQLite, with
// the true size beside it.
type eventLogRow struct {
ID string
CreatedAt time.Time
Method string
ContentType string
Body []byte
BodyBytes int64
}
// view projects a loaded row for rendering.
func (r *eventLogRow) view() EventLogView {
body := r.Body
truncated := r.BodyBytes > int64(len(body))
// Only a cut body can have been left mid-sequence by
// this query. A whole body is passed through exactly as
// stored, however malformed.
if truncated {
body = trimPartialRune(body)
}
return EventLogView{
ID: r.ID,
CreatedAt: r.CreatedAt,
Method: r.Method,
ContentType: r.ContentType,
Body: string(body),
BodyBytes: r.BodyBytes,
BodyTruncated: truncated,
}
}
// trimPartialRune drops a trailing UTF-8 sequence that the
// byte-wise cut left incomplete, so a multi-byte rune severed
// at the cap does not surface as a mojibake tail.
//
// Bytes that are merely invalid UTF-8 are left exactly as
// stored: this service receives binary payloads, and rewriting
// them would misreport what was delivered. The distinction is
// utf8.FullRune's — it reports a complete sequence for an
// invalid encoding too, since that decodes to a width-1 error
// rune, so only a valid prefix still waiting for its
// continuation bytes is removed. A tail with no rune start in
// its last utf8.UTFMax bytes cannot be an incomplete sequence
// either, and is likewise left alone.
func trimPartialRune(b []byte) []byte {
for i := len(b) - 1; i >= 0 && len(b)-i <= utf8.UTFMax; i-- {
if !utf8.RuneStart(b[i]) {
continue
}
if utf8.FullRune(b[i:]) {
return b
}
return b[:i]
}
return b
}

261
internal/handlers/event_log_view_test.go Normal file
View File

@@ -0,0 +1,261 @@
package handlers_test
import (
"net/http"
"net/http/httptest"
"strconv"
"strings"
"testing"
"unicode/utf8"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
// bodyCap is the number of body bytes the event log page is
// allowed to render for one event.
const bodyCap = handlers.MaxRenderedBodyBytesForTest
// snowman is a three-byte rune, so a body of them straddles the
// byte-wise cut: bodyCap is not a multiple of three.
const snowman = "☃"
// seedEventWithBody records one event with the given body in the
// webhook's own database and returns it, so a caller that needs
// the generated event id can have it.
func seedEventWithBody(
t *testing.T,
dbMgr *database.WebhookDBManager,
webhookID string,
body string,
) *database.Event {
t.Helper()
webhookDB, err := dbMgr.GetDB(webhookID)
require.NoError(t, err)
event := &database.Event{
WebhookID: webhookID,
Method: http.MethodPost,
Body: body,
ContentType: "application/octet-stream",
}
require.NoError(t, webhookDB.Omit(
clause.Associations,
).Create(event).Error)
return event
}
// seedAndProject stores one body and returns the projection the
// event log page would be handed for it.
func seedAndProject(
t *testing.T,
body string,
) handlers.EventLogView {
t.Helper()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedEventWithBody(t, dbMgr, wh.ID, body)
views := h.LoadEventLogViewsForTest(
httptest.NewRecorder(), *wh, 1,
)
require.Len(t, views, 1)
return views[0]
}
// TestHandleSourceLogs_BoundsOversizeBody proves the rendered
// page is bounded by the cap rather than by the stored payload:
// the body here is 64 times the cap, and the ingest path would
// accept twice as much again.
func TestHandleSourceLogs_BoundsOversizeBody(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
const (
sentinel = "TAIL-SENTINEL-1f4a9c"
storedBytes = 512 * 1024
)
wh := seedWebhook(t, db)
seedEventWithBody(
t, dbMgr, wh.ID,
strings.Repeat("A", storedBytes-len(sentinel))+sentinel,
)
page := renderSourceLogsPage(t, h, sess, wh.ID)
// Nothing past the cap reaches the page, and the whole page
// stays far below the stored body it is reporting on.
assert.NotContains(t, page, sentinel)
assert.Less(t, len(page), 4*bodyCap)
// The marker states the true stored size, not the cut one.
assert.Contains(
t, page,
"showing "+strconv.Itoa(bodyCap)+
" of "+strconv.Itoa(storedBytes)+" bytes",
)
}
// TestHandleSourceLogs_SmallBodyRendersWhole guards the other
// side of the cap: a body under it is shown in full and carries
// no truncation marker.
func TestHandleSourceLogs_SmallBodyRendersWhole(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedEventWithBody(t, dbMgr, wh.ID, `{"kept":"whole"}`)
page := renderSourceLogsPage(t, h, sess, wh.ID)
assert.Contains(t, page, "&#34;kept&#34;")
assert.NotContains(t, page, "Body truncated for display")
}
// TestEventLogView_CutMidRune proves a multi-byte rune severed
// by the byte-wise cut is dropped rather than surfaced as a
// mojibake tail.
func TestEventLogView_CutMidRune(t *testing.T) {
t.Parallel()
body := strings.Repeat(snowman, 4096)
view := seedAndProject(t, body)
// bodyCap bytes hold bodyCap/3 whole snowmen and two bytes
// of the next one; those two are dropped.
whole := bodyCap / len(snowman)
assert.True(t, view.BodyTruncated)
assert.Equal(t, int64(len(body)), view.BodyBytes)
assert.Equal(t, strings.Repeat(snowman, whole), view.Body)
assert.True(t, utf8.ValidString(view.Body))
assert.LessOrEqual(t, len(view.Body), bodyCap)
}
// TestEventLogView_BinaryBodyLeftAsStored proves a binary
// payload is passed through byte for byte. Its tail is invalid
// UTF-8 however the cut falls, so repairing it would misreport
// what the sender delivered.
func TestEventLogView_BinaryBodyLeftAsStored(t *testing.T) {
t.Parallel()
raw := make([]byte, bodyCap+808)
for i := range raw {
// 0x80..0xBF: continuation bytes, never a rune start.
raw[i] = 0x80 | byte(i%0x40)
}
view := seedAndProject(t, string(raw))
assert.True(t, view.BodyTruncated)
assert.Equal(t, int64(len(raw)), view.BodyBytes)
assert.Equal(t, string(raw[:bodyCap]), view.Body)
assert.False(t, utf8.ValidString(view.Body))
}
// TestTrimPartialRune covers the distinction the cut repair
// turns on: an incomplete but valid sequence is dropped, while
// bytes that are merely invalid UTF-8 are left alone.
func TestTrimPartialRune(t *testing.T) {
t.Parallel()
cases := []struct {
name string
in []byte
want []byte
}{{
name: "complete ascii",
in: []byte("abc"),
want: []byte("abc"),
}, {
name: "complete multibyte",
in: []byte("ab" + snowman),
want: []byte("ab" + snowman),
}, {
name: "two byte rune cut",
in: []byte{'a', 0xC3},
want: []byte{'a'},
}, {
name: "three byte rune cut after one",
in: []byte{'a', 0xE2},
want: []byte{'a'},
}, {
name: "three byte rune cut after two",
in: []byte{'a', 0xE2, 0x98},
want: []byte{'a'},
}, {
name: "four byte rune cut",
in: []byte{'a', 0xF0, 0x9F, 0x92}, // U+1F4A9 cut
want: []byte{'a'},
}, {
name: "invalid start byte kept",
in: []byte{'a', 0xFF},
want: []byte{'a', 0xFF},
}, {
name: "orphan continuation bytes kept",
in: []byte{0x80, 0x81, 0x82, 0x83, 0x84},
want: []byte{0x80, 0x81, 0x82, 0x83, 0x84},
}, {
name: "truncated sequence followed by junk kept",
in: []byte{0xE2, 0x98, 0xFF},
want: []byte{0xE2, 0x98, 0xFF},
}, {
name: "empty",
in: []byte{},
want: []byte{},
}}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
assert.Equal(
t, tc.want,
handlers.TrimPartialRuneForTest(tc.in),
)
})
}
}

78
internal/handlers/export_test.go
View File

@@ -1,6 +1,62 @@
package handlers
import "net/http"
import (
"html/template"
"log/slog"
"net/http"
"sneak.berlin/go/webhooker/internal/database"
)
// SetLogForTest replaces the handler's logger, so the handlers_test
// package can assert on what a log line actually contains rather than
// on what it is meant to contain.
func (s *Handlers) SetLogForTest(log *slog.Logger) {
s.log = log
}
// MaxRenderedBodyBytesForTest exposes the event log's body cap
// to the handlers_test package.
const MaxRenderedBodyBytesForTest = maxRenderedBodyBytes
// DummyVerificationsForTest reports how many equivalent-cost
// verifications were charged for usernames that do not exist. It
// lets a test prove the anti-enumeration path ran without timing
// anything.
func (s *Handlers) DummyVerificationsForTest() uint64 {
return s.dummyVerifications.Load()
}
// TrimPartialRuneForTest exposes trimPartialRune for use in the
// handlers_test package.
func TrimPartialRuneForTest(b []byte) []byte {
return trimPartialRune(b)
}
// LoadEventLogViewsForTest exposes loadEventsWithDeliveries for
// use in the handlers_test package. Assertions on the projected
// body need the bytes as loaded: html/template rewrites invalid
// UTF-8 on the way out, so the rendered page cannot show whether
// a binary body survived the projection intact.
func (s *Handlers) LoadEventLogViewsForTest(
w http.ResponseWriter,
webhook database.Webhook,
page int,
) []EventLogView {
views, _ := s.loadEventsWithDeliveries(w, webhook, nil, page)
return views
}
// AddTemplateForTest registers a template under a page name so that
// the handlers_test package can drive the render path with a
// template of its own.
func (s *Handlers) AddTemplateForTest(
pageTemplate string,
tmpl *template.Template,
) {
s.templates[pageTemplate] = tmpl
}
// RenderTemplateForTest exposes renderTemplate for use in the
// handlers_test package.
@@ -13,12 +69,26 @@ func (s *Handlers) RenderTemplateForTest(
s.renderTemplate(w, r, pageTemplate, data)
}
// BuildSlackTargetConfigForTest exposes buildSlackTargetConfig
// for use in the handlers_test package.
// BuildSlackTargetConfigForTest exposes buildURLTargetConfig
// with the Slack target parameters for use in the
// handlers_test package.
func (s *Handlers) BuildSlackTargetConfigForTest(
w http.ResponseWriter,
r *http.Request,
targetURL string,
) (string, error) {
return s.buildSlackTargetConfig(w, r, targetURL)
return s.buildURLTargetConfig(
w, r, targetURL, "webhookUrl",
"Webhook URL is required for Slack targets",
)
}
// BuildDatabaseTargetConfigForTest exposes
// buildDatabaseTargetConfig for use in the handlers_test
// package.
func (s *Handlers) BuildDatabaseTargetConfigForTest(
w http.ResponseWriter,
expiry string,
) (string, error) {
return s.buildDatabaseTargetConfig(w, expiry)
}

462
internal/handlers/gormlogbound_test.go Normal file
View File

@@ -0,0 +1,462 @@
package handlers_test
import (
"bytes"
"context"
"io"
"log"
"net/http"
"net/http/httptest"
"os"
"strconv"
"strings"
"sync"
"testing"
"time"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
gormlogger "gorm.io/gorm/logger"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/middleware"
)
// gormBoundTailMarker sits at the far end of every client-chosen value
// this file sends. Its presence in the log means the whole value
// reached the log, so a value that merely happened to be short cannot
// pass for a truncated one.
const gormBoundTailMarker = "ENDOFCLIENTVALUE"
// gormBoundFills are the characters a client can drive through the
// receiver path segment and the login username, chosen for what a log
// handler charges for them.
//
// The bare C0 control is the one that matters: both handlers spell
// U+0001 as a six-byte escape for the one byte it costs to send, the
// widest multiplier available below U+10000 and the case a raw-byte
// budget breaks on first. GORM's default logger applies no budget at
// all, so under the mutation every one of these arrives whole.
func gormBoundFills() []struct {
name string
fill string
} {
return []struct {
name string
fill string
}{
{"plain", "x"},
{"quote", `"`},
{"backslash", `\`},
{"tab", "\t"},
{"newline", "\n"},
{"c0_control", "\x01"},
{"astral_nonprintable", "\U0001000C"},
}
}
// syncBuf collects captured output from the goroutine draining the
// pipe.
type syncBuf struct {
mu sync.Mutex
b bytes.Buffer
}
func (s *syncBuf) Write(p []byte) (int, error) {
s.mu.Lock()
defer s.mu.Unlock()
return s.b.Write(p)
}
func (s *syncBuf) String() string {
s.mu.Lock()
defer s.mu.Unlock()
return s.b.String()
}
func (s *syncBuf) reset() {
s.mu.Lock()
defer s.mu.Unlock()
s.b.Reset()
}
// stdoutCapture redirects os.Stdout for the duration of a test.
//
// internal/logger builds its handler over os.Stdout at construction
// time, so redirecting the variable before the application is built
// captures everything the service logger — and therefore the GORM
// adapter, which writes through it — emits.
type stdoutCapture struct {
buf *syncBuf
r *os.File
w *os.File
orig *os.File
done chan struct{}
seq int
}
func captureStdout(t *testing.T) *stdoutCapture {
t.Helper()
r, w, err := os.Pipe()
require.NoError(t, err)
c := &stdoutCapture{
buf: &syncBuf{},
r: r,
w: w,
orig: os.Stdout,
done: make(chan struct{}),
}
os.Stdout = w
go func() {
defer close(c.done)
_, _ = io.Copy(c.buf, r)
}()
t.Cleanup(func() {
os.Stdout = c.orig
_ = w.Close()
<-c.done
_ = r.Close()
})
return c
}
// drain returns everything written since the previous drain and
// clears the buffer.
//
// A sentinel is pushed through the same pipe and waited for, so the
// draining goroutine is known to have caught up before the buffer is
// read. Without it the comparison below would race the reader rather
// than measure the writers.
func (c *stdoutCapture) drain(t *testing.T) string {
t.Helper()
c.seq++
sentinel := "\n<<drain-" + strconv.Itoa(c.seq) + ">>\n"
_, err := c.w.WriteString(sentinel)
require.NoError(t, err)
deadline := time.Now().Add(10 * time.Second)
for !strings.Contains(c.buf.String(), sentinel) {
require.False(
t, time.Now().After(deadline),
"timed out waiting for captured output",
)
time.Sleep(time.Millisecond)
}
out := strings.Replace(c.buf.String(), sentinel, "", 1)
c.buf.reset()
return out
}
// teeStdout writes to a buffer and to whatever os.Stdout is at the
// moment of the write.
//
// The second half is the point. GORM's package-level default logger
// resolves os.Stdout once, at package init, so a logger built over the
// variable would keep writing to the real terminal no matter what a
// test redirects. Resolving it per write puts the bytes a defaulted
// gorm.Config would cost in production into the same capture as
// everything else internal/logger emits, which is what lets the volume
// assertions below measure the whole writer set rather than one member
// of it.
type teeStdout struct {
buf *syncBuf
}
func (w teeStdout) Write(p []byte) (int, error) {
_, _ = os.Stdout.Write(p)
return w.buf.Write(p)
}
// captureGORMDefault replaces GORM's package-level default logger with
// one configured exactly as GORM configures its own, writing to a
// buffer and to os.Stdout.
//
// This is the mutation detector. gormlogger.Default is what a bare
// &gorm.Config{} installs, and its config here is GORM's verbatim —
// Warn, IgnoreRecordNotFoundError false — so a reverted call site
// behaves as it would in production rather than as a test dialed it.
// With every gorm.Open in this service naming its own logger, nothing
// consults this value and the buffer stays empty; revert any one of
// the three and the interpolated SQL lands here.
func captureGORMDefault(t *testing.T) *syncBuf {
t.Helper()
buf := &syncBuf{}
orig := gormlogger.Default
gormlogger.Default = gormlogger.New(
log.New(teeStdout{buf: buf}, "", log.LstdFlags),
gormlogger.Config{
SlowThreshold: 200 * time.Millisecond,
LogLevel: gormlogger.Warn,
IgnoreRecordNotFoundError: false,
Colorful: false,
},
)
t.Cleanup(func() { gormlogger.Default = orig })
return buf
}
// floodUnauthenticated drives reps requests at each of the two
// unauthenticated lookups that miss by design, for every fill, with a
// client-chosen value of size raw bytes.
func floodUnauthenticated(
t *testing.T, h *handlers.Handlers, size, reps int,
) int {
t.Helper()
requests := 0
for _, f := range gormBoundFills() {
var b strings.Builder
for b.Len() < size {
b.WriteString(f.fill)
}
b.WriteString(gormBoundTailMarker)
value := b.String()
for range reps {
postWebhook(t, h, value)
postUnknownLogin(t, h, value)
requests += 2
}
}
return requests
}
// floodPerWebhook drives the same client-chosen values at the second
// gorm.Open site, the per-webhook database internal/database's
// WebhookDBManager opens.
//
// That site is behind authentication in production, so this is not
// part of the unauthenticated flood above and is counted separately.
// It is here because the ceiling the README states covers every
// writer, and the manager is one of them: with nothing driving it, a
// bare &gorm.Config{} could be restored at
// internal/database/webhook_db_manager.go and the whole suite would
// stay green.
func floodPerWebhook(
t *testing.T, mgr *database.WebhookDBManager, size, reps int,
) int {
t.Helper()
requests := 0
for _, f := range gormBoundFills() {
var b strings.Builder
for b.Len() < size {
b.WriteString(f.fill)
}
b.WriteString(gormBoundTailMarker)
value := b.String()
db, err := mgr.GetDB("pin-" + f.name)
require.NoError(t, err)
for range reps {
var got database.Event
err = db.Where("id = ?", value).First(&got).Error
require.ErrorIs(t, err, gorm.ErrRecordNotFound)
requests++
}
}
return requests
}
// postWebhook drives the receiver with an invented entrypoint path.
// The route pattern matches any single segment, so every byte of the
// value is the client's, and the lookup behind it misses by design.
func postWebhook(
t *testing.T, h *handlers.Handlers, entrypoint string,
) {
t.Helper()
req := httptest.NewRequestWithContext(
context.Background(), http.MethodPost, "/webhook/x",
strings.NewReader("{}"),
)
rctx := chi.NewRouteContext()
rctx.URLParams.Add("uuid", entrypoint)
req = req.WithContext(context.WithValue(
req.Context(), chi.RouteCtxKey, rctx,
))
w := httptest.NewRecorder()
h.HandleWebhook().ServeHTTP(w, req)
require.Equal(t, http.StatusNotFound, w.Code)
}
// postUnknownLogin submits the login form with an unknown username,
// through the postLogin helper in logbound_test.go. The field is
// bounded only by the 1 MB body cap, and the lookup behind it misses
// by design.
func postUnknownLogin(
t *testing.T, h *handlers.Handlers, username string,
) {
t.Helper()
// 401 while the client still has failure budget against this
// username, 429 once the login guard has taken it away. Both
// outcomes sit behind the user lookup, which is the query this
// test is here to drive.
require.Contains(
t,
[]int{http.StatusUnauthorized, http.StatusTooManyRequests},
postLogin(t, h, username),
)
}
// assertFloodBounded holds every captured line to the stated ceiling
// and proves nothing carried a whole client value.
func assertFloodBounded(t *testing.T, label, out string) {
t.Helper()
assert.NotContains(
t, out, gormBoundTailMarker,
"%s: the far end of a client-chosen value reached the "+
"log, so nothing truncated it", label,
)
for line := range strings.SplitSeq(
strings.TrimRight(out, "\n"), "\n",
) {
if line == "" {
continue
}
assert.LessOrEqual(
t, len(line), middleware.MaxAccessLogLineBytes,
"%s: log line exceeded its bound: %s",
label, line[:min(len(line), 300)],
)
}
}
// TestFlood_NoWriterGrowsWithTheInput is the definition of done for
// the GORM logger defect, stated over every writer at once, for two of
// this service's three gorm.Open sites: the main database behind the
// two unauthenticated lookups, and the per-webhook database the
// WebhookDBManager opens. The third, the archive writer, is pinned in
// internal/delivery, where its type lives.
//
// What each assertion is worth, since two of the three would pass
// against a service that had never been fixed if the capture were set
// up differently:
//
// - The gormDefault check is the sharp one. It fires the moment any
// gorm.Open in this service goes back to a bare &gorm.Config{}.
// - The volume and per-line checks bite only because the replaced
// default logger tees into os.Stdout, so a reverted call site
// shows up in the same capture as everything internal/logger
// writes — the way it would in production. Without that tee both
// were vacuous: at INFO the two handler misses log at DEBUG and
// the adapter drops the record-not-found, so the capture holds
// nothing but fixed-string warnings.
//
// The level is left where newTestApp leaves it, at INFO: the level an
// operator runs at by default, and the one the defect was visible at.
// The handlers' own miss lines sit at DEBUG and spend the same
// logfield budget as everything else, so they are not what makes
// either assertion above bite at any level.
//
// It is deliberately not parallel: it redirects os.Stdout and replaces
// gormlogger.Default, both of which are process-global. Go runs every
// non-parallel top-level test to completion before it resumes the
// parallel ones, so nothing else in this package is running while the
// capture is installed.
//
//nolint:paralleltest // Deliberately sequential; see above.
func TestFlood_NoWriterGrowsWithTheInput(t *testing.T) {
const (
smallBytes = 128
bigBytes = 8 << 10
reps = 5
)
gormDefault := captureGORMDefault(t)
capture := captureStdout(t)
var (
h *handlers.Handlers
mgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &mgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
// Startup chatter is not what this test measures.
capture.drain(t)
floodUnauthenticated(t, h, smallBytes, reps)
floodPerWebhook(t, mgr, smallBytes, reps)
small := capture.drain(t)
requests := floodUnauthenticated(t, h, bigBytes, reps)
requests += floodPerWebhook(t, mgr, bigBytes, reps)
big := capture.drain(t)
assertFloodBounded(t, "small flood", small)
assertFloodBounded(t, "big flood", big)
// GORM's default logger is what the defect was. Nothing in this
// service may reach it.
got := gormDefault.String()
assert.Empty(
t, got,
"GORM's default logger wrote %d bytes; the first of them: %s",
len(got), got[:min(len(got), 300)],
)
// The same flood, with 64 times the client-chosen input, must not
// buy 64 times the log. A few bytes of slack covers a latency
// field changing width; the input grew by roughly half a megabyte.
const slackPerRequest = 64
assert.LessOrEqual(
t, len(big), len(small)+slackPerRequest*requests,
"log volume tracked the size of the client's input: "+
"%d bytes at %d bytes of input per request, %d bytes "+
"at %d",
len(small), smallBytes, len(big), bigBytes,
)
}

50
internal/handlers/handlers.go
View File

@@ -3,12 +3,14 @@
package handlers
import (
"bytes"
"context"
"encoding/json"
"errors"
"html/template"
"log/slog"
"net/http"
"sync/atomic"
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/database"
@@ -26,15 +28,24 @@ const (
maxBodyShift = 20
// recentEventLimit is the number of recent events to show.
recentEventLimit = 20
// defaultRetentionDays is the default event retention period.
defaultRetentionDays = 30
// paginationPerPage is the number of items per page.
paginationPerPage = 25
// tmplKeyError is the template data key for an error message.
tmplKeyError = "Error"
// tmplKeyWebhook is the template data key for a webhook.
tmplKeyWebhook = "Webhook"
)
// errInvalidPassword is returned when a password does not match.
var errInvalidPassword = errors.New("invalid password")
// errVerificationBusy is returned when no password-verification slot
// became free before the wait elapsed, so no password was verified.
var errVerificationBusy = errors.New(
"password verification capacity exhausted",
)
//nolint:revive // HandlersParams is a standard fx naming convention.
type HandlersParams struct {
fx.In
@@ -45,7 +56,9 @@ type HandlersParams struct {
WebhookDBMgr *database.WebhookDBManager
Healthcheck *healthcheck.Healthcheck
Session *session.Session
Middleware *middleware.Middleware
Notifier delivery.Notifier
Evictor delivery.WebhookEvictor
}
// Handlers provides HTTP handler methods for all application
@@ -57,8 +70,15 @@ type Handlers struct {
db *database.Database
dbMgr *database.WebhookDBManager
session *session.Session
mw *middleware.Middleware
notifier delivery.Notifier
evictor delivery.WebhookEvictor
templates map[string]*template.Template
// dummyVerifications counts the equivalent-cost verifications
// charged for usernames that do not exist. It exists so a test
// can prove that path runs without measuring wall-clock time.
dummyVerifications atomic.Uint64
}
// parsePageTemplate parses a page-specific template set from the
@@ -91,7 +111,9 @@ func New(
s.db = params.Database
s.dbMgr = params.WebhookDBMgr
s.session = params.Session
s.mw = params.Middleware
s.notifier = params.Notifier
s.evictor = params.Evictor
// Parse all page templates once at startup
s.templates = map[string]*template.Template{
@@ -218,13 +240,22 @@ func (s *Handlers) renderTemplate(
s.executeTemplate(w, tmpl, wrapper)
}
// executeTemplate runs the template and handles errors.
// executeTemplate renders the template into a buffer and writes to
// the response only once rendering has fully succeeded. Executing
// straight into the ResponseWriter commits a partial body and a 200
// status before a mid-render error can be reported, leaving no way
// to serve a 500. Buffering makes a page's rendered size resident
// memory per concurrent viewer, so every page owes it a bound: the
// event log caps each stored body at maxRenderedBodyBytes for exactly
// this reason.
func (s *Handlers) executeTemplate(
w http.ResponseWriter,
tmpl *template.Template,
data any,
) {
err := tmpl.Execute(w, data)
var buf bytes.Buffer
err := tmpl.Execute(&buf, data)
if err != nil {
s.log.Error(
"failed to execute template", "error", err,
@@ -233,5 +264,16 @@ func (s *Handlers) executeTemplate(
w, "Internal server error",
http.StatusInternalServerError,
)
return
}
w.Header().Set("Content-Type", "text/html; charset=utf-8")
_, err = buf.WriteTo(w)
if err != nil {
s.log.Error(
"failed to write rendered page", "error", err,
)
}
}

153
internal/handlers/handlers_test.go
View File

@@ -2,8 +2,11 @@ package handlers_test
import (
"context"
"errors"
"html/template"
"net/http"
"net/http/httptest"
"sync"
"testing"
"github.com/stretchr/testify/assert"
@@ -17,6 +20,7 @@ import (
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/healthcheck"
"sneak.berlin/go/webhooker/internal/logger"
"sneak.berlin/go/webhooker/internal/middleware"
"sneak.berlin/go/webhooker/internal/session"
)
@@ -24,6 +28,32 @@ type noopNotifier struct{}
func (n *noopNotifier) Notify([]delivery.Task) {}
// recordingEvictor is a delivery.WebhookEvictor that records
// the webhook ids it was asked to evict, so a test can prove
// that a deletion path reached the delivery engine.
type recordingEvictor struct {
mu sync.Mutex
evicted []string
}
func (r *recordingEvictor) EvictWebhook(webhookID string) {
r.mu.Lock()
defer r.mu.Unlock()
r.evicted = append(r.evicted, webhookID)
}
// Evicted returns a copy of the recorded webhook ids.
func (r *recordingEvictor) Evicted() []string {
r.mu.Lock()
defer r.mu.Unlock()
out := make([]string, len(r.evicted))
copy(out, r.evicted)
return out
}
func newTestApp(
t *testing.T,
targets ...any,
@@ -47,6 +77,13 @@ func newTestApp(
func() delivery.Notifier {
return &noopNotifier{}
},
func() *recordingEvictor {
return &recordingEvictor{}
},
func(r *recordingEvictor) delivery.WebhookEvictor {
return r
},
middleware.New,
handlers.New,
),
fx.Populate(targets...),
@@ -186,3 +223,119 @@ func TestRenderTemplate(t *testing.T) {
t, http.StatusInternalServerError, w.Code,
)
}
// errMidRender is the failure a test template raises partway through
// rendering.
var errMidRender = errors.New("deliberate mid-render failure")
// midRenderFailure is template data whose first method renders and
// whose second fails, so the template aborts after output has
// already been produced.
type midRenderFailure struct{}
// Prefix is the output a streaming renderer would flush before the
// failure below aborts the template.
func (midRenderFailure) Prefix() string { return partialPageMarker }
// Boom aborts template execution.
func (midRenderFailure) Boom() (string, error) {
return "", errMidRender
}
// partialPageMarker is content the failing template emits before it
// aborts.
const partialPageMarker = "PARTIAL PAGE CONTENT"
// TestRenderTemplateMidRenderErrorSendsNoPartialBody proves the
// renderer does not commit output it cannot finish: a template that
// fails partway through must yield a 500 and a body carrying none of
// the content emitted before the failure. Against a renderer that
// executes straight into the ResponseWriter this fails on both
// counts, returning 200 with the prefix already flushed.
func TestRenderTemplateMidRenderErrorSendsNoPartialBody(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
app := newTestApp(t, &h)
app.RequireStart()
t.Cleanup(app.RequireStop)
h.AddTemplateForTest("failing.html", template.Must(
template.New("failing").Parse(
`{{.Data.Prefix}}{{.Data.Boom}}TAIL`,
),
))
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/", nil)
w := httptest.NewRecorder()
h.RenderTemplateForTest(
w, req, "failing.html", midRenderFailure{},
)
assert.Equal(
t, http.StatusInternalServerError, w.Code,
"a failed render must report a 500",
)
assert.Equal(
t, "Internal server error\n", w.Body.String(),
"the response must carry no part of the aborted page",
)
}
func TestBuildDatabaseTargetConfig_Valid(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
app := newTestApp(t, &h)
app.RequireStart()
t.Cleanup(app.RequireStop)
// Empty expiry: the keep-forever default, empty config.
w := httptest.NewRecorder()
cfg, err := h.BuildDatabaseTargetConfigForTest(w, "")
require.NoError(t, err)
assert.Empty(t, cfg)
// Explicit never is stored as config.
w = httptest.NewRecorder()
cfg, err = h.BuildDatabaseTargetConfigForTest(w, "never")
require.NoError(t, err)
assert.JSONEq(t, `{"expiry":"never"}`, cfg)
// A positive duration is stored as config.
w = httptest.NewRecorder()
cfg, err = h.BuildDatabaseTargetConfigForTest(w, "720h")
require.NoError(t, err)
assert.JSONEq(t, `{"expiry":"720h"}`, cfg)
}
func TestBuildDatabaseTargetConfig_RejectsBadExpiry(
t *testing.T,
) {
t.Parallel()
var h *handlers.Handlers
app := newTestApp(t, &h)
app.RequireStart()
t.Cleanup(app.RequireStop)
for _, bad := range []string{"nonsense", "7d", "-5h"} {
w := httptest.NewRecorder()
cfg, err := h.BuildDatabaseTargetConfigForTest(w, bad)
require.Error(t, err, "expiry %q", bad)
assert.Empty(t, cfg)
assert.Equal(
t, http.StatusBadRequest, w.Code,
"expiry %q should be rejected with 400", bad,
)
}
}

543
internal/handlers/logbound_test.go Normal file
View File

@@ -0,0 +1,543 @@
package handlers_test
// The handler-side half of the log-field audit. Two slog calls in
// this package reach a value an UNAUTHENTICATED client picks outright
// and of a length it picks outright:
//
// - the unknown-entrypoint DEBUG line on /webhook/{uuid}, whose
// path segment matched no stored entrypoint and so is bounded by
// nothing;
// - the failed-login DEBUG lines, whose username is a form field.
//
// Both are at DEBUG, which is off in production by default. That is
// not a bound: an operator turning DEBUG on to diagnose a flood must
// not thereby hand the flood an unbounded write. Both spend the same
// internal/logfield budget as the access log, and both are held here
// to middleware.MaxAccessLogLineBytes.
//
// The two login lines past the username lookup — "invalid password"
// and "user logged in" — carry the same cap without needing it, since
// by then the value is a stored row rather than the client's. They are
// pinned here too, so the caps cannot be dropped silently.
//
// So is the "password verification capacity exhausted" WARN line,
// whose path chi pins to the constant "/pages/login" on the one route
// that reaches it. Its cap is defensive, and the test below drives the
// handler directly with the path a parameterised route would give it,
// because an unasserted cap is one a later edit removes for free.
import (
"bytes"
"context"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/middleware"
)
// floodRequests is the number of distinct invented values each flood
// drives through the call site under test.
const floodRequests = 32
// oversizedFillBytes is the length of the single client-chosen value
// used to show that line size does not track input size.
const oversizedFillBytes = 8192
// attackerMarker and tailMarker sit at the END of every oversized
// value, past every budget. Their absence from the log is what
// proves the value was cut rather than merely being short.
const (
attackerMarker = "QQATTACKERTEXTQQ"
tailMarker = "QQTRUNCATEDTAILQQ"
)
// escapeFills are the characters the log handlers escape, so a value
// built out of them costs more on the line than it did on the wire. A
// budget counted in raw bytes passes the plain case and fails these.
//
// U+1000C is unassigned, hence non-printable, and strconv.Quote
// spells it as a ten-byte \UXXXXXXXX while the JSON handler passes
// its four UTF-8 bytes through; only the text shape of these tests
// reaches that charge.
func escapeFills() map[string]string {
return map[string]string{
"plain": "x",
"quote": `"`,
"backslash": `\`,
"tab": "\t",
"newline": "\n",
// A C0 control neither handler has a short escape for, so
// each one costs six bytes on the line against the single
// byte it cost to send: the widest multiplier a client can
// drive, and the case a raw-byte budget breaks on first.
//
// This fill is load-bearing, not decoration. Budgeting raw
// bytes instead of encoded is caught by this fill alone,
// and only under the JSON handler, at 3,072 bytes against
// the 2,560 ceiling. Drop it and that mutation passes.
"control": "\x01",
"astral": "\U0001000C",
}
}
// logHandlers are the two handlers internal/logger can install.
func logHandlers() map[string]func(
io.Writer, *slog.HandlerOptions,
) slog.Handler {
return map[string]func(
io.Writer, *slog.HandlerOptions,
) slog.Handler{
"json": func(
w io.Writer, o *slog.HandlerOptions,
) slog.Handler {
return slog.NewJSONHandler(w, o)
},
"text": func(
w io.Writer, o *slog.HandlerOptions,
) slog.Handler {
return slog.NewTextHandler(w, o)
},
}
}
// oversizedFill builds an 8 KB client-chosen value out of
// repetitions of ch, with both markers at its far end.
func oversizedFill(ch string) string {
return "x" + strings.Repeat(ch, oversizedFillBytes) +
attackerMarker + tailMarker
}
// capturingHandlers builds a Handlers whose log is captured into the
// returned buffer at DEBUG through the named handler.
//
// extra is passed to fx.Populate alongside the Handlers, for the call
// sites that also need the database the client's value is looked up
// in, or the Middleware whose resource has to be exhausted before the
// branch under test is reached.
func capturingHandlers(
t *testing.T,
newHandler func(io.Writer, *slog.HandlerOptions) slog.Handler,
extra ...any,
) (*handlers.Handlers, *bytes.Buffer) {
t.Helper()
var h *handlers.Handlers
app := newTestApp(t, append([]any{&h}, extra...)...)
app.RequireStart()
t.Cleanup(app.RequireStop)
buf := new(bytes.Buffer)
h.SetLogForTest(slog.New(newHandler(
buf, &slog.HandlerOptions{Level: slog.LevelDebug},
)))
return h, buf
}
// logLines splits the captured buffer into non-empty lines, holding
// each to the stated per-line ceiling.
func logLines(t *testing.T, buf *bytes.Buffer) []string {
t.Helper()
var lines []string
for line := range strings.SplitSeq(
strings.TrimSpace(buf.String()), "\n",
) {
if line == "" {
continue
}
require.LessOrEqual(
t, len(line), middleware.MaxAccessLogLineBytes,
"log line exceeded its bound: %s", line,
)
lines = append(lines, line)
}
return lines
}
// assertNoClientText fails if the far end of the client-chosen input
// survived into the log.
func assertNoClientText(t *testing.T, buf *bytes.Buffer) {
t.Helper()
assert.NotContains(
t, buf.String(), attackerMarker,
"log carried attacker-chosen text",
)
assert.NotContains(
t, buf.String(), tailMarker,
"log carried the tail of the attacker-chosen text",
)
}
// receiverRouter mounts the real receiver handler at the production
// route pattern.
func receiverRouter(h *handlers.Handlers) *chi.Mux {
router := chi.NewRouter()
router.Post("/webhook/{uuid}", h.HandleWebhook())
return router
}
// postReceiver sends one POST at /webhook/<segment>.
//
// RawPath is cleared after parsing so chi routes on the decoded path
// and the handler sees the raw bytes rather than their percent-escaped
// spelling. That is the harder case for the budget: the escaped
// spelling is plain ASCII, which costs one byte per byte, while the
// decoded bytes are what the log handler has to escape.
func postReceiver(
t *testing.T, router *chi.Mux, segment string,
) int {
t.Helper()
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost,
"/webhook/"+url.PathEscape(segment),
strings.NewReader(""),
)
req.URL.RawPath = ""
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
return w.Code
}
// postLogin submits the login form with the given username and a
// non-empty password.
func postLogin(
t *testing.T, h *handlers.Handlers, username string,
) int {
t.Helper()
return postLoginWithPassword(t, h, username, "not-the-password")
}
// postLoginWithPassword submits the login form with both credentials
// chosen by the caller, so a test can reach the branches past the
// username lookup.
func postLoginWithPassword(
t *testing.T, h *handlers.Handlers, username, password string,
) int {
t.Helper()
form := url.Values{
"username": {username},
"password": {password},
}
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost,
"/pages/login",
strings.NewReader(form.Encode()),
)
req.Header.Set(
"Content-Type", "application/x-www-form-urlencoded",
)
w := httptest.NewRecorder()
h.HandleLoginSubmit().ServeHTTP(w, req)
return w.Code
}
// TestUnknownEntrypoint_LogLineDoesNotTrackPathSize drives 8 KB of
// client-chosen path at the unauthenticated receiver's
// unknown-entrypoint DEBUG line and holds it to the same ceiling the
// access log states.
func TestUnknownEntrypoint_LogLineDoesNotTrackPathSize(t *testing.T) {
t.Parallel()
for handlerName, newHandler := range logHandlers() {
for fillName, fill := range escapeFills() {
t.Run(handlerName+"/"+fillName, func(t *testing.T) {
t.Parallel()
h, buf := capturingHandlers(t, newHandler)
router := receiverRouter(h)
for i := range floodRequests {
assert.Equal(
t,
http.StatusNotFound,
postReceiver(
t, router,
oversizedFill(fill)+
strings.Repeat("y", i),
),
)
}
lines := logLines(t, buf)
require.Len(t, lines, floodRequests)
assertNoClientText(t, buf)
assertBoundedFlood(t, buf.Len())
})
}
}
}
// TestFailedLogin_LogLineDoesNotTrackUsernameSize drives 8 KB of
// client-chosen username at the unauthenticated login endpoint's
// DEBUG line and holds it to the same ceiling.
func TestFailedLogin_LogLineDoesNotTrackUsernameSize(t *testing.T) {
t.Parallel()
for handlerName, newHandler := range logHandlers() {
for fillName, fill := range escapeFills() {
t.Run(handlerName+"/"+fillName, func(t *testing.T) {
t.Parallel()
h, buf := capturingHandlers(t, newHandler)
for i := range floodRequests {
assert.Equal(
t,
http.StatusUnauthorized,
postLogin(
t, h,
oversizedFill(fill)+
strings.Repeat("y", i),
),
)
}
lines := logLines(t, buf)
require.Len(t, lines, floodRequests)
assertNoClientText(t, buf)
assertBoundedFlood(t, buf.Len())
})
}
}
}
// storedUserPassword is the password held by the oversize accounts
// the test below creates.
const storedUserPassword = "correct-horse-battery-staple"
// storedFillBytes is the raw length of the client-chosen value in
// those accounts' usernames. It is well past the 512-byte field
// budget, so the line is still truncated, but short enough that the
// session cookie a successful login writes stays inside
// securecookie's 4 KB limit: the cookie is written BEFORE the
// "user logged in" line, so an 8 KB username answers 500 and never
// reaches it.
const storedFillBytes = 1024
// storedFill builds a username fill of storedFillBytes raw bytes out
// of repetitions of ch, with both markers at its far end.
func storedFill(ch string) string {
return "x" + strings.Repeat(ch, storedFillBytes/len(ch)) +
attackerMarker + tailMarker
}
// TestStoredUsername_LogLinesDoNotTrackUsernameSize pins the two
// login lines that are reached only AFTER the username matched a
// stored row: "invalid password" and "user logged in". Neither
// strictly needs its cap — the value is the operator's own data by
// then, not the client's — but both carry one so that every username
// this unauthenticated endpoint logs is capped, and an unasserted cap
// is one a later edit removes for free.
//
// One app per handler with the accounts created inside it, and no
// parallelism below that level: every account costs an Argon2id hash
// and every attempt costs a verification.
func TestStoredUsername_LogLinesDoNotTrackUsernameSize(t *testing.T) {
t.Parallel()
for handlerName, newHandler := range logHandlers() {
t.Run(handlerName, func(t *testing.T) {
t.Parallel()
var db *database.Database
h, buf := capturingHandlers(t, newHandler, &db)
hash, err := database.HashPassword(storedUserPassword)
require.NoError(t, err)
fills := escapeFills()
for fillName, fill := range fills {
username := storedFill(fill) + fillName
require.NoError(t, db.DB().Create(&database.User{
Username: username,
Password: hash,
}).Error)
// Matched the row, wrong secret: "invalid
// password".
assert.Equal(
t, http.StatusUnauthorized,
postLoginWithPassword(
t, h, username, "not-the-password",
),
)
// Matched the row, right secret: "user logged
// in".
assert.Equal(
t, http.StatusSeeOther,
postLoginWithPassword(
t, h, username, storedUserPassword,
),
)
}
lines := logLines(t, buf)
require.Len(t, lines, 2*len(fills))
assertNoClientText(t, buf)
})
}
}
// maxVerificationSlots bounds how many slots the loop below will
// take before it gives up, so a semaphore that never fills fails the
// test instead of hanging it. It is deliberately larger than the
// real concurrency bound, which is not exported to this package.
const maxVerificationSlots = 64
// canceledContext returns a context that is already done. A
// verification request carrying one takes the ctx.Done() branch of
// the semaphore's bounded wait immediately, so these cases turn on
// the semaphore being full rather than on a five-second timer firing.
// Nothing here is timing-dependent.
func canceledContext() context.Context {
ctx, cancel := context.WithCancel(context.Background())
cancel()
return ctx
}
// holdEveryVerificationSlot takes verification slots until one is
// refused, and releases them when the test ends. A free slot is
// handed out before any context is consulted, so a canceled context
// cannot make this loop stop early: it stops exactly when the slots
// are gone.
func holdEveryVerificationSlot(
t *testing.T, mw *middleware.Middleware,
) {
t.Helper()
for range maxVerificationSlots {
release, ok := mw.BeginPasswordVerification(canceledContext())
if !ok {
return
}
t.Cleanup(release)
}
require.Fail(t, "the verification semaphore never filled")
}
// postLoginAtPath submits the login form at a path of the caller's
// choosing, with a canceled context.
func postLoginAtPath(
t *testing.T, h *handlers.Handlers, path string,
) int {
t.Helper()
form := url.Values{
"username": {"someone"},
"password": {"not-the-password"},
}
req := httptest.NewRequestWithContext(
canceledContext(),
http.MethodPost,
path,
strings.NewReader(form.Encode()),
)
req.Header.Set(
"Content-Type", "application/x-www-form-urlencoded",
)
w := httptest.NewRecorder()
h.HandleLoginSubmit().ServeHTTP(w, req)
return w.Code
}
// TestVerificationCapacity_LogLineDoesNotTrackPathSize pins the cap
// on the "password verification capacity exhausted" WARN line.
//
// The one route that reaches it is chi's static "/pages/login", so no
// request through the mux can widen the line; the handler is driven
// directly here with the path a parameterised route would give it,
// which is what that cap exists for. Without this test, removing the
// logfield.Truncate there fails nothing.
func TestVerificationCapacity_LogLineDoesNotTrackPathSize(
t *testing.T,
) {
t.Parallel()
for handlerName, newHandler := range logHandlers() {
for fillName, fill := range escapeFills() {
t.Run(handlerName+"/"+fillName, func(t *testing.T) {
t.Parallel()
var mw *middleware.Middleware
h, buf := capturingHandlers(t, newHandler, &mw)
holdEveryVerificationSlot(t, mw)
assert.Equal(
t,
http.StatusServiceUnavailable,
postLoginAtPath(
t, h,
"/source/"+url.PathEscape(
oversizedFill(fill),
)+"/login",
),
)
lines := logLines(t, buf)
require.Len(t, lines, 1)
assertNoClientText(t, buf)
})
}
}
}
// assertBoundedFlood holds the whole flood's log output to what the
// stated per-line ceiling allows. The flood sent
// floodRequests * oversizedFillBytes bytes of client-chosen text;
// this is the assertion that the log did not grow with it.
func assertBoundedFlood(t *testing.T, got int) {
t.Helper()
sent := floodRequests * oversizedFillBytes
require.Less(
t, got, sent/2,
"log volume tracked the size of the flood's input",
)
require.LessOrEqual(
t, got,
floodRequests*middleware.MaxAccessLogLineBytes,
)
}

203
internal/handlers/profile.go
View File

@@ -1,66 +1,229 @@
package handlers
import (
"context"
"net/http"
"github.com/go-chi/chi"
"sneak.berlin/go/webhooker/internal/database"
)
// HandleProfile returns a handler for the user profile page
func (h *Handlers) HandleProfile() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
// Get username from URL
sessionUserID, sessionUsername, ok :=
h.profileOwnerOrDeny(w, r)
if !ok {
return
}
h.renderProfile(w, r, sessionUserID, sessionUsername, "", "")
}
}
// HandlePasswordChange returns a handler that lets an authenticated
// user change their own password. It is served by the CSRF- and
// auth-protected POST /password route under /user/{username}.
func (h *Handlers) HandlePasswordChange() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
sessionUserID, sessionUsername, ok :=
h.profileOwnerOrDeny(w, r)
if !ok {
return
}
// The body size cap is enforced by the MaxBodySize
// middleware, which runs before CSRF parses the form.
err := r.ParseForm()
if err != nil {
h.log.Error("failed to parse form", "error", err)
http.Error(w, "Bad request", http.StatusBadRequest)
return
}
successMessage, errorMessage, handled := h.applyPasswordChange(
r.Context(),
w,
sessionUsername,
// PostFormValue, not FormValue: the credential must
// come from the body, never from the query string.
r.PostFormValue("current_password"),
r.PostFormValue("new_password"),
r.PostFormValue("confirm_password"),
)
if !handled {
return
}
h.renderProfile(
w, r, sessionUserID, sessionUsername,
successMessage, errorMessage,
)
}
}
// applyPasswordChange verifies the current password and, on success,
// persists a fresh hash for the user, reusing the same helpers that
// bootstrap the admin user. It returns the success and error messages
// to display on the profile page. On an internal failure it writes a
// 500 response itself and returns handled=false, signalling the caller
// to stop without re-rendering the page.
func (h *Handlers) applyPasswordChange(
ctx context.Context,
w http.ResponseWriter,
username, currentPassword, newPassword, confirmPassword string,
) (string, string, bool) {
// This endpoint verifies one password and hashes another, at
// 64 MB each, so it takes a slot from the same bound the login
// endpoint uses. The bound is per hash, not per endpoint: leaving
// this path outside it would leave a hole in it. The slot is held
// across both hashes.
release, ok := h.mw.BeginPasswordVerification(ctx)
if !ok {
h.log.Warn("password verification capacity exhausted")
http.Error(
w,
"The server is busy verifying credentials. "+
"Please try again.",
http.StatusServiceUnavailable,
)
return "", "", false
}
defer release()
// Load the user row so we can verify the current password and
// persist the new hash.
var user database.User
err := h.db.DB().Where(
"username = ?", username,
).First(&user).Error
if err != nil {
h.serverError(
w, "failed to load user for password change", err,
)
return "", "", false
}
valid, err := database.VerifyPassword(
currentPassword, user.Password,
)
if err != nil {
h.serverError(w, "failed to verify password", err)
return "", "", false
}
if !valid {
return "", "Current password is incorrect.", true
}
if newPassword == "" {
return "", "New password must not be empty.", true
}
if newPassword != confirmPassword {
return "", "New password and confirmation do not match.", true
}
hashedPassword, err := database.HashPassword(newPassword)
if err != nil {
h.serverError(w, "failed to hash new password", err)
return "", "", false
}
err = h.db.DB().Model(&user).Update(
"password", hashedPassword,
).Error
if err != nil {
h.serverError(w, "failed to update password", err)
return "", "", false
}
h.log.Info("user changed password", "username", username)
return "Password changed successfully.", "", true
}
// profileOwnerOrDeny resolves the session identity and enforces that a
// user may only act on their own profile (the requested username in the
// URL must equal the session username). On any failure it writes the
// appropriate HTTP response and returns ok=false; callers must stop
// when ok is false.
func (h *Handlers) profileOwnerOrDeny(
w http.ResponseWriter,
r *http.Request,
) (string, string, bool) {
requestedUsername := chi.URLParam(r, "username")
if requestedUsername == "" {
http.NotFound(w, r)
return
return "", "", false
}
// Get session. RequireAuth middleware guarantees an
// authenticated session before this handler runs, so we
// only need to guard against an unexpected retrieval error.
// RequireAuth middleware guarantees an authenticated session
// before this handler runs, so we only need to guard against an
// unexpected retrieval error.
sess, err := h.session.Get(r)
if err != nil {
h.log.Error("failed to get session", "error", err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
h.serverError(w, "failed to get session", err)
return
return "", "", false
}
// Get user info from session
sessionUsername, ok := h.session.GetUsername(sess)
if !ok {
h.log.Error("authenticated session missing username")
http.Error(w, "Internal server error", http.StatusInternalServerError)
http.Error(
w, "Internal server error",
http.StatusInternalServerError,
)
return
return "", "", false
}
sessionUserID, ok := h.session.GetUserID(sess)
if !ok {
h.log.Error("authenticated session missing user ID")
http.Error(w, "Internal server error", http.StatusInternalServerError)
http.Error(
w, "Internal server error",
http.StatusInternalServerError,
)
return
return "", "", false
}
// For now, only allow users to view their own profile
// Only allow users to act on their own profile.
if requestedUsername != sessionUsername {
http.Error(w, "Forbidden", http.StatusForbidden)
return
return "", "", false
}
// Prepare data for template
return sessionUserID, sessionUsername, true
}
// renderProfile renders the profile page for the given user,
// optionally including a success or error message.
func (h *Handlers) renderProfile(
w http.ResponseWriter,
r *http.Request,
userID, username, successMessage, errorMessage string,
) {
data := map[string]any{
"User": &UserInfo{
ID: sessionUserID,
Username: sessionUsername,
ID: userID,
Username: username,
},
"SuccessMessage": successMessage,
"ErrorMessage": errorMessage,
}
// Render the profile page
h.renderTemplate(w, r, "profile.html", data)
}
}

134
internal/handlers/profile_test.go
View File

@@ -4,12 +4,15 @@ import (
"context"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/logger"
"sneak.berlin/go/webhooker/internal/middleware"
@@ -157,3 +160,134 @@ func TestUserRoute_Unauthenticated_RedirectedByMiddleware(t *testing.T) {
assert.Equal(t, http.StatusSeeOther, w.Code)
assert.Equal(t, "/pages/login", w.Header().Get("Location"))
}
// passwordChangeRequest builds a POST request to the password-change
// endpoint for the given username, attaching the supplied cookies, an
// urlencoded form body, and the chi URL parameter the handler reads.
func passwordChangeRequest(
username string,
cookies []*http.Cookie,
form url.Values,
) *http.Request {
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost,
"/user/"+username+"/password",
strings.NewReader(form.Encode()),
)
req.Header.Set(
"Content-Type", "application/x-www-form-urlencoded",
)
for _, c := range cookies {
req.AddCookie(c)
}
rctx := chi.NewRouteContext()
rctx.URLParams.Add("username", username)
return req.WithContext(
context.WithValue(req.Context(), chi.RouteCtxKey, rctx),
)
}
func TestHandlePasswordChange_Success(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
var sess *session.Session
var db *database.Database
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
oldHash, err := database.HashPassword("oldpassword")
require.NoError(t, err)
user := &database.User{Username: "pwuser", Password: oldHash}
require.NoError(t, db.DB().Create(user).Error)
cookies := authenticatedCookies(t, sess, user.ID, "pwuser")
form := url.Values{}
form.Set("current_password", "oldpassword")
form.Set("new_password", "newpassword")
form.Set("confirm_password", "newpassword")
req := passwordChangeRequest("pwuser", cookies, form)
w := httptest.NewRecorder()
h.HandlePasswordChange().ServeHTTP(w, req)
assert.Equal(t, http.StatusOK, w.Code)
assert.Contains(
t, w.Body.String(), "Password changed successfully.",
)
var updated database.User
require.NoError(t,
db.DB().Where("username = ?", "pwuser").First(&updated).Error,
)
assert.NotEqual(t, oldHash, updated.Password)
valid, err := database.VerifyPassword(
"newpassword", updated.Password,
)
require.NoError(t, err)
assert.True(t, valid, "new password should verify against new hash")
}
func TestHandlePasswordChange_WrongCurrentPassword(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
var sess *session.Session
var db *database.Database
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
oldHash, err := database.HashPassword("oldpassword")
require.NoError(t, err)
user := &database.User{Username: "pwuser2", Password: oldHash}
require.NoError(t, db.DB().Create(user).Error)
cookies := authenticatedCookies(t, sess, user.ID, "pwuser2")
form := url.Values{}
form.Set("current_password", "wrongpassword")
form.Set("new_password", "newpassword")
form.Set("confirm_password", "newpassword")
req := passwordChangeRequest("pwuser2", cookies, form)
w := httptest.NewRecorder()
h.HandlePasswordChange().ServeHTTP(w, req)
assert.Equal(t, http.StatusOK, w.Code)
assert.Contains(
t, w.Body.String(), "Current password is incorrect.",
)
var unchanged database.User
require.NoError(t,
db.DB().Where(
"username = ?", "pwuser2",
).First(&unchanged).Error,
)
assert.Equal(
t, oldHash, unchanged.Password,
"stored hash must be unchanged after a rejected change",
)
}

356
internal/handlers/source_delete_test.go Normal file
View File

@@ -0,0 +1,356 @@
package handlers_test
import (
"context"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
const (
deleteTestUserID = "test-user-id"
deleteTestUsername = "testuser"
// paramSourceID and paramTargetID are the chi URL parameter
// names the deletion handlers read.
paramSourceID = "sourceID"
paramTargetID = "targetID"
)
// seedWebhook inserts a webhook owned by the test user and
// returns it.
func seedWebhook(
t *testing.T,
db *database.Database,
) *database.Webhook {
t.Helper()
wh := &database.Webhook{
UserID: deleteTestUserID,
Name: "delete-me",
}
require.NoError(
t,
db.DB().Omit(clause.Associations).Create(wh).Error,
)
return wh
}
// seedTarget inserts a target of the given type for a webhook
// and returns it.
func seedTarget(
t *testing.T,
db *database.Database,
webhookID string,
targetType database.TargetType,
) *database.Target {
t.Helper()
tgt := &database.Target{
WebhookID: webhookID,
Name: "t-" + string(targetType),
Type: targetType,
Active: true,
}
require.NoError(
t,
db.DB().Omit(clause.Associations).Create(tgt).Error,
)
return tgt
}
// archivePathFor returns the archive database path the
// delivery engine would use for a webhook: beside the webhook's
// event database in the data directory.
func archivePathFor(
t *testing.T,
mgr *database.WebhookDBManager,
webhookID string,
) string {
t.Helper()
return filepath.Join(
filepath.Dir(mgr.DBPath(webhookID)),
"archive-"+webhookID+".db",
)
}
// writeArchivePlaceholder creates a stand-in archive file so a
// test can assert the file survives webhook deletion.
func writeArchivePlaceholder(path string) error {
return os.WriteFile(path, []byte("archive"), 0o600)
}
// postRequest builds an authenticated POST request carrying the
// given chi URL parameters.
func postRequest(
path string,
cookies []*http.Cookie,
params map[string]string,
) *http.Request {
req := httptest.NewRequestWithContext(
context.Background(), http.MethodPost, path, nil,
)
for _, c := range cookies {
req.AddCookie(c)
}
rctx := chi.NewRouteContext()
for k, v := range params {
rctx.URLParams.Add(k, v)
}
return req.WithContext(
context.WithValue(req.Context(), chi.RouteCtxKey, rctx),
)
}
// TestHandleSourceDelete_EvictsArchiveWriter proves that
// deleting a webhook reaches the delivery engine and releases
// the webhook's archive writer, exercised through the real
// deletion handler rather than by calling the evictor directly.
func TestHandleSourceDelete_EvictsArchiveWriter(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
ev *recordingEvictor
)
app := newTestApp(t, &h, &sess, &db, &ev)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedTarget(t, db, wh.ID, database.TargetTypeDatabase)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/delete",
cookies,
map[string]string{paramSourceID: wh.ID},
)
w := httptest.NewRecorder()
h.HandleSourceDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Equal(
t, []string{wh.ID}, ev.Evicted(),
"deleting a webhook should evict its archive writer",
)
}
// TestHandleSourceDelete_KeepsArchiveFile proves that deleting
// a webhook does not remove its archive database file: the
// archive is long-term storage the operator owns.
func TestHandleSourceDelete_KeepsArchiveFile(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
mgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &mgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
// Place an archive file where the delivery engine would.
archivePath := archivePathFor(t, mgr, wh.ID)
require.NoError(
t,
writeArchivePlaceholder(archivePath),
)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/delete",
cookies,
map[string]string{paramSourceID: wh.ID},
)
w := httptest.NewRecorder()
h.HandleSourceDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
assert.FileExists(
t, archivePath,
"webhook deletion must not destroy the archive file",
)
}
// TestHandleTargetDelete_EvictsWhenLastDatabaseTargetGone
// proves that removing the last database target releases the
// archive writer.
func TestHandleTargetDelete_EvictsWhenLastDatabaseTargetGone(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
ev *recordingEvictor
)
app := newTestApp(t, &h, &sess, &db, &ev)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
tgt := seedTarget(
t, db, wh.ID, database.TargetTypeDatabase,
)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/targets/"+tgt.ID+"/delete",
cookies,
map[string]string{
paramSourceID: wh.ID,
paramTargetID: tgt.ID,
},
)
w := httptest.NewRecorder()
h.HandleTargetDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Equal(
t, []string{wh.ID}, ev.Evicted(),
"removing the last database target should evict",
)
}
// TestHandleTargetDelete_KeepsWriterWhenDatabaseTargetRemains
// proves that deleting one of several database targets leaves
// the still-needed archive writer alone: the surviving target
// keeps archiving to the same file, so the writer must stay.
func TestHandleTargetDelete_KeepsWriterWhenDatabaseTargetRemains(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
ev *recordingEvictor
)
app := newTestApp(t, &h, &sess, &db, &ev)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
doomed := seedTarget(
t, db, wh.ID, database.TargetTypeDatabase,
)
seedTarget(t, db, wh.ID, database.TargetTypeDatabase)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/targets/"+doomed.ID+"/delete",
cookies,
map[string]string{
paramSourceID: wh.ID,
paramTargetID: doomed.ID,
},
)
w := httptest.NewRecorder()
h.HandleTargetDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Empty(
t, ev.Evicted(),
"a second database target still needs the writer",
)
}
// TestHandleTargetDelete_KeepsWriterWhenOtherTypeDeleted proves
// that deleting a target of an unrelated type leaves a
// still-needed archive writer alone: the webhook's database
// target is untouched, so its writer must stay.
func TestHandleTargetDelete_KeepsWriterWhenOtherTypeDeleted(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
ev *recordingEvictor
)
app := newTestApp(t, &h, &sess, &db, &ev)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedTarget(t, db, wh.ID, database.TargetTypeDatabase)
other := seedTarget(t, db, wh.ID, database.TargetTypeLog)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/targets/"+other.ID+"/delete",
cookies,
map[string]string{
paramSourceID: wh.ID,
paramTargetID: other.ID,
},
)
w := httptest.NewRecorder()
h.HandleTargetDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Empty(
t, ev.Evicted(),
"a surviving database target must keep its writer",
)
}

228
internal/handlers/source_detail_test.go Normal file
View File

@@ -0,0 +1,228 @@
package handlers_test
import (
"context"
"net/http"
"net/http/httptest"
"testing"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
// The secret path segments of a Slack incoming webhook URL.
// Holding them is enough to post to the channel forever, so
// they must never reach the rendered page.
const (
slackSecretPath = "/services/T00000000/B00000000/" +
"XXXXXXXXXXXXXXXXXXXXXXXX"
slackWebhookURL = "https://hooks.slack.com" +
slackSecretPath
)
// seedConfiguredTarget inserts a target with a stored config
// blob and returns it.
func seedConfiguredTarget(
t *testing.T,
db *database.Database,
webhookID string,
targetType database.TargetType,
config string,
) *database.Target {
t.Helper()
tgt := &database.Target{
WebhookID: webhookID,
Name: "t-" + string(targetType),
Type: targetType,
Active: true,
Config: config,
}
require.NoError(
t,
db.DB().Omit(clause.Associations).Create(tgt).Error,
)
return tgt
}
// renderSourceDetailPage runs the real source detail handler
// for a webhook and returns the rendered HTML.
func renderSourceDetailPage(
t *testing.T,
h *handlers.Handlers,
sess *session.Session,
webhookID string,
) string {
t.Helper()
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet,
"/source/"+webhookID,
nil,
)
for _, c := range authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
) {
req.AddCookie(c)
}
rctx := chi.NewRouteContext()
rctx.URLParams.Add(paramSourceID, webhookID)
req = req.WithContext(
context.WithValue(
req.Context(), chi.RouteCtxKey, rctx,
),
)
w := httptest.NewRecorder()
h.HandleSourceDetail().ServeHTTP(w, req)
require.Equal(t, http.StatusOK, w.Code)
return w.Body.String()
}
// TestHandleSourceDetail_MasksSlackWebhookURL is the
// load-bearing regression test for the credential leak: the
// rendered page must show the Slack target without any of the
// secret path segments of its webhook URL.
func TestHandleSourceDetail_MasksSlackWebhookURL(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
)
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedConfiguredTarget(
t, db, wh.ID,
database.TargetTypeSlack,
`{"webhookUrl":"`+slackWebhookURL+`"}`,
)
body := renderSourceDetailPage(t, h, sess, wh.ID)
assert.NotContains(t, body, slackSecretPath)
assert.NotContains(t, body, "T00000000")
assert.NotContains(t, body, "B00000000")
assert.NotContains(
t, body, "XXXXXXXXXXXXXXXXXXXXXXXX",
)
assert.NotContains(t, body, "webhookUrl")
assert.Contains(t, body, "Webhook URL")
assert.Contains(t, body, "https://hooks.slack.com/...")
}
// TestHandleSourceDetail_MasksHTTPDestinationURL is the
// regression test for the same leak reached through the http
// target: its destination is routinely an incoming-webhook
// endpoint whose path segments are the credential, so the
// rendered page must not contain them.
func TestHandleSourceDetail_MasksHTTPDestinationURL(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
)
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedConfiguredTarget(
t, db, wh.ID,
database.TargetTypeHTTP,
`{"url":"`+slackWebhookURL+`"}`,
)
body := renderSourceDetailPage(t, h, sess, wh.ID)
assert.NotContains(t, body, slackSecretPath)
assert.NotContains(t, body, "T00000000")
assert.NotContains(t, body, "B00000000")
assert.NotContains(
t, body, "XXXXXXXXXXXXXXXXXXXXXXXX",
)
assert.Contains(t, body, "Destination URL")
assert.Contains(t, body, "https://hooks.slack.com/...")
}
// TestHandleSourceDetail_RendersNamedTargetFields proves the
// other target types render labelled fields rather than the
// stored blob.
func TestHandleSourceDetail_RendersNamedTargetFields(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
)
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedConfiguredTarget(
t, db, wh.ID,
database.TargetTypeHTTP,
`{"url":"https://example.com/hook","timeout":30,`+
`"headers":{"Authorization":"Bearer sekrit"}}`,
)
seedConfiguredTarget(
t, db, wh.ID,
database.TargetTypeDatabase,
`{"expiry":"720h"}`,
)
seedConfiguredTarget(
t, db, wh.ID,
database.TargetType("carrier-pigeon"),
`{"beak":"sharp"}`,
)
body := renderSourceDetailPage(t, h, sess, wh.ID)
assert.Contains(t, body, "Destination URL")
assert.Contains(t, body, "https://example.com/...")
assert.Contains(t, body, "Timeout")
assert.Contains(t, body, "1 configured")
assert.NotContains(t, body, "sekrit")
assert.Contains(t, body, "Archive Expiry")
assert.Contains(t, body, "720h")
// An unknown type gets the neutral placeholder, never the
// stored blob.
assert.Contains(t, body, "(unavailable)")
assert.NotContains(t, body, "beak")
}

134
internal/handlers/source_logs_test.go Normal file
View File

@@ -0,0 +1,134 @@
package handlers_test
import (
"context"
"net/http"
"net/http/httptest"
"testing"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
// seedDeliveredEvent records an event and a delivery for it in
// the webhook's own database, so the log page has a delivery
// to render against the target.
func seedDeliveredEvent(
t *testing.T,
dbMgr *database.WebhookDBManager,
webhookID, targetID string,
) {
t.Helper()
webhookDB, err := dbMgr.GetDB(webhookID)
require.NoError(t, err)
event := &database.Event{
WebhookID: webhookID,
Method: http.MethodPost,
Body: `{"test":true}`,
ContentType: "application/json",
}
require.NoError(t, webhookDB.Omit(
clause.Associations,
).Create(event).Error)
dlv := &database.Delivery{
EventID: event.ID,
TargetID: targetID,
Status: database.DeliveryStatusDelivered,
}
require.NoError(t, webhookDB.Omit(
clause.Associations,
).Create(dlv).Error)
}
// renderSourceLogsPage runs the real event log handler for a
// webhook and returns the rendered HTML.
func renderSourceLogsPage(
t *testing.T,
h *handlers.Handlers,
sess *session.Session,
webhookID string,
) string {
t.Helper()
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet,
"/source/"+webhookID+"/logs",
nil,
)
for _, c := range authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
) {
req.AddCookie(c)
}
rctx := chi.NewRouteContext()
rctx.URLParams.Add(paramSourceID, webhookID)
req = req.WithContext(
context.WithValue(
req.Context(), chi.RouteCtxKey, rctx,
),
)
w := httptest.NewRecorder()
h.HandleSourceLogs().ServeHTTP(w, req)
require.Equal(t, http.StatusOK, w.Code)
return w.Body.String()
}
// TestHandleSourceLogs_MasksSlackWebhookURL proves the event
// log page is handed a display-safe projection of each target
// rather than the stored row, so the credential cannot be
// rendered from its template data.
func TestHandleSourceLogs_MasksSlackWebhookURL(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
tgt := seedConfiguredTarget(
t, db, wh.ID,
database.TargetTypeSlack,
`{"webhookUrl":"`+slackWebhookURL+`"}`,
)
seedDeliveredEvent(t, dbMgr, wh.ID, tgt.ID)
body := renderSourceLogsPage(t, h, sess, wh.ID)
assert.NotContains(t, body, slackSecretPath)
assert.NotContains(t, body, "T00000000")
assert.NotContains(t, body, "B00000000")
assert.NotContains(
t, body, "XXXXXXXXXXXXXXXXXXXXXXXX",
)
assert.NotContains(t, body, "webhookUrl")
// The page still identifies the delivery's target.
assert.Contains(t, body, tgt.Name)
assert.Contains(t, body, "delivered")
}

522
internal/handlers/source_management.go
View File

@@ -5,6 +5,7 @@ import (
"errors"
"net/http"
"strconv"
"strings"
"github.com/go-chi/chi"
"github.com/google/uuid"
@@ -24,11 +25,81 @@ type WebhookListItem struct {
// errMissingURL signals that a required URL was not provided.
var errMissingURL = errors.New("missing URL")
// EventWithDeliveries holds an event and its deliveries.
type EventWithDeliveries struct {
database.Event
// errInvalidRetention signals a retention_days form value that is not
// a non-negative whole number.
var errInvalidRetention = errors.New("invalid retention days")
Deliveries []database.Delivery
// errRetentionTooLarge signals a retention_days form value that is a
// whole number but larger than the reaper's cutoff arithmetic can
// represent. It is distinguished from errInvalidRetention so the form
// can tell the user the actual ceiling instead of implying their input
// was not a number.
var errRetentionTooLarge = errors.New("retention days out of range")
// retentionErrorMessage returns the message the create and edit forms
// show the user for a rejected retention_days value. Any error other
// than errRetentionTooLarge falls back to the generic wording, so an
// unrecognised parse failure still produces a sensible 400 rather than
// an empty alert.
func retentionErrorMessage(err error) string {
if errors.Is(err, errRetentionTooLarge) {
return "Retention must be at most " +
strconv.Itoa(database.MaxFiniteRetentionDays) +
" days, or 0 to retain events forever."
}
return "Retention must be a whole number of days, or 0 to " +
"retain events forever."
}
// parseRetentionDays interprets a retention_days form value.
//
// An empty value yields fallback, which lets the create path apply the
// default and the edit path leave the stored value unchanged. A value
// of 0 is returned as 0 and is rewritten to the retain-forever
// sentinel by database.Webhook's BeforeSave hook. Anything unparseable
// or negative is an error rather than a silently substituted default.
//
// The upper bound is not cosmetic. The reaper computes its cutoff as a
// time.Duration, an int64 nanosecond count, so a day count above
// database.MaxFiniteRetentionDays overflows, puts the cutoff in the
// future, and deletes every event the webhook has. A finite value
// above that ceiling is therefore a 400.
//
// A value at or above the retain-forever sentinel is not out of range:
// it is what the edit form pre-fills for a retain-forever webhook, so
// submitting the form back unchanged has to keep meaning "forever"
// rather than being rejected.
func parseRetentionDays(raw string, fallback int) (int, error) {
raw = strings.TrimSpace(raw)
if raw == "" {
return fallback, nil
}
v, err := strconv.Atoi(raw)
if err != nil || v < 0 {
return 0, errInvalidRetention
}
if v >= database.RetentionForeverDays {
return database.RetentionForeverDays, nil
}
if v > database.MaxFiniteRetentionDays {
return 0, errRetentionTooLarge
}
return v, nil
}
// DeliveryView is the display-safe projection of a delivery
// for the event log page. Its target is a TargetView, so the
// stored configuration blob — which holds the target's
// credential — has no path to the template.
type DeliveryView struct {
ID string
Status database.DeliveryStatus
Target delivery.TargetView
}
// HandleSourceList shows a list of user's webhooks.
@@ -105,11 +176,30 @@ func (h *Handlers) buildWebhookListItems(
// HandleSourceCreate shows the form to create a new webhook.
func (h *Handlers) HandleSourceCreate() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
data := map[string]any{
"Error": "",
h.renderTemplate(
w, r, "sources_new.html",
newSourceFormData("", "", ""),
)
}
}
h.renderTemplate(w, r, "sources_new.html", data)
// newSourceFormData builds the template data for the webhook creation
// form.
//
// It carries the retention default so the pre-filled value comes from
// database.DefaultRetentionDays rather than being a third hardcoded
// copy of the same policy, and it carries the submitted name and
// description so that re-rendering the form after a validation failure
// gives the user their input back instead of a blank form. The edit
// form already behaves that way; create now matches it.
func newSourceFormData(
errMsg, name, description string,
) map[string]any {
return map[string]any{
tmplKeyError: errMsg,
"Name": name,
"Description": description,
"DefaultRetentionDays": database.DefaultRetentionDays,
}
}
@@ -126,10 +216,8 @@ func (h *Handlers) HandleSourceCreateSubmit() http.HandlerFunc {
return
}
r.Body = http.MaxBytesReader(
w, r.Body, 1<<maxBodyShift,
)
// The body size cap is enforced by the MaxBodySize
// middleware, which runs before CSRF parses the form.
err := r.ParseForm()
if err != nil {
http.Error(
@@ -139,28 +227,36 @@ func (h *Handlers) HandleSourceCreateSubmit() http.HandlerFunc {
return
}
name := r.FormValue("name")
description := r.FormValue("description")
retentionStr := r.FormValue("retention_days")
name := r.PostFormValue("name")
description := r.PostFormValue("description")
retentionStr := r.PostFormValue("retention_days")
if name == "" {
data := map[string]any{
"Error": "Name is required",
}
w.WriteHeader(http.StatusBadRequest)
h.renderTemplate(w, r, "sources_new.html", data)
h.renderTemplate(
w, r, "sources_new.html",
newSourceFormData(
"Name is required", name, description,
),
)
return
}
retentionDays := defaultRetentionDays
retentionDays, retErr := parseRetentionDays(
retentionStr, database.DefaultRetentionDays,
)
if retErr != nil {
w.WriteHeader(http.StatusBadRequest)
h.renderTemplate(
w, r, "sources_new.html",
newSourceFormData(
retentionErrorMessage(retErr),
name, description,
),
)
if retentionStr != "" {
v, convErr := strconv.Atoi(retentionStr)
if convErr == nil && v > 0 {
retentionDays = v
}
return
}
h.createWebhookWithEntrypoint(
@@ -314,10 +410,15 @@ func (h *Handlers) renderSourceDetail(
scheme = fwdProto
}
// The template calls Webhook methods, which take pointer
// receivers; html/template cannot address a value stored in a map.
data := map[string]any{
"Webhook": webhook,
tmplKeyWebhook: &webhook,
"Entrypoints": entrypoints,
"Targets": targets,
// Targets are projected to a display-safe view: the
// stored config blob holds credentials and must never
// reach a template.
"Targets": delivery.NewTargetViews(targets),
"Events": events,
"BaseURL": scheme + "://" + host,
}
@@ -351,8 +452,8 @@ func (h *Handlers) HandleSourceEdit() http.HandlerFunc {
}
data := map[string]any{
"Webhook": webhook,
"Error": "",
tmplKeyWebhook: &webhook,
tmplKeyError: "",
}
h.renderTemplate(w, r, "source_edit.html", data)
@@ -385,10 +486,8 @@ func (h *Handlers) HandleSourceEditSubmit() http.HandlerFunc {
return
}
r.Body = http.MaxBytesReader(
w, r.Body, 1<<maxBodyShift,
)
// The body size cap is enforced by the MaxBodySize
// middleware, which runs before CSRF parses the form.
err = r.ParseForm()
if err != nil {
http.Error(
@@ -408,15 +507,13 @@ func (h *Handlers) applyWebhookEdit(
r *http.Request,
webhook *database.Webhook,
) {
r.Body = http.MaxBytesReader(
w, r.Body, 1<<maxBodyShift,
)
name := r.FormValue("name")
// The body size cap is enforced by the MaxBodySize middleware,
// which runs before CSRF parses the form.
name := r.PostFormValue("name")
if name == "" {
data := map[string]any{
"Webhook": *webhook,
"Error": "Name is required",
tmplKeyWebhook: webhook,
tmplKeyError: "Name is required",
}
w.WriteHeader(http.StatusBadRequest)
@@ -426,8 +523,26 @@ func (h *Handlers) applyWebhookEdit(
}
webhook.Name = name
webhook.Description = r.FormValue("description")
h.parseRetention(r, webhook)
webhook.Description = r.PostFormValue("description")
// An empty field falls back to the stored value, so submitting the
// form without touching retention leaves the policy alone.
retentionDays, retErr := parseRetentionDays(
r.PostFormValue("retention_days"), webhook.RetentionDays,
)
if retErr != nil {
data := map[string]any{
tmplKeyWebhook: webhook,
tmplKeyError: retentionErrorMessage(retErr),
}
w.WriteHeader(http.StatusBadRequest)
h.renderTemplate(w, r, "source_edit.html", data)
return
}
webhook.RetentionDays = retentionDays
err := h.db.DB().Save(webhook).Error
if err != nil {
@@ -441,23 +556,6 @@ func (h *Handlers) applyWebhookEdit(
)
}
// parseRetention parses and applies retention_days from the
// form.
func (h *Handlers) parseRetention(
r *http.Request,
webhook *database.Webhook,
) {
retStr := r.FormValue("retention_days")
if retStr == "" {
return
}
v, err := strconv.Atoi(retStr)
if err == nil && v > 0 {
webhook.RetentionDays = v
}
}
// HandleSourceDelete handles webhook deletion.
func (h *Handlers) HandleSourceDelete() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
@@ -532,6 +630,13 @@ func (h *Handlers) deleteWebhookResources(
return
}
// Release the delivery engine's per-webhook archiving state
// so a deleted webhook's archive writer (and any handle open
// within its debounce window) does not linger for the
// process lifetime. The archive file itself is deliberately
// left on disk; see evictArchiveWriter.
h.evictArchiveWriter(webhook.ID)
err = h.dbMgr.DeleteDB(webhook.ID)
if err != nil {
h.log.Error(
@@ -550,29 +655,113 @@ func (h *Handlers) deleteWebhookResources(
http.Redirect(w, r, "/sources", http.StatusSeeOther)
}
// HandleSourceLogs shows the request/response logs for a
// webhook.
func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
// evictArchiveWriter asks the delivery engine to drop its
// cached archive writer for a webhook, closing the archive file
// handle.
//
// The archive database file is NOT deleted. Unlike the event
// database — which is per-webhook working storage and is
// hard-deleted with the webhook — an archive is explicitly
// long-term storage that an operator may want to keep or move
// away for offline retention. Destroying it as a side effect of
// deleting a webhook would be a surprising and unrecoverable
// data loss, so the file is left for the operator to handle.
func (h *Handlers) evictArchiveWriter(webhookID string) {
if h.evictor == nil {
return
}
h.evictor.EvictWebhook(webhookID)
}
// evictArchiveWriterIfUnused releases a webhook's archive
// writer once the webhook has no database target left to feed
// it.
//
// It is called after any child resource of a webhook is
// deleted, and is correct without knowing which kind was: it
// evicts only when no database target remains, so deleting one
// of several database targets — or deleting an unrelated
// target type — leaves a still-needed writer alone. When no
// database target ever existed there is no writer and eviction
// is a no-op. Soft-deleted targets are excluded by GORM's
// default scope, so the row just deleted is not counted.
func (h *Handlers) evictArchiveWriterIfUnused(webhookID string) {
var remaining int64
err := h.db.DB().
Model(&database.Target{}).
Where(
"webhook_id = ? AND type = ?",
webhookID, database.TargetTypeDatabase,
).
Count(&remaining).Error
if err != nil {
h.log.Error(
"failed to count remaining database targets",
"webhook_id", webhookID,
"error", err,
)
return
}
if remaining > 0 {
return
}
h.evictArchiveWriter(webhookID)
}
// ownedWebhook resolves the request's sourceID parameter to a
// webhook the session's user owns.
//
// Ownership and existence are decided by one query, so a
// webhook belonging to another user is indistinguishable from
// one that does not exist: both are a 404, and neither confirms
// the id. Callers that reach further into a webhook's data —
// the event log page and the event body download — share this
// one check rather than restating it, so the download cannot
// come to authorize differently from the page that links to it.
//
// It reports false once it has written the response, which is a
// redirect to the login page for an unauthenticated request and
// a 404 otherwise. The caller returns without writing more.
func (h *Handlers) ownedWebhook(
w http.ResponseWriter,
r *http.Request,
) (database.Webhook, bool) {
var webhook database.Webhook
userID, ok := h.getUserID(r)
if !ok {
http.Redirect(
w, r, "/pages/login", http.StatusSeeOther,
)
return
return database.Webhook{}, false
}
sourceID := chi.URLParam(r, "sourceID")
var webhook database.Webhook
err := h.db.DB().Where(
"id = ? AND user_id = ?", sourceID, userID,
).First(&webhook).Error
if err != nil {
http.NotFound(w, r)
return database.Webhook{}, false
}
return webhook, true
}
// HandleSourceLogs shows the request/response logs for a
// webhook.
func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
webhook, ok := h.ownedWebhook(w, r)
if !ok {
return
}
@@ -589,7 +778,7 @@ func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
}
data := map[string]any{
"Webhook": webhook,
tmplKeyWebhook: &webhook,
"Events": evts,
"Page": page,
"TotalPages": totalPages,
@@ -604,22 +793,27 @@ func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
}
}
// loadTargetMap loads targets into a map keyed by target ID.
// loadTargetMap loads targets into a map of display-safe
// views keyed by target ID. The projection happens here so
// that no caller can hand a raw target, configuration blob
// and all, to a template.
func (h *Handlers) loadTargetMap(
webhookID string,
) map[string]database.Target {
) map[string]delivery.TargetView {
var targets []database.Target
h.db.DB().Where(
"webhook_id = ?", webhookID,
).Find(&targets)
views := delivery.NewTargetViews(targets)
targetMap := make(
map[string]database.Target, len(targets),
map[string]delivery.TargetView, len(views),
)
for _, t := range targets {
targetMap[t.ID] = t
for _, v := range views {
targetMap[v.ID] = v
}
return targetMap
@@ -640,16 +834,18 @@ func (h *Handlers) parsePage(r *http.Request) int {
}
// loadEventsWithDeliveries loads paginated events and their
// deliveries from the per-webhook database.
// deliveries from the per-webhook database. Events come back
// as capped projections rather than database.Event rows: see
// eventLogColumns for why the cut happens in SQL.
func (h *Handlers) loadEventsWithDeliveries(
w http.ResponseWriter,
webhook database.Webhook,
targetMap map[string]database.Target,
targetMap map[string]delivery.TargetView,
page int,
) ([]EventWithDeliveries, int64) {
) ([]EventLogView, int64) {
var totalEvents int64
var result []EventWithDeliveries
var result []EventLogView
if !h.dbMgr.DBExists(webhook.ID) {
return result, totalEvents
@@ -670,35 +866,54 @@ func (h *Handlers) loadEventsWithDeliveries(
offset := (page - 1) * paginationPerPage
var events []database.Event
var rows []eventLogRow
webhookDB.Where(
webhookDB.Model(&database.Event{}).Select(
eventLogColumns, maxRenderedBodyBytes,
).Where(
"webhook_id = ?", webhook.ID,
).Order("created_at DESC").Offset(offset).Limit(
paginationPerPage,
).Find(&events)
).Find(&rows)
result = make([]EventWithDeliveries, len(events))
result = make([]EventLogView, len(rows))
for i := range events {
result[i].Event = events[i]
for i := range rows {
result[i] = rows[i].view()
var deliveries []database.Delivery
webhookDB.Where(
"event_id = ?", events[i].ID,
).Find(&result[i].Deliveries)
"event_id = ?", rows[i].ID,
).Find(&deliveries)
for j := range result[i].Deliveries {
tid := result[i].Deliveries[j].TargetID
if target, ok := targetMap[tid]; ok {
result[i].Deliveries[j].Target = target
}
}
result[i].Deliveries = newDeliveryViews(
deliveries, targetMap,
)
}
return result, totalEvents
}
// newDeliveryViews projects deliveries for rendering,
// resolving each one's target to its display-safe view.
func newDeliveryViews(
deliveries []database.Delivery,
targetMap map[string]delivery.TargetView,
) []DeliveryView {
views := make([]DeliveryView, len(deliveries))
for i := range deliveries {
views[i] = DeliveryView{
ID: deliveries[i].ID,
Status: deliveries[i].Status,
Target: targetMap[deliveries[i].TargetID],
}
}
return views
}
// HandleEntrypointCreate handles adding a new entrypoint.
func (h *Handlers) HandleEntrypointCreate() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
@@ -724,10 +939,8 @@ func (h *Handlers) HandleEntrypointCreate() http.HandlerFunc {
return
}
r.Body = http.MaxBytesReader(
w, r.Body, 1<<maxBodyShift,
)
// The body size cap is enforced by the MaxBodySize
// middleware, which runs before CSRF parses the form.
err = r.ParseForm()
if err != nil {
http.Error(
@@ -737,7 +950,7 @@ func (h *Handlers) HandleEntrypointCreate() http.HandlerFunc {
return
}
description := r.FormValue("description")
description := r.PostFormValue("description")
entrypoint := &database.Entrypoint{
WebhookID: webhook.ID,
@@ -784,10 +997,8 @@ func (h *Handlers) HandleTargetCreate() http.HandlerFunc {
return
}
r.Body = http.MaxBytesReader(
w, r.Body, 1<<maxBodyShift,
)
// The body size cap is enforced by the MaxBodySize
// middleware, which runs before CSRF parses the form.
err = r.ParseForm()
if err != nil {
http.Error(
@@ -807,14 +1018,20 @@ func (h *Handlers) processTargetCreate(
r *http.Request,
webhook database.Webhook,
) {
r.Body = http.MaxBytesReader(
w, r.Body, 1<<maxBodyShift,
)
name := r.FormValue("name")
targetType := database.TargetType(r.FormValue("type"))
targetURL := r.FormValue("url")
maxRetriesStr := r.FormValue("max_retries")
// The body size cap is enforced by the MaxBodySize middleware,
// which runs before CSRF parses the form.
//
// Every field here is read with PostFormValue, not FormValue.
// FormValue falls back to the query string, which would let
// `POST /source/{id}/targets?url=https://hooks.slack.com/...`
// configure a target from a value the request line carries — and
// the request line, unlike the body, is what logs, proxies,
// Referer headers and error trackers record.
name := r.PostFormValue("name")
targetType := database.TargetType(r.PostFormValue("type"))
targetURL := r.PostFormValue("url")
maxRetriesStr := r.PostFormValue("max_retries")
expiry := r.PostFormValue("expiry")
if name == "" {
http.Error(
@@ -834,7 +1051,7 @@ func (h *Handlers) processTargetCreate(
}
configJSON, err := h.buildTargetConfig(
w, r, targetType, targetURL,
w, r, targetType, targetURL, expiry,
)
if err != nil {
return
@@ -892,18 +1109,28 @@ func parseNonNegativeInt(s string) int {
}
// buildTargetConfig builds the JSON config string for a target.
// The expiry form value is read by the caller (which bounds the
// request body) and applies to database targets only.
func (h *Handlers) buildTargetConfig(
w http.ResponseWriter,
r *http.Request,
targetType database.TargetType,
targetURL string,
targetURL, expiry string,
) (string, error) {
switch targetType {
case database.TargetTypeHTTP:
return h.buildHTTPTargetConfig(w, r, targetURL)
return h.buildURLTargetConfig(
w, r, targetURL, "url",
"URL is required for HTTP targets",
)
case database.TargetTypeSlack:
return h.buildSlackTargetConfig(w, r, targetURL)
case database.TargetTypeDatabase, database.TargetTypeLog:
return h.buildURLTargetConfig(
w, r, targetURL, "webhookUrl",
"Webhook URL is required for Slack targets",
)
case database.TargetTypeDatabase:
return h.buildDatabaseTargetConfig(w, expiry)
case database.TargetTypeLog:
return "", nil
default:
http.Error(
@@ -915,16 +1142,18 @@ func (h *Handlers) buildTargetConfig(
}
}
// buildHTTPTargetConfig builds config JSON for an HTTP target.
func (h *Handlers) buildHTTPTargetConfig(
// buildURLTargetConfig builds config JSON for a target whose
// configuration is a single SSRF-validated URL stored under
// configKey. missingMsg is the error shown when no URL is given.
func (h *Handlers) buildURLTargetConfig(
w http.ResponseWriter,
r *http.Request,
targetURL string,
targetURL, configKey, missingMsg string,
) (string, error) {
if targetURL == "" {
http.Error(
w,
"URL is required for HTTP targets",
missingMsg,
http.StatusBadRequest,
)
@@ -935,9 +1164,12 @@ func (h *Handlers) buildHTTPTargetConfig(
r.Context(), targetURL,
)
if err != nil {
// The submitted URL can be a credential (a Slack
// incoming webhook URL is a bearer token), so the log
// records only its scheme and host.
h.log.Warn(
"target URL blocked by SSRF protection",
"url", targetURL,
"url", delivery.MaskURL(targetURL),
"error", err,
)
http.Error(
@@ -949,7 +1181,7 @@ func (h *Handlers) buildHTTPTargetConfig(
return "", err
}
cfg := map[string]any{"url": targetURL}
cfg := map[string]any{configKey: targetURL}
configBytes, err := json.Marshal(cfg)
if err != nil {
@@ -964,41 +1196,33 @@ func (h *Handlers) buildHTTPTargetConfig(
return string(configBytes), nil
}
// buildSlackTargetConfig builds config JSON for a Slack target.
func (h *Handlers) buildSlackTargetConfig(
// buildDatabaseTargetConfig builds config JSON for a database
// (archive) target. The optional expiry (a form value read by
// the caller, which bounds the request body) is validated here,
// at creation time, so an unparseable value is rejected with a
// 400 instead of failing every subsequent delivery. An empty
// expiry yields an empty config (the keep-forever default).
func (h *Handlers) buildDatabaseTargetConfig(
w http.ResponseWriter,
r *http.Request,
targetURL string,
expiry string,
) (string, error) {
if targetURL == "" {
http.Error(
w,
"Webhook URL is required for Slack targets",
http.StatusBadRequest,
)
return "", errMissingURL
expiry = strings.TrimSpace(expiry)
if expiry == "" {
return "", nil
}
err := delivery.ValidateTargetURL(
r.Context(), targetURL,
)
err := delivery.ValidateArchiveExpiry(expiry)
if err != nil {
h.log.Warn(
"target URL blocked by SSRF protection",
"url", targetURL,
"error", err,
)
http.Error(
w,
"Invalid target URL: "+err.Error(),
"Invalid archive expiry: "+err.Error(),
http.StatusBadRequest,
)
return "", err
}
cfg := map[string]any{"webhookUrl": targetURL}
cfg := map[string]any{"expiry": expiry}
configBytes, err := json.Marshal(cfg)
if err != nil {
@@ -1018,23 +1242,31 @@ func (h *Handlers) HandleEntrypointDelete() http.HandlerFunc {
return h.deleteChildResource(
"entrypointID", &database.Entrypoint{},
"failed to delete entrypoint",
nil,
)
}
// HandleTargetDelete handles deleting a target.
// HandleTargetDelete handles deleting a target. Deleting the
// last database target of a webhook leaves its archive writer
// with nothing to write, so the writer is evicted and its
// handle closed; the archive file is left on disk.
func (h *Handlers) HandleTargetDelete() http.HandlerFunc {
return h.deleteChildResource(
"targetID", &database.Target{},
"failed to delete target",
h.evictArchiveWriterIfUnused,
)
}
// deleteChildResource returns a handler that deletes a child
// resource (entrypoint or target) belonging to a webhook.
// resource (entrypoint or target) belonging to a webhook. The
// optional afterDelete hook runs with the webhook's id once the
// delete has succeeded, before the redirect.
func (h *Handlers) deleteChildResource(
idParam string,
model any,
errMsg string,
afterDelete func(webhookID string),
) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
userID, ok := h.getUserID(r)
@@ -1074,6 +1306,10 @@ func (h *Handlers) deleteChildResource(
return
}
if afterDelete != nil {
afterDelete(webhook.ID)
}
http.Redirect(
w, r,
"/source/"+webhook.ID,

589
internal/handlers/source_management_test.go Normal file
View File

@@ -0,0 +1,589 @@
package handlers_test
import (
"context"
"net/http"
"net/http/httptest"
"net/url"
"strconv"
"strings"
"testing"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
const (
// sourceTestUserID is the session user id used by the webhook
// management tests.
sourceTestUserID = "source-test-user"
// sourceIDParam is the chi URL parameter naming a webhook.
sourceIDParam = "sourceID"
)
// formRequest builds an urlencoded POST to path carrying the given
// cookies, plus any chi URL parameters the handler reads.
func formRequest(
path string,
cookies []*http.Cookie,
form url.Values,
urlParams map[string]string,
) *http.Request {
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost,
path,
strings.NewReader(form.Encode()),
)
req.Header.Set(
"Content-Type", "application/x-www-form-urlencoded",
)
for _, c := range cookies {
req.AddCookie(c)
}
rctx := chi.NewRouteContext()
for k, v := range urlParams {
rctx.URLParams.Add(k, v)
}
return req.WithContext(
context.WithValue(req.Context(), chi.RouteCtxKey, rctx),
)
}
// getRequest builds a GET to path carrying the given cookies, plus any
// chi URL parameters the handler reads.
func getRequest(
t *testing.T,
path string,
cookies []*http.Cookie,
urlParams map[string]string,
) *http.Request {
t.Helper()
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, path, nil,
)
for _, c := range cookies {
req.AddCookie(c)
}
rctx := chi.NewRouteContext()
for k, v := range urlParams {
rctx.URLParams.Add(k, v)
}
return req.WithContext(
context.WithValue(req.Context(), chi.RouteCtxKey, rctx),
)
}
// submitCreate posts the webhook creation form with the given
// retention_days value (omitted entirely when retention is nil) and
// returns the recorder.
func submitCreate(
t *testing.T,
h *handlers.Handlers,
cookies []*http.Cookie,
name string,
retention *string,
) *httptest.ResponseRecorder {
t.Helper()
form := url.Values{}
form.Set("name", name)
if retention != nil {
form.Set("retention_days", *retention)
}
req := formRequest("/sources/new", cookies, form, nil)
w := httptest.NewRecorder()
h.HandleSourceCreateSubmit().ServeHTTP(w, req)
return w
}
// onlyWebhook loads the single webhook belonging to the test user.
func onlyWebhook(
t *testing.T,
db *database.Database,
) database.Webhook {
t.Helper()
var webhooks []database.Webhook
require.NoError(
t,
db.DB().Where("user_id = ?", sourceTestUserID).
Find(&webhooks).Error,
)
require.Len(t, webhooks, 1)
return webhooks[0]
}
// seedWebhookWithRetention inserts a webhook owned by the test user
// with an exact stored retention value, bypassing Webhook.BeforeSave
// via a column-level update so that legacy rows can be planted too.
func seedWebhookWithRetention(
t *testing.T,
db *database.Database,
retentionDays int,
) database.Webhook {
t.Helper()
wh := &database.Webhook{
UserID: sourceTestUserID,
Name: "seeded",
RetentionDays: retentionDays,
}
require.NoError(
t,
db.DB().Omit(clause.Associations).Create(wh).Error,
)
require.NoError(
t,
db.DB().Model(wh).
Update("retention_days", retentionDays).Error,
)
wh.RetentionDays = retentionDays
return *wh
}
// storedRetentionDays reads the retention_days column for a webhook.
func storedRetentionDays(
t *testing.T,
db *database.Database,
id string,
) int {
t.Helper()
var got int
require.NoError(
t,
db.DB().Model(&database.Webhook{}).
Where("id = ?", id).
Pluck("retention_days", &got).Error,
)
return got
}
// sourceTestEnv bundles the handler, session, and database a webhook
// management test drives.
type sourceTestEnv struct {
handlers *handlers.Handlers
db *database.Database
cookies []*http.Cookie
}
func setupSourceTest(t *testing.T) *sourceTestEnv {
t.Helper()
var h *handlers.Handlers
var sess *session.Session
var db *database.Database
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
return &sourceTestEnv{
handlers: h,
db: db,
cookies: authenticatedCookies(
t, sess, sourceTestUserID, "sourceuser",
),
}
}
// TestHandleSourceCreateSubmit_ZeroRetentionPersistsForever is the core
// regression test for the bug: the create form's 0 must reach the
// database as the retain-forever sentinel rather than being replaced by
// the column's default of 30.
func TestHandleSourceCreateSubmit_ZeroRetentionPersistsForever(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
zero := "0"
w := submitCreate(t, env.handlers, env.cookies, "forever", &zero)
require.Equal(t, http.StatusSeeOther, w.Code)
wh := onlyWebhook(t, env.db)
assert.Equal(
t,
database.RetentionForeverDays,
storedRetentionDays(t, env.db, wh.ID),
)
assert.True(t, wh.RetainsForever())
}
func TestHandleSourceCreateSubmit_OmittedRetentionUsesDefault(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
w := submitCreate(t, env.handlers, env.cookies, "defaulted", nil)
require.Equal(t, http.StatusSeeOther, w.Code)
wh := onlyWebhook(t, env.db)
assert.Equal(
t,
database.DefaultRetentionDays,
storedRetentionDays(t, env.db, wh.ID),
)
}
// TestHandleSourceCreate_PrefillsDefaultFromConstant keeps the create
// form's pre-filled retention from becoming a third hardcoded copy of
// the 30-day policy.
func TestHandleSourceCreate_PrefillsDefaultFromConstant(t *testing.T) {
t.Parallel()
env := setupSourceTest(t)
w := httptest.NewRecorder()
env.handlers.HandleSourceCreate().ServeHTTP(
w, getRequest(t, "/sources/new", env.cookies, nil),
)
require.Equal(t, http.StatusOK, w.Code)
body := w.Body.String()
assert.Contains(
t, body,
`value="`+strconv.Itoa(database.DefaultRetentionDays)+`"`,
)
assert.NotContains(
t, body, `max="365"`,
"a max below the sentinel would block retain-forever",
)
assert.Contains(t, body, `min="0"`)
}
func TestHandleSourceCreateSubmit_InvalidRetentionIsRejected(
t *testing.T,
) {
t.Parallel()
for _, raw := range []string{"abc", "-1", "3.5"} {
t.Run(raw, func(t *testing.T) {
t.Parallel()
env := setupSourceTest(t)
w := submitCreate(
t, env.handlers, env.cookies, "bad", &raw,
)
assert.Equal(t, http.StatusBadRequest, w.Code)
assert.Contains(
t, w.Body.String(), "Retention must be",
)
var count int64
require.NoError(
t,
env.db.DB().Model(&database.Webhook{}).
Where("user_id = ?", sourceTestUserID).
Count(&count).Error,
)
assert.Zero(
t, count,
"no webhook may be created from a rejected form",
)
})
}
}
// TestHandleSourceCreateSubmit_OverflowingRetentionIsRejected covers
// the data-loss path directly: a finite retention above the largest one
// the reaper's cutoff arithmetic can represent must never reach the
// database, because the sweep would compute a future cutoff and delete
// every event the webhook has.
func TestHandleSourceCreateSubmit_OverflowingRetentionIsRejected(
t *testing.T,
) {
t.Parallel()
tooBig := strconv.Itoa(database.MaxFiniteRetentionDays + 1)
env := setupSourceTest(t)
w := submitCreate(t, env.handlers, env.cookies, "huge", &tooBig)
assert.Equal(t, http.StatusBadRequest, w.Code)
assert.Contains(
t, w.Body.String(),
strconv.Itoa(database.MaxFiniteRetentionDays),
"the form tells the user the actual ceiling",
)
var count int64
require.NoError(
t,
env.db.DB().Model(&database.Webhook{}).
Where("user_id = ?", sourceTestUserID).
Count(&count).Error,
)
assert.Zero(
t, count,
"no webhook may be created from a rejected form",
)
}
// TestHandleSourceCreateSubmit_SentinelIsAcceptedAsForever guards the
// boundary between "too large to represent" and "retain forever": the
// sentinel is above MaxFiniteRetentionDays, but it is the value the
// edit form pre-fills, so it must be accepted rather than rejected as
// out of range.
func TestHandleSourceCreateSubmit_SentinelIsAcceptedAsForever(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
sentinel := strconv.Itoa(database.RetentionForeverDays)
w := submitCreate(t, env.handlers, env.cookies, "forever", &sentinel)
require.Equal(t, http.StatusSeeOther, w.Code)
wh := onlyWebhook(t, env.db)
assert.Equal(
t,
database.RetentionForeverDays,
storedRetentionDays(t, env.db, wh.ID),
)
}
// TestHandleSourceCreateSubmit_RejectedFormKeepsUserInput checks that a
// validation failure hands the user's typing back, matching what the
// edit form already does. Losing a long description to a mistyped
// retention value is the kind of thing that makes people give up on a
// form.
func TestHandleSourceCreateSubmit_RejectedFormKeepsUserInput(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
const (
name = "kept-name"
description = "a description worth not losing"
)
form := url.Values{}
form.Set("name", name)
form.Set("description", description)
form.Set("retention_days", "nonsense")
req := formRequest("/sources/new", env.cookies, form, nil)
w := httptest.NewRecorder()
env.handlers.HandleSourceCreateSubmit().ServeHTTP(w, req)
require.Equal(t, http.StatusBadRequest, w.Code)
body := w.Body.String()
assert.Contains(t, body, `value="`+name+`"`)
assert.Contains(t, body, description)
}
// submitEdit posts the webhook edit form for the given webhook.
func submitEdit(
t *testing.T,
env *sourceTestEnv,
wh database.Webhook,
retention string,
) *httptest.ResponseRecorder {
t.Helper()
form := url.Values{}
form.Set("name", wh.Name)
form.Set("description", wh.Description)
form.Set("retention_days", retention)
req := formRequest(
"/source/"+wh.ID+"/edit",
env.cookies,
form,
map[string]string{sourceIDParam: wh.ID},
)
w := httptest.NewRecorder()
env.handlers.HandleSourceEditSubmit().ServeHTTP(w, req)
return w
}
func TestHandleSourceEditSubmit_ZeroRetentionPersistsForever(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
wh := seedWebhookWithRetention(
t, env.db, database.DefaultRetentionDays,
)
w := submitEdit(t, env, wh, "0")
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Equal(
t,
database.RetentionForeverDays,
storedRetentionDays(t, env.db, wh.ID),
)
}
func TestHandleSourceEditSubmit_InvalidRetentionIsRejected(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
wh := seedWebhookWithRetention(
t, env.db, database.DefaultRetentionDays,
)
w := submitEdit(t, env, wh, "not-a-number")
assert.Equal(t, http.StatusBadRequest, w.Code)
assert.Contains(t, w.Body.String(), "Retention must be")
assert.Equal(
t,
database.DefaultRetentionDays,
storedRetentionDays(t, env.db, wh.ID),
"a rejected form must not change the stored retention",
)
}
func TestHandleSourceEditSubmit_EmptyRetentionLeavesValueUnchanged(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
wh := seedWebhookWithRetention(t, env.db, 7)
w := submitEdit(t, env, wh, "")
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Equal(t, 7, storedRetentionDays(t, env.db, wh.ID))
}
// TestSourceEditForm_ForeverWebhookRoundTrips walks the exact path that
// the removed max="365" cap used to break: render the edit form for a
// retain-forever webhook, confirm the pre-filled sentinel is not capped
// by browser validation, then submit that pre-filled value straight
// back and confirm the retention policy survives untouched.
func TestSourceEditForm_ForeverWebhookRoundTrips(t *testing.T) {
t.Parallel()
env := setupSourceTest(t)
wh := seedWebhookWithRetention(
t, env.db, database.RetentionForeverDays,
)
req := getRequest(
t, "/source/"+wh.ID+"/edit", env.cookies,
map[string]string{sourceIDParam: wh.ID},
)
w := httptest.NewRecorder()
env.handlers.HandleSourceEdit().ServeHTTP(w, req)
require.Equal(t, http.StatusOK, w.Code)
sentinel := strconv.Itoa(database.RetentionForeverDays)
body := w.Body.String()
assert.Contains(
t, body, `value="`+sentinel+`"`,
"the edit form pre-fills the stored retention",
)
assert.NotContains(
t, body, `max="365"`,
"a max below the sentinel would block saving any edit",
)
// "Currently forever." is the rendered RetentionLabel, not the
// static hint below the input, which says "Enter 0 to retain events
// forever." A bare Contains of "forever" would pass for any
// webhook and would assert nothing about this one.
assert.Contains(
t, body, "Currently forever.",
"the form reports this webhook's policy as forever",
)
// Submit the pre-filled value back, exactly as a browser would.
post := submitEdit(t, env, wh, sentinel)
require.Equal(t, http.StatusSeeOther, post.Code)
assert.Equal(
t,
database.RetentionForeverDays,
storedRetentionDays(t, env.db, wh.ID),
)
}
// TestSourceListAndDetail_ShowForeverNotTheSentinelNumber checks that
// the retain-forever value is never rendered to the user as a raw day
// count on either read-only view.
func TestSourceListAndDetail_ShowForeverNotTheSentinelNumber(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
wh := seedWebhookWithRetention(
t, env.db, database.RetentionForeverDays,
)
sentinel := strconv.Itoa(database.RetentionForeverDays)
listW := httptest.NewRecorder()
env.handlers.HandleSourceList().ServeHTTP(
listW, getRequest(t, "/sources", env.cookies, nil),
)
require.Equal(t, http.StatusOK, listW.Code)
assert.Contains(t, listW.Body.String(), "Retention: forever")
assert.NotContains(t, listW.Body.String(), sentinel)
detailW := httptest.NewRecorder()
env.handlers.HandleSourceDetail().ServeHTTP(
detailW,
getRequest(
t, "/source/"+wh.ID, env.cookies,
map[string]string{sourceIDParam: wh.ID},
),
)
require.Equal(t, http.StatusOK, detailW.Code)
assert.Contains(t, detailW.Body.String(), "Retention: forever")
assert.NotContains(t, detailW.Body.String(), sentinel)
}

206
internal/handlers/target_create_query_test.go Normal file
View File

@@ -0,0 +1,206 @@
package handlers_test
import (
"bytes"
"context"
"log/slog"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/middleware"
)
// targetSecretSegments are the path segments of an incoming-webhook
// URL. For Slack, Discord and Teams the path IS the bearer credential,
// so this string must not reach storage or the access log by way of
// the request line.
const targetSecretSegments = "T00000000/B00000000/QQTARGETSECRETQQ"
// targetSecretURL is a destination whose secret lives in its path. It
// uses a literal public address rather than a hostname so the SSRF
// check resolves nothing: with a hostname, a sandbox without DNS would
// reject the URL for the wrong reason and the test would pass even
// with the defect reintroduced.
const targetSecretURL = "https://93.184.216.34/services/" +
targetSecretSegments
// targetsForWebhook returns every target stored against a webhook.
func targetsForWebhook(
t *testing.T,
db *database.Database,
webhookID string,
) []database.Target {
t.Helper()
var targets []database.Target
require.NoError(
t,
db.DB().Where("webhook_id = ?", webhookID).
Find(&targets).Error,
)
return targets
}
// postTargetCreate drives HandleTargetCreate through the production
// access-log middleware and a chi route, so the logged url field is
// produced exactly as it ships, and returns the recorder plus the
// captured log.
func postTargetCreate(
t *testing.T,
env *sourceTestEnv,
webhookID string,
query string,
form url.Values,
) (*httptest.ResponseRecorder, string) {
t.Helper()
logBuf := new(bytes.Buffer)
mw := middleware.NewForTest(
slog.New(slog.NewJSONHandler(
logBuf, &slog.HandlerOptions{Level: slog.LevelInfo},
)),
&config.Config{Environment: config.EnvironmentDev},
nil,
)
router := chi.NewRouter()
router.Use(mw.Logging())
router.Post(
"/source/{sourceID}/targets",
env.handlers.HandleTargetCreate(),
)
target := "/source/" + webhookID + "/targets"
if query != "" {
target += "?" + query
}
body := ""
if form != nil {
body = form.Encode()
}
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost,
target,
strings.NewReader(body),
)
req.Header.Set(
"Content-Type", "application/x-www-form-urlencoded",
)
for _, c := range env.cookies {
req.AddCookie(c)
}
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
return w, logBuf.String()
}
// TestHandleTargetCreate_QueryStringURLDoesNotConfigureATarget is the
// regression test for the ingress leak. r.FormValue falls back to the
// query string when a field is absent from the POST body, so
//
// POST /source/{id}/targets?url=https://hooks.slack.com/services/...
//
// with an empty url field used to create a working target from a value
// carried on the request line — where logs, proxies, Referer headers
// and error trackers record it. The handler reads the body only, so
// the request is rejected for a missing URL and stores nothing.
//
// name and type are sent in the BODY on purpose: the request has to
// get past those two validations for the assertion to be about the url
// read specifically.
func TestHandleTargetCreate_QueryStringURLDoesNotConfigureATarget(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
webhook := seedWebhookWithRetention(t, env.db, 30)
body := url.Values{}
body.Set("name", "leaky")
body.Set("type", string(database.TargetTypeSlack))
w, logged := postTargetCreate(
t, env, webhook.ID,
"url="+url.QueryEscape(targetSecretURL),
body,
)
assert.Equal(t, http.StatusBadRequest, w.Code)
targets := targetsForWebhook(t, env.db, webhook.ID)
assert.Empty(
t, targets,
"a query-string value must not populate a target config",
)
assert.NotContains(t, logged, targetSecretSegments)
assert.NotContains(t, logged, "93.184.216.34")
assert.NotEmpty(t, logged, "the access log line must still be written")
}
// TestHandleTargetCreate_BodyURLStillCreatesTheTarget is the positive
// control for the test above: the rejection has to come from where the
// value was read, not from the handler being broken.
func TestHandleTargetCreate_BodyURLStillCreatesTheTarget(t *testing.T) {
t.Parallel()
env := setupSourceTest(t)
webhook := seedWebhookWithRetention(t, env.db, 30)
body := url.Values{}
body.Set("name", "legit")
body.Set("type", string(database.TargetTypeSlack))
body.Set("url", targetSecretURL)
w, logged := postTargetCreate(t, env, webhook.ID, "", body)
assert.Equal(t, http.StatusSeeOther, w.Code)
targets := targetsForWebhook(t, env.db, webhook.ID)
require.Len(t, targets, 1)
assert.Contains(t, targets[0].Config, targetSecretSegments)
// The body carried the credential, so the access log must still
// not have it: the log records the request line only.
assert.NotContains(t, logged, targetSecretSegments)
}
// TestHandleTargetCreate_QueryStringCannotSupplyNameOrType covers the
// rest of the converted reads on this handler in one request: with an
// empty body, nothing the query carries is visible to it.
func TestHandleTargetCreate_QueryStringCannotSupplyNameOrType(
t *testing.T,
) {
t.Parallel()
env := setupSourceTest(t)
webhook := seedWebhookWithRetention(t, env.db, 30)
w, _ := postTargetCreate(
t, env, webhook.ID,
"name=leaky&type=slack&max_retries=9&expiry=30d&url="+
url.QueryEscape(targetSecretURL),
url.Values{},
)
assert.Equal(t, http.StatusBadRequest, w.Code)
assert.Contains(t, w.Body.String(), "Name is required")
assert.Empty(t, targetsForWebhook(t, env.db, webhook.ID))
}

299
internal/handlers/ui_copy_test.go Normal file
View File

@@ -0,0 +1,299 @@
package handlers_test
import (
"context"
"net/http"
"net/http/httptest"
"strconv"
"testing"
"time"
"github.com/stretchr/testify/assert"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
// Template data keys the page templates read. The handlers package has
// its own unexported constants for these; this is the external test
// package, so it needs its own.
const (
dataKeyWebhook = "Webhook"
dataKeyError = "Error"
)
// testWebhookID is the identifier given to the webhook under test on
// pages that render one.
const testWebhookID = "wh-1"
// renderPage renders a page template through the real template set as
// an authenticated user and returns the resulting HTML.
func renderPage(
t *testing.T,
h *handlers.Handlers,
sess *session.Session,
page string,
data map[string]any,
) string {
t.Helper()
cookies := authenticatedCookies(t, sess, "test-user-id", "testuser")
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/", nil,
)
for _, c := range cookies {
req.AddCookie(c)
}
w := httptest.NewRecorder()
h.RenderTemplateForTest(w, req, page, data)
return w.Body.String()
}
// TestNavbarUsesWebhookTerminology pins the user-visible navigation
// label to "Webhooks". The /sources route is deliberately unchanged, so
// the assertion targets the link text rather than the href.
func TestNavbarUsesWebhookTerminology(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
var sess *session.Session
app := newTestApp(t, &h, &sess)
app.RequireStart()
t.Cleanup(app.RequireStop)
// One item, so the list body renders too: it calls
// WebhookListItem.RetentionLabel, promoted from the embedded
// Webhook and therefore a pointer method. An empty list would
// skip that call and hide a template error behind the
// navigation assertions below.
item := handlers.WebhookListItem{}
item.Name = "wh"
item.ID = testWebhookID
item.RetentionDays = 14
body := renderPage(t, h, sess, "sources_list.html", map[string]any{
"Webhooks": []handlers.WebhookListItem{item},
})
assert.Contains(t, body, "Retention: 14 days")
assert.Contains(t, body, `class="btn-text">Webhooks</a>`)
assert.Contains(
t, body, `class="btn-text w-full text-left">Webhooks</a>`,
)
assert.Contains(
t, body,
`<h1 class="text-2xl font-medium text-gray-900">Webhooks</h1>`,
)
assert.NotContains(
t, body, ">Sources<",
"no user-visible element may still be labelled Sources",
)
assert.Contains(
t, body, `href="/sources"`,
"the /sources route itself must not change",
)
}
// TestEditPageUsesWebhookTerminology pins the edit page's heading and
// its back link. The link's href still points at /source/{id}, which is
// intentional: only user-visible copy changes.
func TestEditPageUsesWebhookTerminology(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
var sess *session.Session
app := newTestApp(t, &h, &sess)
app.RequireStart()
t.Cleanup(app.RequireStop)
// The webhook goes in as a pointer because source_edit.html calls
// Webhook.RetentionLabel, a pointer method: a map element is not
// addressable, so a value here renders an error instead of the
// page.
webhook := &database.Webhook{Name: "wh", RetentionDays: 14}
webhook.ID = testWebhookID
body := renderPage(t, h, sess, "source_edit.html", map[string]any{
dataKeyWebhook: webhook,
dataKeyError: "",
})
assert.Contains(t, body, "Edit Webhook")
assert.NotContains(t, body, ">Sources<")
assert.Contains(t, body, `href="/source/wh-1"`)
}
// TestCreateFormRetentionCopyMatchesBehaviour pins the create form's
// retention copy to what the code does: the reaper permanently deletes
// events past the cutoff, an empty field falls back to
// DefaultRetentionDays, and 0 is rewritten to the retain-forever
// sentinel by Webhook.BeforeSave.
func TestCreateFormRetentionCopyMatchesBehaviour(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
var sess *session.Session
app := newTestApp(t, &h, &sess)
app.RequireStart()
t.Cleanup(app.RequireStop)
body := renderPage(t, h, sess, "sources_new.html", map[string]any{
"Name": "",
"Description": "",
"DefaultRetentionDays": database.DefaultRetentionDays,
dataKeyError: "",
})
assert.Contains(
t, body,
"permanently deletes events older than this",
"the form must say retention is enforced by deletion",
)
assert.Contains(t, body, "Enter 0 to retain events forever")
assert.Contains(
t, body,
"leave blank to use the default of "+
strconv.Itoa(database.DefaultRetentionDays)+" days",
"blank means the default, not forever",
)
}
// TestEditFormRetentionCopyMatchesBehaviour pins the edit form's
// retention copy, including that it states the stored policy via
// RetentionLabel and that an empty field leaves that policy unchanged
// rather than meaning forever.
func TestEditFormRetentionCopyMatchesBehaviour(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
var sess *session.Session
app := newTestApp(t, &h, &sess)
app.RequireStart()
t.Cleanup(app.RequireStop)
finite := &database.Webhook{Name: "wh", RetentionDays: 14}
finite.ID = testWebhookID
body := renderPage(t, h, sess, "source_edit.html", map[string]any{
dataKeyWebhook: finite,
dataKeyError: "",
})
assert.Contains(t, body, "Currently 14 days.")
assert.Contains(
t, body,
"permanently deletes events older than this",
)
assert.Contains(t, body, "Enter 0 to retain events forever")
assert.Contains(
t, body,
"leave blank to keep the current setting",
"blank means unchanged, not forever",
)
forever := &database.Webhook{
Name: "wh",
RetentionDays: database.RetentionForeverDays,
}
forever.ID = "wh-2"
foreverBody := renderPage(
t, h, sess, "source_edit.html", map[string]any{
dataKeyWebhook: forever,
dataKeyError: "",
},
)
assert.Contains(
t, foreverBody, "Currently forever.",
"a retain-forever webhook must not read as a day count",
)
assert.Contains(
t, foreverBody,
"No events are deleted while retention is set to forever",
)
assert.NotContains(
t, foreverBody,
"permanently deletes events older than this",
"the reaper skips retain-forever webhooks, so the form "+
"must not claim it deletes their events",
)
}
// TestEntrypointCopyButtonIsProgressiveEnhancement proves the copy
// affordance degrades: the button ships with the hidden attribute, so a
// browser that never runs app.js shows no dead control, and the URL is
// rendered as ordinary selectable text either way.
func TestEntrypointCopyButtonIsProgressiveEnhancement(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
var sess *session.Session
app := newTestApp(t, &h, &sess)
app.RequireStart()
t.Cleanup(app.RequireStop)
entrypoint := database.Entrypoint{Path: "abc123"}
entrypoint.ID = "ep-1"
// The webhook goes in as a pointer because source_detail.html
// calls Webhook.RetentionLabel, a pointer method: a map element
// is not addressable, so a value here aborts execution partway
// down the page, after the copy button has already been flushed
// to the response.
webhook := &database.Webhook{Name: "wh", RetentionDays: 14}
webhook.ID = testWebhookID
webhook.CreatedAt = time.Date(
2026, time.January, 2, 3, 4, 5, 0, time.UTC,
)
body := renderPage(t, h, sess, "source_detail.html", map[string]any{
dataKeyWebhook: webhook,
"Entrypoints": []database.Entrypoint{entrypoint},
// The handler passes delivery.NewTargetViews(targets), never
// raw targets, so the test data has to have that same shape.
"Targets": delivery.NewTargetViews(nil),
"Events": []database.Event{},
"BaseURL": "https://hooks.example.com",
})
assert.Contains(
t, body,
`<code id="entrypoint-url-ep-1"`,
)
assert.Contains(t, body, "https://hooks.example.com/webhook/abc123")
assert.Contains(
t, body,
`hidden data-copy-target="entrypoint-url-ep-1"`,
"the button must start hidden and be revealed by script",
)
// renderTemplate streams to the ResponseWriter, so an abort
// midway still leaves everything above it in the body. This pins
// content from the last line of the template, which is below the
// assertions above: without it, a page that renders the copy
// button and then 500s passes.
assert.Contains(
t, body, "Retention: 14 days",
"the page must render to completion, not abort partway",
)
}

29
internal/handlers/webhook.go
View File

@@ -9,6 +9,7 @@ import (
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
"sneak.berlin/go/webhooker/internal/logfield"
)
const (
@@ -39,12 +40,6 @@ func (h *Handlers) HandleWebhook() http.HandlerFunc {
return
}
h.log.Info("webhook request received",
"entrypoint_uuid", entrypointUUID,
"method", r.Method,
"remote_addr", r.RemoteAddr,
)
entrypoint, ok := h.lookupEntrypoint(
w, r, entrypointUUID,
)
@@ -52,6 +47,18 @@ func (h *Handlers) HandleWebhook() http.HandlerFunc {
return
}
// Logged only once the UUID is known to name a real
// entrypoint. The UUID comes straight out of the path on
// the one unauthenticated endpoint, so logging it before
// the lookup let a client write an INFO line per invented
// path; the request itself is already in the access log
// and a miss is already logged at DEBUG.
h.log.Info("webhook request received",
"entrypoint_uuid", entrypointUUID,
"method", r.Method,
"remote_addr", r.RemoteAddr,
)
if !entrypoint.Active {
http.Error(w, "Gone", http.StatusGone)
@@ -119,9 +126,16 @@ func (h *Handlers) lookupEntrypoint(
"path = ?", entrypointUUID,
).First(&entrypoint)
if result.Error != nil {
// The receiver is unauthenticated and /webhook/{uuid}
// matches any single segment, so this value is entirely
// client-chosen on exactly the branch where the lookup
// failed. DEBUG is off by default; the cap is what keeps
// turning it on from restoring an unbounded write.
h.log.Debug(
"entrypoint not found",
"path", entrypointUUID,
"path", logfield.Truncate(
entrypointUUID, logfield.MaxBytes,
),
)
http.NotFound(w, r)
@@ -329,6 +343,7 @@ func (h *Handlers) buildDeliveryTasks(
DeliveryID: dlv.ID,
EventID: event.ID,
WebhookID: entrypoint.WebhookID,
EntrypointID: entrypoint.ID,
TargetID: targets[i].ID,
TargetName: targets[i].Name,
TargetType: targets[i].Type,

21
internal/lifecycle/export_test.go Normal file
View File

@@ -0,0 +1,21 @@
package lifecycle
import (
"context"
"log/slog"
)
// WaitDone exposes waitDone to the external test package. Only the
// unexported waiter can be handed a channel that is already closed
// before the call, which is the state the preamble exists for;
// through WaitForShutdown the waiter goroutine may or may not have
// closed the channel yet, so the case is not reachable
// deterministically from outside.
func WaitDone(
ctx context.Context,
log *slog.Logger,
component string,
done <-chan struct{},
) error {
return waitDone(ctx, log, component, done)
}

80
internal/lifecycle/lifecycle.go Normal file
View File

@@ -0,0 +1,80 @@
// Package lifecycle holds helpers shared by the components that
// register fx start and stop hooks.
package lifecycle
import (
"context"
"fmt"
"log/slog"
"sync"
)
// WaitForShutdown waits for wg to drain, bounded by ctx.
//
// fx hands OnStop a context carrying the application's stop
// timeout. A bare wg.Wait() discards that deadline, so a single
// goroutine that never observes cancellation — a delivery target
// that never returns, a SQLite operation blocked on a lock —
// hangs the process forever instead of letting it exit when the
// timeout expires, which is exactly when a clean shutdown matters
// most.
//
// On timeout it logs at error naming component and returns an
// error: the goroutines are still running, and reporting success
// would hide an unclean shutdown from the operator. The waiting
// goroutine outlives this call and exits when (if) wg drains; it
// holds nothing but the channel it closes.
func WaitForShutdown(
ctx context.Context,
log *slog.Logger,
component string,
wg *sync.WaitGroup,
) error {
done := make(chan struct{})
go func() {
defer close(done)
wg.Wait()
}()
return waitDone(ctx, log, component, done)
}
// waitDone waits for done to close, bounded by ctx.
//
// The non-blocking preamble is load-bearing. When the component has
// already drained and ctx has already expired, both cases of the
// bounded select are ready and Go picks between them uniformly at
// random, so a clean shutdown would be reported as a timeout about
// half the time. Draining wins: the goroutines are gone, and there
// is nothing left for the operator to act on.
func waitDone(
ctx context.Context,
log *slog.Logger,
component string,
done <-chan struct{},
) error {
select {
case <-done:
return nil
default:
}
select {
case <-done:
return nil
case <-ctx.Done():
log.Error(
"shutdown timed out, goroutines still running",
"component", component,
"error", ctx.Err(),
)
return fmt.Errorf(
"%s: shutdown timed out, "+
"goroutines still running: %w",
component, ctx.Err(),
)
}
}

113
internal/lifecycle/lifecycle_test.go Normal file
View File

@@ -0,0 +1,113 @@
package lifecycle_test
import (
"context"
"log/slog"
"sync"
"testing"
"time"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/lifecycle"
)
// waitTimeout is the stop budget the timeout case gives a
// goroutine that never returns. The test's own patience is the
// go test deadline, so the only thing this value affects is how
// long the case takes.
const waitTimeout = 100 * time.Millisecond
func discardLogger() *slog.Logger {
return slog.New(slog.DiscardHandler)
}
func TestWaitForShutdown_DrainedGroup(t *testing.T) {
t.Parallel()
var wg sync.WaitGroup
wg.Go(func() {})
require.NoError(
t,
lifecycle.WaitForShutdown(
context.Background(), discardLogger(),
"test component", &wg,
),
)
}
// racePasses is how many times the both-cases-ready race is run.
// Without the preamble each pass is an independent coin flip, so
// the probability of the whole loop passing by luck is 2^-N: at
// this N the test is deterministic in practice, and it involves no
// wall-clock waiting at all.
const racePasses = 1000
// TestWaitDone_DrainedBeforeExpiredContext covers the case where a
// component drained cleanly but the stop context had already
// expired. Both select cases are ready, and Go chooses among ready
// cases uniformly at random, so the drained case must be settled by
// the preamble before the bounded select ever runs.
func TestWaitDone_DrainedBeforeExpiredContext(t *testing.T) {
t.Parallel()
done := make(chan struct{})
close(done)
ctx, cancel := context.WithCancel(context.Background())
cancel()
for pass := range racePasses {
require.NoErrorf(
t,
lifecycle.WaitDone(
ctx, discardLogger(), "test component", done,
),
"pass %d reported a timeout for a drained component",
pass,
)
}
}
// TestWaitDone_ExpiredContext pins the other side of the preamble:
// an expired context with a component that has not drained is still
// a timeout.
func TestWaitDone_ExpiredContext(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithCancel(context.Background())
cancel()
err := lifecycle.WaitDone(
ctx, discardLogger(), "test component",
make(chan struct{}),
)
require.ErrorIs(t, err, context.Canceled)
require.ErrorContains(t, err, "test component")
}
func TestWaitForShutdown_ContextExpires(t *testing.T) {
t.Parallel()
release := make(chan struct{})
t.Cleanup(func() { close(release) })
var wg sync.WaitGroup
wg.Go(func() { <-release })
ctx, cancel := context.WithTimeout(
context.Background(), waitTimeout,
)
defer cancel()
err := lifecycle.WaitForShutdown(
ctx, discardLogger(), "test component", &wg,
)
require.ErrorIs(t, err, context.DeadlineExceeded)
require.ErrorContains(t, err, "test component")
}

143
internal/logfield/logfield.go Normal file
View File

@@ -0,0 +1,143 @@
// Package logfield bounds the client-supplied values this service
// writes into its logs.
//
// Any log field whose content a client picks is spent against a budget
// here, in ENCODED bytes rather than in the bytes the client sent, so
// that escaping cannot multiply a field past its nominal size. One
// budget and one implementation serves the access log in
// internal/middleware and every other slog call that reaches a
// client-chosen path, header or form value; a second, ad-hoc
// truncation somewhere else in the tree is the thing this package
// exists to prevent.
package logfield
import (
"strings"
"unicode"
"unicode/utf8"
)
const (
// MaxBytes is the default budget for a log field whose value the
// client supplies outright: a URL, a path, a header, a form value.
// The budget is spent in ENCODED bytes (see Truncate), so 512 still
// holds a real browser's User-Agent whole — those are plain ASCII,
// which encodes one byte for one — while a value built from
// characters the encoder escapes keeps a shorter prefix. That is
// the intended trade: 500 quotation marks are not a debugging
// asset.
MaxBytes = 512
// TruncationMarker is appended to any field that was cut, so a
// short value and a truncated one cannot be confused. It is charged
// on top of the budget, not inside it.
TruncationMarker = "[truncated]"
)
// EncodedBytes is what r costs on the line once the log handler has
// escaped it, taking the worse of the two handlers internal/logger
// configures.
//
// slog's JSON handler escapes quote, backslash, newline, carriage
// return and tab to two bytes each, and every other C0 control plus
// LINE SEPARATOR and PARAGRAPH SEPARATOR to a six-byte \u escape; it
// passes every other rune through as its own UTF-8. Its text handler
// quotes with strconv.Quote, which spells a non-printable rune below
// U+10000 as \uXXXX but one at or above U+10000 as \UXXXXXXXX — ten
// bytes, not six. The text handler is therefore the worse of the two
// for every non-printable rune, and by four bytes apiece for the
// 955,086 unassigned, private-use and format code points on planes 1
// to 16.
//
// Charging ten there is what makes the stated line ceilings hold for
// the tty handler as well: U+1000C encodes as F0 90 80 8C, every byte
// >= 0x80, which httpguts.ValidHeaderFieldValue accepts and
// net/textproto does not strip, so a header can be filled with them.
//
// Both handlers pass printable runes through as their own UTF-8, so
// unicode.IsPrint separates the escaped cases from the plain ones for
// either handler.
func EncodedBytes(r rune) int {
const (
// A backslash and the character itself.
shortEscapeBytes = 2
// \uXXXX, which is also the width of \u00XX.
escapedRuneBytes = 6
// \UXXXXXXXX, strconv.Quote's spelling of a non-printable
// rune outside the basic multilingual plane.
escapedAstralRuneBytes = 10
// The first code point strconv.Quote spells with \U.
firstAstralRune = 0x10000
)
switch {
case r == '"' || r == '\\' || r == '\n' || r == '\r' || r == '\t':
return shortEscapeBytes
case !unicode.IsPrint(r) && r >= firstAstralRune:
return escapedAstralRuneBytes
case !unicode.IsPrint(r):
return escapedRuneBytes
default:
return utf8.RuneLen(r)
}
}
// Truncate caps s at maxBytes of ENCODED output, marking the value
// when it cuts.
//
// Budgeting raw bytes would not bound the line. Escaping only ever
// grows a value, so a raw budget spent on characters the encoder
// escapes buys a field several times its nominal size — and the line
// is the thing an operator is told to multiply by their request rate.
// Charging each rune what it will actually cost is what makes a stated
// ceiling true rather than merely larger. The visible consequence is
// that an escape-heavy value keeps a shorter prefix than a plain one,
// which is the correct trade.
//
// The result is always valid UTF-8. A cut on a byte boundary can split
// a multi-byte rune, and a header can carry bytes that were never
// valid UTF-8 to begin with; both are dropped rather than kept, since
// an encoder would otherwise spend six bytes replacing each one.
func Truncate(s string, maxBytes int) string {
// No rune encodes to fewer bytes than it occupies, so nothing past
// maxBytes raw can fit the budget. Slicing first bounds the scan
// below to the budget rather than to the size of the header the
// client sent.
window, cut := s, false
if len(window) > maxBytes {
window, cut = window[:maxBytes], true
}
var (
kept strings.Builder
spent int
)
for i := 0; i < len(window); {
r, size := utf8.DecodeRuneInString(window[i:])
if r == utf8.RuneError && size == 1 {
i += size
continue
}
cost := EncodedBytes(r)
if spent+cost > maxBytes {
cut = true
break
}
spent += cost
kept.WriteString(window[i : i+size])
i += size
}
if !cut {
return kept.String()
}
return kept.String() + TruncationMarker
}

321
internal/logfield/logfield_test.go Normal file
View File

@@ -0,0 +1,321 @@
package logfield_test
import (
"bytes"
"io"
"log/slog"
"strings"
"testing"
"unicode/utf8"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/logfield"
)
// budget is the field budget these tests spend. Small enough that a
// cut is unambiguous, large enough to hold several runes of every
// width.
const budget = 64
// sampleRunes is how many runes wide the values in the charge test
// are. The handlers add a constant per field — a pair of quotes when
// the value needs quoting — so the per-rune charge is only visible
// once it is amortised over a run of them.
const sampleRunes = 64
// quotingSlack is that constant: the pair of quotes a handler adds to
// a value that needs them and omits from one that does not.
const quotingSlack = 2
// newHandlers are the two handlers internal/logger can install. Time
// is dropped so a line's width is a function of its value alone —
// RFC3339Nano trims trailing zeros, so two consecutive timestamps do
// not render to the same number of bytes.
func newHandlers() map[string]func(io.Writer) slog.Handler {
opts := &slog.HandlerOptions{
Level: slog.LevelDebug,
ReplaceAttr: func(_ []string, a slog.Attr) slog.Attr {
if a.Key == slog.TimeKey {
return slog.Attr{}
}
return a
},
}
return map[string]func(io.Writer) slog.Handler{
"json": func(w io.Writer) slog.Handler {
return slog.NewJSONHandler(w, opts)
},
"text": func(w io.Writer) slog.Handler {
return slog.NewTextHandler(w, opts)
},
}
}
// renderedWidth is the number of bytes a handler writes for a line
// carrying value in a single attribute.
func renderedWidth(
newHandler func(io.Writer) slog.Handler,
value string,
) int {
buf := new(bytes.Buffer)
slog.New(newHandler(buf)).Info("m", "v", value)
return buf.Len()
}
// chargeTestRunes is the set of code points the charge test measures:
// every rune in the first two planes' worth of the BMP that the
// handlers are most likely to treat specially, the separators that
// only slog's JSON handler escapes, and a stratified sample across
// the rest of Unicode so the astral charge is exercised on more than
// one hand-picked rune.
func chargeTestRunes() []rune {
const (
denseCeiling = 0x800
stride = 1021
surrogateLo = 0xD800
surrogateHi = 0xDFFF
)
var runes []rune
keep := func(r rune) {
if r >= surrogateLo && r <= surrogateHi {
return
}
runes = append(runes, r)
}
for r := range rune(denseCeiling) {
keep(r)
}
for _, r := range []rune{
0x2028, 0x2029, 0x200B, 0x4E00, 0xE000, 0xFFFD,
0x1000C, 0x1F600, 0xE0001, 0x10FFFF,
} {
keep(r)
}
for r := rune(denseCeiling); r <= utf8.MaxRune; r += stride {
keep(r)
}
return runes
}
// TestEncodedBytes_ChargesAtLeastWhatTheHandlersEmit is the property
// the whole capping scheme rests on: a rune may not cost more on the
// line than the budget was charged for it. An undercharged rune is
// how a stated ceiling becomes false without any test noticing, so
// the charge is measured against what the handlers actually write
// rather than against the escaping rules as read.
func TestEncodedBytes_ChargesAtLeastWhatTheHandlersEmit(t *testing.T) {
t.Parallel()
for name, newHandler := range newHandlers() {
t.Run(name, func(t *testing.T) {
t.Parallel()
// 'a' is a printable ASCII rune, charged exactly one
// byte, so it is the zero point the other runes are
// measured against.
base := renderedWidth(
newHandler, strings.Repeat("a", sampleRunes),
)
for _, r := range chargeTestRunes() {
got := renderedWidth(
newHandler,
strings.Repeat(string(r), sampleRunes),
)
charged := sampleRunes *
(logfield.EncodedBytes(r) - 1)
require.LessOrEqual(
t, got-base, charged+quotingSlack,
"U+%04X costs more on the line than "+
"EncodedBytes charges for it",
r,
)
}
})
}
}
// TestTruncate_SpendsNoMoreThanTheBudget holds the result to the
// budget in ENCODED bytes, which is the unit the budget is stated in.
// A raw-byte cap passes the ASCII case here and fails every other
// one.
func TestTruncate_SpendsNoMoreThanTheBudget(t *testing.T) {
t.Parallel()
for name, fill := range map[string]string{
"plain": "x",
"quote": `"`,
"backslash": `\`,
"tab": "\t",
"newline": "\n",
"control": "\x01",
"astral": "\U0001000C",
// U+4E00, a printable multi-byte rune, charged its three
// UTF-8 bytes rather than an escape. Spelled numerically
// because gosmopolitan rejects Han in a string literal.
"cjk": string(rune(0x4E00)),
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
got := logfield.Truncate(
strings.Repeat(fill, budget*8), budget,
)
require.True(
t, strings.HasSuffix(
got, logfield.TruncationMarker,
),
"an oversized value must be marked as cut",
)
spent := 0
for _, r := range strings.TrimSuffix(
got, logfield.TruncationMarker,
) {
spent += logfield.EncodedBytes(r)
}
assert.LessOrEqual(t, spent, budget)
assert.True(t, utf8.ValidString(got))
})
}
}
// TestTruncate_LeavesShortValuesAlone keeps the marker meaningful: a
// value that fits comes back byte for byte, so a marked value is
// always a cut one.
func TestTruncate_LeavesShortValuesAlone(t *testing.T) {
t.Parallel()
for _, s := range []string{
"", "GET", "/source/abc/edit", "Mozilla/5.0 (X11)",
} {
assert.Equal(t, s, logfield.Truncate(s, budget))
}
}
// TestTruncate_DropsInvalidUTF8 covers the bytes a header can carry
// that were never valid UTF-8. Keeping them would make the encoder
// spend six bytes apiece replacing them, which is exactly the
// amplification the budget exists to prevent.
func TestTruncate_DropsInvalidUTF8(t *testing.T) {
t.Parallel()
got := logfield.Truncate("a\xffb\xfe\xfec", budget)
assert.Equal(t, "abc", got)
assert.True(t, utf8.ValidString(got))
}
// encodedCost is what a whole string costs on a line, by the same
// accounting Truncate spends its budget with.
func encodedCost(s string) int {
total := 0
for _, r := range s {
total += logfield.EncodedBytes(r)
}
return total
}
// TestTruncate_SpendsEncodedBytesNotRawBytes is the zero-headroom
// version of TestTruncate_SpendsNoMoreThanTheBudget above, and of the
// line-length assertions elsewhere.
//
// A line ceiling has slack in it by construction, and a LessOrEqual
// against the budget cannot tell a budget spent exactly from one
// spent under. Here the budget is checked against exactly what it
// bought: a value built from a single rune must keep exactly
// MaxBytes/EncodedBytes(r) of them, with nothing spare. A raw-byte
// budget — cost := utf8.RuneLen(r) — fails this for every rune the
// handlers escape.
func TestTruncate_SpendsEncodedBytesNotRawBytes(t *testing.T) {
t.Parallel()
for name, r := range map[string]rune{
"plain": 'x',
"quote": '"',
"backslash": '\\',
"tab": '\t',
"newline": '\n',
"carriage_return": '\r',
"c0_control": '\x01',
"del": '\x7f',
// U+2028 LINE SEPARATOR, which only the JSON handler
// escapes.
"line_separator": '
',
"astral_nonprintable": '\U0001000C',
"multibyte_printable": 'é',
// U+20AC, a three-byte printable rune, charged its own
// UTF-8 bytes rather than an escape.
"three_byte_printable": '€',
"emoji_printable": '\U0001F600',
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
cost := logfield.EncodedBytes(r)
want := logfield.MaxBytes / cost
// Far past the budget under either accounting.
in := strings.Repeat(string(r), logfield.MaxBytes*2)
got := logfield.Truncate(in, logfield.MaxBytes)
require.True(
t, strings.HasSuffix(
got, logfield.TruncationMarker,
),
"a value past the budget must be marked",
)
kept := strings.TrimSuffix(
got, logfield.TruncationMarker,
)
assert.Equal(
t, want, utf8.RuneCountInString(kept),
"budget bought the wrong number of runes at "+
"%d encoded bytes each", cost,
)
assert.LessOrEqual(
t, encodedCost(kept), logfield.MaxBytes,
)
})
}
}
// TestTruncate_NeverSplitsARune covers a cut landing inside a
// multi-byte encoding rather than between two of them.
func TestTruncate_NeverSplitsARune(t *testing.T) {
t.Parallel()
// U+20AC, three bytes and printable, so a small budget lands
// inside an encoding rather than on a boundary.
in := strings.Repeat("€", logfield.MaxBytes)
for b := 1; b <= 16; b++ {
got := strings.TrimSuffix(
logfield.Truncate(in, b), logfield.TruncationMarker,
)
assert.True(
t, utf8.ValidString(got),
"budget %d produced invalid UTF-8", b,
)
assert.LessOrEqual(t, encodedCost(got), b)
}
}

658
internal/middleware/accesslog_test.go Normal file
View File

@@ -0,0 +1,658 @@
package middleware_test
import (
"bytes"
"context"
"encoding/json"
"log/slog"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/go-chi/chi"
chimw "github.com/go-chi/chi/middleware"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/middleware"
)
// floodRequests is the number of distinct invented paths each flood
// test drives through the access log.
const floodRequests = 64
// attackerMarker is embedded in every invented path. No access log
// line for a redirected or rejected request may contain it.
const attackerMarker = "QQATTACKERTEXTQQ"
// maxLineBytes bounds a single access log line whose client-supplied
// fields are of ordinary size. Well above what the fixed fields need,
// well below the length of the oversized input the amplification tests
// send.
const maxLineBytes = 1024
// maxCappedLineBytes bounds a single access log line when every
// client-supplied field arrives oversized and is truncated to its
// budget. This is the number the README quotes as the per-line cost an
// operator sizes log storage against, and it is a bound on the
// ENCODED line, which is what the operator's disk holds.
const maxCappedLineBytes = 2560
// oversizedSegmentBytes is the length of the single attacker-chosen
// path segment, query string or header used to show line size does not
// track input size.
const oversizedSegmentBytes = 8192
// tailMarker is placed at the END of an oversized header value, so its
// absence from the log proves the value was truncated rather than
// merely being short.
const tailMarker = "QQTRUNCATEDTAILQQ"
// These mirror the middleware's own budgets, which are unexported.
// They are duplicated rather than exported so that widening a budget
// in the middleware has to be restated here deliberately.
const (
maxFieldBytes = 512
maxRequestIDBytes = 128
maxMethodBytes = 32
truncationSuffix = "[truncated]"
unmatchedRouteLiteral = "(unmatched)"
)
// capturingMiddleware returns a Middleware whose logger writes JSON
// lines into the returned buffer, so the access log can be asserted
// on directly.
func capturingMiddleware(t *testing.T) (*middleware.Middleware, *bytes.Buffer) {
t.Helper()
buf := new(bytes.Buffer)
log := slog.New(slog.NewJSONHandler(
buf,
&slog.HandlerOptions{Level: slog.LevelInfo},
))
cfg := &config.Config{Environment: config.EnvironmentDev}
return middleware.NewForTest(log, cfg, nil), buf
}
// capturingTextMiddleware is capturingMiddleware for the other handler
// internal/logger can select: slog's text handler, which
// internal/logger/logger.go installs when stderr is a tty. It escapes
// differently from the JSON one, so the line bound has to be asserted
// against both.
func capturingTextMiddleware(
t *testing.T,
) (*middleware.Middleware, *bytes.Buffer) {
t.Helper()
buf := new(bytes.Buffer)
log := slog.New(slog.NewTextHandler(
buf,
&slog.HandlerOptions{Level: slog.LevelInfo},
))
cfg := &config.Config{Environment: config.EnvironmentDev}
return middleware.NewForTest(log, cfg, nil), buf
}
// accessLogRouter mirrors the production route shapes that an
// unauthenticated client can reach: the public receiver, the
// authenticated profile route (which redirects to login rather than
// rejecting outright), the health check (which answers 200 to anyone,
// behind no rate limiter at all), and a plain static route.
func accessLogRouter(m *middleware.Middleware) *chi.Mux {
router := chi.NewRouter()
// Production registers RequestID ahead of Logging, and chi's
// RequestID passes an inbound X-Request-Id header straight
// through, so the request_id field is client-supplied too.
router.Use(chimw.RequestID)
router.Use(m.Logging())
router.Get(
"/.well-known/healthcheck",
func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
},
)
router.HandleFunc(
"/webhook/{uuid}",
func(w http.ResponseWriter, r *http.Request) {
// Stands in for the real handler: an unknown entrypoint
// UUID 404s, a known one succeeds.
if chi.URLParam(r, "uuid") != "known" {
http.Error(w, "not found", http.StatusNotFound)
return
}
w.WriteHeader(http.StatusOK)
},
)
router.Route("/user/{username}", func(r chi.Router) {
r.Get("/", func(w http.ResponseWriter, r *http.Request) {
http.Redirect(
w, r, "/pages/login", http.StatusSeeOther,
)
})
})
boom := func(w http.ResponseWriter, _ *http.Request) {
http.Error(w, "boom", http.StatusInternalServerError)
}
router.Get("/boom", boom)
// The 5xx branch keeps the concrete path, so it needs a route that
// answers 500 to a path of the client's choosing: that is where the
// url field and the header fields are both at their budget on the
// same line.
router.Get("/boom/*", boom)
return router
}
// accessLogEntries decodes the captured buffer into one map per
// logged line, holding every line to maxLineBytes.
func accessLogEntries(
t *testing.T,
buf *bytes.Buffer,
) []map[string]any {
t.Helper()
return accessLogEntriesWithin(t, buf, maxLineBytes)
}
// accessLogEntriesWithin decodes the captured buffer into one map per
// logged line, holding every line to bound bytes.
func accessLogEntriesWithin(
t *testing.T,
buf *bytes.Buffer,
bound int,
) []map[string]any {
t.Helper()
var entries []map[string]any
for line := range strings.SplitSeq(
strings.TrimSpace(buf.String()), "\n",
) {
if line == "" {
continue
}
require.LessOrEqual(
t, len(line), bound,
"access log line exceeded its bound",
)
var entry map[string]any
require.NoError(t, json.Unmarshal([]byte(line), &entry))
entries = append(entries, entry)
}
return entries
}
// get drives one GET through the router.
func get(t *testing.T, router *chi.Mux, target string) int {
t.Helper()
return getWithHeaders(t, router, target, nil)
}
// getWithHeaders drives one GET through the router with the supplied
// request headers set.
func getWithHeaders(
t *testing.T,
router *chi.Mux,
target string,
headers map[string]string,
) int {
t.Helper()
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, target, nil,
)
for name, value := range headers {
req.Header.Set(name, value)
}
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
return w.Code
}
// assertFloodIsBounded drives floodRequests distinct invented paths
// built by pathFor and asserts every logged line names wantURL, that
// none carries the invented text, and that the line count is exactly
// one per request.
func assertFloodIsBounded(
t *testing.T,
pathFor func(i int) string,
wantStatus int,
wantURL string,
) {
t.Helper()
m, buf := capturingMiddleware(t)
router := accessLogRouter(m)
for i := range floodRequests {
assert.Equal(t, wantStatus, get(t, router, pathFor(i)))
}
assert.NotContains(
t, buf.String(), attackerMarker,
"access log carried attacker-chosen path text",
)
entries := accessLogEntries(t, buf)
require.Len(t, entries, floodRequests)
for _, entry := range entries {
assert.Equal(t, wantURL, entry["url"])
assert.InDelta(
t, float64(wantStatus), entry["status"], 0,
)
}
}
func TestAccessLog_InventedReceiverPathsLogRoutePattern(t *testing.T) {
t.Parallel()
assertFloodIsBounded(
t,
func(i int) string {
return "/webhook/" + attackerMarker +
strings.Repeat("x", i) + "?q=" + attackerMarker
},
http.StatusNotFound,
"/webhook/{uuid}",
)
}
func TestAccessLog_InventedProfilePathsLogRoutePattern(t *testing.T) {
t.Parallel()
// The login redirect is a 3xx, not a 4xx, but it is just as free
// for an unauthenticated client to drive with invented input.
// The doubled slash is what chi's RoutePattern yields for a
// mounted subrouter's index route.
assertFloodIsBounded(
t,
func(i int) string {
return "/user/" + attackerMarker +
strings.Repeat("x", i) + "/"
},
http.StatusSeeOther,
"/user/{username}//",
)
}
func TestAccessLog_UnroutablePathsLogFixedLiteral(t *testing.T) {
t.Parallel()
assertFloodIsBounded(
t,
func(i int) string {
return "/" + attackerMarker + strings.Repeat("x", i)
},
http.StatusNotFound,
"(unmatched)",
)
}
// oversizedValue builds an 8 KB header value out of repetitions of ch,
// with the tail marker at its end.
//
// The leading 'x' is load-bearing for tab: net/textproto strips leading
// and trailing whitespace from a header value, so a value that were
// nothing but tabs would arrive empty over a real connection and the
// case would prove nothing.
func oversizedValue(ch string) string {
return "x" + strings.Repeat(ch, oversizedSegmentBytes) + tailMarker
}
// oversizedHeaders fills every client-supplied header the access log
// reads with the same value.
func oversizedHeaders(value string) map[string]string {
return map[string]string{
"User-Agent": value,
"Referer": value,
"X-Request-Id": value,
}
}
// sizeCase is one way of pointing 8 KB of client-chosen text at the
// access log.
type sizeCase struct {
target string
headers map[string]string
wantStatus int
wantURL string
bound int
}
// lineSizeCases enumerates every part of a request that reaches the
// access log, at 8 KB apiece.
func lineSizeCases() map[string]sizeCase {
cases := map[string]sizeCase{
"oversized path segment": {
target: "/webhook/" + attackerMarker +
strings.Repeat("x", oversizedSegmentBytes),
wantStatus: http.StatusNotFound,
wantURL: "/webhook/{uuid}",
bound: maxLineBytes,
},
// /.well-known/healthcheck answers 200 to anyone and has no
// rate limiter in front of it, so an oversized query appended
// to it would otherwise buy the same amplification as an
// invented 404 path, unauthenticated and unthrottled.
"oversized query on an unauthenticated 200": {
target: "/.well-known/healthcheck?q=" + attackerMarker +
strings.Repeat("x", oversizedSegmentBytes),
wantStatus: http.StatusOK,
wantURL: "/.well-known/healthcheck?(redacted)",
bound: maxLineBytes,
},
// These reach the line on every request, including one whose
// url field is correctly redacted.
"oversized headers": {
target: "/" + attackerMarker,
headers: oversizedHeaders(oversizedValue("h")),
wantStatus: http.StatusNotFound,
wantURL: unmatchedRouteLiteral,
bound: maxCappedLineBytes,
},
}
// The url field on a 5xx keeps the concrete path, so it reaches its
// own budget on the same line as the three header fields. That is
// the widest access log line the service can be made to write.
longPath := "/boom/" + strings.Repeat("x", oversizedSegmentBytes)
wantLongURL := longPath[:maxFieldBytes] + truncationSuffix
// escapeChars are the runes Go's header parser accepts in a header
// value and the log handler then escapes, coming out wider than
// they went in. A budget counted in raw bytes lets any of them buy
// a field several times its nominal size, so every one of them
// gets a case.
//
// The astral one is the case the JSON handler alone does not
// reach: U+1000C is unassigned, so it is non-printable, and
// strconv.Quote spells a non-printable rune at or above U+10000
// as a ten-byte \UXXXXXXXX. The JSON handler passes it through as
// its four UTF-8 bytes, so only the text-handler shape of this
// test holds the ten-byte charge honest.
escapeChars := map[string]string{
"quote": `"`,
"backslash": `\`,
"tab": "\t",
"astral": "\U0001000C",
}
for kind, char := range escapeChars {
fill := oversizedValue(char)
cases["oversized "+kind+" headers"] = sizeCase{
target: "/" + attackerMarker,
headers: oversizedHeaders(fill),
wantStatus: http.StatusNotFound,
wantURL: unmatchedRouteLiteral,
bound: maxCappedLineBytes,
}
cases["oversized "+kind+" headers with a 5xx concrete url"] =
sizeCase{
target: longPath,
headers: oversizedHeaders(fill),
wantStatus: http.StatusInternalServerError,
wantURL: wantLongURL,
bound: maxCappedLineBytes,
}
}
return cases
}
// TestAccessLog_LineSizeDoesNotTrackInputSize drives 8 KB of
// client-chosen text at the access log through each part of the
// request that reaches it, and holds the resulting line to a fixed
// bound in every case.
//
// The bound is on the ENCODED line, so the cases built out of
// characters the handler escapes are the ones that matter: a budget
// spent in raw bytes passes every plain-ASCII case here and still
// writes a line half again as long as the stated ceiling.
func TestAccessLog_LineSizeDoesNotTrackInputSize(t *testing.T) {
t.Parallel()
require.Equal(
t, middleware.MaxAccessLogLineBytes, maxCappedLineBytes,
"the README quotes this ceiling and the middleware derives "+
"it; they have to agree",
)
for name, tc := range lineSizeCases() {
t.Run(name, func(t *testing.T) {
t.Parallel()
m, buf := capturingMiddleware(t)
router := accessLogRouter(m)
assert.Equal(
t,
tc.wantStatus,
getWithHeaders(t, router, tc.target, tc.headers),
)
// accessLogEntriesWithin enforces the bound, which is
// orders of magnitude smaller than the input just sent.
entries := accessLogEntriesWithin(t, buf, tc.bound)
require.Len(t, entries, 1)
assert.Equal(t, tc.wantURL, entries[0]["url"])
// The markers sit at the far end of the client-chosen
// text, so their absence is what proves the redaction and
// the truncation actually ran.
assert.NotContains(
t, buf.String(), attackerMarker,
"access log carried attacker-chosen text",
)
assert.NotContains(
t, buf.String(), tailMarker,
"access log carried an untruncated client field",
)
})
}
}
// TestAccessLog_LineSizeDoesNotTrackInputSizeOnTheTextHandler runs the
// same cases through slog's text handler, which internal/logger
// selects on a tty.
//
// MaxAccessLogLineBytes is quoted to operators unqualified, so it has
// to hold for whichever handler is installed — and the two do not
// escape alike. The astral case is the one that separates them: the
// JSON handler emits U+1000C as its four UTF-8 bytes, while
// strconv.Quote spells it \U0001000C at ten. Charging six for it, as
// this code did, put a real 2,676-byte line on the wire here while
// every JSON case stayed comfortably inside the bound.
//
// Only the size bound is asserted; the url field's contents are the
// JSON shape's business above.
func TestAccessLog_LineSizeDoesNotTrackInputSizeOnTheTextHandler(
t *testing.T,
) {
t.Parallel()
for name, tc := range lineSizeCases() {
t.Run(name, func(t *testing.T) {
t.Parallel()
m, buf := capturingTextMiddleware(t)
router := accessLogRouter(m)
assert.Equal(
t,
tc.wantStatus,
getWithHeaders(t, router, tc.target, tc.headers),
)
line := strings.TrimSpace(buf.String())
require.NotEmpty(t, line)
assert.NotContains(
t, line, "\n", "expected exactly one log line",
)
require.LessOrEqual(
t, len(line), tc.bound,
"access log line exceeded its bound",
)
assert.Contains(t, line, "url=")
assert.NotContains(
t, line, attackerMarker,
"access log carried attacker-chosen text",
)
assert.NotContains(
t, line, tailMarker,
"access log carried an untruncated client field",
)
})
}
}
// TestAccessLog_OversizedMethodIsTruncated covers the last term in the
// MaxAccessLogLineBytes arithmetic that the size cases above cannot
// reach: Go accepts any RFC 7230 token as a method, and getWithHeaders
// only ever sends GET.
func TestAccessLog_OversizedMethodIsTruncated(t *testing.T) {
t.Parallel()
m, buf := capturingMiddleware(t)
router := accessLogRouter(m)
method := strings.Repeat("M", oversizedSegmentBytes) + attackerMarker
req := httptest.NewRequestWithContext(
context.Background(), method, "/"+attackerMarker, nil,
)
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
entries := accessLogEntriesWithin(t, buf, maxLineBytes)
require.Len(t, entries, 1)
assert.Equal(
t,
strings.Repeat("M", maxMethodBytes)+truncationSuffix,
entries[0]["method"],
)
assert.NotContains(
t, buf.String(), attackerMarker,
"access log carried attacker-chosen text",
)
}
// TestAccessLog_OversizedHeadersKeepATruncatedPrefix checks the other
// half of the header cap: the fields are cut, not dropped, so a
// truncated User-Agent is still worth reading.
func TestAccessLog_OversizedHeadersKeepATruncatedPrefix(t *testing.T) {
t.Parallel()
m, buf := capturingMiddleware(t)
router := accessLogRouter(m)
assert.Equal(
t,
http.StatusNotFound,
getWithHeaders(
t, router, "/nope",
oversizedHeaders(oversizedValue("h")),
),
)
entries := accessLogEntriesWithin(t, buf, maxCappedLineBytes)
require.Len(t, entries, 1)
for key, budget := range map[string]int{
"useragent": maxFieldBytes,
"referer": maxFieldBytes,
"request_id": maxRequestIDBytes,
} {
value, ok := entries[0][key].(string)
require.True(t, ok, key)
assert.LessOrEqual(
t, len(value), budget+len(truncationSuffix), key,
)
assert.Contains(t, value, truncationSuffix, key)
assert.Contains(t, value, "hhhh", key)
}
}
func TestAccessLog_SuccessKeepsConcretePathAndRedactsQuery(
t *testing.T,
) {
t.Parallel()
m, buf := capturingMiddleware(t)
router := accessLogRouter(m)
assert.Equal(
t, http.StatusOK, get(t, router, "/webhook/known?src=ci"),
)
// The path resolved against a stored entrypoint, so it stays. The
// query never does: see TestAccessLog_UnauthenticatedSuccess...
entries := accessLogEntries(t, buf)
require.Len(t, entries, 1)
assert.Equal(t, "/webhook/known?(redacted)", entries[0]["url"])
assert.NotContains(t, buf.String(), "src=ci")
}
func TestAccessLog_ServerErrorKeepsConcreteURL(t *testing.T) {
t.Parallel()
m, buf := capturingMiddleware(t)
router := accessLogRouter(m)
assert.Equal(
t, http.StatusInternalServerError, get(t, router, "/boom"),
)
entries := accessLogEntries(t, buf)
require.Len(t, entries, 1)
assert.Equal(t, "/boom", entries[0]["url"])
}
func TestAccessLog_RetainsEveryOtherField(t *testing.T) {
t.Parallel()
m, buf := capturingMiddleware(t)
router := accessLogRouter(m)
assert.Equal(
t,
http.StatusNotFound,
get(t, router, "/webhook/"+attackerMarker),
)
entries := accessLogEntries(t, buf)
require.Len(t, entries, 1)
for _, key := range []string{
"request_start", "method", "url", "useragent", "request_id",
"referer", "proto", "remoteIP", "status", "latency_ms",
} {
assert.Contains(t, entries[0], key)
}
assert.Equal(t, http.MethodGet, entries[0]["method"])
assert.Equal(t, "HTTP/1.1", entries[0]["proto"])
}

18
internal/middleware/csrf.go
View File

@@ -4,6 +4,7 @@ import (
"net/http"
"github.com/gorilla/csrf"
"sneak.berlin/go/webhooker/internal/logfield"
)
// CSRFToken retrieves the CSRF token from the request context.
@@ -42,9 +43,22 @@ func isClientTLS(r *http.Request) bool {
// csrf.Secure option is set at creation time, not per-request.
func (m *Middleware) CSRF() func(http.Handler) http.Handler {
csrfErrorHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// CSRF is registered ahead of RequireAuth on every route
// group that uses it, so this WARN is reachable by an
// unauthenticated client: a POST with no token to
// /source/<any length of any text>/edit lands here. The
// method and path are capped against the same budgets as
// the access log. remote_addr is set by net/http from the
// accepted connection rather than by the client, and
// csrf.FailureReason returns one of gorilla/csrf's own
// fixed error values, so neither is client-sized.
m.log.Warn("csrf: token validation failed",
"method", r.Method,
"path", r.URL.Path,
"method", logfield.Truncate(
r.Method, maxLogMethodBytes,
),
"path", logfield.Truncate(
r.URL.Path, logfield.MaxBytes,
),
"remote_addr", r.RemoteAddr,
"reason", csrf.FailureReason(r),
)

93
internal/middleware/export_test.go
View File

@@ -1,7 +1,9 @@
package middleware
import (
"context"
"net/http"
"time"
)
// NewLoggingResponseWriterForTest wraps newLoggingResponseWriter
@@ -25,10 +27,99 @@ func IPFromHostPort(hp string) string {
return ipFromHostPort(hp)
}
// ClientKeyForTest exposes clientKey for testing.
func ClientKeyForTest(m *Middleware, r *http.Request) string {
return m.clientKey(r)
}
// IsClientTLS exposes isClientTLS for testing.
func IsClientTLS(r *http.Request) bool {
return isClientTLS(r)
}
// LoginRateLimitConst exposes the loginRateLimit constant.
// LoginRateLimitConst exposes the loginRateLimit constant: the
// number of FAILED login attempts one client may make against one
// submitted username per interval.
const LoginRateLimitConst = loginRateLimit
// LoginFailureMaxKeysConst exposes the cap on each of the login
// guard's key sets.
const LoginFailureMaxKeysConst = loginFailureMaxKeys
// PasswordVerifyConcurrencyConst exposes the bound on concurrent
// Argon2id verifications.
const PasswordVerifyConcurrencyConst = passwordVerifyConcurrency
// PasswordVerifyMaxWaitersConst exposes the bound on how many
// requests may queue for a verification slot.
const PasswordVerifyMaxWaitersConst = passwordVerifyMaxWaiters
// LoginGuard is the login failure counter and verification
// semaphore, exposed for direct testing.
type LoginGuard = loginGuard
// NewLoginGuardForTest builds a guard with test-sized parameters.
func NewLoginGuardForTest(
limit int,
interval time.Duration,
maxKeys, concurrency, maxWaiters int,
wait time.Duration,
) *LoginGuard {
return newLoginGuard(
limit, interval, maxKeys, concurrency, maxWaiters, wait,
)
}
// QueuedWaitersForTest reports how many requests are currently
// queued for a verification slot.
func (g *LoginGuard) QueuedWaitersForTest() int {
return len(g.queue)
}
// SetNowForTest replaces the guard's clock.
func (g *LoginGuard) SetNowForTest(now func() time.Time) {
g.mu.Lock()
defer g.mu.Unlock()
g.now = now
}
// FailForTest exposes fail.
func (g *LoginGuard) FailForTest(clientKey, username string) bool {
return g.fail(clientKey, username)
}
// SucceedForTest exposes succeed.
func (g *LoginGuard) SucceedForTest(clientKey, username string) {
g.succeed(clientKey, username)
}
// AcquireForTest exposes acquire.
func (g *LoginGuard) AcquireForTest(
ctx context.Context,
) (func(), bool) {
return g.acquire(ctx)
}
// TrackedKeysForTest reports how many failure counters the guard
// holds, per-username and per-address respectively.
func (g *LoginGuard) TrackedKeysForTest() (int, int) {
g.mu.Lock()
defer g.mu.Unlock()
return len(g.byUser), len(g.byAddr)
}
// PasswordChangeRateLimitConst exposes the
// passwordChangeRateLimit constant.
const PasswordChangeRateLimitConst = passwordChangeRateLimit
// ReceiverAggregateMultiplierConst exposes the
// receiverAggregateMultiplier constant.
const ReceiverAggregateMultiplierConst = receiverAggregateMultiplier
// ReceiverAggregateLimitForTest exposes receiverAggregateLimit for
// testing.
func ReceiverAggregateLimitForTest(perEntrypoint int) int {
return receiverAggregateLimit(perEntrypoint)
}

544
internal/middleware/logbound_test.go Normal file
View File

@@ -0,0 +1,544 @@
package middleware_test
// This file covers the log lines OUTSIDE the access log that carry a
// client-chosen value. accesslog_test.go bounds the one INFO line the
// Logging middleware writes; these are the separate slog calls that
// were never in that sweep and so never got the budget:
//
// - MaxBodySize's 413 rejection, at WARN, registered ahead of
// RequireAuth and therefore reachable unauthenticated at a URL of
// the client's choosing.
// - CSRF's 403 rejection, at WARN, also registered ahead of
// RequireAuth.
// - The rate limiters' 429 rejection, at WARN, on the
// unauthenticated receiver among others.
// - RequireAuth's own unauthenticated-request line, at DEBUG.
// - RecordLoginFailure's throttle rejection, at WARN. Its cap is
// defensive rather than load-bearing today: chi pins the one
// route that calls it to the constant path "/pages/login". The
// method is exported and takes any *http.Request, so the test
// below hands it the request a caller on a parameterised route
// would, which is what the cap exists for.
//
// Every case here holds the ENCODED line to
// middleware.MaxAccessLogLineBytes, under both handlers
// internal/logger can install, against 8 KB of client-chosen text
// built out of the characters those handlers escape. A budget spent
// in raw bytes passes the plain-ASCII cases and fails the rest.
import (
"bytes"
"context"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/middleware"
)
// bodyLimitBytes is the MaxBodySize cap these tests install. Any
// declared Content-Length above it takes the 413 branch.
const bodyLimitBytes = 1024
// declaredBodyBytes is the Content-Length an oversize request
// declares. Nothing is actually sent: the 413 branch fires off the
// declaration alone, which is what makes the attack free.
const declaredBodyBytes = bodyLimitBytes * 2
// receiverLimitPerMinute is the per-entrypoint receiver limit these
// tests install. The aggregate limiter sits at ten times this, so a
// flood stays under it and the rejections come from the
// per-entrypoint limiter, which is the one that logs the path.
const receiverLimitPerMinute = 8
// escapeFills are the characters a client can put in a request that
// the log handlers then escape, coming out wider than they went in.
// A budget counted in raw bytes lets any of them buy a field several
// times its nominal size.
//
// U+1000C is the case the JSON handler alone does not reach: it is
// unassigned, so it is non-printable, and strconv.Quote spells a
// non-printable rune at or above U+10000 as a ten-byte \UXXXXXXXX
// while the JSON handler passes its four UTF-8 bytes through. Only
// the text-handler shape of these tests holds that charge honest.
func escapeFills() map[string]string {
return map[string]string{
"plain": "x",
"quote": `"`,
"backslash": `\`,
"tab": "\t",
"newline": "\n",
// A C0 control neither handler has a short escape for, so
// each one costs six bytes on the line against the single
// byte it cost to send. This is the widest multiplier a
// client can drive, and the case a raw-byte budget breaks
// on first.
//
// This fill is load-bearing, not decoration. Budgeting raw
// bytes instead of encoded is caught by this fill alone,
// and only under the JSON handler, at 3,072 bytes against
// the 2,560 ceiling. Drop it and that mutation passes.
"control": "\x01",
"astral": "\U0001000C",
}
}
// logHandlers are the two handlers internal/logger can install: the
// JSON one, and the text one it selects when stderr is a tty. They do
// not escape alike, and MaxAccessLogLineBytes is quoted unqualified,
// so every case runs through both.
func logHandlers() map[string]func(
io.Writer, *slog.HandlerOptions,
) slog.Handler {
return map[string]func(
io.Writer, *slog.HandlerOptions,
) slog.Handler{
"json": func(
w io.Writer, o *slog.HandlerOptions,
) slog.Handler {
return slog.NewJSONHandler(w, o)
},
"text": func(
w io.Writer, o *slog.HandlerOptions,
) slog.Handler {
return slog.NewTextHandler(w, o)
},
}
}
// oversizedPathSegment builds an 8 KB client-chosen path segment out
// of repetitions of ch, percent-encoded so it survives URL parsing
// into r.URL.Path the way it would arriving off a socket.
//
// Both markers sit at the END, past every budget, so their absence
// from the log is what proves the value was cut rather than merely
// being short. The leading 'x' keeps the segment non-empty for fills
// that a parser might otherwise fold away.
func oversizedPathSegment(ch string) string {
return url.PathEscape(
"x" + strings.Repeat(ch, oversizedSegmentBytes) +
attackerMarker + tailMarker,
)
}
// capturingLogger returns a logger at DEBUG writing into the returned
// buffer through the named handler.
func capturingLogger(
newHandler func(io.Writer, *slog.HandlerOptions) slog.Handler,
) (*slog.Logger, *bytes.Buffer) {
buf := new(bytes.Buffer)
opts := &slog.HandlerOptions{Level: slog.LevelDebug}
return slog.New(newHandler(buf, opts)), buf
}
// capturingBoundMiddleware builds a Middleware with a real session
// manager (CSRF needs its key, RequireAuth needs its store) whose log
// is captured at DEBUG.
func capturingBoundMiddleware(
t *testing.T,
newHandler func(io.Writer, *slog.HandlerOptions) slog.Handler,
) (*middleware.Middleware, *bytes.Buffer) {
t.Helper()
log, buf := capturingLogger(newHandler)
cfg := &config.Config{
Environment: config.EnvironmentDev,
ReceiverRateLimit: receiverLimitPerMinute,
}
sess := newTestSessionManager(cfg, log, nil)
return middleware.NewForTest(log, cfg, sess), buf
}
// unreachable is a next-handler that fails the test if the middleware
// under test let the request through. Every site here rejects.
func unreachable(t *testing.T) http.Handler {
t.Helper()
return http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
assert.Fail(t, "rejected request reached the next handler")
})
}
// logSite is one non-access-log call site that logs a client-chosen
// path. drive sends requests at it that all take the rejecting
// branch; linesPerRequest is how many log lines one such request
// produces there.
type logSite struct {
// build wraps the site's middleware around a handler that must
// not be reached.
build func(
t *testing.T, m *middleware.Middleware,
) http.Handler
// send issues one request for the given client-chosen path and
// returns the status. Some sites need a warm-up request before
// they reject, which send performs itself.
send func(h http.Handler, path string) int
// wantStatus is the status the rejecting branch answers with.
wantStatus int
}
// postOversize sends a POST whose declared Content-Length exceeds the
// body limit without sending a body, which is the whole cost of the
// attack on the MaxBodySize branch.
func postOversize(h http.Handler, path string) int {
req := httptest.NewRequestWithContext(
context.Background(), http.MethodPost, path, nil,
)
req.ContentLength = declaredBodyBytes
req.Header.Set(
"Content-Type", "application/x-www-form-urlencoded",
)
w := httptest.NewRecorder()
h.ServeHTTP(w, req)
return w.Code
}
// postNoToken sends a POST carrying no CSRF token and no session
// cookie, which is what an unauthenticated client sends.
func postNoToken(h http.Handler, path string) int {
req := httptest.NewRequestWithContext(
context.Background(), http.MethodPost, path,
strings.NewReader(""),
)
req.Header.Set(
"Content-Type", "application/x-www-form-urlencoded",
)
w := httptest.NewRecorder()
h.ServeHTTP(w, req)
return w.Code
}
// getNoSession sends a GET with no session cookie.
func getNoSession(h http.Handler, path string) int {
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, path, nil,
)
w := httptest.NewRecorder()
h.ServeHTTP(w, req)
return w.Code
}
// logSites enumerates the call sites under test.
func logSites() map[string]logSite {
return map[string]logSite{
// The site this file exists for: WARN, on by default, and
// registered ahead of RequireAuth.
"maxbodysize 413": {
build: func(
t *testing.T, m *middleware.Middleware,
) http.Handler {
t.Helper()
return m.MaxBodySize(bodyLimitBytes)(
unreachable(t),
)
},
send: postOversize,
wantStatus: http.StatusRequestEntityTooLarge,
},
// Also ahead of RequireAuth, also WARN.
"csrf 403": {
build: func(
t *testing.T, m *middleware.Middleware,
) http.Handler {
t.Helper()
return m.CSRF()(unreachable(t))
},
send: postNoToken,
wantStatus: http.StatusForbidden,
},
// The per-entrypoint receiver limiter, unauthenticated. Its
// bucket is keyed on the path, so the first request through a
// fresh path is served and only the ones after it are
// rejected; sendUntilLimited absorbs that.
"receiver rate limit 429": {
build: func(
t *testing.T, m *middleware.Middleware,
) http.Handler {
t.Helper()
return m.ReceiverRateLimit()(okHandler())
},
send: sendUntilLimited,
wantStatus: http.StatusTooManyRequests,
},
// RequireAuth's own line. DEBUG is off in production by
// default, but turning it on to diagnose a flood must not
// restore an unbounded write.
"requireauth redirect": {
build: func(
t *testing.T, m *middleware.Middleware,
) http.Handler {
t.Helper()
return m.RequireAuth()(unreachable(t))
},
send: getNoSession,
wantStatus: http.StatusSeeOther,
},
}
}
// sendUntilLimited drives the per-entrypoint receiver limiter past
// its allowance on one path and returns the status of the rejected
// request. Every request before the last is served, and only the last
// one logs.
func sendUntilLimited(h http.Handler, path string) int {
code := http.StatusOK
for range receiverLimitPerMinute + 1 {
req := httptest.NewRequestWithContext(
context.Background(), http.MethodPost, path, nil,
)
req.RemoteAddr = "203.0.113.7:5555"
w := httptest.NewRecorder()
h.ServeHTTP(w, req)
code = w.Code
}
return code
}
// logLines splits the captured buffer into non-empty lines, holding
// each to bound bytes.
func logLines(t *testing.T, buf *bytes.Buffer, bound int) []string {
t.Helper()
var lines []string
for line := range strings.SplitSeq(
strings.TrimSpace(buf.String()), "\n",
) {
if line == "" {
continue
}
require.LessOrEqual(
t, len(line), bound,
"log line exceeded its bound: %s", line,
)
lines = append(lines, line)
}
return lines
}
// assertNoClientText fails if any marker from the far end of the
// client-chosen input survived into the log. Their absence is what
// distinguishes a real cut from a value that merely happened to be
// short.
func assertNoClientText(t *testing.T, buf *bytes.Buffer) {
t.Helper()
assert.NotContains(
t, buf.String(), attackerMarker,
"log carried attacker-chosen text",
)
assert.NotContains(
t, buf.String(), tailMarker,
"log carried the tail of the attacker-chosen text",
)
}
// TestLogLines_ClientChosenPathDoesNotSizeTheLine points 8 KB of
// client-chosen path at each non-access-log call site that logs one,
// through both handlers and through every character those handlers
// escape, and holds the resulting line to MaxAccessLogLineBytes.
//
// Removing any one of the logfield.Truncate calls at those sites
// fails this test: the line grows to roughly the size of the input,
// or to several times it on the escaping fills.
func TestLogLines_ClientChosenPathDoesNotSizeTheLine(t *testing.T) {
t.Parallel()
for siteName, site := range logSites() {
for handlerName, newHandler := range logHandlers() {
for fillName, fill := range escapeFills() {
name := siteName + "/" + handlerName + "/" + fillName
t.Run(name, func(t *testing.T) {
t.Parallel()
m, buf := capturingBoundMiddleware(
t, newHandler,
)
path := "/source/" +
oversizedPathSegment(fill) + "/edit"
assert.Equal(
t,
site.wantStatus,
site.send(site.build(t, m), path),
)
lines := logLines(
t, buf,
middleware.MaxAccessLogLineBytes,
)
require.NotEmpty(
t, lines,
"the site under test logged nothing, "+
"so the bound proves nothing",
)
assertNoClientText(t, buf)
})
}
}
}
}
// TestLoginThrottle_LogLineDoesNotTrackPathSize pins the cap on
// RecordLoginFailure's "login failure limit exceeded" WARN line.
//
// That site does not fit logSites above: it is not a middleware
// wrapping a handler but an exported method the login handler calls,
// and the only route that calls it today is chi's static
// "/pages/login", so no request through the mux can widen the line.
// Driving the method directly is therefore the whole point rather
// than a shortcut — it is exactly the call a second caller on a route
// with a URL parameter would make, and without this test removing the
// logfield.Truncate there fails nothing.
func TestLoginThrottle_LogLineDoesNotTrackPathSize(t *testing.T) {
t.Parallel()
for handlerName, newHandler := range logHandlers() {
for fillName, fill := range escapeFills() {
t.Run(handlerName+"/"+fillName, func(t *testing.T) {
t.Parallel()
m, buf := capturingBoundMiddleware(
t, newHandler,
)
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost,
"/source/"+
oversizedPathSegment(fill)+"/login",
nil,
)
req.RemoteAddr = "203.0.113.9:5555"
// The budget is spent per client and username,
// so one more failure than the budget allows is
// what takes the throttled branch.
var throttled bool
for range middleware.LoginRateLimitConst + 1 {
throttled = m.RecordLoginFailure(
req, "someone",
)
}
require.True(
t, throttled,
"the throttled branch never ran, so the "+
"bound proves nothing",
)
lines := logLines(
t, buf, middleware.MaxAccessLogLineBytes,
)
require.NotEmpty(t, lines)
assertNoClientText(t, buf)
})
}
}
}
// TestMaxBodySize_FloodOfOversizePathsDoesNotGrowTheLog is the
// flood shape from the issue: an unauthenticated client posting
// oversize declarations at invented 8 KB paths, as fast as it likes.
//
// It asserts the property directly rather than by proxy — the bytes
// the flood writes to the operator's log do not track the bytes the
// flood sent. The same flood at a one-character path is the control:
// 8 KB of extra input per request buys at most the field budget, not
// 8 KB of log.
func TestMaxBodySize_FloodOfOversizePathsDoesNotGrowTheLog(
t *testing.T,
) {
t.Parallel()
for handlerName, newHandler := range logHandlers() {
for fillName, fill := range escapeFills() {
t.Run(handlerName+"/"+fillName, func(t *testing.T) {
t.Parallel()
flood := func(segment func(i int) string) int {
m, buf := capturingBoundMiddleware(
t, newHandler,
)
h := m.MaxBodySize(bodyLimitBytes)(
unreachable(t),
)
for i := range floodRequests {
assert.Equal(
t,
http.StatusRequestEntityTooLarge,
postOversize(
h,
"/source/"+segment(i)+"/edit",
),
)
}
lines := logLines(
t, buf,
middleware.MaxAccessLogLineBytes,
)
require.Len(t, lines, floodRequests)
assertNoClientText(t, buf)
return buf.Len()
}
sent := oversizedSegmentBytes * floodRequests
oversize := flood(func(i int) string {
return oversizedPathSegment(fill) +
strings.Repeat("y", i)
})
control := flood(func(i int) string {
return "a" + strings.Repeat("y", i)
})
// The whole point: 8 KB per request of extra
// client-chosen input bought a bounded amount of
// log, not a proportional amount.
assert.Less(
t, oversize-control, sent/2,
"log volume tracked the size of the flood's "+
"input",
)
assert.LessOrEqual(
t,
oversize,
floodRequests*
middleware.MaxAccessLogLineBytes,
)
})
}
}
}

409
internal/middleware/loginguard.go Normal file
View File

@@ -0,0 +1,409 @@
package middleware
import (
"context"
"crypto/sha256"
"encoding/hex"
"net/http"
"sync"
"time"
"sneak.berlin/go/webhooker/internal/logfield"
)
const (
// loginFailureMaxKeys bounds how many distinct failure counters
// each of the guard's two key sets holds. The submitted username
// is part of a key, so the key set is attacker-influenced and
// needs a hard cap or the limiter becomes the memory
// amplification surface it exists to protect.
//
// A single-admin deployment has a handful of legitimate (client,
// username) pairs, so 1024 is three orders of magnitude of
// headroom before a real operator can be pushed onto the
// fallback. It costs little: a counter is a ~64-byte key string,
// a 32-byte window and map overhead, call it 170 bytes, so both
// key sets full is 2 * 1024 * 170 bytes, under 0.4 MB.
loginFailureMaxKeys = 1024
// passwordVerifyConcurrency bounds how many Argon2id
// verifications may run at once across every password-verifying
// endpoint. Because credentials are now verified before any
// limiter budget is spent, an attacker can force one hash per
// request, and each hash allocates argon2Memory — 64 MB. Two
// slots commit at most 128 MB to password hashing, which fits
// inside the smallest container this service is realistically
// given alongside its own working set; four would commit 256 MB
// and crowd it. A single-admin product needs no concurrent
// logins at all, so the second slot exists only so that one
// stalled request does not serialise the endpoint.
passwordVerifyConcurrency = 2
// passwordVerifyWait is how long a request waits for a
// verification slot before it is answered 503. Slots are handed
// out in arrival order, so a legitimate request queues behind
// the requests already waiting rather than behind the flood as a
// whole. The wait is well inside the 60s request timeout.
passwordVerifyWait = 5 * time.Second
// passwordVerifyMaxWaiters bounds how many requests may be
// queued for a slot at once. Past it, acquire sheds immediately
// with 503 instead of joining the queue.
//
// The wait bounds how long one request occupies memory; this
// bounds how many do so at the same time, and without it the
// 128 MB hashing budget above is the smaller half of the real
// footprint. At the 400 req/s a saturation attack can offer, an
// unbounded queue would park ~2000 requests for the full five
// seconds.
//
// A waiter costs far more than maxFormBodySize suggests: that
// caps the raw body read, not what the parse retains. MaxBodySize,
// CSRF and ParseForm all run before acquire, so a parked waiter
// holds r.Form plus r.PostForm plus its header block for the
// whole wait. Measured on the pinned go1.26.1 toolchain, as the
// HeapAlloc delta across two GCs with 64 waiters parked in the
// handler: an ordinary two-field login form retains ~0 MB, but a
// 1 MB urlencoded body at Go's 10,000-parameter parse cap retains
// 2.82 MB (3.09 MB with %41 escapes), and adding the ~0.9 MB of
// headers httpMaxHeaderBytes allows takes it to 4.18 MB. The
// retained parse and the header block dominate; the raw body does
// not.
//
// Arithmetic, from the measured 4.18 MB worst case: 16 waiters
// commit ~67 MB of queue memory, and peak commitment for the
// endpoint is 128 MB of Argon2id plus the 18 requests that retain
// a parsed form — 16 queued and the 2 being hashed — at
// 18 * 4.18 MB, so ~75 MB: about 203 MB in all. Cross-check
// against the deadline: two slots at the ~27 verifications/s
// measured on a review host (with the race detector on, so the
// real rate is higher) drain a full 16-deep queue in about 0.6 s,
// far inside passwordVerifyWait.
//
// Those 203 MB are live bytes, not resident bytes: the Go
// collector lets the heap reach roughly twice the live set before
// collecting, with transient parse garbage on top. The review
// measured a peak HeapAlloc of 392 MB against this guard under 18
// adversarial requests, so provision on the order of 400 MB rather
// than 203 MB.
passwordVerifyMaxWaiters = 16
// failureKeyHashBytes is how much of the username digest goes
// into a failure key. 64 bits over at most loginFailureMaxKeys
// live keys makes a collision negligible, and a collision would
// only merge two usernames' failure counters, which throttles
// sooner rather than later.
failureKeyHashBytes = 8
)
// failureWindow counts failed credential verifications for one
// bucket, and records when that count lapses.
type failureWindow struct {
count int
resetAt time.Time
}
// loginGuard is what replaced the pre-emptive rate limiter on the
// login POST.
//
// A limiter that spends budget on arrival cannot protect a
// single-admin product: behind the reverse proxy the deployment
// requires, with TRUSTED_PROXIES unset, every client keys on the
// proxy, so a stranger trickling five POSTs a minute keeps the one
// bucket full and the operator's own correct password is answered 429
// forever. There is no second administrative path.
//
// So budget is spent only by a FAILED verification. A correct
// password is never throttled, whatever the counters say, which is
// the only shape that guarantees the operator can get in. Two
// consequences follow and are handled here:
//
// - Every login request now costs an Argon2id hash, so the number
// running concurrently is bounded by slots. Without that bound
// this trades an admin lockout for memory exhaustion, which is
// strictly worse.
// - Counting per (client, username) makes the key set
// attacker-influenced, so both key sets are capped. Beyond the
// per-username cap, failures fall back to a counter keyed on the
// client alone; beyond that cap too, a failure is answered as
// throttled without being recorded, since refusing to answer a
// wrong password costs the operator nothing.
type loginGuard struct {
mu sync.Mutex
byUser map[string]*failureWindow
byAddr map[string]*failureWindow
slots chan struct{}
// queue holds one token per request waiting for a slot. A token
// is taken non-blockingly, so a request that finds it full is
// shed rather than queued, and is given up as soon as the wait
// ends however it ends.
queue chan struct{}
limit int
interval time.Duration
maxKeys int
wait time.Duration
// now is time.Now outside tests.
now func() time.Time
}
// newLoginGuard builds a guard with the given failure limit per
// interval, key-set cap, verification concurrency, queue depth and
// slot wait.
func newLoginGuard(
limit int,
interval time.Duration,
maxKeys, concurrency, maxWaiters int,
wait time.Duration,
) *loginGuard {
return &loginGuard{
byUser: make(map[string]*failureWindow),
byAddr: make(map[string]*failureWindow),
slots: make(chan struct{}, concurrency),
queue: make(chan struct{}, maxWaiters),
limit: limit,
interval: interval,
maxKeys: maxKeys,
wait: wait,
now: time.Now,
}
}
// acquire reserves a verification slot, waiting up to the guard's
// wait for one. It reports false when the queue of waiters is
// already full, when no slot became available in time, or when the
// request was cancelled while waiting; the caller must then answer
// 503 without verifying anything. The returned function releases the
// slot and must be called exactly once.
//
// ctx is consulted only once the request has to wait: a slot that is
// free on arrival is handed out without looking at it, so an
// already-cancelled request can be granted one. That is deliberate
// and matches lifecycle.waitDone — the caller abandons the work on
// its own ctx and releases the slot immediately, so nothing is spent
// on it, and refusing instead would mean shedding a request with
// capacity standing free.
func (g *loginGuard) acquire(ctx context.Context) (func(), bool) {
// A free slot is taken before any timer is armed, and before a
// queue place is claimed: a request that never waits is not a
// waiter. Without this preamble the bounded select below can find
// its slot send and an already-expired timer ready at the same
// time, and Go picks among ready cases uniformly at random — so a
// process descheduled for longer than the wait sheds a request
// with slots standing free, which is precisely when shedding is
// least defensible.
//
// This cannot let a late arrival barge past a queued waiter. A
// waiter can only be parked on a FULL buffer, and a release
// refills that buffer from the head of the send queue under the
// channel lock, so the buffer never appears non-full while anyone
// is parked and this send fails whenever there is a waiter.
select {
case g.slots <- struct{}{}:
return func() { <-g.slots }, true
default:
}
// Shedding past the queue depth is what keeps waiting memory
// bounded; the wait alone only bounds how long one waiter holds
// its parsed form, not how many hold one at once.
select {
case g.queue <- struct{}{}:
default:
return nil, false
}
// Held only for the wait. A request that gets a slot gives its
// queue token back before it starts hashing, so the depth is a
// bound on waiters rather than on requests in the handler.
defer func() { <-g.queue }()
timer := time.NewTimer(g.wait)
defer timer.Stop()
// The blocking send is deliberate: a receive on a full buffered
// channel hands the slot straight to the head of the send queue,
// so slots go out in arrival order and a later arrival cannot
// barge past a request already waiting.
select {
case g.slots <- struct{}{}:
return func() { <-g.slots }, true
case <-timer.C:
return nil, false
case <-ctx.Done():
return nil, false
}
}
// fail records one failed credential verification by clientKey
// against username, and reports whether this client has now spent
// its failure budget and should be answered 429.
func (g *loginGuard) fail(clientKey, username string) bool {
g.mu.Lock()
defer g.mu.Unlock()
now := g.now()
window := g.window(
g.byUser, userFailureKey(clientKey, username), now,
)
if window == nil {
window = g.window(g.byAddr, clientKey, now)
}
if window == nil {
// Both key sets are full and neither already tracks this
// client, so nothing can be counted without unbounded
// growth. Answering the failure as throttled is the safe
// direction: it never touches a correct password.
return true
}
window.count++
return window.count >= g.limit
}
// succeed forgives clientKey's failures against username. A correct
// password clears the counters, so an operator who mistypes several
// times and then gets it right is not throttled afterwards.
func (g *loginGuard) succeed(clientKey, username string) {
g.mu.Lock()
defer g.mu.Unlock()
delete(g.byUser, userFailureKey(clientKey, username))
delete(g.byAddr, clientKey)
}
// window returns the live counter for key in set, resetting a lapsed
// one and creating a missing one when the cap allows. It returns nil
// only when key is absent and set is full even after lapsed entries
// are swept.
func (g *loginGuard) window(
set map[string]*failureWindow,
key string,
now time.Time,
) *failureWindow {
window, ok := set[key]
if ok {
if !now.Before(window.resetAt) {
window.count = 0
window.resetAt = now.Add(g.interval)
}
return window
}
if len(set) >= g.maxKeys {
sweepLapsed(set, now)
}
if len(set) >= g.maxKeys {
return nil
}
window = &failureWindow{resetAt: now.Add(g.interval)}
set[key] = window
return window
}
// sweepLapsed drops counters whose interval has elapsed.
func sweepLapsed(set map[string]*failureWindow, now time.Time) {
for key, window := range set {
if !now.Before(window.resetAt) {
delete(set, key)
}
}
}
// userFailureKey identifies one (client, submitted username) pair.
// The username is hashed rather than embedded: a submitted username
// is attacker-controlled text of attacker-chosen length, and hashing
// makes every key the same size whatever was sent.
func userFailureKey(clientKey, username string) string {
sum := sha256.Sum256([]byte(username))
return clientKey + "|" +
hex.EncodeToString(sum[:failureKeyHashBytes])
}
// guard returns the middleware's login guard, building it on first
// use so that every construction path — fx and the test constructor
// alike — gets one.
func (m *Middleware) guard() *loginGuard {
m.loginGuardOnce.Do(func() {
m.loginGuard = newLoginGuard(
loginRateLimit,
loginRateInterval,
loginFailureMaxKeys,
passwordVerifyConcurrency,
passwordVerifyMaxWaiters,
passwordVerifyWait,
)
})
return m.loginGuard
}
// BeginPasswordVerification reserves one of the bounded Argon2id
// verification slots. It reports false when the queue of waiting
// requests is already at passwordVerifyMaxWaiters, or when no slot
// became free within passwordVerifyWait; in either case the caller
// must answer 503 and must not verify a password. The returned
// function releases the slot and must be called exactly once.
//
// Every endpoint that hashes a password on request must go through
// this, or the bound has a hole: the memory is committed per hash,
// not per endpoint.
func (m *Middleware) BeginPasswordVerification(
ctx context.Context,
) (func(), bool) {
return m.guard().acquire(ctx)
}
// RecordLoginFailure counts a failed credential verification for the
// request's client against the submitted username, and reports
// whether the response should be 429 rather than 401.
func (m *Middleware) RecordLoginFailure(
r *http.Request,
username string,
) bool {
throttled := m.guard().fail(m.clientKey(r), username)
if throttled {
// Truncated even though chi pins this route's path to
// the 12-byte constant "/pages/login": RecordLoginFailure
// is exported and takes any *http.Request, so a caller on
// a route with a URL parameter would otherwise widen this
// line. logbound_test.go pins the cap by making exactly
// that call, since no request through the mux can.
m.log.Warn(
"login failure limit exceeded",
"path", logfield.Truncate(
r.URL.Path, logfield.MaxBytes,
),
)
}
return throttled
}
// ForgiveLoginFailures clears the failure counters for the request's
// client and the submitted username after a successful
// authentication.
func (m *Middleware) ForgiveLoginFailures(
r *http.Request,
username string,
) {
m.guard().succeed(m.clientKey(r), username)
}
// LoginFailureInterval is how long a spent login failure budget
// takes to refill, which is what a throttled login answers as
// Retry-After.
func (m *Middleware) LoginFailureInterval() time.Duration {
return m.guard().interval
}

629
internal/middleware/loginguard_test.go Normal file
View File

@@ -0,0 +1,629 @@
package middleware_test
import (
"context"
"fmt"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/middleware"
)
// mib converts the Argon2id memory parameter, which is in KiB, to MB.
const mib = 1024
const (
// guardInterval is the failure window these tests use. It is
// long enough that nothing lapses mid-test on its own; tests
// that need a lapse drive the clock instead.
guardInterval = time.Minute
// guardWait is the slot wait for tests that expect to get a
// slot. Tests that expect to be refused set their own.
guardWait = 2 * time.Second
guardClient = "198.51.100.7"
guardUser = "admin"
// racePasses is how many times a both-cases-ready select race is
// run. A pass can only go the wrong way once the zero-duration
// timer has fired, so the per-pass detection probability is
// somewhere below 1/2 rather than exactly it; the bound that
// matters is that passes are independent, so a regression that
// survives is exponentially unlikely in N. The test still waits
// on nothing.
racePasses = 1000
)
// newGuard builds a guard with production-shaped defaults and the
// given key-set cap and verification concurrency.
func newGuard(maxKeys, concurrency int) *middleware.LoginGuard {
return middleware.NewLoginGuardForTest(
middleware.LoginRateLimitConst,
guardInterval,
maxKeys,
concurrency,
middleware.PasswordVerifyMaxWaitersConst,
guardWait,
)
}
// TestLoginGuard_ThrottlesRepeatedFailures is the brute-force half:
// wrong passwords for one username from one client key still run out
// of budget and are answered 429.
func TestLoginGuard_ThrottlesRepeatedFailures(t *testing.T) {
t.Parallel()
g := newGuard(middleware.LoginFailureMaxKeysConst, 1)
for i := range middleware.LoginRateLimitConst - 1 {
assert.False(
t, g.FailForTest(guardClient, guardUser),
"failure %d is still inside the budget", i,
)
}
assert.True(
t, g.FailForTest(guardClient, guardUser),
"the last failure of the budget must throttle",
)
assert.True(
t, g.FailForTest(guardClient, guardUser),
"failures past the budget must stay throttled",
)
}
// TestLoginGuard_SuccessForgivesFailures pins the forgiveness rule:
// an operator who mistypes several times and then gets it right must
// not be left throttled.
func TestLoginGuard_SuccessForgivesFailures(t *testing.T) {
t.Parallel()
g := newGuard(middleware.LoginFailureMaxKeysConst, 1)
for range middleware.LoginRateLimitConst {
g.FailForTest(guardClient, guardUser)
}
g.SucceedForTest(guardClient, guardUser)
assert.False(
t, g.FailForTest(guardClient, guardUser),
"a success must reset the counter, so the next mistake "+
"starts a fresh budget",
)
}
// TestLoginGuard_FailuresAreKeyedPerUsername proves the second half
// of the keying: one username's spent budget does not throttle
// another's from the same client.
func TestLoginGuard_FailuresAreKeyedPerUsername(t *testing.T) {
t.Parallel()
g := newGuard(middleware.LoginFailureMaxKeysConst, 1)
for range middleware.LoginRateLimitConst {
g.FailForTest(guardClient, guardUser)
}
assert.True(t, g.FailForTest(guardClient, guardUser))
assert.False(
t, g.FailForTest(guardClient, "someone-else"),
"a different submitted username must have its own budget",
)
}
// TestLoginGuard_WindowLapses covers the interval: a counter that has
// gone quiet for the whole window starts again from zero.
func TestLoginGuard_WindowLapses(t *testing.T) {
t.Parallel()
g := newGuard(middleware.LoginFailureMaxKeysConst, 1)
var now atomic.Int64
now.Store(time.Now().UnixNano())
g.SetNowForTest(func() time.Time {
return time.Unix(0, now.Load())
})
for range middleware.LoginRateLimitConst {
g.FailForTest(guardClient, guardUser)
}
assert.True(t, g.FailForTest(guardClient, guardUser))
now.Add(int64(guardInterval) + 1)
assert.False(
t, g.FailForTest(guardClient, guardUser),
"a lapsed window must start a fresh budget",
)
}
// TestLoginGuard_UsernameKeySetIsBounded is the memory bound. The
// submitted username is attacker-controlled, so an attacker rotating
// usernames must not be able to grow the guard without limit: past
// the cap, tracking falls back to a counter keyed on the client
// address alone.
func TestLoginGuard_UsernameKeySetIsBounded(t *testing.T) {
t.Parallel()
const (
maxKeys = 8
attempts = 500
)
g := newGuard(maxKeys, 1)
for i := range attempts {
g.FailForTest(guardClient, fmt.Sprintf("user-%d", i))
}
byUser, byAddr := g.TrackedKeysForTest()
assert.LessOrEqual(
t, byUser, maxKeys,
"the per-username key set must not grow past its cap",
)
assert.LessOrEqual(
t, byAddr, maxKeys,
"the fallback key set must not grow past its cap either",
)
assert.Positive(
t, byAddr,
"past the cap, failures must fall back to the address "+
"bucket rather than being dropped",
)
assert.Less(
t, byUser+byAddr, attempts,
"memory must not grow with the number of distinct "+
"usernames submitted",
)
}
// TestLoginGuard_BeyondBothCapsStaysThrottled covers the hard stop.
// When both key sets are full of live counters and the client is in
// neither, there is nothing to count without unbounded growth, so the
// failure is answered as throttled. That costs the operator nothing:
// a correct password never reaches this path.
func TestLoginGuard_BeyondBothCapsStaysThrottled(t *testing.T) {
t.Parallel()
const maxKeys = 4
g := newGuard(maxKeys, 1)
// Fill the per-username set from one client, then fill the
// address set from distinct clients.
for i := range maxKeys {
g.FailForTest(guardClient, fmt.Sprintf("user-%d", i))
}
for i := range maxKeys {
g.FailForTest(fmt.Sprintf("203.0.113.%d", i), "whoever")
}
assert.True(
t, g.FailForTest("203.0.113.200", "brand-new"),
"a client that fits in neither full key set must be "+
"answered as throttled rather than tracked",
)
byUser, byAddr := g.TrackedKeysForTest()
assert.LessOrEqual(t, byUser, maxKeys)
assert.LessOrEqual(t, byAddr, maxKeys)
}
// TestLoginGuard_SemaphoreBoundsConcurrentVerifications is the memory
// bound on the hashing itself. Verifying credentials before spending
// limiter budget means an attacker can force one Argon2id hash per
// request, and each allocates 64 MB; without this bound the fix for
// an admin lockout would be a memory-exhaustion DoS instead.
func TestLoginGuard_SemaphoreBoundsConcurrentVerifications(
t *testing.T,
) {
t.Parallel()
const (
concurrency = 2
workers = 12
// rendezvousDeadlock is the deadlock guard described below.
// It is orders of magnitude longer than any scheduling delay,
// so it never decides the result, and well inside script/test's
// 30s timeout, so a wedge fails on the assertion instead of
// blowing the package timeout.
rendezvousDeadlock = 5 * time.Second
)
g := newGuard(middleware.LoginFailureMaxKeysConst, concurrency)
var (
mu sync.Mutex
inside int
highest int
wg sync.WaitGroup
recorded sync.WaitGroup
once sync.Once
)
// Slot holders rendezvous instead of sleeping, and they hold until
// every worker has been answered. A sleep only makes overlap
// likely — on a host loaded enough to deschedule a goroutine for
// longer than the sleep the workers serialise and the maximum
// observed comes back as 1 — so the rendezvous is what makes the
// overlap a fact rather than a race won.
//
// The barrier must not open at the concurrency-th holder, which
// would fix the lower bound at the cost of the upper one this test
// exists to enforce: holders would leave as soon as the count
// reached concurrency, so a guard admitting extra requests would
// let them arrive after the first holders had already left and
// highest would report concurrency however many were really let
// in. It opens instead once every worker's acquire has returned
// and any slot it won has been counted, so under a broken guard
// every admitted worker is inside simultaneously and highest is
// the true maximum. Under a correct guard the refused workers
// return within the guard's own wait, which decides nothing beyond
// how long that takes.
overlapped := make(chan struct{})
closeOverlapped := func() {
once.Do(func() { close(overlapped) })
}
// Deadlock guard, not a timing margin: no assertion depends on its
// length, and the only way to reach it is a worker that never
// returns from acquire at all. It is here so that such a wedge
// fails legibly on the assertion below instead of hanging until
// the package test timeout.
abandon := time.AfterFunc(rendezvousDeadlock, closeOverlapped)
defer abandon.Stop()
recorded.Add(workers)
go func() {
recorded.Wait()
closeOverlapped()
}()
for range workers {
wg.Go(func() {
release, ok := g.AcquireForTest(context.Background())
if !ok {
recorded.Done()
return
}
defer release()
mu.Lock()
inside++
if inside > highest {
highest = inside
}
mu.Unlock()
// Counted before signalling, so the barrier can never open
// while an admitted worker is still on its way to being
// counted.
recorded.Done()
<-overlapped
mu.Lock()
inside--
mu.Unlock()
})
}
wg.Wait()
mu.Lock()
defer mu.Unlock()
assert.Equal(
t, concurrency, highest,
"no more than %d verifications may run at once", concurrency,
)
}
// TestLoginGuard_SaturatedSemaphoreRefusesRatherThanQueueing pins
// what happens when every slot is taken for longer than the wait: the
// request is refused, so the caller answers 503 without allocating
// another 64 MB hash.
//
// Neither half of this rides on the wait being long enough. The
// refusal holds the only slot across the whole of the second call, so
// there is no wait it could get lucky with — the wait fixes only how
// long the refusal takes, not whether it happens. The reuse after
// release is settled by acquire's non-blocking preamble, which is
// pinned separately by TestLoginGuard_FreeSlotBeatsAnExpiredWait. So
// the wait below is sized to keep the test quick, not to win a race.
func TestLoginGuard_SaturatedSemaphoreRefusesRatherThanQueueing(
t *testing.T,
) {
t.Parallel()
g := middleware.NewLoginGuardForTest(
middleware.LoginRateLimitConst,
guardInterval,
middleware.LoginFailureMaxKeysConst,
1,
middleware.PasswordVerifyMaxWaitersConst,
10*time.Millisecond,
)
release, ok := g.AcquireForTest(context.Background())
require.True(t, ok, "the first acquire must get the only slot")
_, ok = g.AcquireForTest(context.Background())
assert.False(
t, ok,
"with the only slot held, a second request must be refused "+
"rather than wait indefinitely",
)
release()
release, ok = g.AcquireForTest(context.Background())
// require, not assert: acquire returns a nil release alongside a
// false ok, so calling it after a non-fatal assertion turns one
// failed test into a segfault that takes the whole package test
// binary down. Every assertion whose value is dereferenced later
// has to stop the test.
require.True(
t, ok, "the slot must be reusable once released",
)
release()
}
// TestLoginGuard_FreeSlotBeatsAnExpiredWait is the determinism this
// file used to lack. acquire selects over a slot send and a wait
// timer, and Go chooses among ready cases uniformly at random, so a
// call made after the timer had already fired was a coin flip: on a
// loaded host the previous test's third acquire could be refused
// with its slot standing free, and then dereference the nil release
// it got back.
//
// The wait here is already elapsed on arrival, which is the worst
// case that scheduling can produce, so a free slot must still be
// granted every time. Without acquire's non-blocking preamble each
// pass is an independent coin flip and the loop fails within a few
// passes; with it the property holds by construction and no wall
// clock is involved.
func TestLoginGuard_FreeSlotBeatsAnExpiredWait(t *testing.T) {
t.Parallel()
g := middleware.NewLoginGuardForTest(
middleware.LoginRateLimitConst,
guardInterval,
middleware.LoginFailureMaxKeysConst,
1,
middleware.PasswordVerifyMaxWaitersConst,
0,
)
for pass := range racePasses {
release, ok := g.AcquireForTest(context.Background())
require.Truef(
t, ok,
"pass %d was refused a slot that was free; an expired "+
"wait must never beat an available slot",
pass,
)
release()
}
}
// TestLoginGuard_AcquireHonoursCancellation proves a client that
// disconnects while queued frees its place immediately instead of
// holding it for the full wait.
func TestLoginGuard_AcquireHonoursCancellation(t *testing.T) {
t.Parallel()
g := newGuard(middleware.LoginFailureMaxKeysConst, 1)
release, ok := g.AcquireForTest(context.Background())
require.True(t, ok)
defer release()
ctx, cancel := context.WithCancel(context.Background())
cancel()
_, ok = g.AcquireForTest(ctx)
assert.False(
t, ok, "a cancelled request must not wait for a slot",
)
}
// TestPasswordVerifyConcurrency_MatchesMemoryBudget pins the
// concurrency constant to the arithmetic behind it: the number of
// slots is the hashing budget divided by what one Argon2id hash
// actually costs.
//
// The per-hash figure is read out of the shipped password
// parameters rather than copied here. A guard that asserts a literal
// against a literal cannot see the thing it guards: raising
// argon2Memory would leave it green while the real ceiling doubled.
func TestPasswordVerifyConcurrency_MatchesMemoryBudget(t *testing.T) {
t.Parallel()
// Memory is the real argon2Memory, in KiB.
perHashMB := int(database.DefaultPasswordConfig().Memory) / mib
require.Positive(
t, perHashMB,
"the Argon2id memory parameter must be readable in MB",
)
// The memory this service commits to password hashing.
const budgetMB = 128
assert.Equal(
t,
middleware.PasswordVerifyConcurrencyConst,
budgetMB/perHashMB,
"the verification concurrency must be the %d MB hashing "+
"budget divided by the %d MB one Argon2id hash costs; "+
"if the Argon2id parameters changed, the slot count "+
"must change with them",
budgetMB, perHashMB,
)
}
// TestLoginGuard_ShedsPastTheQueueCap pins the memory bound on
// waiting, as distinct from the bound on hashing. A waiter arrives
// with its form already parsed, and the retained parse plus its
// header block cost several MB — far more than maxFormBodySize
// suggests, since that caps only the raw body read — so an unbounded
// queue would hold that much per waiting request for the whole wait;
// past the cap the guard must refuse instantly rather than grow.
func TestLoginGuard_ShedsPastTheQueueCap(t *testing.T) {
t.Parallel()
const (
maxWaiters = 2
// Long enough that a queued waiter never times out on its
// own, so anything the test observes leaving the queue left
// because it was shed.
neverElapses = time.Minute
// The probe carries its own deadline, so a guard that queues
// the probe instead of shedding it fails here rather than
// hanging until the package test timeout.
//
// This is a patience budget, not a margin to be won. A shed
// returns in microseconds and a probe that queued instead
// would not return for neverElapses, so the two are a whole
// minute apart and any budget between them separates them. It
// is set far above any scheduling stall a loaded host can
// produce, because the previous 200 ms — and the 100 ms
// elapsed-time assertion it fed — bounded the latency of a
// goroutine hand-off, which is a false red waiting to happen
// on the machine this suite runs on. What actually proves the
// probe was not queued is the queue depth asserted below.
probePatience = 5 * time.Second
)
g := middleware.NewLoginGuardForTest(
middleware.LoginRateLimitConst,
guardInterval,
middleware.LoginFailureMaxKeysConst,
1,
maxWaiters,
neverElapses,
)
// Occupy the only slot, so everything after this queues.
release, ok := g.AcquireForTest(context.Background())
require.True(t, ok)
defer release()
defer fillQueue(t, g, maxWaiters)()
granted, answered := probeQueueCap(g, probePatience)
require.True(
t, answered,
"a request arriving past the queue cap is still waiting to "+
"be queued; it must have been shed",
)
assert.False(
t, granted,
"a request arriving past the queue cap must be shed",
)
assert.Equal(
t, maxWaiters, g.QueuedWaitersForTest(),
"a shed request must not have grown the queue",
)
}
// fillQueue starts n waiters on g and returns once all of them are
// queued for a slot. The returned function releases them and waits
// for them to exit.
func fillQueue(
t *testing.T,
g *middleware.LoginGuard,
n int,
) func() {
t.Helper()
ctx, cancel := context.WithCancel(context.Background())
var wg sync.WaitGroup
for range n {
wg.Go(func() {
done, got := g.AcquireForTest(ctx)
if got {
done()
}
})
}
// Patience budget, not a margin: the waiters park in microseconds
// and nothing releases them, so the only way to exhaust this is a
// guard that never queues. One second is the same order as the
// scheduling stalls this suite has to survive, so it is not one.
require.Eventually(
t,
func() bool { return g.QueuedWaitersForTest() == n },
5*time.Second, time.Millisecond,
"the waiters must reach the queue before the cap is tested",
)
return func() {
cancel()
wg.Wait()
}
}
// probeQueueCap acquires from another goroutine. It reports, in
// order, whether the call was granted a slot and whether it was
// answered at all within wait; a call that never returned reports
// false for both.
//
// It runs off the test goroutine deliberately. Joining a full queue
// is not cancellable by context — refusing to join is the property
// under test — so a guard that fails this would otherwise hang the
// package until the test timeout instead of failing here.
//
// It reports no elapsed time. Timing a goroutine hand-off measures
// the host, not the guard, and the caller distinguishes shedding from
// queueing by the queue depth instead.
func probeQueueCap(
g *middleware.LoginGuard,
wait time.Duration,
) (bool, bool) {
probed := make(chan bool, 1)
go func() {
release, ok := g.AcquireForTest(context.Background())
if ok {
release()
}
probed <- ok
}()
select {
case result := <-probed:
return result, true
case <-time.After(wait):
return false, false
}
}

347
internal/middleware/middleware.go
View File

@@ -6,9 +6,11 @@ import (
"log/slog"
"net"
"net/http"
"sync"
"time"
basicauth "github.com/99designs/basicauth-go"
"github.com/go-chi/chi"
"github.com/go-chi/chi/middleware"
"github.com/go-chi/cors"
metrics "github.com/slok/go-http-metrics/metrics/prometheus"
@@ -17,6 +19,7 @@ import (
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/globals"
"sneak.berlin/go/webhooker/internal/logfield"
"sneak.berlin/go/webhooker/internal/logger"
"sneak.berlin/go/webhooker/internal/session"
)
@@ -25,6 +28,118 @@ const (
// corsMaxAge is the maximum time (in seconds) that a
// preflight response can be cached.
corsMaxAge = 300
// unmatchedRoute is logged in the access log's url field when a
// redirected or rejected request matched no route pattern at
// all. Every byte of such a path is client-chosen, so none of it
// is logged.
unmatchedRoute = "(unmatched)"
// redactedQuery stands in for the query string on the access log
// branches that keep the concrete URL. The query is client-chosen
// on every route, including the ones that answer an
// unauthenticated 200, so logging it verbatim would let a client
// pick the size of the line it writes.
redactedQuery = "?(redacted)"
// maxLogRequestIDBytes bounds the request id, which is also
// client-supplied: chi's RequestID middleware passes an inbound
// X-Request-Id header through verbatim. Its generated form is an
// order of magnitude shorter than this.
maxLogRequestIDBytes = 128
// maxLogMethodBytes bounds the method. Go accepts any RFC 7230
// token there, bounded only by the header size limit, so it is
// client-chosen text like the rest. The longest registered method
// is half this.
maxLogMethodBytes = 32
// MaxAccessLogLineBytes is the ceiling on one JSON access log line,
// and the number an operator multiplies by the request rate to size
// log storage. It is not an observation of a sample: it is the sum
// of the budgets above, each of which logfield.Truncate enforces in
// ENCODED bytes, plus the part of the line no client can influence.
//
// url, useragent, referer 3*(512+11) = 1569
// request_id 128+11 = 139
// method 32+11 = 43
// fixed portion = 336
// ----
// 2087
//
// The 512 is logfield.MaxBytes; the 11 is the truncation marker,
// charged on top of each budget rather than inside it.
//
// The fixed portion is the JSON punctuation, the field names, the
// level and the message, both timestamps at their longest, an IPv6
// remoteIP with a zone, a three-digit status and a full-width int64
// latency. Stated at 2560 so the figure carries headroom rather
// than sitting on the arithmetic.
//
// The tty text handler in internal/logger is covered by the same
// figure. logfield.EncodedBytes charges every rune at least what
// the wider of the two handlers emits for it — including the ten
// bytes strconv.Quote spends on a non-printable rune at or above
// U+10000, which is four more than the JSON handler ever spends —
// so each budget bounds the encoded field under either handler.
// The text handler's fixed portion is 286, the smaller of the two,
// which puts its worst case at 2037.
//
// It is also the ceiling on every OTHER line this service writes
// THROUGH SLOG that carries text an UNAUTHENTICATED client
// supplies. Those lines — the MaxBodySize rejection, the CSRF
// rejection, the rate-limit rejection, the unauthenticated-request
// and unknown-entrypoint DEBUG lines, the failed-login DEBUG
// lines, and the two login-throttle WARN lines ("login failure
// limit exceeded" in loginguard.go and "password verification
// capacity exhausted" in internal/handlers/auth.go) — spend the
// same per-field budgets, and each carries strictly fewer
// client-supplied fields than the access log does,
// so none of them can reach a width the access log cannot. That is
// asserted directly, per line and under both handlers, rather than
// left to the reasoning: see logbound_test.go in this package and
// in internal/handlers.
//
// The two login-throttle lines are capped defensively: chi pins
// their route to the constant path "/pages/login", so no request
// through the mux can widen either one. Their assertions call
// RecordLoginFailure and the login handler directly with the path
// a caller on a parameterised route would supply, which is the
// only way those caps can be pinned at all.
//
// One writer in that set is not a handler's own slog call. The
// GORM adapter in internal/gormlog logs the statement with the
// client-chosen parameter already interpolated into it, which on
// the receiver and login lookups is exactly the value the budgets
// above exist for. Its widest line spends two logfield.MaxBytes
// budgets — the statement and the driver error — against a fixed
// portion smaller than this one's, and
// internal/gormlog/gormlog_test.go asserts every line it emits
// against this constant directly rather than leaving it as
// arithmetic.
//
// What it does NOT cover, so that the figure above is not read as
// more than it is:
//
// - Lines carrying an AUTHENTICATED operator's own input, which
// are not truncated at all: the webhook name on "webhook
// created" and the target host on "target URL blocked by SSRF
// protection" (both internal/handlers/source_management.go),
// and target_name in internal/delivery/engine.go and
// target_http.go. Each is bounded only by the 1 MB form body
// cap, so a 100 KB name writes one line of roughly 600 KB.
// Deliberate: truncating the operator's own configuration
// echoed back costs debuggability against no adversary.
// - The "log" delivery target, which exists to write the whole
// inbound event to the log. Deliberate; see
// internal/delivery/target_log.go.
// - The record a recovered panic writes, which is not an access
// log line: its client-supplied fields are charged the same
// budgets, but it carries a whole goroutine stack as well and
// is wider than this figure. It has its own stated ceiling,
// MaxPanicLogLineBytes in recoverer.go, and is written once
// per recovered panic rather than once per request.
MaxAccessLogLineBytes = 2560
)
//nolint:revive // MiddlewareParams is a standard fx naming convention.
@@ -43,6 +158,12 @@ type Middleware struct {
log *slog.Logger
params *MiddlewareParams
session *session.Session
// loginGuard counts failed credential verifications and bounds
// concurrent password hashing. It is built on first use so that
// every construction path gets one; see guard().
loginGuardOnce sync.Once
loginGuard *loginGuard
}
// New creates a Middleware from the provided fx parameters.
@@ -94,6 +215,70 @@ func (lrw *loggingResponseWriter) WriteHeader(code int) {
lrw.ResponseWriter.WriteHeader(code)
}
// concreteLogURL renders the request's own URL for the access log
// branches that keep it, with the query string replaced by a fixed
// marker.
//
// The path on those branches is bounded by the service's routes or by
// the operator's data — a 2xx on the receiver means the UUID named a
// stored entrypoint, a 2xx under /s means the file is in the embedded
// tree. The query is not bounded by anything: /.well-known/healthcheck
// and /s/* take no authentication and sit behind no rate limiter, and
// /pages/login behind only the login limiter, so any of them will
// answer 200 to a URL carrying an arbitrary number of arbitrary bytes
// after the '?'. Keeping the path and dropping the query is what makes
// this branch as bounded as the pattern branches below.
//
// Nothing debuggable is lost. One route in the service reads a query
// parameter at all — `page`, on the authenticated pagination links in
// internal/handlers/source_management.go — and the alternatives that
// would preserve more (a key count, a key allowlist) all require
// parsing an attacker-sized query on every request, which is work an
// unauthenticated client would then be choosing for us.
func concreteLogURL(r *http.Request) string {
path := r.URL.EscapedPath()
if r.URL.RawQuery == "" && !r.URL.ForceQuery {
return path
}
return path + redactedQuery
}
// accessLogURL returns the value for the access log's url field.
//
// 2xx and 5xx responses get the concrete path (see concreteLogURL). A
// success resolved against a static route or against the operator's
// own data — on the receiver, a 2xx means the UUID named a stored
// entrypoint — and a server error is our own bug, where the exact URL
// is the primary evidence and which no client can provoke at will.
//
// 3xx and 4xx responses get the chi route pattern instead. Those are
// the outcomes an unauthenticated client drives for free: 404 or 429
// on any invented /webhook/ path, 303 to the login page on any
// invented /user/ path. Logging the concrete URL there lets a flood
// write attacker-chosen text, of attacker-chosen length, into the
// operator's log at one line per request. The pattern comes from the
// router's own table, so it is bounded by the service's routes while
// still naming which class of request was rejected.
//
// The pattern is only populated once routing has run, so this must be
// called after the handler returns, not before.
func accessLogURL(r *http.Request, status int) string {
if status < http.StatusMultipleChoices ||
status >= http.StatusInternalServerError {
return concreteLogURL(r)
}
if rc := chi.RouteContext(r.Context()); rc != nil {
if pattern := rc.RoutePattern(); pattern != "" {
return pattern
}
}
return unmatchedRoute
}
// Logging returns middleware that logs each HTTP request with
// timing and metadata.
func (s *Middleware) Logging() func(http.Handler) http.Handler {
@@ -118,13 +303,27 @@ func (s *Middleware) Logging() func(http.Handler) http.Handler {
}
}
// Every field below that a client can influence is
// truncated to a fixed budget, so the size of this
// line does not track the size of the request.
s.log.Info("http request",
"request_start", start,
"method", r.Method,
"url", r.URL.String(),
"useragent", r.UserAgent(),
"request_id", requestID,
"referer", r.Referer(),
"method", logfield.Truncate(
r.Method, maxLogMethodBytes,
),
"url", logfield.Truncate(
accessLogURL(r, lrw.statusCode),
logfield.MaxBytes,
),
"useragent", logfield.Truncate(
r.UserAgent(), logfield.MaxBytes,
),
"request_id", logfield.Truncate(
requestID, maxLogRequestIDBytes,
),
"referer", logfield.Truncate(
r.Referer(), logfield.MaxBytes,
),
"proto", r.Proto,
"remoteIP", ipFromHostPort(r.RemoteAddr),
"status", lrw.statusCode,
@@ -186,11 +385,26 @@ func (s *Middleware) RequireAuth() func(http.Handler) http.Handler {
return
}
// IsAuthenticated also enforces both session expiry
// deadlines, so an idle-expired or absolutely-expired
// session lands here and is sent back to the login
// page.
if !s.session.IsAuthenticated(sess) {
// This is the unauthenticated branch, so both
// fields are entirely client-chosen and neither
// is bounded by anything the router did. DEBUG
// is off by default, but turning it on to
// diagnose a problem must not hand a client an
// unbounded write into the log, so the same
// budgets apply here as in the access log.
s.log.Debug(
"auth middleware: unauthenticated request",
"path", r.URL.Path,
"method", r.Method,
"path", logfield.Truncate(
r.URL.Path, logfield.MaxBytes,
),
"method", logfield.Truncate(
r.Method, maxLogMethodBytes,
),
)
http.Redirect(
w, r, "/pages/login", http.StatusSeeOther,
@@ -199,6 +413,26 @@ func (s *Middleware) RequireAuth() func(http.Handler) http.Handler {
return
}
// This request authenticated with the session, so it
// counts as activity: push the idle deadline forward.
// This is the only place sessions are refreshed, which
// is what keeps an unauthenticated request from
// extending someone else's session. Touch advances the
// idle clock only -- the absolute cap is untouched --
// and reports false when nothing changed, so most
// requests do not re-issue the cookie. Save before the
// handler runs, while the headers are still ours to
// write.
if s.session.Touch(sess) {
saveErr := s.session.Save(r, w, sess)
if saveErr != nil {
s.log.Error(
"auth middleware: failed to refresh session",
"error", saveErr,
)
}
}
next.ServeHTTP(w, r)
})
}
@@ -265,10 +499,56 @@ func (s *Middleware) SecurityHeaders() func(http.Handler) http.Handler {
}
}
// MaxBodySize returns middleware that limits the request body size
// for POST requests. If the body exceeds the given limit in
// bytes, the server returns 413 Request Entity Too Large. This
// prevents clients from sending arbitrarily large form bodies.
// NoCache returns middleware that instructs browsers and
// intermediary proxies not to cache the response. It sets
// Cache-Control: no-store and Pragma: no-cache (the latter for
// older HTTP/1.0 intermediaries). Apply it to authenticated pages
// so webhook configuration and captured event data are not stored
// by caches.
func (s *Middleware) NoCache() func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(
w http.ResponseWriter,
r *http.Request,
) {
w.Header().Set("Cache-Control", "no-store")
w.Header().Set("Pragma", "no-cache")
next.ServeHTTP(w, r)
})
}
}
// bodyLimitedMethod reports whether the request method carries a
// body that the MaxBodySize middleware should cap.
func bodyLimitedMethod(method string) bool {
return method == http.MethodPost ||
method == http.MethodPut ||
method == http.MethodPatch
}
// MaxBodySize returns middleware that limits the size of
// POST/PUT/PATCH request bodies to maxBytes. It must be registered
// before any middleware that parses the body — notably CSRF, which
// calls r.PostFormValue — so that form parsing happens under this
// cap rather than net/http's 10 MB default.
//
// Two enforcement paths exist, because http.MaxBytesReader alone
// cannot produce a 413: it reports the overflow as an error from
// Read, by which point the body parser downstream has already
// converted that error into its own response.
//
// - Declared oversize: the request announces a Content-Length
// greater than maxBytes. The middleware answers 413 Request
// Entity Too Large immediately and does not call the next
// handler, so neither CSRF nor the endpoint handler runs.
// - Undeclared oversize: the request is chunked (Content-Length
// of -1) or lies about its Content-Length. There is nothing to
// check up front, so http.MaxBytesReader hard-caps the body at
// maxBytes and the request fails downstream — the form parse
// errors out and CSRF rejects it with 403. The response is less
// precise than a 413, but the body is still never buffered
// beyond the cap, which is the property that matters.
func (s *Middleware) MaxBodySize(
maxBytes int64,
) func(http.Handler) http.Handler {
@@ -277,14 +557,47 @@ func (s *Middleware) MaxBodySize(
w http.ResponseWriter,
r *http.Request,
) {
if r.Method == http.MethodPost ||
r.Method == http.MethodPut ||
r.Method == http.MethodPatch {
r.Body = http.MaxBytesReader(
w, r.Body, maxBytes,
)
if !bodyLimitedMethod(r.Method) {
next.ServeHTTP(w, r)
return
}
if r.ContentLength > maxBytes {
// This runs ahead of RequireAuth (see
// setupUserRoutes and friends in
// internal/server/routes.go), so an
// unauthenticated client reaches it with a path
// of its own choosing and its own length —
// POST /source/<8 KB>/edit with an oversize
// declared Content-Length costs nothing to
// send. At WARN, on by default, that is a
// write into the operator's log sized by the
// attacker unless the path is capped. Same
// budgets as the access log, so this line
// cannot be wider than that one.
s.log.Warn(
"request body exceeds limit",
"method", logfield.Truncate(
r.Method, maxLogMethodBytes,
),
"path", logfield.Truncate(
r.URL.Path, logfield.MaxBytes,
),
"content_length", r.ContentLength,
"limit", maxBytes,
)
http.Error(
w,
"Request Entity Too Large",
http.StatusRequestEntityTooLarge,
)
return
}
r.Body = http.MaxBytesReader(w, r.Body, maxBytes)
next.ServeHTTP(w, r)
})
}

471
internal/middleware/middleware_test.go
View File

@@ -3,11 +3,14 @@ package middleware_test
import (
"context"
"encoding/base64"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"time"
"github.com/gorilla/sessions"
"github.com/stretchr/testify/assert"
@@ -28,6 +31,22 @@ func testMiddleware(
) (*middleware.Middleware, *session.Session) {
t.Helper()
m, s, _ := testMiddlewareWithSessionClock(t, env, 0, nil)
return m, s
}
// testMiddlewareWithSessionClock is testMiddleware with a
// configurable session idle timeout and a manually advanced clock,
// for the session-expiry tests. A nil clock uses the real one.
func testMiddlewareWithSessionClock(
t *testing.T,
env string,
idleTimeout time.Duration,
clock *fakeClock,
) (*middleware.Middleware, *session.Session, *fakeClock) {
t.Helper()
log := slog.New(slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{Level: slog.LevelDebug},
@@ -35,9 +54,24 @@ func testMiddleware(
cfg := &config.Config{
Environment: env,
SessionIdleTimeout: idleTimeout,
}
// Create a real session manager with a known key
sessManager := newTestSessionManager(cfg, log, clock)
m := middleware.NewForTest(log, cfg, sessManager)
return m, sessManager, clock
}
// newTestSessionManager builds the real session.Session the
// middleware tests run against: an in-memory cookie store with a
// known key, and optionally a manually advanced clock.
func newTestSessionManager(
cfg *config.Config,
log *slog.Logger,
clock *fakeClock,
) *session.Session {
key := make([]byte, testKeySize)
for i := range key {
@@ -53,11 +87,36 @@ func testMiddleware(
SameSite: http.SameSiteLaxMode,
}
sessManager := session.NewForTest(store, cfg, log, key)
var now func() time.Time
m := middleware.NewForTest(log, cfg, sessManager)
if clock != nil {
now = clock.Now
}
return m, sessManager
return session.NewForTest(store, cfg, log, key, now)
}
// fakeClock is a manually advanced clock, so session expiry can be
// tested without sleeping.
type fakeClock struct {
t time.Time
}
func (c *fakeClock) Now() time.Time {
return c.t
}
func (c *fakeClock) Advance(d time.Duration) {
c.t = c.t.Add(d)
}
// newFakeClock returns a clock started at a fixed instant.
func newFakeClock() *fakeClock {
return &fakeClock{
t: time.Date(
2026, time.January, 2, 3, 4, 5, 0, time.UTC,
),
}
}
// --- Logging Middleware Tests ---
@@ -387,6 +446,367 @@ func TestRequireAuth_UnauthenticatedSession_RedirectsToLogin(
assert.Equal(t, "/pages/login", w.Header().Get("Location"))
}
// --- RequireAuth Session Expiry Tests ---
// loginCookies authenticates a new session and returns the cookies
// a browser would then send back.
func loginCookies(
t *testing.T,
sessManager *session.Session,
) []*http.Cookie {
t.Helper()
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/login", nil)
w := httptest.NewRecorder()
sess, err := sessManager.Get(req)
require.NoError(t, err)
sessManager.SetUser(sess, "user-123", "testuser")
require.NoError(t, sessManager.Save(req, w, sess))
cookies := w.Result().Cookies()
require.NotEmpty(t, cookies, "session cookie should be set")
return cookies
}
// runAuthed sends a request carrying cookies through RequireAuth
// and reports whether the protected handler ran, plus the response.
func runAuthed(
t *testing.T,
m *middleware.Middleware,
cookies []*http.Cookie,
) (bool, *httptest.ResponseRecorder) {
t.Helper()
var called bool
handler := m.RequireAuth()(http.HandlerFunc(
func(_ http.ResponseWriter, _ *http.Request) {
called = true
},
))
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet, "/dashboard", nil,
)
for _, c := range cookies {
req.AddCookie(c)
}
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
return called, w
}
// sessionCookies filters a response's cookies down to the session
// cookie, so tests can tell whether the session was re-issued.
func sessionCookies(
w *httptest.ResponseRecorder,
) []*http.Cookie {
var out []*http.Cookie
for _, c := range w.Result().Cookies() {
if c.Name == session.SessionName {
out = append(out, c)
}
}
return out
}
func TestRequireAuth_IdleExpiredSession_RedirectsToLogin(
t *testing.T,
) {
t.Parallel()
idle := time.Hour
m, sessManager, clock := testMiddlewareWithSessionClock(
t, config.EnvironmentDev, idle, newFakeClock(),
)
cookies := loginCookies(t, sessManager)
clock.Advance(idle)
called, w := runAuthed(t, m, cookies)
assert.False(
t, called,
"handler should not run for an idle-expired session",
)
assert.Equal(t, http.StatusSeeOther, w.Code)
assert.Equal(t, "/pages/login", w.Header().Get("Location"))
assert.Empty(
t, sessionCookies(w),
"an expired session must not be refreshed",
)
}
func TestRequireAuth_RefreshesIdleDeadlineOnActivity(
t *testing.T,
) {
t.Parallel()
idle := time.Hour
m, sessManager, clock := testMiddlewareWithSessionClock(
t, config.EnvironmentDev, idle, newFakeClock(),
)
cookies := loginCookies(t, sessManager)
// Activity halfway through the idle window.
clock.Advance(idle / 2)
called, w := runAuthed(t, m, cookies)
require.True(t, called, "handler should run while valid")
refreshed := sessionCookies(w)
require.NotEmpty(
t, refreshed,
"activity should re-issue the session cookie",
)
// Past the original deadline. The refreshed cookie is still
// good; the original one is not.
clock.Advance(idle - time.Second)
calledRefreshed, _ := runAuthed(t, m, refreshed)
assert.True(
t, calledRefreshed,
"refreshed session should outlive the original deadline",
)
calledStale, staleW := runAuthed(t, m, cookies)
assert.False(
t, calledStale,
"the pre-refresh cookie carries the old idle deadline",
)
assert.Equal(t, http.StatusSeeOther, staleW.Code)
}
func TestRequireAuth_UnauthenticatedRequestDoesNotRefresh(
t *testing.T,
) {
t.Parallel()
m, sessManager, _ := testMiddlewareWithSessionClock(
t, config.EnvironmentDev, time.Hour, newFakeClock(),
)
// A session cookie that exists but was never authenticated.
req := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/setup", nil)
setupW := httptest.NewRecorder()
sess, err := sessManager.Get(req)
require.NoError(t, err)
require.NoError(t, sessManager.Save(req, setupW, sess))
cookies := setupW.Result().Cookies()
require.NotEmpty(t, cookies)
called, w := runAuthed(t, m, cookies)
assert.False(t, called)
assert.Empty(
t, sessionCookies(w),
"an unauthenticated request must not stamp the session",
)
}
// --- NoCache Middleware Tests ---
func TestNoCache_SetsHeaders(t *testing.T) {
t.Parallel()
m, _ := testMiddleware(t, config.EnvironmentDev)
var called bool
handler := m.NoCache()(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
called = true
w.WriteHeader(http.StatusOK)
},
))
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet, "/sources", nil,
)
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
assert.True(
t, called,
"NoCache middleware should call the next handler",
)
assert.Equal(
t, "no-store",
w.Header().Get("Cache-Control"),
)
assert.Equal(
t, "no-cache",
w.Header().Get("Pragma"),
)
}
// --- MaxBodySize Middleware Tests ---
const testBodyLimit int64 = 64
// maxBodySizeHandler wraps a sentinel handler in MaxBodySize with
// testBodyLimit. The sentinel records whether it ran and how much of
// the body it managed to read, so tests can distinguish "never
// reached" from "reached but truncated".
type maxBodySizeResult struct {
called bool
read int
readErr error
response *httptest.ResponseRecorder
}
func runMaxBodySize(
t *testing.T,
req *http.Request,
) *maxBodySizeResult {
t.Helper()
m, _ := testMiddleware(t, config.EnvironmentDev)
res := &maxBodySizeResult{response: httptest.NewRecorder()}
handler := m.MaxBodySize(testBodyLimit)(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
res.called = true
body, err := io.ReadAll(r.Body)
res.read = len(body)
res.readErr = err
w.WriteHeader(http.StatusOK)
},
))
handler.ServeHTTP(res.response, req)
return res
}
// postWithBody builds a POST request whose Content-Length is
// accurate for the given payload size.
func postWithBody(size int) *http.Request {
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost, "/pages/login",
strings.NewReader(strings.Repeat("a", size)),
)
req.Header.Set(
"Content-Type", "application/x-www-form-urlencoded",
)
return req
}
func TestMaxBodySize_DeclaredOversize_413AndHandlerNotReached(
t *testing.T,
) {
t.Parallel()
res := runMaxBodySize(t, postWithBody(int(testBodyLimit)+1))
assert.False(
t, res.called,
"handler must not be reached for an oversized body",
)
assert.Equal(
t, http.StatusRequestEntityTooLarge, res.response.Code,
)
}
func TestMaxBodySize_AtLimit_PassesThrough(t *testing.T) {
t.Parallel()
res := runMaxBodySize(t, postWithBody(int(testBodyLimit)))
assert.True(
t, res.called,
"handler should be reached for a body at the limit",
)
require.NoError(t, res.readErr)
assert.Equal(t, int(testBodyLimit), res.read)
assert.Equal(t, http.StatusOK, res.response.Code)
}
func TestMaxBodySize_UnderLimit_PassesThrough(t *testing.T) {
t.Parallel()
res := runMaxBodySize(t, postWithBody(1))
assert.True(t, res.called)
require.NoError(t, res.readErr)
assert.Equal(t, 1, res.read)
assert.Equal(t, http.StatusOK, res.response.Code)
}
func TestMaxBodySize_GetWithOversizeBody_NotCapped(t *testing.T) {
t.Parallel()
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet, "/pages/login",
strings.NewReader(
strings.Repeat("a", int(testBodyLimit)+1),
),
)
res := runMaxBodySize(t, req)
assert.True(
t, res.called,
"GET requests are not subject to the POST body cap",
)
require.NoError(t, res.readErr)
assert.Equal(t, int(testBodyLimit)+1, res.read)
}
// TestMaxBodySize_UndeclaredOversize_TruncatedAtCap covers the
// chunked / lying-Content-Length case: there is nothing to check up
// front, so the request reaches the handler but MaxBytesReader
// hard-caps the body and the read fails at the limit.
func TestMaxBodySize_UndeclaredOversize_TruncatedAtCap(
t *testing.T,
) {
t.Parallel()
req := postWithBody(int(testBodyLimit) + 1)
// Simulate a chunked request: no declared length.
req.ContentLength = -1
res := runMaxBodySize(t, req)
assert.True(
t, res.called,
"an undeclared oversize body cannot be rejected up front",
)
require.Error(
t, res.readErr,
"reading past the cap must fail",
)
assert.Equal(
t, int(testBodyLimit), res.read,
"the handler must not see more than the cap",
)
}
// --- Helper Tests ---
func TestIpFromHostPort(t *testing.T) {
@@ -440,13 +860,18 @@ func metricsAuthMiddleware(
store := sessions.NewCookieStore(key)
store.Options = &sessions.Options{Path: "/", MaxAge: 86400}
sessManager := session.NewForTest(store, cfg, log, key)
sessManager := session.NewForTest(store, cfg, log, key, nil)
return middleware.NewForTest(log, cfg, sessManager)
}
func TestMetricsAuth_ValidCredentials(t *testing.T) {
t.Parallel()
// runMetricsAuthRequest sends a GET /metrics request with the
// given basic-auth password through MetricsAuth and reports
// whether the wrapped handler ran plus the recorded response.
func runMetricsAuthRequest(
t *testing.T, password string,
) (bool, *httptest.ResponseRecorder) {
t.Helper()
m := metricsAuthMiddleware(t)
@@ -464,12 +889,20 @@ func TestMetricsAuth_ValidCredentials(t *testing.T) {
context.Background(),
http.MethodGet, "/metrics", nil,
)
req.SetBasicAuth("admin", "secret")
req.SetBasicAuth("admin", password)
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
return called, w
}
func TestMetricsAuth_ValidCredentials(t *testing.T) {
t.Parallel()
called, w := runMetricsAuthRequest(t, "secret")
assert.True(
t, called,
"handler should be called with valid basic auth",
@@ -480,27 +913,7 @@ func TestMetricsAuth_ValidCredentials(t *testing.T) {
func TestMetricsAuth_InvalidCredentials(t *testing.T) {
t.Parallel()
m := metricsAuthMiddleware(t)
var called bool
handler := m.MetricsAuth()(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
called = true
w.WriteHeader(http.StatusOK)
},
))
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet, "/metrics", nil,
)
req.SetBasicAuth("admin", "wrong-password")
w := httptest.NewRecorder()
handler.ServeHTTP(w, req)
called, w := runMetricsAuthRequest(t, "wrong-password")
assert.False(
t, called,

379
internal/middleware/ratelimit.go
View File

@@ -1,46 +1,312 @@
package middleware
import (
"math"
"net/http"
"net/netip"
"slices"
"strings"
"time"
"github.com/go-chi/httprate"
"sneak.berlin/go/webhooker/internal/logfield"
)
const (
// loginRateLimit is the maximum number of login attempts
// per interval.
// loginRateLimit is the maximum number of FAILED login attempts
// one client may make against one submitted username per
// interval before further failures are answered 429. Successful
// attempts are never counted and never throttled — see
// loginGuard.
loginRateLimit = 5
// loginRateInterval is the time window for the rate limit.
// loginRateInterval is the time window for the login failure
// limit.
loginRateInterval = 1 * time.Minute
// passwordChangeRateLimit is the maximum number of password
// change attempts per interval. Each attempt verifies the
// current password, so the endpoint must be rate-limited
// like any other password-based authentication endpoint.
passwordChangeRateLimit = 5
// passwordChangeRateInterval is the time window for the
// password change rate limit.
passwordChangeRateInterval = 1 * time.Minute
// receiverRateInterval is the time window for the webhook
// receiver rate limit. The configured limit is expressed in
// requests per minute.
receiverRateInterval = 1 * time.Minute
// receiverAggregateMultiplier scales the configured
// per-entrypoint receiver limit into the aggregate limit one
// client IP may spend across the whole /webhook/* route. Ten
// entrypoints' worth lets a single sender address drive several
// entrypoints at their full rate, while still capping what one
// address costs the unauthenticated receiver.
receiverAggregateMultiplier = 10
// maxForwardedHops bounds how many X-Forwarded-For entries the
// chain walk examines. Real chains are one to three hops, but a
// client can pad the header up to MaxHeaderBytes, so without a
// bound every request pays a walk proportional to whatever the
// client sent.
maxForwardedHops = 64
// ipv6BucketBits is the prefix length IPv6 clients are bucketed
// on. A routed /64 is the normal residential and mobile
// allocation, so it is the unit an attacker gets addresses in
// and therefore the unit worth limiting.
ipv6BucketBits = 64
)
// LoginRateLimit returns middleware that enforces per-IP rate
// limiting on login attempts using go-chi/httprate. Only POST
// requests are rate-limited; GET requests (rendering the login
// form) pass through unaffected. When the rate limit is exceeded,
// a 429 Too Many Requests response is returned. IP extraction
// honours X-Forwarded-For, X-Real-IP, and True-Client-IP headers
// for reverse-proxy setups.
func (m *Middleware) LoginRateLimit() func(http.Handler) http.Handler {
limiter := httprate.Limit(
loginRateLimit,
loginRateInterval,
httprate.WithKeyFuncs(httprate.KeyByRealIP),
httprate.WithLimitHandler(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
m.log.Warn("login rate limit exceeded",
"path", r.URL.Path,
// normalizeAddr strips the IPv4-in-IPv6 wrapper and any zone from
// addr so that comparisons and bucket keys are canonical.
func normalizeAddr(addr netip.Addr) netip.Addr {
return addr.Unmap().WithZone("")
}
// bucketKey is the rate-limit bucket identity of a client address.
// IPv4 keys on the full address; IPv6 keys on its /64 prefix,
// because keying IPv6 per /128 lets one ordinary subscriber rotate
// source addresses inside its own routed /64 and mint a fresh bucket
// per request — evading every limiter here at the network layer,
// with no spoofing and nothing to detect.
//
// An IPv4-mapped address (::ffff:1.2.3.4) is keyed as the IPv4
// address it carries, never masked to a /64: mapped form all shares
// the ::ffff:0:0/96 prefix, so masking would collapse every IPv4
// client reaching a proxy that emits it into one bucket. Callers
// pass addresses through normalizeAddr, which already unmaps; the
// unmap here keeps the property true of the key function itself.
//
// The two families cannot collide: an IPv4 key is a bare dotted
// quad, and an IPv6 key always carries a "/64" suffix.
func bucketKey(addr netip.Addr) string {
addr = addr.Unmap()
if addr.Is4() {
return addr.String()
}
// Prefix errors only on a negative bit count, on over 32 bits
// for an IPv4 address, or on over 128 for IPv6. The count here
// is the constant 64 and the IPv4 case returned above, so the
// error is unreachable. (The zero Addr does not error either: it
// yields the zero Prefix. Neither call site can produce one,
// since both parse the address first.)
prefix, _ := addr.Prefix(ipv6BucketBits)
return prefix.String()
}
// isTrustedProxy reports whether addr belongs to a network the
// operator listed in TRUSTED_PROXIES. The list is empty by default,
// so by default nothing is trusted.
func (m *Middleware) isTrustedProxy(addr netip.Addr) bool {
for _, prefix := range m.params.Config.TrustedProxies {
if prefix.Contains(addr) {
return true
}
}
return false
}
// forwardedClientAddr returns the client address named by this
// request's X-Forwarded-For chain. It is consulted only for requests
// whose direct peer is a trusted proxy.
//
// X-Forwarded-For is the only header read. X-Real-IP and
// True-Client-IP are deliberately ignored: the reverse proxies in
// common use append to X-Forwarded-For and pass any other header the
// client sent through untouched, so believing a single-valued header
// would let a client behind the trusted proxy name its own bucket —
// the very bypass this gating exists to close.
//
// The chain is walked right to left, because the rightmost entry is
// the one the nearest proxy appended and everything to its left may
// have been written by the client. The first hop that is not itself
// a trusted proxy is the client. A hop that cannot be read as a bare
// address ends the walk: past it the chain is not the shape assumed
// here, so the caller falls back to the peer address.
//
// Only the last maxForwardedHops entries are examined. A longer chain
// is padding, and running out of hops falls back to the peer address
// the same way an unreadable hop does.
//
// The entries are cut off the right end of each header value in place
// rather than split out of it: the receiver is unauthenticated and a
// client can pad the header up to MaxHeaderBytes, so splitting would
// allocate in proportion to the padding (about 8 MB for a 1 MB
// header) before the cap could discard any of it. Multiple header
// values are walked in reverse for the same reason, since joining
// them copies the whole chain.
func (m *Middleware) forwardedClientAddr(
r *http.Request,
) (netip.Addr, bool) {
seen := 0
for _, value := range slices.Backward(
r.Header.Values("X-Forwarded-For"),
) {
for last := false; !last && seen < maxForwardedHops; seen++ {
hop := value
comma := strings.LastIndexByte(value, ',')
if comma < 0 {
last = true
} else {
hop, value = value[comma+1:], value[:comma]
}
hop = strings.TrimSpace(hop)
if hop == "" {
continue
}
addr, err := netip.ParseAddr(hop)
if err != nil {
return netip.Addr{}, false
}
if addr = normalizeAddr(addr); !m.isTrustedProxy(addr) {
return addr, true
}
}
}
return netip.Addr{}, false
}
// rateLimitKey is the client identity every rate limiter in this
// package buckets on. Forwarded headers are honoured only when the
// direct peer (RemoteAddr) is inside the configured trusted-proxy
// set; otherwise the peer address itself is the key. Without that
// gate any client could mint a fresh bucket per request, or starve
// another client's bucket, by picking an X-Forwarded-For value —
// which makes every limit here decorative against a deliberate
// attacker.
//
// The address that identifies the client is then reduced to a bucket
// by bucketKey: full address for IPv4, /64 prefix for IPv6.
func (m *Middleware) rateLimitKey(r *http.Request) (string, error) {
return m.clientKey(r), nil
}
// clientKey computes the bucket key described on rateLimitKey.
func (m *Middleware) clientKey(r *http.Request) string {
peer, err := netip.ParseAddr(ipFromHostPort(r.RemoteAddr))
if err != nil {
// Not an address we can reason about; key on the raw
// value, the most specific identity left. Distinct
// RemoteAddr values stay in distinct buckets, so this
// path cannot silently collapse unrelated clients
// together. On a Unix-socket listener every peer
// carries the same RemoteAddr and so shares one bucket,
// which is the fail-closed direction.
return r.RemoteAddr
}
peer = normalizeAddr(peer)
if !m.isTrustedProxy(peer) {
return bucketKey(peer)
}
if addr, ok := m.forwardedClientAddr(r); ok {
return bucketKey(addr)
}
return bucketKey(peer)
}
// tooManyRequests returns the 429 handler used by the
// password-change and per-entrypoint receiver limiters: it logs the
// rejection with logMessage and answers with responseMessage.
// httprate adds the Retry-After header (RFC 6585). The aggregate
// receiver limiter uses floodTooManyRequests instead.
//
// The path is capped against the same budget as the access log's url
// field. The per-entrypoint receiver limiter is unauthenticated and
// its path is a client-chosen segment of client-chosen length, so at
// WARN an uncapped path would let a sender pick the size of the line
// it writes — the same defect the access log capping closed.
func (m *Middleware) tooManyRequests(
logMessage, responseMessage string,
) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
m.log.Warn(
logMessage,
"path", logfield.Truncate(
r.URL.Path, logfield.MaxBytes,
),
)
http.Error(
w,
"Too many login attempts. "+
http.Error(w, responseMessage, http.StatusTooManyRequests)
}
}
// floodTooManyRequests returns the 429 handler for a limiter whose
// rejections are themselves the flood: it logs at DEBUG and without
// the path, then answers with responseMessage.
//
// The aggregate receiver limiter trips exactly when one address is
// sending faster than the receiver wants to serve, so its rejection
// log is one line per request of that flood. At WARN with "path" that
// hands a client a way to write its own text into the operator's log,
// at a level that trips alerting, once per request — the log-volume
// problem this limiter exists to bound. DEBUG is off in production by
// default, so a flood costs nothing here; the path is dropped so that
// turning DEBUG on to diagnose one does not restore the problem.
//
// This limiter bounds the database work an invented path costs, not
// the number of log lines it produces: the access log in
// middleware.go still records every request, served or rejected.
func (m *Middleware) floodTooManyRequests(
logMessage, responseMessage string,
) http.HandlerFunc {
return func(w http.ResponseWriter, _ *http.Request) {
m.log.Debug(logMessage)
http.Error(w, responseMessage, http.StatusTooManyRequests)
}
}
// PasswordChangeRateLimit returns middleware that enforces
// per-IP rate limiting on password change attempts. The change
// endpoint verifies the current password, so without a limit a
// stolen session could be used to brute-force it.
//
// Unlike the login POST this limit is still spent on arrival, which
// is safe here: RequireAuth runs ahead of it, so only a request
// already carrying a valid session can reach the bucket, and an
// operator locked out of changing a password can still log in.
func (m *Middleware) PasswordChangeRateLimit() func(http.Handler) http.Handler {
return m.postRateLimit(
passwordChangeRateLimit,
passwordChangeRateInterval,
"password change rate limit exceeded",
"Too many password change attempts. "+
"Please try again later.",
http.StatusTooManyRequests,
)
},
)),
}
// postRateLimit builds middleware that enforces a per-IP rate
// limit on POST requests only; all other methods pass through
// unaffected. Requests over the limit receive a 429 with the
// given response message, and each rejection is logged with the
// given log message. Clients are identified by rateLimitKey.
func (m *Middleware) postRateLimit(
limit int,
interval time.Duration,
logMessage, responseMessage string,
) func(http.Handler) http.Handler {
limiter := httprate.Limit(
limit,
interval,
httprate.WithKeyFuncs(m.rateLimitKey),
httprate.WithLimitHandler(
m.tooManyRequests(logMessage, responseMessage),
),
)
return func(next http.Handler) http.Handler {
@@ -50,8 +316,7 @@ func (m *Middleware) LoginRateLimit() func(http.Handler) http.Handler {
w http.ResponseWriter,
r *http.Request,
) {
// Only rate-limit POST requests (actual login
// attempts)
// Only rate-limit POST requests.
if r.Method != http.MethodPost {
next.ServeHTTP(w, r)
@@ -62,3 +327,63 @@ func (m *Middleware) LoginRateLimit() func(http.Handler) http.Handler {
})
}
}
// ReceiverRateLimit returns middleware that rate-limits the public
// webhook receiver endpoint with two limits in series.
//
// The inner limit is per client IP per request path: the path
// contains the entrypoint UUID, so each sender is limited per
// entrypoint without affecting other senders or other entrypoints.
// It is Config.ReceiverRateLimit requests per minute.
//
// That limit alone bounds nothing in aggregate. The route pattern
// /webhook/{uuid} matches any single segment, so a client that
// invents a fresh path per request mints a fresh bucket per request
// and never refills one — and every such request still reaches the
// handler's entrypoint lookup before it 404s. The outer limit is
// therefore keyed on the client IP alone, capping what one address
// can spend across the whole route however it varies the path.
//
// Requests over either limit receive a 429. Clients are identified
// by rateLimitKey.
func (m *Middleware) ReceiverRateLimit() func(http.Handler) http.Handler {
perEntrypoint := httprate.Limit(
m.params.Config.ReceiverRateLimit,
receiverRateInterval,
httprate.WithKeyFuncs(
m.rateLimitKey,
httprate.KeyByEndpoint,
),
httprate.WithLimitHandler(m.tooManyRequests(
"webhook receiver rate limit exceeded",
"Too many requests. Please slow down.",
)),
)
aggregate := httprate.Limit(
receiverAggregateLimit(m.params.Config.ReceiverRateLimit),
receiverRateInterval,
httprate.WithKeyFuncs(m.rateLimitKey),
httprate.WithLimitHandler(m.floodTooManyRequests(
"webhook receiver aggregate rate limit exceeded",
"Too many requests. Please slow down.",
)),
)
return func(next http.Handler) http.Handler {
return aggregate(perEntrypoint(next))
}
}
// receiverAggregateLimit is the per-IP aggregate limit derived from
// the configured per-entrypoint limit. The operator sets the latter
// and nothing bounds it from above, so the multiplication is
// saturated rather than allowed to wrap into a negative limit that
// would reject every request.
func receiverAggregateLimit(perEntrypoint int) int {
if perEntrypoint > math.MaxInt/receiverAggregateMultiplier {
return math.MaxInt
}
return perEntrypoint * receiverAggregateMultiplier
}

1176
internal/middleware/ratelimit_test.go
View File

File diff suppressed because it is too large Load Diff

209
internal/middleware/recoverer.go Normal file
View File

@@ -0,0 +1,209 @@
package middleware
import (
"errors"
"fmt"
"net/http"
"runtime/debug"
"github.com/go-chi/chi/middleware"
"sneak.berlin/go/webhooker/internal/logfield"
)
const (
// maxPanicValueBytes bounds the recovered panic value. The value
// is our own text, but a handler is free to build one out of the
// request — panic(fmt.Sprintf("bad %q", r.URL.Path)) — so it is
// charged the same budget the access log gives a field the
// client supplies outright.
maxPanicValueBytes = logfield.MaxBytes
// maxPanicStackBytes bounds the stack, in the same ENCODED bytes
// logfield.Truncate charges everywhere else. Nothing a client
// sends chooses the depth of our own call stack, so this is not
// a safety limit; it is what makes MaxPanicLogLineBytes an
// arithmetic ceiling rather than an observation. A stack is cut
// at its far end, which is net/http's accept frames — the panic
// site and the handler that reached it are at the near end and
// are always kept.
//
// For scale: a handler panicking under the full shipped
// middleware chain produces a stack of roughly 3,690 bytes in a
// record of roughly 3,960, so this budget holds better than
// twice the depth that case reaches. Neither number is an
// invariant — debug.Stack() embeds absolute source paths, so
// both move with where the tree sits, and four checkouts have
// reported records of 3,959, 3,961, 3,984 and 4,026 bytes.
// internal/server's TestPanicThroughProductionRouter asserts the
// ceiling and that the stack arrived uncut, not the figures.
maxPanicStackBytes = 8192
// MaxPanicLogLineBytes is the ceiling on the single line a
// recovered panic writes. It is wider than
// MaxAccessLogLineBytes, which bounds a line written once per
// request, where this one is written once per recovered panic.
//
// panic 512+11 = 523
// stack 8192+11 = 8203
// request_id 128+11 = 139
// fixed portion = 256
// ----
// 9121
//
// The fixed portion is the JSON punctuation, the field names,
// the level, the message, the timestamp at its longest and the
// response_committed boolean.
//
// Stated at 10240 so the figure carries headroom rather than
// sitting on the arithmetic, exactly as MaxAccessLogLineBytes
// is. Both handlers internal/logger can install are covered, for
// the reason given there: logfield.EncodedBytes charges every
// rune the wider of the two.
//
// The 9121 and the 10240 are the invariants here. What follows
// is illustration: with all three growable fields — the stack,
// the panic value and a client-supplied X-Request-Id — driven
// past their budgets at once, TestRecovererBoundsTheStack
// measured 9,009 bytes on the JSON handler and 8,982 to 8,983 on
// the text one in this checkout. Neither is fixed: the stack's
// own content decides where its cut lands, so the figures move by
// a byte or so between runs and with the checkout. The test
// asserts the ceiling and that every growable field was cut,
// never the figures.
MaxPanicLogLineBytes = 10240
)
// recoverResponseWriter records whether the response has been
// committed, which is the one thing the recoverer cannot learn from
// the panic itself: a handler that panics after writing a status has
// already spent the response, and a second WriteHeader would only
// draw net/http's "superfluous response.WriteHeader" complaint
// without changing what the client received.
type recoverResponseWriter struct {
http.ResponseWriter
committed bool
}
func (w *recoverResponseWriter) WriteHeader(code int) {
w.committed = true
w.ResponseWriter.WriteHeader(code)
}
func (w *recoverResponseWriter) Write(b []byte) (int, error) {
// An unheralded Write commits the response just as surely as
// WriteHeader does: net/http sends 200 in front of it.
w.committed = true
//nolint:wrapcheck // Pass the writer's own error through unchanged.
return w.ResponseWriter.Write(b)
}
// Unwrap lets http.ResponseController reach the writer underneath, so
// a handler can still flush or set a write deadline through this
// wrapper.
func (w *recoverResponseWriter) Unwrap() http.ResponseWriter {
return w.ResponseWriter
}
// Recoverer returns middleware that turns a handler panic into one
// structured ERROR record and a 500, rather than a dropped
// connection.
//
// It replaces chi's middleware.Recoverer, which does neither on a
// current Go release. chi v1.5.5's pretty-printer scans the stack for
// a frame beginning "panic(0x", which the runtime has not emitted
// since it started printing "panic({0x...}"; the scan therefore never
// terminates early, every line reaches decorateFuncCallLine, and that
// function slices pkg[strings.Index(pkg, "."):] without checking for
// -1. The resulting second panic escapes chi's own deferred function,
// so its WriteHeader(500) never runs and net/http closes the
// connection reporting its own crash instead of the original one.
// See https://git.eeqj.de/sneak/webhooker/issues/187.
//
// chi v5.3.1 has since fixed both halves of that — it scans for
// "panic(" and guards the index — so upgrading would restore the 500.
// It would not give what this does: v5 still writes an ANSI-coloured
// pretty stack straight to os.Stderr, outside internal/logger, outside
// any budget, at no level the operator set.
//
// Where this sits in the chain is load-bearing, and routes.go states
// it: inside everything that observes the response, so the 500 is
// what the access log records and the metrics count, and outside the
// sentryhttp handler, whose Repanic option depends on something
// further out recovering what it re-raises.
func (s *Middleware) Recoverer() func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(
w http.ResponseWriter,
r *http.Request,
) {
rw := &recoverResponseWriter{ResponseWriter: w}
defer func() {
rvr := recover()
if rvr == nil {
return
}
// http.ErrAbortHandler is a handler stating that it
// is abandoning the connection on purpose, not a
// fault. net/http special-cases it, suppressing both
// the stack trace and any response, so it is passed
// straight back out rather than logged and answered.
err, isError := rvr.(error)
if isError &&
errors.Is(err, http.ErrAbortHandler) {
panic(rvr)
}
s.logPanic(r, rvr, rw.committed)
if rw.committed {
return
}
http.Error(
rw,
http.StatusText(
http.StatusInternalServerError,
),
http.StatusInternalServerError,
)
}()
next.ServeHTTP(rw, r)
})
}
}
// logPanic writes the record. Every field it can grow is truncated to
// a fixed budget, so MaxPanicLogLineBytes holds.
//
// The request is identified by request_id alone rather than by
// repeating the method, URL and address: the access log line for the
// same request carries all of those, already bounded, and — because
// the recoverer runs inside the logging middleware — now carries the
// 500 as its status too. Repeating them here would double those
// budgets against a line already wider than the access log's ceiling,
// to say a second time what one join already says.
func (s *Middleware) logPanic(
r *http.Request,
rvr any,
committed bool,
) {
s.log.Error("handler panic",
"panic", logfield.Truncate(
fmt.Sprint(rvr), maxPanicValueBytes,
),
"stack", logfield.Truncate(
string(debug.Stack()), maxPanicStackBytes,
),
"request_id", logfield.Truncate(
middleware.GetReqID(r.Context()),
maxLogRequestIDBytes,
),
"response_committed", committed,
)
}

645
internal/middleware/recoverer_test.go Normal file
View File

@@ -0,0 +1,645 @@
package middleware_test
import (
"bytes"
"encoding/json"
"io"
"log"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/go-chi/chi"
chimw "github.com/go-chi/chi/middleware"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/middleware"
)
// panicMarker is the panic value the probe handlers raise. The
// recoverer's whole job is to put this string, and not some second
// panic's, in front of an operator.
const panicMarker = "QQORIGINALPANICVALUEQQ"
// probeFuncName appears in the stack of every panic raised below,
// since that is the function raising it. Its presence is how these
// tests tell a real stack from an empty field.
const probeFuncName = "panicProbe"
// committedStatus is the status a handler sends before panicking in
// the already-committed case. It is deliberately not 200, so a test
// cannot pass on net/http's implicit default.
const committedStatus = http.StatusMultiStatus
// recovererProbe is a test server carrying one panicking route,
// behind the production recoverer.
type recovererProbe struct {
server *httptest.Server
// logs holds every record the middleware wrote.
logs *bytes.Buffer
// serverErrors holds everything net/http wrote to its own error
// log. A working recoverer leaves it empty: net/http only reports
// a request when a panic escapes the handler chain, which is the
// failure this issue is about.
serverErrors *bytes.Buffer
}
// newRecovererProbe stands up a real HTTP server — a real listener, a
// real connection, a real client — behind the production recoverer.
//
// A real server rather than an httptest.ResponseRecorder, because a
// recorder cannot express the outcome that made this a defect: chi's
// Recoverer left net/http to close the connection, which a recorder
// records as an ordinary unwritten response while a client sees EOF.
// The status a client actually receives is only observable over a
// socket.
func newRecovererProbe(
t *testing.T,
textHandler bool,
handler http.HandlerFunc,
) *recovererProbe {
t.Helper()
newMiddleware := capturingMiddleware
if textHandler {
newMiddleware = capturingTextMiddleware
}
m, logs := newMiddleware(t)
router := chi.NewRouter()
// The registration order the production router uses: RequestID
// outside so the recoverer's record can name the request,
// Logging outside so the recovered 500 is the status it records.
router.Use(chimw.RequestID)
router.Use(m.Logging())
router.Use(m.Recoverer())
router.Get("/probe", handler)
serverErrors := new(bytes.Buffer)
server := httptest.NewUnstartedServer(router)
server.Config.ErrorLog = log.New(serverErrors, "", 0)
server.Start()
t.Cleanup(server.Close)
return &recovererProbe{
server: server,
logs: logs,
serverErrors: serverErrors,
}
}
// get drives one request at the probe route and returns the response,
// or the transport error if the connection was dropped instead.
func (p *recovererProbe) get(t *testing.T) (*http.Response, error) {
t.Helper()
return p.getWithRequestID(t, "")
}
// getWithRequestID drives the same request carrying a client-supplied
// X-Request-Id. chi's RequestID middleware adopts that header verbatim
// when it is present and only generates a value when it is absent, so
// this is the third growable field on the panic record and the only
// one a client fills outright.
func (p *recovererProbe) getWithRequestID(
t *testing.T,
requestID string,
) (*http.Response, error) {
t.Helper()
req, err := http.NewRequestWithContext(
t.Context(), http.MethodGet, p.server.URL+"/probe", nil,
)
require.NoError(t, err)
if requestID != "" {
req.Header.Set(chimw.RequestIDHeader, requestID)
}
return p.server.Client().Do(req)
}
// wait shuts the server down and blocks until every in-flight request
// has finished, which is what makes the log buffer safe to read.
//
// A client returns as soon as the response is complete — or, for a
// deliberately aborted connection, as soon as it is closed — while the
// access log line for the same request is still being written on the
// server goroutine. It is idempotent, so a test may call it directly
// before reading the buffer itself.
func (p *recovererProbe) wait() {
p.server.Close()
}
// records decodes every JSON log line the probe captured.
func (p *recovererProbe) records(t *testing.T) []map[string]any {
t.Helper()
p.wait()
var out []map[string]any
for line := range strings.SplitSeq(
strings.TrimSpace(p.logs.String()), "\n",
) {
if line == "" {
continue
}
record := map[string]any{}
require.NoError(t, json.Unmarshal([]byte(line), &record))
out = append(out, record)
}
return out
}
// panicRecord returns the single "handler panic" record, failing if
// there is not exactly one.
func (p *recovererProbe) panicRecord(t *testing.T) map[string]any {
t.Helper()
var found []map[string]any
for _, record := range p.records(t) {
if record["msg"] == "handler panic" {
found = append(found, record)
}
}
require.Len(
t, found, 1,
"exactly one panic record expected, log was:\n%s",
p.logs.String(),
)
return found[0]
}
// panicProbe panics with the marker. It is a named function so the
// stack assertions have something to look for.
func panicProbe(http.ResponseWriter, *http.Request) {
panic(panicMarker)
}
func TestRecovererAnswers500AndLogsTheOriginalPanic(t *testing.T) {
t.Parallel()
probe := newRecovererProbe(t, false, panicProbe)
resp, err := probe.get(t)
require.NoError(
t, err,
"a panicking handler must answer, not drop the connection",
)
defer func() { _ = resp.Body.Close() }()
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
assert.Equal(t, http.StatusInternalServerError, resp.StatusCode)
assert.Contains(t, string(body), "Internal Server Error")
record := probe.panicRecord(t)
assert.Equal(t, "ERROR", record["level"])
assert.Equal(t, panicMarker, record["panic"])
assert.Equal(t, false, record["response_committed"])
stack, ok := record["stack"].(string)
require.True(t, ok, "the record must carry a stack")
assert.Contains(
t, stack, probeFuncName,
"the stack must reach the function that panicked",
)
assert.NotContains(
t, stack, "slice bounds out of range",
"a secondary panic must not have occurred",
)
assert.Empty(
t, probe.serverErrors.String(),
"net/http must not have had to report anything",
)
}
// TestRecovererStatusReachesTheAccessLog pins the placement. The
// recoverer runs inside the logging middleware precisely so the status
// it writes is the one the access log records; registered outside it,
// as chi's Recoverer was, the same request is logged as a 200 that the
// client never received.
func TestRecovererStatusReachesTheAccessLog(t *testing.T) {
t.Parallel()
probe := newRecovererProbe(t, false, panicProbe)
resp, err := probe.get(t)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(t, http.StatusInternalServerError, resp.StatusCode)
var access map[string]any
for _, record := range probe.records(t) {
if record["msg"] == "http request" {
access = record
}
}
require.NotNil(t, access, "the request must still be logged")
assert.EqualValues(
t, http.StatusInternalServerError, access["status"],
"the access log must record the status the client got",
)
// The panic record identifies its request by request_id alone,
// so that join has to work.
assert.Equal(
t, access["request_id"],
probe.panicRecord(t)["request_id"],
)
assert.NotEmpty(t, access["request_id"])
}
// TestRecovererRepanicsErrAbortHandler covers the one panic value that
// must not be turned into a 500. net/http documents it as the way a
// handler abandons a connection deliberately and special-cases it,
// suppressing both the response and its own stack report.
func TestRecovererRepanicsErrAbortHandler(t *testing.T) {
t.Parallel()
probe := newRecovererProbe(
t, false,
func(http.ResponseWriter, *http.Request) {
panic(http.ErrAbortHandler)
},
)
resp, err := probe.get(t)
if err == nil {
_ = resp.Body.Close()
}
require.Error(
t, err,
"an aborted handler must not answer with a status",
)
for _, record := range probe.records(t) {
assert.NotEqual(
t, "handler panic", record["msg"],
"a deliberate abort is not a fault to report",
)
}
assert.Empty(
t, probe.serverErrors.String(),
"net/http suppresses ErrAbortHandler; it must still see it",
)
}
// TestRecovererKeepsAnAlreadyCommittedResponse covers a handler that
// panics after sending its status. The bytes are already on the wire,
// so a second WriteHeader would change nothing the client sees and
// would draw net/http's "superfluous response.WriteHeader" report.
func TestRecovererKeepsAnAlreadyCommittedResponse(t *testing.T) {
t.Parallel()
probe := newRecovererProbe(
t, false,
func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(committedStatus)
_, _ = w.Write([]byte("partial"))
panic(panicMarker)
},
)
resp, err := probe.get(t)
require.NoError(t, err)
defer func() { _ = resp.Body.Close() }()
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
assert.Equal(t, committedStatus, resp.StatusCode)
assert.Equal(t, "partial", string(body))
record := probe.panicRecord(t)
assert.Equal(t, panicMarker, record["panic"])
assert.Equal(
t, true, record["response_committed"],
"the record must say why no 500 was sent",
)
assert.NotContains(
t, probe.serverErrors.String(),
"superfluous response.WriteHeader",
)
}
// TestRecovererKeepsAnImplicitlyCommittedResponse is the same case
// without an explicit WriteHeader: a bare Write commits the response
// to 200 just as surely.
func TestRecovererKeepsAnImplicitlyCommittedResponse(t *testing.T) {
t.Parallel()
probe := newRecovererProbe(
t, false,
func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte("partial"))
panic(panicMarker)
},
)
resp, err := probe.get(t)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Equal(
t, true, probe.panicRecord(t)["response_committed"],
)
assert.NotContains(
t, probe.serverErrors.String(),
"superfluous response.WriteHeader",
)
}
// panicLogHandler names one of the two handlers internal/logger can
// install. The recoverer's probe selects between them with a bool
// rather than by constructing one, which is why this does not reuse
// logHandlers() the way the fills reuse escapeFills().
type panicLogHandler struct {
name string
text bool
}
func panicLogHandlers() []panicLogHandler {
return []panicLogHandler{{"json", false}, {"text", true}}
}
// TestRecovererBoundsThePanicRecord holds the record to its stated
// ceiling with a panic value the size of a request. A handler is free
// to build a panic value out of what the client sent, so the value is
// charged a client-sized budget even though the stack is not.
func TestRecovererBoundsThePanicRecord(t *testing.T) {
t.Parallel()
for _, handler := range panicLogHandlers() {
// The fills are internal/middleware's own access log fills,
// shared rather than restated: plain text, the characters
// both handlers escape to two bytes, a bare C0 control, and
// an astral non-printable the text handler spells with a
// ten-byte \U escape.
for fillName, fillRune := range escapeFills() {
t.Run(handler.name+"/"+fillName, func(t *testing.T) {
t.Parallel()
value := strings.Repeat(
fillRune, oversizedSegmentBytes,
) + tailMarker
probe := newRecovererProbe(
t, handler.text,
func(http.ResponseWriter, *http.Request) {
panic(value)
},
)
resp, err := probe.get(t)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(
t, http.StatusInternalServerError,
resp.StatusCode,
)
probe.wait()
for line := range strings.SplitSeq(
strings.TrimSpace(probe.logs.String()), "\n",
) {
assert.LessOrEqual(
t, len(line),
middleware.MaxPanicLogLineBytes,
"log line exceeded its stated bound",
)
assert.NotContains(
t, line, tailMarker,
"the far end of the panic value reached "+
"the log, so nothing truncated it",
)
}
})
}
}
}
// deepPanic recurses to depth and then panics, so the stack itself
// overruns its budget. It is the only way to exercise the stack cut:
// the shipped middleware chain does not come close (see
// TestPanicThroughProductionRouter in internal/server).
func deepPanic(depth int, value string) int {
if depth == 0 {
panic(value)
}
return deepPanic(depth-1, value) + 1
}
// assertEveryFieldWasCut holds each of the record's three growable
// fields to its own budget, which is what the ceiling is the sum of.
// The stack is cut at its far end, so its near end — the panic site —
// has to survive; the request id is the client's own bytes, so its
// cut is the one that bounds an attacker rather than our own call
// depth.
func assertEveryFieldWasCut(t *testing.T, record map[string]any) {
t.Helper()
stack, ok := record["stack"].(string)
require.True(t, ok)
assert.True(
t, strings.HasSuffix(stack, truncationSuffix),
"an oversized stack must be marked as cut",
)
assert.Contains(
t, stack, "deepPanic",
"the near end of the stack must survive the cut",
)
assert.NotContains(
t, stack, "net/http.(*conn).serve",
"the far end is what a cut discards",
)
id, ok := record["request_id"].(string)
require.True(t, ok)
assert.True(
t, strings.HasSuffix(id, truncationSuffix),
"an oversized request id must be marked as cut",
)
assert.LessOrEqual(
t, len(id), maxRequestIDBytes+len(truncationSuffix),
"the request id must be held to its own budget",
)
value, ok := record["panic"].(string)
require.True(t, ok)
assert.True(
t, strings.HasSuffix(value, truncationSuffix),
"an oversized panic value must be marked as cut",
)
}
// TestRecovererBoundsTheStack drives every growable field on the
// record past its budget at once — an oversized stack, an oversized
// panic value and an oversized client-supplied X-Request-Id — over
// both log handlers. It holds that line to the stated ceiling and
// reports what it measured, and it pins that a cut stack keeps its
// near end — the panic site — rather than its far one.
//
// The two fields the test picks the content of — the panic value and
// the request id — are filled with the quotation mark. Both handlers
// escape it to two bytes, which is exactly what logfield charges for
// it, so each of those fields emits every byte of its budget; no fill
// emits more, since logfield charges each rune the wider of the two
// handlers and a field can therefore never emit more than it spent.
// The stack is not a fill: recursion drives it past its budget and
// the cut lands wherever its own content puts it, which is why the
// measured widths move by a byte between runs.
func TestRecovererBoundsTheStack(t *testing.T) {
t.Parallel()
for _, handler := range panicLogHandlers() {
t.Run(handler.name, func(t *testing.T) {
t.Parallel()
value := strings.Repeat(`"`, oversizedSegmentBytes) +
tailMarker
requestID := strings.Repeat(`"`, oversizedSegmentBytes) +
tailMarker
probe := newRecovererProbe(
t, handler.text,
func(http.ResponseWriter, *http.Request) {
_ = deepPanic(512, value)
},
)
resp, err := probe.getWithRequestID(t, requestID)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
require.Equal(
t, http.StatusInternalServerError, resp.StatusCode,
)
probe.wait()
// The text handler does not emit JSON, so the field-level
// assertions run on the JSON one; the line bound below
// is asserted on both, which is the point of the sweep.
if !handler.text {
assertEveryFieldWasCut(t, probe.panicRecord(t))
}
widest := 0
for line := range strings.SplitSeq(
strings.TrimSpace(probe.logs.String()), "\n",
) {
assert.LessOrEqual(
t, len(line), middleware.MaxPanicLogLineBytes,
)
assert.NotContains(t, line, tailMarker)
widest = max(widest, len(line))
}
t.Logf(
"widest line measured: %d bytes (ceiling %d)",
widest, middleware.MaxPanicLogLineBytes,
)
})
}
}
// TestRecovererIgnoresANonPanickingHandler is the negative control:
// the middleware must be inert on the ordinary path.
func TestRecovererIgnoresANonPanickingHandler(t *testing.T) {
t.Parallel()
probe := newRecovererProbe(
t, false,
func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusTeapot)
},
)
resp, err := probe.get(t)
require.NoError(t, err)
require.NoError(t, resp.Body.Close())
assert.Equal(t, http.StatusTeapot, resp.StatusCode)
for _, record := range probe.records(t) {
assert.NotEqual(t, "handler panic", record["msg"])
}
}
// TestRecovererKeepsResponseControllerWorking pins the Unwrap method.
// The middleware wraps the ResponseWriter to learn whether the
// response was committed, and a wrapper without Unwrap hides
// net/http's own writer from http.ResponseController, so a handler
// that flushes or sets a deadline starts failing.
//
// The recoverer is the only middleware in the chain here. The access
// logger's own wrapper does not implement Unwrap, so a chain
// containing it fails this regardless of what the recoverer does;
// what is being pinned is that the recoverer adds no such opacity of
// its own.
func TestRecovererKeepsResponseControllerWorking(t *testing.T) {
t.Parallel()
m, _ := capturingMiddleware(t)
handler := m.Recoverer()(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte("chunk"))
flushErr := http.NewResponseController(w).Flush()
if flushErr != nil {
http.Error(
w, "flush failed",
http.StatusInternalServerError,
)
return
}
},
))
server := httptest.NewServer(handler)
t.Cleanup(server.Close)
req, err := http.NewRequestWithContext(
t.Context(), http.MethodGet, server.URL, nil,
)
require.NoError(t, err)
resp, err := server.Client().Do(req)
require.NoError(t, err)
defer func() { _ = resp.Body.Close() }()
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Equal(t, "chunk", string(body))
}

93
internal/server/export_test.go Normal file
View File

@@ -0,0 +1,93 @@
package server
import (
"log/slog"
"net/http"
"github.com/getsentry/sentry-go"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/middleware"
)
// MaxFormBodySizeForTest exposes the form body cap so tests can
// build requests that sit exactly at, below, and above it.
const MaxFormBodySizeForTest = maxFormBodySize
// ScrubSentryRequestForTest exposes the BeforeSend hook that
// enableSentry installs, so a test can assert on what it leaves in an
// event without standing up a Sentry client.
func ScrubSentryRequestForTest(
event *sentry.Event,
hint *sentry.EventHint,
) *sentry.Event {
return scrubSentryRequest(event, hint)
}
// SentryClientOptionsForTest exposes the exact options enableSentry
// initialises the SDK with, so a test can capture events through the
// production hook wiring rather than a hand-built equivalent.
func SentryClientOptionsForTest(
dsn, release string,
) sentry.ClientOptions {
return sentryClientOptions(dsn, release)
}
// NewRouterForTest builds the real route tree via SetupRoutes with
// the supplied middleware and handlers, bypassing the fx lifecycle
// and the HTTP listener. Tests use it so that route-group middleware
// registration order is exercised exactly as it ships, rather than
// against a hand-rebuilt chain that could drift from routes.go.
func NewRouterForTest(
log *slog.Logger,
cfg *config.Config,
mw *middleware.Middleware,
h *handlers.Handlers,
) http.Handler {
s := &Server{
log: log,
mw: mw,
h: h,
params: ServerParams{Config: cfg},
}
s.SetupRoutes()
return s.router
}
// ProbePattern is the route NewRouterWithProbeForTest adds to the
// production route tree.
const ProbePattern = "/probe"
// NewRouterWithProbeForTest builds the production route tree exactly
// as NewRouterForTest does and then registers probe at ProbePattern,
// so a test can drive a handler that panics through the shipped
// global middleware chain rather than a hand-assembled one. Nothing
// about the chain is rebuilt here: the probe is an extra leaf under
// the same Use() registrations every other route gets.
//
// sentryEnabled selects whether the sentryhttp handler is registered,
// which in production a configured SENTRY_DSN decides. It is a
// parameter because the relationship between that handler's Repanic
// option and the recoverer registered outside it is the thing a test
// has to be able to pin.
func NewRouterWithProbeForTest(
log *slog.Logger,
cfg *config.Config,
mw *middleware.Middleware,
h *handlers.Handlers,
sentryEnabled bool,
probe http.HandlerFunc,
) http.Handler {
s := &Server{
log: log,
mw: mw,
h: h,
params: ServerParams{Config: cfg},
sentryEnabled: sentryEnabled,
}
s.SetupRoutes()
s.router.Handle(ProbePattern, probe)
return s.router
}

Some files were not shown because too many files have changed in this diff Show More