diff --git a/TODO.md b/TODO.md index f99594f..6929c87 100644 --- a/TODO.md +++ b/TODO.md @@ -12,24 +12,14 @@ One issue per unit of work, one branch and one PR per issue: `main` without notice. One `next` -> `main` PR accumulates the milestone; releases are cut from `main` separately. -Issue branches do NOT touch this file — the manager maintains it on `next`. -Every branch editing `TODO.md` conflicts with every other (#112). +The tracker (https://git.eeqj.de/sneak/webhooker/issues) is the record of what +is done and what is under way. This file keeps no list of either. # Status -The milestone (https://git.eeqj.de/sneak/webhooker/milestone/9) is the -authoritative list, and the only place to read a count or a state of play from. -This file records where the project is, not what is in flight: a sentence whose -truth depends on a branch being unmerged is wrong the moment it merges, and this -file has been wrong that way before. - -The durability defect that held the tag has landed -(https://git.eeqj.de/sneak/webhooker/issues/256, commit `8d64259`). Every SQLite -handle opens with WAL journaling and a busy timeout, a bookkeeping write that -fails leaves its delivery in a recoverable state rather than a lying one, and -recovery skips a delivery that already has a successful result row. Final -pre-tag verification exercised it and confirmed it holds. Whatever the milestone -still shows open is what remains before `v1.0.0`. +What is still open for 1.0.0 is in its milestone +(https://git.eeqj.de/sneak/webhooker/milestone/9). `v1.0.0` is not tagged, and +only @sneak tags it. Delivery is at-least-once by design, not by accident: a send whose result row does not land is attempted again, so a receiver can see a duplicate. That is @@ -38,338 +28,31 @@ under Rationale. It is not a defect to re-file. # Next Step -Clear the rest of the open 1.0.0 milestone -(https://git.eeqj.de/sneak/webhooker/milestone/9) and tag `v1.0.0`. Merging -`next` into `main` is a separate act from tagging and waits on neither of those: -`next` is kept mergeable at all times, which is the point of the branch. - -# Completed Steps - -- 2026-08-24 Bind the plaintext HTTP listener deliberately, via `BIND_ADDRESS` - defaulting to `127.0.0.1`, and document the reverse-proxy deployment. A - hostname, an empty value or a value carrying a port is a startup error, and - the `Dockerfile` sets `0.0.0.0` because a loopback bind inside a container is - unreachable (https://git.eeqj.de/sneak/webhooker/issues/268). The same commit - removed the shutdown race: `httpServer` is built in the constructor rather - than assigned from the serving goroutine, which orders the write before every - fx hook and rules out the nil dereference a SIGTERM arriving first would have - caused, and `sentryEnabled` is an `atomic.Bool` - (https://git.eeqj.de/sneak/webhooker/issues/226) -- 2026-08-24 Remove inbound request signature verification. The entrypoint UUID - is the authentication secret, so the per-entrypoint shared secret, the - `internal/signature` package, the receiver check, the model fields and the - forms are all gone. This reverses the feature that landed earlier in the same - milestone (https://git.eeqj.de/sneak/webhooker/issues/67, - https://git.eeqj.de/sneak/webhooker/issues/279) -- 2026-08-24 Stamp the build version into the binary and render it in the UI - footer. `script/version` is the single source — `$VERSION`, else - `git describe --tags --always --dirty`, else `unknown` — so a `make build` - binary and a `make docker` image from one checkout report the same thing, and - nothing in it varies between two builds of the same commit, which the release - gate's byte-identical assertion would catch - (https://git.eeqj.de/sneak/webhooker/issues/253) -- 2026-08-24 Derive cookie `Secure` and CSRF strictness from the request - transport rather than from `WEBHOOKER_ENVIRONMENT`. Behind a real TLS proxy - with the environment left at its `dev` default, the session cookie silently - lost `Secure` while the CSRF cookie on the same response kept it. - `X-Forwarded-Proto` is now matched case-insensitively on its first - comma-separated element, so `HTTPS` and `https, http` no longer fall to the - relaxed CSRF path (https://git.eeqj.de/sneak/webhooker/issues/269) -- 2026-08-24 Roll back a failed webhook deletion instead of committing it. A - failing delete committed whatever had already succeeded, hard-deleted the - per-webhook event database anyway, and redirected as though it had worked — - orphaned config plus permanently destroyed history, reported as success. All - three delete positions now roll back with the event database intact - (https://git.eeqj.de/sneak/webhooker/issues/262) -- 2026-08-24 Name a deleted target on its historical deliveries, marked - `(deleted)`, rather than leaving the event log unable to say where a delivery - went. A deleted target's credentials stay masked exactly as a live one's, and - it cannot become deliverable again through the receiver, resubmit, replay, the - edit form or the toggle (https://git.eeqj.de/sneak/webhooker/issues/211) -- 2026-08-24 Bound both request-controlled `/metrics` label dimensions, so the - unauthenticated receiver is no longer a memory-exhaustion vector: `handler` - carries the chi route pattern, and `method` folds anything chi cannot route - onto a single `(unmatched)` sentinel. Both were reproduced before the fix — - 300 random method tokens took the series count from 106 to 7,631, and path - flooding reached 62,532 — and a label audit across a live scrape found no - third unbounded dimension (https://git.eeqj.de/sneak/webhooker/issues/254, - https://git.eeqj.de/sneak/webhooker/issues/261) -- 2026-08-24 Validate `max_retries` on both target forms. `abc`, `2.7` and `-5` - silently became 0 — fire-and-forget — including on the edit path, where it - destroyed a working value, and `999999999` stored verbatim. The ceiling of 20 - is the `max` both templates already declared - (https://git.eeqj.de/sneak/webhooker/issues/221) -- 2026-08-24 Resubmit a stored event as a new undelivered event, so a backend - under development can be tested against real captured traffic. Per-delivery - replay cannot serve that: it re-sends one finished delivery to its own - original target, and a target created for a dev backend has no prior delivery - to replay. Resubmit re-injects the stored event at the top of the receiver - path and fans it out to whatever targets are active now - (https://git.eeqj.de/sneak/webhooker/issues/250) -- 2026-08-20 Take an exclusive lock on `DATA_DIR` at startup, so two instances - on one directory cannot both deliver - (https://git.eeqj.de/sneak/webhooker/issues/201) -- 2026-08-20 Shut down the app when the HTTP listener fails. The `OnStart` hook - returned as soon as the serving goroutine was spawned, so a failed listen left - fx reporting RUNNING and a live process with nothing bound — invisible to - systemd and Docker restart policies - (https://git.eeqj.de/sneak/webhooker/issues/200) -- 2026-08-20 Stop target credentials leaking into the per-webhook event - databases (https://git.eeqj.de/sneak/webhooker/issues/206), log SQL with - placeholders rather than bound values - (https://git.eeqj.de/sneak/webhooker/issues/207), and fail loudly on half-set - metrics auth credentials (https://git.eeqj.de/sneak/webhooker/issues/205) -- 2026-08-20 Read queue depths with `Find`, not `Scan`. `Scan` swaps GORM's own - trace recorder in for the logging adapter, and that recorder does not - implement `gorm.ParamsFilter`, so those statements logged their bound values - interpolated and bypassed the suppression above. The two units gated green - against a `next` that lacked the other, and `next` went red when both landed - (https://git.eeqj.de/sneak/webhooker/issues/234) -- 2026-08-20 Render per-attempt delivery detail in the event log - (https://git.eeqj.de/sneak/webhooker/issues/202) and add replay of a - terminally failed delivery (https://git.eeqj.de/sneak/webhooker/issues/203) -- 2026-08-20 Expose delivery metrics on `/metrics` - (https://git.eeqj.de/sneak/webhooker/issues/209) and document the backup, - restore and upgrade procedures - (https://git.eeqj.de/sneak/webhooker/issues/210) -- 2026-08-20 Add a `webhooker resetpw` subcommand and a bootstrap banner. The - admin bootstrap password was printed once among roughly 45 fx lines, and under - `docker run -d` went to container logs subject to rotation; there was no reset - path at all, so recovery meant hand-deleting the users row, documented - nowhere. The password is read from stdin or generated, never from argv where - `/proc` would publish it (https://git.eeqj.de/sneak/webhooker/issues/208) -- 2026-08-20 Add `ALLOWED_EGRESS_CIDRS`, an allowlist-only escape hatch for the - SSRF guard, so a self-hosted proxy can forward into the operator's own - network. The guard's always-blocked set cannot be reopened by configuration - (https://git.eeqj.de/sneak/webhooker/issues/204) -- 2026-08-20 Harden operator-set target headers, which were carried unsafely - across a redirect (https://git.eeqj.de/sneak/webhooker/issues/233) -- 2026-08-20 Add a target edit form with headers and timeout fields - (https://git.eeqj.de/sneak/webhooker/issues/127) -- 2026-08-18 Raise `script/test`'s per-package timeout from 30s to 90s, matching - the org-wide backstop. `go test` applies `-timeout` per package, and - `internal/handlers` had grown past the old budget: a cache-defeated build - failed outright at `GOMAXPROCS=4`, and every run under deliberate host load - breached 30s. The measurement table lives in the script (#194) -- 2026-08-18 Re-sync `REPO_POLICIES.md` from `prompts`. The local copy was stale - and still mandated a 20s test target with a 30s timeout, which the org - replaced with a 60s cap and a 90s backstop. A synced copy is not a source; - reading it as one nearly produced a PR against `prompts` proposing a change - already merged there (#196) -- 2026-08-18 Report handler panics through the logger and answer 500. chi - v1.5.5's `Recoverer` scans for a `panic(0x` frame the runtime no longer emits, - then indexes `pkg[-1:]`, so it panicked inside its own stack printer before - writing a byte: the recovery never ran, the client got a dropped connection - instead of a 500, and the original panic was lost. A local middleware replaces - it, bounded by `MaxPanicLogLineBytes` (#187) -- 2026-08-18 Route GORM's logger through `slog` and bound it. Every `gorm.Open` - left `logger.Default` in place at `Warn` with `IgnoreRecordNotFoundError` - false, so **every record-not-found printed the fully interpolated SQL to - stdout** — including the client-chosen path on `/webhook/{uuid}` and the - submitted username on the login form, at no level the operator set and outside - `internal/logger` entirely. Three call sites, not the two the issue named - (#178) -- 2026-08-18 Bound every `slog` line against client-chosen text. Eight sites - reachable unauthenticated, found by reading every `slog` call in the tree - rather than only the one reported; the budget moved to a shared - `internal/logfield` so no second truncation exists. `DEBUG` being off by - default is not a bound and is not treated as one (#176) -- 2026-08-18 Stop a slow host turning a login-guard test into a segfault. A - non-fatal `assert` on an acquire result was dereferenced on the next line, so - one timing miss killed the whole `internal/middleware` binary and reddened CI - for unrelated PRs. The fix also removed a real production race — `acquire` - could shed a request with a slot standing free, because Go picks uniformly - among ready `select` cases (#186) -- 2026-08-18 Send the chi route pattern to Sentry rather than the concrete path. - The receiver's path carries the entrypoint capability token, so every Sentry - event from `/webhook/{uuid}` shipped a live credential to a third party. - Request `Data`, `QueryString`, `Cookies` and `Env` are dropped and headers - reduced to an allowlist (#179) -- 2026-08-18 Read form fields from the POST body only. `r.FormValue` merges the - query string, so a login could be driven by URL parameters — putting the - password somewhere that lands in access logs, proxy logs and browser history - (#160) -- 2026-08-18 Verify login credentials before spending rate-limit budget, so a - flood of wrong passwords cannot lock out the account it is guessing at. The - manager took this decision rather than stall the queue; it is flagged on the - issue for reversal (#150) -- 2026-08-18 Run all linting in Docker via `Dockerfile.lint`. Host lint was - wrong in both directions from version skew and shared caches. `script/lint` - asserts the summary line, because `--no-cache-filter` silently ignores a stage - name it does not match — the flag that makes the gate meaningful fails open - (#109) -- 2026-08-18 Serve an event's full stored body over HTTP. The list query - truncates for rendering, and that truncated value was the only way to read a - body, so the full payload was unreachable (#157) -- 2026-08-18 Bound the access log line against client-chosen text. - `internal/logfield` budgets by _encoded_ bytes, not runes, so a handler's JSON - escaping cannot multiply a field past its allowance (#146) -- 2026-08-18 Mark superseded CI commits `failure` rather than `skipped`. A - skipped run rolls up green, so a commit that was never tested reported success - (#152) -- 2026-08-18 Set `fx.StopTimeout` inside the container stop grace, so shutdown - hooks are bounded by a deadline the orchestrator will actually honour rather - than being killed mid-flush (#134) -- 2026-08-17 Bucket IPv6 rate-limit keys by `/64`. A single allocation hands out - 2^64 addresses, so per-address keying let one client mint unlimited buckets. - Manager decision, recorded on the issue (#125) -- 2026-08-17 Correct release-blocking README and startup-warning inaccuracies, - including claims about behaviour the code does not have (#151) -- 2026-08-17 Fetch and verify Alpine.js at build time against - `static/vendor.sha256` instead of committing the minified blob, so the - dependency is pinned by hash rather than by trust (#145) -- 2026-08-17 Bound the event log's rendered bodies in the query itself, so a - large stored payload cannot be read into memory just to be truncated for - display (#135) -- 2026-08-17 Mask the `http` target's destination URL in the UI: it can carry a - bearer credential in its path or query, and was rendered verbatim. Manager - decision to mask unconditionally (#115) -- 2026-08-14 Bound shutdown hooks by their stop context, so a hook that hangs - cannot hold the process past its grace period (#102) -- 2026-08-14 Render templates via a buffer rather than the `ResponseWriter`, so - a template error part-way through cannot commit a 200 and then fail — the - response is written only once it is whole (#123) -- 2026-08-14 Align the session codec's max-age with the 7-day absolute cap. The - codec accepted cookies the session layer considered expired, so the cap was - enforced in one place and not the other (#108) -- 2026-08-12 Warn when `TRUSTED_PROXIES` is empty in production, where the safe - default silently discards forwarded headers and every client rate-limits as - the proxy's address (#149) -- 2026-08-12 Bound the receiver rate limit per client IP across the whole - `/webhook/*` route. The existing limiter keyed on the request path and - `/webhook/{uuid}` matches any single segment, so a client that invented a - fresh path per request minted a fresh bucket per request: the limit on the - only unauthenticated endpoint bounded nothing in aggregate, and every request - still cost an entrypoint lookup before it 404ed. An outer limiter keyed on the - client address alone now bounds that, chained in front of the unchanged - per-entrypoint limiter (#139) -- 2026-08-12 Correct release-blocking documentation inaccuracies: the README - promised manual redelivery in the present tense in three places when nothing - implements it (the same false claim also sat in the doc comment that was its - source text), the env table omitted `RETENTION_SWEEP_INTERVAL`, and `TODO.md` - itself omitted five landed units (#141) -- 2026-08-12 Make the CI gate execute the checks it reports on. The workflow now - writes a build-context fingerprint before calling `script/cibuild`, so a code - commit invalidates the `COPY` layer of the lint and builder stages while a - docs-only commit still replays from cache; a superseding run also rewrites the - `failure` status Gitea leaves on commits it cancelled and never tested. - Verified by pushing a deliberately broken test and watching CI go red (#119) -- 2026-08-12 Require a positive `RETENTION_SWEEP_INTERVAL`: a non-positive value - reached `time.NewTicker` in both the retention reaper and the archive sweeper, - panicking two goroutines with no recover after startup had already reported - success (#140) -- 2026-08-12 Bound the `X-Forwarded-For` scan's allocation to the hop cap: the - reverse walk cuts entries with `strings.LastIndexByte` instead of joining and - splitting, so a 1 MB header allocates 16 bytes rather than 1.6 MB per request - on the unauthenticated receiver. Semantics proven unchanged by differential - testing against the previous implementation (#133) -- 2026-08-12 Cap the `X-Forwarded-For` hop walk at 64 entries, so an - attacker-supplied chain cannot burn unbounded CPU in the rate-limit key - function; running off the end falls back to the peer address (#124) -- 2026-08-12 Gate forwarded-header trust behind a `TRUSTED_PROXIES` CIDR list: - all three rate limiters key on the connection's own address unless the direct - peer is a configured proxy, in which case `X-Forwarded-For` is walked right to - left for the first non-proxy hop. Default trusts nothing, and a - set-but-unparseable value aborts startup. Before this, any client could mint a - fresh bucket or drain another's by rotating a spoofed header (#88) -- 2026-08-11 Web UI cleanup: nav terminology unified on Webhooks, the Profile - settings placeholder removed, a progressive-enhancement copy button for the - entrypoint URL, and retention form copy that states the actual policy - (deletion by the reaper, 0 retains forever) (#57) -- 2026-08-11 Mask the webhook credential in delivery errors and logs: Go embeds - the request URL in `*url.Error`, so every transport failure persisted the full - Slack webhook URL into the per-webhook event database via - `DeliveryResult.Error`, a field a future REST API would have served. - `maskURLError` drops path, query and userinfo while preserving the wrapped - cause, so `errors.Is`/`As` and `Timeout()` still work and DNS, TLS and timeout - failures still read differently (#118) -- 2026-08-11 Rate-limit the public webhook receiver endpoint - (`RECEIVER_RATE_LIMIT`, default 120/min), keyed on client IP plus entrypoint - path so one entrypoint cannot exhaust another's budget; over-limit requests - get 429 with `Retry-After`. It was the one unauthenticated, internet-facing - endpoint with no limit at all (#64) -- 2026-08-11 Enforce the body size limit before CSRF parses the form: - `MaxBodySize` is now first in all four form-parsing route groups, so an - oversized request is rejected with 413 instead of being read in full by the - CSRF middleware before any cap applied (#90) -- 2026-08-11 Mask target config on the source detail page, which rendered the - stored blob verbatim and so exposed the Slack incoming-webhook URL — a bearer - credential that cannot be revoked per-holder. Config reaches the template only - as a `TargetView` of labelled fields, and header values are rendered as a - count (#113) -- 2026-08-11 Allow `retention_days` of 0 to mean retain forever, via a sentinel - written in `BeforeSave` so the GORM column default cannot win the race. Also - bounds the reaper's cutoff arithmetic: day counts above 106751 overflowed - `time.Duration` and wrapped the cutoff into the future, where every row - matched and the sweep deleted everything (#79) -- 2026-08-09 Inactivity-based session timeout: sliding idle expiry - (`SESSION_IDLE_TIMEOUT`, default `24h`) refreshed on authenticated requests, - with the 7-day absolute cap kept as an independent backstop that activity - never extends (#66) -- 2026-08-09 Restart recovery and the 60s retry sweep terminally fail an - orphaned `retrying` delivery whose target type no longer supports retries, - recording a `DeliveryResult` with the reason instead of leaving the delivery - stuck forever (#82) -- 2026-08-09 Root the delivery engine's worker pool and the retention reaper's - sweep loop at `context.Background()` rather than the fx `OnStart` hook context - (#97), which carries fx's 15s start timeout and killed both roughly fifteen - seconds after boot: the proxy silently stopped delivering webhooks entirely, - and the reaper never ran a single sweep under its default one-hour interval -- 2026-08-09 Archive writer lifecycle (#89): deleting a webhook (or its last - `database` target) evicts the cached archive writer and closes its handle - while deliberately leaving `archive-{webhookID}.db` on disk, and a new - `ArchiveSweeper` prunes idle archives on the existing - `RETENTION_SWEEP_INTERVAL` without ever creating an archive file -- 2026-08-09 Configuration parsing fails loudly on set-but-unparseable - environment values: `envInt` removed in favour of `envPositiveInt` plus a - `PORT` range check, `envBool` now parses with `strconv.ParseBool`, and - defaults apply only to unset variables (#80) -- 2026-08-07 Automatic event retention cleanup based on `retention_days`, - deleting expired events, deliveries, and delivery results from each - per-webhook event database (#63) -- 2026-08-07 Update golangci-lint to v2.12.2 (Docker image digest in - `Dockerfile`, release-archive sha256 pins in `script/bootstrap`), adopt the - canonical `.golangci.yml` (v2 `linters.settings` layout so - `lll`/`funlen`/`cyclop`/`dupl` thresholds actually apply), and fix all newly - surfaced lint findings -- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile - shims, README Entrypoints section -- 2026-03-25 pin golangci-lint Docker image for linting (#55) -- 2026-03-18 CSRF middleware detects TLS per-request, fixing login over plain - HTTP and behind reverse proxies (#54) -- 2026-03-17 root path redirects based on auth state (#52) -- 2026-03-17 CSRF protection, SSRF prevention for HTTP delivery targets with DNS - rebinding defense, and per-IP login rate limiting (#42) -- 2026-03-17 Slack target type for incoming webhook notifications (#47) -- 2026-03-17 Dockerfile absolute paths and static linking (#49); absolute dev - DATA_DIR default and clarified env docs (#46) -- 2026-03-05 security headers middleware, session regeneration on login, request - body size limits (#41) -- 2026-03-04 tests for delivery, middleware, and session packages (#32); removed - the build-architecture global (#31) -- 2026-03-04 1.0 MVP merge: Webhook/Entrypoint/Target rename, core delivery - engine with bounded worker pool and circuit breaker, parallel fan-out, - per-webhook event databases, management UI (#16) -- 2026-03-01 repo brought to REPO_POLICIES standards; TODO.md folded into README - (#6) +@sneak runs webhooker in limited production, then tags `v1.0.0`. Merging `next` +into `main` is a separate act from tagging and waits on neither of those: `next` +is kept mergeable at all times, which is the point of the branch. # Future Steps -- Delivery status and retry management UI. Replay of a terminally failed - delivery and per-attempt detail already landed - (https://git.eeqj.de/sneak/webhooker/issues/203, - https://git.eeqj.de/sneak/webhooker/issues/202) -- Per-webhook rate limiting in the receiver handler (per-webhook config plus - handler enforcement; global limits must not apply to receiver endpoints) -- API key authentication for programmatic access (APIKey model exists; Bearer - token middleware does not) +- Per-webhook rate limits, set in the web UI and enforced in the receiver + handler, on top of the receiver's existing limits, which `RECEIVER_RATE_LIMIT` + sets for the whole server (see Rate Limiting in the README) +- API key authentication for programmatic access (the `APIKey` model exists, but + nothing creates a key and no middleware checks one) - REST API v1 - CRUD for webhooks, entrypoints, targets - event viewing and filtering endpoints - event redelivery endpoint - OpenAPI specification -- Analytics dashboard: success rates, response times, volume -- A remember-me option at login +- Analytics in the web UI: response times summarised per webhook or per target, + and figures for all webhooks together. The webhook page already shows event, + delivery and failure counts and failure percentages, the target list counts + each target's deliveries, the event log and each event's page show every + attempt's duration, and `/metrics` carries server-wide counts and delivery + durations +- A remember-me option at login. Every session already survives a browser + restart: it lasts up to 7 days, and ends sooner after `SESSION_IDLE_TIMEOUT` + (default 24 hours) without activity - Password reset flow for a forgotten password over the web. The authenticated password _change_ flow already landed, and a lost password is recoverable from the console with `webhooker resetpw` @@ -377,7 +60,13 @@ Clear the rest of the open 1.0.0 milestone - Later, nice to have - email delivery target type - SNS and S3 delivery targets - - data transformations (e.g. webhook to Slack message formatting) - - JSONL file delivery with periodic S3 upload - - webhook event search and filtering - - multi-user with role-based access control + - data transformations (e.g. a configurable Slack message format; the Slack + target sends one fixed format today) + - JSONL file delivery with periodic S3 upload (the `database` target already + archives events to SQLite files, rotated hourly, daily or monthly if set, + and the webhook page offers them as one gzipped JSON download) + - webhook event search, and filtering beyond the event log's failed and + pending views + - multi-user with role-based access control (each webhook already belongs to + one account, but there is only the account made at first boot and no way + to add another)