clawbot
9969694a47
All checks were successful
check / check (push) Successful in 3m25s
The admin bootstrap password was printed once, as one line among roughly 45 fx lines, and under docker run -d went to container logs subject to rotation. There was no reset path at all -- no subcommand, no forgot-password flow, no env override -- so recovery meant hand-deleting the users row from webhooker.db, which was documented nowhere. Adds webhooker resetpw [-generate] <username>. The password is read from stdin or generated with the existing crypto/rand helper, never taken from argv where /proc would publish it. It reuses the existing Argon2id hashing rather than reimplementing the parameters, and writes a single UPDATE only after the hash is complete, so no failure can leave an account with no usable password. An unknown username is a hard error and never creates an account. It refuses to run against a DATA_DIR held by a live instance, via the exclusive lock from #201. DATA_DIR and webhooker.db are checked to exist before the lock is acquired, so a mistyped path creates nothing -- neither a directory tree nor a stray lock file. The bootstrap password now appears exactly once, in a distinct banner written straight to a caller-named writer rather than as an fx log line.
2545 lines
134 KiB
Markdown
2545 lines
134 KiB
Markdown
# webhooker
|
||
|
||
webhooker is a self-hosted webhook proxy and store-and-forward service
|
||
written in [Go](https://golang.org) by
|
||
[@sneak](https://sneak.berlin). It receives webhooks from external
|
||
services, durably stores them, and delivers them to configured targets
|
||
with retry support, logging, and observability. Category: infrastructure
|
||
/ web service. License: MIT.
|
||
|
||
## Getting Started
|
||
|
||
### Prerequisites
|
||
|
||
- Go 1.26.1+ (the version in `go.mod`)
|
||
- Docker (for linting, for the test stage of the CI gate, and for
|
||
containerized deployment)
|
||
- `curl`, used by `script/fetch-assets` to download the third-party
|
||
browser assets, which are not committed (`make bootstrap` installs
|
||
it if missing)
|
||
|
||
golangci-lint is not a prerequisite and must not be installed on the
|
||
host: `script/bootstrap` does not install it, and `make lint` runs the
|
||
digest-pinned linter image via `Dockerfile.lint`.
|
||
|
||
### Quick Start
|
||
|
||
```bash
|
||
# Clone the repo
|
||
git clone https://git.eeqj.de/sneak/webhooker.git
|
||
cd webhooker
|
||
|
||
# Install Go dependencies and the third-party browser assets.
|
||
# `make deps` alone is not enough: it only runs go mod download/tidy,
|
||
# and the checks below need the fetched assets.
|
||
make bootstrap
|
||
|
||
# Run all checks (test, lint, format check)
|
||
make check
|
||
|
||
# Run in development mode. DATA_DIR defaults to /var/lib/webhooker in
|
||
# every environment, so set it (in .env or the shell) to a writable
|
||
# directory when running from a clone.
|
||
DATA_DIR=./data make dev
|
||
|
||
# Build Docker image
|
||
make docker
|
||
```
|
||
|
||
### Development Commands
|
||
|
||
```bash
|
||
make bootstrap # Install all dependencies (idempotent)
|
||
make setup # Bootstrap + install git pre-commit hook
|
||
make assets # Fetch + verify third-party browser assets
|
||
make fmt # Format code (gofmt + goimports)
|
||
make fmt-check # Fail if gofmt would change anything (writes nothing)
|
||
make lint # Run golangci-lint in Docker (Dockerfile.lint)
|
||
make test # Run tests with race detection
|
||
make check # test + lint + fmt-check (CI gate)
|
||
make build # Build binary to bin/webhooker
|
||
make run # build, then run ./bin/webhooker
|
||
make dev # go run ./cmd/webhooker
|
||
make deps # go mod download + go mod tidy
|
||
make docker # Build Docker image
|
||
make hooks # Install git pre-commit hook that runs script/precommit
|
||
make css # Regenerate static/css/tailwind.css (needs tailwindcss)
|
||
make clean # Remove bin/
|
||
```
|
||
|
||
### Configuration
|
||
|
||
All configuration is via environment variables. For local development,
|
||
you can place variables in a `.env` file in the project root (loaded
|
||
automatically via `godotenv/autoload`).
|
||
|
||
The environment is selected by setting `WEBHOOKER_ENVIRONMENT` to `dev`
|
||
or `prod` (default: `dev`). The setting controls several behaviors:
|
||
|
||
| Behavior | `dev` | `prod` |
|
||
| --------------------- | -------------------------------- | ------------------------------- |
|
||
| CORS | Allows any origin (`*`) | Disabled (no-op) |
|
||
| Session cookie Secure | `false` (works over plain HTTP) | `true` (requires HTTPS) |
|
||
|
||
The CSRF cookie's `Secure` flag and Origin/Referer validation mode are
|
||
determined per-request based on the actual transport protocol, not the
|
||
environment setting. The middleware checks `r.TLS` (direct TLS) and the
|
||
`X-Forwarded-Proto` header (TLS-terminating reverse proxy) to decide:
|
||
|
||
- **Direct TLS or `X-Forwarded-Proto: https`**: Secure cookies, strict
|
||
Origin/Referer validation.
|
||
- **Plaintext HTTP**: Non-Secure cookies, relaxed Origin/Referer
|
||
checks (token validation still enforced).
|
||
|
||
This means CSRF protection works correctly in all deployment scenarios:
|
||
behind a TLS-terminating reverse proxy, with direct TLS, or over plain
|
||
HTTP during development. When running behind a reverse proxy, ensure it
|
||
sets the `X-Forwarded-Proto: https` header.
|
||
|
||
All other differences (log format, security headers, etc.) are
|
||
independent of the environment setting — log format is determined by
|
||
TTY detection, and security headers are always applied.
|
||
|
||
| Variable | Description | Default |
|
||
| ----------------------- | ----------------------------------- | -------- |
|
||
| `WEBHOOKER_ENVIRONMENT` | `dev` or `prod` | `dev` |
|
||
| `PORT` | HTTP listen port | `8080` |
|
||
| `DATA_DIR` | Directory for all SQLite databases | `/var/lib/webhooker` |
|
||
| `DEBUG` | Enable debug logging | `false` |
|
||
| `MAINTENANCE_MODE` | Report `maintenanceMode: true` in the healthcheck JSON. It does not change how any request is served — no maintenance page exists | `false` |
|
||
| `METRICS_USERNAME` | Basic auth username for `/metrics`. Must be set together with `METRICS_PASSWORD`; one without the other fails startup | `""` |
|
||
| `METRICS_PASSWORD` | Basic auth password for `/metrics`. Must be set together with `METRICS_USERNAME`; one without the other fails startup | `""` |
|
||
| `SENTRY_DSN` | Sentry error reporting DSN | `""` |
|
||
| `RETENTION_SWEEP_INTERVAL` | How often the retention reaper and archive sweeper run (Go duration, must be positive) | `1h` |
|
||
| `SESSION_IDLE_TIMEOUT` | Idle session timeout (Go duration) | `24h` |
|
||
| `RECEIVER_RATE_LIMIT` | Receiver requests/minute per IP per entrypoint (10x that per IP across the route) | `120` |
|
||
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted (unset: all clients behind a proxy share one rate-limit bucket; a correct login password is never throttled either way) | `""` (none) |
|
||
|
||
#### Metrics credentials
|
||
|
||
`METRICS_USERNAME` and `METRICS_PASSWORD` are set together or not at
|
||
all. With both set, `/metrics` is served behind basic auth. With
|
||
neither set, the route is not registered and returns 404. With one set
|
||
and the other empty or unset, the process refuses to start and exits
|
||
non-zero with an error naming both variables — mounting the endpoint
|
||
on the username alone would publish it behind a password that is the
|
||
empty string, and quietly withholding it would deny an endpoint that
|
||
was asked for. The `hasMetricsAuth` field in the startup log and the
|
||
existence of the route are the same value, so they cannot disagree.
|
||
|
||
#### Single-instance lock
|
||
|
||
Exactly one webhooker process may use a `DATA_DIR` at a time. Two
|
||
processes sharing one open the same databases and each run delivery
|
||
recovery over the same rows, so every pending delivery goes out twice —
|
||
duplicate delivery to your endpoints, from nothing worse than an
|
||
overlapping deploy or a double start.
|
||
|
||
At startup, before anything opens a database, the process takes an
|
||
exclusive advisory lock (`flock(2)`) on `{DATA_DIR}/webhooker.lock` and
|
||
holds it for its lifetime. A second process pointed at the same
|
||
directory prints a message naming it and exits non-zero:
|
||
|
||
```
|
||
webhooker: data directory is already in use by another instance: /var/lib/webhooker (/var/lib/webhooker/webhooker.lock). Only one webhooker may use a data directory: two both run delivery recovery over the same rows and both deliver
|
||
```
|
||
|
||
The lock is the kernel's, not the file's: it is released when the
|
||
process exits, including `kill -9`, so a leftover `webhooker.lock`
|
||
never blocks a restart and must not be deleted by hand. The file is
|
||
also left in place on a clean shutdown, deliberately — unlinking it
|
||
would let the next process lock a fresh inode while a third still held
|
||
the old one.
|
||
|
||
To run two webhookers on one host, give each its own `DATA_DIR`.
|
||
|
||
`flock(2)` is host-local and per-inode: it arbitrates between processes
|
||
and containers sharing a volume or bind mount on one machine, but not
|
||
between hosts on a network filesystem, and a `DATA_DIR` inside a
|
||
container's own writable layer is not shared with anything. On a
|
||
filesystem that refuses `flock` outright, startup fails closed — the
|
||
process reports the error and refuses to start rather than running
|
||
unlocked.
|
||
|
||
#### Trusted proxies
|
||
|
||
`TRUSTED_PROXIES` is a comma-separated list of CIDR blocks (a bare
|
||
address such as `192.168.1.7` is accepted and treated as a single
|
||
host), for example `192.168.1.7, 2001:db8::5`. It decides whose
|
||
`X-Forwarded-For` header the rate limiters believe, so it should name
|
||
the addresses of your reverse proxies and nothing else.
|
||
|
||
`X-Forwarded-For` is honoured **only** when the connecting peer is
|
||
inside one of these blocks; for every other peer the client identity is
|
||
the connection's own address and the header is ignored. The default is
|
||
the empty list, which trusts nobody — anything else would let any
|
||
client pick its own rate limit bucket, minting a fresh one per request
|
||
or draining someone else's. Set it to the address of your reverse
|
||
proxy, and to nothing wider. A set but unparseable value aborts
|
||
startup.
|
||
|
||
That default is safe against forged headers, but leaving it unset in
|
||
production has a cost you must know about. Production runs behind a
|
||
TLS-terminating reverse proxy, so with `TRUSTED_PROXIES` unset every
|
||
request keys on the proxy's own address and all clients share a single
|
||
bucket per limit. The receiver limits become service-wide ceilings,
|
||
and the login endpoint's failure counting collapses onto one key, so a
|
||
stranger's wrong passwords throttle every other client's wrong
|
||
passwords.
|
||
|
||
What it cannot do is lock the operator out. The login endpoint
|
||
verifies credentials **before** it consults any limit and charges only
|
||
failures, so a correct password is never throttled no matter how full
|
||
the bucket is. See [Rate Limiting](#rate-limiting).
|
||
|
||
The remedy is to set `TRUSTED_PROXIES` to your reverse proxy's
|
||
address, which restores per-client buckets. webhooker logs a warning
|
||
at startup whenever `TRUSTED_PROXIES` is empty, in every environment —
|
||
not only when `WEBHOOKER_ENVIRONMENT=prod`, because that variable
|
||
defaults to `dev` and an operator who never set it is precisely the
|
||
one at risk. The warning is informational when nothing proxies to the
|
||
process: with no proxy in front, the peer address is the client's own
|
||
and the buckets are already per-client. See
|
||
[Rate Limiting](#rate-limiting) for what each limit shares.
|
||
|
||
`X-Real-IP` and `True-Client-IP` are **never** read, from any peer.
|
||
Reverse proxies append to `X-Forwarded-For` but forward other client
|
||
headers verbatim, so a single-valued header is client-controlled even
|
||
behind a trusted proxy.
|
||
|
||
Within a trusted request, `X-Forwarded-For` is read right to left,
|
||
because the rightmost entry is the one the nearest proxy appended and
|
||
everything left of it may have been written by the client. The first
|
||
hop that is not itself a trusted proxy is taken as the client. A hop
|
||
that is not a bare IP address — `ip:port`, a bracketed IPv6 literal,
|
||
the token `unknown` — ends the walk and the peer address is used
|
||
instead, since past such an entry the chain is not the shape assumed
|
||
here. The peer address is likewise used when the header is absent or
|
||
every hop in it is a trusted proxy.
|
||
|
||
Two operator requirements follow:
|
||
|
||
- Your proxy must **append** the peer address to `X-Forwarded-For`
|
||
(nginx `$proxy_add_x_forwarded_for`, HAProxy `option forwardfor`,
|
||
Caddy and AWS ALB by default), and must append a bare address with
|
||
no port.
|
||
- List proxy hosts **only**. Any address inside `TRUSTED_PROXIES`
|
||
chooses its own rate-limit key: its `X-Forwarded-For` is walked, so
|
||
it can name a different address on every request to get a fresh
|
||
bucket each time, or name another client's address to drain that
|
||
client's bucket. Never list a block that also covers clients — a
|
||
broad `10.0.0.0/8` on a network where clients live in the same range
|
||
makes all three limits, including the unauthenticated webhook
|
||
receiver, silently bypassable by every client in the block.
|
||
|
||
#### Sessions
|
||
|
||
Sessions are bounded by two independent clocks, and end at whichever
|
||
one runs out first:
|
||
|
||
- **Idle expiry** (`SESSION_IDLE_TIMEOUT`, default `24h`) is a sliding
|
||
window. Every authenticated request pushes it forward, so a session
|
||
in continuous use never hits it, while an abandoned one expires a day
|
||
after its last use. Any non-positive value (`0`, or a negative
|
||
duration such as `-1s`) disables idle expiry entirely; the absolute
|
||
cap below still applies. A set-but-unparseable value aborts startup
|
||
rather than silently falling back to the default.
|
||
- **Absolute expiry** is a fixed 7 days from login. Activity does
|
||
**not** extend it: after a week, every session ends and the user
|
||
authenticates again.
|
||
|
||
Only requests that authenticate with the session count as activity, so
|
||
an unauthenticated request carrying the cookie cannot keep a session
|
||
alive. The idle timestamp is rewritten at most once per tenth of the
|
||
idle window rather than on every request, which means a session may
|
||
expire up to 10% early relative to the user's true last request, but
|
||
never late.
|
||
|
||
Both clocks are anchored by timestamps stored in the session cookie.
|
||
Sessions issued before this feature existed carry neither, so they are
|
||
treated as expired: upgrading to a build that has it logs every
|
||
existing session out once, and those users sign in again.
|
||
|
||
#### Invalid values abort startup
|
||
|
||
The defaults above apply **only** to variables that are unset (or set
|
||
to an empty string). A variable that is set but cannot be parsed is a
|
||
fatal configuration error: webhooker logs the offending variable and
|
||
its value and refuses to start, rather than silently running with a
|
||
substituted default. `PORT=eighty`, `DEBUG=ture`, and
|
||
`RETENTION_SWEEP_INTERVAL=1 hour` all abort startup. `PORT` must
|
||
additionally be a number in the range 1–65535,
|
||
`RECEIVER_RATE_LIMIT` must be at least 1,
|
||
`RETENTION_SWEEP_INTERVAL` must be greater than zero (it is a ticker
|
||
period, so `0s` or a negative value would crash the reaper after
|
||
startup), and every entry in `TRUSTED_PROXIES` must be a CIDR block or
|
||
a bare IP address. `SESSION_IDLE_TIMEOUT` is the exception: a
|
||
non-positive value there means idle expiry is disabled, not invalid.
|
||
|
||
Boolean variables (`DEBUG`, `MAINTENANCE_MODE`) accept exactly the
|
||
spellings Go's `strconv.ParseBool` accepts — `1`, `t`, `T`, `TRUE`,
|
||
`true`, `True`, `0`, `f`, `F`, `FALSE`, `false`, `False` — and nothing
|
||
else. `yes`, `on`, and `off` are rejected rather than quietly treated
|
||
as false.
|
||
|
||
On first startup, webhooker automatically generates a cryptographically
|
||
secure session encryption key and stores it in the database. This key
|
||
persists across restarts — no manual key management is needed.
|
||
|
||
#### The admin account
|
||
|
||
On first startup — a `DATA_DIR` with no accounts in it — webhooker
|
||
creates an `admin` user with a randomly generated password and prints
|
||
it to standard output as a ruled banner:
|
||
|
||
```
|
||
========================================================================
|
||
WEBHOOKER FIRST BOOT: an admin account has been created.
|
||
|
||
username: admin
|
||
password: 3xamPl3-p4ssw0rd
|
||
|
||
Save this password now: it is shown only here, and only once.
|
||
If it is lost, run `webhooker resetpw admin` on a stopped deployment.
|
||
========================================================================
|
||
```
|
||
|
||
It is a banner rather than a log line because that is the only time it
|
||
is ever shown: as one `INFO` record it sat among the roughly 45 fx
|
||
`PROVIDE`/`RUN`/`HOOK` lines a boot writes, and under `docker run -d`
|
||
it is one line in a log subject to rotation. The database stores only
|
||
its Argon2id hash. There is no second account and no forgot-password
|
||
flow, so the banner and the reset command below are the only two ways
|
||
in.
|
||
|
||
#### Recovering a lost admin password
|
||
|
||
`webhooker resetpw` sets an existing account's password from the
|
||
command line:
|
||
|
||
```bash
|
||
# Generate a new password and print it.
|
||
DATA_DIR=/var/lib/webhooker webhooker resetpw -generate admin
|
||
|
||
# Or supply one on standard input (minimum 8 characters).
|
||
printf '%s' "$NEW_PASSWORD" | \
|
||
DATA_DIR=/var/lib/webhooker webhooker resetpw admin
|
||
```
|
||
|
||
In a container it is the same binary, which the image sets as `CMD`
|
||
rather than `ENTRYPOINT`, so the whole command has to be given:
|
||
|
||
```bash
|
||
docker run --rm -v webhooker-data:/var/lib/webhooker \
|
||
webhooker /app/webhooker resetpw -generate admin
|
||
```
|
||
|
||
Stop the service first — with the volume still attached to a running
|
||
container, the command refuses.
|
||
|
||
The password is never taken as a command-line argument: on Linux argv
|
||
is readable through `/proc` by every account on the host for as long as
|
||
the process lives. Standard input is echoed when it is a terminal — the
|
||
prompt says so — so `-generate` or a pipe is preferable on a shared
|
||
machine.
|
||
|
||
What it will not do:
|
||
|
||
- **Run against a live deployment.** It takes the same exclusive
|
||
`DATA_DIR` lock the server does (see
|
||
[Single-instance lock](#single-instance-lock)) and refuses while a
|
||
running instance holds it, naming the directory and exiting non-zero.
|
||
A running process keeps serving every session that authenticated with
|
||
the old password, so a reset underneath it would report a change the
|
||
service does not honour.
|
||
- **Create anything.** A `DATA_DIR` that does not exist, or that holds
|
||
no `webhooker.db`, is an error rather than a new empty deployment —
|
||
a mistyped path must not be built out and then reported as a success.
|
||
- **Create an account.** A username that does not exist is an error.
|
||
`resetpw` changes an existing account's password and nothing else.
|
||
|
||
`DATA_DIR` selects the deployment exactly as it does for the server. A
|
||
password that changes on disk takes effect at the next login; sessions
|
||
that are already authenticated are unaffected either way.
|
||
|
||
Changing a password you still know needs none of this — use
|
||
`POST /user/{username}/password` in the web UI.
|
||
|
||
#### What `DEBUG=true` exposes
|
||
|
||
`DEBUG=true` lowers the log level to `DEBUG`, which turns on every
|
||
statement GORM runs, the two by-design lookup misses on the
|
||
unauthenticated routes, and the rate limiter's own rejections. It is
|
||
meant to be safe to turn on while diagnosing a live service and safe to
|
||
paste the output of into a bug report.
|
||
|
||
What it does **not** put in the log:
|
||
|
||
- **Values bound to a SQL statement.** Statements are logged with their
|
||
placeholders, never with the values substituted into them, at every
|
||
level. That is what keeps the session encryption key out of the first
|
||
boot's `INSERT INTO settings` and the `admin` account's Argon2id
|
||
password hash out of its `INSERT INTO users` — the two statements
|
||
that made a debug log worth stealing. It applies to every table and
|
||
every statement rather than to a list of tables known to hold a
|
||
secret, so a table added later is covered without anyone remembering
|
||
to add it. The cost is that a failing statement can no longer be
|
||
replayed from the log alone: the statement, the table, the driver
|
||
error and the row count are all still there, but its values have to
|
||
come from the database.
|
||
`internal/gormlog/firstboot_test.go` boots the real graph with
|
||
`DEBUG=true` against an empty `DATA_DIR` and asserts that neither
|
||
secret appears in what that boot wrote to stdout.
|
||
The one exception is `(*gorm.DB).Scan`, which GORM logs through its
|
||
own trace recorder rather than through this filter. No production
|
||
code path calls it, and `internal/gormlog/scan_guard_test.go` fails
|
||
if a non-test file adds one.
|
||
- **Session cookies, API keys or target credentials.** None of these is
|
||
logged at any level.
|
||
|
||
What is in the log regardless of `DEBUG`, and is not a debug-logging
|
||
decision:
|
||
|
||
- **The initial `admin` password**, in the clear, once, on the first
|
||
boot that creates the account — as the banner described under
|
||
[The admin account](#the-admin-account), written straight to standard
|
||
output rather than through the logger. That banner is the only place
|
||
it is ever shown; the database stores the hash. A first boot's output
|
||
is not safe to paste anywhere until that account's password has been
|
||
changed. The same applies to `webhooker resetpw -generate`, which
|
||
prints the password it generated in the same form.
|
||
- **An authenticated operator's own configuration**, echoed back
|
||
untruncated — webhook names, target hostnames. See the logging
|
||
section under Security for the full list and for the per-line size
|
||
bound that covers unauthenticated traffic.
|
||
|
||
### Running with Docker
|
||
|
||
```bash
|
||
docker run -d \
|
||
-p 8080:8080 \
|
||
-v /path/to/data:/var/lib/webhooker \
|
||
-e WEBHOOKER_ENVIRONMENT=prod \
|
||
webhooker:latest
|
||
```
|
||
|
||
The container runs as a non-root user (`webhooker`, UID 1000), exposes
|
||
port 8080, and includes a health check against
|
||
`/.well-known/healthcheck`. The `/var/lib/webhooker` volume holds all
|
||
SQLite databases: the main application database (`webhooker.db`), the
|
||
per-webhook event databases (`events-{uuid}.db`), and any archive
|
||
databases written by `database` targets (`archive-{uuid}.db`). Mount
|
||
this as a persistent volume to preserve data across container
|
||
restarts.
|
||
|
||
## Backup, Restore, and Upgrades
|
||
|
||
### What to back up
|
||
|
||
All persistent state lives in `DATA_DIR` (`/var/lib/webhooker` by
|
||
default). Back up that directory in full — the state is spread across
|
||
several files whose names depend on your data, so copying the directory
|
||
is both the simplest and the only complete rule:
|
||
|
||
- `webhooker.db` — one per install. Settings (including the session
|
||
encryption key), users, API keys, webhooks, entrypoints, targets.
|
||
- `events-{webhook_uuid}.db` — **one per webhook**. Events, deliveries,
|
||
delivery results.
|
||
- `archive-{webhook_uuid}.db` — **one per webhook that has a `database`
|
||
target**. Archived events. Keyed on the webhook UUID, not the target
|
||
UUID: a webhook with several `database` targets still has exactly one
|
||
archive file.
|
||
|
||
`{webhook_uuid}` is the webhook's UUID primary key in its canonical
|
||
36-character hyphenated form, so a real filename looks like
|
||
`events-3f2a1c9e-....db`. The only other file is `webhooker.lock`, the
|
||
always-empty [single-instance lock](#single-instance-lock); it holds no
|
||
state and is not part of the backup set — a copied one is stale and
|
||
blocks nothing. No `-wal` or `-shm` files are produced (see below); a
|
||
transient `{name}.db-journal` may exist beside a database while a write
|
||
is in flight and is not part of the backup set either.
|
||
|
||
Configuration is **not** in `DATA_DIR` — it comes from the environment
|
||
and from a `.env` file read out of the process working directory. Back
|
||
that up with your deployment config, separately.
|
||
|
||
### A hot copy is not safe
|
||
|
||
No `journal_mode` pragma is ever issued on any database webhooker opens,
|
||
so all of them run on SQLite's default rollback journal. There is no
|
||
WAL. The main and event databases are also held open for the entire
|
||
process lifetime — `WebhookDBManager` caches event database handles and
|
||
closes them only on webhook deletion or shutdown — so "it looked idle"
|
||
is not a guarantee that nothing was mid-transaction.
|
||
|
||
That means `cp`, `rsync`, `tar` or a filesystem snapshot taken against a
|
||
running instance can capture a database mid-transaction and yield a file
|
||
that is corrupt or missing state the journal would have rolled back. Use
|
||
one of the two procedures below instead.
|
||
|
||
**Stop, copy, start.** The simplest, needs no extra tooling, and the
|
||
only one that gives a single point in time across every file:
|
||
|
||
```bash
|
||
docker stop webhooker
|
||
cp -a /path/to/data /path/to/backup-$(date -u +%Y%m%dT%H%M%SZ)
|
||
docker start webhooker
|
||
```
|
||
|
||
**SQLite online backup.** No downtime, one file at a time:
|
||
|
||
```bash
|
||
for db in /path/to/data/*.db; do
|
||
sqlite3 "$db" ".backup '/path/to/backup/$(basename "$db")'"
|
||
done
|
||
```
|
||
|
||
`.backup` takes the proper locks and writes a consistent file. Two
|
||
caveats. First, the runtime image is `alpine:3.21` with only
|
||
`ca-certificates` added — the `sqlite3` CLI is **not** in it, so run
|
||
this on the host against the volume path, or from a throwaway container
|
||
that mounts the volume. Second, each file is captured at its own
|
||
instant, so a webhook created or an event delivered between two files
|
||
being copied lands in one and not the other. If you need the whole set
|
||
coherent as of a single moment, stop the service.
|
||
|
||
Archive databases are the one exception the service is built for: the
|
||
archive writer closes its handle after each write (debounced to at most
|
||
one reopen per second), so an operator can move `archive-{uuid}.db`
|
||
away for offline retention while the service runs, and it is recreated
|
||
on the next write (see
|
||
[Database Architecture](#database-architecture)). That is a
|
||
move-the-file-away workflow, not a substitute for the backup procedures
|
||
above.
|
||
|
||
### Restore
|
||
|
||
1. Stop the service.
|
||
|
||
2. Restore the **whole set together**: `webhooker.db` *and* every
|
||
`events-*.db` *and* every `archive-*.db`. A partial restore fails
|
||
quietly rather than loudly. Every database is opened `mode=rwc`, so a
|
||
missing `events-{uuid}.db` is **created empty** on first access
|
||
instead of erroring — the webhook comes back with its configuration
|
||
intact and its entire event history silently gone. Event databases
|
||
restored without `webhooker.db` are simply orphaned; nothing
|
||
references their UUIDs.
|
||
|
||
3. Do not carry `*.db-journal` files into the restore. Backups taken by
|
||
either procedure above are self-consistent and do not need one.
|
||
|
||
4. **Fix ownership.** The container runs as the non-root `webhooker`
|
||
user, UID 1000 / GID 1000. Restored files must be owned by (or
|
||
writable by) that UID, and so must the directory itself — SQLite
|
||
creates the rollback journal beside the database, so a writable file
|
||
inside a directory it cannot write is not enough:
|
||
|
||
```bash
|
||
chown -R 1000:1000 /path/to/data
|
||
```
|
||
|
||
Restoring as `root` on the host and forgetting this step is the
|
||
usual way a restore fails.
|
||
|
||
5. Start the service. `AutoMigrate` runs against each restored database
|
||
as it is opened.
|
||
|
||
### Upgrades
|
||
|
||
**Back up before every upgrade.** Every start runs GORM `AutoMigrate`
|
||
unconditionally against whatever files it finds:
|
||
|
||
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
|
||
`Entrypoint`, `Target`
|
||
- each event database when it is lazily opened — `Event`, `Delivery`,
|
||
`DeliveryResult`
|
||
- each archive database on every open and reopen
|
||
|
||
There is no schema version table, no migration ledger, and no down
|
||
migrations. Nothing in the files records which version wrote them, and
|
||
no code path undoes a migration.
|
||
|
||
Upgrade procedure:
|
||
|
||
1. Stop the service.
|
||
2. Back up `DATA_DIR` using one of the procedures above.
|
||
3. Pull the new image and start it.
|
||
4. Confirm `database migrations completed` in the logs before putting
|
||
traffic back on it.
|
||
|
||
**Downgrade is unsupported.** Once a newer binary has migrated the files
|
||
there is no way to move them back. `AutoMigrate` is additive — it adds
|
||
tables, columns and indexes and never drops or rewrites them — so an
|
||
older binary will generally open migrated files and appear to work while
|
||
writing against a schema it does not know about. The failure mode is
|
||
silent divergence, not a startup error. The only supported way back to
|
||
an older version is restoring the pre-upgrade backup, which discards
|
||
everything received since that backup was taken.
|
||
|
||
### Backups contain secrets
|
||
|
||
Treat a backup with the same care as the credentials inside it. Encrypt
|
||
backups at rest and restrict who can read them.
|
||
|
||
- `events-{uuid}.db` and `archive-{uuid}.db` hold the **full payload
|
||
body and headers** of every event as received, including whatever the
|
||
sending service put in them — tokens, signatures, personal data.
|
||
- Event databases written before
|
||
[issue #206](https://git.eeqj.de/sneak/webhooker/issues/206) was fixed
|
||
**also contain target credentials**: a GORM association upsert on the
|
||
delivery and retry write path copied `targets` rows, `config`
|
||
included, into the per-webhook database. For a Slack target the
|
||
`webhookUrl` *is* the bearer credential, and an `http` target's URL
|
||
can embed userinfo. This version never writes those rows; the first
|
||
time it opens such a file it deletes them and vacuums the file, which
|
||
removes the credential bytes rather than only unlinking the rows.
|
||
Deleting alone would not: the bytes stay readable in the file's free
|
||
pages until it is rewritten. The sweep is recorded in the file's
|
||
`user_version` only once the vacuum returns, so a sweep that fails or
|
||
is interrupted fails the open and is retried on the next one, and a
|
||
file this version has opened without error holds no leaked rows and
|
||
no recoverable bytes from them. On upgrade this rewrites each
|
||
existing `events-{uuid}.db` once, on its first open. Two cases still
|
||
hand over live delivery destinations: a backup taken from an older
|
||
build, and a backup of a file this version has not yet opened
|
||
successfully. Copies already made stay affected — the sweep only
|
||
rewrites the file it opens, and freed blocks may persist in
|
||
filesystem snapshots and on the underlying storage. Rotate any target
|
||
credential that was in a backup you cannot account for.
|
||
- `webhooker.db` stores target config **unencrypted**, tracked at
|
||
[issue #212](https://git.eeqj.de/sneak/webhooker/issues/212), next to
|
||
the session encryption key and the Argon2id password hashes. It also
|
||
holds each entrypoint's inbound signature secret in the clear, for
|
||
the reason given under
|
||
[Inbound Signature Verification](#inbound-signature-verification):
|
||
HMAC verification needs the key itself, so it cannot be hashed.
|
||
|
||
## Inbound Signature Verification
|
||
|
||
A receiver URL is a bare v4 UUID in a path. That is unguessable, but it
|
||
is not a credential: anyone who learns it — from a browser history, a
|
||
proxy log, a screenshot, a copy-pasted support ticket — can post events
|
||
that webhooker stores and forwards, and the inbound headers are passed
|
||
on to your targets almost verbatim, so they also choose what the
|
||
downstream service sees. Verification is how an entrypoint stops
|
||
accepting anything that reaches its URL.
|
||
|
||
It is optional and configured per entrypoint. An entrypoint with no
|
||
scheme selected is not verified, which is what every entrypoint was
|
||
before this existed and what every entrypoint remains after an
|
||
upgrade — enabling verification is always a deliberate act, and no
|
||
existing deployment is locked out of its own receivers by installing a
|
||
new version.
|
||
|
||
When a scheme **is** selected, a request whose signature is missing,
|
||
malformed or wrong is answered `401` and **nothing is stored**: no
|
||
event row, no delivery row, no delivery attempt. Rejection happens
|
||
after the body is read (the signature covers it) and before the first
|
||
write.
|
||
|
||
### Supported schemes
|
||
|
||
| Scheme | Header | Check |
|
||
| -------- | --------------------- | ----- |
|
||
| `github` | `X-Hub-Signature-256` | HMAC-SHA256 of the raw request body under the shared secret, hex-encoded, prefixed `sha256=` |
|
||
| `gitlab` | `X-Gitlab-Token` | The header is the shared secret itself, compared as-is |
|
||
|
||
Both comparisons run in constant time (`hmac.Equal`). The HMAC is
|
||
computed over the request body exactly as received, before any parsing,
|
||
and under the same 1 MB body cap every other request obeys — an
|
||
unsigned sender cannot make webhooker buffer more than a signed one.
|
||
|
||
GitHub's older SHA-1 `X-Hub-Signature` is **not** accepted. Neither is
|
||
a GitHub digest sent without its `sha256=` prefix.
|
||
|
||
### Configuring a sender
|
||
|
||
The secret is a value you choose and enter in two places: at the sender
|
||
and in webhooker. webhooker never generates or displays one, so there
|
||
is no stored credential the UI can be made to reveal.
|
||
|
||
1. Generate a secret, e.g. `openssl rand -hex 32`.
|
||
2. In webhooker, open the webhook's page, find the entrypoint, and
|
||
click **Configure** (or **Rotate**, if it already has one). Select
|
||
the scheme and paste the secret. Surrounding whitespace is stripped,
|
||
so a value pasted with a trailing space still works; a secret whose
|
||
own first or last character is a space cannot be stored.
|
||
3. At the sender:
|
||
- **GitHub** — repository (or organization) → Settings → Webhooks →
|
||
the hook → **Secret**. GitHub then signs every delivery with
|
||
`X-Hub-Signature-256`.
|
||
- **GitLab** — project → Settings → Webhooks → the hook → **Secret
|
||
token**. GitLab sends it verbatim as `X-Gitlab-Token`.
|
||
|
||
**Rotation** is the same form: submit the new secret. Deliveries signed
|
||
with the old secret are rejected from that moment, so change it at the
|
||
sender in the same sitting. Selecting **None** removes verification and
|
||
deletes the stored secret with it.
|
||
|
||
The page shows which scheme an entrypoint uses and which header it
|
||
reads, never the secret. The value is stored in the clear — HMAC
|
||
verification needs the key itself, and a hash of it cannot recompute a
|
||
sender's digest — so it is handled like the other credentials
|
||
webhooker holds: excluded from JSON, kept out of templates by a
|
||
projection (`handlers.EntrypointView`), and absent from every log line,
|
||
including the ones written when verification fails.
|
||
|
||
### The credential is not stored or forwarded
|
||
|
||
Under the `gitlab` scheme the signature header **is** the secret. An
|
||
accepted request's headers are persisted on the event and forwarded to
|
||
every delivery target, so `X-Gitlab-Token` is removed from that copy
|
||
before the event is written — otherwise every target operator, every
|
||
backup and everyone with read access to `events-*.db` would hold the
|
||
value needed to forge signed requests to the entrypoint it protects.
|
||
The sender's other headers are untouched, and the request the receiver
|
||
itself verifies against is not modified.
|
||
|
||
The stripping is driven by the scheme's own description rather than by
|
||
a header name, and a scheme is stripped unless it declares that its
|
||
header carries a digest. `github` declares it: `X-Hub-Signature-256` is
|
||
an HMAC over the body, from which the key cannot be recovered, so it is
|
||
stored and forwarded intact. A scheme added later is stripped by
|
||
default.
|
||
|
||
### When configuration is broken
|
||
|
||
An entrypoint whose stored scheme this build does not recognise, or
|
||
which has one half of the scheme/secret pair and not the other, is
|
||
answered `500` and stores nothing. It is not treated as unverified. The
|
||
UI cannot create such a row — it rejects an unknown scheme with a `400`
|
||
— so this covers a hand-edited database or a downgrade to a build that
|
||
predates a scheme. Failing closed is the point: an entrypoint the
|
||
operator believes is protected must never quietly go back to accepting
|
||
anything.
|
||
|
||
## Entrypoints
|
||
|
||
This repository adheres to the
|
||
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
||
standard: normalized scripts in `script/` are the entrypoints for the
|
||
development workflow. Ten of the Makefile's sixteen targets are thin
|
||
shims that call them; `build`, `run`, `dev`, `deps`, `clean` and `css`
|
||
are inline commands with no script behind them. We provide:
|
||
|
||
- `script/bootstrap` — install all dependencies (idempotent)
|
||
- `script/setup` — make a fresh clone ready for development
|
||
(bootstrap, then install-precommit)
|
||
- `script/projectname` — output the project name ("webhooker")
|
||
- `script/fetch-assets` — download the third-party browser assets into
|
||
`static/`, verifying each against its pinned sha256
|
||
- `script/test` — run the test suite
|
||
- `script/lint` — run golangci-lint in Docker (see Linting below)
|
||
- `script/fmt` — format all code (writes)
|
||
- `script/fmt-check` — check formatting (read-only)
|
||
- `script/check` — run test, lint, and fmt-check
|
||
- `script/docker` — build the Docker image tagged via `script/projectname`
|
||
- `script/cibuild` — CI entrypoint: `docker build .` (the Dockerfile
|
||
runs the checks, so a green build implies a green repo)
|
||
- `script/ci-mark-superseded` — CI helper: mark the commits whose run a
|
||
newer push cancelled (see [CI gate honesty](#ci-gate-honesty))
|
||
- `script/precommit` — pre-commit checks (`go mod tidy` guard, then
|
||
`script/check`)
|
||
- `script/install-precommit` — install the git pre-commit hook that
|
||
runs `script/precommit`
|
||
|
||
## Third-party browser assets
|
||
|
||
The web UI serves one third-party script, Alpine.js. It is **not** committed:
|
||
a minified bundle in the tree is unreviewable, and `REPO_POLICIES.md` bars
|
||
both committed build artifacts and unpinned external references.
|
||
|
||
Instead `script/fetch-assets` downloads it from a pinned URL, checks the
|
||
download against a hardcoded sha256, and installs it under `static/`. The
|
||
sha256 of every installed asset is recorded in `static/vendor.sha256`, and
|
||
`static/vendor_test.go` re-hashes the bytes `go:embed` put in the binary
|
||
against that manifest — so the pin is enforced on what actually ships, not
|
||
merely written down. Any mismatch fails the build.
|
||
|
||
`make bootstrap` runs the fetch for local development, and the Dockerfile
|
||
runs it in the build stage; `.gitignore` and `.dockerignore` keep the
|
||
artifact out of both the repo and the build context.
|
||
|
||
To move to a new version: update the version, URL, and tarball sha256 in
|
||
`script/fetch-assets` and the asset sha256 in `static/vendor.sha256`, then
|
||
run `make assets && make check`.
|
||
|
||
## Rationale
|
||
|
||
Webhook integrations between services are inherently fragile. The
|
||
receiving service must be online when the webhook fires, most webhook
|
||
senders provide no built-in retry mechanism, and there is no standard
|
||
way to inspect what was sent, when it was sent, or whether delivery
|
||
succeeded.
|
||
|
||
webhooker solves this by acting as a durable intermediary:
|
||
|
||
1. **Reliable ingestion** — webhooker is always ready to accept incoming
|
||
webhooks. It stores every received event before attempting any
|
||
delivery, so nothing is lost if downstream targets are unavailable.
|
||
|
||
2. **Guaranteed delivery** — Events are queued for delivery to each
|
||
configured target. Failed deliveries are retried with configurable
|
||
backoff. Every delivery attempt is logged with status codes, response
|
||
bodies, and timing.
|
||
|
||
3. **Observability** — Full request/response logging for every webhook
|
||
received and every delivery attempted. Prometheus metrics expose
|
||
volume, latency, and error rates. The web UI provides real-time
|
||
visibility into event flow.
|
||
|
||
4. **Fan-out** — A single incoming webhook can be delivered to multiple
|
||
targets simultaneously. This enables patterns like forwarding a
|
||
GitHub webhook to both a deployment service and a Slack channel.
|
||
|
||
5. **Replay** (not yet implemented) — Every received event is stored in
|
||
full, which is what manual redelivery for debugging or testing will
|
||
be built on. No redelivery exists today, in the web UI or the API;
|
||
see [TODO.md](TODO.md).
|
||
|
||
### Use Cases
|
||
|
||
- **Store-and-forward** with configurable retries for unreliable
|
||
receivers
|
||
- **Observability** via Prometheus metrics on webhook frequency, payload
|
||
size, and delivery performance
|
||
- **Debugging** and introspection of webhook payloads in the web UI
|
||
- **Replay** of webhook events for application testing and development
|
||
(planned; not yet implemented)
|
||
- **Fan-out** delivery of a single webhook to multiple downstream
|
||
targets
|
||
- **High-availability ingestion** for delivery to less reliable backend
|
||
systems
|
||
|
||
## Design
|
||
|
||
### Architecture Overview
|
||
|
||
webhooker is structured as a standard Go HTTP server following the
|
||
[sneak/prompts GO_HTTP_SERVER_CONVENTIONS](https://git.eeqj.de/sneak/prompts/src/branch/main/prompts/GO_HTTP_SERVER_CONVENTIONS.md).
|
||
It uses:
|
||
|
||
- **[Uber fx](https://go.uber.org/fx)** for dependency injection and
|
||
lifecycle management
|
||
- **[go-chi](https://github.com/go-chi/chi)** for HTTP routing
|
||
- **[GORM](https://gorm.io)** for database access with
|
||
**[modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite)** as
|
||
the runtime SQLite driver. Note: `gorm.io/driver/sqlite` transitively
|
||
depends on `mattn/go-sqlite3`, which requires CGO at build time (see
|
||
[Docker](#docker) section)
|
||
- **[slog](https://pkg.go.dev/log/slog)** (stdlib) for structured
|
||
logging with TTY detection (text for dev, JSON for prod)
|
||
- **[gorilla/sessions](https://github.com/gorilla/sessions)** for
|
||
encrypted cookie-based session management
|
||
- **[gorilla/csrf](https://github.com/gorilla/csrf)** for CSRF
|
||
protection (cookie-based double-submit tokens)
|
||
- **[go-chi/httprate](https://github.com/go-chi/httprate)** for
|
||
sliding-window rate limiting of the password-change and webhook
|
||
receiver endpoints. The bucket is per client IP only when
|
||
`TRUSTED_PROXIES` names the reverse proxy; unset, every client
|
||
behind that proxy shares one bucket per limit. The login endpoint
|
||
counts failed attempts itself instead, so that a correct password is
|
||
never throttled (see [Rate Limiting](#rate-limiting))
|
||
- **[Prometheus](https://prometheus.io)** for metrics, served at
|
||
`/metrics` behind basic auth
|
||
- **[Sentry](https://sentry.io)** for optional error reporting
|
||
|
||
### Naming Conventions
|
||
|
||
The codebase uses consistent naming throughout (rename completed in
|
||
[issue #12](https://git.eeqj.de/sneak/webhooker/issues/12)):
|
||
|
||
| Entity | Description |
|
||
| ---------------- | ----------- |
|
||
| **Webhook** | Top-level configuration entity grouping entrypoints and targets |
|
||
| **Entrypoint** | A receiver URL where external services POST events |
|
||
| **Target** | A delivery destination for events |
|
||
|
||
### Data Model
|
||
|
||
webhooker's data model has nine entities organized into two tiers: the
|
||
**application tier** (user and webhook configuration) and the **event
|
||
tier** (event ingestion, delivery, and logging).
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ APPLICATION TIER │
|
||
│ (main application database) │
|
||
│ │
|
||
│ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │
|
||
│ │ User │──1:N──│ Webhook │──1:N──│ Entrypoint │ │
|
||
│ │ │ │ │ │ │ │
|
||
│ │ │ │ │──1:N──│ Target │ │
|
||
│ │ │ └──────────┘ └──────────────┘ │
|
||
│ │ │──1:N──│ APIKey │ │
|
||
│ └──────────┘ └──────────┘ │
|
||
│ │
|
||
│ ┌──────────┐ │
|
||
│ │ Setting │ (key-value application config) │
|
||
│ └──────────┘ │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ EVENT TIER │
|
||
│ (per-webhook dedicated databases) │
|
||
│ │
|
||
│ ┌──────────┐ ┌──────────┐ ┌─────────────────┐ │
|
||
│ │ Event │──1:N──│ Delivery │──1:N──│ DeliveryResult │ │
|
||
│ └──────────┘ └──────────┘ └─────────────────┘ │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
#### Setting
|
||
|
||
A key-value pair for application-level configuration that is
|
||
auto-managed rather than user-provided. Used to store the session
|
||
encryption key and any future auto-generated settings.
|
||
|
||
| Field | Type | Description |
|
||
| ------- | ------ | ----------- |
|
||
| `key` | string | Primary key (setting name) |
|
||
| `value` | text | Setting value |
|
||
|
||
Currently stored settings:
|
||
|
||
- **`session_key`** — Base64-encoded 32-byte session encryption key,
|
||
auto-generated on first startup.
|
||
|
||
#### User
|
||
|
||
A registered user of the webhooker service.
|
||
|
||
| Field | Type | Description |
|
||
| ---------- | -------- | ----------- |
|
||
| `id` | UUID | Primary key |
|
||
| `username` | string | Unique login name |
|
||
| `password` | string | Argon2id hash (never exposed via API) |
|
||
|
||
**Relations:** Has many Webhooks. Has many APIKeys.
|
||
|
||
Passwords are hashed with Argon2id using secure defaults (64 MB memory,
|
||
1 iteration, 4 threads, 32-byte key, 16-byte salt). On first startup,
|
||
an `admin` user is created with a randomly generated 16-character
|
||
password printed once to stdout; `webhooker resetpw` sets it again if
|
||
it is lost (see [The admin account](#the-admin-account)). Every one of
|
||
those paths hashes through the same `internal/database` code, so the
|
||
parameters cannot drift between them.
|
||
|
||
#### Webhook
|
||
|
||
The top-level configuration entity. A webhook groups together one or
|
||
more entrypoints (receiver URLs) and one or more targets (delivery
|
||
destinations) into a logical unit. A user creates a webhook to set up
|
||
event routing.
|
||
|
||
| Field | Type | Description |
|
||
| ---------------- | ------- | ----------- |
|
||
| `id` | UUID | Primary key |
|
||
| `user_id` | UUID | Foreign key → User |
|
||
| `name` | string | Human-readable name |
|
||
| `description` | string | Optional description |
|
||
| `retention_days` | integer | Days to retain events (default: 30; 0 means retain forever) |
|
||
|
||
**Relations:** Belongs to User. Has many Entrypoints. Has many Targets.
|
||
|
||
The `retention_days` field controls how long event data is kept in the
|
||
webhook's dedicated database before automatic cleanup.
|
||
|
||
Setting `retention_days` to `0` means "retain events forever". Because
|
||
the column carries a default of 30, a literal zero cannot survive an
|
||
insert, so a zero is rewritten on save to a sentinel of `365 * 1000`
|
||
days (`database.RetentionForeverDays`). The retention reaper recognises
|
||
that sentinel and skips the webhook entirely, and the web UI displays
|
||
such a webhook's retention as "forever" rather than as a day count.
|
||
|
||
Submitted `retention_days` values therefore fall into three bands, not
|
||
two:
|
||
|
||
- `1` up to `database.MaxFiniteRetentionDays` (106751 days, about 292
|
||
years) is accepted as a finite retention.
|
||
- Above that ceiling but below the retain-forever sentinel of 365000
|
||
(`database.RetentionForeverDays`) is rejected with a 400. This is the
|
||
band the cap exists for.
|
||
- `0`, and `365000` or above, are accepted and mean retain forever,
|
||
collapsing to the sentinel — `0` in `Webhook.BeforeSave`, the large
|
||
values in `parseRetentionDays`. The large values are not out of
|
||
range: the edit form pre-fills the sentinel for a retain-forever
|
||
webhook, so submitting that form back unchanged has to keep meaning
|
||
"forever".
|
||
|
||
A negative value is in none of the three: `parseRetentionDays` rejects
|
||
it with a 400 before `BeforeSave` ever sees it.
|
||
|
||
The cap is not arbitrary: the reaper computes its cutoff as a
|
||
`time.Duration`, an int64 nanosecond count, and a longer period
|
||
overflows it. An overflowed cutoff lands in the future, where it
|
||
matches every row, so the sweep would delete every event the webhook
|
||
has instead of none. The reaper also clamps the value it is given, so a
|
||
row written by an older version cannot trigger that either.
|
||
|
||
#### Entrypoint
|
||
|
||
A receiver URL where external services POST webhook events. Each
|
||
entrypoint has a unique UUID-based path.
|
||
When an HTTP request arrives at an entrypoint's path, webhooker captures
|
||
the full request and creates an Event.
|
||
|
||
| Field | Type | Description |
|
||
| -------------- | ------- | ----------- |
|
||
| `id` | UUID | Primary key |
|
||
| `webhook_id` | UUID | Foreign key → Webhook |
|
||
| `path` | string | Unique bare UUID, generated at creation. The `/webhook/` prefix is route only and is not stored: the receiver matches this column against the raw `{uuid}` path segment |
|
||
| `description` | string | Optional description |
|
||
| `active` | boolean | Whether this entrypoint accepts events (default: true) |
|
||
| `signature_scheme` | string | How inbound requests are authenticated: `github`, `gitlab`, or empty for no verification (default: empty). See [Inbound Signature Verification](#inbound-signature-verification) |
|
||
| `signature_secret` | string | The secret shared with the sender, stored in the clear because HMAC verification needs the key itself. Never marshalled to JSON, never rendered, never logged. Empty when no scheme is set |
|
||
|
||
**Relations:** Belongs to Webhook.
|
||
|
||
Both signature columns arrive through `AutoMigrate` with an empty
|
||
default, so every entrypoint written before they existed migrates to
|
||
"not configured" and keeps accepting the traffic it already accepted.
|
||
|
||
A webhook can have multiple entrypoints. This allows separate URLs for
|
||
different event sources that all feed into the same processing pipeline
|
||
(e.g., one entrypoint for GitHub, another for Stripe, both routing to
|
||
the same targets).
|
||
|
||
#### Target
|
||
|
||
A delivery destination for events. Each target defines where and how
|
||
events should be forwarded.
|
||
|
||
| Field | Type | Description |
|
||
| ---------------- | ---------- | ----------- |
|
||
| `id` | UUID | Primary key |
|
||
| `webhook_id` | UUID | Foreign key → Webhook |
|
||
| `name` | string | Human-readable name |
|
||
| `type` | TargetType | One of: `http`, `slack`, `database`, `log` |
|
||
| `active` | boolean | Whether deliveries are enabled (default: true) |
|
||
| `config` | JSON text | Type-specific configuration |
|
||
| `max_retries` | integer | Maximum retry attempts for `http` and `slack` targets (0 = fire-and-forget, >0 = retries with backoff and a circuit breaker). Ignored by `database` and `log` targets |
|
||
| `max_queue_size` | integer | Stored and shown on the target's detail view, but not enforced anywhere yet: nothing in the delivery engine consults it. Queue depth is set by the two fixed 10,000-entry channels |
|
||
|
||
**Relations:** Belongs to Webhook. Has many Deliveries.
|
||
|
||
**Target types:**
|
||
|
||
- **`http`** — Forward the event as an HTTP POST to a configured URL.
|
||
Behavior depends on `max_retries`: when `max_retries` is 0 (the
|
||
default), the target operates in fire-and-forget mode — a single
|
||
attempt with no retries and no circuit breaker. When `max_retries` is
|
||
greater than 0, failed deliveries are retried with exponential backoff
|
||
up to `max_retries` attempts, protected by a per-target circuit
|
||
breaker.
|
||
- **`slack`** — Post the event as a formatted message to a
|
||
Slack-compatible incoming webhook URL (`webhookUrl` in `config`). It
|
||
is built on the same HTTP core as `http` and honours `max_retries`
|
||
identically, circuit breaker included. See the Slack target section
|
||
under "Per-Webhook Event Databases" for the message format.
|
||
- **`database`** — Archive the full event as a row into a separate
|
||
per-webhook archive database (`archive-{webhookID}.db`) for long-term
|
||
retention, with an optional creation-validated expiry (default: keep
|
||
forever). No external delivery and no retries; an archive write
|
||
failure fails the delivery. See the database target section under
|
||
"Per-Webhook Event Databases" for the full semantics.
|
||
- **`log`** — Write the event to the application log (stdout). Useful
|
||
for debugging.
|
||
|
||
The `config` field stores type-specific configuration as JSON (e.g.,
|
||
destination URL, custom headers, timeout settings).
|
||
|
||
#### APIKey
|
||
|
||
A programmatic access credential for API authentication.
|
||
|
||
| Field | Type | Description |
|
||
| -------------- | --------- | ----------- |
|
||
| `id` | UUID | Primary key |
|
||
| `user_id` | UUID | Foreign key → User |
|
||
| `key` | string | Unique API key value |
|
||
| `description` | string | Optional description |
|
||
| `last_used_at` | timestamp | Last time this key was used (nullable) |
|
||
|
||
**Relations:** Belongs to User.
|
||
|
||
#### Event
|
||
|
||
A captured incoming webhook request. Stores the complete HTTP request
|
||
data for auditing and for the planned replay capability.
|
||
|
||
| Field | Type | Description |
|
||
| -------------- | ------ | ----------- |
|
||
| `id` | UUID | Primary key |
|
||
| `webhook_id` | UUID | Foreign key → Webhook |
|
||
| `entrypoint_id` | UUID | Foreign key → Entrypoint |
|
||
| `method` | string | HTTP method of the captured request. Always `POST`: the receiver answers every other method with 405 before an Event is created |
|
||
| `headers` | JSON | Complete request headers |
|
||
| `body` | text | Raw request body |
|
||
| `content_type` | string | Content-Type header value |
|
||
|
||
**Relations:** Belongs to Webhook. Belongs to Entrypoint. Has many
|
||
Deliveries.
|
||
|
||
When a request arrives at an entrypoint, the full request (method,
|
||
headers, body) is captured as an Event. The event is then queued for
|
||
delivery to every active target configured on the parent webhook.
|
||
|
||
#### Delivery
|
||
|
||
The pairing of an event with a target. Tracks the overall delivery
|
||
status across potentially multiple attempts.
|
||
|
||
| Field | Type | Description |
|
||
| ---------- | -------------- | ----------- |
|
||
| `id` | UUID | Primary key |
|
||
| `event_id` | UUID | Foreign key → Event |
|
||
| `target_id`| UUID | Foreign key → Target |
|
||
| `status` | DeliveryStatus | One of: `pending`, `delivered`, `failed`, `retrying` |
|
||
|
||
**Relations:** Belongs to Event. Belongs to Target. Has many
|
||
DeliveryResults.
|
||
|
||
**Delivery statuses:**
|
||
|
||
- **`pending`** — Created but not yet attempted.
|
||
- **`retrying`** — At least one attempt failed; more attempts remain.
|
||
- **`delivered`** — Successfully delivered (at least one attempt
|
||
succeeded).
|
||
- **`failed`** — All retry attempts exhausted without success.
|
||
|
||
#### DeliveryResult
|
||
|
||
The result of a single delivery attempt. Every attempt (including
|
||
retries) is individually logged for full observability.
|
||
|
||
| Field | Type | Description |
|
||
| --------------- | ------- | ----------- |
|
||
| `id` | UUID | Primary key |
|
||
| `delivery_id` | UUID | Foreign key → Delivery |
|
||
| `attempt_num` | integer | Attempt number (1-based) |
|
||
| `success` | boolean | Whether this attempt succeeded |
|
||
| `status_code` | integer | HTTP response status code (if applicable) |
|
||
| `response_body` | text | Response body (if applicable) |
|
||
| `error` | string | Error message (on failure) |
|
||
| `duration` | integer | Request duration in milliseconds |
|
||
|
||
**Relations:** Belongs to Delivery.
|
||
|
||
#### Common Fields
|
||
|
||
Every entity except `Setting` includes these fields from `BaseModel`.
|
||
`Setting` is a bare key-value row with no `id`, no timestamps and no
|
||
soft delete:
|
||
|
||
| Field | Type | Description |
|
||
| ------------ | --------- | ----------- |
|
||
| `id` | UUID | Auto-generated UUIDv4 primary key |
|
||
| `created_at` | timestamp | Record creation time |
|
||
| `updated_at` | timestamp | Last modification time |
|
||
| `deleted_at` | timestamp | Soft-delete timestamp (nullable; GORM soft deletes) |
|
||
|
||
### Database Architecture
|
||
|
||
#### Per-Webhook Event Databases
|
||
|
||
webhooker uses **separate SQLite database files**: a main application
|
||
database for configuration data and per-webhook databases for event
|
||
storage. All database files live in the `DATA_DIR` directory.
|
||
|
||
**Main Application Database** (`{DATA_DIR}/webhooker.db`) — stores
|
||
configuration and application state:
|
||
|
||
- **Settings** — auto-managed key-value config (e.g. session encryption
|
||
key)
|
||
- **Users** — accounts and Argon2id password hashes
|
||
- **Webhooks** — webhook configurations
|
||
- **Entrypoints** — receiver URL definitions
|
||
- **Targets** — delivery destination configurations
|
||
- **APIKeys** — programmatic access credentials
|
||
|
||
On first startup the main database is auto-migrated, a session
|
||
encryption key is generated and stored, and an `admin` user is created.
|
||
|
||
**Per-Webhook Event Databases** (`{DATA_DIR}/events-{webhook_uuid}.db`)
|
||
— each webhook gets its own dedicated SQLite file containing:
|
||
|
||
- **Events** — captured incoming webhook payloads
|
||
- **Deliveries** — event-to-target pairings and their status
|
||
- **DeliveryResults** — individual delivery attempt logs
|
||
|
||
Per-webhook databases are created automatically when a webhook is
|
||
created (and lazily on first access for webhooks that predate this
|
||
feature). They are managed by the `WebhookDBManager` component, which
|
||
handles connection pooling, lazy opening, migrations, and cleanup.
|
||
|
||
This separation provides:
|
||
|
||
- **Isolation** — a high-volume webhook won't cause lock contention or
|
||
journal growth affecting the main application or other webhooks.
|
||
- **Independent lifecycle** — event databases can be independently
|
||
backed up, archived, rotated, or size-limited without impacting the
|
||
application.
|
||
- **Clean deletion** — removing a webhook and all its history is as
|
||
simple as deleting one file. Configuration is soft-deleted in the main
|
||
DB; the event database file is hard-deleted (permanently removed).
|
||
- **Per-webhook retention** — the `retention_days` field on each webhook
|
||
controls automatic cleanup of old events in that webhook's database
|
||
only, or disables cleanup entirely when set to `0` (retain forever).
|
||
- **Performance** — each webhook's database has its own page cache and
|
||
its own lock, so concurrent event ingestion across webhooks won't
|
||
contend. No write-ahead log is involved: both DSNs are
|
||
`file:{path}?cache=shared&mode=rwc` and no `journal_mode` pragma is
|
||
ever issued, so every database runs on SQLite's default rollback
|
||
journal.
|
||
|
||
The **database target type** builds on this architecture to provide
|
||
long-term archiving, separate from the per-webhook event database (which
|
||
may prune events under its own retention). Delivering to a database
|
||
target writes the full event — body, headers, method, content type, and
|
||
webhook/entrypoint/event identifiers — as a row into a dedicated archive
|
||
database, `archive-{webhookID}.db`, stored under the data directory
|
||
beside the event database. After each write the archive handle is closed
|
||
and reopened, debounced to at most once per second, so an operator can
|
||
move the archive file away for offline archiving without stopping the
|
||
service; a moved or removed archive file is recreated automatically on
|
||
the next write. An optional `expiry` in the target's config JSON (e.g.
|
||
`{"expiry":"720h"}`) is validated when the target is created — the
|
||
default (unset or the literal `never`) keeps rows forever — and rows
|
||
older than the expiry are pruned each time the archive is (re)opened. An
|
||
archive write failure is never silent success: the delivery records a
|
||
failed attempt with the error and is marked failed.
|
||
|
||
Because reopens only happen on writes, an archive belonging to a webhook
|
||
that has stopped receiving events would never be pruned. A background
|
||
**archive sweeper** closes that gap: on the same interval as the event
|
||
retention reaper (`RETENTION_SWEEP_INTERVAL`) it prunes every archive
|
||
whose database target declares a positive expiry, whether or not the
|
||
webhook is still receiving traffic. The sweep never creates an archive —
|
||
a webhook whose archive file does not yet exist is skipped, not
|
||
initialised — it takes the same per-webhook lock the write path uses, so
|
||
it can never interleave with a write, and it leaves the archive closed
|
||
afterwards so the move-the-file-away workflow keeps working. Archives
|
||
with no expiry, or the expiry `never`, are not touched by the sweep at
|
||
all.
|
||
|
||
Note that a webhook has one archive file but may carry more than one
|
||
`database` target, each with its own `expiry`. The shortest expiry
|
||
configured on any of them therefore governs the whole archive, and the
|
||
sweep applies it whether or not the webhook is still receiving events.
|
||
Configure a single `database` target per webhook unless you intend that.
|
||
|
||
Deleting a webhook releases its archive: the delivery engine's cached
|
||
archive writer is dropped and its file handle closed, so nothing lingers
|
||
after the webhook is gone. The archive **file itself is deliberately
|
||
left on disk**. Unlike the event database — per-webhook working storage
|
||
that is hard-deleted with the webhook — an archive is long-term storage
|
||
an operator may still want to keep or move away for offline retention,
|
||
and destroying it as a side effect of deleting a webhook would be
|
||
unrecoverable. Removing `archive-{webhookID}.db` is the operator's call.
|
||
Deleting a webhook's last `database` target releases the writer the same
|
||
way, and for the same reason leaves the file alone.
|
||
|
||
The **Slack target type** sends webhook events as formatted messages to
|
||
any Slack-compatible incoming webhook URL (works with Slack, Mattermost,
|
||
and other compatible services). Each message includes event metadata
|
||
(HTTP method, content type, timestamp, body size) and the payload
|
||
pretty-printed in a code block. JSON payloads are automatically
|
||
formatted with indentation for readability; non-JSON payloads are shown
|
||
as raw text. Large payloads are truncated to keep messages reasonable.
|
||
Config stores `webhookUrl` — the Slack/Mattermost incoming webhook
|
||
endpoint. That is the JSON key; the error text for a missing one reads
|
||
`webhook_url is required`, which is the message, not the key.
|
||
|
||
The database uses the
|
||
[modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite) driver at
|
||
runtime, though CGO is required at build time due to the transitive
|
||
`mattn/go-sqlite3` dependency from `gorm.io/driver/sqlite`.
|
||
|
||
### Request Flow
|
||
|
||
```
|
||
External Service
|
||
│
|
||
│ POST /webhook/{uuid}
|
||
▼
|
||
┌─────────────┐ ┌──────────────┐ ┌──────────────┐
|
||
│ chi Router │────►│ Middleware │────►│ Webhook │
|
||
│ │ │ Stack │ │ Handler │
|
||
└─────────────┘ └──────────────┘ └──────┬───────┘
|
||
│
|
||
1. Look up Entrypoint by UUID
|
||
2. Read the body under the 1 MB cap
|
||
3. Verify the signature, if the entrypoint has
|
||
one configured — 401 and no writes if it
|
||
fails (see Inbound Signature Verification)
|
||
4. Capture full request as Event
|
||
5. Create Delivery records for each active Target
|
||
6. Build self-contained delivery.Task structs
|
||
(target config + event data inline for
|
||
bodies < 16 KiB)
|
||
7. Notify Engine via channel (no DB read needed)
|
||
│
|
||
▼
|
||
┌──────────────┐
|
||
│ Delivery │◄── retry timers
|
||
│ Engine │ (backoff)
|
||
│ (worker │
|
||
│ pool) │
|
||
└──────┬───────┘
|
||
│
|
||
┌── bounded worker pool (N workers) ──┐
|
||
▼ ▼ ▼
|
||
┌────────────┐ ┌────────────┐ ┌────────────┐
|
||
│ HTTP Target│ │ HTTP Target│ │ Log Target │
|
||
│(max_retries│ │(max_retries│ │ (stdout) │
|
||
│ == 0) │ │ > 0, │ └────────────┘
|
||
│ fire+forget│ │ backoff + │
|
||
└────────────┘ │ circuit │
|
||
│ breaker) │
|
||
└────────────┘
|
||
```
|
||
|
||
### Bounded Worker Pool
|
||
|
||
The delivery engine uses a **fixed-size worker pool** (default: 10
|
||
workers) to process all deliveries. At most N deliveries are in-flight
|
||
at any time, preventing goroutine explosions regardless of queue depth.
|
||
|
||
**Architecture:**
|
||
|
||
- **Channels as queues:** Two buffered channels serve as bounded queues:
|
||
a delivery channel (new tasks from the webhook handler) and a retry
|
||
channel (tasks from backoff timers). Both are buffered to 10,000.
|
||
- **Fan-out via channel, not goroutines:** When an event arrives with
|
||
multiple targets, each `delivery.Task` is sent to the delivery channel.
|
||
Workers pick them up and process them — no goroutine-per-target.
|
||
- **Worker goroutines:** A fixed number of worker goroutines select from
|
||
both channels. Each worker processes one task at a time, then picks up
|
||
the next. Workers are the ONLY goroutines doing actual HTTP delivery.
|
||
- **Retry backpressure with DB fallback:** When a retry timer fires and
|
||
the retry channel is full, the timer is dropped — the delivery stays
|
||
in `retrying` status in the database. A periodic sweep (every 60s)
|
||
scans for these "orphaned" retries and re-queues them. No blocked
|
||
goroutines, no unbounded timer chains.
|
||
- **Bounded concurrency:** At most N deliveries (N = number of workers)
|
||
are in-flight simultaneously. Even if a circuit breaker is open for
|
||
hours and thousands of retries queue up in the channels, the workers
|
||
drain them at a controlled rate when the circuit closes.
|
||
|
||
This means:
|
||
|
||
- **No goroutine explosion** — even with 10,000 queued retries, only
|
||
N worker goroutines exist.
|
||
- **Natural backpressure** — if workers are busy, new tasks wait in the
|
||
channel buffer rather than spawning more goroutines.
|
||
- **Independent results** — each worker records its own delivery result
|
||
in the per-webhook database without coordination.
|
||
- **Graceful shutdown** — cancel the context, workers finish their
|
||
current task and exit. The stop hook waits for the pool via
|
||
`lifecycle.WaitForShutdown`, which bounds that wait by fx's stop
|
||
timeout rather than blocking forever on a wedged worker. On timeout
|
||
it logs at `ERROR` and returns an error, and the goroutines that
|
||
did not finish are still running — an unclean shutdown is reported
|
||
rather than hidden.
|
||
|
||
**Recovery paths:**
|
||
|
||
1. **Startup recovery:** When the engine starts, it scans all per-webhook
|
||
databases for `pending` and `retrying` deliveries. Pending deliveries
|
||
are sent to the delivery channel; retrying deliveries get backoff
|
||
timers scheduled.
|
||
2. **Periodic retry sweep (DB-mediated fallback):** Every 60 seconds the
|
||
engine scans for `retrying` deliveries whose backoff period has
|
||
elapsed. This catches "orphaned" retries — ones whose in-memory timer
|
||
was dropped because the retry channel was full. The database is the
|
||
durable fallback that ensures no retry is permanently lost, even under
|
||
extreme backpressure.
|
||
|
||
**Changing a target's type does not migrate in-flight deliveries.** Only
|
||
`http` and `slack` targets own durable retries; `database` and `log`
|
||
targets are fire-and-forget and never produce a `retrying` delivery. If a
|
||
target's `type` is edited from a retrying type to a non-retrying (or
|
||
unknown) one while one of its deliveries is still `retrying`, both
|
||
recovery paths above terminally mark that delivery `failed` and record a
|
||
`DeliveryResult` naming the current target type as the reason, logging it
|
||
at warn level. The delivery is not re-dispatched under the new type — the
|
||
operator never asked for that delivery — and while the event itself
|
||
remains stored in the per-webhook event database, there is no way to
|
||
redeliver it: manual redelivery is planned, not implemented (see
|
||
[TODO.md](TODO.md)).
|
||
|
||
### Circuit Breaker (HTTP and Slack Targets with Retries)
|
||
|
||
`http` and `slack` targets with `max_retries` > 0 are protected by a
|
||
**per-target circuit breaker** that prevents hammering a down target
|
||
with repeated failed delivery attempts. The circuit breaker is
|
||
in-memory only and resets on restart (which is fine — startup recovery
|
||
rescans the database anyway).
|
||
|
||
**States:**
|
||
|
||
| State | Behavior |
|
||
| ----------- | -------- |
|
||
| **Closed** | Normal operation. Deliveries flow through. Consecutive failures are counted. |
|
||
| **Open** | Target appears down. Deliveries are skipped and rescheduled for after the cooldown. |
|
||
| **Half-Open** | Cooldown expired. One probe delivery is allowed to test if the target has recovered. |
|
||
|
||
**Transitions:**
|
||
|
||
```
|
||
success ┌──────────┐
|
||
┌────────────────────► │ Closed │ ◄─── probe succeeds
|
||
│ │ (normal) │
|
||
│ └────┬─────┘
|
||
│ │ N consecutive failures
|
||
│ ▼
|
||
│ ┌──────────┐
|
||
│ │ Open │ ◄─── probe fails
|
||
│ │(tripped) │
|
||
│ └────┬─────┘
|
||
│ │ cooldown expires
|
||
│ ▼
|
||
│ ┌──────────┐
|
||
└──────────────────────│Half-Open │
|
||
│ (probe) │
|
||
└──────────┘
|
||
```
|
||
|
||
**Defaults:**
|
||
|
||
- **Failure threshold:** 5 consecutive failures before opening
|
||
- **Cooldown:** 30 seconds in open state before probing
|
||
|
||
**Scope:** Circuit breakers apply to **`http` and `slack` targets with
|
||
`max_retries` > 0**. The Slack target is built on the same HTTP core
|
||
and hands its own `max_retries` to the same retry path, so it gets a
|
||
breaker with the same 5-failure / 30-second defaults. Fire-and-forget
|
||
targets of either type (`max_retries` == 0), database targets (local
|
||
operations), and log targets (stdout) do not use circuit breakers.
|
||
|
||
When a circuit is open and a new delivery arrives, the engine marks the
|
||
delivery as `retrying` and schedules a retry timer for after the
|
||
remaining cooldown period. This ensures no deliveries are lost — they're
|
||
just delayed until the target is healthy again.
|
||
|
||
### Metrics
|
||
|
||
`/metrics` serves one Prometheus registry behind basic auth (see
|
||
[Infrastructure Endpoints](#infrastructure-endpoints)). Alongside the
|
||
inbound HTTP metrics recorded by the middleware, it exposes the
|
||
delivery pipeline — the part of the service that can be failing while
|
||
the receive side looks perfectly healthy, because it is: events are
|
||
arriving and being stored, they are just not getting anywhere.
|
||
|
||
| Metric | Type | Meaning |
|
||
| ------ | ---- | ------- |
|
||
| `webhooker_events_received_total` | counter | Events received and durably stored. Compare against the delivery counters on one dashboard |
|
||
| `webhooker_delivery_attempts_total` | counter | Delivery attempts actually dispatched to a target. A delivery an open circuit breaker refused is not one: it is counted as a retry instead |
|
||
| `webhooker_deliveries_succeeded_total` | counter | Deliveries that reached `delivered` |
|
||
| `webhooker_deliveries_failed_total` | counter | Deliveries that failed terminally and will not be retried |
|
||
| `webhooker_delivery_retries_total` | counter | Deliveries put back into `retrying` |
|
||
| `webhooker_delivery_duration_seconds` | histogram | Wall time of a single dispatched delivery attempt, the same duration the attempt's `DeliveryResult` records |
|
||
| `webhooker_deliveries_pending` | gauge | Deliveries currently in `pending` |
|
||
| `webhooker_deliveries_retrying` | gauge | Deliveries currently in `retrying` |
|
||
| `webhooker_circuit_breakers_open` | gauge | Circuit breakers currently open |
|
||
|
||
Every delivery metric carries exactly one label, `target_type`, and
|
||
cardinality is the whole reason for that restriction. A target type is
|
||
one of four compile-time constants, so the label domain is bounded by
|
||
construction; a value outside that set collapses to `unknown` rather
|
||
than minting a series of its own. Target ids, event ids and entrypoint
|
||
ids are deliberately not labels: they are UUIDs minted per operator
|
||
action or per inbound request, a series is never reclaimed once it
|
||
exists, and labelling by any of them would make `/metrics` a memory
|
||
leak that grows with traffic.
|
||
|
||
The two queue-depth gauges are counted out of the databases by a
|
||
sampler that runs every 30 seconds for as long as the delivery engine
|
||
does, rather than tracked as deltas alongside the status transitions: a
|
||
delta would have to be seeded at startup from rows a previous process
|
||
wrote, and would drift permanently on any transition that failed to
|
||
persist.
|
||
|
||
Those two gauges also publish an `unknown` series, from startup rather
|
||
than on first occurrence. Deliveries queued against a target that has
|
||
since been deleted are counted there: that backlog is the one nobody is
|
||
watching, so it is the one that must not silently vanish from the
|
||
gauge. The outcome counters move only after the status change has been
|
||
written, so a transition the database rejected is never reported as an
|
||
outcome that happened.
|
||
|
||
### Rate Limiting
|
||
|
||
Global blanket rate limiting middleware (e.g., a per-IP throttle shared
|
||
with the web UI) **must not** apply to webhook receiver endpoints.
|
||
Webhook endpoints receive automated traffic from external services at
|
||
unpredictable rates, and blanket limits shared with other routes would
|
||
cause legitimate deliveries to be dropped.
|
||
|
||
The receiver instead has its own dedicated abuse limit, scoped to the
|
||
`/webhook/{uuid}` route only and keyed per client IP per request path
|
||
(`httprate.KeyByEndpoint`): one misbehaving sender is throttled without
|
||
affecting other senders of the same entrypoint or the same sender's
|
||
other entrypoints. Keying on the path rather than on the entrypoint
|
||
matters — see the aggregate limit below. The limit is
|
||
`RECEIVER_RATE_LIMIT` requests per minute (default 120, generous for
|
||
legitimate webhook senders). Requests over the limit receive HTTP 429
|
||
with a `Retry-After` header. A set-but-invalid `RECEIVER_RATE_LIMIT`
|
||
value aborts startup rather than silently falling back to the default.
|
||
|
||
A second limit sits in front of that one, keyed on the client IP alone
|
||
and covering the whole route at ten times `RECEIVER_RATE_LIMIT` requests
|
||
per minute (default 1200). The per-entrypoint limit needs it: the route
|
||
pattern matches any single path segment, so a client that invents a
|
||
fresh path per request gets a fresh per-entrypoint bucket every time and
|
||
would otherwise have no aggregate limit at all — while each of those
|
||
requests still costs an entrypoint lookup before it 404s. The aggregate
|
||
limit leaves room for one address to drive several entrypoints at their
|
||
full rate, and it is not configurable separately.
|
||
|
||
What that aggregate limit bounds is the database work an invented path
|
||
costs; log volume it caps rather than eliminates. A path that names no
|
||
entrypoint is recorded by the handler at `DEBUG`, and the aggregate
|
||
limiter logs its own rejections at `DEBUG` and without the path, so
|
||
neither appears at all under the default level. The per-entrypoint
|
||
limiter is the loud one: it logs every rejection at `WARN` with the
|
||
request path, which on this route is attacker-controlled text. A client
|
||
hammering a single invented path is served `RECEIVER_RATE_LIMIT`
|
||
requests and has the rest of its aggregate budget rejected there, so
|
||
the aggregate limit is what bounds the _number_ of those `WARN` lines —
|
||
to under ten times `RECEIVER_RATE_LIMIT` per minute per client IP, 1080
|
||
at the defaults, where before it there was no bound at all. Their
|
||
_width_ is bounded by the field budgets below, the same ones the access
|
||
log spends. The access log is bounded by neither limit: every request
|
||
is recorded once at `INFO`, served or rejected alike.
|
||
|
||
What the access log does bound is the _content_ of those lines. A 3xx
|
||
or 4xx response logs the chi route pattern — `/webhook/{uuid}`,
|
||
`/user/{username}//`, or the literal `(unmatched)` when the request hit
|
||
no route at all — in place of the concrete URL. Those are the outcomes
|
||
an unauthenticated client can drive for free: 404 and 429 on any
|
||
invented receiver path, a login redirect on any invented profile path.
|
||
Logging the URL there would let a flood write text of its own choosing,
|
||
at a length of its own choosing, into the log. 2xx and 5xx responses
|
||
keep the concrete path — a success resolved against a static route or
|
||
against the operator's own data (on the receiver, against a stored
|
||
entrypoint UUID), and a 5xx is a bug in this service, where the exact
|
||
path is the evidence and no client can provoke one at will.
|
||
|
||
The query string is never logged; it is replaced by the fixed marker
|
||
`?(redacted)`. It is client-chosen on every route, and
|
||
`/.well-known/healthcheck` and `/s/*` answer 200 to anyone with no rate
|
||
limiter in front of them, so a query on a fixed 200 URL would otherwise
|
||
buy the same amplification as an invented path. Nothing debuggable is
|
||
lost: `page`, on the authenticated pagination links, is the only query
|
||
parameter this service reads.
|
||
|
||
Client-supplied request content does not leave the host by the other
|
||
route either. The Sentry SDK attaches the request to every event it
|
||
captures, independently of the access log, and `SendDefaultPII=false`
|
||
does not cover all of what it copies: the raw query string and the
|
||
first 10 KiB of the request body are both taken unconditionally, the
|
||
body precisely because these handlers call `ParseForm`. A `BeforeSend`
|
||
hook therefore replaces the query string and the body with
|
||
`(redacted)`, drops cookies and the remote-address environment, and
|
||
reduces the headers to a fixed allowlist — `Accept`, `Content-Length`,
|
||
`Content-Type`, `Host`, `Origin`, `Referer`, `User-Agent` and
|
||
`X-Request-Id`.
|
||
|
||
The same hook rewrites the request URL. The SDK builds it as
|
||
`scheme://host/path` from the concrete path, which on the receiver
|
||
route is `/webhook/<uuid>` in full — and that UUID is a write
|
||
capability, not an identifier: anyone holding it can post events this
|
||
service accepts and its targets then deliver. A tracker has its own
|
||
retention, access control and deletion policy, so the rule the access
|
||
log follows above does not carry across that boundary. What is sent is
|
||
the chi route pattern instead: `http://host/webhook/{uuid}`.
|
||
|
||
The scheme and the host are kept, and everything else in the URL is
|
||
discarded rather than edited, so a future SDK version that starts
|
||
appending a query string cannot widen this. The scheme has to survive
|
||
for the reason given below. The host is whatever the request's `Host`
|
||
header carried — this service validates no hostname, so on a directly
|
||
exposed deployment a client sets it — and that same header is on the
|
||
allowlist above, so scrubbing the host out of the URL would withhold
|
||
nothing that is not sent anyway.
|
||
|
||
The body, the query string and the URL are all handled on every route
|
||
rather than filtered by route. For the URL that is also what keeps the
|
||
event locatable: an error event is grouped by its exception and stack
|
||
trace, not by its URL, so replacing the path with the pattern costs no
|
||
grouping and the pattern still names the route in the UI. And an
|
||
unconditional rule cannot leak on a route somebody forgets to add to
|
||
it, which a route-conditional one can. For the body there is a second
|
||
reason: nothing debuggable is lost, because every handler reads its
|
||
fields with `PostFormValue`, so the body is exactly where the
|
||
credentials are — the target destination URL, the login password, both
|
||
password-change fields — and the one route whose body is genuine
|
||
signal is the receiver, whose body is already stored on the event and
|
||
served from the UI, so a tracker is not where anyone reads it.
|
||
|
||
The route is reachable from the hook only on the error dispatch.
|
||
`sentryhttp`'s recover path puts the request on the context it hands
|
||
to `RecoverWithContext`, and the SDK carries that context through to
|
||
`BeforeSend` as `hint.Context`, so
|
||
`hint.Context.Value(sentry.RequestContextKey)` yields the live request
|
||
and chi's `RoutePattern()` yields the matched pattern off it. The
|
||
transaction dispatch has no such request: a finished span captures
|
||
with a nil hint, which the client replaces with an empty one, so
|
||
`BeforeSendTransaction` sees no context at all. Tracing is off in this
|
||
service, so no transaction event is produced today, but the hook is
|
||
installed on both dispatches as a floor.
|
||
|
||
Where the pattern is out of reach — the transaction dispatch, an event
|
||
captured outside the router, or a request that matched no route — the
|
||
fallback is never the concrete path. The path becomes the literal
|
||
`/(redacted)`, so the URL reads `http://host/(redacted)`; a URL the
|
||
rewrite cannot parse into a scheme is withheld whole. A transaction
|
||
event additionally carries the SDK's own `METHOD /path` name, built
|
||
from the concrete path as well; it is rewritten on the same terms, to
|
||
`POST /webhook/{uuid}` where the pattern is known and `POST
|
||
/(redacted)` where it is not.
|
||
|
||
The headers are an allowlist for the same reason the rules above are
|
||
unconditional: the SDK's own filter removes four names and passes
|
||
everything else, which would ship `X-CSRF-Token` and the shared
|
||
secrets senders put on the receiver route. What survives still names
|
||
the failing route — scheme, host, route pattern, method — and
|
||
`X-Request-Id` ties the event to the local access log line that holds
|
||
the rest. Nothing dropped is needed for the likeliest use, debugging a
|
||
CSRF rejection. Its three inputs are the TLS decision, `Origin` and
|
||
`Referer`; the latter two are kept, and the first is the scheme of the
|
||
retained URL, because the SDK derives that scheme from
|
||
`r.TLS != nil || r.Header.Get("X-Forwarded-Proto") == "https"` — byte
|
||
for byte the predicate `internal/middleware/csrf.go` uses to choose
|
||
between the `csrf.Secure(true)` and `csrf.Secure(false)` handlers.
|
||
That is what the rewrite above preserves it for, and it is why
|
||
dropping `X-Forwarded-Proto` costs nothing. The dropped provider
|
||
headers (`X-GitHub-Event`, `X-Gitlab-Event` and the like) are real
|
||
signal but are recorded locally on the event, and
|
||
`Sentry-Trace`/`Baggage` are already reflected in the event's trace
|
||
context.
|
||
|
||
The remaining client-supplied fields are truncated rather than dropped,
|
||
each to a fixed budget: 512 bytes for `url`, `useragent` and `referer`,
|
||
128 for `request_id` (chi passes an inbound `X-Request-Id` header
|
||
through), and 32 for `method`. A truncated `User-Agent` is still worth
|
||
reading; an absent one is not. A cut value ends in `[truncated]`, which
|
||
is charged on top of the budget rather than inside it.
|
||
|
||
Each budget is spent in _encoded_ bytes, not in the bytes the client
|
||
sent. Every rune is charged what the wider of the two log handlers
|
||
emits for it: two bytes for a quotation mark, a backslash or a tab; six
|
||
for a non-printable rune below U+10000; ten for one at or above it,
|
||
which the text handler spells `\UXXXXXXXX`. Go's header parser accepts
|
||
all of them in a header value, so a budget counted raw would buy a
|
||
field several times its nominal size — and the line, not the header, is
|
||
what an operator has to store. Plain ASCII encodes one byte for one, so
|
||
a real browser's `User-Agent` still fits whole; a value built out of
|
||
escapes keeps a proportionally shorter prefix, which is the right
|
||
trade.
|
||
|
||
Net: **one `INFO` line per request, of at most 2,560 bytes.** That
|
||
ceiling is arithmetic, not an observation: 3 × (512 + 11) for `url`,
|
||
`useragent` and `referer`, plus 128 + 11 for `request_id`, plus 32 + 11
|
||
for `method`, plus a 336-byte fixed portion (the field names, the
|
||
punctuation, both timestamps at their longest, an IPv6 `remoteIP` with
|
||
a zone, the status and the latency) — 2,087 bytes, stated at 2,560 so
|
||
the figure has headroom. `internal/middleware/accesslog_test.go`
|
||
asserts it against 8 KB of client-chosen text in the path, in the
|
||
query, and in each of `User-Agent`, `Referer` and `X-Request-Id`,
|
||
including cases built from the characters the handlers escape, and
|
||
against the widest access log line the service can be made to write: a
|
||
5xx that keeps its concrete path while all three header fields are also
|
||
at their budget. Every case runs through both handlers
|
||
`internal/logger` can select — the JSON one and the text one it installs
|
||
on a tty — since the two do not escape alike and the ceiling is quoted
|
||
unqualified. Measured over a real connection, the widest access log line
|
||
is 1,972 bytes.
|
||
|
||
Multiply that ceiling by the request rate to size log storage. Note
|
||
that the rate is not bounded by the limits above on every route:
|
||
`/.well-known/healthcheck` and `/s/*` sit behind no limiter, so there
|
||
the multiplier is whatever the deployment will serve.
|
||
|
||
**The same ceiling covers every other line the service writes through
|
||
`slog` that carries text an unauthenticated client supplies**, with one
|
||
exception stated below it: the recovered-panic record, which carries a
|
||
whole goroutine stack alongside its client-supplied fields and so has
|
||
its own wider ceiling. The access log is not the only line a client can
|
||
put its own text into, and a budget that held for one line and not the
|
||
others would be worse than no stated budget at all. Every `slog` call an
|
||
unauthenticated request can reach spends the same per-field budget
|
||
through `internal/logfield`, and each carries strictly fewer
|
||
client-supplied fields than the access log does, so none of them can be
|
||
wider than it:
|
||
|
||
| Log line | Level | Client-chosen value | Reachable unauthenticated |
|
||
| ------------------------------------------ | ------- | ------------------- | ----------------------------------------- |
|
||
| `request body exceeds limit` (413) | `WARN` | path, method | yes — `MaxBodySize` precedes `RequireAuth` |
|
||
| `csrf: token validation failed` (403) | `WARN` | path, method | yes — `CSRF` precedes `RequireAuth` |
|
||
| `... rate limit exceeded` (429) | `WARN` | path | yes, on the receiver |
|
||
| `auth middleware: unauthenticated request` | `DEBUG` | path, method | yes, by definition |
|
||
| `entrypoint not found` | `DEBUG` | entrypoint UUID | yes, on the receiver |
|
||
| `user not found` / `invalid password` | `DEBUG` | username | yes, on the login form |
|
||
| `login failure limit exceeded` (429) | `WARN` | path | yes, on the login form |
|
||
| `password verification capacity exhausted` | `WARN` | path | yes, on the login form |
|
||
|
||
`DEBUG` being off by default is not a bound. An operator turning it on
|
||
to diagnose a flood must not thereby hand the flood an unbounded write,
|
||
so those lines are capped too.
|
||
|
||
The last two rows are capped defensively rather than against a
|
||
demonstrated width: chi routes `POST /pages/login` on a static pattern,
|
||
so `r.URL.Path` there is the 12-byte constant `/pages/login` and each
|
||
line lands near 120 bytes. `RecordLoginFailure` is nonetheless an
|
||
exported method taking any `*http.Request`, and a future caller on a
|
||
route with a URL parameter would widen the line. Since no request
|
||
through the mux can, both caps are pinned by tests that call those two
|
||
entry points directly with the path such a caller would supply.
|
||
Removing either cap fails 14 subtests.
|
||
|
||
`internal/middleware/logbound_test.go` and
|
||
`internal/handlers/logbound_test.go` drive 8 KB of client-chosen text
|
||
at each of these — 1 KB at `invalid password`, whose accounts are
|
||
shared with the successful-login line, where a username past 4 KB
|
||
overflows the session cookie and answers 500 before that line is
|
||
written — through both handlers, and through seven fills: plain text
|
||
as the baseline, and then the quotation mark, backslash, tab, newline,
|
||
C0 control and astral non-printable, six characters the wider of the
|
||
two handlers spends more on than the client spent sending them. Every
|
||
case holds each line to the 2,560-byte ceiling. That per-line ceiling
|
||
is what the figure above states, and every row establishes it.
|
||
|
||
Three of the sites go further and bound the whole flood's output — the
|
||
total bytes a run of distinct invented values wrote, which is the
|
||
shape an operator sizing storage cares about. They are
|
||
`request body exceeds limit`
|
||
(`TestMaxBodySize_FloodOfOversizePathsDoesNotGrowTheLog`),
|
||
`entrypoint not found` and `user not found` (the last two through
|
||
`assertBoundedFlood`). The other rows carry no aggregate assertion;
|
||
the per-line ceiling is what they establish.
|
||
|
||
`internal/logfield/logfield_test.go` measures the per-rune charge
|
||
against what the handlers really emit, over roughly 3,000 code points on
|
||
each, so an undercharged rune fails a test rather than quietly
|
||
falsifying the ceiling.
|
||
|
||
**It covers GORM's statement logging as well.** GORM's own default
|
||
logger printed the fully interpolated SQL — parameters and all — to
|
||
standard output on every statement that returned an error, including a
|
||
plain record-not-found, at a level no operator setting reached. Two of
|
||
this service's lookups miss by design on unauthenticated routes: the
|
||
entrypoint lookup behind `/webhook/{uuid}` and the user lookup behind
|
||
the login form, whose path segment and submitted username the client
|
||
picks outright. Every
|
||
`gorm.Open` in the service now installs the adapter in
|
||
`internal/gormlog` instead. It writes through the same `slog` logger as
|
||
everything else, so its lines take the level the operator set and the
|
||
handler `internal/logger` selected, and every value it emits is spent
|
||
through the same `internal/logfield` budget. A record-not-found is not
|
||
logged as an error: it is the expected outcome on both of those paths,
|
||
and each handler already records its own miss at `DEBUG` — bounded, per
|
||
the table above — without the SQL. Slow statements are kept, at `WARN`,
|
||
above the same 200 ms threshold GORM used and with the statement
|
||
bounded, because that report is the one thing GORM's logger gave an
|
||
operator that nothing else here does. The adapter orders its cases
|
||
exactly as GORM's own `Trace` orders them, so a statement that both
|
||
missed and ran slow is still reported as slow, and dropping the miss
|
||
costs an operator no report `IgnoreRecordNotFoundError` would have
|
||
kept. A GORM line spends at most two of those budgets — the statement
|
||
and the driver error — against a smaller fixed portion than the access
|
||
log's, and `internal/gormlog/gormlog_test.go` asserts each line against
|
||
`MaxAccessLogLineBytes` directly rather than leaving it as arithmetic.
|
||
|
||
The adapter also logs no bound value at all: it implements
|
||
`gorm.ParamsFilter` and discards the parameters, so GORM renders the
|
||
statement with its placeholders intact instead of substituting the
|
||
values into it. That is a separate property from the size bound and it
|
||
is what a bound is no substitute for — the session encryption key is 44
|
||
base64 characters and an Argon2id hash under 100, so both fit inside
|
||
every budget above and a truncated secret is still a secret. It holds
|
||
on all three arms of `Trace`, including the routine one an operator
|
||
reaches at `DEBUG`, which is the only level at which a successful
|
||
`INSERT` is written at all. One GORM path does not consult the filter —
|
||
`(*gorm.DB).Scan`, which records the statement through GORM's own trace
|
||
recorder. No production code path calls it; its one caller is
|
||
`internal/database/database_test.go:91`, whose `SELECT 1` binds
|
||
nothing, and `internal/gormlog/scan_guard_test.go` fails if a non-test
|
||
file calls it. `Pluck`, `Row` and `Raw` all run through the normal
|
||
callback processor and are filtered.
|
||
See `#### What DEBUG=true exposes` under Configuration.
|
||
|
||
What that ceiling does **not** cover, stated here so the figure is not
|
||
read as more than it is:
|
||
|
||
- **Lines carrying an authenticated operator's own input**, which are
|
||
not truncated at all. `webhook created` logs the submitted `name`
|
||
verbatim and `target URL blocked by SSRF protection` logs the target
|
||
host (both `internal/handlers/source_management.go`), as do the
|
||
`target_name` lines in `internal/delivery/engine.go` and
|
||
`internal/delivery/target_http.go`. The only bound on any of them is
|
||
the 1 MB form body cap, so a 100 KB `name` writes a single line of
|
||
roughly 600 KB — measured. This is deliberate: every one of these
|
||
requires an authenticated operator on a service with no
|
||
self-registration, and truncating the operator's own configuration
|
||
echoed back would cost debuggability against no adversary. It does
|
||
mean the 2,560-byte figure sizes unauthenticated traffic, not the
|
||
operator's own administrative requests.
|
||
- The **`log` delivery target**, which writes the whole inbound event —
|
||
headers and body — to the log. This one is deliberate: capping it
|
||
would defeat the target, since emitting the payload is the delivery.
|
||
It costs nothing unless an authenticated operator creates a target of
|
||
that type on a specific webhook, and each line it writes is bounded
|
||
per event by the 1 MB receiver body cap. Adding one is a decision to
|
||
spend log volume on that webhook's payloads.
|
||
- **Two writers that do not go through `internal/logger` at all**, both
|
||
on standard error. `fx` prints the dependency graph and the lifecycle
|
||
hooks through its default console logger at startup and shutdown —
|
||
nothing calls `fx.WithLogger`, and `fx.New` builds that logger over
|
||
`os.Stderr`. The Go runtime writes a panic or a fatal error itself; a
|
||
panic in a background worker rather than in a request handler is the
|
||
case that reaches it, since nothing recovers those. Neither carries a
|
||
client-chosen value at a client-chosen length: the five `panic` calls
|
||
in this service are invariant guards over constants and over
|
||
`crypto/rand`.
|
||
- **`net/http`'s own faults**, which are _not_ a separate writer.
|
||
`internal/server/http.go` builds its server with a nil `ErrorLog`, so
|
||
`net/http` falls back to the `log` package's default logger — and
|
||
`internal/logger` calls `slog.SetDefault`, which redirects that logger
|
||
into whichever handler it installed. Those lines therefore arrive on
|
||
standard output, shaped like every other line, at `INFO`. They are not
|
||
truncated. A handler panic is no longer one of them: the recover
|
||
middleware below answers it and writes it as the bounded record
|
||
described there instead, and `internal/server/recoverer_test.go`
|
||
requires that `http: panic serving` appear in neither of the process's
|
||
two streams when a panic is driven through the production router. The
|
||
one panic still handed back to `net/http` is `http.ErrAbortHandler`,
|
||
which it special-cases and does not log at all. What is left on this
|
||
path is `net/http`'s own diagnostics, whose values are the runtime's,
|
||
not a client's.
|
||
|
||
Wider than that 2,560-byte ceiling, and stated separately rather than
|
||
carved out of it: the record a recovered panic produces. The recover
|
||
middleware in `internal/middleware` answers `500` and writes one `ERROR`
|
||
record through `internal/logger` carrying the panic value, the stack and
|
||
the request id — the same `request_id` the access log line for that
|
||
request carries, which is how the two are joined. It replaced chi's
|
||
`middleware.Recoverer`, which on a current Go release crashed inside its
|
||
own stack pretty-printer: the connection was dropped rather than
|
||
answered, and what reached the operator described that crash rather than
|
||
the fault behind it.
|
||
|
||
That record is bounded the same way, in the same encoded bytes and
|
||
through the same `internal/logfield` budget: 512 for the panic value,
|
||
because a handler is free to build one out of the request, 128 for the
|
||
request id, which a client supplies outright through `X-Request-Id`,
|
||
and 8,192 for the stack, cut at its far end so that the panic site
|
||
survives a cut and `net/http`'s accept frames are what is lost. Net:
|
||
**at most 10,240 bytes, once per recovered panic** — 9,121 by the
|
||
arithmetic (523 + 8,203 + 139 + a 256-byte fixed portion), stated at
|
||
10,240 for headroom.
|
||
|
||
Those two numbers are the claim; the measurements below only
|
||
illustrate it. `internal/middleware/recoverer_test.go` drives all
|
||
three growable fields past their budgets on one record, over both
|
||
handlers, and measured 9,009 bytes on the JSON handler and
|
||
8,982–8,983 on the text one in one checkout. Neither is an invariant:
|
||
the stack's own content decides where its cut lands, so the figures
|
||
move by a byte or so between runs. The real case is far below both —
|
||
through the shipped middleware chain the whole record measures
|
||
roughly 3,960 bytes over a roughly 3,690-byte stack, taken by
|
||
`internal/server/recoverer_test.go` from the process's own file
|
||
descriptors while driving a panic through the production router over
|
||
a real server in a subprocess. That pair moves further still, since
|
||
`debug.Stack()` embeds absolute source paths and so depends on where
|
||
the tree is checked out: four checkouts have reported 3,959, 3,961,
|
||
3,984 and 4,026. What the tests assert is the ceiling, that every
|
||
client-supplied field was cut, and that the shipped chain's stack
|
||
arrived uncut — never the numbers.
|
||
|
||
Every limiter here — receiver, login, and password change — identifies
|
||
the client the same way, through one shared key function: the
|
||
connection's own address, unless the peer is listed in
|
||
`TRUSTED_PROXIES`, in which case the forwarded client address is used
|
||
instead. That address becomes a bucket by family: IPv4 keys on the full
|
||
address, IPv6 on its `/64` prefix. A routed `/64` is the normal
|
||
residential and mobile IPv6 allocation, so keying IPv6 per address would
|
||
let one subscriber rotate source addresses and mint a fresh bucket per
|
||
request, evading these limits at the network layer without spoofing
|
||
anything; the cost is that distinct clients inside one `/64` share a
|
||
bucket. IPv4-mapped addresses (`::ffff:1.2.3.4`) key as the IPv4 address
|
||
they carry. See [Trusted proxies](#trusted-proxies). Deployed without that
|
||
variable set, a client behind a reverse proxy shares one bucket with
|
||
every other client behind the same proxy. Set `TRUSTED_PROXIES` to the
|
||
proxy's address to get per-client limits back. What the shared bucket
|
||
costs is not the same for every limiter, and the two cases pull in
|
||
opposite directions:
|
||
|
||
- For the **receiver** limits it costs throughput, which is the safe
|
||
direction to be wrong in: sharing can only make a limit bind sooner,
|
||
never let a sender past it. It matters more for the aggregate limit
|
||
than for the per-entrypoint one: with `TRUSTED_PROXIES` unset behind
|
||
the reverse proxy a production deployment is required to run behind,
|
||
every request keys on the proxy, so the aggregate limit becomes a
|
||
service-wide ceiling of 1200 requests per minute across all senders
|
||
and all entrypoints, where the per-entrypoint limit's capacity still
|
||
grows with the number of entrypoints. Any deployment with more than a
|
||
handful of busy entrypoints must set `TRUSTED_PROXIES`.
|
||
- For the **login and password-change** limits it costs precision, not
|
||
availability. Login failures from every client land in one counter,
|
||
so a stranger's wrong passwords make the operator's own wrong
|
||
passwords answer `429` sooner; the operator's _correct_ password is
|
||
never affected, because it is never counted. Production deployments
|
||
should still set `TRUSTED_PROXIES`; webhooker warns at startup
|
||
whenever it is empty, in any environment.
|
||
|
||
#### The login endpoint
|
||
|
||
The login `POST` is the one endpoint with no pre-emptive limiter in
|
||
front of it, and that is deliberate. A limiter that spends budget on
|
||
arrival is a lockout in this deployment shape: sharing one bucket, a
|
||
stranger sending five POSTs a minute — about 0.08 requests per second,
|
||
from anywhere — keeps it permanently full, and the operator has no
|
||
second administrative path. So the handler inverts the order:
|
||
|
||
1. **Credentials are verified first, and only a failed attempt spends
|
||
budget.** A correct password is never rate-limited, whatever the
|
||
counters hold. This is what guarantees the admin UI stays
|
||
reachable.
|
||
2. **Failures are counted per (client bucket, submitted username)**,
|
||
five per minute, after which further _failures_ from that pair are
|
||
answered `429` with a `Retry-After`. That `429` is a label on the
|
||
response, not a gate in front of the work: the credential check has
|
||
already run by the time the counter is consulted, so a throttled
|
||
client's guess is still evaluated. See the guessing rate below. A
|
||
successful login clears the counter, so mistyping a few times and
|
||
then getting it right leaves you unthrottled. Because the submitted
|
||
username is attacker-controlled, at most 1024 username counters and
|
||
1024 fallback address counters are tracked; past the first cap
|
||
failures fall back to the address counter, and past both they are
|
||
answered as throttled without being recorded. Total tracked state
|
||
is under half a megabyte and does not grow with the number of
|
||
usernames an attacker invents.
|
||
3. **Concurrent password verifications are capped at two, and the
|
||
queue for them at 16.** Verifying before counting means every login
|
||
request costs an Argon2id hash, and Argon2id here is 64 MB per
|
||
hash — two slots is a 128 MB ceiling on password hashing. Every
|
||
endpoint that hashes a password takes a slot, including the
|
||
password-change endpoint, which holds one across both the
|
||
verification and the new hash. A request that waits five seconds
|
||
without getting a slot is answered `503 Service Unavailable` and no
|
||
hash is computed for it. The wait alone does not bound memory, only
|
||
how long one request holds some, so the number of waiters is capped
|
||
as well. Size the queue from what a parked waiter actually retains,
|
||
not from the 1 MB body cap: that caps the raw body read, while the
|
||
body-cap, CSRF and form-parsing middleware all run before the
|
||
guard, so a waiter holds its parsed form plus its request header
|
||
block for the whole wait. Measured on the pinned Go 1.26.1
|
||
toolchain, as the heap delta with 64 waiters parked in the handler,
|
||
an ordinary two-field login form retains ~0 MB, a 1 MB urlencoded
|
||
body at Go's 10,000-parameter parse cap retains 2.82 MB (3.09 MB
|
||
with `%41` escapes), and the ~0.9 MB of headers the 1 MB header cap
|
||
allows takes it to **4.18 MB** — the retained parse and the headers
|
||
dominate, not the raw body. So the cap is 16 waiters: 16 x 4.18 MB
|
||
is about 67 MB of committed queue memory, and two slots drain a
|
||
full 16-deep queue in roughly 0.6 s, far inside the five-second
|
||
deadline. A request arriving past the cap is shed with `503`
|
||
immediately instead of joining the queue. **Peak commitment for the
|
||
endpoint is therefore about 203 MB**: 128 MB of Argon2id, plus the
|
||
18 requests holding a parsed form — 16 queued and the 2 being
|
||
hashed — at about 75 MB. That 203 MB is _live_ commitment, not
|
||
resident size: the Go collector lets the heap reach roughly twice
|
||
the live set before collecting, with transient parse garbage on top
|
||
of it. The independent review of this endpoint fired 18 adversarial
|
||
requests at an idle guard and measured a peak `HeapAlloc` of
|
||
392 MB. **Provision on the order of 400 MB**, not for the 203 MB
|
||
itemised here and not for the hashing budget alone.
|
||
|
||
An unknown username is verified against a dummy hash rather than
|
||
rejected early, so a nonexistent account costs the same time as a real
|
||
one and the response cannot be used to enumerate usernames.
|
||
|
||
**This raises online guessing throughput by about 300x, and that is
|
||
the trade.** Because the credential check always precedes the counter,
|
||
what bounds online brute force is the semaphore, not the failure
|
||
counter. Two slots at the cost of one Argon2id verification is on the
|
||
order of **27 guesses per second, about 2.3 million per day**, against
|
||
5 per minute under the pre-emptive limiter this replaced. Treat that
|
||
figure as a lower bound rather than a ceiling: it was measured with
|
||
Go's race detector enabled, so real hardware verifies faster and
|
||
guesses faster. Choose the admin password to survive millions of
|
||
online guesses per day — a long random passphrase, not a memorable
|
||
one. Rate-limiting `POST /pages/login` at the reverse proxy, where the
|
||
real client address is visible, is the way to put a cheaper bound back
|
||
on top.
|
||
|
||
The residual exposure is a bounded, self-clearing loss of login
|
||
**availability** — not merely of latency. A flood can keep both
|
||
verification slots busy, and a request that neither gets a slot within
|
||
five seconds nor finds room in the queue is answered `503`. Above
|
||
roughly 27 requests per second the operator is not served slowly, it
|
||
is shed: its chance per attempt is about the ratio of service rate to
|
||
flood rate, so at 400 requests per second it is roughly one attempt in
|
||
fourteen. A sufficiently determined flood still denies login for as
|
||
long as it runs.
|
||
|
||
What changed is the price and the aftermath. Denying login used to
|
||
cost an attacker 0.08 requests per second from anywhere; it now costs
|
||
30 or more sustained, about 400 times as much. Nothing accumulates
|
||
while the flood runs, nothing needs resetting when it stops, and the
|
||
operator's correct password succeeds on the first attempt afterwards.
|
||
Restarting the service is **not** a remedy: a restart clears the
|
||
failure counters, which are not what is saturated, and the flood
|
||
re-fills both verification slots on its first two requests. The
|
||
remedies are to block the source at the reverse proxy, or to
|
||
rate-limit `POST /pages/login` there — the one place a limit can be
|
||
applied without reintroducing the lockout, because the proxy sees the
|
||
real client address. Setting `TRUSTED_PROXIES` does not stop the
|
||
saturation, but it makes the source visible in the failure logs.
|
||
|
||
Finer-grained per-webhook rate limits (configured in the web UI and
|
||
enforced in the webhook handler) can layer on top of this env-level
|
||
abuse limit later; they are tracked as future work.
|
||
|
||
### API Endpoints
|
||
|
||
#### Public Endpoints
|
||
|
||
| Method | Path | Description |
|
||
| ------ | --------------------------- | ----------- |
|
||
| `GET` | `/` | Root redirect, 303 (authenticated → `/sources`, unauthenticated → `/pages/login`) |
|
||
| `GET` | `/.well-known/healthcheck` | Health check (JSON: `status`, `now`, `uptimeSeconds`, `uptimeHuman`, `version`, `appname`, `maintenanceMode`) |
|
||
| any | `/s/*` | Static file serving (embedded CSS, JS). Mounted for every method, not just `GET`/`HEAD`: chi's `Mount` registers all methods and `http.FileServer` special-cases only `HEAD` (by omitting the body), so a `POST` or `DELETE` to an asset is answered `200` with the file. Pinned by `TestStaticServesEveryMethod` |
|
||
| `POST` | `/webhook/{uuid}` | Webhook receiver endpoint. `POST` only — every other method is answered `405 Method Not Allowed` with `Allow: POST`. Rate limited (see [Rate Limiting](#rate-limiting)) |
|
||
|
||
#### Authentication Endpoints
|
||
|
||
| Method | Path | Description |
|
||
| ------ | --------------- | ----------- |
|
||
| `GET` | `/pages/login` | Login page (not rate limited) |
|
||
| `POST` | `/pages/login` | Login form submission. Credentials are verified before any limit is consulted, so a correct password is never throttled; 5 FAILED attempts per minute per bucket per submitted username, then `429`. `503` if no verification slot frees up within 5s, or immediately if 16 requests are already queued for one (see [Rate Limiting](#rate-limiting)) |
|
||
| `POST` | `/pages/logout` | Logout (destroys session) |
|
||
|
||
#### Authenticated Endpoints
|
||
|
||
| Method | Path | Description |
|
||
| ------ | ------------------------ | ----------- |
|
||
| `GET` | `/user/{username}` | User profile page |
|
||
| `POST` | `/user/{username}/password` | Change the user's password (5 per minute per bucket, then `429`; `503` if no verification slot frees up within 5s, or immediately if 16 requests are already queued for one) |
|
||
| `GET` | `/sources` | List user's webhooks |
|
||
| `GET` | `/sources/new` | Create webhook form |
|
||
| `POST` | `/sources/new` | Create webhook submission |
|
||
| `GET` | `/source/{id}` | Webhook detail view |
|
||
| `GET` | `/source/{id}/edit` | Edit webhook form |
|
||
| `POST` | `/source/{id}/edit` | Edit webhook submission |
|
||
| `POST` | `/source/{id}/delete` | Delete webhook |
|
||
| `GET` | `/source/{id}/logs` | Webhook event logs |
|
||
| `POST` | `/source/{id}/entrypoints` | Add entrypoint to webhook |
|
||
| `POST` | `/source/{id}/entrypoints/{entrypointID}/delete` | Delete an entrypoint |
|
||
| `POST` | `/source/{id}/entrypoints/{entrypointID}/toggle` | Enable or disable an entrypoint |
|
||
| `POST` | `/source/{id}/entrypoints/{entrypointID}/secret` | Set, rotate or remove the entrypoint's inbound signature scheme and secret (see [Inbound Signature Verification](#inbound-signature-verification)) |
|
||
| `POST` | `/source/{id}/targets` | Add target to webhook |
|
||
| `POST` | `/source/{id}/targets/{targetID}/delete` | Delete a target |
|
||
| `POST` | `/source/{id}/targets/{targetID}/toggle` | Enable or disable a target |
|
||
|
||
#### Infrastructure Endpoints
|
||
|
||
| Method | Path | Description |
|
||
| ------ | ---------- | ----------- |
|
||
| `GET` | `/metrics` | Prometheus metrics, behind basic auth. The route is registered only when `METRICS_USERNAME` and `METRICS_PASSWORD` are both set; with neither set it does not exist and returns 404, and with only one set the process refuses to start |
|
||
|
||
#### API (Planned)
|
||
|
||
| Method | Path | Description |
|
||
| -------- | ------------------------------ | ----------- |
|
||
| `GET` | `/api/v1/webhooks` | List webhooks |
|
||
| `POST` | `/api/v1/webhooks` | Create webhook |
|
||
| `GET` | `/api/v1/webhooks/{id}` | Get webhook details |
|
||
| `PUT` | `/api/v1/webhooks/{id}` | Update webhook |
|
||
| `DELETE` | `/api/v1/webhooks/{id}` | Delete webhook |
|
||
| `GET` | `/api/v1/webhooks/{id}/events` | List events for webhook |
|
||
| `POST` | `/api/v1/events/{id}/redeliver`| Redeliver an event |
|
||
|
||
None of these exist yet. `/api/v1` is mounted with no routes, so every
|
||
path under it returns 404 today. API authentication will use API keys
|
||
passed via `Authorization: Bearer <key>` header; no Bearer middleware
|
||
is implemented either.
|
||
|
||
### Package Layout
|
||
|
||
All application code lives under `internal/` to prevent external
|
||
imports. The entry point is `cmd/webhooker/main.go`.
|
||
|
||
```
|
||
webhooker/
|
||
├── cmd/webhooker/
|
||
│ └── main.go # Entry point: subcommand dispatch; no args locks DATA_DIR and wires fx
|
||
├── internal/
|
||
│ ├── banner/
|
||
│ │ └── banner.go # Ruled block for the one credential shown in the clear
|
||
│ ├── resetpw/
|
||
│ │ └── resetpw.go # `webhooker resetpw`: set an account's password, stopped deployments only
|
||
│ ├── config/
|
||
│ │ └── config.go # Configuration loading from environment variables
|
||
│ ├── database/
|
||
│ │ ├── base_model.go # BaseModel with UUID primary keys
|
||
│ │ ├── database.go # GORM connection, migrations, admin seed
|
||
│ │ ├── models.go # AutoMigrate for config-tier models
|
||
│ │ ├── model_setting.go # Setting entity (key-value app config)
|
||
│ │ ├── model_user.go # User entity
|
||
│ │ ├── model_webhook.go # Webhook entity
|
||
│ │ ├── model_entrypoint.go # Entrypoint entity and SignatureScheme enum
|
||
│ │ ├── model_target.go # Target entity and TargetType enum
|
||
│ │ ├── model_event.go # Event entity (per-webhook DB)
|
||
│ │ ├── model_delivery.go # Delivery entity (per-webhook DB)
|
||
│ │ ├── model_delivery_result.go # DeliveryResult entity (per-webhook DB)
|
||
│ │ ├── model_apikey.go # APIKey entity
|
||
│ │ ├── password.go # Argon2id hashing and verification
|
||
│ │ ├── retention.go # Retention reaper (per-webhook event expiry)
|
||
│ │ ├── testing.go # NewTestDatabase: wrapper for tests, no fx lifecycle
|
||
│ │ └── webhook_db_manager.go # Per-webhook DB lifecycle manager
|
||
│ ├── datadir/
|
||
│ │ └── lock.go # Exclusive advisory lock on DATA_DIR (one instance)
|
||
│ ├── globals/
|
||
│ │ └── globals.go # Build-time variables (appname, version, arch)
|
||
│ ├── gormlog/
|
||
│ │ └── gormlog.go # GORM's logger.Interface on top of slog, bounded
|
||
│ ├── logfield/
|
||
│ │ └── logfield.go # Encoded-byte budget for client-supplied log values
|
||
│ ├── delivery/
|
||
│ │ ├── engine.go # Event-driven delivery engine (channel + timer based)
|
||
│ │ ├── circuit_breaker.go # Per-target circuit breaker for http/slack targets with retries
|
||
│ │ ├── target.go # Target interface, Task, Scheduler
|
||
│ │ ├── target_http.go # HTTP target (retries, circuit breaker)
|
||
│ │ ├── target_slack.go # Slack/Mattermost incoming-webhook target
|
||
│ │ ├── target_database.go # Database archive target
|
||
│ │ ├── target_database_archive.go # Archive file lifecycle and pruning
|
||
│ │ ├── target_log.go # Log target (stdout)
|
||
│ │ ├── target_config_view.go # Masked target config for templates
|
||
│ │ ├── archive_sweeper.go # Periodic pruning of idle archives
|
||
│ │ ├── queue_depth.go # Periodic sampler behind the queue-depth gauges
|
||
│ │ ├── url_mask.go # Strips credentials from *url.Error
|
||
│ │ └── ssrf.go # SSRF prevention (IP validation, safe HTTP transport)
|
||
│ ├── handlers/
|
||
│ │ ├── handlers.go # Base handler struct, JSON helpers, template rendering
|
||
│ │ ├── auth.go # Login, logout handlers
|
||
│ │ ├── entrypoint_view.go # Masked entrypoint view for templates
|
||
│ │ ├── event_log_view.go # Event log projection, byte-capped in SQL
|
||
│ │ ├── healthcheck.go # Health check handler
|
||
│ │ ├── index.go # Index page handler
|
||
│ │ ├── profile.go # User profile handler
|
||
│ │ ├── source_management.go # Webhook CRUD handlers
|
||
│ │ └── webhook.go # Webhook receiver handler
|
||
│ ├── healthcheck/
|
||
│ │ └── healthcheck.go # Health check service (uptime, version)
|
||
│ ├── lifecycle/
|
||
│ │ └── lifecycle.go # Shared stop-hook waiter, bounded by the stop context
|
||
│ ├── logger/
|
||
│ │ └── logger.go # slog setup with TTY detection
|
||
│ ├── metrics/
|
||
│ │ └── metrics.go # Delivery Prometheus collectors, labelled by target type
|
||
│ ├── middleware/
|
||
│ │ ├── middleware.go # Logging, CORS, Auth, Metrics, MetricsAuth, SecurityHeaders, MaxBodySize
|
||
│ │ ├── csrf.go # CSRF protection middleware (gorilla/csrf)
|
||
│ │ ├── ratelimit.go # Per-IP rate limiting middleware (go-chi/httprate)
|
||
│ │ ├── loginguard.go # Login failure counters and the Argon2id verification semaphore
|
||
│ │ └── testing.go # NewForTest: Middleware without the fx lifecycle
|
||
│ ├── server/
|
||
│ │ ├── server.go # Server struct, fx lifecycle, signal handling
|
||
│ │ ├── http.go # HTTP server setup with timeouts
|
||
│ │ └── routes.go # All route definitions
|
||
│ ├── session/
|
||
│ │ ├── session.go # Cookie-based session management
|
||
│ │ └── testing.go # NewForTest: Session without the fx lifecycle
|
||
│ └── signature/
|
||
│ └── signature.go # Inbound signature verification (GitHub, GitLab)
|
||
├── static/
|
||
│ ├── static.go # //go:embed directive
|
||
│ ├── css/input.css # Tailwind input, source for tailwind.css (make css)
|
||
│ ├── css/tailwind.css # Generated stylesheet the pages load
|
||
│ ├── css/style.css # Older hand-written stylesheet, no longer loaded
|
||
│ ├── js/app.js # Progressive-enhancement copy-to-clipboard
|
||
│ ├── js/alpine.min.js # Alpine.js, fetched by script/fetch-assets, not committed
|
||
│ └── vendor.sha256 # Pinned hashes the fetched assets are verified against
|
||
├── templates/ # Go HTML templates (base, login, sources, etc.)
|
||
├── script/ # Scripts to Rule Them All entrypoints
|
||
├── Dockerfile # Three stages: lint, test+build, Alpine runtime
|
||
├── Dockerfile.lint # Lint-only image built by script/lint
|
||
├── Makefile # 10 of 16 targets shim script/; 6 are inline
|
||
├── go.mod / go.sum
|
||
└── .golangci.yml # Linter configuration
|
||
```
|
||
|
||
### Dependency Injection
|
||
|
||
Components are wired via Uber fx in this order:
|
||
|
||
1. `globals.New` — Build-time variables (appname, version, arch)
|
||
2. `logger.New` — Structured logging (slog with TTY detection)
|
||
3. `config.New` — Configuration loading (environment variables)
|
||
4. `database.New` — Main SQLite connection, config migrations, admin
|
||
user seed
|
||
5. `database.NewWebhookDBManager` — Per-webhook event database
|
||
lifecycle manager
|
||
6. `database.NewRetentionReaper` — Per-webhook event retention sweep
|
||
7. `healthcheck.New` — Health check service
|
||
8. `session.New` — Cookie-based session manager (key from database)
|
||
9. `handlers.New` — HTTP handlers
|
||
10. `middleware.New` — HTTP middleware
|
||
11. `delivery.New` — Event-driven delivery engine
|
||
12. `delivery.NewArchiveSweeper` — Periodic pruning of idle archives
|
||
13. `delivery.Engine` → `delivery.Notifier` — interface bridge
|
||
14. `delivery.Engine` → `delivery.WebhookEvictor` — interface bridge so
|
||
deleting a webhook releases its archive writer
|
||
15. `server.New` — HTTP server and router
|
||
|
||
The server starts via `fx.Invoke(func(*server.Server, *delivery.Engine,
|
||
*database.RetentionReaper, *delivery.ArchiveSweeper) {})`, which
|
||
triggers the fx lifecycle hooks in dependency order. The
|
||
`delivery.Notifier` interface allows the webhook handler to send
|
||
self-contained `delivery.Task` slices to the engine without a direct
|
||
package dependency. Each task carries all target config and event data
|
||
inline (for bodies under 16 KiB, `delivery.MaxInlineBodySize`), so the
|
||
engine can deliver without reading from any database — it only writes
|
||
to record results.
|
||
|
||
### Middleware Stack
|
||
|
||
Applied to all routes in this order:
|
||
|
||
1. **RequestID** — Generate unique request IDs (chi built-in)
|
||
2. **SecurityHeaders** — Production security headers on every response
|
||
(HSTS, X-Content-Type-Options, X-Frame-Options, CSP, Referrer-Policy,
|
||
Permissions-Policy)
|
||
3. **Logging** — Structured request logging (method, URL, status,
|
||
latency, remote IP, user agent, request ID)
|
||
4. **Metrics** — Prometheus HTTP metrics (if `METRICS_USERNAME` and
|
||
`METRICS_PASSWORD` are both set)
|
||
5. **CORS** — Cross-origin resource sharing headers
|
||
6. **Timeout** — 60-second request timeout
|
||
7. **Recoverer** — Panic recovery: one `ERROR` record through
|
||
`internal/logger` and a `500`
|
||
8. **Sentry** — Error reporting to Sentry (if `SENTRY_DSN` is set;
|
||
configured with `Repanic: true` so panics still reach Recoverer)
|
||
|
||
Recoverer sits seventh rather than first, and both neighbours are the
|
||
reason. It runs **inside** everything that observes the response, so
|
||
the `500` it writes for a panicking handler is the status the access
|
||
log records and the metrics count; registered first, as chi's own
|
||
`middleware.Recoverer` was, the same request was logged as a `200` that
|
||
the client never received. It runs **outside** the Sentry handler, so
|
||
`Repanic: true` has something to re-raise into: an operator with
|
||
`SENTRY_DSN` set keeps the report, and one without it now gets the
|
||
local record instead of nothing. What that placement gives up is
|
||
recovery of a panic in the six entries above it, none of which does
|
||
more than set a header or start a timer.
|
||
|
||
Additionally, form endpoints (`/pages`, `/user/*`, `/sources`,
|
||
`/source/*`) apply a **MaxBodySize** middleware that limits
|
||
POST/PUT/PATCH request bodies to 1 MB. It is registered ahead of the
|
||
CSRF middleware in every one of those route groups, because
|
||
gorilla/csrf parses the form; if the cap were installed after it, form
|
||
parsing would run under net/http's 10 MB default and the 1 MB limit
|
||
would never apply. A request that declares a `Content-Length` over the
|
||
limit is answered with `413 Request Entity Too Large` without its body
|
||
being read and without reaching CSRF, the route group's remaining
|
||
middleware, or the handler. It is not rejected before *any* other
|
||
middleware, though: the global entries listed above all run first, so
|
||
such a request is still logged and given the security headers — and
|
||
counted in the metrics, on a deployment where the `/metrics`
|
||
credentials are set and the Metrics middleware is therefore registered
|
||
at all. The
|
||
rejection itself is logged at `WARN` with the method, path and
|
||
declared length. A chunked request, or
|
||
one that lies about its length, is hard-capped by
|
||
`http.MaxBytesReader` and fails downstream at form-parse time.
|
||
|
||
Those same four route groups then apply **CSRF** and **NoCache**
|
||
(`Cache-Control: no-store`, `Pragma: no-cache`), and every group except
|
||
`/pages` applies **RequireAuth**. The rate limiters are per-route
|
||
rather than global: **PasswordChangeRateLimit** on
|
||
`/user/{username}/password` and **ReceiverRateLimit** on
|
||
`/webhook/{uuid}`. There is deliberately none on `/pages/login` — that
|
||
endpoint counts failures inside the handler, after the credential
|
||
check, see [The login endpoint](#the-login-endpoint).
|
||
|
||
### Authentication
|
||
|
||
- **Web UI:** Cookie-based sessions using gorilla/sessions with
|
||
encrypted cookies. Sessions are configured with HttpOnly, SameSite
|
||
Lax, and Secure (in production). Absolute session lifetime is 7 days,
|
||
with a sliding idle timeout on top of it (see
|
||
[Sessions](#sessions)).
|
||
- **API (planned):** API key authentication via `Authorization: Bearer`
|
||
header. API keys are stored per-user with usage tracking
|
||
(`last_used_at`).
|
||
- **Metrics:** Basic authentication protecting the `/metrics` endpoint.
|
||
- **Recovery:** `webhooker resetpw <username>` on a stopped deployment
|
||
is the only way back into an account whose password was lost (see
|
||
[Recovering a lost admin password](#recovering-a-lost-admin-password)).
|
||
|
||
### Security
|
||
|
||
- Passwords hashed with Argon2id (64 MB memory cost)
|
||
- Session cookies are HttpOnly, SameSite Lax, Secure (prod only)
|
||
- Session regeneration on login to prevent session fixation attacks
|
||
- Session key is a 32-byte value auto-generated on first startup and
|
||
stored in the database
|
||
- Production security headers on all responses: HSTS, X-Content-Type-Options
|
||
(`nosniff`), X-Frame-Options (`DENY`), Content-Security-Policy, Referrer-Policy,
|
||
and Permissions-Policy
|
||
- Request body size limits (1 MB) on all form POST endpoints, enforced
|
||
by middleware that runs before CSRF parses the form
|
||
- **CSRF protection** via [gorilla/csrf](https://github.com/gorilla/csrf)
|
||
on all state-changing forms (cookie-based double-submit tokens with
|
||
HMAC authentication). Applied to `/pages`, `/sources`, `/source`, and
|
||
`/user` routes. Excluded from `/webhook` (inbound webhook POSTs) and
|
||
`/api` (stateless API). The middleware auto-detects TLS status
|
||
per-request (via `r.TLS` and `X-Forwarded-Proto`) to set appropriate
|
||
cookie security flags and Origin/Referer validation mode
|
||
- **Optional inbound signature verification** per entrypoint (GitHub
|
||
`X-Hub-Signature-256`, GitLab `X-Gitlab-Token`). Off by default and
|
||
off after an upgrade, so behaviour is unchanged until an operator
|
||
turns it on. Where it is on, an unsigned or wrongly signed request
|
||
is `401` and is not persisted, and a configuration the receiver
|
||
cannot apply fails closed rather than reverting to unverified. The
|
||
comparison is constant time and the secret never reaches a template,
|
||
a JSON response or a log line (see
|
||
[Inbound Signature Verification](#inbound-signature-verification))
|
||
- **SSRF prevention** for HTTP delivery targets: private/reserved IP
|
||
ranges (RFC 1918, loopback, link-local, cloud metadata) are blocked
|
||
both at target creation time (URL validation) and at delivery time
|
||
(custom HTTP transport with SSRF-safe dialer that validates resolved
|
||
IPs before connecting, preventing DNS rebinding attacks)
|
||
- **Login limiting is inverted, deliberately.** The login `POST` has
|
||
no pre-emptive rate limiter in front of it. Credentials are
|
||
verified first and only a _failed_ attempt spends budget, so a
|
||
correct password is never throttled and no flood of wrong ones can
|
||
deny the operator the only administrative path. Failures are
|
||
counted per (bucket, submitted username), five per minute, after
|
||
which further failures are answered `429` with a `Retry-After`.
|
||
What bounds brute force is not that counter but the cap of two
|
||
concurrent Argon2id verifications: a throttled client's guess is
|
||
still evaluated, so roughly 27 guesses a second get through and the
|
||
admin password has to carry that load (see
|
||
[The login endpoint](#the-login-endpoint)). `GET` requests to the
|
||
login page are not limited
|
||
- **Password-change rate limiting** via [go-chi/httprate](https://github.com/go-chi/httprate):
|
||
sliding-window rate limiter, 5 POST attempts per minute per bucket.
|
||
It runs behind session auth, so only a client already holding a
|
||
valid session reaches it, and an operator throttled out of changing
|
||
a password can still log in. The bucket is per client IP only when
|
||
`TRUSTED_PROXIES` names the reverse proxy; unset, every client
|
||
shares one bucket, which costs precision rather than availability
|
||
(see [Rate Limiting](#rate-limiting)). webhooker warns at startup
|
||
whenever `TRUSTED_PROXIES` is empty
|
||
- Prometheus metrics behind basic auth
|
||
- Static assets embedded in binary (no filesystem access needed at
|
||
runtime)
|
||
- Container runs as non-root user (UID 1000)
|
||
- GORM soft deletes on every entity that carries `BaseModel`, which is
|
||
all of them but `Setting` (data preserved for audit)
|
||
|
||
### Shutdown
|
||
|
||
On SIGINT or SIGTERM, fx runs the registered stop hooks in reverse
|
||
dependency order under a **5 second budget** (`fx.StopTimeout` in
|
||
`cmd/webhooker/main.go`). That budget covers the whole sequence, not
|
||
each hook. The order, read off the fx stop-hook log:
|
||
|
||
1. `ArchiveSweeper`
|
||
2. `RetentionReaper`
|
||
3. `server` — the HTTP drain, bounded separately by
|
||
`server.ShutdownTimeout` (**3 seconds**), then a Sentry flush if
|
||
`SENTRY_DSN` is set
|
||
4. `delivery.Engine`
|
||
5. `healthcheck`
|
||
6. `WebhookDBManager`
|
||
7. the database close
|
||
|
||
The two components that can realistically hold the budget run
|
||
first: a retention sweep or an archive prune caught mid-tick each
|
||
waits on its `WaitGroup` bounded by the stop context, so a wedge
|
||
there consumes the 5 seconds before the HTTP server hook is ever
|
||
entered. The hooks after the server are microsecond-scale in normal
|
||
operation.
|
||
|
||
The HTTP drain budget is deliberately **shorter** than the sequence
|
||
budget. Were the two equal, a drain that used its whole budget would
|
||
exhaust the sequence budget at the instant it finished, and every
|
||
later hook — the delivery engine, the healthcheck, the webhook DB
|
||
manager and the database close — would be skipped in exactly the
|
||
case where the drain mattered. 3 seconds leaves 2 seconds
|
||
(`server.TailHookReserve`) for the tail, which is far more than the
|
||
microseconds it needs.
|
||
|
||
That reserve belongs to the tail hooks, not to the server hook, and
|
||
the Sentry flush is what could take it: it runs after the drain
|
||
**inside the same hook**, and `sentry.Flush` takes a bare duration
|
||
and honours no context, so an unreachable Sentry endpoint would add
|
||
its own timeout on top of a full-length drain and consume the whole
|
||
sequence budget by itself. It is therefore clamped to whatever is
|
||
left on the stop context minus the reserve, and skipped when that
|
||
leaves too little to be worth attempting — so a full-length drain
|
||
means Sentry events are dropped rather than the database close being
|
||
skipped.
|
||
|
||
This does not make the database close unconditional: a wedged
|
||
`ArchiveSweeper` or `RetentionReaper` still runs first and can
|
||
consume the whole budget on its own.
|
||
|
||
The value is chosen to sit inside the container stop grace period.
|
||
Docker's default `docker stop` grace is 10 seconds and the Dockerfile
|
||
sets no `STOPSIGNAL` or grace override, so the process must be gone
|
||
before that. fx's own default is 15 seconds, which is past the grace:
|
||
the container would be SIGKILLed (exit 137) before the bound could
|
||
fire, and nothing that depends on it — including the
|
||
`shutdown timed out, goroutines still running` error log that tells
|
||
an operator a component is wedged — would ever be reached.
|
||
|
||
Two operational consequences follow from bounding the sequence:
|
||
|
||
- **A wedged component aborts the rest of the shutdown.** fx checks
|
||
the stop context before each remaining hook and returns outright
|
||
once it has expired, skipping the hooks it has not reached. If the
|
||
first-stopped component consumes the whole budget, the later hooks
|
||
never run — **the database close among them**. SQLite is crash-safe,
|
||
so this is not corruption, but it is not a clean close either.
|
||
- **Lowering the grace below 5 seconds reintroduces the silent
|
||
truncation.** `docker stop --time`, Compose's `stop_grace_period`,
|
||
or Kubernetes' `terminationGracePeriodSeconds` set under 5 seconds
|
||
put SIGKILL back in front of the bound, and the process dies with
|
||
no shutdown diagnostics at all. Keep the deployment's grace above
|
||
the stop timeout.
|
||
|
||
### Linting
|
||
|
||
golangci-lint never runs on the host. `script/lint` builds
|
||
`Dockerfile.lint`, which copies the repo into the digest-pinned
|
||
golangci-lint image and lints as a build step, so a successful build is
|
||
a clean lint. A host binary would share one cache and one lock with
|
||
every other checkout on the machine, which has produced both invented
|
||
findings attributed to other worktrees and unearned passes.
|
||
|
||
Three properties are load-bearing:
|
||
|
||
- `script/lint` passes `--no-cache-filter=lint`. Without it an unchanged
|
||
tree replays the lint layer from cache and the build exits 0 in under
|
||
a second having linted nothing. The `deps` stage stays cacheable, so
|
||
module downloads are not repeated. Invalidation is scoped to the one
|
||
stage; never prune the shared build cache.
|
||
- `script/lint` does not trust that flag. Docker silently ignores
|
||
`--no-cache-filter` for a stage name that does not match, so a stage
|
||
rename or a one-character typo would restore the cached false green
|
||
with no warning and a fast exit 0. The script therefore tees the
|
||
build output and treats a run as a pass only if golangci-lint's own
|
||
summary line (`N issues.` / `N issues:`) appears in it: no summary,
|
||
no lint, whatever the exit code says.
|
||
- Both lint steps use `RUN --network=none`. `golangci-lint config
|
||
verify` is documented as fetching its JSON schema over HTTPS, which
|
||
would be an unpinned remote dependency; the pinned image resolves the
|
||
schema without network access, and `--network=none` enforces that
|
||
instead of trusting it. Verify is worth keeping because
|
||
`golangci-lint run` silently ignores config keys it does not
|
||
recognize, so a typo would disable a setting with no warning.
|
||
|
||
### Docker
|
||
|
||
The Dockerfile uses a three-stage build. Each stage is pinned by
|
||
digest, and the two check stages are separate images so the linter's
|
||
version is fixed independently of the compiler's:
|
||
|
||
1. **Lint stage** (`golangci/golangci-lint:v2.12.2`, Debian-based) —
|
||
installs `make`, downloads dependencies, copies the source, and runs
|
||
`make fmt-check`, then `golangci-lint config verify` and
|
||
`golangci-lint run`, both with `--network=none`.
|
||
2. **Builder stage** (`golang:1.26.1-bookworm`) — depends on the lint
|
||
stage passing (it copies a file from it), runs `script/fetch-assets`
|
||
to download and verify the third-party browser assets, then runs
|
||
`make test` and `make build`, and finally rebuilds the binary with
|
||
`CGO_ENABLED=1` and static linking so it runs on musl.
|
||
3. **Runtime stage** (`alpine:3.21`) — copies the static binary,
|
||
creates the `/var/lib/webhooker` directory for all SQLite databases,
|
||
runs as the non-root `webhooker` user (UID 1000), exposes port 8080,
|
||
and includes a health check against `/.well-known/healthcheck`.
|
||
|
||
The lint stage invokes `golangci-lint` directly rather than `make lint`:
|
||
it is already the pinned linter image, and `make lint` builds
|
||
`Dockerfile.lint`, which would need a docker daemon inside this build.
|
||
|
||
Both check stages use Debian rather than Alpine because
|
||
`gorm.io/driver/sqlite` pulls in `mattn/go-sqlite3`, which needs CGO
|
||
and does not compile against musl. Only the final binary is statically
|
||
linked, which is what lets it run on the Alpine runtime image.
|
||
|
||
`script/cibuild` — `docker build .` — is the CI gate: the checks run
|
||
inside the image, so a build that succeeds is a repo that is formatted,
|
||
linted, tested and compiled. `script/lint` also uses Docker
|
||
(`Dockerfile.lint`, see Linting above), so `make lint` and `make check`
|
||
run the same pinned linter version the gate does; only `script/test`
|
||
and `script/fmt-check` run on the host.
|
||
|
||
#### CI gate honesty
|
||
|
||
A layer cache lets `docker build .` exit 0 in seconds with the lint and
|
||
test stages replayed rather than executed, which would make a green
|
||
check meaningless. The `check` workflow therefore writes
|
||
`.ci-fingerprint` into the build context before building. Its value is
|
||
the hash of the last commit that touched the build context, so:
|
||
|
||
- Any commit that changes code (including a squash merge whose tree
|
||
matches an already-built branch) gets a new fingerprint, invalidates
|
||
the `COPY . .` layer of both check stages, and really runs
|
||
`make fmt-check`, `golangci-lint`, `make test`, and `make build`. A
|
||
run that reports success ran them.
|
||
- A docs-only commit leaves the fingerprint unchanged — `.dockerignore`
|
||
excludes `*.md`, `LICENSE` and `.editorconfig` from the context
|
||
anyway — so the image replays from cache and costs seconds.
|
||
|
||
The module download layer sits above `COPY . .` and stays cached either
|
||
way.
|
||
|
||
A separate workflow step, run before the fingerprint is written, covers
|
||
a second way the gate lied: Gitea cancels an in-flight run when a newer
|
||
commit lands on the same branch and records that cancellation as a
|
||
`failure` status, so a commit nothing ever tested reads as a test
|
||
result. Cancellation is unconditional server-side for push events, so
|
||
the superseding run calls `script/ci-mark-superseded`, which rewrites
|
||
that exact status to `failure` /
|
||
`Superseded by a newer commit; never tested`.
|
||
|
||
The state stays `failure` on purpose: Gitea's combined status folds
|
||
`skipped` into `success`, so marking a never-tested commit `skipped`
|
||
made the status API report green for it, indistinguishable from a commit
|
||
that passed. Reading a commit's status on this repo therefore goes:
|
||
|
||
- `success` / `Successful in ...` — the checks ran and passed.
|
||
- `failure` / `Failing after ...` — the checks ran and failed.
|
||
- `failure` / `Superseded by a newer commit; never tested` — the run was
|
||
cancelled, by a newer push or by hand, and nothing was verified about
|
||
this commit. Test the commit itself before concluding anything about
|
||
it.
|
||
|
||
Genuine failures and successes are never touched, and no status is left
|
||
`pending`, which would block the commit indefinitely. The step derives
|
||
its context string from the workflow name, the job **id** and the event.
|
||
That is deliberately not byte-identical to Gitea's own rule, which uses
|
||
the job's display `name:` where the runner exports the id, so giving the
|
||
job a `name:` — or renaming the workflow — makes the derived context
|
||
stop matching. The step fails loudly when no status on the commit
|
||
carries that context, so no rename can silently disable the rewrite.
|
||
|
||
## TODO
|
||
|
||
See [TODO.md](TODO.md).
|
||
|
||
## License
|
||
|
||
MIT
|
||
|
||
## Author
|
||
|
||
[@sneak](https://sneak.berlin)
|