1991 lines
106 KiB
Markdown
1991 lines
106 KiB
Markdown
# webhooker
|
||
|
||
webhooker is a self-hosted webhook proxy and store-and-forward service
|
||
written in [Go](https://golang.org) by
|
||
[@sneak](https://sneak.berlin). It receives webhooks from external
|
||
services, durably stores them, and delivers them to configured targets
|
||
with retry support, logging, and observability. Category: infrastructure
|
||
/ web service. License: MIT.
|
||
|
||
## Getting Started
|
||
|
||
### Prerequisites
|
||
|
||
- Go 1.26.1+ (the version in `go.mod`)
|
||
- Docker (for linting, for the test stage of the CI gate, and for
|
||
containerized deployment)
|
||
- `curl`, used by `script/fetch-assets` to download the third-party
|
||
browser assets, which are not committed (`make bootstrap` installs
|
||
it if missing)
|
||
|
||
golangci-lint is not a prerequisite and must not be installed on the
|
||
host: `script/bootstrap` does not install it, and `make lint` runs the
|
||
digest-pinned linter image via `Dockerfile.lint`.
|
||
|
||
### Quick Start
|
||
|
||
```bash
|
||
# Clone the repo
|
||
git clone https://git.eeqj.de/sneak/webhooker.git
|
||
cd webhooker
|
||
|
||
# Install Go dependencies and the third-party browser assets.
|
||
# `make deps` alone is not enough: it only runs go mod download/tidy,
|
||
# and the checks below need the fetched assets.
|
||
make bootstrap
|
||
|
||
# Run all checks (test, lint, format check)
|
||
make check
|
||
|
||
# Run in development mode. DATA_DIR defaults to /var/lib/webhooker in
|
||
# every environment, so set it (in .env or the shell) to a writable
|
||
# directory when running from a clone.
|
||
DATA_DIR=./data make dev
|
||
|
||
# Build Docker image
|
||
make docker
|
||
```
|
||
|
||
### Development Commands
|
||
|
||
```bash
|
||
make bootstrap # Install all dependencies (idempotent)
|
||
make setup # Bootstrap + install git pre-commit hook
|
||
make assets # Fetch + verify third-party browser assets
|
||
make fmt # Format code (gofmt + goimports)
|
||
make fmt-check # Fail if gofmt would change anything (writes nothing)
|
||
make lint # Run golangci-lint in Docker (Dockerfile.lint)
|
||
make test # Run tests with race detection
|
||
make check # test + lint + fmt-check (CI gate)
|
||
make build # Build binary to bin/webhooker
|
||
make run # build, then run ./bin/webhooker
|
||
make dev # go run ./cmd/webhooker
|
||
make deps # go mod download + go mod tidy
|
||
make docker # Build Docker image
|
||
make hooks # Install git pre-commit hook that runs script/precommit
|
||
make css # Regenerate static/css/tailwind.css (needs tailwindcss)
|
||
make clean # Remove bin/
|
||
```
|
||
|
||
### Configuration
|
||
|
||
All configuration is via environment variables. For local development,
|
||
you can place variables in a `.env` file in the project root (loaded
|
||
automatically via `godotenv/autoload`).
|
||
|
||
The environment is selected by setting `WEBHOOKER_ENVIRONMENT` to `dev`
|
||
or `prod` (default: `dev`). The setting controls several behaviors:
|
||
|
||
| Behavior | `dev` | `prod` |
|
||
| --------------------- | -------------------------------- | ------------------------------- |
|
||
| CORS | Allows any origin (`*`) | Disabled (no-op) |
|
||
| Session cookie Secure | `false` (works over plain HTTP) | `true` (requires HTTPS) |
|
||
|
||
The CSRF cookie's `Secure` flag and Origin/Referer validation mode are
|
||
determined per-request based on the actual transport protocol, not the
|
||
environment setting. The middleware checks `r.TLS` (direct TLS) and the
|
||
`X-Forwarded-Proto` header (TLS-terminating reverse proxy) to decide:
|
||
|
||
- **Direct TLS or `X-Forwarded-Proto: https`**: Secure cookies, strict
|
||
Origin/Referer validation.
|
||
- **Plaintext HTTP**: Non-Secure cookies, relaxed Origin/Referer
|
||
checks (token validation still enforced).
|
||
|
||
This means CSRF protection works correctly in all deployment scenarios:
|
||
behind a TLS-terminating reverse proxy, with direct TLS, or over plain
|
||
HTTP during development. When running behind a reverse proxy, ensure it
|
||
sets the `X-Forwarded-Proto: https` header.
|
||
|
||
All other differences (log format, security headers, etc.) are
|
||
independent of the environment setting — log format is determined by
|
||
TTY detection, and security headers are always applied.
|
||
|
||
| Variable | Description | Default |
|
||
| ----------------------- | ----------------------------------- | -------- |
|
||
| `WEBHOOKER_ENVIRONMENT` | `dev` or `prod` | `dev` |
|
||
| `PORT` | HTTP listen port | `8080` |
|
||
| `DATA_DIR` | Directory for all SQLite databases | `/var/lib/webhooker` |
|
||
| `DEBUG` | Enable debug logging | `false` |
|
||
| `MAINTENANCE_MODE` | Report `maintenanceMode: true` in the healthcheck JSON. It does not change how any request is served — no maintenance page exists | `false` |
|
||
| `METRICS_USERNAME` | Basic auth username for `/metrics` | `""` |
|
||
| `METRICS_PASSWORD` | Basic auth password for `/metrics` | `""` |
|
||
| `SENTRY_DSN` | Sentry error reporting DSN | `""` |
|
||
| `RETENTION_SWEEP_INTERVAL` | How often the retention reaper and archive sweeper run (Go duration, must be positive) | `1h` |
|
||
| `SESSION_IDLE_TIMEOUT` | Idle session timeout (Go duration) | `24h` |
|
||
| `RECEIVER_RATE_LIMIT` | Receiver requests/minute per IP per entrypoint (10x that per IP across the route) | `120` |
|
||
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted (unset: all clients behind a proxy share one rate-limit bucket; a correct login password is never throttled either way) | `""` (none) |
|
||
|
||
#### Trusted proxies
|
||
|
||
`TRUSTED_PROXIES` is a comma-separated list of CIDR blocks (a bare
|
||
address such as `192.168.1.7` is accepted and treated as a single
|
||
host), for example `192.168.1.7, 2001:db8::5`. It decides whose
|
||
`X-Forwarded-For` header the rate limiters believe, so it should name
|
||
the addresses of your reverse proxies and nothing else.
|
||
|
||
`X-Forwarded-For` is honoured **only** when the connecting peer is
|
||
inside one of these blocks; for every other peer the client identity is
|
||
the connection's own address and the header is ignored. The default is
|
||
the empty list, which trusts nobody — anything else would let any
|
||
client pick its own rate limit bucket, minting a fresh one per request
|
||
or draining someone else's. Set it to the address of your reverse
|
||
proxy, and to nothing wider. A set but unparseable value aborts
|
||
startup.
|
||
|
||
That default is safe against forged headers, but leaving it unset in
|
||
production has a cost you must know about. Production runs behind a
|
||
TLS-terminating reverse proxy, so with `TRUSTED_PROXIES` unset every
|
||
request keys on the proxy's own address and all clients share a single
|
||
bucket per limit. The receiver limits become service-wide ceilings,
|
||
and the login endpoint's failure counting collapses onto one key, so a
|
||
stranger's wrong passwords throttle every other client's wrong
|
||
passwords.
|
||
|
||
What it cannot do is lock the operator out. The login endpoint
|
||
verifies credentials **before** it consults any limit and charges only
|
||
failures, so a correct password is never throttled no matter how full
|
||
the bucket is. See [Rate Limiting](#rate-limiting).
|
||
|
||
The remedy is to set `TRUSTED_PROXIES` to your reverse proxy's
|
||
address, which restores per-client buckets. webhooker logs a warning
|
||
at startup whenever `TRUSTED_PROXIES` is empty, in every environment —
|
||
not only when `WEBHOOKER_ENVIRONMENT=prod`, because that variable
|
||
defaults to `dev` and an operator who never set it is precisely the
|
||
one at risk. The warning is informational when nothing proxies to the
|
||
process: with no proxy in front, the peer address is the client's own
|
||
and the buckets are already per-client. See
|
||
[Rate Limiting](#rate-limiting) for what each limit shares.
|
||
|
||
`X-Real-IP` and `True-Client-IP` are **never** read, from any peer.
|
||
Reverse proxies append to `X-Forwarded-For` but forward other client
|
||
headers verbatim, so a single-valued header is client-controlled even
|
||
behind a trusted proxy.
|
||
|
||
Within a trusted request, `X-Forwarded-For` is read right to left,
|
||
because the rightmost entry is the one the nearest proxy appended and
|
||
everything left of it may have been written by the client. The first
|
||
hop that is not itself a trusted proxy is taken as the client. A hop
|
||
that is not a bare IP address — `ip:port`, a bracketed IPv6 literal,
|
||
the token `unknown` — ends the walk and the peer address is used
|
||
instead, since past such an entry the chain is not the shape assumed
|
||
here. The peer address is likewise used when the header is absent or
|
||
every hop in it is a trusted proxy.
|
||
|
||
Two operator requirements follow:
|
||
|
||
- Your proxy must **append** the peer address to `X-Forwarded-For`
|
||
(nginx `$proxy_add_x_forwarded_for`, HAProxy `option forwardfor`,
|
||
Caddy and AWS ALB by default), and must append a bare address with
|
||
no port.
|
||
- List proxy hosts **only**. Any address inside `TRUSTED_PROXIES`
|
||
chooses its own rate-limit key: its `X-Forwarded-For` is walked, so
|
||
it can name a different address on every request to get a fresh
|
||
bucket each time, or name another client's address to drain that
|
||
client's bucket. Never list a block that also covers clients — a
|
||
broad `10.0.0.0/8` on a network where clients live in the same range
|
||
makes all three limits, including the unauthenticated webhook
|
||
receiver, silently bypassable by every client in the block.
|
||
|
||
#### Sessions
|
||
|
||
Sessions are bounded by two independent clocks, and end at whichever
|
||
one runs out first:
|
||
|
||
- **Idle expiry** (`SESSION_IDLE_TIMEOUT`, default `24h`) is a sliding
|
||
window. Every authenticated request pushes it forward, so a session
|
||
in continuous use never hits it, while an abandoned one expires a day
|
||
after its last use. Any non-positive value (`0`, or a negative
|
||
duration such as `-1s`) disables idle expiry entirely; the absolute
|
||
cap below still applies. A set-but-unparseable value aborts startup
|
||
rather than silently falling back to the default.
|
||
- **Absolute expiry** is a fixed 7 days from login. Activity does
|
||
**not** extend it: after a week, every session ends and the user
|
||
authenticates again.
|
||
|
||
Only requests that authenticate with the session count as activity, so
|
||
an unauthenticated request carrying the cookie cannot keep a session
|
||
alive. The idle timestamp is rewritten at most once per tenth of the
|
||
idle window rather than on every request, which means a session may
|
||
expire up to 10% early relative to the user's true last request, but
|
||
never late.
|
||
|
||
Both clocks are anchored by timestamps stored in the session cookie.
|
||
Sessions issued before this feature existed carry neither, so they are
|
||
treated as expired: upgrading to a build that has it logs every
|
||
existing session out once, and those users sign in again.
|
||
|
||
#### Invalid values abort startup
|
||
|
||
The defaults above apply **only** to variables that are unset (or set
|
||
to an empty string). A variable that is set but cannot be parsed is a
|
||
fatal configuration error: webhooker logs the offending variable and
|
||
its value and refuses to start, rather than silently running with a
|
||
substituted default. `PORT=eighty`, `DEBUG=ture`, and
|
||
`RETENTION_SWEEP_INTERVAL=1 hour` all abort startup. `PORT` must
|
||
additionally be a number in the range 1–65535,
|
||
`RECEIVER_RATE_LIMIT` must be at least 1,
|
||
`RETENTION_SWEEP_INTERVAL` must be greater than zero (it is a ticker
|
||
period, so `0s` or a negative value would crash the reaper after
|
||
startup), and every entry in `TRUSTED_PROXIES` must be a CIDR block or
|
||
a bare IP address. `SESSION_IDLE_TIMEOUT` is the exception: a
|
||
non-positive value there means idle expiry is disabled, not invalid.
|
||
|
||
Boolean variables (`DEBUG`, `MAINTENANCE_MODE`) accept exactly the
|
||
spellings Go's `strconv.ParseBool` accepts — `1`, `t`, `T`, `TRUE`,
|
||
`true`, `True`, `0`, `f`, `F`, `FALSE`, `false`, `False` — and nothing
|
||
else. `yes`, `on`, and `off` are rejected rather than quietly treated
|
||
as false.
|
||
|
||
On first startup, webhooker automatically generates a cryptographically
|
||
secure session encryption key and stores it in the database. This key
|
||
persists across restarts — no manual key management is needed.
|
||
|
||
On first startup, webhooker creates an `admin` user
|
||
with a randomly generated password and logs it to stdout. This password
|
||
is only displayed once.
|
||
|
||
### Running with Docker
|
||
|
||
```bash
|
||
docker run -d \
|
||
-p 8080:8080 \
|
||
-v /path/to/data:/var/lib/webhooker \
|
||
-e WEBHOOKER_ENVIRONMENT=prod \
|
||
webhooker:latest
|
||
```
|
||
|
||
The container runs as a non-root user (`webhooker`, UID 1000), exposes
|
||
port 8080, and includes a health check against
|
||
`/.well-known/healthcheck`. The `/var/lib/webhooker` volume holds all
|
||
SQLite databases: the main application database (`webhooker.db`), the
|
||
per-webhook event databases (`events-{uuid}.db`), and any archive
|
||
databases written by `database` targets (`archive-{uuid}.db`). Mount
|
||
this as a persistent volume to preserve data across container
|
||
restarts.
|
||
|
||
## Entrypoints
|
||
|
||
This repository adheres to the
|
||
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
||
standard: normalized scripts in `script/` are the entrypoints for the
|
||
development workflow. Ten of the Makefile's sixteen targets are thin
|
||
shims that call them; `build`, `run`, `dev`, `deps`, `clean` and `css`
|
||
are inline commands with no script behind them. We provide:
|
||
|
||
- `script/bootstrap` — install all dependencies (idempotent)
|
||
- `script/setup` — make a fresh clone ready for development
|
||
(bootstrap, then install-precommit)
|
||
- `script/projectname` — output the project name ("webhooker")
|
||
- `script/fetch-assets` — download the third-party browser assets into
|
||
`static/`, verifying each against its pinned sha256
|
||
- `script/test` — run the test suite
|
||
- `script/lint` — run golangci-lint in Docker (see Linting below)
|
||
- `script/fmt` — format all code (writes)
|
||
- `script/fmt-check` — check formatting (read-only)
|
||
- `script/check` — run test, lint, and fmt-check
|
||
- `script/docker` — build the Docker image tagged via `script/projectname`
|
||
- `script/cibuild` — CI entrypoint: `docker build .` (the Dockerfile
|
||
runs the checks, so a green build implies a green repo)
|
||
- `script/ci-mark-superseded` — CI helper: mark the commits whose run a
|
||
newer push cancelled (see [CI gate honesty](#ci-gate-honesty))
|
||
- `script/precommit` — pre-commit checks (`go mod tidy` guard, then
|
||
`script/check`)
|
||
- `script/install-precommit` — install the git pre-commit hook that
|
||
runs `script/precommit`
|
||
|
||
## Third-party browser assets
|
||
|
||
The web UI serves one third-party script, Alpine.js. It is **not** committed:
|
||
a minified bundle in the tree is unreviewable, and `REPO_POLICIES.md` bars
|
||
both committed build artifacts and unpinned external references.
|
||
|
||
Instead `script/fetch-assets` downloads it from a pinned URL, checks the
|
||
download against a hardcoded sha256, and installs it under `static/`. The
|
||
sha256 of every installed asset is recorded in `static/vendor.sha256`, and
|
||
`static/vendor_test.go` re-hashes the bytes `go:embed` put in the binary
|
||
against that manifest — so the pin is enforced on what actually ships, not
|
||
merely written down. Any mismatch fails the build.
|
||
|
||
`make bootstrap` runs the fetch for local development, and the Dockerfile
|
||
runs it in the build stage; `.gitignore` and `.dockerignore` keep the
|
||
artifact out of both the repo and the build context.
|
||
|
||
To move to a new version: update the version, URL, and tarball sha256 in
|
||
`script/fetch-assets` and the asset sha256 in `static/vendor.sha256`, then
|
||
run `make assets && make check`.
|
||
|
||
## Rationale
|
||
|
||
Webhook integrations between services are inherently fragile. The
|
||
receiving service must be online when the webhook fires, most webhook
|
||
senders provide no built-in retry mechanism, and there is no standard
|
||
way to inspect what was sent, when it was sent, or whether delivery
|
||
succeeded.
|
||
|
||
webhooker solves this by acting as a durable intermediary:
|
||
|
||
1. **Reliable ingestion** — webhooker is always ready to accept incoming
|
||
webhooks. It stores every received event before attempting any
|
||
delivery, so nothing is lost if downstream targets are unavailable.
|
||
|
||
2. **Guaranteed delivery** — Events are queued for delivery to each
|
||
configured target. Failed deliveries are retried with configurable
|
||
backoff. Every delivery attempt is logged with status codes, response
|
||
bodies, and timing.
|
||
|
||
3. **Observability** — Full request/response logging for every webhook
|
||
received and every delivery attempted. Prometheus metrics expose
|
||
volume, latency, and error rates. The web UI provides real-time
|
||
visibility into event flow.
|
||
|
||
4. **Fan-out** — A single incoming webhook can be delivered to multiple
|
||
targets simultaneously. This enables patterns like forwarding a
|
||
GitHub webhook to both a deployment service and a Slack channel.
|
||
|
||
5. **Replay** (not yet implemented) — Every received event is stored in
|
||
full, which is what manual redelivery for debugging or testing will
|
||
be built on. No redelivery exists today, in the web UI or the API;
|
||
see [TODO.md](TODO.md).
|
||
|
||
### Use Cases
|
||
|
||
- **Store-and-forward** with configurable retries for unreliable
|
||
receivers
|
||
- **Observability** via Prometheus metrics on webhook frequency, payload
|
||
size, and delivery performance
|
||
- **Debugging** and introspection of webhook payloads in the web UI
|
||
- **Replay** of webhook events for application testing and development
|
||
(planned; not yet implemented)
|
||
- **Fan-out** delivery of a single webhook to multiple downstream
|
||
targets
|
||
- **High-availability ingestion** for delivery to less reliable backend
|
||
systems
|
||
|
||
## Design
|
||
|
||
### Architecture Overview
|
||
|
||
webhooker is structured as a standard Go HTTP server following the
|
||
[sneak/prompts GO_HTTP_SERVER_CONVENTIONS](https://git.eeqj.de/sneak/prompts/src/branch/main/prompts/GO_HTTP_SERVER_CONVENTIONS.md).
|
||
It uses:
|
||
|
||
- **[Uber fx](https://go.uber.org/fx)** for dependency injection and
|
||
lifecycle management
|
||
- **[go-chi](https://github.com/go-chi/chi)** for HTTP routing
|
||
- **[GORM](https://gorm.io)** for database access with
|
||
**[modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite)** as
|
||
the runtime SQLite driver. Note: `gorm.io/driver/sqlite` transitively
|
||
depends on `mattn/go-sqlite3`, which requires CGO at build time (see
|
||
[Docker](#docker) section)
|
||
- **[slog](https://pkg.go.dev/log/slog)** (stdlib) for structured
|
||
logging with TTY detection (text for dev, JSON for prod)
|
||
- **[gorilla/sessions](https://github.com/gorilla/sessions)** for
|
||
encrypted cookie-based session management
|
||
- **[gorilla/csrf](https://github.com/gorilla/csrf)** for CSRF
|
||
protection (cookie-based double-submit tokens)
|
||
- **[go-chi/httprate](https://github.com/go-chi/httprate)** for
|
||
sliding-window rate limiting of the password-change and webhook
|
||
receiver endpoints. The bucket is per client IP only when
|
||
`TRUSTED_PROXIES` names the reverse proxy; unset, every client
|
||
behind that proxy shares one bucket per limit. The login endpoint
|
||
counts failed attempts itself instead, so that a correct password is
|
||
never throttled (see [Rate Limiting](#rate-limiting))
|
||
- **[Prometheus](https://prometheus.io)** for metrics, served at
|
||
`/metrics` behind basic auth
|
||
- **[Sentry](https://sentry.io)** for optional error reporting
|
||
|
||
### Naming Conventions
|
||
|
||
The codebase uses consistent naming throughout (rename completed in
|
||
[issue #12](https://git.eeqj.de/sneak/webhooker/issues/12)):
|
||
|
||
| Entity | Description |
|
||
| ---------------- | ----------- |
|
||
| **Webhook** | Top-level configuration entity grouping entrypoints and targets |
|
||
| **Entrypoint** | A receiver URL where external services POST events |
|
||
| **Target** | A delivery destination for events |
|
||
|
||
### Data Model
|
||
|
||
webhooker's data model has nine entities organized into two tiers: the
|
||
**application tier** (user and webhook configuration) and the **event
|
||
tier** (event ingestion, delivery, and logging).
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ APPLICATION TIER │
|
||
│ (main application database) │
|
||
│ │
|
||
│ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │
|
||
│ │ User │──1:N──│ Webhook │──1:N──│ Entrypoint │ │
|
||
│ │ │ │ │ │ │ │
|
||
│ │ │ │ │──1:N──│ Target │ │
|
||
│ │ │ └──────────┘ └──────────────┘ │
|
||
│ │ │──1:N──│ APIKey │ │
|
||
│ └──────────┘ └──────────┘ │
|
||
│ │
|
||
│ ┌──────────┐ │
|
||
│ │ Setting │ (key-value application config) │
|
||
│ └──────────┘ │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ EVENT TIER │
|
||
│ (per-webhook dedicated databases) │
|
||
│ │
|
||
│ ┌──────────┐ ┌──────────┐ ┌─────────────────┐ │
|
||
│ │ Event │──1:N──│ Delivery │──1:N──│ DeliveryResult │ │
|
||
│ └──────────┘ └──────────┘ └─────────────────┘ │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
#### Setting
|
||
|
||
A key-value pair for application-level configuration that is
|
||
auto-managed rather than user-provided. Used to store the session
|
||
encryption key and any future auto-generated settings.
|
||
|
||
| Field | Type | Description |
|
||
| ------- | ------ | ----------- |
|
||
| `key` | string | Primary key (setting name) |
|
||
| `value` | text | Setting value |
|
||
|
||
Currently stored settings:
|
||
|
||
- **`session_key`** — Base64-encoded 32-byte session encryption key,
|
||
auto-generated on first startup.
|
||
|
||
#### User
|
||
|
||
A registered user of the webhooker service.
|
||
|
||
| Field | Type | Description |
|
||
| ---------- | -------- | ----------- |
|
||
| `id` | UUID | Primary key |
|
||
| `username` | string | Unique login name |
|
||
| `password` | string | Argon2id hash (never exposed via API) |
|
||
|
||
**Relations:** Has many Webhooks. Has many APIKeys.
|
||
|
||
Passwords are hashed with Argon2id using secure defaults (64 MB memory,
|
||
1 iteration, 4 threads, 32-byte key, 16-byte salt). On first startup,
|
||
an `admin` user is created with a randomly generated 16-character
|
||
password logged to stdout.
|
||
|
||
#### Webhook
|
||
|
||
The top-level configuration entity. A webhook groups together one or
|
||
more entrypoints (receiver URLs) and one or more targets (delivery
|
||
destinations) into a logical unit. A user creates a webhook to set up
|
||
event routing.
|
||
|
||
| Field | Type | Description |
|
||
| ---------------- | ------- | ----------- |
|
||
| `id` | UUID | Primary key |
|
||
| `user_id` | UUID | Foreign key → User |
|
||
| `name` | string | Human-readable name |
|
||
| `description` | string | Optional description |
|
||
| `retention_days` | integer | Days to retain events (default: 30; 0 means retain forever) |
|
||
|
||
**Relations:** Belongs to User. Has many Entrypoints. Has many Targets.
|
||
|
||
The `retention_days` field controls how long event data is kept in the
|
||
webhook's dedicated database before automatic cleanup.
|
||
|
||
Setting `retention_days` to `0` means "retain events forever". Because
|
||
the column carries a default of 30, a literal zero cannot survive an
|
||
insert, so a zero is rewritten on save to a sentinel of `365 * 1000`
|
||
days (`database.RetentionForeverDays`). The retention reaper recognises
|
||
that sentinel and skips the webhook entirely, and the web UI displays
|
||
such a webhook's retention as "forever" rather than as a day count.
|
||
|
||
Submitted `retention_days` values therefore fall into three bands, not
|
||
two:
|
||
|
||
- `1` up to `database.MaxFiniteRetentionDays` (106751 days, about 292
|
||
years) is accepted as a finite retention.
|
||
- Above that ceiling but below the retain-forever sentinel of 365000
|
||
(`database.RetentionForeverDays`) is rejected with a 400. This is the
|
||
band the cap exists for.
|
||
- `0`, and `365000` or above, are accepted and mean retain forever,
|
||
collapsing to the sentinel — `0` in `Webhook.BeforeSave`, the large
|
||
values in `parseRetentionDays`. The large values are not out of
|
||
range: the edit form pre-fills the sentinel for a retain-forever
|
||
webhook, so submitting that form back unchanged has to keep meaning
|
||
"forever".
|
||
|
||
A negative value is in none of the three: `parseRetentionDays` rejects
|
||
it with a 400 before `BeforeSave` ever sees it.
|
||
|
||
The cap is not arbitrary: the reaper computes its cutoff as a
|
||
`time.Duration`, an int64 nanosecond count, and a longer period
|
||
overflows it. An overflowed cutoff lands in the future, where it
|
||
matches every row, so the sweep would delete every event the webhook
|
||
has instead of none. The reaper also clamps the value it is given, so a
|
||
row written by an older version cannot trigger that either.
|
||
|
||
#### Entrypoint
|
||
|
||
A receiver URL where external services POST webhook events. Each
|
||
entrypoint has a unique UUID-based path.
|
||
When an HTTP request arrives at an entrypoint's path, webhooker captures
|
||
the full request and creates an Event.
|
||
|
||
| Field | Type | Description |
|
||
| -------------- | ------- | ----------- |
|
||
| `id` | UUID | Primary key |
|
||
| `webhook_id` | UUID | Foreign key → Webhook |
|
||
| `path` | string | Unique bare UUID, generated at creation. The `/webhook/` prefix is route only and is not stored: the receiver matches this column against the raw `{uuid}` path segment |
|
||
| `description` | string | Optional description |
|
||
| `active` | boolean | Whether this entrypoint accepts events (default: true) |
|
||
|
||
**Relations:** Belongs to Webhook.
|
||
|
||
A webhook can have multiple entrypoints. This allows separate URLs for
|
||
different event sources that all feed into the same processing pipeline
|
||
(e.g., one entrypoint for GitHub, another for Stripe, both routing to
|
||
the same targets).
|
||
|
||
#### Target
|
||
|
||
A delivery destination for events. Each target defines where and how
|
||
events should be forwarded.
|
||
|
||
| Field | Type | Description |
|
||
| ---------------- | ---------- | ----------- |
|
||
| `id` | UUID | Primary key |
|
||
| `webhook_id` | UUID | Foreign key → Webhook |
|
||
| `name` | string | Human-readable name |
|
||
| `type` | TargetType | One of: `http`, `slack`, `database`, `log` |
|
||
| `active` | boolean | Whether deliveries are enabled (default: true) |
|
||
| `config` | JSON text | Type-specific configuration |
|
||
| `max_retries` | integer | Maximum retry attempts for `http` and `slack` targets (0 = fire-and-forget, >0 = retries with backoff and a circuit breaker). Ignored by `database` and `log` targets |
|
||
| `max_queue_size` | integer | Stored and shown on the target's detail view, but not enforced anywhere yet: nothing in the delivery engine consults it. Queue depth is set by the two fixed 10,000-entry channels |
|
||
|
||
**Relations:** Belongs to Webhook. Has many Deliveries.
|
||
|
||
**Target types:**
|
||
|
||
- **`http`** — Forward the event as an HTTP POST to a configured URL.
|
||
Behavior depends on `max_retries`: when `max_retries` is 0 (the
|
||
default), the target operates in fire-and-forget mode — a single
|
||
attempt with no retries and no circuit breaker. When `max_retries` is
|
||
greater than 0, failed deliveries are retried with exponential backoff
|
||
up to `max_retries` attempts, protected by a per-target circuit
|
||
breaker.
|
||
- **`slack`** — Post the event as a formatted message to a
|
||
Slack-compatible incoming webhook URL (`webhookUrl` in `config`). It
|
||
is built on the same HTTP core as `http` and honours `max_retries`
|
||
identically, circuit breaker included. See the Slack target section
|
||
under "Per-Webhook Event Databases" for the message format.
|
||
- **`database`** — Archive the full event as a row into a separate
|
||
per-webhook archive database (`archive-{webhookID}.db`) for long-term
|
||
retention, with an optional creation-validated expiry (default: keep
|
||
forever). No external delivery and no retries; an archive write
|
||
failure fails the delivery. See the database target section under
|
||
"Per-Webhook Event Databases" for the full semantics.
|
||
- **`log`** — Write the event to the application log (stdout). Useful
|
||
for debugging.
|
||
|
||
The `config` field stores type-specific configuration as JSON (e.g.,
|
||
destination URL, custom headers, timeout settings).
|
||
|
||
#### APIKey
|
||
|
||
A programmatic access credential for API authentication.
|
||
|
||
| Field | Type | Description |
|
||
| -------------- | --------- | ----------- |
|
||
| `id` | UUID | Primary key |
|
||
| `user_id` | UUID | Foreign key → User |
|
||
| `key` | string | Unique API key value |
|
||
| `description` | string | Optional description |
|
||
| `last_used_at` | timestamp | Last time this key was used (nullable) |
|
||
|
||
**Relations:** Belongs to User.
|
||
|
||
#### Event
|
||
|
||
A captured incoming webhook request. Stores the complete HTTP request
|
||
data for auditing and for the planned replay capability.
|
||
|
||
| Field | Type | Description |
|
||
| -------------- | ------ | ----------- |
|
||
| `id` | UUID | Primary key |
|
||
| `webhook_id` | UUID | Foreign key → Webhook |
|
||
| `entrypoint_id` | UUID | Foreign key → Entrypoint |
|
||
| `method` | string | HTTP method of the captured request. Always `POST`: the receiver answers every other method with 405 before an Event is created |
|
||
| `headers` | JSON | Complete request headers |
|
||
| `body` | text | Raw request body |
|
||
| `content_type` | string | Content-Type header value |
|
||
|
||
**Relations:** Belongs to Webhook. Belongs to Entrypoint. Has many
|
||
Deliveries.
|
||
|
||
When a request arrives at an entrypoint, the full request (method,
|
||
headers, body) is captured as an Event. The event is then queued for
|
||
delivery to every active target configured on the parent webhook.
|
||
|
||
#### Delivery
|
||
|
||
The pairing of an event with a target. Tracks the overall delivery
|
||
status across potentially multiple attempts.
|
||
|
||
| Field | Type | Description |
|
||
| ---------- | -------------- | ----------- |
|
||
| `id` | UUID | Primary key |
|
||
| `event_id` | UUID | Foreign key → Event |
|
||
| `target_id`| UUID | Foreign key → Target |
|
||
| `status` | DeliveryStatus | One of: `pending`, `delivered`, `failed`, `retrying` |
|
||
|
||
**Relations:** Belongs to Event. Belongs to Target. Has many
|
||
DeliveryResults.
|
||
|
||
**Delivery statuses:**
|
||
|
||
- **`pending`** — Created but not yet attempted.
|
||
- **`retrying`** — At least one attempt failed; more attempts remain.
|
||
- **`delivered`** — Successfully delivered (at least one attempt
|
||
succeeded).
|
||
- **`failed`** — All retry attempts exhausted without success.
|
||
|
||
#### DeliveryResult
|
||
|
||
The result of a single delivery attempt. Every attempt (including
|
||
retries) is individually logged for full observability.
|
||
|
||
| Field | Type | Description |
|
||
| --------------- | ------- | ----------- |
|
||
| `id` | UUID | Primary key |
|
||
| `delivery_id` | UUID | Foreign key → Delivery |
|
||
| `attempt_num` | integer | Attempt number (1-based) |
|
||
| `success` | boolean | Whether this attempt succeeded |
|
||
| `status_code` | integer | HTTP response status code (if applicable) |
|
||
| `response_body` | text | Response body (if applicable) |
|
||
| `error` | string | Error message (on failure) |
|
||
| `duration` | integer | Request duration in milliseconds |
|
||
|
||
**Relations:** Belongs to Delivery.
|
||
|
||
#### Common Fields
|
||
|
||
Every entity except `Setting` includes these fields from `BaseModel`.
|
||
`Setting` is a bare key-value row with no `id`, no timestamps and no
|
||
soft delete:
|
||
|
||
| Field | Type | Description |
|
||
| ------------ | --------- | ----------- |
|
||
| `id` | UUID | Auto-generated UUIDv4 primary key |
|
||
| `created_at` | timestamp | Record creation time |
|
||
| `updated_at` | timestamp | Last modification time |
|
||
| `deleted_at` | timestamp | Soft-delete timestamp (nullable; GORM soft deletes) |
|
||
|
||
### Database Architecture
|
||
|
||
#### Per-Webhook Event Databases
|
||
|
||
webhooker uses **separate SQLite database files**: a main application
|
||
database for configuration data and per-webhook databases for event
|
||
storage. All database files live in the `DATA_DIR` directory.
|
||
|
||
**Main Application Database** (`{DATA_DIR}/webhooker.db`) — stores
|
||
configuration and application state:
|
||
|
||
- **Settings** — auto-managed key-value config (e.g. session encryption
|
||
key)
|
||
- **Users** — accounts and Argon2id password hashes
|
||
- **Webhooks** — webhook configurations
|
||
- **Entrypoints** — receiver URL definitions
|
||
- **Targets** — delivery destination configurations
|
||
- **APIKeys** — programmatic access credentials
|
||
|
||
On first startup the main database is auto-migrated, a session
|
||
encryption key is generated and stored, and an `admin` user is created.
|
||
|
||
**Per-Webhook Event Databases** (`{DATA_DIR}/events-{webhook_uuid}.db`)
|
||
— each webhook gets its own dedicated SQLite file containing:
|
||
|
||
- **Events** — captured incoming webhook payloads
|
||
- **Deliveries** — event-to-target pairings and their status
|
||
- **DeliveryResults** — individual delivery attempt logs
|
||
|
||
Per-webhook databases are created automatically when a webhook is
|
||
created (and lazily on first access for webhooks that predate this
|
||
feature). They are managed by the `WebhookDBManager` component, which
|
||
handles connection pooling, lazy opening, migrations, and cleanup.
|
||
|
||
This separation provides:
|
||
|
||
- **Isolation** — a high-volume webhook won't cause lock contention or
|
||
journal growth affecting the main application or other webhooks.
|
||
- **Independent lifecycle** — event databases can be independently
|
||
backed up, archived, rotated, or size-limited without impacting the
|
||
application.
|
||
- **Clean deletion** — removing a webhook and all its history is as
|
||
simple as deleting one file. Configuration is soft-deleted in the main
|
||
DB; the event database file is hard-deleted (permanently removed).
|
||
- **Per-webhook retention** — the `retention_days` field on each webhook
|
||
controls automatic cleanup of old events in that webhook's database
|
||
only, or disables cleanup entirely when set to `0` (retain forever).
|
||
- **Performance** — each webhook's database has its own page cache and
|
||
its own lock, so concurrent event ingestion across webhooks won't
|
||
contend. No write-ahead log is involved: both DSNs are
|
||
`file:{path}?cache=shared&mode=rwc` and no `journal_mode` pragma is
|
||
ever issued, so every database runs on SQLite's default rollback
|
||
journal.
|
||
|
||
The **database target type** builds on this architecture to provide
|
||
long-term archiving, separate from the per-webhook event database (which
|
||
may prune events under its own retention). Delivering to a database
|
||
target writes the full event — body, headers, method, content type, and
|
||
webhook/entrypoint/event identifiers — as a row into a dedicated archive
|
||
database, `archive-{webhookID}.db`, stored under the data directory
|
||
beside the event database. After each write the archive handle is closed
|
||
and reopened, debounced to at most once per second, so an operator can
|
||
move the archive file away for offline archiving without stopping the
|
||
service; a moved or removed archive file is recreated automatically on
|
||
the next write. An optional `expiry` in the target's config JSON (e.g.
|
||
`{"expiry":"720h"}`) is validated when the target is created — the
|
||
default (unset or the literal `never`) keeps rows forever — and rows
|
||
older than the expiry are pruned each time the archive is (re)opened. An
|
||
archive write failure is never silent success: the delivery records a
|
||
failed attempt with the error and is marked failed.
|
||
|
||
Because reopens only happen on writes, an archive belonging to a webhook
|
||
that has stopped receiving events would never be pruned. A background
|
||
**archive sweeper** closes that gap: on the same interval as the event
|
||
retention reaper (`RETENTION_SWEEP_INTERVAL`) it prunes every archive
|
||
whose database target declares a positive expiry, whether or not the
|
||
webhook is still receiving traffic. The sweep never creates an archive —
|
||
a webhook whose archive file does not yet exist is skipped, not
|
||
initialised — it takes the same per-webhook lock the write path uses, so
|
||
it can never interleave with a write, and it leaves the archive closed
|
||
afterwards so the move-the-file-away workflow keeps working. Archives
|
||
with no expiry, or the expiry `never`, are not touched by the sweep at
|
||
all.
|
||
|
||
Note that a webhook has one archive file but may carry more than one
|
||
`database` target, each with its own `expiry`. The shortest expiry
|
||
configured on any of them therefore governs the whole archive, and the
|
||
sweep applies it whether or not the webhook is still receiving events.
|
||
Configure a single `database` target per webhook unless you intend that.
|
||
|
||
Deleting a webhook releases its archive: the delivery engine's cached
|
||
archive writer is dropped and its file handle closed, so nothing lingers
|
||
after the webhook is gone. The archive **file itself is deliberately
|
||
left on disk**. Unlike the event database — per-webhook working storage
|
||
that is hard-deleted with the webhook — an archive is long-term storage
|
||
an operator may still want to keep or move away for offline retention,
|
||
and destroying it as a side effect of deleting a webhook would be
|
||
unrecoverable. Removing `archive-{webhookID}.db` is the operator's call.
|
||
Deleting a webhook's last `database` target releases the writer the same
|
||
way, and for the same reason leaves the file alone.
|
||
|
||
The **Slack target type** sends webhook events as formatted messages to
|
||
any Slack-compatible incoming webhook URL (works with Slack, Mattermost,
|
||
and other compatible services). Each message includes event metadata
|
||
(HTTP method, content type, timestamp, body size) and the payload
|
||
pretty-printed in a code block. JSON payloads are automatically
|
||
formatted with indentation for readability; non-JSON payloads are shown
|
||
as raw text. Large payloads are truncated to keep messages reasonable.
|
||
Config stores `webhookUrl` — the Slack/Mattermost incoming webhook
|
||
endpoint. That is the JSON key; the error text for a missing one reads
|
||
`webhook_url is required`, which is the message, not the key.
|
||
|
||
The database uses the
|
||
[modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite) driver at
|
||
runtime, though CGO is required at build time due to the transitive
|
||
`mattn/go-sqlite3` dependency from `gorm.io/driver/sqlite`.
|
||
|
||
### Request Flow
|
||
|
||
```
|
||
External Service
|
||
│
|
||
│ POST /webhook/{uuid}
|
||
▼
|
||
┌─────────────┐ ┌──────────────┐ ┌──────────────┐
|
||
│ chi Router │────►│ Middleware │────►│ Webhook │
|
||
│ │ │ Stack │ │ Handler │
|
||
└─────────────┘ └──────────────┘ └──────┬───────┘
|
||
│
|
||
1. Look up Entrypoint by UUID
|
||
2. Capture full request as Event
|
||
3. Create Delivery records for each active Target
|
||
4. Build self-contained delivery.Task structs
|
||
(target config + event data inline for
|
||
bodies < 16 KiB)
|
||
5. Notify Engine via channel (no DB read needed)
|
||
│
|
||
▼
|
||
┌──────────────┐
|
||
│ Delivery │◄── retry timers
|
||
│ Engine │ (backoff)
|
||
│ (worker │
|
||
│ pool) │
|
||
└──────┬───────┘
|
||
│
|
||
┌── bounded worker pool (N workers) ──┐
|
||
▼ ▼ ▼
|
||
┌────────────┐ ┌────────────┐ ┌────────────┐
|
||
│ HTTP Target│ │ HTTP Target│ │ Log Target │
|
||
│(max_retries│ │(max_retries│ │ (stdout) │
|
||
│ == 0) │ │ > 0, │ └────────────┘
|
||
│ fire+forget│ │ backoff + │
|
||
└────────────┘ │ circuit │
|
||
│ breaker) │
|
||
└────────────┘
|
||
```
|
||
|
||
### Bounded Worker Pool
|
||
|
||
The delivery engine uses a **fixed-size worker pool** (default: 10
|
||
workers) to process all deliveries. At most N deliveries are in-flight
|
||
at any time, preventing goroutine explosions regardless of queue depth.
|
||
|
||
**Architecture:**
|
||
|
||
- **Channels as queues:** Two buffered channels serve as bounded queues:
|
||
a delivery channel (new tasks from the webhook handler) and a retry
|
||
channel (tasks from backoff timers). Both are buffered to 10,000.
|
||
- **Fan-out via channel, not goroutines:** When an event arrives with
|
||
multiple targets, each `delivery.Task` is sent to the delivery channel.
|
||
Workers pick them up and process them — no goroutine-per-target.
|
||
- **Worker goroutines:** A fixed number of worker goroutines select from
|
||
both channels. Each worker processes one task at a time, then picks up
|
||
the next. Workers are the ONLY goroutines doing actual HTTP delivery.
|
||
- **Retry backpressure with DB fallback:** When a retry timer fires and
|
||
the retry channel is full, the timer is dropped — the delivery stays
|
||
in `retrying` status in the database. A periodic sweep (every 60s)
|
||
scans for these "orphaned" retries and re-queues them. No blocked
|
||
goroutines, no unbounded timer chains.
|
||
- **Bounded concurrency:** At most N deliveries (N = number of workers)
|
||
are in-flight simultaneously. Even if a circuit breaker is open for
|
||
hours and thousands of retries queue up in the channels, the workers
|
||
drain them at a controlled rate when the circuit closes.
|
||
|
||
This means:
|
||
|
||
- **No goroutine explosion** — even with 10,000 queued retries, only
|
||
N worker goroutines exist.
|
||
- **Natural backpressure** — if workers are busy, new tasks wait in the
|
||
channel buffer rather than spawning more goroutines.
|
||
- **Independent results** — each worker records its own delivery result
|
||
in the per-webhook database without coordination.
|
||
- **Graceful shutdown** — cancel the context, workers finish their
|
||
current task and exit. The stop hook waits for the pool via
|
||
`lifecycle.WaitForShutdown`, which bounds that wait by fx's stop
|
||
timeout rather than blocking forever on a wedged worker. On timeout
|
||
it logs at `ERROR` and returns an error, and the goroutines that
|
||
did not finish are still running — an unclean shutdown is reported
|
||
rather than hidden.
|
||
|
||
**Recovery paths:**
|
||
|
||
1. **Startup recovery:** When the engine starts, it scans all per-webhook
|
||
databases for `pending` and `retrying` deliveries. Pending deliveries
|
||
are sent to the delivery channel; retrying deliveries get backoff
|
||
timers scheduled.
|
||
2. **Periodic retry sweep (DB-mediated fallback):** Every 60 seconds the
|
||
engine scans for `retrying` deliveries whose backoff period has
|
||
elapsed. This catches "orphaned" retries — ones whose in-memory timer
|
||
was dropped because the retry channel was full. The database is the
|
||
durable fallback that ensures no retry is permanently lost, even under
|
||
extreme backpressure.
|
||
|
||
**Changing a target's type does not migrate in-flight deliveries.** Only
|
||
`http` and `slack` targets own durable retries; `database` and `log`
|
||
targets are fire-and-forget and never produce a `retrying` delivery. If a
|
||
target's `type` is edited from a retrying type to a non-retrying (or
|
||
unknown) one while one of its deliveries is still `retrying`, both
|
||
recovery paths above terminally mark that delivery `failed` and record a
|
||
`DeliveryResult` naming the current target type as the reason, logging it
|
||
at warn level. The delivery is not re-dispatched under the new type — the
|
||
operator never asked for that delivery — and while the event itself
|
||
remains stored in the per-webhook event database, there is no way to
|
||
redeliver it: manual redelivery is planned, not implemented (see
|
||
[TODO.md](TODO.md)).
|
||
|
||
### Circuit Breaker (HTTP and Slack Targets with Retries)
|
||
|
||
`http` and `slack` targets with `max_retries` > 0 are protected by a
|
||
**per-target circuit breaker** that prevents hammering a down target
|
||
with repeated failed delivery attempts. The circuit breaker is
|
||
in-memory only and resets on restart (which is fine — startup recovery
|
||
rescans the database anyway).
|
||
|
||
**States:**
|
||
|
||
| State | Behavior |
|
||
| ----------- | -------- |
|
||
| **Closed** | Normal operation. Deliveries flow through. Consecutive failures are counted. |
|
||
| **Open** | Target appears down. Deliveries are skipped and rescheduled for after the cooldown. |
|
||
| **Half-Open** | Cooldown expired. One probe delivery is allowed to test if the target has recovered. |
|
||
|
||
**Transitions:**
|
||
|
||
```
|
||
success ┌──────────┐
|
||
┌────────────────────► │ Closed │ ◄─── probe succeeds
|
||
│ │ (normal) │
|
||
│ └────┬─────┘
|
||
│ │ N consecutive failures
|
||
│ ▼
|
||
│ ┌──────────┐
|
||
│ │ Open │ ◄─── probe fails
|
||
│ │(tripped) │
|
||
│ └────┬─────┘
|
||
│ │ cooldown expires
|
||
│ ▼
|
||
│ ┌──────────┐
|
||
└──────────────────────│Half-Open │
|
||
│ (probe) │
|
||
└──────────┘
|
||
```
|
||
|
||
**Defaults:**
|
||
|
||
- **Failure threshold:** 5 consecutive failures before opening
|
||
- **Cooldown:** 30 seconds in open state before probing
|
||
|
||
**Scope:** Circuit breakers apply to **`http` and `slack` targets with
|
||
`max_retries` > 0**. The Slack target is built on the same HTTP core
|
||
and hands its own `max_retries` to the same retry path, so it gets a
|
||
breaker with the same 5-failure / 30-second defaults. Fire-and-forget
|
||
targets of either type (`max_retries` == 0), database targets (local
|
||
operations), and log targets (stdout) do not use circuit breakers.
|
||
|
||
When a circuit is open and a new delivery arrives, the engine marks the
|
||
delivery as `retrying` and schedules a retry timer for after the
|
||
remaining cooldown period. This ensures no deliveries are lost — they're
|
||
just delayed until the target is healthy again.
|
||
|
||
### Rate Limiting
|
||
|
||
Global blanket rate limiting middleware (e.g., a per-IP throttle shared
|
||
with the web UI) **must not** apply to webhook receiver endpoints.
|
||
Webhook endpoints receive automated traffic from external services at
|
||
unpredictable rates, and blanket limits shared with other routes would
|
||
cause legitimate deliveries to be dropped.
|
||
|
||
The receiver instead has its own dedicated abuse limit, scoped to the
|
||
`/webhook/{uuid}` route only and keyed per client IP per request path
|
||
(`httprate.KeyByEndpoint`): one misbehaving sender is throttled without
|
||
affecting other senders of the same entrypoint or the same sender's
|
||
other entrypoints. Keying on the path rather than on the entrypoint
|
||
matters — see the aggregate limit below. The limit is
|
||
`RECEIVER_RATE_LIMIT` requests per minute (default 120, generous for
|
||
legitimate webhook senders). Requests over the limit receive HTTP 429
|
||
with a `Retry-After` header. A set-but-invalid `RECEIVER_RATE_LIMIT`
|
||
value aborts startup rather than silently falling back to the default.
|
||
|
||
A second limit sits in front of that one, keyed on the client IP alone
|
||
and covering the whole route at ten times `RECEIVER_RATE_LIMIT` requests
|
||
per minute (default 1200). The per-entrypoint limit needs it: the route
|
||
pattern matches any single path segment, so a client that invents a
|
||
fresh path per request gets a fresh per-entrypoint bucket every time and
|
||
would otherwise have no aggregate limit at all — while each of those
|
||
requests still costs an entrypoint lookup before it 404s. The aggregate
|
||
limit leaves room for one address to drive several entrypoints at their
|
||
full rate, and it is not configurable separately.
|
||
|
||
What that aggregate limit bounds is the database work an invented path
|
||
costs; log volume it caps rather than eliminates. A path that names no
|
||
entrypoint is recorded by the handler at `DEBUG`, and the aggregate
|
||
limiter logs its own rejections at `DEBUG` and without the path, so
|
||
neither appears at all under the default level. The per-entrypoint
|
||
limiter is the loud one: it logs every rejection at `WARN` with the
|
||
request path, which on this route is attacker-controlled text. A client
|
||
hammering a single invented path is served `RECEIVER_RATE_LIMIT`
|
||
requests and has the rest of its aggregate budget rejected there, so
|
||
the aggregate limit is what bounds the _number_ of those `WARN` lines —
|
||
to under ten times `RECEIVER_RATE_LIMIT` per minute per client IP, 1080
|
||
at the defaults, where before it there was no bound at all. Their
|
||
_width_ is bounded by the field budgets below, the same ones the access
|
||
log spends. The access log is bounded by neither limit: every request
|
||
is recorded once at `INFO`, served or rejected alike.
|
||
|
||
What the access log does bound is the _content_ of those lines. A 3xx
|
||
or 4xx response logs the chi route pattern — `/webhook/{uuid}`,
|
||
`/user/{username}//`, or the literal `(unmatched)` when the request hit
|
||
no route at all — in place of the concrete URL. Those are the outcomes
|
||
an unauthenticated client can drive for free: 404 and 429 on any
|
||
invented receiver path, a login redirect on any invented profile path.
|
||
Logging the URL there would let a flood write text of its own choosing,
|
||
at a length of its own choosing, into the log. 2xx and 5xx responses
|
||
keep the concrete path — a success resolved against a static route or
|
||
against the operator's own data (on the receiver, against a stored
|
||
entrypoint UUID), and a 5xx is a bug in this service, where the exact
|
||
path is the evidence and no client can provoke one at will.
|
||
|
||
The query string is never logged; it is replaced by the fixed marker
|
||
`?(redacted)`. It is client-chosen on every route, and
|
||
`/.well-known/healthcheck` and `/s/*` answer 200 to anyone with no rate
|
||
limiter in front of them, so a query on a fixed 200 URL would otherwise
|
||
buy the same amplification as an invented path. Nothing debuggable is
|
||
lost: `page`, on the authenticated pagination links, is the only query
|
||
parameter this service reads.
|
||
|
||
Client-supplied request content does not leave the host by the other
|
||
route either. The Sentry SDK attaches the request to every event it
|
||
captures, independently of the access log, and `SendDefaultPII=false`
|
||
does not cover all of what it copies: the raw query string and the
|
||
first 10 KiB of the request body are both taken unconditionally, the
|
||
body precisely because these handlers call `ParseForm`. A `BeforeSend`
|
||
hook therefore replaces the query string and the body with
|
||
`(redacted)`, drops cookies and the remote-address environment, and
|
||
reduces the headers to a fixed allowlist — `Accept`, `Content-Length`,
|
||
`Content-Type`, `Host`, `Origin`, `Referer`, `User-Agent` and
|
||
`X-Request-Id`.
|
||
|
||
The same hook rewrites the request URL. The SDK builds it as
|
||
`scheme://host/path` from the concrete path, which on the receiver
|
||
route is `/webhook/<uuid>` in full — and that UUID is a write
|
||
capability, not an identifier: anyone holding it can post events this
|
||
service accepts and its targets then deliver. A tracker has its own
|
||
retention, access control and deletion policy, so the rule the access
|
||
log follows above does not carry across that boundary. What is sent is
|
||
the chi route pattern instead: `http://host/webhook/{uuid}`.
|
||
|
||
The scheme and the host are kept, and everything else in the URL is
|
||
discarded rather than edited, so a future SDK version that starts
|
||
appending a query string cannot widen this. The scheme has to survive
|
||
for the reason given below. The host is whatever the request's `Host`
|
||
header carried — this service validates no hostname, so on a directly
|
||
exposed deployment a client sets it — and that same header is on the
|
||
allowlist above, so scrubbing the host out of the URL would withhold
|
||
nothing that is not sent anyway.
|
||
|
||
The body, the query string and the URL are all handled on every route
|
||
rather than filtered by route. For the URL that is also what keeps the
|
||
event locatable: an error event is grouped by its exception and stack
|
||
trace, not by its URL, so replacing the path with the pattern costs no
|
||
grouping and the pattern still names the route in the UI. And an
|
||
unconditional rule cannot leak on a route somebody forgets to add to
|
||
it, which a route-conditional one can. For the body there is a second
|
||
reason: nothing debuggable is lost, because every handler reads its
|
||
fields with `PostFormValue`, so the body is exactly where the
|
||
credentials are — the target destination URL, the login password, both
|
||
password-change fields — and the one route whose body is genuine
|
||
signal is the receiver, whose body is already stored on the event and
|
||
served from the UI, so a tracker is not where anyone reads it.
|
||
|
||
The route is reachable from the hook only on the error dispatch.
|
||
`sentryhttp`'s recover path puts the request on the context it hands
|
||
to `RecoverWithContext`, and the SDK carries that context through to
|
||
`BeforeSend` as `hint.Context`, so
|
||
`hint.Context.Value(sentry.RequestContextKey)` yields the live request
|
||
and chi's `RoutePattern()` yields the matched pattern off it. The
|
||
transaction dispatch has no such request: a finished span captures
|
||
with a nil hint, which the client replaces with an empty one, so
|
||
`BeforeSendTransaction` sees no context at all. Tracing is off in this
|
||
service, so no transaction event is produced today, but the hook is
|
||
installed on both dispatches as a floor.
|
||
|
||
Where the pattern is out of reach — the transaction dispatch, an event
|
||
captured outside the router, or a request that matched no route — the
|
||
fallback is never the concrete path. The path becomes the literal
|
||
`/(redacted)`, so the URL reads `http://host/(redacted)`; a URL the
|
||
rewrite cannot parse into a scheme is withheld whole. A transaction
|
||
event additionally carries the SDK's own `METHOD /path` name, built
|
||
from the concrete path as well; it is rewritten on the same terms, to
|
||
`POST /webhook/{uuid}` where the pattern is known and `POST
|
||
/(redacted)` where it is not.
|
||
|
||
The headers are an allowlist for the same reason the rules above are
|
||
unconditional: the SDK's own filter removes four names and passes
|
||
everything else, which would ship `X-CSRF-Token` and the shared
|
||
secrets senders put on the receiver route. What survives still names
|
||
the failing route — scheme, host, route pattern, method — and
|
||
`X-Request-Id` ties the event to the local access log line that holds
|
||
the rest. Nothing dropped is needed for the likeliest use, debugging a
|
||
CSRF rejection. Its three inputs are the TLS decision, `Origin` and
|
||
`Referer`; the latter two are kept, and the first is the scheme of the
|
||
retained URL, because the SDK derives that scheme from
|
||
`r.TLS != nil || r.Header.Get("X-Forwarded-Proto") == "https"` — byte
|
||
for byte the predicate `internal/middleware/csrf.go` uses to choose
|
||
between the `csrf.Secure(true)` and `csrf.Secure(false)` handlers.
|
||
That is what the rewrite above preserves it for, and it is why
|
||
dropping `X-Forwarded-Proto` costs nothing. The dropped provider
|
||
headers (`X-GitHub-Event`, `X-Gitlab-Event` and the like) are real
|
||
signal but are recorded locally on the event, and
|
||
`Sentry-Trace`/`Baggage` are already reflected in the event's trace
|
||
context.
|
||
|
||
The remaining client-supplied fields are truncated rather than dropped,
|
||
each to a fixed budget: 512 bytes for `url`, `useragent` and `referer`,
|
||
128 for `request_id` (chi passes an inbound `X-Request-Id` header
|
||
through), and 32 for `method`. A truncated `User-Agent` is still worth
|
||
reading; an absent one is not. A cut value ends in `[truncated]`, which
|
||
is charged on top of the budget rather than inside it.
|
||
|
||
Each budget is spent in _encoded_ bytes, not in the bytes the client
|
||
sent. Every rune is charged what the wider of the two log handlers
|
||
emits for it: two bytes for a quotation mark, a backslash or a tab; six
|
||
for a non-printable rune below U+10000; ten for one at or above it,
|
||
which the text handler spells `\UXXXXXXXX`. Go's header parser accepts
|
||
all of them in a header value, so a budget counted raw would buy a
|
||
field several times its nominal size — and the line, not the header, is
|
||
what an operator has to store. Plain ASCII encodes one byte for one, so
|
||
a real browser's `User-Agent` still fits whole; a value built out of
|
||
escapes keeps a proportionally shorter prefix, which is the right
|
||
trade.
|
||
|
||
Net: **one `INFO` line per request, of at most 2,560 bytes.** That
|
||
ceiling is arithmetic, not an observation: 3 × (512 + 11) for `url`,
|
||
`useragent` and `referer`, plus 128 + 11 for `request_id`, plus 32 + 11
|
||
for `method`, plus a 336-byte fixed portion (the field names, the
|
||
punctuation, both timestamps at their longest, an IPv6 `remoteIP` with
|
||
a zone, the status and the latency) — 2,087 bytes, stated at 2,560 so
|
||
the figure has headroom. `internal/middleware/accesslog_test.go`
|
||
asserts it against 8 KB of client-chosen text in the path, in the
|
||
query, and in each of `User-Agent`, `Referer` and `X-Request-Id`,
|
||
including cases built from the characters the handlers escape, and
|
||
against the widest access log line the service can be made to write: a
|
||
5xx that keeps its concrete path while all three header fields are also
|
||
at their budget. Every case runs through both handlers
|
||
`internal/logger` can select — the JSON one and the text one it installs
|
||
on a tty — since the two do not escape alike and the ceiling is quoted
|
||
unqualified. Measured over a real connection, the widest access log line
|
||
is 1,972 bytes.
|
||
|
||
Multiply that ceiling by the request rate to size log storage. Note
|
||
that the rate is not bounded by the limits above on every route:
|
||
`/.well-known/healthcheck` and `/s/*` sit behind no limiter, so there
|
||
the multiplier is whatever the deployment will serve.
|
||
|
||
**The same ceiling covers every other line the service writes through
|
||
`slog` that carries text an unauthenticated client supplies**, with one
|
||
exception stated below it: the recovered-panic record, which carries a
|
||
whole goroutine stack alongside its client-supplied fields and so has
|
||
its own wider ceiling. The access log is not the only line a client can
|
||
put its own text into, and a budget that held for one line and not the
|
||
others would be worse than no stated budget at all. Every `slog` call an
|
||
unauthenticated request can reach spends the same per-field budget
|
||
through `internal/logfield`, and each carries strictly fewer
|
||
client-supplied fields than the access log does, so none of them can be
|
||
wider than it:
|
||
|
||
| Log line | Level | Client-chosen value | Reachable unauthenticated |
|
||
| ------------------------------------------ | ------- | ------------------- | ----------------------------------------- |
|
||
| `request body exceeds limit` (413) | `WARN` | path, method | yes — `MaxBodySize` precedes `RequireAuth` |
|
||
| `csrf: token validation failed` (403) | `WARN` | path, method | yes — `CSRF` precedes `RequireAuth` |
|
||
| `... rate limit exceeded` (429) | `WARN` | path | yes, on the receiver |
|
||
| `auth middleware: unauthenticated request` | `DEBUG` | path, method | yes, by definition |
|
||
| `entrypoint not found` | `DEBUG` | entrypoint UUID | yes, on the receiver |
|
||
| `user not found` / `invalid password` | `DEBUG` | username | yes, on the login form |
|
||
| `login failure limit exceeded` (429) | `WARN` | path | yes, on the login form |
|
||
| `password verification capacity exhausted` | `WARN` | path | yes, on the login form |
|
||
|
||
`DEBUG` being off by default is not a bound. An operator turning it on
|
||
to diagnose a flood must not thereby hand the flood an unbounded write,
|
||
so those lines are capped too.
|
||
|
||
The last two rows are capped defensively rather than against a
|
||
demonstrated width: chi routes `POST /pages/login` on a static pattern,
|
||
so `r.URL.Path` there is the 12-byte constant `/pages/login` and each
|
||
line lands near 120 bytes. `RecordLoginFailure` is nonetheless an
|
||
exported method taking any `*http.Request`, and a future caller on a
|
||
route with a URL parameter would widen the line. Since no request
|
||
through the mux can, both caps are pinned by tests that call those two
|
||
entry points directly with the path such a caller would supply.
|
||
Removing either cap fails 14 subtests.
|
||
|
||
`internal/middleware/logbound_test.go` and
|
||
`internal/handlers/logbound_test.go` drive 8 KB of client-chosen text
|
||
at each of these — 1 KB at `invalid password`, whose accounts are
|
||
shared with the successful-login line, where a username past 4 KB
|
||
overflows the session cookie and answers 500 before that line is
|
||
written — through both handlers, and through seven fills: plain text
|
||
as the baseline, and then the quotation mark, backslash, tab, newline,
|
||
C0 control and astral non-printable, six characters the wider of the
|
||
two handlers spends more on than the client spent sending them. Every
|
||
case holds each line to the 2,560-byte ceiling. That per-line ceiling
|
||
is what the figure above states, and every row establishes it.
|
||
|
||
Three of the sites go further and bound the whole flood's output — the
|
||
total bytes a run of distinct invented values wrote, which is the
|
||
shape an operator sizing storage cares about. They are
|
||
`request body exceeds limit`
|
||
(`TestMaxBodySize_FloodOfOversizePathsDoesNotGrowTheLog`),
|
||
`entrypoint not found` and `user not found` (the last two through
|
||
`assertBoundedFlood`). The other rows carry no aggregate assertion;
|
||
the per-line ceiling is what they establish.
|
||
|
||
`internal/logfield/logfield_test.go` measures the per-rune charge
|
||
against what the handlers really emit, over roughly 3,000 code points on
|
||
each, so an undercharged rune fails a test rather than quietly
|
||
falsifying the ceiling.
|
||
|
||
**It covers GORM's statement logging as well.** GORM's own default
|
||
logger printed the fully interpolated SQL — parameters and all — to
|
||
standard output on every statement that returned an error, including a
|
||
plain record-not-found, at a level no operator setting reached. Two of
|
||
this service's lookups miss by design on unauthenticated routes: the
|
||
entrypoint lookup behind `/webhook/{uuid}` and the user lookup behind
|
||
the login form, whose path segment and submitted username the client
|
||
picks outright. Every
|
||
`gorm.Open` in the service now installs the adapter in
|
||
`internal/gormlog` instead. It writes through the same `slog` logger as
|
||
everything else, so its lines take the level the operator set and the
|
||
handler `internal/logger` selected, and every value it emits is spent
|
||
through the same `internal/logfield` budget. A record-not-found is not
|
||
logged as an error: it is the expected outcome on both of those paths,
|
||
and each handler already records its own miss at `DEBUG` — bounded, per
|
||
the table above — without the SQL. Slow statements are kept, at `WARN`,
|
||
above the same 200 ms threshold GORM used and with the statement
|
||
bounded, because that report is the one thing GORM's logger gave an
|
||
operator that nothing else here does. The adapter orders its cases
|
||
exactly as GORM's own `Trace` orders them, so a statement that both
|
||
missed and ran slow is still reported as slow, and dropping the miss
|
||
costs an operator no report `IgnoreRecordNotFoundError` would have
|
||
kept. A GORM line spends at most two of those budgets — the statement
|
||
and the driver error — against a smaller fixed portion than the access
|
||
log's, and `internal/gormlog/gormlog_test.go` asserts each line against
|
||
`MaxAccessLogLineBytes` directly rather than leaving it as arithmetic.
|
||
|
||
What that ceiling does **not** cover, stated here so the figure is not
|
||
read as more than it is:
|
||
|
||
- **Lines carrying an authenticated operator's own input**, which are
|
||
not truncated at all. `webhook created` logs the submitted `name`
|
||
verbatim and `target URL blocked by SSRF protection` logs the target
|
||
host (both `internal/handlers/source_management.go`), as do the
|
||
`target_name` lines in `internal/delivery/engine.go` and
|
||
`internal/delivery/target_http.go`. The only bound on any of them is
|
||
the 1 MB form body cap, so a 100 KB `name` writes a single line of
|
||
roughly 600 KB — measured. This is deliberate: every one of these
|
||
requires an authenticated operator on a service with no
|
||
self-registration, and truncating the operator's own configuration
|
||
echoed back would cost debuggability against no adversary. It does
|
||
mean the 2,560-byte figure sizes unauthenticated traffic, not the
|
||
operator's own administrative requests.
|
||
- The **`log` delivery target**, which writes the whole inbound event —
|
||
headers and body — to the log. This one is deliberate: capping it
|
||
would defeat the target, since emitting the payload is the delivery.
|
||
It costs nothing unless an authenticated operator creates a target of
|
||
that type on a specific webhook, and each line it writes is bounded
|
||
per event by the 1 MB receiver body cap. Adding one is a decision to
|
||
spend log volume on that webhook's payloads.
|
||
- **Two writers that do not go through `internal/logger` at all**, both
|
||
on standard error. `fx` prints the dependency graph and the lifecycle
|
||
hooks through its default console logger at startup and shutdown —
|
||
nothing calls `fx.WithLogger`, and `fx.New` builds that logger over
|
||
`os.Stderr`. The Go runtime writes a panic or a fatal error itself; a
|
||
panic in a background worker rather than in a request handler is the
|
||
case that reaches it, since nothing recovers those. Neither carries a
|
||
client-chosen value at a client-chosen length: the five `panic` calls
|
||
in this service are invariant guards over constants and over
|
||
`crypto/rand`.
|
||
- **`net/http`'s own faults**, which are _not_ a separate writer.
|
||
`internal/server/http.go` builds its server with a nil `ErrorLog`, so
|
||
`net/http` falls back to the `log` package's default logger — and
|
||
`internal/logger` calls `slog.SetDefault`, which redirects that logger
|
||
into whichever handler it installed. Those lines therefore arrive on
|
||
standard output, shaped like every other line, at `INFO`. They are not
|
||
truncated. A handler panic is no longer one of them: the recover
|
||
middleware below answers it and writes it as the bounded record
|
||
described there instead, and `internal/server/recoverer_test.go`
|
||
requires that `http: panic serving` appear in neither of the process's
|
||
two streams when a panic is driven through the production router. The
|
||
one panic still handed back to `net/http` is `http.ErrAbortHandler`,
|
||
which it special-cases and does not log at all. What is left on this
|
||
path is `net/http`'s own diagnostics, whose values are the runtime's,
|
||
not a client's.
|
||
|
||
Wider than that 2,560-byte ceiling, and stated separately rather than
|
||
carved out of it: the record a recovered panic produces. The recover
|
||
middleware in `internal/middleware` answers `500` and writes one `ERROR`
|
||
record through `internal/logger` carrying the panic value, the stack and
|
||
the request id — the same `request_id` the access log line for that
|
||
request carries, which is how the two are joined. It replaced chi's
|
||
`middleware.Recoverer`, which on a current Go release crashed inside its
|
||
own stack pretty-printer: the connection was dropped rather than
|
||
answered, and what reached the operator described that crash rather than
|
||
the fault behind it.
|
||
|
||
That record is bounded the same way, in the same encoded bytes and
|
||
through the same `internal/logfield` budget: 512 for the panic value,
|
||
because a handler is free to build one out of the request, 128 for the
|
||
request id, which a client supplies outright through `X-Request-Id`,
|
||
and 8,192 for the stack, cut at its far end so that the panic site
|
||
survives a cut and `net/http`'s accept frames are what is lost. Net:
|
||
**at most 10,240 bytes, once per recovered panic** — 9,121 by the
|
||
arithmetic (523 + 8,203 + 139 + a 256-byte fixed portion), stated at
|
||
10,240 for headroom.
|
||
|
||
Those two numbers are the claim; the measurements below only
|
||
illustrate it. `internal/middleware/recoverer_test.go` drives all
|
||
three growable fields past their budgets on one record, over both
|
||
handlers, and measured 9,009 bytes on the JSON handler and
|
||
8,982–8,983 on the text one in one checkout. Neither is an invariant:
|
||
the stack's own content decides where its cut lands, so the figures
|
||
move by a byte or so between runs. The real case is far below both —
|
||
through the shipped middleware chain the whole record measures
|
||
roughly 3,960 bytes over a roughly 3,690-byte stack, taken by
|
||
`internal/server/recoverer_test.go` from the process's own file
|
||
descriptors while driving a panic through the production router over
|
||
a real server in a subprocess. That pair moves further still, since
|
||
`debug.Stack()` embeds absolute source paths and so depends on where
|
||
the tree is checked out: four checkouts have reported 3,959, 3,961,
|
||
3,984 and 4,026. What the tests assert is the ceiling, that every
|
||
client-supplied field was cut, and that the shipped chain's stack
|
||
arrived uncut — never the numbers.
|
||
|
||
Every limiter here — receiver, login, and password change — identifies
|
||
the client the same way, through one shared key function: the
|
||
connection's own address, unless the peer is listed in
|
||
`TRUSTED_PROXIES`, in which case the forwarded client address is used
|
||
instead. That address becomes a bucket by family: IPv4 keys on the full
|
||
address, IPv6 on its `/64` prefix. A routed `/64` is the normal
|
||
residential and mobile IPv6 allocation, so keying IPv6 per address would
|
||
let one subscriber rotate source addresses and mint a fresh bucket per
|
||
request, evading these limits at the network layer without spoofing
|
||
anything; the cost is that distinct clients inside one `/64` share a
|
||
bucket. IPv4-mapped addresses (`::ffff:1.2.3.4`) key as the IPv4 address
|
||
they carry. See [Trusted proxies](#trusted-proxies). Deployed without that
|
||
variable set, a client behind a reverse proxy shares one bucket with
|
||
every other client behind the same proxy. Set `TRUSTED_PROXIES` to the
|
||
proxy's address to get per-client limits back. What the shared bucket
|
||
costs is not the same for every limiter, and the two cases pull in
|
||
opposite directions:
|
||
|
||
- For the **receiver** limits it costs throughput, which is the safe
|
||
direction to be wrong in: sharing can only make a limit bind sooner,
|
||
never let a sender past it. It matters more for the aggregate limit
|
||
than for the per-entrypoint one: with `TRUSTED_PROXIES` unset behind
|
||
the reverse proxy a production deployment is required to run behind,
|
||
every request keys on the proxy, so the aggregate limit becomes a
|
||
service-wide ceiling of 1200 requests per minute across all senders
|
||
and all entrypoints, where the per-entrypoint limit's capacity still
|
||
grows with the number of entrypoints. Any deployment with more than a
|
||
handful of busy entrypoints must set `TRUSTED_PROXIES`.
|
||
- For the **login and password-change** limits it costs precision, not
|
||
availability. Login failures from every client land in one counter,
|
||
so a stranger's wrong passwords make the operator's own wrong
|
||
passwords answer `429` sooner; the operator's _correct_ password is
|
||
never affected, because it is never counted. Production deployments
|
||
should still set `TRUSTED_PROXIES`; webhooker warns at startup
|
||
whenever it is empty, in any environment.
|
||
|
||
#### The login endpoint
|
||
|
||
The login `POST` is the one endpoint with no pre-emptive limiter in
|
||
front of it, and that is deliberate. A limiter that spends budget on
|
||
arrival is a lockout in this deployment shape: sharing one bucket, a
|
||
stranger sending five POSTs a minute — about 0.08 requests per second,
|
||
from anywhere — keeps it permanently full, and the operator has no
|
||
second administrative path. So the handler inverts the order:
|
||
|
||
1. **Credentials are verified first, and only a failed attempt spends
|
||
budget.** A correct password is never rate-limited, whatever the
|
||
counters hold. This is what guarantees the admin UI stays
|
||
reachable.
|
||
2. **Failures are counted per (client bucket, submitted username)**,
|
||
five per minute, after which further _failures_ from that pair are
|
||
answered `429` with a `Retry-After`. That `429` is a label on the
|
||
response, not a gate in front of the work: the credential check has
|
||
already run by the time the counter is consulted, so a throttled
|
||
client's guess is still evaluated. See the guessing rate below. A
|
||
successful login clears the counter, so mistyping a few times and
|
||
then getting it right leaves you unthrottled. Because the submitted
|
||
username is attacker-controlled, at most 1024 username counters and
|
||
1024 fallback address counters are tracked; past the first cap
|
||
failures fall back to the address counter, and past both they are
|
||
answered as throttled without being recorded. Total tracked state
|
||
is under half a megabyte and does not grow with the number of
|
||
usernames an attacker invents.
|
||
3. **Concurrent password verifications are capped at two, and the
|
||
queue for them at 16.** Verifying before counting means every login
|
||
request costs an Argon2id hash, and Argon2id here is 64 MB per
|
||
hash — two slots is a 128 MB ceiling on password hashing. Every
|
||
endpoint that hashes a password takes a slot, including the
|
||
password-change endpoint, which holds one across both the
|
||
verification and the new hash. A request that waits five seconds
|
||
without getting a slot is answered `503 Service Unavailable` and no
|
||
hash is computed for it. The wait alone does not bound memory, only
|
||
how long one request holds some, so the number of waiters is capped
|
||
as well. Size the queue from what a parked waiter actually retains,
|
||
not from the 1 MB body cap: that caps the raw body read, while the
|
||
body-cap, CSRF and form-parsing middleware all run before the
|
||
guard, so a waiter holds its parsed form plus its request header
|
||
block for the whole wait. Measured on the pinned Go 1.26.1
|
||
toolchain, as the heap delta with 64 waiters parked in the handler,
|
||
an ordinary two-field login form retains ~0 MB, a 1 MB urlencoded
|
||
body at Go's 10,000-parameter parse cap retains 2.82 MB (3.09 MB
|
||
with `%41` escapes), and the ~0.9 MB of headers the 1 MB header cap
|
||
allows takes it to **4.18 MB** — the retained parse and the headers
|
||
dominate, not the raw body. So the cap is 16 waiters: 16 x 4.18 MB
|
||
is about 67 MB of committed queue memory, and two slots drain a
|
||
full 16-deep queue in roughly 0.6 s, far inside the five-second
|
||
deadline. A request arriving past the cap is shed with `503`
|
||
immediately instead of joining the queue. **Peak commitment for the
|
||
endpoint is therefore about 203 MB**: 128 MB of Argon2id, plus the
|
||
18 requests holding a parsed form — 16 queued and the 2 being
|
||
hashed — at about 75 MB. That 203 MB is _live_ commitment, not
|
||
resident size: the Go collector lets the heap reach roughly twice
|
||
the live set before collecting, with transient parse garbage on top
|
||
of it. The independent review of this endpoint fired 18 adversarial
|
||
requests at an idle guard and measured a peak `HeapAlloc` of
|
||
392 MB. **Provision on the order of 400 MB**, not for the 203 MB
|
||
itemised here and not for the hashing budget alone.
|
||
|
||
An unknown username is verified against a dummy hash rather than
|
||
rejected early, so a nonexistent account costs the same time as a real
|
||
one and the response cannot be used to enumerate usernames.
|
||
|
||
**This raises online guessing throughput by about 300x, and that is
|
||
the trade.** Because the credential check always precedes the counter,
|
||
what bounds online brute force is the semaphore, not the failure
|
||
counter. Two slots at the cost of one Argon2id verification is on the
|
||
order of **27 guesses per second, about 2.3 million per day**, against
|
||
5 per minute under the pre-emptive limiter this replaced. Treat that
|
||
figure as a lower bound rather than a ceiling: it was measured with
|
||
Go's race detector enabled, so real hardware verifies faster and
|
||
guesses faster. Choose the admin password to survive millions of
|
||
online guesses per day — a long random passphrase, not a memorable
|
||
one. Rate-limiting `POST /pages/login` at the reverse proxy, where the
|
||
real client address is visible, is the way to put a cheaper bound back
|
||
on top.
|
||
|
||
The residual exposure is a bounded, self-clearing loss of login
|
||
**availability** — not merely of latency. A flood can keep both
|
||
verification slots busy, and a request that neither gets a slot within
|
||
five seconds nor finds room in the queue is answered `503`. Above
|
||
roughly 27 requests per second the operator is not served slowly, it
|
||
is shed: its chance per attempt is about the ratio of service rate to
|
||
flood rate, so at 400 requests per second it is roughly one attempt in
|
||
fourteen. A sufficiently determined flood still denies login for as
|
||
long as it runs.
|
||
|
||
What changed is the price and the aftermath. Denying login used to
|
||
cost an attacker 0.08 requests per second from anywhere; it now costs
|
||
30 or more sustained, about 400 times as much. Nothing accumulates
|
||
while the flood runs, nothing needs resetting when it stops, and the
|
||
operator's correct password succeeds on the first attempt afterwards.
|
||
Restarting the service is **not** a remedy: a restart clears the
|
||
failure counters, which are not what is saturated, and the flood
|
||
re-fills both verification slots on its first two requests. The
|
||
remedies are to block the source at the reverse proxy, or to
|
||
rate-limit `POST /pages/login` there — the one place a limit can be
|
||
applied without reintroducing the lockout, because the proxy sees the
|
||
real client address. Setting `TRUSTED_PROXIES` does not stop the
|
||
saturation, but it makes the source visible in the failure logs.
|
||
|
||
Finer-grained per-webhook rate limits (configured in the web UI and
|
||
enforced in the webhook handler) can layer on top of this env-level
|
||
abuse limit later; they are tracked as future work.
|
||
|
||
### API Endpoints
|
||
|
||
#### Public Endpoints
|
||
|
||
| Method | Path | Description |
|
||
| ------ | --------------------------- | ----------- |
|
||
| `GET` | `/` | Root redirect, 303 (authenticated → `/sources`, unauthenticated → `/pages/login`) |
|
||
| `GET` | `/.well-known/healthcheck` | Health check (JSON: `status`, `now`, `uptimeSeconds`, `uptimeHuman`, `version`, `appname`, `maintenanceMode`) |
|
||
| any | `/s/*` | Static file serving (embedded CSS, JS). Mounted for every method, not just `GET`/`HEAD`: chi's `Mount` registers all methods and `http.FileServer` special-cases only `HEAD` (by omitting the body), so a `POST` or `DELETE` to an asset is answered `200` with the file. Pinned by `TestStaticServesEveryMethod` |
|
||
| `POST` | `/webhook/{uuid}` | Webhook receiver endpoint. `POST` only — every other method is answered `405 Method Not Allowed` with `Allow: POST`. Rate limited (see [Rate Limiting](#rate-limiting)) |
|
||
|
||
#### Authentication Endpoints
|
||
|
||
| Method | Path | Description |
|
||
| ------ | --------------- | ----------- |
|
||
| `GET` | `/pages/login` | Login page (not rate limited) |
|
||
| `POST` | `/pages/login` | Login form submission. Credentials are verified before any limit is consulted, so a correct password is never throttled; 5 FAILED attempts per minute per bucket per submitted username, then `429`. `503` if no verification slot frees up within 5s, or immediately if 16 requests are already queued for one (see [Rate Limiting](#rate-limiting)) |
|
||
| `POST` | `/pages/logout` | Logout (destroys session) |
|
||
|
||
#### Authenticated Endpoints
|
||
|
||
| Method | Path | Description |
|
||
| ------ | ------------------------ | ----------- |
|
||
| `GET` | `/user/{username}` | User profile page |
|
||
| `POST` | `/user/{username}/password` | Change the user's password (5 per minute per bucket, then `429`; `503` if no verification slot frees up within 5s, or immediately if 16 requests are already queued for one) |
|
||
| `GET` | `/sources` | List user's webhooks |
|
||
| `GET` | `/sources/new` | Create webhook form |
|
||
| `POST` | `/sources/new` | Create webhook submission |
|
||
| `GET` | `/source/{id}` | Webhook detail view |
|
||
| `GET` | `/source/{id}/edit` | Edit webhook form |
|
||
| `POST` | `/source/{id}/edit` | Edit webhook submission |
|
||
| `POST` | `/source/{id}/delete` | Delete webhook |
|
||
| `GET` | `/source/{id}/logs` | Webhook event logs |
|
||
| `POST` | `/source/{id}/entrypoints` | Add entrypoint to webhook |
|
||
| `POST` | `/source/{id}/entrypoints/{entrypointID}/delete` | Delete an entrypoint |
|
||
| `POST` | `/source/{id}/entrypoints/{entrypointID}/toggle` | Enable or disable an entrypoint |
|
||
| `POST` | `/source/{id}/targets` | Add target to webhook |
|
||
| `POST` | `/source/{id}/targets/{targetID}/delete` | Delete a target |
|
||
| `POST` | `/source/{id}/targets/{targetID}/toggle` | Enable or disable a target |
|
||
|
||
#### Infrastructure Endpoints
|
||
|
||
| Method | Path | Description |
|
||
| ------ | ---------- | ----------- |
|
||
| `GET` | `/metrics` | Prometheus metrics, behind basic auth. The route is registered only when `METRICS_USERNAME` is set; otherwise it does not exist and returns 404 |
|
||
|
||
#### API (Planned)
|
||
|
||
| Method | Path | Description |
|
||
| -------- | ------------------------------ | ----------- |
|
||
| `GET` | `/api/v1/webhooks` | List webhooks |
|
||
| `POST` | `/api/v1/webhooks` | Create webhook |
|
||
| `GET` | `/api/v1/webhooks/{id}` | Get webhook details |
|
||
| `PUT` | `/api/v1/webhooks/{id}` | Update webhook |
|
||
| `DELETE` | `/api/v1/webhooks/{id}` | Delete webhook |
|
||
| `GET` | `/api/v1/webhooks/{id}/events` | List events for webhook |
|
||
| `POST` | `/api/v1/events/{id}/redeliver`| Redeliver an event |
|
||
|
||
None of these exist yet. `/api/v1` is mounted with no routes, so every
|
||
path under it returns 404 today. API authentication will use API keys
|
||
passed via `Authorization: Bearer <key>` header; no Bearer middleware
|
||
is implemented either.
|
||
|
||
### Package Layout
|
||
|
||
All application code lives under `internal/` to prevent external
|
||
imports. The entry point is `cmd/webhooker/main.go`.
|
||
|
||
```
|
||
webhooker/
|
||
├── cmd/webhooker/
|
||
│ └── main.go # Entry point: sets globals, wires fx
|
||
├── internal/
|
||
│ ├── config/
|
||
│ │ └── config.go # Configuration loading from environment variables
|
||
│ ├── database/
|
||
│ │ ├── base_model.go # BaseModel with UUID primary keys
|
||
│ │ ├── database.go # GORM connection, migrations, admin seed
|
||
│ │ ├── models.go # AutoMigrate for config-tier models
|
||
│ │ ├── model_setting.go # Setting entity (key-value app config)
|
||
│ │ ├── model_user.go # User entity
|
||
│ │ ├── model_webhook.go # Webhook entity
|
||
│ │ ├── model_entrypoint.go # Entrypoint entity
|
||
│ │ ├── model_target.go # Target entity and TargetType enum
|
||
│ │ ├── model_event.go # Event entity (per-webhook DB)
|
||
│ │ ├── model_delivery.go # Delivery entity (per-webhook DB)
|
||
│ │ ├── model_delivery_result.go # DeliveryResult entity (per-webhook DB)
|
||
│ │ ├── model_apikey.go # APIKey entity
|
||
│ │ ├── password.go # Argon2id hashing and verification
|
||
│ │ ├── retention.go # Retention reaper (per-webhook event expiry)
|
||
│ │ ├── testing.go # NewTestDatabase: wrapper for tests, no fx lifecycle
|
||
│ │ └── webhook_db_manager.go # Per-webhook DB lifecycle manager
|
||
│ ├── globals/
|
||
│ │ └── globals.go # Build-time variables (appname, version, arch)
|
||
│ ├── gormlog/
|
||
│ │ └── gormlog.go # GORM's logger.Interface on top of slog, bounded
|
||
│ ├── logfield/
|
||
│ │ └── logfield.go # Encoded-byte budget for client-supplied log values
|
||
│ ├── delivery/
|
||
│ │ ├── engine.go # Event-driven delivery engine (channel + timer based)
|
||
│ │ ├── circuit_breaker.go # Per-target circuit breaker for http/slack targets with retries
|
||
│ │ ├── target.go # Target interface, Task, Scheduler
|
||
│ │ ├── target_http.go # HTTP target (retries, circuit breaker)
|
||
│ │ ├── target_slack.go # Slack/Mattermost incoming-webhook target
|
||
│ │ ├── target_database.go # Database archive target
|
||
│ │ ├── target_database_archive.go # Archive file lifecycle and pruning
|
||
│ │ ├── target_log.go # Log target (stdout)
|
||
│ │ ├── target_config_view.go # Masked target config for templates
|
||
│ │ ├── archive_sweeper.go # Periodic pruning of idle archives
|
||
│ │ ├── url_mask.go # Strips credentials from *url.Error
|
||
│ │ └── ssrf.go # SSRF prevention (IP validation, safe HTTP transport)
|
||
│ ├── handlers/
|
||
│ │ ├── handlers.go # Base handler struct, JSON helpers, template rendering
|
||
│ │ ├── auth.go # Login, logout handlers
|
||
│ │ ├── event_log_view.go # Event log projection, byte-capped in SQL
|
||
│ │ ├── healthcheck.go # Health check handler
|
||
│ │ ├── index.go # Index page handler
|
||
│ │ ├── profile.go # User profile handler
|
||
│ │ ├── source_management.go # Webhook CRUD handlers
|
||
│ │ └── webhook.go # Webhook receiver handler
|
||
│ ├── healthcheck/
|
||
│ │ └── healthcheck.go # Health check service (uptime, version)
|
||
│ ├── lifecycle/
|
||
│ │ └── lifecycle.go # Shared stop-hook waiter, bounded by the stop context
|
||
│ ├── logger/
|
||
│ │ └── logger.go # slog setup with TTY detection
|
||
│ ├── middleware/
|
||
│ │ ├── middleware.go # Logging, CORS, Auth, Metrics, MetricsAuth, SecurityHeaders, MaxBodySize
|
||
│ │ ├── csrf.go # CSRF protection middleware (gorilla/csrf)
|
||
│ │ ├── ratelimit.go # Per-IP rate limiting middleware (go-chi/httprate)
|
||
│ │ ├── loginguard.go # Login failure counters and the Argon2id verification semaphore
|
||
│ │ └── testing.go # NewForTest: Middleware without the fx lifecycle
|
||
│ ├── server/
|
||
│ │ ├── server.go # Server struct, fx lifecycle, signal handling
|
||
│ │ ├── http.go # HTTP server setup with timeouts
|
||
│ │ └── routes.go # All route definitions
|
||
│ └── session/
|
||
│ ├── session.go # Cookie-based session management
|
||
│ └── testing.go # NewForTest: Session without the fx lifecycle
|
||
├── static/
|
||
│ ├── static.go # //go:embed directive
|
||
│ ├── css/input.css # Tailwind input, source for tailwind.css (make css)
|
||
│ ├── css/tailwind.css # Generated stylesheet the pages load
|
||
│ ├── css/style.css # Older hand-written stylesheet, no longer loaded
|
||
│ ├── js/app.js # Progressive-enhancement copy-to-clipboard
|
||
│ ├── js/alpine.min.js # Alpine.js, fetched by script/fetch-assets, not committed
|
||
│ └── vendor.sha256 # Pinned hashes the fetched assets are verified against
|
||
├── templates/ # Go HTML templates (base, login, sources, etc.)
|
||
├── script/ # Scripts to Rule Them All entrypoints
|
||
├── Dockerfile # Three stages: lint, test+build, Alpine runtime
|
||
├── Dockerfile.lint # Lint-only image built by script/lint
|
||
├── Makefile # 10 of 16 targets shim script/; 6 are inline
|
||
├── go.mod / go.sum
|
||
└── .golangci.yml # Linter configuration
|
||
```
|
||
|
||
### Dependency Injection
|
||
|
||
Components are wired via Uber fx in this order:
|
||
|
||
1. `globals.New` — Build-time variables (appname, version, arch)
|
||
2. `logger.New` — Structured logging (slog with TTY detection)
|
||
3. `config.New` — Configuration loading (environment variables)
|
||
4. `database.New` — Main SQLite connection, config migrations, admin
|
||
user seed
|
||
5. `database.NewWebhookDBManager` — Per-webhook event database
|
||
lifecycle manager
|
||
6. `database.NewRetentionReaper` — Per-webhook event retention sweep
|
||
7. `healthcheck.New` — Health check service
|
||
8. `session.New` — Cookie-based session manager (key from database)
|
||
9. `handlers.New` — HTTP handlers
|
||
10. `middleware.New` — HTTP middleware
|
||
11. `delivery.New` — Event-driven delivery engine
|
||
12. `delivery.NewArchiveSweeper` — Periodic pruning of idle archives
|
||
13. `delivery.Engine` → `delivery.Notifier` — interface bridge
|
||
14. `delivery.Engine` → `delivery.WebhookEvictor` — interface bridge so
|
||
deleting a webhook releases its archive writer
|
||
15. `server.New` — HTTP server and router
|
||
|
||
The server starts via `fx.Invoke(func(*server.Server, *delivery.Engine,
|
||
*database.RetentionReaper, *delivery.ArchiveSweeper) {})`, which
|
||
triggers the fx lifecycle hooks in dependency order. The
|
||
`delivery.Notifier` interface allows the webhook handler to send
|
||
self-contained `delivery.Task` slices to the engine without a direct
|
||
package dependency. Each task carries all target config and event data
|
||
inline (for bodies under 16 KiB, `delivery.MaxInlineBodySize`), so the
|
||
engine can deliver without reading from any database — it only writes
|
||
to record results.
|
||
|
||
### Middleware Stack
|
||
|
||
Applied to all routes in this order:
|
||
|
||
1. **RequestID** — Generate unique request IDs (chi built-in)
|
||
2. **SecurityHeaders** — Production security headers on every response
|
||
(HSTS, X-Content-Type-Options, X-Frame-Options, CSP, Referrer-Policy,
|
||
Permissions-Policy)
|
||
3. **Logging** — Structured request logging (method, URL, status,
|
||
latency, remote IP, user agent, request ID)
|
||
4. **Metrics** — Prometheus HTTP metrics (if `METRICS_USERNAME` is set)
|
||
5. **CORS** — Cross-origin resource sharing headers
|
||
6. **Timeout** — 60-second request timeout
|
||
7. **Recoverer** — Panic recovery: one `ERROR` record through
|
||
`internal/logger` and a `500`
|
||
8. **Sentry** — Error reporting to Sentry (if `SENTRY_DSN` is set;
|
||
configured with `Repanic: true` so panics still reach Recoverer)
|
||
|
||
Recoverer sits seventh rather than first, and both neighbours are the
|
||
reason. It runs **inside** everything that observes the response, so
|
||
the `500` it writes for a panicking handler is the status the access
|
||
log records and the metrics count; registered first, as chi's own
|
||
`middleware.Recoverer` was, the same request was logged as a `200` that
|
||
the client never received. It runs **outside** the Sentry handler, so
|
||
`Repanic: true` has something to re-raise into: an operator with
|
||
`SENTRY_DSN` set keeps the report, and one without it now gets the
|
||
local record instead of nothing. What that placement gives up is
|
||
recovery of a panic in the six entries above it, none of which does
|
||
more than set a header or start a timer.
|
||
|
||
Additionally, form endpoints (`/pages`, `/user/*`, `/sources`,
|
||
`/source/*`) apply a **MaxBodySize** middleware that limits
|
||
POST/PUT/PATCH request bodies to 1 MB. It is registered ahead of the
|
||
CSRF middleware in every one of those route groups, because
|
||
gorilla/csrf parses the form; if the cap were installed after it, form
|
||
parsing would run under net/http's 10 MB default and the 1 MB limit
|
||
would never apply. A request that declares a `Content-Length` over the
|
||
limit is answered with `413 Request Entity Too Large` without its body
|
||
being read and without reaching CSRF, the route group's remaining
|
||
middleware, or the handler. It is not rejected before *any* other
|
||
middleware, though: the global entries listed above all run first, so
|
||
such a request is still logged and given the security headers — and
|
||
counted in the metrics, on a deployment where `METRICS_USERNAME` is
|
||
set and the Metrics middleware is therefore registered at all. The
|
||
rejection itself is logged at `WARN` with the method, path and
|
||
declared length. A chunked request, or
|
||
one that lies about its length, is hard-capped by
|
||
`http.MaxBytesReader` and fails downstream at form-parse time.
|
||
|
||
Those same four route groups then apply **CSRF** and **NoCache**
|
||
(`Cache-Control: no-store`, `Pragma: no-cache`), and every group except
|
||
`/pages` applies **RequireAuth**. The rate limiters are per-route
|
||
rather than global: **PasswordChangeRateLimit** on
|
||
`/user/{username}/password` and **ReceiverRateLimit** on
|
||
`/webhook/{uuid}`. There is deliberately none on `/pages/login` — that
|
||
endpoint counts failures inside the handler, after the credential
|
||
check, see [The login endpoint](#the-login-endpoint).
|
||
|
||
### Authentication
|
||
|
||
- **Web UI:** Cookie-based sessions using gorilla/sessions with
|
||
encrypted cookies. Sessions are configured with HttpOnly, SameSite
|
||
Lax, and Secure (in production). Absolute session lifetime is 7 days,
|
||
with a sliding idle timeout on top of it (see
|
||
[Sessions](#sessions)).
|
||
- **API (planned):** API key authentication via `Authorization: Bearer`
|
||
header. API keys are stored per-user with usage tracking
|
||
(`last_used_at`).
|
||
- **Metrics:** Basic authentication protecting the `/metrics` endpoint.
|
||
|
||
### Security
|
||
|
||
- Passwords hashed with Argon2id (64 MB memory cost)
|
||
- Session cookies are HttpOnly, SameSite Lax, Secure (prod only)
|
||
- Session regeneration on login to prevent session fixation attacks
|
||
- Session key is a 32-byte value auto-generated on first startup and
|
||
stored in the database
|
||
- Production security headers on all responses: HSTS, X-Content-Type-Options
|
||
(`nosniff`), X-Frame-Options (`DENY`), Content-Security-Policy, Referrer-Policy,
|
||
and Permissions-Policy
|
||
- Request body size limits (1 MB) on all form POST endpoints, enforced
|
||
by middleware that runs before CSRF parses the form
|
||
- **CSRF protection** via [gorilla/csrf](https://github.com/gorilla/csrf)
|
||
on all state-changing forms (cookie-based double-submit tokens with
|
||
HMAC authentication). Applied to `/pages`, `/sources`, `/source`, and
|
||
`/user` routes. Excluded from `/webhook` (inbound webhook POSTs) and
|
||
`/api` (stateless API). The middleware auto-detects TLS status
|
||
per-request (via `r.TLS` and `X-Forwarded-Proto`) to set appropriate
|
||
cookie security flags and Origin/Referer validation mode
|
||
- **SSRF prevention** for HTTP delivery targets: private/reserved IP
|
||
ranges (RFC 1918, loopback, link-local, cloud metadata) are blocked
|
||
both at target creation time (URL validation) and at delivery time
|
||
(custom HTTP transport with SSRF-safe dialer that validates resolved
|
||
IPs before connecting, preventing DNS rebinding attacks)
|
||
- **Login limiting is inverted, deliberately.** The login `POST` has
|
||
no pre-emptive rate limiter in front of it. Credentials are
|
||
verified first and only a _failed_ attempt spends budget, so a
|
||
correct password is never throttled and no flood of wrong ones can
|
||
deny the operator the only administrative path. Failures are
|
||
counted per (bucket, submitted username), five per minute, after
|
||
which further failures are answered `429` with a `Retry-After`.
|
||
What bounds brute force is not that counter but the cap of two
|
||
concurrent Argon2id verifications: a throttled client's guess is
|
||
still evaluated, so roughly 27 guesses a second get through and the
|
||
admin password has to carry that load (see
|
||
[The login endpoint](#the-login-endpoint)). `GET` requests to the
|
||
login page are not limited
|
||
- **Password-change rate limiting** via [go-chi/httprate](https://github.com/go-chi/httprate):
|
||
sliding-window rate limiter, 5 POST attempts per minute per bucket.
|
||
It runs behind session auth, so only a client already holding a
|
||
valid session reaches it, and an operator throttled out of changing
|
||
a password can still log in. The bucket is per client IP only when
|
||
`TRUSTED_PROXIES` names the reverse proxy; unset, every client
|
||
shares one bucket, which costs precision rather than availability
|
||
(see [Rate Limiting](#rate-limiting)). webhooker warns at startup
|
||
whenever `TRUSTED_PROXIES` is empty
|
||
- Prometheus metrics behind basic auth
|
||
- Static assets embedded in binary (no filesystem access needed at
|
||
runtime)
|
||
- Container runs as non-root user (UID 1000)
|
||
- GORM soft deletes on every entity that carries `BaseModel`, which is
|
||
all of them but `Setting` (data preserved for audit)
|
||
|
||
### Shutdown
|
||
|
||
On SIGINT or SIGTERM, fx runs the registered stop hooks in reverse
|
||
dependency order under a **5 second budget** (`fx.StopTimeout` in
|
||
`cmd/webhooker/main.go`). That budget covers the whole sequence, not
|
||
each hook. The order, read off the fx stop-hook log:
|
||
|
||
1. `ArchiveSweeper`
|
||
2. `RetentionReaper`
|
||
3. `server` — the HTTP drain, bounded separately by
|
||
`server.ShutdownTimeout` (**3 seconds**), then a Sentry flush if
|
||
`SENTRY_DSN` is set
|
||
4. `delivery.Engine`
|
||
5. `healthcheck`
|
||
6. `WebhookDBManager`
|
||
7. the database close
|
||
|
||
The two components that can realistically hold the budget run
|
||
first: a retention sweep or an archive prune caught mid-tick each
|
||
waits on its `WaitGroup` bounded by the stop context, so a wedge
|
||
there consumes the 5 seconds before the HTTP server hook is ever
|
||
entered. The hooks after the server are microsecond-scale in normal
|
||
operation.
|
||
|
||
The HTTP drain budget is deliberately **shorter** than the sequence
|
||
budget. Were the two equal, a drain that used its whole budget would
|
||
exhaust the sequence budget at the instant it finished, and every
|
||
later hook — the delivery engine, the healthcheck, the webhook DB
|
||
manager and the database close — would be skipped in exactly the
|
||
case where the drain mattered. 3 seconds leaves 2 seconds
|
||
(`server.TailHookReserve`) for the tail, which is far more than the
|
||
microseconds it needs.
|
||
|
||
That reserve belongs to the tail hooks, not to the server hook, and
|
||
the Sentry flush is what could take it: it runs after the drain
|
||
**inside the same hook**, and `sentry.Flush` takes a bare duration
|
||
and honours no context, so an unreachable Sentry endpoint would add
|
||
its own timeout on top of a full-length drain and consume the whole
|
||
sequence budget by itself. It is therefore clamped to whatever is
|
||
left on the stop context minus the reserve, and skipped when that
|
||
leaves too little to be worth attempting — so a full-length drain
|
||
means Sentry events are dropped rather than the database close being
|
||
skipped.
|
||
|
||
This does not make the database close unconditional: a wedged
|
||
`ArchiveSweeper` or `RetentionReaper` still runs first and can
|
||
consume the whole budget on its own.
|
||
|
||
The value is chosen to sit inside the container stop grace period.
|
||
Docker's default `docker stop` grace is 10 seconds and the Dockerfile
|
||
sets no `STOPSIGNAL` or grace override, so the process must be gone
|
||
before that. fx's own default is 15 seconds, which is past the grace:
|
||
the container would be SIGKILLed (exit 137) before the bound could
|
||
fire, and nothing that depends on it — including the
|
||
`shutdown timed out, goroutines still running` error log that tells
|
||
an operator a component is wedged — would ever be reached.
|
||
|
||
Two operational consequences follow from bounding the sequence:
|
||
|
||
- **A wedged component aborts the rest of the shutdown.** fx checks
|
||
the stop context before each remaining hook and returns outright
|
||
once it has expired, skipping the hooks it has not reached. If the
|
||
first-stopped component consumes the whole budget, the later hooks
|
||
never run — **the database close among them**. SQLite is crash-safe,
|
||
so this is not corruption, but it is not a clean close either.
|
||
- **Lowering the grace below 5 seconds reintroduces the silent
|
||
truncation.** `docker stop --time`, Compose's `stop_grace_period`,
|
||
or Kubernetes' `terminationGracePeriodSeconds` set under 5 seconds
|
||
put SIGKILL back in front of the bound, and the process dies with
|
||
no shutdown diagnostics at all. Keep the deployment's grace above
|
||
the stop timeout.
|
||
|
||
### Linting
|
||
|
||
golangci-lint never runs on the host. `script/lint` builds
|
||
`Dockerfile.lint`, which copies the repo into the digest-pinned
|
||
golangci-lint image and lints as a build step, so a successful build is
|
||
a clean lint. A host binary would share one cache and one lock with
|
||
every other checkout on the machine, which has produced both invented
|
||
findings attributed to other worktrees and unearned passes.
|
||
|
||
Three properties are load-bearing:
|
||
|
||
- `script/lint` passes `--no-cache-filter=lint`. Without it an unchanged
|
||
tree replays the lint layer from cache and the build exits 0 in under
|
||
a second having linted nothing. The `deps` stage stays cacheable, so
|
||
module downloads are not repeated. Invalidation is scoped to the one
|
||
stage; never prune the shared build cache.
|
||
- `script/lint` does not trust that flag. Docker silently ignores
|
||
`--no-cache-filter` for a stage name that does not match, so a stage
|
||
rename or a one-character typo would restore the cached false green
|
||
with no warning and a fast exit 0. The script therefore tees the
|
||
build output and treats a run as a pass only if golangci-lint's own
|
||
summary line (`N issues.` / `N issues:`) appears in it: no summary,
|
||
no lint, whatever the exit code says.
|
||
- Both lint steps use `RUN --network=none`. `golangci-lint config
|
||
verify` is documented as fetching its JSON schema over HTTPS, which
|
||
would be an unpinned remote dependency; the pinned image resolves the
|
||
schema without network access, and `--network=none` enforces that
|
||
instead of trusting it. Verify is worth keeping because
|
||
`golangci-lint run` silently ignores config keys it does not
|
||
recognize, so a typo would disable a setting with no warning.
|
||
|
||
### Docker
|
||
|
||
The Dockerfile uses a three-stage build. Each stage is pinned by
|
||
digest, and the two check stages are separate images so the linter's
|
||
version is fixed independently of the compiler's:
|
||
|
||
1. **Lint stage** (`golangci/golangci-lint:v2.12.2`, Debian-based) —
|
||
installs `make`, downloads dependencies, copies the source, and runs
|
||
`make fmt-check`, then `golangci-lint config verify` and
|
||
`golangci-lint run`, both with `--network=none`.
|
||
2. **Builder stage** (`golang:1.26.1-bookworm`) — depends on the lint
|
||
stage passing (it copies a file from it), runs `script/fetch-assets`
|
||
to download and verify the third-party browser assets, then runs
|
||
`make test` and `make build`, and finally rebuilds the binary with
|
||
`CGO_ENABLED=1` and static linking so it runs on musl.
|
||
3. **Runtime stage** (`alpine:3.21`) — copies the static binary,
|
||
creates the `/var/lib/webhooker` directory for all SQLite databases,
|
||
runs as the non-root `webhooker` user (UID 1000), exposes port 8080,
|
||
and includes a health check against `/.well-known/healthcheck`.
|
||
|
||
The lint stage invokes `golangci-lint` directly rather than `make lint`:
|
||
it is already the pinned linter image, and `make lint` builds
|
||
`Dockerfile.lint`, which would need a docker daemon inside this build.
|
||
|
||
Both check stages use Debian rather than Alpine because
|
||
`gorm.io/driver/sqlite` pulls in `mattn/go-sqlite3`, which needs CGO
|
||
and does not compile against musl. Only the final binary is statically
|
||
linked, which is what lets it run on the Alpine runtime image.
|
||
|
||
`script/cibuild` — `docker build .` — is the CI gate: the checks run
|
||
inside the image, so a build that succeeds is a repo that is formatted,
|
||
linted, tested and compiled. `script/lint` also uses Docker
|
||
(`Dockerfile.lint`, see Linting above), so `make lint` and `make check`
|
||
run the same pinned linter version the gate does; only `script/test`
|
||
and `script/fmt-check` run on the host.
|
||
|
||
#### CI gate honesty
|
||
|
||
A layer cache lets `docker build .` exit 0 in seconds with the lint and
|
||
test stages replayed rather than executed, which would make a green
|
||
check meaningless. The `check` workflow therefore writes
|
||
`.ci-fingerprint` into the build context before building. Its value is
|
||
the hash of the last commit that touched the build context, so:
|
||
|
||
- Any commit that changes code (including a squash merge whose tree
|
||
matches an already-built branch) gets a new fingerprint, invalidates
|
||
the `COPY . .` layer of both check stages, and really runs
|
||
`make fmt-check`, `golangci-lint`, `make test`, and `make build`. A
|
||
run that reports success ran them.
|
||
- A docs-only commit leaves the fingerprint unchanged — `.dockerignore`
|
||
excludes `*.md`, `LICENSE` and `.editorconfig` from the context
|
||
anyway — so the image replays from cache and costs seconds.
|
||
|
||
The module download layer sits above `COPY . .` and stays cached either
|
||
way.
|
||
|
||
A separate workflow step, run before the fingerprint is written, covers
|
||
a second way the gate lied: Gitea cancels an in-flight run when a newer
|
||
commit lands on the same branch and records that cancellation as a
|
||
`failure` status, so a commit nothing ever tested reads as a test
|
||
result. Cancellation is unconditional server-side for push events, so
|
||
the superseding run calls `script/ci-mark-superseded`, which rewrites
|
||
that exact status to `failure` /
|
||
`Superseded by a newer commit; never tested`.
|
||
|
||
The state stays `failure` on purpose: Gitea's combined status folds
|
||
`skipped` into `success`, so marking a never-tested commit `skipped`
|
||
made the status API report green for it, indistinguishable from a commit
|
||
that passed. Reading a commit's status on this repo therefore goes:
|
||
|
||
- `success` / `Successful in ...` — the checks ran and passed.
|
||
- `failure` / `Failing after ...` — the checks ran and failed.
|
||
- `failure` / `Superseded by a newer commit; never tested` — the run was
|
||
cancelled, by a newer push or by hand, and nothing was verified about
|
||
this commit. Test the commit itself before concluding anything about
|
||
it.
|
||
|
||
Genuine failures and successes are never touched, and no status is left
|
||
`pending`, which would block the commit indefinitely. The step derives
|
||
its context string from the workflow name, the job **id** and the event.
|
||
That is deliberately not byte-identical to Gitea's own rule, which uses
|
||
the job's display `name:` where the runner exports the id, so giving the
|
||
job a `name:` — or renaming the workflow — makes the derived context
|
||
stop matching. The step fails loudly when no status on the commit
|
||
carries that context, so no rename can silently disable the rewrite.
|
||
|
||
## TODO
|
||
|
||
See [TODO.md](TODO.md).
|
||
|
||
## License
|
||
|
||
MIT
|
||
|
||
## Author
|
||
|
||
[@sneak](https://sneak.berlin)
|