With TRUSTED_PROXIES empty behind the reverse proxy production is required to run behind, every login POST keyed on the proxy's address and shared one 5/minute bucket. A stranger sending five POSTs a minute -- 0.08 requests per second, from anywhere -- kept that bucket permanently full, and the operator's own correct password was answered 429 indefinitely with no second administrative path. The login POST no longer has a pre-emptive limiter. The handler verifies credentials first and spends budget only on a FAILED attempt, so a correct password is never throttled whatever the counters hold. Three things follow, and are implemented together because the first is unsafe without the other two: - Failures are counted per (client bucket, submitted username), five per minute, after which further failures get 429 with a Retry-After. A successful login clears the counter, so mistyping and then succeeding does not leave the operator throttled. - Both key sets are capped at 1024 entries. The submitted username is attacker-controlled, so past the first cap failures fall back to a counter keyed on the client alone, and past both caps a failure is answered as throttled without being recorded. Tracked state stays under half a megabyte and does not grow with invented usernames. - Concurrent Argon2id verifications are capped at two, a 128 MB ceiling at 64 MB per hash, and the queue for those slots is capped at 64 waiters. Every password-hashing endpoint takes a slot, including the password-change endpoint, which holds one across both its hashes. A request that waits five seconds without a slot is answered 503, and one that arrives with the queue already full is shed with 503 immediately rather than joining it. Bounding the wait alone would not bound memory: a waiter reaches the guard with its form parsed, so it holds up to the 1 MB body cap for the whole wait, and at flood rates an unbounded queue is worth gigabytes against a 128 MB hashing budget. 64 waiters is 64 MB of committed queue memory, shallow enough that two slots drain a full queue inside the five-second deadline; peak commitment is 128 MB of hashing plus about 66 MB of parsed bodies. An unknown username is verified against a dummy hash instead of returning early, so a nonexistent account costs the same time as a real one and the response cannot be used to enumerate usernames. The password-change limiter is unchanged: RequireAuth runs ahead of it, so only a request already carrying a valid session reaches its bucket. Two consequences are documented rather than fixed, because they follow from the shape the issue asks for. Online guessing throughput rises from 5 a minute to roughly 27 a second, about 2.3 million a day: the credential check always precedes the counter, so the 429 is a label on the response rather than a gate in front of the hash, and what bounds brute force is the semaphore. And under a sustained flood the residual exposure is a loss of login availability, not merely of latency -- above about 27 requests a second most attempts are shed with 503, so a determined flood still denies login for as long as it runs. It costs roughly 400x more to run, nothing accumulates, and the first attempt after it stops succeeds. Restarting the service does not help: the counters a restart clears are not what is saturated. Also adds the missing test for the third bucketKey call site, where the peer is a trusted proxy but the forwarded chain names no client. Every existing test of that fallback uses an IPv4 proxy, where bucketKey is the identity function, so dropping the /64 masking there left the suite green. README and the TRUSTED_PROXIES startup warning updated: a shared bucket now costs precision, not the availability of the admin path.
webhooker
webhooker is a self-hosted webhook proxy and store-and-forward service written in Go by @sneak. It receives webhooks from external services, durably stores them, and delivers them to configured targets with retry support, logging, and observability. Category: infrastructure / web service. License: MIT.
Getting Started
Prerequisites
- Go 1.26.1+ (the version in
go.mod) - golangci-lint v2.12.2 (the version pinned in
script/bootstrapand in theDockerfile's lint stage;make bootstrapinstalls it) - Docker (for containerized deployment, and for the lint and test stages of the CI gate)
curl, used byscript/fetch-assetsto download the third-party browser assets, which are not committed (make bootstrapinstalls it if missing)
Quick Start
# Clone the repo
git clone https://git.eeqj.de/sneak/webhooker.git
cd webhooker
# Install Go dependencies, the pinned linter, and the third-party
# browser assets. `make deps` alone is not enough: it only runs
# go mod download/tidy, and the checks below need the fetched assets.
make bootstrap
# Run all checks (test, lint, format check)
make check
# Run in development mode. DATA_DIR defaults to /var/lib/webhooker in
# every environment, so set it (in .env or the shell) to a writable
# directory when running from a clone.
DATA_DIR=./data make dev
# Build Docker image
make docker
Development Commands
make bootstrap # Install all dependencies (idempotent)
make setup # Bootstrap + install git pre-commit hook
make assets # Fetch + verify third-party browser assets
make fmt # Format code (gofmt + goimports)
make fmt-check # Fail if gofmt would change anything (writes nothing)
make lint # Run golangci-lint
make test # Run tests with race detection
make check # test + lint + fmt-check (CI gate)
make build # Build binary to bin/webhooker
make run # build, then run ./bin/webhooker
make dev # go run ./cmd/webhooker
make deps # go mod download + go mod tidy
make docker # Build Docker image
make hooks # Install git pre-commit hook that runs script/precommit
make css # Regenerate static/css/tailwind.css (needs tailwindcss)
make clean # Remove bin/
Configuration
All configuration is via environment variables. For local development,
you can place variables in a .env file in the project root (loaded
automatically via godotenv/autoload).
The environment is selected by setting WEBHOOKER_ENVIRONMENT to dev
or prod (default: dev). The setting controls several behaviors:
| Behavior | dev |
prod |
|---|---|---|
| CORS | Allows any origin (*) |
Disabled (no-op) |
| Session cookie Secure | false (works over plain HTTP) |
true (requires HTTPS) |
The CSRF cookie's Secure flag and Origin/Referer validation mode are
determined per-request based on the actual transport protocol, not the
environment setting. The middleware checks r.TLS (direct TLS) and the
X-Forwarded-Proto header (TLS-terminating reverse proxy) to decide:
- Direct TLS or
X-Forwarded-Proto: https: Secure cookies, strict Origin/Referer validation. - Plaintext HTTP: Non-Secure cookies, relaxed Origin/Referer checks (token validation still enforced).
This means CSRF protection works correctly in all deployment scenarios:
behind a TLS-terminating reverse proxy, with direct TLS, or over plain
HTTP during development. When running behind a reverse proxy, ensure it
sets the X-Forwarded-Proto: https header.
All other differences (log format, security headers, etc.) are independent of the environment setting — log format is determined by TTY detection, and security headers are always applied.
| Variable | Description | Default |
|---|---|---|
WEBHOOKER_ENVIRONMENT |
dev or prod |
dev |
PORT |
HTTP listen port | 8080 |
DATA_DIR |
Directory for all SQLite databases | /var/lib/webhooker |
DEBUG |
Enable debug logging | false |
MAINTENANCE_MODE |
Report maintenanceMode: true in the healthcheck JSON. It does not change how any request is served — no maintenance page exists |
false |
METRICS_USERNAME |
Basic auth username for /metrics |
"" |
METRICS_PASSWORD |
Basic auth password for /metrics |
"" |
SENTRY_DSN |
Sentry error reporting DSN | "" |
RETENTION_SWEEP_INTERVAL |
How often the retention reaper and archive sweeper run (Go duration, must be positive) | 1h |
SESSION_IDLE_TIMEOUT |
Idle session timeout (Go duration) | 24h |
RECEIVER_RATE_LIMIT |
Receiver requests/minute per IP per entrypoint (10x that per IP across the route) | 120 |
TRUSTED_PROXIES |
CIDRs whose forwarded headers are trusted (unset: all clients behind a proxy share one rate-limit bucket; a correct login password is never throttled either way) | "" (none) |
Trusted proxies
TRUSTED_PROXIES is a comma-separated list of CIDR blocks (a bare
address such as 192.168.1.7 is accepted and treated as a single
host), for example 192.168.1.7, 2001:db8::5. It decides whose
X-Forwarded-For header the rate limiters believe, so it should name
the addresses of your reverse proxies and nothing else.
X-Forwarded-For is honoured only when the connecting peer is
inside one of these blocks; for every other peer the client identity is
the connection's own address and the header is ignored. The default is
the empty list, which trusts nobody — anything else would let any
client pick its own rate limit bucket, minting a fresh one per request
or draining someone else's. Set it to the address of your reverse
proxy, and to nothing wider. A set but unparseable value aborts
startup.
That default is safe against forged headers, but leaving it unset in
production has a cost you must know about. Production runs behind a
TLS-terminating reverse proxy, so with TRUSTED_PROXIES unset every
request keys on the proxy's own address and all clients share a single
bucket per limit. The receiver limits become service-wide ceilings,
and the login endpoint's failure counting collapses onto one key, so a
stranger's wrong passwords throttle every other client's wrong
passwords.
What it cannot do is lock the operator out. The login endpoint verifies credentials before it consults any limit and charges only failures, so a correct password is never throttled no matter how full the bucket is. See Rate Limiting.
The remedy is to set TRUSTED_PROXIES to your reverse proxy's
address, which restores per-client buckets. webhooker logs a warning
at startup whenever TRUSTED_PROXIES is empty, in every environment —
not only when WEBHOOKER_ENVIRONMENT=prod, because that variable
defaults to dev and an operator who never set it is precisely the
one at risk. The warning is informational when nothing proxies to the
process: with no proxy in front, the peer address is the client's own
and the buckets are already per-client. See
Rate Limiting for what each limit shares.
X-Real-IP and True-Client-IP are never read, from any peer.
Reverse proxies append to X-Forwarded-For but forward other client
headers verbatim, so a single-valued header is client-controlled even
behind a trusted proxy.
Within a trusted request, X-Forwarded-For is read right to left,
because the rightmost entry is the one the nearest proxy appended and
everything left of it may have been written by the client. The first
hop that is not itself a trusted proxy is taken as the client. A hop
that is not a bare IP address — ip:port, a bracketed IPv6 literal,
the token unknown — ends the walk and the peer address is used
instead, since past such an entry the chain is not the shape assumed
here. The peer address is likewise used when the header is absent or
every hop in it is a trusted proxy.
Two operator requirements follow:
- Your proxy must append the peer address to
X-Forwarded-For(nginx$proxy_add_x_forwarded_for, HAProxyoption forwardfor, Caddy and AWS ALB by default), and must append a bare address with no port. - List proxy hosts only. Any address inside
TRUSTED_PROXIESchooses its own rate-limit key: itsX-Forwarded-Foris walked, so it can name a different address on every request to get a fresh bucket each time, or name another client's address to drain that client's bucket. Never list a block that also covers clients — a broad10.0.0.0/8on a network where clients live in the same range makes all three limits, including the unauthenticated webhook receiver, silently bypassable by every client in the block.
Sessions
Sessions are bounded by two independent clocks, and end at whichever one runs out first:
- Idle expiry (
SESSION_IDLE_TIMEOUT, default24h) is a sliding window. Every authenticated request pushes it forward, so a session in continuous use never hits it, while an abandoned one expires a day after its last use. Any non-positive value (0, or a negative duration such as-1s) disables idle expiry entirely; the absolute cap below still applies. A set-but-unparseable value aborts startup rather than silently falling back to the default. - Absolute expiry is a fixed 7 days from login. Activity does not extend it: after a week, every session ends and the user authenticates again.
Only requests that authenticate with the session count as activity, so an unauthenticated request carrying the cookie cannot keep a session alive. The idle timestamp is rewritten at most once per tenth of the idle window rather than on every request, which means a session may expire up to 10% early relative to the user's true last request, but never late.
Both clocks are anchored by timestamps stored in the session cookie. Sessions issued before this feature existed carry neither, so they are treated as expired: upgrading to a build that has it logs every existing session out once, and those users sign in again.
Invalid values abort startup
The defaults above apply only to variables that are unset (or set
to an empty string). A variable that is set but cannot be parsed is a
fatal configuration error: webhooker logs the offending variable and
its value and refuses to start, rather than silently running with a
substituted default. PORT=eighty, DEBUG=ture, and
RETENTION_SWEEP_INTERVAL=1 hour all abort startup. PORT must
additionally be a number in the range 1–65535,
RECEIVER_RATE_LIMIT must be at least 1,
RETENTION_SWEEP_INTERVAL must be greater than zero (it is a ticker
period, so 0s or a negative value would crash the reaper after
startup), and every entry in TRUSTED_PROXIES must be a CIDR block or
a bare IP address. SESSION_IDLE_TIMEOUT is the exception: a
non-positive value there means idle expiry is disabled, not invalid.
Boolean variables (DEBUG, MAINTENANCE_MODE) accept exactly the
spellings Go's strconv.ParseBool accepts — 1, t, T, TRUE,
true, True, 0, f, F, FALSE, false, False — and nothing
else. yes, on, and off are rejected rather than quietly treated
as false.
On first startup, webhooker automatically generates a cryptographically secure session encryption key and stores it in the database. This key persists across restarts — no manual key management is needed.
On first startup, webhooker creates an admin user
with a randomly generated password and logs it to stdout. This password
is only displayed once.
Running with Docker
docker run -d \
-p 8080:8080 \
-v /path/to/data:/var/lib/webhooker \
-e WEBHOOKER_ENVIRONMENT=prod \
webhooker:latest
The container runs as a non-root user (webhooker, UID 1000), exposes
port 8080, and includes a health check against
/.well-known/healthcheck. The /var/lib/webhooker volume holds all
SQLite databases: the main application database (webhooker.db), the
per-webhook event databases (events-{uuid}.db), and any archive
databases written by database targets (archive-{uuid}.db). Mount
this as a persistent volume to preserve data across container
restarts.
Entrypoints
This repository adheres to the
Scripts to Rule Them All
standard: normalized scripts in script/ are the entrypoints for the
development workflow. Ten of the Makefile's sixteen targets are thin
shims that call them; build, run, dev, deps, clean and css
are inline commands with no script behind them. We provide:
script/bootstrap— install all dependencies (idempotent)script/setup— make a fresh clone ready for development (bootstrap, then install-precommit)script/projectname— output the project name ("webhooker")script/fetch-assets— download the third-party browser assets intostatic/, verifying each against its pinned sha256script/test— run the test suitescript/lint— run golangci-lintscript/fmt— format all code (writes)script/fmt-check— check formatting (read-only)script/check— run test, lint, and fmt-checkscript/docker— build the Docker image tagged viascript/projectnamescript/cibuild— CI entrypoint:docker build .(the Dockerfile runs the checks, so a green build implies a green repo)script/ci-mark-superseded— CI helper: mark the commits whose run a newer push cancelled (see CI gate honesty)script/precommit— pre-commit checks (go mod tidyguard, thenscript/check)script/install-precommit— install the git pre-commit hook that runsscript/precommit
Third-party browser assets
The web UI serves one third-party script, Alpine.js. It is not committed:
a minified bundle in the tree is unreviewable, and REPO_POLICIES.md bars
both committed build artifacts and unpinned external references.
Instead script/fetch-assets downloads it from a pinned URL, checks the
download against a hardcoded sha256, and installs it under static/. The
sha256 of every installed asset is recorded in static/vendor.sha256, and
static/vendor_test.go re-hashes the bytes go:embed put in the binary
against that manifest — so the pin is enforced on what actually ships, not
merely written down. Any mismatch fails the build.
make bootstrap runs the fetch for local development, and the Dockerfile
runs it in the build stage; .gitignore and .dockerignore keep the
artifact out of both the repo and the build context.
To move to a new version: update the version, URL, and tarball sha256 in
script/fetch-assets and the asset sha256 in static/vendor.sha256, then
run make assets && make check.
Rationale
Webhook integrations between services are inherently fragile. The receiving service must be online when the webhook fires, most webhook senders provide no built-in retry mechanism, and there is no standard way to inspect what was sent, when it was sent, or whether delivery succeeded.
webhooker solves this by acting as a durable intermediary:
-
Reliable ingestion — webhooker is always ready to accept incoming webhooks. It stores every received event before attempting any delivery, so nothing is lost if downstream targets are unavailable.
-
Guaranteed delivery — Events are queued for delivery to each configured target. Failed deliveries are retried with configurable backoff. Every delivery attempt is logged with status codes, response bodies, and timing.
-
Observability — Full request/response logging for every webhook received and every delivery attempted. Prometheus metrics expose volume, latency, and error rates. The web UI provides real-time visibility into event flow.
-
Fan-out — A single incoming webhook can be delivered to multiple targets simultaneously. This enables patterns like forwarding a GitHub webhook to both a deployment service and a Slack channel.
-
Replay (not yet implemented) — Every received event is stored in full, which is what manual redelivery for debugging or testing will be built on. No redelivery exists today, in the web UI or the API; see TODO.md.
Use Cases
- Store-and-forward with configurable retries for unreliable receivers
- Observability via Prometheus metrics on webhook frequency, payload size, and delivery performance
- Debugging and introspection of webhook payloads in the web UI
- Replay of webhook events for application testing and development (planned; not yet implemented)
- Fan-out delivery of a single webhook to multiple downstream targets
- High-availability ingestion for delivery to less reliable backend systems
Design
Architecture Overview
webhooker is structured as a standard Go HTTP server following the sneak/prompts GO_HTTP_SERVER_CONVENTIONS. It uses:
- Uber fx for dependency injection and lifecycle management
- go-chi for HTTP routing
- GORM for database access with
modernc.org/sqlite as
the runtime SQLite driver. Note:
gorm.io/driver/sqlitetransitively depends onmattn/go-sqlite3, which requires CGO at build time (see Docker section) - slog (stdlib) for structured logging with TTY detection (text for dev, JSON for prod)
- gorilla/sessions for encrypted cookie-based session management
- gorilla/csrf for CSRF protection (cookie-based double-submit tokens)
- go-chi/httprate for
sliding-window rate limiting of the password-change and webhook
receiver endpoints. The bucket is per client IP only when
TRUSTED_PROXIESnames the reverse proxy; unset, every client behind that proxy shares one bucket per limit. The login endpoint counts failed attempts itself instead, so that a correct password is never throttled (see Rate Limiting) - Prometheus for metrics, served at
/metricsbehind basic auth - Sentry for optional error reporting
Naming Conventions
The codebase uses consistent naming throughout (rename completed in issue #12):
| Entity | Description |
|---|---|
| Webhook | Top-level configuration entity grouping entrypoints and targets |
| Entrypoint | A receiver URL where external services POST events |
| Target | A delivery destination for events |
Data Model
webhooker's data model has nine entities organized into two tiers: the application tier (user and webhook configuration) and the event tier (event ingestion, delivery, and logging).
┌─────────────────────────────────────────────────────────────┐
│ APPLICATION TIER │
│ (main application database) │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │
│ │ User │──1:N──│ Webhook │──1:N──│ Entrypoint │ │
│ │ │ │ │ │ │ │
│ │ │ │ │──1:N──│ Target │ │
│ │ │ └──────────┘ └──────────────┘ │
│ │ │──1:N──│ APIKey │ │
│ └──────────┘ └──────────┘ │
│ │
│ ┌──────────┐ │
│ │ Setting │ (key-value application config) │
│ └──────────┘ │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ EVENT TIER │
│ (per-webhook dedicated databases) │
│ │
│ ┌──────────┐ ┌──────────┐ ┌─────────────────┐ │
│ │ Event │──1:N──│ Delivery │──1:N──│ DeliveryResult │ │
│ └──────────┘ └──────────┘ └─────────────────┘ │
└─────────────────────────────────────────────────────────────┘
Setting
A key-value pair for application-level configuration that is auto-managed rather than user-provided. Used to store the session encryption key and any future auto-generated settings.
| Field | Type | Description |
|---|---|---|
key |
string | Primary key (setting name) |
value |
text | Setting value |
Currently stored settings:
session_key— Base64-encoded 32-byte session encryption key, auto-generated on first startup.
User
A registered user of the webhooker service.
| Field | Type | Description |
|---|---|---|
id |
UUID | Primary key |
username |
string | Unique login name |
password |
string | Argon2id hash (never exposed via API) |
Relations: Has many Webhooks. Has many APIKeys.
Passwords are hashed with Argon2id using secure defaults (64 MB memory,
1 iteration, 4 threads, 32-byte key, 16-byte salt). On first startup,
an admin user is created with a randomly generated 16-character
password logged to stdout.
Webhook
The top-level configuration entity. A webhook groups together one or more entrypoints (receiver URLs) and one or more targets (delivery destinations) into a logical unit. A user creates a webhook to set up event routing.
| Field | Type | Description |
|---|---|---|
id |
UUID | Primary key |
user_id |
UUID | Foreign key → User |
name |
string | Human-readable name |
description |
string | Optional description |
retention_days |
integer | Days to retain events (default: 30; 0 means retain forever) |
Relations: Belongs to User. Has many Entrypoints. Has many Targets.
The retention_days field controls how long event data is kept in the
webhook's dedicated database before automatic cleanup.
Setting retention_days to 0 means "retain events forever". Because
the column carries a default of 30, a literal zero cannot survive an
insert, so a zero is rewritten on save to a sentinel of 365 * 1000
days (database.RetentionForeverDays). The retention reaper recognises
that sentinel and skips the webhook entirely, and the web UI displays
such a webhook's retention as "forever" rather than as a day count.
Submitted retention_days values therefore fall into three bands, not
two:
1up todatabase.MaxFiniteRetentionDays(106751 days, about 292 years) is accepted as a finite retention.- Above that ceiling but below the retain-forever sentinel of 365000
(
database.RetentionForeverDays) is rejected with a 400. This is the band the cap exists for. 0, and365000or above, are accepted and mean retain forever, collapsing to the sentinel —0inWebhook.BeforeSave, the large values inparseRetentionDays. The large values are not out of range: the edit form pre-fills the sentinel for a retain-forever webhook, so submitting that form back unchanged has to keep meaning "forever".
A negative value is in none of the three: parseRetentionDays rejects
it with a 400 before BeforeSave ever sees it.
The cap is not arbitrary: the reaper computes its cutoff as a
time.Duration, an int64 nanosecond count, and a longer period
overflows it. An overflowed cutoff lands in the future, where it
matches every row, so the sweep would delete every event the webhook
has instead of none. The reaper also clamps the value it is given, so a
row written by an older version cannot trigger that either.
Entrypoint
A receiver URL where external services POST webhook events. Each entrypoint has a unique UUID-based path. When an HTTP request arrives at an entrypoint's path, webhooker captures the full request and creates an Event.
| Field | Type | Description |
|---|---|---|
id |
UUID | Primary key |
webhook_id |
UUID | Foreign key → Webhook |
path |
string | Unique bare UUID, generated at creation. The /webhook/ prefix is route only and is not stored: the receiver matches this column against the raw {uuid} path segment |
description |
string | Optional description |
active |
boolean | Whether this entrypoint accepts events (default: true) |
Relations: Belongs to Webhook.
A webhook can have multiple entrypoints. This allows separate URLs for different event sources that all feed into the same processing pipeline (e.g., one entrypoint for GitHub, another for Stripe, both routing to the same targets).
Target
A delivery destination for events. Each target defines where and how events should be forwarded.
| Field | Type | Description |
|---|---|---|
id |
UUID | Primary key |
webhook_id |
UUID | Foreign key → Webhook |
name |
string | Human-readable name |
type |
TargetType | One of: http, slack, database, log |
active |
boolean | Whether deliveries are enabled (default: true) |
config |
JSON text | Type-specific configuration |
max_retries |
integer | Maximum retry attempts for http and slack targets (0 = fire-and-forget, >0 = retries with backoff and a circuit breaker). Ignored by database and log targets |
max_queue_size |
integer | Stored and shown on the target's detail view, but not enforced anywhere yet: nothing in the delivery engine consults it. Queue depth is set by the two fixed 10,000-entry channels |
Relations: Belongs to Webhook. Has many Deliveries.
Target types:
http— Forward the event as an HTTP POST to a configured URL. Behavior depends onmax_retries: whenmax_retriesis 0 (the default), the target operates in fire-and-forget mode — a single attempt with no retries and no circuit breaker. Whenmax_retriesis greater than 0, failed deliveries are retried with exponential backoff up tomax_retriesattempts, protected by a per-target circuit breaker.slack— Post the event as a formatted message to a Slack-compatible incoming webhook URL (webhookUrlinconfig). It is built on the same HTTP core ashttpand honoursmax_retriesidentically, circuit breaker included. See the Slack target section under "Per-Webhook Event Databases" for the message format.database— Archive the full event as a row into a separate per-webhook archive database (archive-{webhookID}.db) for long-term retention, with an optional creation-validated expiry (default: keep forever). No external delivery and no retries; an archive write failure fails the delivery. See the database target section under "Per-Webhook Event Databases" for the full semantics.log— Write the event to the application log (stdout). Useful for debugging.
The config field stores type-specific configuration as JSON (e.g.,
destination URL, custom headers, timeout settings).
APIKey
A programmatic access credential for API authentication.
| Field | Type | Description |
|---|---|---|
id |
UUID | Primary key |
user_id |
UUID | Foreign key → User |
key |
string | Unique API key value |
description |
string | Optional description |
last_used_at |
timestamp | Last time this key was used (nullable) |
Relations: Belongs to User.
Event
A captured incoming webhook request. Stores the complete HTTP request data for auditing and for the planned replay capability.
| Field | Type | Description |
|---|---|---|
id |
UUID | Primary key |
webhook_id |
UUID | Foreign key → Webhook |
entrypoint_id |
UUID | Foreign key → Entrypoint |
method |
string | HTTP method of the captured request. Always POST: the receiver answers every other method with 405 before an Event is created |
headers |
JSON | Complete request headers |
body |
text | Raw request body |
content_type |
string | Content-Type header value |
Relations: Belongs to Webhook. Belongs to Entrypoint. Has many Deliveries.
When a request arrives at an entrypoint, the full request (method, headers, body) is captured as an Event. The event is then queued for delivery to every active target configured on the parent webhook.
Delivery
The pairing of an event with a target. Tracks the overall delivery status across potentially multiple attempts.
| Field | Type | Description |
|---|---|---|
id |
UUID | Primary key |
event_id |
UUID | Foreign key → Event |
target_id |
UUID | Foreign key → Target |
status |
DeliveryStatus | One of: pending, delivered, failed, retrying |
Relations: Belongs to Event. Belongs to Target. Has many DeliveryResults.
Delivery statuses:
pending— Created but not yet attempted.retrying— At least one attempt failed; more attempts remain.delivered— Successfully delivered (at least one attempt succeeded).failed— All retry attempts exhausted without success.
DeliveryResult
The result of a single delivery attempt. Every attempt (including retries) is individually logged for full observability.
| Field | Type | Description |
|---|---|---|
id |
UUID | Primary key |
delivery_id |
UUID | Foreign key → Delivery |
attempt_num |
integer | Attempt number (1-based) |
success |
boolean | Whether this attempt succeeded |
status_code |
integer | HTTP response status code (if applicable) |
response_body |
text | Response body (if applicable) |
error |
string | Error message (on failure) |
duration |
integer | Request duration in milliseconds |
Relations: Belongs to Delivery.
Common Fields
Every entity except Setting includes these fields from BaseModel.
Setting is a bare key-value row with no id, no timestamps and no
soft delete:
| Field | Type | Description |
|---|---|---|
id |
UUID | Auto-generated UUIDv4 primary key |
created_at |
timestamp | Record creation time |
updated_at |
timestamp | Last modification time |
deleted_at |
timestamp | Soft-delete timestamp (nullable; GORM soft deletes) |
Database Architecture
Per-Webhook Event Databases
webhooker uses separate SQLite database files: a main application
database for configuration data and per-webhook databases for event
storage. All database files live in the DATA_DIR directory.
Main Application Database ({DATA_DIR}/webhooker.db) — stores
configuration and application state:
- Settings — auto-managed key-value config (e.g. session encryption key)
- Users — accounts and Argon2id password hashes
- Webhooks — webhook configurations
- Entrypoints — receiver URL definitions
- Targets — delivery destination configurations
- APIKeys — programmatic access credentials
On first startup the main database is auto-migrated, a session
encryption key is generated and stored, and an admin user is created.
Per-Webhook Event Databases ({DATA_DIR}/events-{webhook_uuid}.db)
— each webhook gets its own dedicated SQLite file containing:
- Events — captured incoming webhook payloads
- Deliveries — event-to-target pairings and their status
- DeliveryResults — individual delivery attempt logs
Per-webhook databases are created automatically when a webhook is
created (and lazily on first access for webhooks that predate this
feature). They are managed by the WebhookDBManager component, which
handles connection pooling, lazy opening, migrations, and cleanup.
This separation provides:
- Isolation — a high-volume webhook won't cause lock contention or journal growth affecting the main application or other webhooks.
- Independent lifecycle — event databases can be independently backed up, archived, rotated, or size-limited without impacting the application.
- Clean deletion — removing a webhook and all its history is as simple as deleting one file. Configuration is soft-deleted in the main DB; the event database file is hard-deleted (permanently removed).
- Per-webhook retention — the
retention_daysfield on each webhook controls automatic cleanup of old events in that webhook's database only, or disables cleanup entirely when set to0(retain forever). - Performance — each webhook's database has its own page cache and
its own lock, so concurrent event ingestion across webhooks won't
contend. No write-ahead log is involved: both DSNs are
file:{path}?cache=shared&mode=rwcand nojournal_modepragma is ever issued, so every database runs on SQLite's default rollback journal.
The database target type builds on this architecture to provide
long-term archiving, separate from the per-webhook event database (which
may prune events under its own retention). Delivering to a database
target writes the full event — body, headers, method, content type, and
webhook/entrypoint/event identifiers — as a row into a dedicated archive
database, archive-{webhookID}.db, stored under the data directory
beside the event database. After each write the archive handle is closed
and reopened, debounced to at most once per second, so an operator can
move the archive file away for offline archiving without stopping the
service; a moved or removed archive file is recreated automatically on
the next write. An optional expiry in the target's config JSON (e.g.
{"expiry":"720h"}) is validated when the target is created — the
default (unset or the literal never) keeps rows forever — and rows
older than the expiry are pruned each time the archive is (re)opened. An
archive write failure is never silent success: the delivery records a
failed attempt with the error and is marked failed.
Because reopens only happen on writes, an archive belonging to a webhook
that has stopped receiving events would never be pruned. A background
archive sweeper closes that gap: on the same interval as the event
retention reaper (RETENTION_SWEEP_INTERVAL) it prunes every archive
whose database target declares a positive expiry, whether or not the
webhook is still receiving traffic. The sweep never creates an archive —
a webhook whose archive file does not yet exist is skipped, not
initialised — it takes the same per-webhook lock the write path uses, so
it can never interleave with a write, and it leaves the archive closed
afterwards so the move-the-file-away workflow keeps working. Archives
with no expiry, or the expiry never, are not touched by the sweep at
all.
Note that a webhook has one archive file but may carry more than one
database target, each with its own expiry. The shortest expiry
configured on any of them therefore governs the whole archive, and the
sweep applies it whether or not the webhook is still receiving events.
Configure a single database target per webhook unless you intend that.
Deleting a webhook releases its archive: the delivery engine's cached
archive writer is dropped and its file handle closed, so nothing lingers
after the webhook is gone. The archive file itself is deliberately
left on disk. Unlike the event database — per-webhook working storage
that is hard-deleted with the webhook — an archive is long-term storage
an operator may still want to keep or move away for offline retention,
and destroying it as a side effect of deleting a webhook would be
unrecoverable. Removing archive-{webhookID}.db is the operator's call.
Deleting a webhook's last database target releases the writer the same
way, and for the same reason leaves the file alone.
The Slack target type sends webhook events as formatted messages to
any Slack-compatible incoming webhook URL (works with Slack, Mattermost,
and other compatible services). Each message includes event metadata
(HTTP method, content type, timestamp, body size) and the payload
pretty-printed in a code block. JSON payloads are automatically
formatted with indentation for readability; non-JSON payloads are shown
as raw text. Large payloads are truncated to keep messages reasonable.
Config stores webhookUrl — the Slack/Mattermost incoming webhook
endpoint. That is the JSON key; the error text for a missing one reads
webhook_url is required, which is the message, not the key.
The database uses the
modernc.org/sqlite driver at
runtime, though CGO is required at build time due to the transitive
mattn/go-sqlite3 dependency from gorm.io/driver/sqlite.
Request Flow
External Service
│
│ POST /webhook/{uuid}
▼
┌─────────────┐ ┌──────────────┐ ┌──────────────┐
│ chi Router │────►│ Middleware │────►│ Webhook │
│ │ │ Stack │ │ Handler │
└─────────────┘ └──────────────┘ └──────┬───────┘
│
1. Look up Entrypoint by UUID
2. Capture full request as Event
3. Create Delivery records for each active Target
4. Build self-contained delivery.Task structs
(target config + event data inline for
bodies < 16 KiB)
5. Notify Engine via channel (no DB read needed)
│
▼
┌──────────────┐
│ Delivery │◄── retry timers
│ Engine │ (backoff)
│ (worker │
│ pool) │
└──────┬───────┘
│
┌── bounded worker pool (N workers) ──┐
▼ ▼ ▼
┌────────────┐ ┌────────────┐ ┌────────────┐
│ HTTP Target│ │ HTTP Target│ │ Log Target │
│(max_retries│ │(max_retries│ │ (stdout) │
│ == 0) │ │ > 0, │ └────────────┘
│ fire+forget│ │ backoff + │
└────────────┘ │ circuit │
│ breaker) │
└────────────┘
Bounded Worker Pool
The delivery engine uses a fixed-size worker pool (default: 10 workers) to process all deliveries. At most N deliveries are in-flight at any time, preventing goroutine explosions regardless of queue depth.
Architecture:
- Channels as queues: Two buffered channels serve as bounded queues: a delivery channel (new tasks from the webhook handler) and a retry channel (tasks from backoff timers). Both are buffered to 10,000.
- Fan-out via channel, not goroutines: When an event arrives with
multiple targets, each
delivery.Taskis sent to the delivery channel. Workers pick them up and process them — no goroutine-per-target. - Worker goroutines: A fixed number of worker goroutines select from both channels. Each worker processes one task at a time, then picks up the next. Workers are the ONLY goroutines doing actual HTTP delivery.
- Retry backpressure with DB fallback: When a retry timer fires and
the retry channel is full, the timer is dropped — the delivery stays
in
retryingstatus in the database. A periodic sweep (every 60s) scans for these "orphaned" retries and re-queues them. No blocked goroutines, no unbounded timer chains. - Bounded concurrency: At most N deliveries (N = number of workers) are in-flight simultaneously. Even if a circuit breaker is open for hours and thousands of retries queue up in the channels, the workers drain them at a controlled rate when the circuit closes.
This means:
- No goroutine explosion — even with 10,000 queued retries, only N worker goroutines exist.
- Natural backpressure — if workers are busy, new tasks wait in the channel buffer rather than spawning more goroutines.
- Independent results — each worker records its own delivery result in the per-webhook database without coordination.
- Graceful shutdown — cancel the context, workers finish their
current task and exit. The stop hook waits for the pool via
lifecycle.WaitForShutdown, which bounds that wait by fx's stop timeout rather than blocking forever on a wedged worker. On timeout it logs atERRORand returns an error, and the goroutines that did not finish are still running — an unclean shutdown is reported rather than hidden.
Recovery paths:
- Startup recovery: When the engine starts, it scans all per-webhook
databases for
pendingandretryingdeliveries. Pending deliveries are sent to the delivery channel; retrying deliveries get backoff timers scheduled. - Periodic retry sweep (DB-mediated fallback): Every 60 seconds the
engine scans for
retryingdeliveries whose backoff period has elapsed. This catches "orphaned" retries — ones whose in-memory timer was dropped because the retry channel was full. The database is the durable fallback that ensures no retry is permanently lost, even under extreme backpressure.
Changing a target's type does not migrate in-flight deliveries. Only
http and slack targets own durable retries; database and log
targets are fire-and-forget and never produce a retrying delivery. If a
target's type is edited from a retrying type to a non-retrying (or
unknown) one while one of its deliveries is still retrying, both
recovery paths above terminally mark that delivery failed and record a
DeliveryResult naming the current target type as the reason, logging it
at warn level. The delivery is not re-dispatched under the new type — the
operator never asked for that delivery — and while the event itself
remains stored in the per-webhook event database, there is no way to
redeliver it: manual redelivery is planned, not implemented (see
TODO.md).
Circuit Breaker (HTTP and Slack Targets with Retries)
http and slack targets with max_retries > 0 are protected by a
per-target circuit breaker that prevents hammering a down target
with repeated failed delivery attempts. The circuit breaker is
in-memory only and resets on restart (which is fine — startup recovery
rescans the database anyway).
States:
| State | Behavior |
|---|---|
| Closed | Normal operation. Deliveries flow through. Consecutive failures are counted. |
| Open | Target appears down. Deliveries are skipped and rescheduled for after the cooldown. |
| Half-Open | Cooldown expired. One probe delivery is allowed to test if the target has recovered. |
Transitions:
success ┌──────────┐
┌────────────────────► │ Closed │ ◄─── probe succeeds
│ │ (normal) │
│ └────┬─────┘
│ │ N consecutive failures
│ ▼
│ ┌──────────┐
│ │ Open │ ◄─── probe fails
│ │(tripped) │
│ └────┬─────┘
│ │ cooldown expires
│ ▼
│ ┌──────────┐
└──────────────────────│Half-Open │
│ (probe) │
└──────────┘
Defaults:
- Failure threshold: 5 consecutive failures before opening
- Cooldown: 30 seconds in open state before probing
Scope: Circuit breakers apply to http and slack targets with
max_retries > 0. The Slack target is built on the same HTTP core
and hands its own max_retries to the same retry path, so it gets a
breaker with the same 5-failure / 30-second defaults. Fire-and-forget
targets of either type (max_retries == 0), database targets (local
operations), and log targets (stdout) do not use circuit breakers.
When a circuit is open and a new delivery arrives, the engine marks the
delivery as retrying and schedules a retry timer for after the
remaining cooldown period. This ensures no deliveries are lost — they're
just delayed until the target is healthy again.
Rate Limiting
Global blanket rate limiting middleware (e.g., a per-IP throttle shared with the web UI) must not apply to webhook receiver endpoints. Webhook endpoints receive automated traffic from external services at unpredictable rates, and blanket limits shared with other routes would cause legitimate deliveries to be dropped.
The receiver instead has its own dedicated abuse limit, scoped to the
/webhook/{uuid} route only and keyed per client IP per request path
(httprate.KeyByEndpoint): one misbehaving sender is throttled without
affecting other senders of the same entrypoint or the same sender's
other entrypoints. Keying on the path rather than on the entrypoint
matters — see the aggregate limit below. The limit is
RECEIVER_RATE_LIMIT requests per minute (default 120, generous for
legitimate webhook senders). Requests over the limit receive HTTP 429
with a Retry-After header. A set-but-invalid RECEIVER_RATE_LIMIT
value aborts startup rather than silently falling back to the default.
A second limit sits in front of that one, keyed on the client IP alone
and covering the whole route at ten times RECEIVER_RATE_LIMIT requests
per minute (default 1200). The per-entrypoint limit needs it: the route
pattern matches any single path segment, so a client that invents a
fresh path per request gets a fresh per-entrypoint bucket every time and
would otherwise have no aggregate limit at all — while each of those
requests still costs an entrypoint lookup before it 404s. The aggregate
limit leaves room for one address to drive several entrypoints at their
full rate, and it is not configurable separately.
What that aggregate limit bounds is the database work an invented path
costs; log volume it caps rather than eliminates. A path that names no
entrypoint is recorded by the handler at DEBUG, and the aggregate
limiter logs its own rejections at DEBUG and without the path, so
neither appears at all under the default level. The per-entrypoint
limiter is the loud one: it still logs every rejection at WARN with
the request path, which on this route is attacker-controlled text. A
client hammering a single invented path is served RECEIVER_RATE_LIMIT
requests and has the rest of its aggregate budget rejected there, so
the aggregate limit is what bounds those WARN lines — to under ten
times RECEIVER_RATE_LIMIT per minute per client IP, 1080 at the
defaults, where before it there was no bound at all. The access log is
bounded by neither limit: every request is recorded once at INFO,
served or rejected alike.
What the access log does bound is the content of those lines. A 3xx
or 4xx response logs the chi route pattern — /webhook/{uuid},
/user/{username}//, or the literal (unmatched) when the request hit
no route at all — in place of the concrete URL. Those are the outcomes
an unauthenticated client can drive for free: 404 and 429 on any
invented receiver path, a login redirect on any invented profile path.
Logging the URL there would let a flood write text of its own choosing,
at a length of its own choosing, into the log. 2xx and 5xx responses
keep the concrete path — a success resolved against a static route or
against the operator's own data (on the receiver, against a stored
entrypoint UUID), and a 5xx is a bug in this service, where the exact
path is the evidence and no client can provoke one at will.
The query string is never logged; it is replaced by the fixed marker
?(redacted). It is client-chosen on every route, and
/.well-known/healthcheck and /s/* answer 200 to anyone with no rate
limiter in front of them, so a query on a fixed 200 URL would otherwise
buy the same amplification as an invented path. Nothing debuggable is
lost: page, on the authenticated pagination links, is the only query
parameter this service reads.
The remaining client-supplied fields are truncated rather than dropped,
each to a fixed budget: 512 bytes for url, useragent and referer,
128 for request_id (chi passes an inbound X-Request-Id header
through), and 32 for method. A truncated User-Agent is still worth
reading; an absent one is not. A cut value ends in [truncated], which
is charged on top of the budget rather than inside it.
Each budget is spent in encoded bytes, not in the bytes the client
sent. Every rune is charged what the wider of the two log handlers
emits for it: two bytes for a quotation mark, a backslash or a tab; six
for a non-printable rune below U+10000; ten for one at or above it,
which the text handler spells \UXXXXXXXX. Go's header parser accepts
all of them in a header value, so a budget counted raw would buy a
field several times its nominal size — and the line, not the header, is
what an operator has to store. Plain ASCII encodes one byte for one, so
a real browser's User-Agent still fits whole; a value built out of
escapes keeps a proportionally shorter prefix, which is the right
trade.
Net: one INFO line per request, of at most 2,560 bytes. That
ceiling is arithmetic, not an observation: 3 × (512 + 11) for url,
useragent and referer, plus 128 + 11 for request_id, plus 32 + 11
for method, plus a 336-byte fixed portion (the field names, the
punctuation, both timestamps at their longest, an IPv6 remoteIP with
a zone, the status and the latency) — 2,087 bytes, stated at 2,560 so
the figure has headroom. internal/middleware/accesslog_test.go
asserts it against 8 KB of client-chosen text in the path, in the
query, and in each of User-Agent, Referer and X-Request-Id,
including cases built from the characters the handlers escape, and
against the widest line the service can be made to write: a 5xx that
keeps its concrete path while all three header fields are also at their
budget. Every case runs through both handlers internal/logger can
select — the JSON one and the text one it installs on a tty — since the
two do not escape alike and the ceiling is quoted unqualified. Measured
over a real connection, the widest line is 1,972 bytes.
Multiply that ceiling by the request rate to size log storage. Note
that the rate is not bounded by the limits above on every route:
/.well-known/healthcheck and /s/* sit behind no limiter, so there
the multiplier is whatever the deployment will serve.
Every limiter here — receiver, login, and password change — identifies
the client the same way, through one shared key function: the
connection's own address, unless the peer is listed in
TRUSTED_PROXIES, in which case the forwarded client address is used
instead. That address becomes a bucket by family: IPv4 keys on the full
address, IPv6 on its /64 prefix. A routed /64 is the normal
residential and mobile IPv6 allocation, so keying IPv6 per address would
let one subscriber rotate source addresses and mint a fresh bucket per
request, evading these limits at the network layer without spoofing
anything; the cost is that distinct clients inside one /64 share a
bucket. IPv4-mapped addresses (::ffff:1.2.3.4) key as the IPv4 address
they carry. See Trusted proxies. Deployed without that
variable set, a client behind a reverse proxy shares one bucket with
every other client behind the same proxy. Set TRUSTED_PROXIES to the
proxy's address to get per-client limits back. What the shared bucket
costs is not the same for every limiter, and the two cases pull in
opposite directions:
- For the receiver limits it costs throughput, which is the safe
direction to be wrong in: sharing can only make a limit bind sooner,
never let a sender past it. It matters more for the aggregate limit
than for the per-entrypoint one: with
TRUSTED_PROXIESunset behind the reverse proxy a production deployment is required to run behind, every request keys on the proxy, so the aggregate limit becomes a service-wide ceiling of 1200 requests per minute across all senders and all entrypoints, where the per-entrypoint limit's capacity still grows with the number of entrypoints. Any deployment with more than a handful of busy entrypoints must setTRUSTED_PROXIES. - For the login and password-change limits it costs precision, not
availability. Login failures from every client land in one counter,
so a stranger's wrong passwords make the operator's own wrong
passwords answer
429sooner; the operator's correct password is never affected, because it is never counted. Production deployments should still setTRUSTED_PROXIES; webhooker warns at startup whenever it is empty, in any environment.
The login endpoint
The login POST is the one endpoint with no pre-emptive limiter in
front of it, and that is deliberate. A limiter that spends budget on
arrival is a lockout in this deployment shape: sharing one bucket, a
stranger sending five POSTs a minute — about 0.08 requests per second,
from anywhere — keeps it permanently full, and the operator has no
second administrative path. So the handler inverts the order:
- Credentials are verified first, and only a failed attempt spends budget. A correct password is never rate-limited, whatever the counters hold. This is what guarantees the admin UI stays reachable.
- Failures are counted per (client bucket, submitted username),
five per minute, after which further failures from that pair are
answered
429with aRetry-After. That429is a label on the response, not a gate in front of the work: the credential check has already run by the time the counter is consulted, so a throttled client's guess is still evaluated. See the guessing rate below. A successful login clears the counter, so mistyping a few times and then getting it right leaves you unthrottled. Because the submitted username is attacker-controlled, at most 1024 username counters and 1024 fallback address counters are tracked; past the first cap failures fall back to the address counter, and past both they are answered as throttled without being recorded. Total tracked state is under half a megabyte and does not grow with the number of usernames an attacker invents. - Concurrent password verifications are capped at two, and the
queue for them at 64. Verifying before counting means every login
request costs an Argon2id hash, and Argon2id here is 64 MB per
hash — two slots is a 128 MB ceiling on password hashing. Every
endpoint that hashes a password takes a slot, including the
password-change endpoint, which holds one across both the
verification and the new hash. A request that waits five seconds
without getting a slot is answered
503 Service Unavailableand no hash is computed for it. The wait alone does not bound memory, only how long one request holds some: a waiter reaches the guard with its form already parsed, so it holds up to the 1 MB body cap for as long as it waits, and at flood rates an unbounded queue would be worth gigabytes against a 128 MB hashing budget. So the number of waiters is capped as well, at 64 — 1 MB each against 64 MB of committed queue memory, and shallow enough that two slots can drain a full queue inside the five-second deadline. A request arriving past the cap is shed with503immediately instead of joining the queue. Peak commitment for the endpoint is therefore 128 MB of hashing plus about 66 MB of parsed request bodies.
An unknown username is verified against a dummy hash rather than rejected early, so a nonexistent account costs the same time as a real one and the response cannot be used to enumerate usernames.
This raises online guessing throughput by about 300x, and that is
the trade. Because the credential check always precedes the counter,
what bounds online brute force is the semaphore, not the failure
counter. Two slots at the cost of one Argon2id verification is on the
order of 27 guesses per second, about 2.3 million per day, against
5 per minute under the pre-emptive limiter this replaced. Treat that
figure as a lower bound rather than a ceiling: it was measured with
Go's race detector enabled, so real hardware verifies faster and
guesses faster. Choose the admin password to survive millions of
online guesses per day — a long random passphrase, not a memorable
one. Rate-limiting POST /pages/login at the reverse proxy, where the
real client address is visible, is the way to put a cheaper bound back
on top.
The residual exposure is a bounded, self-clearing loss of login
availability — not merely of latency. A flood can keep both
verification slots busy, and a request that neither gets a slot within
five seconds nor finds room in the queue is answered 503. Above
roughly 27 requests per second the operator is not served slowly, it
is shed: its chance per attempt is about the ratio of service rate to
flood rate, so at 400 requests per second it is roughly one attempt in
fourteen. A sufficiently determined flood still denies login for as
long as it runs.
What changed is the price and the aftermath. Denying login used to
cost an attacker 0.08 requests per second from anywhere; it now costs
30 or more sustained, about 400 times as much. Nothing accumulates
while the flood runs, nothing needs resetting when it stops, and the
operator's correct password succeeds on the first attempt afterwards.
Restarting the service is not a remedy: a restart clears the
failure counters, which are not what is saturated, and the flood
re-fills both verification slots on its first two requests. The
remedies are to block the source at the reverse proxy, or to
rate-limit POST /pages/login there — the one place a limit can be
applied without reintroducing the lockout, because the proxy sees the
real client address. Setting TRUSTED_PROXIES does not stop the
saturation, but it makes the source visible in the failure logs.
Finer-grained per-webhook rate limits (configured in the web UI and enforced in the webhook handler) can layer on top of this env-level abuse limit later; they are tracked as future work.
API Endpoints
Public Endpoints
| Method | Path | Description |
|---|---|---|
GET |
/ |
Root redirect, 303 (authenticated → /sources, unauthenticated → /pages/login) |
GET |
/.well-known/healthcheck |
Health check (JSON: status, now, uptimeSeconds, uptimeHuman, version, appname, maintenanceMode) |
| any | /s/* |
Static file serving (embedded CSS, JS). Mounted for every method, not just GET/HEAD: chi's Mount registers all methods and http.FileServer special-cases only HEAD (by omitting the body), so a POST or DELETE to an asset is answered 200 with the file. Pinned by TestStaticServesEveryMethod |
POST |
/webhook/{uuid} |
Webhook receiver endpoint. POST only — every other method is answered 405 Method Not Allowed with Allow: POST. Rate limited (see Rate Limiting) |
Authentication Endpoints
| Method | Path | Description |
|---|---|---|
GET |
/pages/login |
Login page (not rate limited) |
POST |
/pages/login |
Login form submission. Credentials are verified before any limit is consulted, so a correct password is never throttled; 5 FAILED attempts per minute per bucket per submitted username, then 429. 503 if no verification slot frees up within 5s, or immediately if 64 requests are already queued for one (see Rate Limiting) |
POST |
/pages/logout |
Logout (destroys session) |
Authenticated Endpoints
| Method | Path | Description |
|---|---|---|
GET |
/user/{username} |
User profile page |
POST |
/user/{username}/password |
Change the user's password (5 per minute per bucket, then 429; 503 if no verification slot frees up within 5s, or immediately if 64 requests are already queued for one) |
GET |
/sources |
List user's webhooks |
GET |
/sources/new |
Create webhook form |
POST |
/sources/new |
Create webhook submission |
GET |
/source/{id} |
Webhook detail view |
GET |
/source/{id}/edit |
Edit webhook form |
POST |
/source/{id}/edit |
Edit webhook submission |
POST |
/source/{id}/delete |
Delete webhook |
GET |
/source/{id}/logs |
Webhook event logs |
POST |
/source/{id}/entrypoints |
Add entrypoint to webhook |
POST |
/source/{id}/entrypoints/{entrypointID}/delete |
Delete an entrypoint |
POST |
/source/{id}/entrypoints/{entrypointID}/toggle |
Enable or disable an entrypoint |
POST |
/source/{id}/targets |
Add target to webhook |
POST |
/source/{id}/targets/{targetID}/delete |
Delete a target |
POST |
/source/{id}/targets/{targetID}/toggle |
Enable or disable a target |
Infrastructure Endpoints
| Method | Path | Description |
|---|---|---|
GET |
/metrics |
Prometheus metrics, behind basic auth. The route is registered only when METRICS_USERNAME is set; otherwise it does not exist and returns 404 |
API (Planned)
| Method | Path | Description |
|---|---|---|
GET |
/api/v1/webhooks |
List webhooks |
POST |
/api/v1/webhooks |
Create webhook |
GET |
/api/v1/webhooks/{id} |
Get webhook details |
PUT |
/api/v1/webhooks/{id} |
Update webhook |
DELETE |
/api/v1/webhooks/{id} |
Delete webhook |
GET |
/api/v1/webhooks/{id}/events |
List events for webhook |
POST |
/api/v1/events/{id}/redeliver |
Redeliver an event |
None of these exist yet. /api/v1 is mounted with no routes, so every
path under it returns 404 today. API authentication will use API keys
passed via Authorization: Bearer <key> header; no Bearer middleware
is implemented either.
Package Layout
All application code lives under internal/ to prevent external
imports. The entry point is cmd/webhooker/main.go.
webhooker/
├── cmd/webhooker/
│ └── main.go # Entry point: sets globals, wires fx
├── internal/
│ ├── config/
│ │ └── config.go # Configuration loading from environment variables
│ ├── database/
│ │ ├── base_model.go # BaseModel with UUID primary keys
│ │ ├── database.go # GORM connection, migrations, admin seed
│ │ ├── models.go # AutoMigrate for config-tier models
│ │ ├── model_setting.go # Setting entity (key-value app config)
│ │ ├── model_user.go # User entity
│ │ ├── model_webhook.go # Webhook entity
│ │ ├── model_entrypoint.go # Entrypoint entity
│ │ ├── model_target.go # Target entity and TargetType enum
│ │ ├── model_event.go # Event entity (per-webhook DB)
│ │ ├── model_delivery.go # Delivery entity (per-webhook DB)
│ │ ├── model_delivery_result.go # DeliveryResult entity (per-webhook DB)
│ │ ├── model_apikey.go # APIKey entity
│ │ ├── password.go # Argon2id hashing and verification
│ │ ├── retention.go # Retention reaper (per-webhook event expiry)
│ │ ├── testing.go # NewTestDatabase: wrapper for tests, no fx lifecycle
│ │ └── webhook_db_manager.go # Per-webhook DB lifecycle manager
│ ├── globals/
│ │ └── globals.go # Build-time variables (appname, version, arch)
│ ├── delivery/
│ │ ├── engine.go # Event-driven delivery engine (channel + timer based)
│ │ ├── circuit_breaker.go # Per-target circuit breaker for http/slack targets with retries
│ │ ├── target.go # Target interface, Task, Scheduler
│ │ ├── target_http.go # HTTP target (retries, circuit breaker)
│ │ ├── target_slack.go # Slack/Mattermost incoming-webhook target
│ │ ├── target_database.go # Database archive target
│ │ ├── target_database_archive.go # Archive file lifecycle and pruning
│ │ ├── target_log.go # Log target (stdout)
│ │ ├── target_config_view.go # Masked target config for templates
│ │ ├── archive_sweeper.go # Periodic pruning of idle archives
│ │ ├── url_mask.go # Strips credentials from *url.Error
│ │ └── ssrf.go # SSRF prevention (IP validation, safe HTTP transport)
│ ├── handlers/
│ │ ├── handlers.go # Base handler struct, JSON helpers, template rendering
│ │ ├── auth.go # Login, logout handlers
│ │ ├── event_log_view.go # Event log projection, byte-capped in SQL
│ │ ├── healthcheck.go # Health check handler
│ │ ├── index.go # Index page handler
│ │ ├── profile.go # User profile handler
│ │ ├── source_management.go # Webhook CRUD handlers
│ │ └── webhook.go # Webhook receiver handler
│ ├── healthcheck/
│ │ └── healthcheck.go # Health check service (uptime, version)
│ ├── lifecycle/
│ │ └── lifecycle.go # Shared stop-hook waiter, bounded by the stop context
│ ├── logger/
│ │ └── logger.go # slog setup with TTY detection
│ ├── middleware/
│ │ ├── middleware.go # Logging, CORS, Auth, Metrics, MetricsAuth, SecurityHeaders, MaxBodySize
│ │ ├── csrf.go # CSRF protection middleware (gorilla/csrf)
│ │ ├── ratelimit.go # Per-IP rate limiting middleware (go-chi/httprate)
│ │ ├── loginguard.go # Login failure counters and the Argon2id verification semaphore
│ │ └── testing.go # NewForTest: Middleware without the fx lifecycle
│ ├── server/
│ │ ├── server.go # Server struct, fx lifecycle, signal handling
│ │ ├── http.go # HTTP server setup with timeouts
│ │ └── routes.go # All route definitions
│ └── session/
│ ├── session.go # Cookie-based session management
│ └── testing.go # NewForTest: Session without the fx lifecycle
├── static/
│ ├── static.go # //go:embed directive
│ ├── css/input.css # Tailwind input, source for tailwind.css (make css)
│ ├── css/tailwind.css # Generated stylesheet the pages load
│ ├── css/style.css # Older hand-written stylesheet, no longer loaded
│ ├── js/app.js # Progressive-enhancement copy-to-clipboard
│ ├── js/alpine.min.js # Alpine.js, fetched by script/fetch-assets, not committed
│ └── vendor.sha256 # Pinned hashes the fetched assets are verified against
├── templates/ # Go HTML templates (base, login, sources, etc.)
├── script/ # Scripts to Rule Them All entrypoints
├── Dockerfile # Three stages: lint, test+build, Alpine runtime
├── Makefile # 10 of 16 targets shim script/; 6 are inline
├── go.mod / go.sum
└── .golangci.yml # Linter configuration
Dependency Injection
Components are wired via Uber fx in this order:
globals.New— Build-time variables (appname, version, arch)logger.New— Structured logging (slog with TTY detection)config.New— Configuration loading (environment variables)database.New— Main SQLite connection, config migrations, admin user seeddatabase.NewWebhookDBManager— Per-webhook event database lifecycle managerdatabase.NewRetentionReaper— Per-webhook event retention sweephealthcheck.New— Health check servicesession.New— Cookie-based session manager (key from database)handlers.New— HTTP handlersmiddleware.New— HTTP middlewaredelivery.New— Event-driven delivery enginedelivery.NewArchiveSweeper— Periodic pruning of idle archivesdelivery.Engine→delivery.Notifier— interface bridgedelivery.Engine→delivery.WebhookEvictor— interface bridge so deleting a webhook releases its archive writerserver.New— HTTP server and router
The server starts via fx.Invoke(func(*server.Server, *delivery.Engine, *database.RetentionReaper, *delivery.ArchiveSweeper) {}), which
triggers the fx lifecycle hooks in dependency order. The
delivery.Notifier interface allows the webhook handler to send
self-contained delivery.Task slices to the engine without a direct
package dependency. Each task carries all target config and event data
inline (for bodies under 16 KiB, delivery.MaxInlineBodySize), so the
engine can deliver without reading from any database — it only writes
to record results.
Middleware Stack
Applied to all routes in this order:
- Recoverer — Panic recovery (chi built-in)
- RequestID — Generate unique request IDs (chi built-in)
- SecurityHeaders — Production security headers on every response (HSTS, X-Content-Type-Options, X-Frame-Options, CSP, Referrer-Policy, Permissions-Policy)
- Logging — Structured request logging (method, URL, status, latency, remote IP, user agent, request ID)
- Metrics — Prometheus HTTP metrics (if
METRICS_USERNAMEis set) - CORS — Cross-origin resource sharing headers
- Timeout — 60-second request timeout
- Sentry — Error reporting to Sentry (if
SENTRY_DSNis set; configured withRepanic: trueso panics still reach Recoverer)
Additionally, form endpoints (/pages, /user/*, /sources,
/source/*) apply a MaxBodySize middleware that limits
POST/PUT/PATCH request bodies to 1 MB. It is registered ahead of the
CSRF middleware in every one of those route groups, because
gorilla/csrf parses the form; if the cap were installed after it, form
parsing would run under net/http's 10 MB default and the 1 MB limit
would never apply. A request that declares a Content-Length over the
limit is answered with 413 Request Entity Too Large without its body
being read and without reaching CSRF, the route group's remaining
middleware, or the handler. It is not rejected before any other
middleware, though: the global entries listed above all run first, so
such a request is still logged and given the security headers — and
counted in the metrics, on a deployment where METRICS_USERNAME is
set and the Metrics middleware is therefore registered at all. The
rejection itself is logged at WARN with the method, path and
declared length. A chunked request, or
one that lies about its length, is hard-capped by
http.MaxBytesReader and fails downstream at form-parse time.
Those same four route groups then apply CSRF and NoCache
(Cache-Control: no-store, Pragma: no-cache), and every group except
/pages applies RequireAuth. The rate limiters are per-route
rather than global: PasswordChangeRateLimit on
/user/{username}/password and ReceiverRateLimit on
/webhook/{uuid}. There is deliberately none on /pages/login — that
endpoint counts failures inside the handler, after the credential
check, see The login endpoint.
Authentication
- Web UI: Cookie-based sessions using gorilla/sessions with encrypted cookies. Sessions are configured with HttpOnly, SameSite Lax, and Secure (in production). Absolute session lifetime is 7 days, with a sliding idle timeout on top of it (see Sessions).
- API (planned): API key authentication via
Authorization: Bearerheader. API keys are stored per-user with usage tracking (last_used_at). - Metrics: Basic authentication protecting the
/metricsendpoint.
Security
- Passwords hashed with Argon2id (64 MB memory cost)
- Session cookies are HttpOnly, SameSite Lax, Secure (prod only)
- Session regeneration on login to prevent session fixation attacks
- Session key is a 32-byte value auto-generated on first startup and stored in the database
- Production security headers on all responses: HSTS, X-Content-Type-Options
(
nosniff), X-Frame-Options (DENY), Content-Security-Policy, Referrer-Policy, and Permissions-Policy - Request body size limits (1 MB) on all form POST endpoints, enforced by middleware that runs before CSRF parses the form
- CSRF protection via gorilla/csrf
on all state-changing forms (cookie-based double-submit tokens with
HMAC authentication). Applied to
/pages,/sources,/source, and/userroutes. Excluded from/webhook(inbound webhook POSTs) and/api(stateless API). The middleware auto-detects TLS status per-request (viar.TLSandX-Forwarded-Proto) to set appropriate cookie security flags and Origin/Referer validation mode - SSRF prevention for HTTP delivery targets: private/reserved IP ranges (RFC 1918, loopback, link-local, cloud metadata) are blocked both at target creation time (URL validation) and at delivery time (custom HTTP transport with SSRF-safe dialer that validates resolved IPs before connecting, preventing DNS rebinding attacks)
- Login limiting is inverted, deliberately. The login
POSThas no pre-emptive rate limiter in front of it. Credentials are verified first and only a failed attempt spends budget, so a correct password is never throttled and no flood of wrong ones can deny the operator the only administrative path. Failures are counted per (bucket, submitted username), five per minute, after which further failures are answered429with aRetry-After. What bounds brute force is not that counter but the cap of two concurrent Argon2id verifications: a throttled client's guess is still evaluated, so roughly 27 guesses a second get through and the admin password has to carry that load (see The login endpoint).GETrequests to the login page are not limited - Password-change rate limiting via go-chi/httprate:
sliding-window rate limiter, 5 POST attempts per minute per bucket.
It runs behind session auth, so only a client already holding a
valid session reaches it, and an operator throttled out of changing
a password can still log in. The bucket is per client IP only when
TRUSTED_PROXIESnames the reverse proxy; unset, every client shares one bucket, which costs precision rather than availability (see Rate Limiting). webhooker warns at startup wheneverTRUSTED_PROXIESis empty - Prometheus metrics behind basic auth
- Static assets embedded in binary (no filesystem access needed at runtime)
- Container runs as non-root user (UID 1000)
- GORM soft deletes on every entity that carries
BaseModel, which is all of them butSetting(data preserved for audit)
Shutdown
On SIGINT or SIGTERM, fx runs the registered stop hooks in reverse
dependency order under a 5 second budget (fx.StopTimeout in
cmd/webhooker/main.go). That budget covers the whole sequence, not
each hook. The order, read off the fx stop-hook log:
ArchiveSweeperRetentionReaperserver— the HTTP drain, bounded separately byserver.ShutdownTimeout(3 seconds), then a Sentry flush ifSENTRY_DSNis setdelivery.EnginehealthcheckWebhookDBManager- the database close
The two components that can realistically hold the budget run
first: a retention sweep or an archive prune caught mid-tick each
waits on its WaitGroup bounded by the stop context, so a wedge
there consumes the 5 seconds before the HTTP server hook is ever
entered. The hooks after the server are microsecond-scale in normal
operation.
The HTTP drain budget is deliberately shorter than the sequence
budget. Were the two equal, a drain that used its whole budget would
exhaust the sequence budget at the instant it finished, and every
later hook — the delivery engine, the healthcheck, the webhook DB
manager and the database close — would be skipped in exactly the
case where the drain mattered. 3 seconds leaves 2 seconds
(server.TailHookReserve) for the tail, which is far more than the
microseconds it needs.
That reserve belongs to the tail hooks, not to the server hook, and
the Sentry flush is what could take it: it runs after the drain
inside the same hook, and sentry.Flush takes a bare duration
and honours no context, so an unreachable Sentry endpoint would add
its own timeout on top of a full-length drain and consume the whole
sequence budget by itself. It is therefore clamped to whatever is
left on the stop context minus the reserve, and skipped when that
leaves too little to be worth attempting — so a full-length drain
means Sentry events are dropped rather than the database close being
skipped.
This does not make the database close unconditional: a wedged
ArchiveSweeper or RetentionReaper still runs first and can
consume the whole budget on its own.
The value is chosen to sit inside the container stop grace period.
Docker's default docker stop grace is 10 seconds and the Dockerfile
sets no STOPSIGNAL or grace override, so the process must be gone
before that. fx's own default is 15 seconds, which is past the grace:
the container would be SIGKILLed (exit 137) before the bound could
fire, and nothing that depends on it — including the
shutdown timed out, goroutines still running error log that tells
an operator a component is wedged — would ever be reached.
Two operational consequences follow from bounding the sequence:
- A wedged component aborts the rest of the shutdown. fx checks the stop context before each remaining hook and returns outright once it has expired, skipping the hooks it has not reached. If the first-stopped component consumes the whole budget, the later hooks never run — the database close among them. SQLite is crash-safe, so this is not corruption, but it is not a clean close either.
- Lowering the grace below 5 seconds reintroduces the silent
truncation.
docker stop --time, Compose'sstop_grace_period, or Kubernetes'terminationGracePeriodSecondsset under 5 seconds put SIGKILL back in front of the bound, and the process dies with no shutdown diagnostics at all. Keep the deployment's grace above the stop timeout.
Docker
The Dockerfile uses a three-stage build. Each stage is pinned by digest, and the two check stages are separate images so the linter's version is fixed independently of the compiler's:
- Lint stage (
golangci/golangci-lint:v2.12.2, Debian-based) — installsmake, downloads dependencies, copies the source, and runsmake fmt-checkthenmake lint. - Builder stage (
golang:1.26.1-bookworm) — depends on the lint stage passing (it copies a file from it), runsscript/fetch-assetsto download and verify the third-party browser assets, then runsmake testandmake build, and finally rebuilds the binary withCGO_ENABLED=1and static linking so it runs on musl. - Runtime stage (
alpine:3.21) — copies the static binary, creates the/var/lib/webhookerdirectory for all SQLite databases, runs as the non-rootwebhookeruser (UID 1000), exposes port 8080, and includes a health check against/.well-known/healthcheck.
Both check stages use Debian rather than Alpine because
gorm.io/driver/sqlite pulls in mattn/go-sqlite3, which needs CGO
and does not compile against musl. Only the final binary is statically
linked, which is what lets it run on the Alpine runtime image.
script/cibuild — docker build . — is the CI gate: the four check
targets run inside the image, so a build that succeeds is a repo that
is formatted, linted, tested and compiled. Only script/cibuild and
script/docker involve Docker. script/lint, and therefore
make lint and make check, run whatever golangci-lint is on the
host, which can be a different version from the pinned one — so the
container is the authoritative lint result
(issue #109 tracks
routing local linting through it as well).
CI gate honesty
A layer cache lets docker build . exit 0 in seconds with the lint and
test stages replayed rather than executed, which would make a green
check meaningless. The check workflow therefore writes
.ci-fingerprint into the build context before building. Its value is
the hash of the last commit that touched the build context, so:
- Any commit that changes code (including a squash merge whose tree
matches an already-built branch) gets a new fingerprint, invalidates
the
COPY . .layer of both check stages, and really runsmake fmt-check,make lint,make test, andmake build. A run that reports success ran them. - A docs-only commit leaves the fingerprint unchanged —
.dockerignoreexcludes*.md,LICENSEand.editorconfigfrom the context anyway — so the image replays from cache and costs seconds.
The module download layer sits above COPY . . and stays cached either
way.
A separate workflow step, run before the fingerprint is written, covers
a second way the gate lied: Gitea cancels an in-flight run when a newer
commit lands on the same branch and records that cancellation as a
failure status, so a commit nothing ever tested reads as a test
result. Cancellation is unconditional server-side for push events, so
the superseding run calls script/ci-mark-superseded, which rewrites
that exact status to failure /
Superseded by a newer commit; never tested.
The state stays failure on purpose: Gitea's combined status folds
skipped into success, so marking a never-tested commit skipped
made the status API report green for it, indistinguishable from a commit
that passed. Reading a commit's status on this repo therefore goes:
success/Successful in ...— the checks ran and passed.failure/Failing after ...— the checks ran and failed.failure/Superseded by a newer commit; never tested— the run was cancelled, by a newer push or by hand, and nothing was verified about this commit. Test the commit itself before concluding anything about it.
Genuine failures and successes are never touched, and no status is left
pending, which would block the commit indefinitely. The step derives
its context string from the workflow name, the job id and the event.
That is deliberately not byte-identical to Gitea's own rule, which uses
the job's display name: where the runner exports the id, so giving the
job a name: — or renaming the workflow — makes the derived context
stop matching. The step fails loudly when no status on the commit
carries that context, so no rename can silently disable the rewrite.
TODO
See TODO.md.
License
MIT