webhooker/README.md

webhooker

webhooker is an MIT-licensed web service, written in Go by @sneak, that receives the HTTP requests other services send when something happens and forwards them to your own systems.

Many services send an HTTP request to a URL you give them when something happens: a code push, a payment, a new ticket. webhooker receives those requests, stores each one as an event, and forwards it to the places you choose. It is a single binary with a web interface. It stores its data in SQLite, a database engine that keeps each database in one ordinary file.

Rationale

webhooker exists so that a request from another service is not lost while your service is down. It also exists so that you can see what was sent and what happened to it.

It stores each event before it forwards it. It keeps events and their delivery attempts for the retention period you set. It can forward one event to several targets, and it can send a stored event again.

How it works

Five words name the moving parts.

• A webhook is a named configuration. It groups entrypoints and targets, and it owns the events it receives.
• An entrypoint is a receiving URL of the form /h/{uuid}. Senders post to it. The UUID is a random 36-character identifier, and it is the only thing that authorises a sender.
• A target is a place events are delivered to. There are four kinds: an HTTP URL, a Slack or Mattermost message-posting URL, an archive file, and the application log.
• An event is one stored request: its method, query string, headers and body.
• A delivery is one event paired with one target. It records every attempt.

The part of webhooker that answers requests to /h/{uuid} is the receiver. It stores the event and one pending delivery per active target, then answers 200.

Delivery is at least once, not exactly once. If a delivery reaches the target but webhooker cannot record that outcome, it sends the delivery again later. A delivery in flight when the service stops is also sent again at the next start. webhooker adds no delivery identifier of its own to the request, so make your targets tolerate a duplicate.

Getting Started

You need the Go version named in go.mod, and Docker for the checks.

git clone https://git.eeqj.de/sneak/webhooker.git
cd webhooker
make bootstrap
DATA_DIR=../webhooker-data make dev

DATA_DIR is where webhooker keeps its databases. It defaults to /var/lib/webhooker, so point it at a directory you can write to.

The first start creates the admin account and prints its password once, in a ruled box on standard output:

========================================================================
WEBHOOKER FIRST BOOT: an admin account has been created.

    username: admin
    password: 3xamPl3-p4ssw0rd

Save this password now: it is shown only here, and only once.
If it is lost, run `webhooker resetpw admin` on a stopped deployment.
========================================================================

Open http://127.0.0.1:8080/ and sign in. The server listens on 127.0.0.1 port 8080 by default.

To run it in Docker instead, build the image from the clone with make docker, which tags it webhooker. Then:

docker run -d \
  -p 127.0.0.1:8080:8080 \
  -v /path/to/data:/var/lib/webhooker \
  webhooker

In production, put a reverse proxy in front of webhooker: a server that accepts the public connections and passes them on. Read Behind a reverse proxy first.

Using the web interface

The sign-in page is /pages/login. There is one account, admin, and no way to add another through the web interface. A session lasts up to 7 days. It ends sooner after 24 hours without use, unless SESSION_IDLE_TIMEOUT says otherwise.

Webhooks

New Webhook on the webhook list at /hooks asks for a name, an optional description and a retention period in days. Events older than the retention period are deleted, along with their deliveries. The default is 30 days. 0 keeps events forever. Every new webhook gets one entrypoint, described as Default entrypoint.

Entrypoint URLs

The webhook page lists each entrypoint with its full URL and a Copy button. Give that URL to the sender. Anyone who holds it can post events, so treat it like a password.

Add creates another entrypoint with its own URL. Edit changes an entrypoint's description. Its URL never changes. To replace a URL, create a new entrypoint, point the sender at it, then deactivate or delete the old one. A deactivated entrypoint answers 410 Gone. A deleted or unknown one answers 404.

Targets

Add under Targets asks for the type first, then that type's settings.

• An http target posts each event to a URL, with optional request headers and a timeout.
• A slack target posts each event as a message to the message-posting URL Slack or Mattermost give out, which they call an incoming webhook.
• An archive target writes each event as a row in a SQLite file of its own. Its stored type is database. It takes an expiry, which is how long archived events are kept, and a rotation, which is how often a new archive file is started.
• A log target writes each event to the application log.

http and slack targets also take Delivery attempts: how many times each delivery is tried in all. The default is 1. The most is 20. A target's type cannot be changed. Deleting an archive target leaves its files on disk.

Events

The webhook page shows the 50 newest events. Each expands to show its body. A body is shown as text up to 32 KiB. A longer or binary body has a Download the body link. Full Event Log shows the 50 newest events with their deliveries and every attempt. Failed and Pending narrow it to events with a failed delivery, or with one still pending or retrying.

Replay sits beside each finished delivery. It creates a new delivery of the same event to the same target, with the target's settings as they stand now. Resubmit sits beside each event. It stores a copy of the event as a new event and delivers it to every target that is active now. Both actions are limited to 30 per minute per client.

Profile and settings

/user/{username} lets you change your password. /settings lists every configuration variable with the value the running server loaded; METRICS_PASSWORD and SENTRY_DSN show only as set or not set.

Deploying

Running with Docker

No image is published. make docker builds one from the clone and tags it webhooker. Run it as shown under Getting Started.

The image listens on every IPv4 address inside the container: it ships BIND_ADDRESS=0.0.0.0. The -p flag decides what the host exposes. Publish to 127.0.0.1:8080, the address your reverse proxy connects to, and nothing wider.

Do not set BIND_ADDRESS to an empty value in a container. An empty value means unset, and unset falls back to 127.0.0.1, so the container answers nothing on its published port.

The app runs as user webhooker. The image's start-up script, deploy/docker-entrypoint.sh, gives DATA_DIR and its contents to that user before the app starts. So a host directory can be mounted as it is. webhooker creates every SQLite file with mode 0600.

Running under upaas

upaas, a deployment tool, builds the image from this repository's Dockerfile and runs it. Add no port mapping, because upaas publishes every mapped port on all interfaces of the host. Put the app on your reverse proxy's Docker network instead, and mount one host directory at /var/lib/webhooker.

Behind a reverse proxy

webhooker speaks plain HTTP. It does not handle TLS, the encryption behind HTTPS: it holds no certificate and accepts no HTTPS connections. The reverse proxy holds the certificate and reaches webhooker over loopback, the address only this host can reach.

Four things must be right.

1. Keep the app port private. Only the proxy may reach it: 127.0.0.1 on a bare host, or a port published to 127.0.0.1 for the image.
2. Send X-Forwarded-Proto: https. This header tells webhooker the request arrived over TLS. It decides the Secure flag on cookies, and whether a form submission must carry a matching Origin or Referer. webhooker reads it from any connecting machine, so the proxy must overwrite whatever the client sent.
3. Cover the proxy in TRUSTED_PROXIES. Otherwise every client shares one rate-limit bucket, the count a limit keeps per client. The default list does not cover a proxy that connects over loopback.
4. Send Host as $http_host, not $host. webhooker checks a form's Origin against the Host header it receives. $host drops the port, so on any port other than 443 every form submission fails with 403.

Protect the proxy's access log: its request paths hold the entrypoint UUIDs.

An nginx server block for this, with TRUSTED_PROXIES=127.0.0.1 on webhooker's side:

server {
    listen 443 ssl;
    server_name webhooker.example.com;
    ssl_certificate     /etc/ssl/certs/webhooker.example.com.crt;
    ssl_certificate_key /etc/ssl/private/webhooker.example.com.key;
    client_max_body_size 1m;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host              $http_host;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 70s;
    }
}

The block matches webhooker's 1 MB body cap and its 60 second request timeout.

HSTS

HSTS is a response header that tells browsers to use HTTPS only for this hostname and every name under it, for max-age seconds. Every response carries it unless HSTS_MAX_AGE is 0. A browser keeps the rule until max-age runs out, so serve webhooker on a hostname you will keep on HTTPS.

Recovering a lost admin password

webhooker resetpw sets an existing account's password from the command line. The deployment must be stopped first: the command takes the same DATA_DIR lock the server does and refuses while a running instance holds it.

DATA_DIR=/var/lib/webhooker webhooker resetpw -generate admin

Without -generate, the new password is read from standard input, at least 8 characters. It is never taken as a command-line argument, because other accounts on the host can read those. In a container, run the same command with docker run --rm and the stopped container's volume mounted. The command creates nothing: a missing DATA_DIR, a missing webhooker.db or an unknown username is an error.

Backup and restore

All state lives in DATA_DIR. Back up the whole directory. It holds:

• the main database, webhooker.db: the session key, the admin account's password hash, and the webhooks, entrypoints and targets;
• one event database per webhook, events-{webhook_uuid}.db: its events, deliveries and attempts;
• the archive files, one per archive target;
• the lock file, webhooker.lock, which is always empty.

Every database runs in WAL mode. WAL is SQLite's write-ahead log: finished writes land in a -wal file beside the database before they reach the .db itself. A copy of the .db without its -wal is missing data. A clean shutdown removes the -wal and -shm files; a crash leaves them, and they must be carried with the .db. Keep DATA_DIR on a local disk: WAL mode is not reliable on network filesystems, and webhooker does not check for one.

A copy taken while the service runs can catch a .db and its -wal at two different instants. Stop the service, copy the directory, and start it again. Without downtime, copy each database with SQLite's own backup command instead, which writes one complete file:

for db in /path/to/data/*.db; do
    sqlite3 "$db" ".backup '/path/to/backup/$(basename "$db")'"
done

Each file is captured at its own instant. A webhook created during the backup can be in the copy of webhooker.db with no event database in the backup; a restore then creates that event database empty.

To restore, stop the service and put back every file together, with any -wal and -shm files the backup holds. A missing or empty database is created again when it is opened, and created a new, empty database is logged at WARN with its path. For webhooker.db that means the main database was lost, most often because the volume holding DATA_DIR is not mounted. For an event database it means that one file was lost; the webhook comes back with its settings but no history.

A backup holds every event's headers and body, and the targets' credentials in plain text. Protect it accordingly.

Security

The entrypoint URL is the credential

Each entrypoint is a random version 4 UUID served at /h/{uuid}. Holding that URL is what authorises a sender. The receiver checks nothing else: no shared secret, no HMAC signature (a hash of the body made with a shared key), no token header. Signature headers a sender adds anyway are stored and forwarded as ordinary headers.

Keep the URL out of tickets, chat and screenshots. webhooker's own log never writes it. The reverse proxy's access log does hold it, in the request path.

The web interface

• Passwords are hashed with Argon2id, a slow password hashing function.
• Session cookies are signed, HttpOnly (hidden from page scripts) and SameSite=Lax (not sent with cross-site form posts). They carry Secure when the request carries X-Forwarded-Proto: https.
• Every form is protected against cross-site request forgery (CSRF), where another site makes your browser submit a form here: each form carries a token tied to a cookie, and a submission's Origin must name this host.
• Form bodies are capped at 1 MB.
• Every response carries these headers:
  • Strict-Transport-Security, unless HSTS_MAX_AGE is 0;
  • X-Content-Type-Options: nosniff;
  • X-Frame-Options: DENY;
  • a Content-Security-Policy (CSP), the header that tells the browser where scripts and styles may come from, set to default-src 'self'; script-src 'self'; style-src 'self'.
• /metrics sits behind HTTP basic authentication, a username and password sent with each request.

Sign-in limits

The password is checked first, and only a failed sign-in is counted. A correct password is never throttled. Failures are counted per client and submitted username. From the fifth failure within a minute, each failure is answered 429. At most two password checks run at once; a request that gets no turn within 5 seconds is answered 503. So those two checks, not the failure count, bound how fast passwords can be guessed. Choose a long random admin password, or rate-limit POST /pages/login at the reverse proxy.

Delivery destinations

A delivery target could be used to make webhooker send requests into the network it sits in and read the answers back. This is called server-side request forgery (SSRF). webhooker refuses it with the SSRF guard.

Every target URL must use http or https and resolve to a public address. The SSRF guard refuses these ranges:

• loopback, 127.0.0.0/8 and ::1;
• the RFC 1918 private ranges, 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16;
• link-local space, which is valid only on one network segment;
• carrier-grade NAT space, which internet providers use inside their own networks;
• multicast, the IPv6 unique local range, and the other reserved ranges.

The SSRF guard runs when a target is saved and again when each delivery connects, so a hostname that changes its address later is still checked. Redirects are followed up to ten hops, and each hop is checked. A hop to another host or port, or from https to http, carries none of the configured or forwarded headers, so a redirect cannot pass a credential to another host.

Egress means the connections webhooker opens outward to deliver. ALLOWED_EGRESS_CIDRS opens named networks to it.

What the logs contain

webhooker writes one access log record per request to standard output, as plain text on a terminal and as JSON otherwise. The access log never records the query string. On the receiver route it records the URL as /h/{uuid}, never the real path, so entrypoint UUIDs stay out of the log.

The admin password is printed in the clear on the first boot and by webhooker resetpw -generate. A log target writes the whole event to the log: query string, headers and body.

When SENTRY_DSN is set, error reports go to Sentry, an error reporting service. A report carries no query string, body or cookies, and the route pattern (the route as registered, such as /h/{uuid}) in place of the path.

Design

webhooker is one process. An HTTP server receives requests, a delivery engine sends them on in the background, and SQLite holds everything in between.

The receiver stores the event and one pending delivery per active target in one database write, then queues the deliveries. Background workers run them and record each attempt. At start and then every minute, a recovery routine queues again any delivery left pending or retrying, so a stop or a crash loses no delivery.

The main database holds configuration. Each webhook has its own event database, so writes for one webhook never wait on another webhook's database. The Go code is under internal/, one package per concern, and the pages' HTML, CSS and JavaScript are embedded in the binary.

Configuration reference

All configuration comes from environment variables. For development, variables can also go in a .env file in the working directory. A variable set in the real environment wins over the file. A .env file that cannot be parsed aborts startup.

A variable that is unset or empty takes its default. A variable that is set but cannot be parsed aborts startup with a message naming it. A Go duration is a number with a unit, such as 30s, 15m or 24h.

WEBHOOKER_ENVIRONMENT

dev or prod. Default prod. This controls one thing: CORS, the headers that let a page on another site call this one from a browser. In dev, every origin is allowed. In prod, no CORS headers are sent.

PORT

The TCP port the listener binds. Default 8080.

BIND_ADDRESS

The IP address the listener binds. Default 127.0.0.1. The Docker image sets 0.0.0.0. It must be an IP address literal: localhost and a value with a port are refused. 0.0.0.0 listens on IPv4 only; :: listens on IPv6 and IPv4.

DATA_DIR

The directory for every SQLite database and the lock file. Default /var/lib/webhooker. It is created if it is missing.

Exactly one webhooker process may use a DATA_DIR at a time; two would both run the recovery routine and send every pending delivery twice. At startup the process locks {DATA_DIR}/webhooker.lock with flock, the kernel's file lock, and a second process pointed at the same directory exits with a message naming it.

DEBUG

true or false. Default false. It accepts the spellings Go's strconv.ParseBool accepts; yes and on are refused. DEBUG=true lowers the log level to DEBUG, which adds every SQL statement, with ? in place of each value, and other diagnostic lines.

METRICS_USERNAME and METRICS_PASSWORD

The basic authentication credentials for /metrics. Default empty. Set both or neither. With both set, /metrics is served behind basic authentication. With neither set, the route does not exist. With only one set, startup fails.

SENTRY_DSN

The DSN for Sentry error reporting: the URL that tells the Sentry library where to send reports. Default empty, which leaves reporting off. A value the Sentry library cannot parse aborts startup.

RETENTION_SWEEP_INTERVAL

How often two background jobs run. The retention sweep deletes events older than each webhook's retention period, with their deliveries. The archive sweep deletes archive rows older than their archive target's expiry. A Go duration greater than zero. Default 1h.

SESSION_IDLE_TIMEOUT

How long a session may go unused before it ends. A Go duration. Default 24h. A zero or negative value disables idle expiry. The 7 day absolute limit always applies.

HSTS_MAX_AGE

The max-age of the Strict-Transport-Security header, as a Go duration in whole seconds. Default 87600h (ten years). 0 sends no header. Read HSTS before changing it.

RECEIVER_RATE_LIMIT

How many requests per minute one client may send to one entrypoint. Default 120. A client over the limit is answered 429. A second limit of ten times this value covers the whole /h/ route per client.

TRUSTED_PROXIES

A comma-separated list of networks in CIDR notation, which names a network as an address and a prefix length, such as 10.0.0.0/8. A bare address counts as a single host. Default 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16, the RFC 1918 private ranges. A set value replaces the default.

The list decides whose X-Forwarded-For header is believed. X-Forwarded-For is the header a proxy uses to pass on the address of the client it is serving. It is read only when the connecting machine is inside the list. X-Real-IP and True-Client-IP are never read. If any client can reach webhooker, or the proxy in front of it, from a private address, set the list to the proxy's address alone. Otherwise those clients can choose their own X-Forwarded-For and bypass every rate limit.

Allowing egress to your own network

ALLOWED_EGRESS_CIDRS is a comma-separated list of networks in CIDR notation that delivery targets may reach although the SSRF guard refuses them by default. Default empty. Example: 10.0.0.0/8, 172.17.0.0/16. When the list is set, startup logs it at WARN.

Anyone with the admin account can make webhooker send requests to every network you list, and read the responses in the event log. List the smallest blocks that cover your destinations, and never 0.0.0.0/0 or ::/0.

These addresses stay blocked whatever you list. They are link-local space, the unspecified addresses, and cloud metadata endpoints, the addresses a cloud gives each virtual machine for its own credentials:

• 169.254.0.0/16 and fe80::/10;
• fd00:ec2::254/128 and fd00:ec2::23/128;
• fd20:ce::254/128;
• fd00:c1::a9fe:a9fe/128;
• fd00:42::42/128;
• fd00:a9fe:a9fe::1/128;
• 100.100.100.200/32;
• 192.0.0.192/32;
• 0.0.0.0/32 and ::/128;
• ::a9fe:a9fe/128 and 64:ff9b::a9fe:a9fe/128.

This list is not every cloud's metadata address. If yours is not here, do not list the block that contains it. Azure's public 168.63.129.16 is refused by default but is reopened by a list that covers it.

Delivery reference

An http target posts the event body to its URL with the event's Content-Type and headers, then the target's configured headers, which replace event headers of the same name. A 2xx response is a success. The first 4 KiB of the response body is stored with the attempt. timeout_seconds is at most 300, and blank means the 30 second default. forward_query appends the event's query string to the URL.

A slack target posts a message that describes the event and shows its body in a code block, cut at 3500 bytes. An archive target, whose stored type is database, appends the event as a row to its own SQLite file. A log target writes the event as one INFO line.

http and slack targets take max_attempts, from 1 to 20. A failed attempt marks the delivery retrying and schedules the next attempt; the wait doubles each time, starting at one second. After the last allowed attempt fails, the delivery is marked failed.

A target with max_attempts above 1 has a circuit breaker, which stops hammering a destination that is down. After 5 failures in a row, deliveries to the target pause for 30 seconds. Then one delivery is sent as a probe; the others wait until a probe succeeds. The webhook page says when a target is paused.

Archives

An archive target writes to archive-{webhook_name}-{target_name}-{target_uuid}.db in DATA_DIR, the two names made safe for a file name. A rotation of monthly, daily or hourly starts a new file for each period, in UTC, with the period before .db, such as -2026-10-01.db. Rows older than the expiry are deleted on each write and by the archive sweep. Renaming the webhook or the target renames the files. Download on the webhook page returns the target's whole archive as one gzipped JSON file.

HTTP routes reference

• GET /.well-known/healthcheck returns JSON with the version and the uptime.
• POST /h/{uuid} is the receiver. Its answers:
  • any method but POST gets 405;
  • a client over a rate limit gets 429;
  • an unknown UUID gets 404;
  • a deactivated entrypoint gets 410;
  • a body over 1 MB gets 413;
  • a failure to store the event gets 500;
  • a stored event gets 200 with {"status":"ok"}.
• GET /pages/login shows the sign-in form, POST /pages/login signs in, and POST /pages/logout signs out.
• The pages behind the sign-in live under /user/, /settings, /hooks and /hook/. A GET to one of them without a session is redirected to /pages/login.
• GET /metrics exists only when METRICS_USERNAME and METRICS_PASSWORD are set.

Metrics reference

/metrics serves these metrics for Prometheus, the monitoring system, plus the Go runtime and process metrics the Prometheus library provides. Each delivery metric carries one label, target_type, which is http, slack, database, log or unknown.

• webhooker_events_received_total and webhooker_events_resubmitted_total, two counters (totals that only grow) with no label;
• webhooker_delivery_attempts_total;
• webhooker_deliveries_succeeded_total and webhooker_deliveries_failed_total;
• webhooker_delivery_retries_total and webhooker_delivery_replays_total;
• webhooker_delivery_duration_seconds, a histogram (a distribution) of one attempt's wall time;
• webhooker_deliveries_pending and webhooker_deliveries_retrying, gauges (current values) sampled every 30 seconds;
• webhooker_circuit_breakers_open.

Three inbound HTTP metrics, named http_..., carry the route pattern as their handler label, never the real path.

Development

You need the Go version named in go.mod, and Docker. The tests, the linters, prettier and the stylesheet build run in Docker, at versions pinned (fixed to one exact release) in the Dockerfile, package.json and yarn.lock. Only gofmt and goimports run on the host.

make bootstrap    # Install git, make, Go and the Go modules
make dev          # go run ./cmd/webhooker
make check        # test, lint, fmt-check and css-check
make test         # Run the tests in Docker
make lint         # Run gofmt, golangci-lint and ESLint in Docker
make fmt          # Format Go and Markdown
make build        # Build bin/webhooker, version-stamped
make docker       # Build the Docker image

Each make target except build, run, dev, deps and clean runs a script in script/, listed under Entrypoints. Every docker build in script/ passes --no-cache, so a check that passes has really run.

The binary's version comes from script/version: git describe on the checkout, or VERSION when that is set. It is shown at /.well-known/healthcheck and in the page footer.

The pages use one third-party script, Alpine.js, in its CSP build @alpinejs/csp, which runs under a Content-Security-Policy that forbids eval. The package file in 3p/ is byte for byte the one the npm registry, the JavaScript package registry, publishes.

A browser test in internal/server loads the pages in a headless browser, one without a window, under the real Content-Security-Policy. make test leaves it out; run make test-browser after changing templates/ or static/js/. static/css/tailwind.css is generated by Tailwind and committed; make check fails when it is out of date, and make css regenerates it.

Entrypoints

This repository adheres to the Scripts to Rule Them All standard: the scripts in script/ are the entry points for the development workflow. This section is about those scripts. The receiving URLs of the application are under Entrypoint URLs.

• script/bootstrap installs git, make, Go and the Go modules.
• script/setup runs script/bootstrap, then script/install-precommit.
• script/projectname prints webhooker.
• script/assets extracts Alpine.js from 3p/ to static/js/alpine.min.js.
• script/test runs the tests in Docker.
• script/test-browser runs the browser test in Docker.
• script/lint runs gofmt, golangci-lint and ESLint in Docker.
• script/fmt formats the Go code on the host and the Markdown in Docker.
• script/fmt-check checks the same and fails instead of writing.
• script/css regenerates static/css/tailwind.css in Docker.
• script/css-check fails when static/css/tailwind.css is out of date.
• script/check runs script/test, script/lint, script/fmt-check and script/css-check.
• script/version prints the version to stamp into the binary.
• script/docker builds the image tagged webhooker.
• script/cibuild is what CI, the check run on every push, runs: script/bootstrap, script/check, then the image build.
• script/precommit checks that go mod tidy changes nothing, then runs script/check.
• script/install-precommit installs the git pre-commit hook that runs script/precommit.

TODO

See TODO.md.

License

MIT.

Author

@sneak