webhooker

webhooker is an MIT-licensed web service, written in Go by @sneak, that receives the HTTP notifications other services send and forwards them to your own systems.

Many services send an HTTP request to a URL you give them when something happens: a code push, a payment, a new ticket. webhooker receives those requests, stores them, and forwards them to the places you choose.

webhooker is a single binary with a web interface. It stores its data in SQLite, a database engine that keeps each database in one ordinary file. It writes each notification to disk before answering the sender. It then delivers it to each target you configured. The web interface shows the newest notifications, with every delivery and attempt they had. From there you can send a stored notification again. webhooker also exports metrics for Prometheus, the monitoring system, so you can watch the whole flow.

Rationale

Senders of these notifications usually make one attempt and keep no record. If your service is down at that moment, the notification is lost. Few senders let you see what they sent, or when, or whether it arrived.

webhooker sits between the sender and your service and fixes those three things. It is always ready to receive, and it stores before it forwards, so nothing is lost while your service is down. It keeps every notification and every delivery attempt, so you can read what was sent and what happened to it. It can forward one notification to several targets, and it can send a stored one again.

How it works

Five words name the moving parts.

• A webhook is a named configuration. It groups entrypoints and targets, and it owns the events it receives.
• An entrypoint is a receiving URL of the form /h/{uuid}. Senders post to it. The UUID is a random 36-character identifier, and it is the only thing that authorises a sender.
• A target is a place events are delivered to. There are four kinds: an HTTP URL, a Slack or Mattermost message-posting URL, an archive database file, and the application log.
• An event is one stored notification: its method, query string, headers and body.
• A delivery is one event paired with one target. It records every attempt.

A request to an entrypoint is stored as an event. In the same database transaction, one pending delivery is created for each active target of the webhook. The sender gets 200 once that transaction has committed. Workers then carry out the deliveries. A delivery to an HTTP or Slack target that fails is retried, up to the number of attempts the target allows.

Delivery is at least once, not exactly once. If a delivery reaches the target but webhooker cannot record that outcome, it sends the delivery again later. A delivery in flight when the service stops is also sent again at the next start. webhooker adds no delivery identifier of its own to the request, so make your targets tolerate a duplicate.

Getting Started

You need the Go version named in go.mod, and Docker for the checks.

git clone https://git.eeqj.de/sneak/webhooker.git
cd webhooker
make bootstrap
DATA_DIR=../webhooker-data make dev

DATA_DIR is where webhooker keeps its databases. It defaults to /var/lib/webhooker in every environment, so point it at a directory you can write to. Keep it outside the clone: the databases hold the session key.

The first start creates the admin account and prints its password once, in a ruled box on standard output:

========================================================================
WEBHOOKER FIRST BOOT: an admin account has been created.

    username: admin
    password: 3xamPl3-p4ssw0rd

Save this password now: it is shown only here, and only once.
If it is lost, run `webhooker resetpw admin` on a stopped deployment.
========================================================================

Open http://127.0.0.1:8080/ and sign in. The server listens on 127.0.0.1 port 8080 by default.

To run it in Docker instead, build the image from the clone with make docker, which tags it webhooker. Then:

docker run -d \
  -p 127.0.0.1:8080:8080 \
  -v /path/to/data:/var/lib/webhooker \
  webhooker

Read Running with Docker before publishing the port anywhere but 127.0.0.1.

Using the web interface

Signing in

The sign-in page is /pages/login. There is one account, admin, and no way to add another through the web interface. A session lasts up to 7 days. It ends sooner after 24 hours without use, unless SESSION_IDLE_TIMEOUT says otherwise.

If a page asked for a sign-in, signing in returns you to that page.

Creating a webhook

The webhook list is at /hooks. New Webhook asks for a name, an optional description and a retention period in days. Events older than the retention period are deleted, along with their deliveries. The default is 30 days. 0 keeps events forever.

The form can also create the first targets. An HTTP target URL creates an http target named HTTP. The Archive checkbox creates an archive target named Archive, with the expiry and rotation chosen beside it.

Every new webhook gets one entrypoint, described as Default entrypoint.

Entrypoint URLs

The webhook page lists each entrypoint with its full URL and a Copy button. Give that URL to the sender. Anyone who holds it can post events, so treat it like a password. Read The entrypoint URL is the credential.

Add creates another entrypoint with its own URL. Several entrypoints on one webhook let you tell senders apart while delivering to the same targets.

Edit changes an entrypoint's description. Its URL never changes. To retire a URL, deactivate the entrypoint or delete it, then create a new one.

A deactivated entrypoint answers 410 Gone. A deleted or unknown one answers 404.

Beside each entrypoint the page shows when its last event arrived, and how many events arrived on it within the retention period.

Targets

Add under Targets asks for the type first, then that type's settings.

• An http target posts each event to a URL. It takes optional request headers, a timeout in seconds, and a switch to pass the event's query string on.
• A slack target posts each event as a message to Slack or Mattermost. It takes the message-posting URL those services give out, which they call an incoming webhook.
• An archive target writes each event as a row in a SQLite file of its own. Its stored type is database. It takes an expiry and a rotation.
• A log target writes each event to the application log. It has no settings beyond its name.

http and slack targets also take Delivery attempts: how many times each delivery is tried in all. The default is 1. The most is 20.

The target list shows each target's settings. A destination URL is shown as its scheme and host, with /... standing for its path. Headers are shown as a count. The target edit page is the one place that shows a URL and its headers in full.

The list also shows, for each target, how many deliveries have ever succeeded and failed, and how many did so in the last 24 hours. For an archive target it names the current archive file, the size of all its files together, and when the last write happened.

Deactivate stops new events reaching the target. Deliveries already created are unaffected. Delete removes the target. An archive target's files stay on disk.

A target's type cannot be changed. Create a new target to deliver another way.

Statistics

The top of the webhook page shows counts of entrypoints and targets, deliveries in progress, the time of the last event, and the retention setting.

Below that are events, deliveries and failures, both lifetime and within retention. Next are figures for the last 10 minutes and the last 24 hours: events, failures and the failure percentage. The percentage is failed deliveries out of all deliveries that finished in the window.

The webhook list shows three of these figures per webhook: events within retention, the last event, and failed deliveries in the last 24 hours.

Recent events and the event log

The webhook page shows the 50 newest events. Each expands to show its body. A body is shown as text up to 32 KiB. A longer or binary body has a Download the body link to the stored bytes.

Full Event Log at /hook/{id}/events shows the 50 newest events with their deliveries and every attempt. Failed narrows it to events with a failed delivery. Pending narrows it to events with a delivery still pending or retrying. Each attempt shows its time, status code, duration, error, and the first 4 KiB of the response.

Each event has a page of its own at /hook/{id}/events/{eventID}. It shows the whole request: query string, headers, body and every delivery.

Replay and resubmit

Replay sits beside each finished delivery. It creates a new delivery of the same event to the same target and sends it on the ordinary path. The target's settings as they stand now are used, so a replay is how you resend after fixing the destination. The original delivery is left as it was. The new one is marked as a replay.

A replay is refused in three cases:

• the delivery has not finished;
• its target is deactivated or deleted;
• another delivery of that event to that target is already in flight.

A delivery whose target was deleted shows no Replay button.

Resubmit sits beside each event. It stores a copy of the event as a new event and delivers it to every target that is active now. That includes targets created after the original arrived. The copy's row names the event it came from. The original's row shows how many copies were made. Resubmitting the same event many times is allowed.

Both actions are limited to 30 per minute per client.

Profile and settings

/user/{username} shows your account and lets you change your password. The current password is checked first. The new password must not be empty and must match its confirmation. Password changes are limited to 5 per minute per client.

/settings lists every configuration variable with the value the running server loaded. It is read only. METRICS_PASSWORD and SENTRY_DSN show only as set or not set.

Deploying

Running with Docker

No image is published. make docker builds one from the clone and tags it webhooker. Then:

docker run -d \
  -p 127.0.0.1:8080:8080 \
  -v /path/to/data:/var/lib/webhooker \
  webhooker

The image listens on every IPv4 address inside the container: it ships BIND_ADDRESS=0.0.0.0. A container has its own network namespace, which means its own addresses, separate from the host's. That is the boundary, and the -p flag decides what the host exposes. Publish to 127.0.0.1:8080, the address your reverse proxy connects from, and nothing wider. A bare -p 8080:8080 opens the port on every interface of the host, and Docker's forwarding rules sit ahead of most host firewalls.

Do not set BIND_ADDRESS to an empty value in a container. An empty value means unset, and unset falls back to the binary's default of 127.0.0.1. The container then answers nothing on its published port and its health check fails. A templated Compose file with an empty value is the usual way this happens.

A container on an IPv6-only Docker network needs BIND_ADDRESS=::.

The app runs as user webhooker (UID 1000). The image's entrypoint script starts as root. It creates DATA_DIR if it is missing, gives it and its contents to webhooker, and sets the directory to mode 0750. Then it runs the app as webhooker. So a host directory can be mounted as it is, whoever owns it. With --user, the script changes nothing and runs the app as that user.

webhooker creates every SQLite file with mode 0600, and resets a file found with another mode when it opens it. webhooker.db holds target settings in plain text, including bearer tokens and Slack URLs, so the file mode is what protects them.

The image's health check requests /.well-known/healthcheck on the container's own address every 30 seconds.

The same binary runs the password reset. The image's CMD is /app/webhooker, and a command given to docker run replaces all of it:

docker run --rm -v webhooker-data:/var/lib/webhooker \
    webhooker /app/webhooker resetpw -generate admin

Running under upaas

upaas, a deployment tool, builds the image from this repository's Dockerfile and runs it.

• Add no port mapping. upaas publishes every mapped port on all interfaces of the host. Instead, set the app's Docker network to your reverse proxy's network. The proxy then reaches the app at upaas- followed by the app name, on port 8080.
• Leave PORT, BIND_ADDRESS and DATA_DIR unset. The image's health check probes port 8080.
• Mount one host directory at /var/lib/webhooker.
• Set TRUSTED_PROXIES to your reverse proxy's address if clients can reach the proxy from a private address. See TRUSTED_PROXIES.

The first start prints the admin password to the container's log. For an app named webhooker:

docker logs upaas-webhooker

If the password is lost, stop the container, reset the password with the app's own image and volume, and start it again:

docker stop upaas-webhooker
docker run --rm --volumes-from upaas-webhooker \
    "$(docker inspect -f '{{.Image}}' upaas-webhooker)" \
    /app/webhooker resetpw -generate admin
docker start upaas-webhooker

Behind a reverse proxy

webhooker speaks plain HTTP. It does not terminate TLS, the encryption behind HTTPS. Put a reverse proxy in front of it: a server that accepts the public connections and passes them on. The proxy holds the certificate and reaches webhooker over loopback, the address only this host can reach.

Five things must be right.

1. Keep the app port private. The binary binds 127.0.0.1 by default. The image binds every IPv4 address inside its container, so publish its port to 127.0.0.1 only. The plain port serves the sign-in form and the receiver without TLS.
2. Send X-Forwarded-Proto: https. This header tells webhooker the request arrived over TLS. It decides the Secure flag on cookies and the strict origin check on forms. Without it, cookies are sent without Secure. webhooker reads this header from any peer, so the proxy must overwrite whatever the client sent.
3. Cover the proxy in TRUSTED_PROXIES. Otherwise every client shares one rate-limit bucket. A bucket is the count a limit keeps per client, and here it would be keyed on the proxy's address. The default list is the private ranges, which does not cover a proxy that connects over loopback.
4. Send Host as $http_host, not $host. $host drops the port. The form origin check compares against the host it was given. On any port other than 443 every form submission then fails with 403.
5. Keep the proxy's access log. It records the client address whatever TRUSTED_PROXIES says. It also records the full request path, which holds the entrypoint UUID of every event. Protect that log as you would the entrypoint URLs.

WEBHOOKER_ENVIRONMENT decides none of this. It controls only CORS, the headers that let a page on another site call this one from a browser. Leave it unset in production.

nginx

A complete server block. Replace the server_name and the certificate paths.

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;  # nginx 1.25.1+; older: listen 443 ssl http2;

    server_name webhooker.example.com;

    ssl_certificate     /etc/ssl/certs/webhooker.example.com.crt;
    ssl_certificate_key /etc/ssl/private/webhooker.example.com.key;
    ssl_protocols       TLSv1.2 TLSv1.3;

    # webhooker caps form and receiver bodies at 1 MB.
    client_max_body_size 1m;

    # The request path holds the entrypoint UUID. Protect this
    # file like the entrypoint URL itself.
    access_log /var/log/nginx/webhooker.access.log combined;

    location / {
        # A literal address, not localhost: localhost may resolve
        # to ::1 first, where webhooker is not listening.
        proxy_pass http://127.0.0.1:8080;

        # $http_host, not $host: $host drops the port.
        proxy_set_header Host              $http_host;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Above webhooker's own 60 second request timeout, so its
        # 504 reaches the client before nginx gives up.
        proxy_read_timeout 70s;
    }
}

server {
    listen 80;
    listen [::]:80;
    server_name webhooker.example.com;
    return 308 https://$host$request_uri;
}

With that block, webhooker's environment is:

BIND_ADDRESS=127.0.0.1   # the default
TRUSTED_PROXIES=127.0.0.1

If nginx runs on another host, BIND_ADDRESS becomes the address nginx connects to, and TRUSTED_PROXIES becomes nginx's address. Firewall the port to that address: the traffic between them is plain text.

HSTS

HSTS is a response header that tells browsers to use HTTPS only for this hostname. Every response carries it unless HSTS_MAX_AGE is 0. With the default of ten years it reads:

Strict-Transport-Security: max-age=315360000; includeSubDomains; preload

A browser that receives this header over HTTPS refuses plain HTTP to the hostname until max-age runs out. Browsers ignore the header over plain HTTP. So serve webhooker on a hostname you will keep on HTTPS.

includeSubDomains extends the rule to every name under the hostname. preload invites browsers to put the domain on their built-in HSTS lists. The header alone lists nothing. A listing happens when someone submits a registered domain at hstspreload.org, and removal takes months.

Setting HSTS_MAX_AGE=0 stops the header. Browsers that already received it keep the rule until their max-age runs out. To end the rule sooner, send a short max-age such as 1s until returning browsers have picked it up.

Recovering a lost admin password

webhooker resetpw sets an existing account's password from the command line. The deployment must be stopped first: the command takes the same DATA_DIR lock the server does and refuses while a running instance holds it.

# Generate a new password and print it.
DATA_DIR=/var/lib/webhooker webhooker resetpw -generate admin

# Or supply one on standard input (at least 8 characters).
printf '%s' "$NEW_PASSWORD" | \
    DATA_DIR=/var/lib/webhooker webhooker resetpw admin

The password is never taken as a command-line argument, because on Linux every account on the host can read a running process's arguments. Standard input is echoed when it is a terminal, and the prompt says so.

The command creates nothing. A DATA_DIR that does not exist, or that holds no webhooker.db, is an error. An unknown username is an error. The new password takes effect at the next sign-in. Sessions already signed in are unaffected.

webhooker help lists the subcommands.

Backup and restore

What to back up

All state lives in DATA_DIR. Back up the whole directory. It holds:

• webhooker.db, one per install. Settings, the session key, users, webhooks, entrypoints and targets.
• events-{webhook_uuid}.db, one per webhook. Events, deliveries and delivery attempts.
• archive-{webhook_name}-{target_name}-{target_uuid}.db, one per archive target, or one per period for a target that rotates. The period goes before .db.
• webhooker.lock, which is always empty and holds no state.

Every database runs in WAL mode. WAL is SQLite's write-ahead log: committed changes land in a -wal file beside the database before they reach the .db itself. So a running service has two more files beside each {name}.db: {name}.db-wal, the log itself, and {name}.db-shm, the shared index SQLite keeps for that log. The -wal file is part of the database. A copy of the .db without it is missing data and may have no readable tables at all.

A clean shutdown closes every database and removes its -wal and -shm files. A crash leaves them, and they must be carried with the .db.

DATA_DIR must be on a local filesystem. SQLite maps the -shm file into memory, which network filesystems do not support reliably. webhooker does not check this. An error reading disk I/O error (522) points at the storage under DATA_DIR.

Configuration is not in DATA_DIR. It comes from the environment and from a .env file in the working directory. Back those up with your deployment.

A running copy is not safe

Databases are held open for the life of the process. cp, rsync, tar or a filesystem snapshot against a running service can catch a .db and its -wal at two different instants. Use one of these two procedures.

Stop, copy, start. This is the only way to get one point in time across every file:

docker stop webhooker
cp -a /path/to/data /path/to/backup-$(date -u +%Y%m%dT%H%M%SZ)
docker start webhooker

SQLite online backup, one file at a time, with no downtime:

for db in /path/to/data/*.db; do
    sqlite3 "$db" ".backup '/path/to/backup/$(basename "$db")'"
done

.backup reads through the WAL and writes one complete file. The runtime image has no sqlite3, so run this on the host or from another container with the volume mounted. Each file is captured at its own instant, so an event stored between two copies lands in one file and not the other.

sqlite3 <db> .dump is an export, not a backup. It is safe to run but holds a read transaction open for its whole duration.

Archive files are the exception. The archive writer closes and reopens its file after a write, at most once per second. So you can move a finished archive file away while the service runs. Move its -wal and -shm with it, or wait until there are none. The next write recreates a missing archive file.

Restore

1. Stop the service.
2. Restore webhooker.db, every events-*.db and every archive-*.db together. A missing events-{uuid}.db is created empty at the next start, and the webhook comes back with its settings but no history. That start logs created a new, empty database at WARN with the file's path. A missing archive file is recreated silently at the next write.
3. Carry any -wal and -shm files that are in the backup.
4. Start the service. The container fixes the owner of the restored files.

Backups contain secrets

Treat a backup like the credentials inside it. Encrypt it at rest and restrict who can read it.

• The event and archive databases hold the full headers and body of every event, including whatever the sender put in them.
• webhooker.db holds target settings unencrypted, next to the session key and the password hashes.

Security

The entrypoint URL is the credential

Each entrypoint is a random version 4 UUID served at /h/{uuid}. Holding that URL is what authorises a sender. The receiver checks nothing else: no shared secret, no HMAC signature (a hash of the body made with a shared key), no token header. None will be added. Signature headers a sender adds anyway, such as X-Hub-Signature, are stored and forwarded as ordinary headers.

What that means in practice:

• Keep the URL out of tickets, chat and screenshots. webhooker's own log never writes it, except in the line a log target writes. The reverse proxy's access log does hold it, in the request path.
• To rotate, create a new entrypoint and point the sender at it. Then deactivate or delete the old one.
• Some senders cannot be given a secret URL: they post to one fixed address and prove themselves with a signature instead. webhooker checks no signatures, so such a sender cannot use it. Pick another integration path for it.

The web interface

• Passwords are hashed with Argon2id, a password hashing function designed to be slow and to need a large amount of memory. The parameters are:
  • 64 MB of memory;
  • 1 pass;
  • 4 threads;
  • a 32-byte key;
  • a 16-byte salt.
• Session cookies are signed, HttpOnly (hidden from page scripts) and SameSite=Lax (not sent with cross-site form posts). They carry Secure when the request arrived over TLS, judged per request from a direct TLS connection or X-Forwarded-Proto: https.
• A new session is issued at sign-in. This prevents session fixation, an attack where a victim is made to use a session the attacker already knows.
• The key that signs the cookies is 32 random bytes made on first start and stored in webhooker.db.
• Every form is protected against cross-site request forgery (CSRF), where another site makes your browser submit a form here. Each form carries a token tied to a cookie. A submission with an Origin header must name this host. Over TLS, a submission without Origin must carry a Referer that names this host.
• Form bodies are capped at 1 MB. A larger declared body is answered 413 before anything reads it.
• Every response carries these headers:
  • Strict-Transport-Security, unless HSTS_MAX_AGE is 0;
  • X-Content-Type-Options: nosniff;
  • X-Frame-Options: DENY;
  • Content-Security-Policy (CSP), which tells the browser where scripts and styles may come from, set to default-src 'self'; script-src 'self'; style-src 'self';
  • Referrer-Policy: strict-origin-when-cross-origin;
  • a Permissions-Policy that denies camera, microphone and location.
• Authenticated pages are sent with Cache-Control: no-store.
• /metrics sits behind HTTP basic authentication, a username and password sent with each request.

Sign-in limits

The sign-in form has no rate limiter in front of it, on purpose. Behind a proxy that TRUSTED_PROXIES does not cover, every client shares one bucket. A limit that counted every request would let any stranger lock the operator out with a few requests a minute.

Instead, the password is checked first, and only a failure is counted. A correct password is never throttled. Failures are counted per client and submitted username. From the fifth failure within a minute, each failure is answered 429 with Retry-After. A successful sign-in clears the count. At most 1024 username counters and 1024 client counters are kept.

Two password checks can run at once; each running check holds one of two slots. Each check takes 64 MB of memory, so hashing is capped at 128 MB. A request that gets no slot within 5 seconds is answered 503 without hashing anything. So is one that arrives when 16 requests are already waiting for a slot. The password change form shares these slots.

The check runs before the count, so the two slots, not the failure count, bound how fast passwords can be guessed. A flood of wrong passwords is tried as fast as two Argon2id checks allow. Choose a long random admin password. To put a cheaper limit on top, rate-limit POST /pages/login at the reverse proxy, where the real client address is visible.

An unknown username is checked against a placeholder hash, so it costs the same time as a real one.

A flood can hold both slots and make sign-in answer 503 for as long as it runs. Nothing accumulates: the operator's correct password works on the first try once the flood stops. Restarting webhooker is no remedy. Block the source at the proxy.

Delivery destinations

A delivery target could be used to make webhooker send requests into the network it sits in and read the answers back. This is called server-side request forgery. webhooker refuses it by default, with a check called the destination check below.

Every target URL must use http or https and resolve to a public address. These private and reserved ranges are refused:

• loopback, 127.0.0.0/8 and ::1;
• the RFC 1918 private ranges, 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16;
• carrier-grade NAT space, 100.64.0.0/10, which internet providers use inside their own networks;
• link-local space, 169.254.0.0/16 and fe80::/10, which is valid only on one network segment;
• the IPv6 unique local range, fc00::/7;
• multicast, the ranges reserved for documentation, and the other reserved ranges.

One public address is refused too: 168.63.129.16, the Azure service that hands a virtual machine its credentials.

The destination check runs when a target is saved and again when each delivery connects. Both use the same function. The address is resolved again at connect time, so a hostname that changes its address after validation is still checked.

Redirects are followed up to ten hops. Each hop is checked the same way. A hop that leaves the target's origin carries none of the target's configured headers and none of the forwarded event headers. Leaving the origin means a different host, a different port, or a step from https to http. This stops an open redirect, a destination that forwards requests to any URL a caller names, from handing a credential to another host.

ALLOWED_EGRESS_CIDRS opens named networks for the common case of forwarding to your own network. Read Allowing egress to your own network before setting it.

What the logs contain

webhooker writes structured logs, one record per line with named fields, to standard output: plain text on a terminal, JSON otherwise. Each request writes one access log line, http request, at INFO. A request to a known entrypoint writes a second line, webhook request received. A stored event writes a third, webhook event created.

Fields a client can influence are cut to a fixed size, so one access log line is at most 2,560 bytes. The access log never records the query string. On the receiver route the URL is logged as /h/{uuid}, never the real path, so entrypoint UUIDs stay out of the log. On other routes a 3xx or 4xx response logs the route pattern, which is the route as registered, such as /user/{username}. A 2xx or 5xx logs the real path. The Referer is logged as its scheme and host only.

Every line that names a client carries two fields. remoteIP is the connecting peer, which behind a proxy is the proxy. clientIP is the client as decided by TRUSTED_PROXIES.

SQL statements are logged with ? in place of each value, never with the values themselves. Session cookies, API keys and target credentials are never logged.

Two things are logged in the clear. The first is the admin password, once, on the first boot, and again when webhooker resetpw -generate is run. The second is the operator's own configuration echoed back: webhook names and target hostnames.

A log target writes the whole event to the log: query string, headers and body. That is its purpose.

A recovered panic is a crash inside one request that the server caught. It writes one ERROR record of at most 10,240 bytes. The record carries the request id of the request that caused it. A request id is a random token given to each request and written in its log lines.

When SENTRY_DSN is set, error reports go to Sentry, an error reporting service, with the request reduced to this:

• the URL with its path replaced by the route pattern;
• the query string and the body replaced by (redacted);
• no cookies;
• only these headers: Accept, Content-Length, Content-Type, Host, Origin, Referer (scheme and host only), User-Agent and X-Request-Id.

Design

Request flow

One request to an entrypoint goes through these steps:

1. The router matches POST /h/{uuid}. Any other method gets 405.
2. The rate limits run: one per client per entrypoint, one per client across the whole route.
3. The handler looks up the entrypoint. Unknown gets 404, deactivated 410.
4. The handler reads the body, up to 1 MB. A larger body gets 413.
5. One transaction stores the event, one pending delivery per active target, and the running totals. The sender gets 200.
6. One task per delivery goes onto the delivery queue.
7. A worker takes the task and runs the target's delivery. The outcome is recorded as an attempt, and the delivery's status is updated.

Components and middleware

The components are wired by Uber fx, a library that builds each component when another needs it. It runs their start hooks (the code each component runs at start) in that order, and their stop hooks in reverse. The components are:

• the globals, the logger and the config;
• the main database and the per-webhook database manager;
• the retention sweep and the health check;
• the session and the handlers;
• the metrics registry and collectors;
• the middleware and the destination check;
• the delivery engine, the archive writers and the archive sweep;
• the server.

Middleware is code that runs around every request. The global middleware runs in this order:

1. request id;
2. security headers;
3. access log;
4. metrics, when /metrics is served;
5. CORS;
6. a 60 second request timeout, which answers 504 when it fires;
7. panic recovery;
8. Sentry, when configured.

A page route group is a set of routes under one prefix, such as /hook/. Each page route group then adds its own middleware, in this order:

1. panic recovery;
2. Sentry, when configured;
3. the 1 MB body cap;
4. CSRF;
5. Cache-Control: no-store;
6. the sign-in check, in every group except /pages, which serves the sign-in form.

The body cap comes before CSRF because CSRF parses the form.

Libraries

• Uber fx wires the components, as above.
• chi, an HTTP router, matches requests to handlers.
• GORM, a Go database library, with modernc.org/sqlite, a pure Go SQLite driver, handles the databases.
• log/slog, the standard library's structured logger, writes the logs.
• gorilla/sessions and gorilla/csrf handle sessions and the CSRF protection.
• go-chi/httprate rate-limits the receiver, the password change, replay and resubmit.
• The Prometheus client and go-http-metrics serve the metrics.
• The Sentry client reports errors when configured.

Package layout

All application code is under internal/. The program starts in cmd/webhooker/main.go, which dispatches subcommands, takes the DATA_DIR lock and wires the components.

• internal/archive: archive files, rotation, pruning, renaming and download.
• internal/banner: the ruled box that prints a credential.
• internal/config: configuration from the environment and .env.
• internal/database: the main database, the models, Argon2id, the retention sweep and the per-webhook database manager.
• internal/datadir: the exclusive lock on DATA_DIR.
• internal/db/migrations: the SQL schema files and their application.
• internal/delivery: the engine, the four target types, retries, the circuit breaker and the destination check.
• internal/globals: build-time variables.
• internal/gormlog: GORM's logger on top of slog, with values withheld.
• internal/handlers: every HTTP handler and page.
• internal/healthcheck: the health check response.
• internal/lifecycle: the bounded wait used by stop hooks.
• internal/logfield: the size budget for client-supplied log values.
• internal/logger: slog setup and the fx event logger.
• internal/metrics: the delivery metrics.
• internal/middleware: logging, security headers, CORS, sessions, CSRF, rate limits, body caps, metrics and panic recovery.
• internal/referer: cuts a Referer to its scheme and host.
• internal/reqtls: the one function that decides whether a request is TLS.
• internal/resetpw: the webhooker resetpw subcommand.
• internal/server: the HTTP server, routes, middleware order and Sentry.
• internal/session: the cookie session.
• internal/versionscript: tests for script/version and the build files.
• static/: embedded CSS and JavaScript. templates/: the HTML templates.

Configuration reference

All configuration comes from environment variables. For development, variables can also go in a .env file in the working directory. It is read once at startup. A variable set in the real environment wins over the file. A .env file that cannot be parsed aborts startup, because a single bad line would otherwise silently drop every variable in the file.

A variable that is unset or empty takes its default. A variable that is set but cannot be parsed aborts startup with a message naming it. PORT=eighty, DEBUG=ture and RETENTION_SWEEP_INTERVAL=1 hour all abort. Each section below says what range its variable accepts.

A Go duration is a number with a unit, such as 30s, 15m, 24h or 1h30m.

The startup log line Configuration loaded shows the loaded values, with three differences. TRUSTED_PROXIES and ALLOWED_EGRESS_CIDRS appear as counts. SENTRY_DSN appears as on or off. The metrics credentials appear as one on-or-off field. The /settings page shows every value; there, METRICS_PASSWORD and SENTRY_DSN show only as set or not set.

WEBHOOKER_ENVIRONMENT

dev or prod. Default prod.

This controls one thing: CORS, the headers that let a page on another site call this one from a browser. In dev, every origin is allowed with Access-Control-Allow-Origin: *, for local testing. In prod, no CORS headers are sent. Cookie security and log format do not depend on it.

PORT

The TCP port the listener binds. Default 8080. It must be a number from 1 to 65535.

BIND_ADDRESS

The IP address the listener binds. Default 127.0.0.1. The Docker image sets 0.0.0.0.

The value must be an IP address literal. localhost is refused, because it may mean either 127.0.0.1 or ::1. A value with a port, such as 127.0.0.1:8080, is refused; the port belongs to PORT.

• 127.0.0.1 answers only on this host. Use it with a reverse proxy on the same host.
• 0.0.0.0 answers on every IPv4 address and no IPv6 address.
• :: answers on every IPv6 and IPv4 address.
• A specific address such as 10.0.0.5 answers on that interface only.

An address that parses but is not on this host passes configuration and fails at bind time. The process logs the error and exits non-zero. The bindAddress field of the startup log shows what was bound.

DATA_DIR

The directory for every SQLite database and the lock file. Default /var/lib/webhooker. It is created if it is missing.

Exactly one webhooker process may use a DATA_DIR at a time. Two processes on one directory would both run delivery recovery and deliver every pending event twice. At startup the process locks {DATA_DIR}/webhooker.lock with flock, the kernel's file lock, and holds the lock until it exits. A second process pointed at the same directory prints a message naming it and exits non-zero.

The kernel releases the lock when the process exits, even on kill -9. A leftover webhooker.lock blocks nothing and must not be deleted by hand. To run two webhookers on one host, give each its own DATA_DIR.

flock works between processes and containers that share a directory on one host. It does not work across hosts on a network filesystem.

DEBUG

true or false. Default false. It accepts the spellings Go's strconv.ParseBool accepts: 1, t, T, TRUE, true, True, 0, f, F, FALSE, false, False. yes and on are refused.

DEBUG=true lowers the log level to DEBUG. Lines at that level include:

• every SQL statement, with ? in place of each value;
• the lookup misses on the receiver and sign-in routes;
• signed-out requests sent to the sign-in page;
• retries being scheduled, and retries skipped because the delivery had already been handled;
• the receiver's whole-route rate limit refusing a request;
• a target deleted while the archive sweep was running;
• the records of how the components were wired at startup.

Debug lines never carry SQL values, session cookies or target credentials. The first-boot password is printed whatever the level, and so is every event a log target writes.

METRICS_USERNAME and METRICS_PASSWORD

The basic authentication credentials for /metrics. Default empty. Set both or neither. With both set, /metrics is served behind basic authentication. With neither set, the route does not exist and answers 404. With only one set, startup fails with an error naming both variables.

SENTRY_DSN

The DSN for Sentry error reporting: the URL that tells the Sentry client where to send reports. Default empty, which leaves reporting off. A set value is parsed with the Sentry client's own parser, and one it cannot parse aborts startup. The release name sent to Sentry is webhooker-{version}.

RETENTION_SWEEP_INTERVAL

How often the retention sweep and the archive sweep run. A Go duration greater than zero. Default 1h.

The retention sweep deletes events older than each webhook's retention period, with their deliveries and attempts, in batches of 1000. The archive sweep prunes rows past each archive target's expiry.

SESSION_IDLE_TIMEOUT

How long a session may go unused before it ends. A Go duration. Default 24h. A zero or negative value disables idle expiry. The 7 day absolute limit always applies.

Only authenticated requests count as use. The idle timestamp in the cookie is rewritten at most once per tenth of the window, so a session can end up to 10% early, never late.

HSTS_MAX_AGE

The max-age of the Strict-Transport-Security header, as a Go duration in whole seconds. Default 87600h (ten years). 0 sends no header. 1.5s and -1h are refused. Read HSTS before changing it.

RECEIVER_RATE_LIMIT

How many requests per minute one client may send to one entrypoint. Default 120. It must be at least 1. A client over the limit is answered 429 with Retry-After.

A second limit covers the whole /h/ route per client at ten times this value, 1200 by default. It exists because the route matches any single path segment. A client inventing a new path per request would otherwise get a fresh bucket each time. Each such request still costs a database lookup. The second limit is not configurable on its own.

TRUSTED_PROXIES

A comma-separated list of networks in CIDR notation, which names a network as an address and a prefix length, such as 10.0.0.0/8. A bare address such as 192.168.1.7 counts as a single host. Default 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16, the RFC 1918 private ranges. A set value replaces the default.

The list decides whose X-Forwarded-For header is believed. X-Forwarded-For is the header a proxy uses to pass on the address of the client it is serving. When the connecting peer is inside the list, the client address is read from that header. For any other peer, the client is the peer itself and the header is ignored. X-Real-IP and True-Client-IP are never read.

The header is read from right to left. The first hop that is not itself a trusted proxy is the client. A hop that is not a bare IP address ends the walk, and the peer address is used instead. At most 64 hops are examined. Your proxy must therefore append the peer address to X-Forwarded-For, as nginx's $proxy_add_x_forwarded_for does.

If any client can reach webhooker, or the proxy in front of it, from a private address, set the list to the proxy's address alone. Otherwise those clients can choose their own X-Forwarded-For and bypass every rate limit. The address to set is the remoteIP field of an http request log line for a request that came through the proxy.

Behind a proxy the list does not cover, every client shares one bucket per limit. The receiver limits become service-wide ceilings. Sign-in failures from every client land in one count, though a correct password is still never throttled.

Rate-limit buckets are per IPv4 address and per IPv6 /64 prefix.

Allowing egress to your own network

ALLOWED_EGRESS_CIDRS is a comma-separated list of networks in CIDR notation that delivery targets may reach although the default rules refuse them. Default empty. A bare address counts as a single host. Example: 10.0.0.0/8, 172.17.0.0/16.

This only ever adds destinations. The default refusals stay in force for every address not listed. When the list is set, startup logs the whole list at WARN.

Anyone with the admin account can make webhooker send requests to every network you list. They can read the responses in the delivery log. List the smallest blocks that cover your destinations. Prefer 10.1.2.3/32 over 10.0.0.0/8. 0.0.0.0/0 or ::/0 opens every private and reserved range at once, including this host's own loopback. Do not list them.

Some addresses stay blocked whatever you list. They are link-local space, the unspecified addresses 0.0.0.0 and ::, and cloud metadata endpoints at fixed non-public addresses. A cloud metadata endpoint is the address a cloud gives each virtual machine for its own credentials and settings. The blocked set is:

• 169.254.0.0/16 and fe80::/10, the link-local ranges. 169.254.169.254 is the metadata address on AWS, Azure, DigitalOcean, Hetzner, OpenStack and others.
• fd00:ec2::254/128 and fd00:ec2::23/128, AWS metadata and EKS Pod Identity over IPv6.
• fd20:ce::254/128, Google Cloud metadata over IPv6.
• fd00:c1::a9fe:a9fe/128, Oracle Cloud metadata over IPv6.
• fd00:42::42/128, Scaleway metadata over IPv6.
• fd00:a9fe:a9fe::1/128, Linode and Akamai metadata over IPv6.
• 100.100.100.200/32, Alibaba Cloud metadata, inside the carrier-grade NAT range.
• 192.0.0.192/32, Oracle Cloud Classic metadata.
• 0.0.0.0/32 and ::/128, the unspecified addresses, which reach this host's loopback on Linux.
• ::a9fe:a9fe/128 and 64:ff9b::a9fe:a9fe/128, two other spellings of 169.254.169.254.

The six fd00::/8 entries matter because fd00::/8 is an ordinary block to list for your own IPv6 network. Without them, that one line would open cloud credentials on five providers.

This list is not every cloud's metadata address. If yours is not here, do not list the block that contains it. Azure's public 168.63.129.16 is refused by default but is reopened by a list that covers it.

Delivery reference

Target types

An http target posts the event body to its URL. The request carries the event's Content-Type, the event's headers, then the target's configured headers, then User-Agent: webhooker/{version}. A configured header replaces an event header of the same name. A 2xx response is a success. The first 4 KiB of the response body is stored with the attempt.

These event headers are never forwarded:

• Host
• Connection and Keep-Alive
• Transfer-Encoding and Content-Length
• Te and Trailer
• Upgrade
• Proxy-Authorization and Proxy-Connection

An http target's settings:

• url, the destination. Only http and https are accepted.
• headers, one Name: value per line. Content-Type is allowed and replaces the event's. These names are refused, because the delivery sets them itself:
  • Host
  • Content-Length
  • Transfer-Encoding
  • Connection
  • Trailer
  • User-Agent
• timeout_seconds, the per-request timeout, at most 300. Blank means the shared 30 second default. A long timeout holds one of the ten delivery workers for its whole duration.
• forward_query, off by default. On, the event's query string is appended to the URL, joined with & if the URL already has one.

A slack target posts a JSON message with one text field to its URL. The message names the method, content type, time received and body size, then shows the body in a code block. A JSON body is indented. A body over 3500 bytes is cut. The URL is any message-posting URL that accepts Slack's format, which includes Mattermost's.

A database target, called an archive in the web interface, appends the event as a row to its own SQLite file. See Archives. It makes no network request and never retries. A write failure fails the delivery.

A log target writes the event, headers and body included, as one INFO line. It never retries.

Attempts and retries

http and slack targets take max_attempts, from 1 to 20. With 1, a failed delivery is marked failed after its one attempt. With more, a failed attempt marks the delivery retrying and schedules the next attempt. The wait doubles each time, starting at one second. After the last allowed attempt fails, the delivery is marked failed.

A delivery has one of four statuses. pending means created but not yet attempted. retrying means at least one attempt failed and more remain. delivered means an attempt succeeded. failed means every allowed attempt failed.

Every attempt is recorded with its number, success, status code, response body, error and duration. One case is not recorded: an attempt cut off by shutdown.

Circuit breaker

An http or slack target with max_attempts above 1 has a circuit breaker. A circuit breaker stops hammering a destination that is down. It is in memory only and resets when the service restarts.

The breaker is closed in normal operation. After 5 failures in a row it opens for 30 seconds. That wait is the cooldown. While the breaker is open, each delivery to the target is marked retrying and scheduled for when the cooldown ends. When the cooldown ends, one delivery is sent as a probe. The others wait one more cooldown. If the probe succeeds, the breaker closes. If it fails, the breaker opens again.

The webhook page says when a target's deliveries are paused and until when. Each waiting delivery in the event log shows the earliest time it can be tried.

Workers and queues

Ten workers carry out deliveries. New deliveries wait in one queue and retries in another. Each queue holds 10,000 tasks. At most ten deliveries are in flight at once, however deep the queues get.

When a retry falls due and its queue is full, the retry is dropped and the delivery stays retrying in the database. The recovery sweep picks it up.

Each task carries the target's settings and the event. A body under 16 KiB travels with the task. A larger one is read from the database when the attempt runs.

Recovery

A recovery routine runs once at start and then every 60 seconds. For every webhook it reads the deliveries still pending or retrying. One that already has a successful attempt recorded is marked delivered and not sent again. A retrying delivery is scheduled for whatever remains of its wait. A pending one is queued. A delivery the engine is already working on is left alone.

At start, every pending delivery is taken, however recent. So a delivery cut off by a stop moments earlier is sent again at once. The periodic sweep takes only pending deliveries untouched for 15 minutes, at most 500 per webhook each time.

If a target was deleted while a delivery to it was still open, recovery marks the delivery failed and records the reason.

Shutdown

On SIGINT or SIGTERM, components stop in reverse start order under a 5 second budget for the whole sequence. The HTTP server drains open requests, which means it waits for them to finish, for at most 3 seconds. That leaves 2 seconds for the components after it. A Sentry flush, if enabled, fits inside the same budget and is skipped when little is left.

The delivery engine cancels any http or slack request in flight. That attempt is not recorded, and the delivery keeps its status, so it is sent again at the next start. A database or log task finishes as normal.

A component that does not stop in time is logged at ERROR, and the components after it are skipped. SQLite is crash-safe, so this loses no data, but the database is not closed cleanly.

Keep the container's stop grace period above 5 seconds. That is the time Docker waits after docker stop before it kills the process. Docker's default is 10. A shorter grace kills the process before the shutdown can run.

Archives

An archive target writes to archive-{webhook_name}-{target_name}-{target_uuid}.db in DATA_DIR. The two names are made safe for a file name:

• lowercased;
• only the ASCII letters a to z and the digits 0 to 9 kept, and every other run of characters turned into one -;
• cut to 40 characters;
• unnamed when nothing is left.

A webhook named Orders (EU) with a target named Long-term archive writes archive-orders-eu-long-term-archive-{target_uuid}.db.

Each row holds:

• the event's id, and the ids of its webhook and entrypoint;
• the method, query string, headers, body and content type;
• the time the row was archived.

archive_rotation is none by default, which keeps one file. monthly, daily and hourly start a new file for each period of the event's receive time, in UTC, with the period before .db: -2026-10.db, -2026-10-01.db or -2026-10-01-19.db. A changed rotation applies from the next event. Files already written keep their names and are still shown, pruned and downloaded.

archive_expiry is never by default. The forms offer 1h, 12h, 24h, 30d, 90d and 365d. Rows older than the expiry are deleted each time the file is opened for a write, and by the archive sweep every RETENTION_SWEEP_INTERVAL. The sweep deletes a period file it has emptied. It never creates a file.

Renaming the webhook or the target renames every file of the target. If a file with the new name already exists, the edit is refused and nothing moves. A file the operator has moved away is skipped.

Deleting a webhook or a target leaves its archive files on disk. Removing them is the operator's decision.

Download on the webhook page returns the target's whole archive as one gzipped JSON file named archive-{webhook_name}-{target_name}-{YYYYMMDDTHHMMSSZ}.json.gz. It holds webhook and target (each an id and a name), exported_at, and archived_events, one object per row with every column. Rows from a period file carry a period field. A body that is not valid UTF-8 is written in base64 with "body_encoding": "base64" beside it. The download streams one file at a time and creates nothing.

Data reference

Files

webhooker.db holds the application tier: settings, users, API keys, webhooks, entrypoints and targets. Each events-{webhook_uuid}.db holds that webhook's event tier: events, deliveries, delivery attempts and running totals. Each archive-*.db holds one archive target's rows.

Every database is opened through one code path, in WAL mode, with at most 4 connections. A connection waits up to 10 seconds for a locked database before giving up. Under WAL a reader never blocks a writer.

An event database is created with its webhook. Every webhook's database is opened at start. One that is missing or empty is created again and logged with created a new, empty database at WARN. On a deployment that has run before, that line means the file was lost, most often because the volume holding DATA_DIR is not mounted. A damaged file fails that webhook alone, with an ERROR naming it and a 500 to its senders.

Deleting a webhook soft-deletes its row, entrypoints and targets in webhooker.db and deletes its event database file. A soft delete marks the row with a deletion time and hides it from the application; the row stays in the file. Deleting an entrypoint or a target soft-deletes it.

Schema

Each kind of database file has its schema in SQL files built into the binary, under internal/db/migrations/: main/ for webhooker.db, events/ for event databases and archive/ for archive files. In each, 000_migration.sql creates the schema_migrations table and 001_schema.sql holds the schema. Opening a database applies every file its schema_migrations does not list, in one transaction.

Application tier

Every entity below except Setting has id (a version 4 UUID), created_at, updated_at and deleted_at. A row with deleted_at set is soft-deleted.

Setting is a key-value row with no other fields. The one key stored is session_key, the base64 session key made on first start.

User has username (unique, at most 1024 bytes) and password (the Argon2id hash). The first start creates admin with a random 16-character password.

Webhook has user_id, name, description and retention_days. retention_days is 0 for forever, or a whole number from 1 to 106751. The cap keeps the retention cutoff inside the range of a Go duration.

Entrypoint has webhook_id, path, description and active. path is the bare UUID. The /h/ prefix is route only and is not stored.

Target has these fields. Each type fills only its own settings.

• webhook_id, name, type and active. type is one of http, slack, database or log.
• url, headers, timeout_seconds and forward_query, the http settings. A slack target uses url alone.
• archive_expiry and archive_rotation, the database settings. archive_expiry is stored in nanoseconds, 0 for never.
• max_attempts, for http and slack.

APIKey has user_id, key, description and last_used_at. Its table, api_keys, exists, but nothing creates a key and nothing checks one.

Event tier

Event has these fields:

• webhook_id and entrypoint_id;
• method, which is always POST;
• raw_query, headers (as JSON), body, body_bytes and content_type;
• resubmitted_from_id, the event a resubmit copied. It is empty for an event that arrived on the receiver.

Delivery has event_id, target_id, status, finished_at and replay. finished_at is set when the status becomes delivered or failed.

DeliveryResult is one attempt. It has these fields:

• delivery_id and attempt_num;
• success and status_code;
• response_body, the first 4 KiB;
• error;
• duration, in milliseconds.

Three tables hold running totals so the pages need not count rows. Each count changes in the transaction that writes or deletes the rows it counts.

• EventTotals is one row: events, events_removed and last_event_at.
• TargetTotals is one row per target, with:
  • deliveries, delivered and failed;
  • deliveries_removed and failed_removed.
• EntrypointTotals is one row per entrypoint: last_event_at.

Retention raises the _removed counts and leaves last_event_at alone, so a webhook's last event is still shown after its events have expired.

HTTP routes reference

Public routes:

• GET / redirects to /hooks when signed in and to /pages/login otherwise.
• GET /.well-known/healthcheck returns JSON with status, now, uptimeSeconds, uptimeHuman, version and appname.
• GET /s/* and HEAD /s/* serve the embedded CSS and JavaScript. Other methods get 405 with Allow: GET, HEAD.
• POST /h/{uuid} is the receiver. Any other method gets 405 with Allow: POST. An unknown UUID gets 404, a deactivated entrypoint 410, a body over 1 MB 413, and a stored event 200 with {"status":"ok"}.

Sign-in routes:

• GET /pages/login shows the sign-in form. Its next parameter names the page to return to. Anything but a path on this site is replaced with /.
• POST /pages/login signs in. See Sign-in limits.
• POST /pages/logout signs out.

Routes behind the sign-in. A GET without a session is redirected to /pages/login with the path as next:

• GET /user/{username} is the profile page. Another user's name gets 403.
• POST /user/{username}/password changes the password, 5 per minute per client.
• GET /settings lists the loaded configuration.
• GET /hooks lists webhooks. GET and POST /hooks/new create one.
• GET /hook/{id} is the webhook page. GET and POST /hook/{id}/edit edit it. POST /hook/{id}/delete deletes it.
• GET /hook/{id}/events is the event log. ?show=failed and ?show=pending narrow it.
• GET /hook/{id}/events/{eventID} is one event's page, and GET /hook/{id}/events/{eventID}/body downloads its stored body.
• POST /hook/{id}/deliveries/{deliveryID}/replay replays a delivery, and POST /hook/{id}/events/{eventID}/resubmit resubmits an event, each 30 per minute per client.
• POST /hook/{id}/entrypoints adds an entrypoint.
• POST /hook/{id}/entrypoints/{entrypointID}/edit changes its description.
• POST /hook/{id}/entrypoints/{entrypointID}/delete deletes it.
• POST /hook/{id}/entrypoints/{entrypointID}/toggle activates or deactivates it.
• POST /hook/{id}/targets adds a target.
• GET /hook/{id}/targets/{targetID}/edit shows the target edit form, and POST to the same path saves it.
• POST /hook/{id}/targets/{targetID}/delete deletes the target.
• POST /hook/{id}/targets/{targetID}/toggle activates or deactivates it.
• GET /hook/{id}/targets/{targetID}/download downloads an archive.

GET /metrics exists only when METRICS_USERNAME and METRICS_PASSWORD are set, behind basic authentication.

/api/v1 is mounted with no routes. Every path under it answers 404.

Metrics reference

/metrics serves one Prometheus registry, the set of metrics it exposes. It carries the Go runtime and process collectors, which report the process's memory, threads and CPU use, the delivery metrics below, and three inbound HTTP metrics.

Nine of the eleven metrics below carry one label, target_type, with one of http, slack, database, log or unknown. The two event counters carry none. Target ids and event ids are never labels, because a label value that grows with traffic would grow the registry without bound.

• webhooker_events_received_total (counter, no label): events stored, resubmitted copies included.
• webhooker_delivery_attempts_total (counter): attempts sent to a target. A delivery the circuit breaker turned away is not an attempt.
• webhooker_deliveries_succeeded_total (counter): deliveries that became delivered.
• webhooker_deliveries_failed_total (counter): deliveries that became failed.
• webhooker_delivery_retries_total (counter): deliveries put into retrying.
• webhooker_delivery_replays_total (counter): deliveries created by Replay.
• webhooker_events_resubmitted_total (counter, no label): events created by Resubmit.
• webhooker_delivery_duration_seconds (histogram): wall time of one attempt.
• webhooker_deliveries_pending and webhooker_deliveries_retrying (gauges): deliveries in those statuses, counted from the databases every 30 seconds. Deliveries to a deleted target count under unknown.
• webhooker_circuit_breakers_open (gauge): breakers currently open.

The success and failure counters move only after the new status has been written.

The inbound HTTP metrics are http_request_duration_seconds, http_response_size_bytes and http_requests_inflight. Their handler label is the route pattern, such as /h/{uuid}, never the real path. A request that matched no route carries (unmatched). http_requests_inflight is one series (one metric with one set of label values) for the whole service, with handler set to (all).

Development

Prerequisites

• The Go version named in go.mod.
• Docker. The tests, the linters, prettier and the stylesheet build run in Docker.

gofmt and goimports run on the host. These tools run only inside Docker, at the versions pinned (fixed to one exact release) in the Dockerfile, package.json and yarn.lock:

• golangci-lint, the Go linter;
• ESLint, the JavaScript linter;
• prettier, the Markdown formatter;
• node and yarn, which run the two above;
• the Tailwind CLI, which generates the stylesheet.

Commands

make bootstrap    # Install git, make, Go and the Go modules (safe to rerun)
make setup        # bootstrap, then install the git pre-commit hook
make assets       # Extract Alpine.js from 3p/ to static/js/alpine.min.js
make fmt          # Format Go (gofmt, goimports) and Markdown (prettier, in Docker)
make fmt-check    # Fail if gofmt or prettier would change anything
make lint         # Run gofmt, golangci-lint and ESLint in Docker
make test         # Run the tests in Docker, with cgo off
make test-browser # Run the browser test in Docker (Dockerfile.browser)
make check        # test, lint, fmt-check and css-check: what CI runs
make build        # Build bin/webhooker, version-stamped
make version      # Print the version this checkout would stamp
make run          # build, then run ./bin/webhooker
make dev          # go run ./cmd/webhooker
make deps         # go mod download, then go mod tidy
make docker       # Build the Docker image
make hooks        # Install the git pre-commit hook
make css          # Regenerate static/css/tailwind.css in Docker
make css-check    # Fail if static/css/tailwind.css is out of date
make clean        # Remove bin/

Each make target above, except build, run, dev, deps, version and clean, is a thin shim over a script in script/. make hooks calls script/install-precommit; every other shim calls the script of the same name. The scripts are listed under Entrypoints.

The tests run with cgo, Go's bridge to C code, turned off. Go's race detector, which finds unsynchronised access to shared memory, needs cgo, so the tests run without it.

How the checks run

Each check is a stage of the Dockerfile, which is one named step of the image build. script/test builds the test stage, script/lint the lint stage and script/css-check the css-check stage. script/fmt-check runs gofmt on the host and builds the markdown-check stage. Every docker build in script/ passes --no-cache, so Docker reruns every step instead of reusing an earlier result, and a check that reports success ran. A bare docker build . carries no such guarantee.

The lint stage runs the gofmt check, golangci-lint config verify and golangci-lint run with --network=none, and ESLint over static/js/. The test stage runs go test -cover on four packages at a time, each running at most eight tests in parallel, with a 90 second timeout per package. On a failure it runs only the failed tests again with -v.

The image build depends on the lint, test, css-check and markdown-check stages, so an image that built is a tree that passed every check. The runtime stage is Alpine with ca-certificates and su-exec, a tool that switches user, added. It has no sqlite3.

The check workflow in .gitea/workflows/check.yml checks out the repository and runs script/cibuild, nothing else.

Version stamping

The binary reports its version at /.well-known/healthcheck, in the page footer, in the starting log line and as the Sentry release. The value is stamped into the binary at build time.

script/version produces it. With VERSION set in the environment, that value is used. Otherwise it is git describe --tags --always --dirty on this checkout, which gives:

• the tag at a clean tagged commit, such as v1.0.0;
• the tag, the commit count and the short hash past a tag, such as v1.0.0-3-g1a2b3c4;
• the short hash alone when no tag is reachable;
• a -dirty suffix on any of these for uncommitted changes.

With no git metadata it is unknown.

make build VERSION=v1.2.3 overrides it. script/docker and script/cibuild run the same git describe on the host and pass the result in as the VERSION build argument. Without that argument the image build reads the .git in the build context, and fails if .git is present but the version comes out empty, dev or unknown. .dockerignore leaves out every git config file, which can carry a credential, and keeps the rest of .git.

make build passes -trimpath and -s -w, so the binary carries no build paths, symbol table or debug information. Nothing that differs between two builds of one commit is stamped in.

Third-party browser assets

The pages use one third-party script: Alpine.js, a small JavaScript library. They use its CSP build, @alpinejs/csp, which runs under a Content-Security-Policy that forbids eval. The standard build needs eval. With the CSP build, every Alpine directive in templates/ names a component registered in static/js/app.js, such as x-data="collapsible", and never an inline expression.

The npm package file is committed as 3p/alpinejs-csp-3.14.9.tgz, byte for byte as the registry publishes it. script/assets extracts package/dist/cdn.min.js from it to static/js/alpine.min.js, which the binary embeds. make build, make dev and the Docker stages run it first. The extracted file is not committed, and a build that skips the extraction fails with an error naming js/alpine.min.js.

To move to a new version:

1. Download https://registry.npmjs.org/@alpinejs/csp/-/csp-<version>.tgz.
2. Check it against the dist.integrity hash at https://registry.npmjs.org/@alpinejs/csp/<version>.
3. Replace the file in 3p/ and update the file name in script/assets.
4. Run make check and make test-browser.

Browser test

internal/server has a test that loads the pages in a headless browser and drives the interface under the real Content-Security-Policy. A headless browser is one without a window. The test is built only with the browser build tag, a flag given to the compiler, so make test leaves it out. make test-browser runs it in Docker with a pinned headless browser image. Run it after changing templates/ or static/js/.

Stylesheet

static/css/tailwind.css is generated by Tailwind and committed. To change the styles, edit the templates, static/js/app.js, internal/handlers/recent_events.go or static/css/input.css, then run make css and commit the result. Tailwind reads classes only from the files that input.css names in its @source lines. make check fails when the committed file differs from what make css generates. static/css/style.css is hand-written. The Content-Security-Policy refuses inline styles, so templates style elements with classes only.

Entrypoints

This repository adheres to the Scripts to Rule Them All standard: the scripts in script/ are the entry points for the development workflow. This section is about those scripts. The receiving URLs of the application are under Entrypoint URLs.

• script/bootstrap installs git, make, Go and the Go modules. It is safe to rerun.
• script/setup runs script/bootstrap, then script/install-precommit.
• script/projectname prints webhooker, the name the other scripts use.
• script/assets extracts Alpine.js from 3p/ to static/js/alpine.min.js.
• script/test builds the test stage of the Dockerfile, which runs the tests.
• script/test-browser builds Dockerfile.browser, which runs the browser test.
• script/lint builds the lint stage, which runs gofmt, golangci-lint and ESLint.
• script/fmt formats the Go code with gofmt and goimports on the host, and the Markdown with prettier in Docker. It writes.
• script/fmt-check runs the same checks and fails instead of writing.
• script/css regenerates static/css/tailwind.css in Docker. It writes.
• script/css-check fails when static/css/tailwind.css differs from what script/css would generate.
• script/check runs script/test, script/lint, script/fmt-check and script/css-check.
• script/version prints the version to stamp into the binary.
• script/docker builds the image tagged webhooker, passing that version in.
• script/cibuild is what CI runs: script/bootstrap, then script/check, then the same image build as script/docker.
• script/precommit checks that go mod tidy changes nothing, then runs script/check.
• script/install-precommit installs the git pre-commit hook that runs script/precommit.

TODO

See TODO.md.

License

MIT.

Author

@sneak