webhooker/README.md

# webhooker

webhooker is an MIT-licensed web service, written in [Go](https://golang.org) by
[@sneak](https://sneak.berlin), that receives the HTTP requests other services
send when something happens and forwards them to your own systems.

Many services send an HTTP request to a URL you give them when something
happens: a code push, a payment, a new ticket. webhooker receives those
requests, stores each one as an event, and forwards it to the places you choose.

webhooker is a single binary with a web interface. It stores its data in SQLite,
a database engine that keeps each database in one ordinary file. It writes each
event to disk before answering the sender. It then delivers the event to each
target you configured. The web interface shows the newest events, with every
delivery and attempt they had. From there you can send a stored event again.
webhooker also exports metrics for Prometheus, the monitoring system, so you can
watch the whole flow.

## Rationale

webhooker exists so that a request from another service is not lost while your
service is down. It also exists so that you can see what was sent and what
happened to it.

It stores each event before it forwards it, so an event that arrives while your
service is down is not lost. It keeps events and their delivery attempts for the
retention period you set. It can forward one event to several targets, and it
can send a stored event again.

## How it works

Five words name the moving parts.

- A **webhook** is a named configuration. It groups entrypoints and targets, and
  it owns the events it receives.
- An **entrypoint** is a receiving URL of the form `/h/{uuid}`. Senders post to
  it. The UUID is a random 36-character identifier, and it is the only thing
  that authorises a sender.
- A **target** is a place events are delivered to. There are four kinds: an HTTP
  URL, a Slack or Mattermost message-posting URL, an archive database file, and
  the application log.
- An **event** is one stored request: its method, query string, headers and
  body.
- A **delivery** is one event paired with one target. It records every attempt.

The part of webhooker that answers requests to `/h/{uuid}` is the receiver.

A request to an entrypoint is stored as an event. One pending delivery is
created for each active target of the webhook, in the same database write as the
event: either all of them are stored or none. The sender gets `200` once they
are stored. Background workers, the part of webhooker that sends deliveries,
then carry them out. A delivery to an HTTP or Slack target that fails is
retried, up to the number of attempts the target allows.

Delivery is at least once, not exactly once. If a delivery reaches the target
but webhooker cannot record that outcome, it sends the delivery again later. A
delivery in flight when the service stops is also sent again at the next start.
webhooker adds no delivery identifier of its own to the request, so make your
targets tolerate a duplicate.

## Getting Started

You need the Go version named in `go.mod`, and Docker for the checks.

```bash
git clone https://git.eeqj.de/sneak/webhooker.git
cd webhooker
make bootstrap
DATA_DIR=../webhooker-data make dev
```

`DATA_DIR` is where webhooker keeps its databases. It defaults to
`/var/lib/webhooker` in every environment, so point it at a directory you can
write to. Keep it outside the clone: the databases hold the session key.

The first start creates the `admin` account and prints its password once, in a
ruled box on standard output:

```
========================================================================
WEBHOOKER FIRST BOOT: an admin account has been created.

    username: admin
    password: 3xamPl3-p4ssw0rd

Save this password now: it is shown only here, and only once.
If it is lost, run `webhooker resetpw admin` on a stopped deployment.
========================================================================
```

Open `http://127.0.0.1:8080/` and sign in. The server listens on `127.0.0.1`
port `8080` by default.

To run it in Docker instead, build the image from the clone with `make docker`,
which tags it `webhooker`. Then:

```bash
docker run -d \
  -p 127.0.0.1:8080:8080 \
  -v /path/to/data:/var/lib/webhooker \
  webhooker
```

Read [Running with Docker](#running-with-docker) before publishing the port
anywhere but `127.0.0.1`.

## Using the web interface

### Signing in

The sign-in page is `/pages/login`. There is one account, `admin`, and no way to
add another through the web interface. A session lasts up to 7 days. It ends
sooner after 24 hours without use, unless `SESSION_IDLE_TIMEOUT` says otherwise.

If a page asked for a sign-in, signing in returns you to that page.

### Creating a webhook

The webhook list is at `/hooks`. **New Webhook** asks for a name, an optional
description and a retention period in days. Events older than the retention
period are deleted, along with their deliveries. The default is 30 days. `0`
keeps events forever.

The form can also create the first targets. An HTTP target URL creates an `http`
target named `HTTP`. The **Archive** checkbox creates an archive target named
`Archive`. Beside it you choose the archive's expiry, which is how long archived
events are kept, and its rotation, which is how often a new archive file is
started.

Every new webhook gets one entrypoint, described as `Default entrypoint`.

### Entrypoint URLs

The webhook page lists each entrypoint with its full URL and a **Copy** button.
Give that URL to the sender. Anyone who holds it can post events, so treat it
like a password. Read
[The entrypoint URL is the credential](#the-entrypoint-url-is-the-credential).

**Add** creates another entrypoint with its own URL. Several entrypoints on one
webhook let you tell senders apart while delivering to the same targets.

**Edit** changes an entrypoint's description. Its URL never changes. To replace
a URL, deactivate the entrypoint or delete it, then create a new one.

A deactivated entrypoint answers `410 Gone`. A deleted or unknown one answers
`404`.

Beside each entrypoint the page shows when its last event arrived, and how many
events arrived on it within the retention period.

### Targets

**Add** under **Targets** asks for the type first, then that type's settings.

- An `http` target posts each event to a URL. It takes optional request headers,
  a timeout in seconds, and a switch to pass the event's query string on.
- A `slack` target posts each event as a message to Slack or Mattermost. It
  takes the message-posting URL those services give out, which they call an
  incoming webhook.
- An archive target writes each event as a row in a SQLite file of its own. Its
  stored type is `database`. It takes an expiry and a rotation.
- A `log` target writes each event to the application log. It has no settings
  beyond its name.

`http` and `slack` targets also take **Delivery attempts**: how many times each
delivery is tried in all. The default is 1. The most is 20.

The target list shows each target's settings. A destination URL is shown as its
scheme and host, with `/...` standing for its path. Headers are shown as a
count. The target edit page is the one place that shows a URL and its headers in
full.

The list also shows, for each target, how many deliveries have ever succeeded
and failed, and how many did so in the last 24 hours. For an archive target it
names the current archive file, the size of all its files together, and when the
last write happened.

**Deactivate** stops new events reaching the target. Deliveries already created
are unaffected. **Delete** removes the target. An archive target's files stay on
disk.

A target's type cannot be changed. Create a new target to deliver another way.

### Statistics

The top of the webhook page shows:

- the number of entrypoints, and how many are active;
- the number of targets, and how many are active;
- deliveries in progress;
- the time of the last event;
- the retention setting.

Below that are events, deliveries and failures, both lifetime and within
retention. Next are figures for the last 10 minutes and the last 24 hours:
events, failures and the failure percentage. The percentage is failed deliveries
out of all deliveries that finished in the window.

The webhook list shows, for each webhook:

- its entrypoint and target counts, with how many are inactive;
- its retention setting;
- its events within retention;
- its last event;
- its failed deliveries in the last 24 hours.

### Recent events and the event log

The webhook page shows the 50 newest events. Each expands to show its body. A
body is shown as text up to 32 KiB. A longer or binary body has a **Download the
body** link to the stored bytes.

**Full Event Log** at `/hook/{id}/events` shows the 50 newest events with their
deliveries and every attempt. **Failed** narrows it to events with a failed
delivery. **Pending** narrows it to events with a delivery still pending or
retrying. Each attempt shows:

- when it ran;
- its status code;
- its duration;
- its error, if any;
- the first 4 KiB of the response.

Each event has a page of its own at `/hook/{id}/events/{eventID}`. It shows the
whole request: query string, headers, body and every delivery.

### Replay and resubmit

**Replay** sits beside each finished delivery. It creates a new delivery of the
same event to the same target and sends it on the ordinary path. The target's
settings as they stand now are used, so a replay is how you resend after fixing
the destination. The original delivery is left as it was. The new one is marked
as a replay.

A replay is refused in three cases:

- the delivery has not finished;
- its target is deactivated or deleted;
- another delivery of that event to that target is already in flight.

A delivery whose target was deleted shows no **Replay** button.

**Resubmit** sits beside each event. It stores a copy of the event as a new
event and delivers it to every target that is active now. That includes targets
created after the original arrived. The copy's row names the event it came from.
The original's row shows how many copies were made. Resubmitting the same event
many times is allowed.

Both actions are limited to 30 per minute per client.

### Profile and settings

`/user/{username}` shows your account and lets you change your password. The
current password is checked first. The new password must not be empty and must
match its confirmation. Password changes are limited to 5 per minute per client.

`/settings` lists every configuration variable with the value the running server
loaded. It is read only. `METRICS_PASSWORD` and `SENTRY_DSN` show only as set or
not set.

## Deploying

### Running with Docker

No image is published. `make docker` builds one from the clone and tags it
`webhooker`. Then:

```bash
docker run -d \
  -p 127.0.0.1:8080:8080 \
  -v /path/to/data:/var/lib/webhooker \
  webhooker
```

The image listens on every IPv4 address inside the container: it ships
`BIND_ADDRESS=0.0.0.0`. A container has its own network namespace, which means
its own addresses, separate from the host's. The namespace is what keeps the
container's ports apart from the host's, and the `-p` flag decides what the host
exposes. Publish to `127.0.0.1:8080`, the address your reverse proxy connects
to, and nothing wider. A bare `-p 8080:8080` publishes the port on every
interface of the host.

Do not set `BIND_ADDRESS` to an empty value in a container. An empty value means
unset, and unset falls back to the binary's default of `127.0.0.1`. The
container then answers nothing on its published port and its health check fails.

A container on an IPv6-only Docker network needs `BIND_ADDRESS=::`.

The app runs as user `webhooker` (UID 1000). The image's start-up script,
`deploy/docker-entrypoint.sh`, starts as root. It creates `DATA_DIR` if it is
missing, gives it and its contents to `webhooker`, and sets the directory to
mode `0750`. Then it runs the app as `webhooker`. So a host directory can be
mounted as it is, whoever owns it. With `--user`, the script changes nothing and
runs the app as that user.

webhooker creates every SQLite file with mode `0600`, and resets a file found
with another mode when it opens it. `webhooker.db` holds target settings in
plain text, including bearer tokens and Slack URLs, so the file mode is what
protects them.

The image's health check requests `/.well-known/healthcheck` on the container's
own address every 30 seconds.

The same binary runs the password reset. The image's `CMD` is `/app/webhooker`,
and a command given to `docker run` replaces all of it:

```bash
docker run --rm -v webhooker-data:/var/lib/webhooker \
    webhooker /app/webhooker resetpw -generate admin
```

### Running under upaas

[upaas](https://git.eeqj.de/sneak/upaas), a deployment tool, builds the image
from this repository's `Dockerfile` and runs it.

- Add no port mapping. upaas publishes every mapped port on all interfaces of
  the host. Instead, set the app's Docker network to your reverse proxy's
  network. The proxy then reaches the app at `upaas-` followed by the app name,
  on port `8080`.
- Leave `PORT`, `BIND_ADDRESS` and `DATA_DIR` unset. The image's health check
  probes port `8080`.
- Mount one host directory at `/var/lib/webhooker`.
- Set `TRUSTED_PROXIES` to your reverse proxy's address if clients can reach the
  proxy from a private address. See [`TRUSTED_PROXIES`](#trusted_proxies).

The first start prints the `admin` password to the container's log. For an app
named `webhooker`:

```bash
docker logs upaas-webhooker
```

If the password is lost, stop the container, reset the password with the app's
own image and volume, and start it again:

```bash
docker stop upaas-webhooker
docker run --rm --volumes-from upaas-webhooker \
    "$(docker inspect -f '{{.Image}}' upaas-webhooker)" \
    /app/webhooker resetpw -generate admin
docker start upaas-webhooker
```

### Behind a reverse proxy

webhooker speaks plain HTTP. It does not handle TLS, the encryption behind
HTTPS: it holds no certificate and accepts no HTTPS connections. Put a reverse
proxy in front of it: a server that accepts the public connections and passes
them on. The proxy holds the certificate and reaches webhooker over loopback,
the address only this host can reach.

Five things must be right.

1. **Keep the app port private.** The binary binds `127.0.0.1` by default. The
   image binds every IPv4 address inside its container, so publish its port to
   `127.0.0.1` only. The plain port serves the sign-in form and the receiver
   without TLS.
2. **Send `X-Forwarded-Proto: https`.** This header tells webhooker the request
   arrived over TLS. It decides the `Secure` flag on cookies, and whether a form
   submission must carry a matching `Origin` or `Referer` (see
   [The web interface](#the-web-interface) under Security). Without it, cookies
   are sent without `Secure`. webhooker reads this header from any connecting
   machine, so the proxy must overwrite whatever the client sent.
3. **Cover the proxy in `TRUSTED_PROXIES`.** Otherwise every client shares one
   rate-limit bucket. A bucket is the count a limit keeps per client, and here
   it would be keyed on the proxy's address. The default list is the private
   ranges, which does not cover a proxy that connects over loopback.
4. **Send `Host` as `$http_host`, not `$host`.** webhooker checks a form's
   `Origin` against the `Host` header it receives. `$host` drops the port, so on
   any port other than 443 the two differ and every form submission fails with
   `403`.
5. **Keep the proxy's access log.** It records the client address whatever
   `TRUSTED_PROXIES` says. It also records the full request path, which holds
   the entrypoint UUID of every event. Protect that log as you would the
   entrypoint URLs.

`WEBHOOKER_ENVIRONMENT` decides none of this. It controls only CORS, the headers
that let a page on another site call this one from a browser. Leave it unset in
production.

#### nginx

A complete server block. Replace the `server_name` and the certificate paths.

```nginx
server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;  # nginx 1.25.1+; older: listen 443 ssl http2;

    server_name webhooker.example.com;

    ssl_certificate     /etc/ssl/certs/webhooker.example.com.crt;
    ssl_certificate_key /etc/ssl/private/webhooker.example.com.key;
    ssl_protocols       TLSv1.2 TLSv1.3;

    # webhooker caps form and receiver bodies at 1 MB.
    client_max_body_size 1m;

    # The request path holds the entrypoint UUID. Protect this
    # file like the entrypoint URL itself.
    access_log /var/log/nginx/webhooker.access.log combined;

    location / {
        # A literal address, not localhost: localhost may resolve
        # to ::1 first, where webhooker is not listening.
        proxy_pass http://127.0.0.1:8080;

        # $http_host, not $host: $host drops the port.
        proxy_set_header Host              $http_host;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Above webhooker's own 60 second request timeout, so its
        # 504 reaches the client before nginx gives up.
        proxy_read_timeout 70s;
    }
}

server {
    listen 80;
    listen [::]:80;
    server_name webhooker.example.com;
    return 308 https://$host$request_uri;
}
```

With that block, webhooker's environment is:

```sh
BIND_ADDRESS=127.0.0.1   # the default
TRUSTED_PROXIES=127.0.0.1
```

If nginx runs on another host, `BIND_ADDRESS` becomes the address nginx connects
to, and `TRUSTED_PROXIES` becomes nginx's address. Firewall the port to that
address: the traffic between them is plain text.

### HSTS

HSTS is a response header that tells browsers to use HTTPS only for this
hostname. Every response carries it unless `HSTS_MAX_AGE` is `0`. With the
default of ten years it reads:

```
Strict-Transport-Security: max-age=315360000; includeSubDomains; preload
```

A browser that receives this header over HTTPS refuses plain HTTP to the
hostname until `max-age` runs out. Browsers ignore the header over plain HTTP.
So serve webhooker on a hostname you will keep on HTTPS.

`includeSubDomains` extends the rule to every name under the hostname. `preload`
invites browsers to put the domain on their built-in HSTS lists. The header
alone lists nothing. A listing happens when someone submits a registered domain
at `hstspreload.org`, and removal takes months.

Setting `HSTS_MAX_AGE=0` stops the header. Browsers that already received it
keep the rule until their `max-age` runs out. To end the rule sooner, send a
short `max-age` such as `1s` until returning browsers have picked it up.

## Recovering a lost admin password

`webhooker resetpw` sets an existing account's password from the command line.
The deployment must be stopped first: the command takes the same `DATA_DIR` lock
the server does and refuses while a running instance holds it.

```bash
# Generate a new password and print it.
DATA_DIR=/var/lib/webhooker webhooker resetpw -generate admin

# Or supply one on standard input (at least 8 characters).
printf '%s' "$NEW_PASSWORD" | \
    DATA_DIR=/var/lib/webhooker webhooker resetpw admin
```

The password is never taken as a command-line argument, because on Linux every
account on the host can read a running process's arguments. Standard input is
echoed when it is a terminal, and the prompt says so.

The command creates nothing. A `DATA_DIR` that does not exist, or that holds no
`webhooker.db`, is an error. An unknown username is an error. The new password
takes effect at the next sign-in. Sessions already signed in are unaffected.

`webhooker help` lists the subcommands.

## Backup and restore

### What to back up

All state lives in `DATA_DIR`. Back up the whole directory. It holds:

- `webhooker.db`, the main database, one per install;
- `events-{webhook_uuid}.db`, one event database per webhook;
- `archive-{webhook_name}-{target_name}-{target_uuid}.db`, one per archive
  target, or one per period for a target that rotates, with the period before
  `.db`;
- `webhooker.lock`, which is always empty and holds no state.

[Data files](#data-files) says what each database holds.

Every database runs in WAL mode. WAL is SQLite's write-ahead log: finished
writes land in a `-wal` file beside the database before they reach the `.db`
itself. So a running service has two more files beside each `{name}.db`:

- `{name}.db-wal`, the log itself;
- `{name}.db-shm`, the shared index SQLite keeps for that log.

The `-wal` file is part of the database. A copy of the `.db` without it is
missing data and may have no readable tables at all.

A clean shutdown closes every database and removes its `-wal` and `-shm` files.
A crash leaves them, and they must be carried with the `.db`.

Keep `DATA_DIR` on a local disk. WAL mode is not reliable on network
filesystems, and webhooker does not check for one.

Configuration is not in `DATA_DIR`. It comes from the environment and from a
`.env` file in the working directory. Back those up with your deployment.

### A running copy is not safe

Databases are held open for the life of the process. `cp`, `rsync`, `tar` or a
filesystem snapshot against a running service can catch a `.db` and its `-wal`
at two different instants. Use one of these two procedures.

Stop, copy, start. This is the only way to get one point in time across every
file:

```bash
docker stop webhooker
cp -a /path/to/data /path/to/backup-$(date -u +%Y%m%dT%H%M%SZ)
docker start webhooker
```

SQLite online backup, one file at a time, with no downtime:

```bash
for db in /path/to/data/*.db; do
    sqlite3 "$db" ".backup '/path/to/backup/$(basename "$db")'"
done
```

`.backup` writes one complete file, with no `-wal` beside it. The runtime image
has no `sqlite3`, so run this on the host or from another container with the
volume mounted. Each file is captured at its own instant. So `webhooker.db` and
an event database copied at different moments can disagree: a webhook created
between the two copies is in one and has no event database in the other.

Archive files are the exception. The archive writer closes and reopens its file
after a write, at most once per second. So you can move a finished archive file
away while the service runs. Move its `-wal` and `-shm` with it, or wait until
there are none. The next write recreates a missing archive file.

### Restore

1. Stop the service.
2. Restore `webhooker.db`, every `events-*.db` and every `archive-*.db`
   together. A missing `events-{uuid}.db` is created empty at the next start,
   and the webhook comes back with its settings but no history. That start logs
   `created a new, empty database` at `WARN` with the file's path. A missing
   archive file is recreated silently at the next write.
3. Carry any `-wal` and `-shm` files that are in the backup.
4. Start the service. The container fixes the owner of the restored files.

### Backups contain secrets

Treat a backup like the credentials inside it. Encrypt it at rest and restrict
who can read it.

- The event and archive databases hold the full headers and body of every event,
  including whatever the sender put in them.
- `webhooker.db` holds target settings unencrypted, next to the session key and
  the password hash.

## Security

### The entrypoint URL is the credential

Each entrypoint is a random version 4 UUID served at `/h/{uuid}`. Holding that
URL is what authorises a sender. The receiver checks nothing else: no shared
secret, no HMAC signature (a hash of the body made with a shared key), no token
header. Signature headers a sender adds anyway, such as `X-Hub-Signature`, are
stored and forwarded as ordinary headers.

What that means in practice:

- Keep the URL out of tickets, chat and screenshots. webhooker's own log never
  writes it. The reverse proxy's access log does hold it, in the request path.
- To replace a URL, create a new entrypoint and point the sender at it. Then
  deactivate or delete the old one.
- Some senders cannot be given a secret URL: they post to one fixed address and
  prove themselves with a signature instead. webhooker checks no signatures, so
  such a sender cannot use it. Pick another integration path for it.

### The web interface

- Passwords are hashed with Argon2id, a password hashing function designed to be
  slow and to need a large amount of memory. The parameters are:
    - 64 MB of memory;
    - 1 pass;
    - 4 threads;
    - a 32-byte key;
    - a 16-byte salt.
- Session cookies are signed, `HttpOnly` (hidden from page scripts) and
  `SameSite=Lax` (not sent with cross-site form posts). They carry `Secure` when
  the request arrived over TLS, judged per request from a direct TLS connection
  or `X-Forwarded-Proto: https`.
- A new session is issued at sign-in. This prevents session fixation, an attack
  where a victim is made to use a session the attacker already knows.
- The key that signs the cookies is 32 random bytes made on first start and
  stored in `webhooker.db`.
- Every form is protected against cross-site request forgery (CSRF), where
  another site makes your browser submit a form here. Each form carries a token
  tied to a cookie. A submission with an `Origin` header must name this host.
  Over TLS, a submission without `Origin` must carry a `Referer` that names this
  host.
- Form bodies are capped at 1 MB. A larger declared body is answered `413`
  before anything reads it.
- Every response carries these headers:
    - `Strict-Transport-Security`, unless `HSTS_MAX_AGE` is `0`;
    - `X-Content-Type-Options: nosniff`;
    - `X-Frame-Options: DENY`;
    - `Content-Security-Policy` (CSP), which tells the browser where scripts and
      styles may come from, set to
      `default-src 'self'; script-src 'self'; style-src 'self'`;
    - `Referrer-Policy: strict-origin-when-cross-origin`;
    - a `Permissions-Policy` that denies camera, microphone and location.
- Authenticated pages are sent with `Cache-Control: no-store`.
- `/metrics` sits behind HTTP basic authentication, a username and password sent
  with each request.

### Sign-in limits

The sign-in form has no rate limiter in front of it, on purpose. Behind a proxy
that `TRUSTED_PROXIES` does not cover, every client shares one bucket. A limit
that counted every request would let any stranger lock the operator out with a
few requests a minute.

Instead, the password is checked first, and only a failure is counted. A correct
password is never throttled. Failures are counted per client and submitted
username. From the fifth failure within a minute, each failure is answered `429`
with `Retry-After`. A successful sign-in clears the count. At most 1024 username
counters and 1024 client counters are kept.

Two password checks can run at once; each running check holds one of two slots.
Each check takes 64 MB of memory, so hashing is capped at 128 MB. A request that
gets no slot within 5 seconds is answered `503` without hashing anything. So is
one that arrives when 16 requests are already waiting for a slot. The password
change form shares these slots.

The check runs before the count, so the two slots, not the failure count, bound
how fast passwords can be guessed. A flood of wrong passwords is tried as fast
as two Argon2id checks allow. Choose a long random admin password. To put a
cheaper limit on top, rate-limit `POST /pages/login` at the reverse proxy, where
the real client address is visible.

An unknown username is checked against a placeholder hash, so it costs the same
time as a real one.

A flood can hold both slots and make sign-in answer `503` for as long as it
runs. Nothing accumulates: the operator's correct password works on the first
try once the flood stops. Restarting webhooker is no remedy. Block the source at
the proxy.

### Delivery destinations

A delivery target could be used to make webhooker send requests into the network
it sits in and read the answers back. This is called server-side request forgery
(SSRF). webhooker refuses it with the SSRF guard, described here.

Every target URL must use `http` or `https` and resolve to a public address.
These private and reserved ranges are refused:

- loopback, `127.0.0.0/8` and `::1`;
- the RFC 1918 private ranges, `10.0.0.0/8`, `172.16.0.0/12` and
  `192.168.0.0/16`;
- carrier-grade NAT space, `100.64.0.0/10`, which internet providers use inside
  their own networks;
- link-local space, `169.254.0.0/16` and `fe80::/10`, which is valid only on one
  network segment;
- the IPv6 unique local range, `fc00::/7`;
- multicast, the ranges reserved for documentation, and the other reserved
  ranges.

One public address is refused too: `168.63.129.16`, the Azure service that hands
a virtual machine its credentials.

The SSRF guard runs when a target is saved and again when each delivery
connects. Both use the same function. The address is resolved again at connect
time, so a hostname that changes its address after validation is still checked.

Redirects are followed up to ten hops. Each hop is checked the same way. A hop
that leaves the target's origin carries none of the target's configured headers
and none of the forwarded event headers. Leaving the origin means a different
host, a different port, or a step from `https` to `http`. An open redirect is a
destination that forwards requests to any URL a caller names. Because of this
rule, an open redirect cannot pass those headers to another host.

Egress means the connections webhooker opens outward to deliver.
`ALLOWED_EGRESS_CIDRS` opens named networks to it, for the common case of
forwarding to your own network. Read
[Allowing egress to your own network](#allowing-egress-to-your-own-network)
before setting it.

### What the logs contain

webhooker writes structured logs, one record per line with named fields, to
standard output: plain text on a terminal, JSON otherwise. Each request writes
one access log line, `http request`, at `INFO`. A request to a known entrypoint
writes a second line, `webhook request received`. A stored event writes a third,
`webhook event created`.

Fields a client can influence are cut to a fixed size, so one access log line is
at most 2,560 bytes. The access log never records the query string. On the
receiver route the URL is logged as `/h/{uuid}`, never the real path, so
entrypoint UUIDs stay out of the log. On other routes a `3xx` or `4xx` response
logs the route pattern, which is the route as registered, such as
`/user/{username}`. A `2xx` or `5xx` logs the real path. The `Referer` is logged
as its scheme and host only.

Every line that names a client carries two fields. `remoteIP` is the connecting
machine, which behind a proxy is the proxy. `clientIP` is the client as decided
by `TRUSTED_PROXIES`.

SQL statements are logged with `?` in place of each value, never with the values
themselves. Outside a `log` target's line, no line carries session cookies or
target credentials.

The `admin` password is logged in the clear once, on the first boot, and again
when `webhooker resetpw -generate` is run. The operator's own configuration is
echoed back in the clear: webhook names and target hostnames.

A `log` target writes the whole event to the log: query string, headers and
body. That is its purpose.

A recovered panic is a crash inside one request that the server caught. It
writes one `ERROR` record of at most 10,240 bytes. The record carries the same
`request_id` as the request's access log line, which ties the two together.

When `SENTRY_DSN` is set, error reports go to Sentry, an error reporting
service, with the request reduced to this:

- the URL with its path replaced by the route pattern;
- the query string and the body replaced by `(redacted)`;
- no cookies;
- only these headers: `Accept`, `Content-Length`, `Content-Type`, `Host`,
  `Origin`, `Referer` (scheme and host only), `User-Agent` and `X-Request-Id`.

## Design

webhooker is one process. An HTTP server receives requests, a delivery engine
sends them on, and SQLite holds everything in between.

The HTTP server runs every request through middleware, code that runs around
each request. The middleware writes the access log, sets the security headers,
applies a 60 second timeout that answers `504`, and recovers from a crash inside
the request. On the page routes it also caps the body at 1 MB, applies the CSRF
protection and checks the sign-in. On the receiver route it applies the two rate
limits. The handler for the route then does the work.

The receiver's handler stores the event, one pending delivery per active target,
and the running totals in one database write. It then queues one task per
delivery. The delivery engine runs ten background workers that take tasks from
two queues, one for new deliveries and one for retries, and run the target's
delivery. Each attempt is recorded, and the delivery's status is updated. A
recovery routine re-queues deliveries the engine lost track of.

The main database holds configuration. Each webhook has its own event database,
so a busy webhook does not slow the others. Archive targets write to files of
their own. [Data files](#data-files) names each file.

The components are wired by [Uber fx](https://go.uber.org/fx), a library that
builds each component when another needs it and runs their start and stop hooks.
The other libraries are:

- [chi](https://github.com/go-chi/chi), an HTTP router, matches requests to
  handlers.
- [GORM](https://gorm.io), a Go database library, with
  [modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite), a pure Go SQLite
  driver, handles the databases.
- `log/slog`, the standard library's structured logger, writes the logs.
- [gorilla/sessions](https://github.com/gorilla/sessions) and
  [gorilla/csrf](https://github.com/gorilla/csrf) handle sessions and the CSRF
  protection.
- [go-chi/httprate](https://github.com/go-chi/httprate) rate-limits the
  receiver, the password change, replay and resubmit.
- The [Prometheus](https://prometheus.io) library and
  [go-http-metrics](https://github.com/slok/go-http-metrics) serve the metrics.
- The [Sentry](https://sentry.io) library reports errors when configured.

Most of the Go code is under `internal/`, one package per concern. The program
starts in `cmd/webhooker/main.go`. `templates/` holds the pages' HTML, and
`static/` their CSS and JavaScript, both embedded in the binary.

## Configuration reference

All configuration comes from environment variables. For development, variables
can also go in a `.env` file in the working directory. It is read once at
startup. A variable set in the real environment wins over the file. A `.env`
file that cannot be parsed aborts startup, because a single bad line would
otherwise silently drop every variable in the file.

A variable that is unset or empty takes its default. A variable that is set but
cannot be parsed aborts startup with a message naming it. `PORT=eighty`,
`DEBUG=ture` and `RETENTION_SWEEP_INTERVAL=1 hour` all abort. Each section below
says what range its variable accepts.

A Go duration is a number with a unit, such as `30s`, `15m`, `24h` or `1h30m`.

The startup log line `Configuration loaded` shows the loaded values, with three
differences. `TRUSTED_PROXIES` and `ALLOWED_EGRESS_CIDRS` appear as counts.
`SENTRY_DSN` appears as on or off. The metrics credentials appear as one
on-or-off field. The `/settings` page shows every value; there,
`METRICS_PASSWORD` and `SENTRY_DSN` show only as set or not set.

### `WEBHOOKER_ENVIRONMENT`

`dev` or `prod`. Default `prod`.

This controls one thing: CORS. In `dev`, every origin is allowed with
`Access-Control-Allow-Origin: *`, for local testing. In `prod`, no CORS headers
are sent. Cookie security and log format do not depend on it.

### `PORT`

The TCP port the listener binds. Default `8080`. It must be a number from 1
to 65535.

### `BIND_ADDRESS`

The IP address the listener binds. Default `127.0.0.1`. The Docker image sets
`0.0.0.0`.

The value must be an IP address literal. `localhost` is refused, because it may
mean either `127.0.0.1` or `::1`. A value with a port, such as `127.0.0.1:8080`,
is refused; the port belongs to `PORT`.

- `127.0.0.1` answers only on this host. Use it with a reverse proxy on the same
  host.
- `0.0.0.0` answers on every IPv4 address and no IPv6 address.
- `::` answers on every IPv6 and IPv4 address.
- A specific address such as `10.0.0.5` answers on that interface only.

An address that parses but is not on this host passes configuration and fails at
bind time. The process logs the error and exits non-zero. The `bindAddress`
field of the `Configuration loaded` line shows the configured address.

### `DATA_DIR`

The directory for every SQLite database and the lock file. Default
`/var/lib/webhooker`. It is created if it is missing.

Exactly one webhooker process may use a `DATA_DIR` at a time. Two processes on
one directory would both run the recovery routine and deliver every pending
event twice. At startup the process locks `{DATA_DIR}/webhooker.lock` with
`flock`, the kernel's file lock, and holds the lock until it exits. A second
process pointed at the same directory prints a message naming it and exits
non-zero.

The kernel releases the lock when the process exits, even on `kill -9`. A
leftover `webhooker.lock` blocks nothing and must not be deleted by hand. To run
two webhookers on one host, give each its own `DATA_DIR`.

`flock` works between processes and containers that share a directory on one
host. It does not work across hosts on a network filesystem.

### `DEBUG`

`true` or `false`. Default `false`. It accepts the twelve spellings Go's
`strconv.ParseBool` accepts, such as `true`, `1` and `false`. `yes` and `on` are
refused.

`DEBUG=true` lowers the log level to `DEBUG`. Lines at that level include:

- every SQL statement, with `?` in place of each value;
- the lookup misses on the receiver and sign-in routes;
- signed-out requests sent to the sign-in page;
- retries being scheduled, and retries skipped because the delivery had already
  been handled;
- the receiver's whole-route rate limit refusing a request;
- a target deleted while the archive sweep was running;
- the records of how the components were wired at startup.

Debug lines never carry SQL values, session cookies or target credentials. The
first-boot password is printed whatever the level, and so is every event a `log`
target writes.

### `METRICS_USERNAME` and `METRICS_PASSWORD`

The basic authentication credentials for `/metrics`. Default empty. Set both or
neither. With both set, `/metrics` is served behind basic authentication. With
neither set, the route does not exist and answers `404`. With only one set,
startup fails with an error naming both variables.

### `SENTRY_DSN`

The DSN for Sentry error reporting: the URL that tells the Sentry library where
to send reports. Default empty, which leaves reporting off. A set value is
parsed with the Sentry library's own parser, and one it cannot parse aborts
startup. The release name sent to Sentry is `webhooker-{version}`.

### `RETENTION_SWEEP_INTERVAL`

How often the retention sweep and the archive sweep run. A Go duration greater
than zero. Default `1h`.

The retention sweep deletes events older than each webhook's retention period,
with their deliveries and attempts, in batches of 1000. The archive sweep prunes
rows past each archive target's expiry.

### `SESSION_IDLE_TIMEOUT`

How long a session may go unused before it ends. A Go duration. Default `24h`. A
zero or negative value disables idle expiry. The 7 day absolute limit always
applies.

Only authenticated requests count as use. The idle timestamp in the cookie is
rewritten at most once per tenth of the window, so a session can end up to 10%
early, never late.

### `HSTS_MAX_AGE`

The `max-age` of the `Strict-Transport-Security` header, as a Go duration in
whole seconds. Default `87600h` (ten years). `0` sends no header. `1.5s` and
`-1h` are refused. Read [HSTS](#hsts) before changing it.

### `RECEIVER_RATE_LIMIT`

How many requests per minute one client may send to one entrypoint. Default
`120`. It must be at least 1. A client over the limit is answered `429` with
`Retry-After`.

A second limit covers the whole `/h/` route per client at ten times this value,
1200 by default. It exists because the route matches any single path segment. A
client inventing a new path per request would otherwise get a fresh bucket each
time. Each such request still costs a database lookup. The second limit is not
configurable on its own.

### `TRUSTED_PROXIES`

A comma-separated list of networks in CIDR notation, which names a network as an
address and a prefix length, such as `10.0.0.0/8`. A bare address such as
`192.168.1.7` counts as a single host. Default
`10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`, the RFC 1918 private ranges. A set
value replaces the default.

The list decides whose `X-Forwarded-For` header is believed. `X-Forwarded-For`
is the header a proxy uses to pass on the address of the client it is serving.
When the connecting machine is inside the list, the client address is read from
that header. For any other connecting machine, the client is that machine itself
and the header is ignored. `X-Real-IP` and `True-Client-IP` are never read.

The header is read from right to left. The first entry that is not itself a
trusted proxy is the client. An entry that is not a bare IP address ends the
walk, and the connecting machine's address is used instead. At most 64 entries
are examined. Your proxy must therefore append the connecting address to
`X-Forwarded-For`, as nginx's `$proxy_add_x_forwarded_for` does.

If any client can reach webhooker, or the proxy in front of it, from a private
address, set the list to the proxy's address alone. Otherwise those clients can
choose their own `X-Forwarded-For` and bypass every rate limit. The address to
set is the `remoteIP` field of an `http request` log line for a request that
came through the proxy.

Behind a proxy the list does not cover, every client shares one bucket per
limit. The receiver limits become service-wide ceilings. Sign-in failures from
every client land in one count, though a correct password is still never
throttled.

Rate-limit buckets are per IPv4 address and per IPv6 `/64` prefix.

### Allowing egress to your own network

Egress means the connections webhooker opens outward to deliver.
`ALLOWED_EGRESS_CIDRS` is a comma-separated list of networks in CIDR notation
that delivery targets may reach although the default rules refuse them. Default
empty. A bare address counts as a single host. Example:
`10.0.0.0/8, 172.17.0.0/16`.

This only ever adds destinations. The default refusals stay in force for every
address not listed. When the list is set, startup logs the whole list at `WARN`.

Anyone with the admin account can make webhooker send requests to every network
you list. They can read the responses in the event log. List the smallest blocks
that cover your destinations. Prefer `10.1.2.3/32` over `10.0.0.0/8`.
`0.0.0.0/0` or `::/0` opens every private and reserved range at once, including
this host's own loopback. Do not list them.

Some addresses stay blocked whatever you list. They are link-local space, the
unspecified addresses `0.0.0.0` and `::`, and cloud metadata endpoints at fixed
non-public addresses. A cloud metadata endpoint is the address a cloud gives
each virtual machine for its own credentials and settings. The blocked set is:

- `169.254.0.0/16` and `fe80::/10`, the link-local ranges. `169.254.169.254` is
  the metadata address on AWS, Azure, DigitalOcean, Hetzner, OpenStack and
  others.
- `fd00:ec2::254/128` and `fd00:ec2::23/128`, AWS metadata and EKS Pod Identity
  over IPv6.
- `fd20:ce::254/128`, Google Cloud metadata over IPv6.
- `fd00:c1::a9fe:a9fe/128`, Oracle Cloud metadata over IPv6.
- `fd00:42::42/128`, Scaleway metadata over IPv6.
- `fd00:a9fe:a9fe::1/128`, Linode and Akamai metadata over IPv6.
- `100.100.100.200/32`, Alibaba Cloud metadata, inside the carrier-grade NAT
  range.
- `192.0.0.192/32`, Oracle Cloud Classic metadata.
- `0.0.0.0/32` and `::/128`, the unspecified addresses, which reach this host's
  loopback on Linux.
- `::a9fe:a9fe/128` and `64:ff9b::a9fe:a9fe/128`, two other spellings of
  `169.254.169.254`.

The six `fd00::/8` entries matter because `fd00::/8` is an ordinary block to
list for your own IPv6 network. Without them, that one line would open cloud
credentials on five providers.

This list is not every cloud's metadata address. If yours is not here, do not
list the block that contains it. Azure's public `168.63.129.16` is refused by
default but is reopened by a list that covers it.

## Delivery reference

### Target types

An `http` target posts the event body to its URL. The request carries, in this
order:

1. the event's `Content-Type`;
2. the event's headers;
3. the target's configured headers, each replacing an event header of the same
   name;
4. `User-Agent: webhooker/{version}`.

A `2xx` response is a success. The first 4 KiB of the response body is stored
with the attempt.

These event headers are never forwarded:

- `Host`
- `Connection` and `Keep-Alive`
- `Transfer-Encoding` and `Content-Length`
- `Te` and `Trailer`
- `Upgrade`
- `Proxy-Authorization` and `Proxy-Connection`

An `http` target's settings:

- `url`, the destination. Only `http` and `https` are accepted.
- `headers`, one `Name: value` per line. `Content-Type` is allowed and replaces
  the event's. These names are refused, because the delivery sets them itself:
    - `Host`
    - `Content-Length`
    - `Transfer-Encoding`
    - `Connection`
    - `Trailer`
    - `User-Agent`
- `timeout_seconds`, the per-request timeout, at most 300. Blank means the
  shared 30 second default. A long timeout holds one of the ten background
  workers for its whole duration.
- `forward_query`, off by default. On, the event's query string is appended to
  the URL, joined with `&` if the URL already has one.

A `slack` target posts a JSON message with one `text` field to its URL. The
message names the method, content type, time received and body size, then shows
the body in a code block. A JSON body is indented. A body over 3500 bytes is
cut. The URL is any message-posting URL that accepts Slack's format, which
includes Mattermost's.

An archive target, whose stored type is `database`, appends the event as a row
to its own SQLite file. See [Archives](#archives). It makes no network request
and never retries. A write failure fails the delivery.

A `log` target writes the event, headers and body included, as one `INFO` line.
It never retries.

### Attempts and retries

`http` and `slack` targets take `max_attempts`, from 1 to 20. With 1, a failed
delivery is marked `failed` after its one attempt. With more, a failed attempt
marks the delivery `retrying` and schedules the next attempt. The wait doubles
each time, starting at one second. After the last allowed attempt fails, the
delivery is marked `failed`.

A delivery has one of four statuses. `pending` means created but not yet
attempted. `retrying` means at least one attempt failed and more remain.
`delivered` means an attempt succeeded. `failed` means every allowed attempt
failed.

Every attempt is recorded, except one cut off by shutdown.

### Circuit breaker

An `http` or `slack` target with `max_attempts` above 1 has a circuit breaker. A
circuit breaker stops hammering a destination that is down. It is in memory only
and resets when the service restarts.

The breaker is closed in normal operation. After 5 failures in a row it opens
for 30 seconds. That wait is the cooldown. While the breaker is open, each
delivery to the target is marked `retrying` and scheduled for when the cooldown
ends. When the cooldown ends, one delivery is sent as a probe. The others wait
one more cooldown. If the probe succeeds, the breaker closes. If it fails, the
breaker opens again.

The webhook page says when a target's deliveries are paused and until when. Each
waiting delivery in the event log shows the earliest time it can be tried.

### Recovery

The recovery routine finds deliveries the delivery engine is not working on and
queues them again. It runs once at start and then every 60 seconds; each
periodic run is a sweep. It reads the deliveries still `pending` or `retrying`
in every event database. One that already has a successful attempt recorded is
marked `delivered` and not sent again. A `retrying` delivery is scheduled for
whatever remains of its wait. A `pending` one is queued. A delivery the engine
is already working on is left alone.

At start, every `pending` delivery is taken, however recent. So a delivery cut
off by a stop moments earlier is sent again at once. A sweep takes only
`pending` deliveries untouched for 15 minutes, at most 500 per webhook each
time.

If a target was deleted while a delivery to it was still open, the recovery
routine marks the delivery `failed` and records the reason.

### Shutdown

On `SIGINT` or `SIGTERM`, the whole shutdown has a 5 second budget. The HTTP
server first waits for open requests to finish, for at most 3 seconds. That
leaves 2 seconds for the rest. A Sentry flush, if enabled, fits inside the same
budget and is skipped when little is left.

The delivery engine cancels any `http` or `slack` request in flight. That
attempt is not recorded, and the delivery keeps its status, so it is sent again
at the next start. An archive or `log` task finishes as normal.

A component that does not stop in time is logged at `ERROR`, and the components
after it are skipped, the database close included. The database files are then
left as a crash leaves them, with their `-wal` and `-shm` files beside them.

Keep the container's stop grace period above 5 seconds. That is the time Docker
waits after `docker stop` before it kills the process. Docker's default is 10.
The shutdown starts at the signal, and a shorter grace cuts it off before it
finishes.

### Archives

An archive target writes to
`archive-{webhook_name}-{target_name}-{target_uuid}.db` in `DATA_DIR`. The two
names are made safe for a file name:

- lowercased;
- only the ASCII letters `a` to `z` and the digits `0` to `9` kept, and every
  other run of characters turned into one `-`;
- cut to 40 characters;
- `unnamed` when nothing is left.

A webhook named `Orders (EU)` with a target named `Long-term archive` writes
`archive-orders-eu-long-term-archive-{target_uuid}.db`.

Each row holds:

- the event's id, and the ids of its webhook and entrypoint;
- the method, query string, headers, body and content type;
- the time the row was archived.

`archive_rotation` is `none` by default, which keeps one file. `monthly`,
`daily` and `hourly` start a new file for each period of the event's receive
time, in UTC, with the period before `.db`: `-2026-10.db`, `-2026-10-01.db` or
`-2026-10-01-19.db`. A changed rotation applies from the next event. Files
already written keep their names and are still shown, pruned and downloaded.

`archive_expiry` is `never` by default. The forms offer `1h`, `12h`, `24h`,
`30d`, `90d` and `365d`. Rows older than the expiry are deleted each time the
file is opened for a write, and by the archive sweep every
`RETENTION_SWEEP_INTERVAL`. The sweep deletes a period file it has emptied. It
never creates a file.

Renaming the webhook or the target renames every file of the target. If a file
with the new name already exists, the edit is refused and nothing moves. A file
the operator has moved away is skipped.

Deleting a webhook or a target leaves its archive files on disk. Removing them
is the operator's decision.

**Download** on the webhook page returns the target's whole archive as one
gzipped JSON file named
`archive-{webhook_name}-{target_name}-{YYYYMMDDTHHMMSSZ}.json.gz`. It holds one
object with these fields:

- `webhook`, with the webhook's `id` and `name`;
- `target`, with the target's `id` and `name`;
- `exported_at`;
- `archived_events`, one object per row with every column.

Rows from a period file carry a `period` field. A body that is not valid UTF-8
is written in base64 with `"body_encoding": "base64"` beside it. The download
streams one file at a time and creates nothing.

## Data files

The main database, `webhooker.db`, holds:

- the settings, which today are only the session key;
- the `admin` account and its password hash;
- the webhooks;
- the entrypoints;
- the targets.

Each event database, `events-{webhook_uuid}.db`, holds that webhook's:

- events;
- deliveries;
- delivery attempts;
- running totals, which the pages read instead of counting rows.

Each archive file, `archive-*.db`, holds one archive target's rows.

Every database is opened in WAL mode, with at most 4 connections. A connection
waits up to 10 seconds for a locked database before giving up.

An event database is created with its webhook. Every webhook's event database is
opened at start. One that is missing or empty is created again and logged with
`created a new, empty database` at `WARN`. On a deployment that has run before,
that line means the file was lost, most often because the volume holding
`DATA_DIR` is not mounted. A damaged file fails that webhook alone, with an
`ERROR` naming it and a `500` to its senders.

Deleting a webhook soft-deletes its row, entrypoints and targets in
`webhooker.db` and deletes its event database file. A soft delete marks the row
with a deletion time and hides it from the application; the row stays in the
file. Deleting an entrypoint or a target soft-deletes it.

Each kind of database file has its schema in SQL files built into the binary,
under `internal/db/migrations/`: `main/` for `webhooker.db`, `events/` for event
databases and `archive/` for archive files. Opening a database applies the files
it has not had yet.

## HTTP routes reference

Public routes:

- `GET /` redirects to `/hooks` when signed in and to `/pages/login` otherwise.
- `GET /.well-known/healthcheck` returns a JSON object with these fields:
    - `status`, always `ok`;
    - `now`, the current time;
    - `uptimeSeconds` and `uptimeHuman`;
    - `version` and `appname`.
- `GET /s/*` and `HEAD /s/*` serve the embedded CSS and JavaScript. Other
  methods get `405` with `Allow: GET, HEAD`.
- `POST /h/{uuid}` is the receiver. Its answers:
    - any method but `POST` gets `405` with `Allow: POST`;
    - an unknown UUID gets `404`;
    - a deactivated entrypoint gets `410`;
    - a body over 1 MB gets `413`;
    - a stored event gets `200` with `{"status":"ok"}`.

Sign-in routes:

- `GET /pages/login` shows the sign-in form. Its `next` parameter names the page
  to return to. Anything but a path on this site is replaced with `/`.
- `POST /pages/login` signs in. See [Sign-in limits](#sign-in-limits).
- `POST /pages/logout` signs out.

The pages behind the sign-in live under `/user/`, `/settings`, `/hooks` and
`/hook/`. They are reached from the web interface. A `GET` to one of them
without a session is redirected to `/pages/login`, with the page's path as
`next`.

`GET /metrics` exists only when `METRICS_USERNAME` and `METRICS_PASSWORD` are
set, behind basic authentication.

`/api/v1` is mounted with no routes. Every path under it answers `404`.

## Metrics reference

`/metrics` serves the delivery metrics below, three inbound HTTP metrics, and
the Go runtime and process metrics the Prometheus library provides.

Nine of the eleven metrics below carry one label, `target_type`, with one of
`http`, `slack`, `database`, `log` or `unknown`. The two event counters carry
none. Target ids and event ids are never labels, because a label value that
grows with traffic would grow the set of metrics without bound.

- `webhooker_events_received_total` (counter, no label): events stored,
  resubmitted copies included.
- `webhooker_delivery_attempts_total` (counter): attempts sent to a target. A
  delivery the circuit breaker turned away is not an attempt.
- `webhooker_deliveries_succeeded_total` (counter): deliveries that became
  `delivered`.
- `webhooker_deliveries_failed_total` (counter): deliveries that became
  `failed`.
- `webhooker_delivery_retries_total` (counter): deliveries put into `retrying`.
- `webhooker_delivery_replays_total` (counter): deliveries created by
  **Replay**.
- `webhooker_events_resubmitted_total` (counter, no label): events created by
  **Resubmit**.
- `webhooker_delivery_duration_seconds` (histogram): wall time of one attempt.
- `webhooker_deliveries_pending` and `webhooker_deliveries_retrying` (gauges):
  deliveries in those statuses, counted from the event databases every 30
  seconds. Deliveries to a deleted target count under `unknown`.
- `webhooker_circuit_breakers_open` (gauge): breakers currently open.

The success and failure counters move only after the new status has been
written.

The inbound HTTP metrics are `http_request_duration_seconds`,
`http_response_size_bytes` and `http_requests_inflight`. Their `handler` label
is the route pattern, such as `/h/{uuid}`, never the real path. A request that
matched no route carries `(unmatched)`. `http_requests_inflight` is one series
(one metric with one set of label values) for the whole service, with `handler`
set to `(all)`.

## Development

### Prerequisites

- The Go version named in `go.mod`.
- Docker. The tests, the linters, prettier and the stylesheet build run in
  Docker.

`gofmt` and `goimports` run on the host. The tools below run only inside Docker,
at the versions pinned (fixed to one exact release) in the `Dockerfile`,
`package.json` and `yarn.lock`:

- `golangci-lint`, the Go linter;
- ESLint, the JavaScript linter;
- prettier, the Markdown formatter;
- node and yarn, which run the two above;
- the Tailwind CLI, which generates the stylesheet.

### Commands

```bash
make bootstrap    # Install git, make, Go and the Go modules (safe to rerun)
make setup        # bootstrap, then install the git pre-commit hook
make assets       # Extract Alpine.js from 3p/ to static/js/alpine.min.js
make fmt          # Format Go (gofmt, goimports) and Markdown (prettier, in Docker)
make fmt-check    # Fail if gofmt or prettier would change anything
make lint         # Run gofmt, golangci-lint and ESLint in Docker
make test         # Run the tests in Docker, with cgo off
make test-browser # Run the browser test in Docker (Dockerfile.browser)
make check        # test, lint, fmt-check and css-check
make build        # Build bin/webhooker, version-stamped
make version      # Print the version this checkout would stamp
make run          # build, then run ./bin/webhooker
make dev          # go run ./cmd/webhooker
make deps         # go mod download, then go mod tidy
make docker       # Build the Docker image
make hooks        # Install the git pre-commit hook
make css          # Regenerate static/css/tailwind.css in Docker
make css-check    # Fail if static/css/tailwind.css is out of date
make clean        # Remove bin/
```

Most `make` targets above are one-line wrappers: each runs one script in
`script/`. The exceptions are `build`, `run`, `dev`, `deps`, `version` and
`clean`, which have no script. `make hooks` runs `script/install-precommit`;
every other wrapper runs the script of the same name. The scripts are listed
under [Entrypoints](#entrypoints).

The tests run with cgo, Go's bridge to C code, turned off. Go's race detector,
which finds unsynchronised access to shared memory, needs cgo, so the tests run
without it.

### How the checks run

Each check is a stage of the `Dockerfile`. A stage is one named step of the
image build. `script/test` builds the `test` stage, `script/lint` the `lint`
stage and `script/css-check` the `css-check` stage. `script/fmt-check` runs
`gofmt` on the host and builds the `markdown-check` stage. Every `docker build`
in `script/` passes `--no-cache`, so Docker reruns every step instead of reusing
an earlier result. So a check that passes has really run. A bare
`docker build .` carries no such guarantee.

The `lint` stage runs these steps:

1. the `gofmt` check;
2. `golangci-lint config verify`, with `--network=none`;
3. `golangci-lint run`, with `--network=none`;
4. ESLint over `static/js/`, in the `js-lint` stage, with `--network=none`.

The `test` stage runs `go test -cover`. On a failure it runs only the failed
tests again with `-v`.

The image build depends on the `lint`, `test`, `css-check` and `markdown-check`
stages, so an image that built is a tree that passed every check. The runtime
stage is Alpine Linux. It adds `ca-certificates` and `su-exec`, a tool that
switches user. It has no `sqlite3`.

CI, the check run on every push, is the `check` workflow in
`.gitea/workflows/check.yml`. It checks out the repository and runs
`script/cibuild`, nothing else.

### Version stamping

The binary reports its version at `/.well-known/healthcheck`, in the page
footer, in the `starting` log line and as the Sentry release. The value is
stamped into the binary at build time.

`script/version` produces it. With `VERSION` set in the environment, that value
is used. Otherwise it is `git describe --tags --always --dirty` on this
checkout, which gives:

- the tag at a clean tagged commit, such as `v1.0.0`;
- the tag, the commit count and the short hash past a tag, such as
  `v1.0.0-3-g1a2b3c4`;
- the short hash alone when no tag is reachable;
- a `-dirty` suffix on any of these for uncommitted changes.

With no git metadata it is `unknown`.

`make build VERSION=v1.2.3` overrides it. `script/docker` and `script/cibuild`
run the same `git describe` on the host and pass the result in as the `VERSION`
build argument. Without that argument the image build reads the `.git` among the
files sent to it, and fails if `.git` is present but the version comes out
empty, `dev` or `unknown`. `.dockerignore` leaves out every git `config` file,
which can carry a credential, and keeps the rest of `.git`.

`make build` passes `-trimpath` and `-s -w`, so the binary carries no build
paths, symbol table or debug information.

### Third-party browser assets

The pages use one third-party script: Alpine.js, a small JavaScript library.
They use its CSP build, `@alpinejs/csp`, which runs under a
Content-Security-Policy that forbids `eval`. The standard build needs `eval`.
With the CSP build, every Alpine.js directive in `templates/` names a component
registered in `static/js/app.js`, such as `x-data="collapsible"`, and never an
inline expression.

The package file is committed as `3p/alpinejs-csp-3.14.9.tgz`. It is byte for
byte the file the npm registry, the JavaScript package registry, publishes.
`script/assets` extracts `package/dist/cdn.min.js` from it to
`static/js/alpine.min.js`, which the binary embeds. `make build`, `make dev` and
the Docker stages run it first. The extracted file is not committed, and a build
that skips the extraction fails with an error naming `js/alpine.min.js`.

To move to a new version:

1. Download `https://registry.npmjs.org/@alpinejs/csp/-/csp-<version>.tgz`.
2. Check it against the `dist.integrity` hash at
   `https://registry.npmjs.org/@alpinejs/csp/<version>`.
3. Replace the file in `3p/` and update the file name in `script/assets`.
4. Run `make check` and `make test-browser`.

### Browser test

`internal/server` has a test that loads the pages in a headless browser and
drives the interface under the real Content-Security-Policy. A headless browser
is one without a window. The test is built only with the `browser` build tag, a
flag given to the compiler, so `make test` leaves it out. `make test-browser`
runs it in Docker with a pinned headless browser image. Run it after changing
`templates/` or `static/js/`.

### Stylesheet

`static/css/tailwind.css` is generated by Tailwind and committed. To change the
styles, edit the templates, `static/js/app.js`,
`internal/handlers/recent_events.go` or `static/css/input.css`, then run
`make css` and commit the result. Tailwind reads classes only from the files
that `input.css` names in its `@source` lines. `make check` fails when the
committed file differs from what `make css` generates. `static/css/style.css` is
hand-written. The Content-Security-Policy refuses inline styles, so templates
style elements with classes only.

## Entrypoints

This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: the scripts in `script/` are the entry points for the development
workflow. This section is about those scripts. The receiving URLs of the
application are under [Entrypoint URLs](#entrypoint-urls).

- `script/bootstrap` installs git, make, Go and the Go modules. It is safe to
  rerun.
- `script/setup` runs `script/bootstrap`, then `script/install-precommit`.
- `script/projectname` prints `webhooker`, the name the other scripts use.
- `script/assets` extracts Alpine.js from `3p/` to `static/js/alpine.min.js`.
- `script/test` builds the `test` stage of the `Dockerfile`, which runs the
  tests.
- `script/test-browser` builds `Dockerfile.browser`, which runs the browser
  test.
- `script/lint` builds the `lint` stage, which runs `gofmt`, `golangci-lint` and
  ESLint.
- `script/fmt` formats the Go code with `gofmt` and `goimports` on the host, and
  the Markdown with prettier in Docker. It writes.
- `script/fmt-check` checks `gofmt` and prettier and fails instead of writing.
- `script/css` regenerates `static/css/tailwind.css` in Docker. It writes.
- `script/css-check` fails when `static/css/tailwind.css` differs from what
  `script/css` would generate.
- `script/check` runs `script/test`, `script/lint`, `script/fmt-check` and
  `script/css-check`.
- `script/version` prints the version to stamp into the binary.
- `script/docker` builds the image tagged `webhooker`, passing that version in.
- `script/cibuild` is what CI runs: `script/bootstrap`, then `script/check`,
  then the same image build as `script/docker`.
- `script/precommit` checks that `go mod tidy` changes nothing, then runs
  `script/check`.
- `script/install-precommit` installs the git pre-commit hook that runs
  `script/precommit`.

## TODO

See [`TODO.md`](TODO.md).

## License

MIT.

## Author

[@sneak](https://sneak.berlin)