webhooker/README.md

# webhooker

webhooker receives webhooks, stores them, and forwards them to the places you
choose. A webhook is an HTTP request that one service sends to another when
something happens. Senders usually make one attempt and keep no record. If your
receiver is down at that moment, the event is lost.

webhooker sits between the sender and your receiver. It is a single Go binary
with a web interface and SQLite storage. It accepts every event the sender posts
and writes it to disk before answering. It then delivers the event to each
target you configured, with retries. The web interface shows each webhook's
newest events, with every delivery and attempt they had. From there you can send
a stored event again. It also exports metrics for Prometheus, the monitoring
system, so you can watch the whole flow.

webhooker is written in [Go](https://golang.org) by
[@sneak](https://sneak.berlin). The license is MIT.

## How it works

Five words name the moving parts.

- A **webhook** is a named configuration. It groups entrypoints and targets, and
  it owns the events it receives.
- An **entrypoint** is a receiving URL of the form `/h/{uuid}`. Senders post to
  it. The UUID is a random 36-character identifier, and it is the only thing
  that authorises a sender.
- A **target** is a place events are delivered to. There are four kinds: an HTTP
  URL, a Slack or Mattermost incoming webhook, an archive database file, and the
  application log.
- An **event** is one stored request: its method, query string, headers and
  body.
- A **delivery** is one event paired with one target. It records every attempt.

A request to an entrypoint is stored as an event. In the same database
transaction, one pending delivery is created for each active target of the
webhook. The sender gets `200` once that transaction has committed. Workers then
carry out the deliveries. A delivery to an HTTP or Slack target that fails is
retried, up to the number of attempts the target allows.

Delivery is at least once, not exactly once. If a delivery reaches the target
but webhooker cannot record that outcome, it sends the delivery again later. A
delivery in flight when the service stops is also sent again at the next start.
webhooker adds no delivery identifier of its own to the request, so make your
receiver tolerate a duplicate.

## Quick start

You need the Go version named in `go.mod`, and Docker for the checks.

```bash
git clone https://git.eeqj.de/sneak/webhooker.git
cd webhooker
make bootstrap
DATA_DIR=../webhooker-data make dev
```

`DATA_DIR` is where webhooker keeps its databases. It defaults to
`/var/lib/webhooker` in every environment, so point it at a directory you can
write to. Keep it outside the clone: the databases hold the session key.

The first start creates the `admin` account and prints its password once, in a
ruled box on standard output:

```
========================================================================
WEBHOOKER FIRST BOOT: an admin account has been created.

    username: admin
    password: 3xamPl3-p4ssw0rd

Save this password now: it is shown only here, and only once.
If it is lost, run `webhooker resetpw admin` on a stopped deployment.
========================================================================
```

Open `http://127.0.0.1:8080/` and sign in. The server listens on `127.0.0.1`
port `8080` by default.

To run it in Docker instead, build the image from the clone with `make docker`,
which tags it `webhooker`. Then:

```bash
docker run -d \
  -p 127.0.0.1:8080:8080 \
  -v /path/to/data:/var/lib/webhooker \
  webhooker
```

Read [Running with Docker](#running-with-docker) before publishing the port
anywhere but `127.0.0.1`.

## Using the web interface

### Signing in

The sign-in page is `/pages/login`. There is one account, `admin`, and no way to
add another through the web interface. A session lasts up to 7 days. It ends
sooner after 24 hours without use, unless `SESSION_IDLE_TIMEOUT` says otherwise.

If a page asked for a sign-in, signing in returns you to that page.

### Creating a webhook

The webhook list is at `/hooks`. **New Webhook** asks for a name, an optional
description and a retention period in days. Events older than the retention
period are deleted, along with their deliveries. The default is 30 days. `0`
keeps events forever.

The form can also create the first targets. An HTTP target URL creates an `http`
target named `HTTP`. The **Archive** checkbox creates an archive target named
`Archive`, with the expiry and rotation chosen beside it.

Every new webhook gets one entrypoint, described as `Default entrypoint`.

### Entrypoints

The webhook page lists each entrypoint with its full URL and a **Copy** button.
Give that URL to the sender. Anyone who holds it can post events, so treat it
like a password. Read
[The entrypoint URL is the credential](#the-entrypoint-url-is-the-credential).

**Add** creates another entrypoint with its own URL. Several entrypoints on one
webhook let you tell senders apart while delivering to the same targets.

**Edit** changes an entrypoint's description. Its URL never changes. To retire a
URL, deactivate the entrypoint or delete it, then create a new one.

A deactivated entrypoint answers `410 Gone`. A deleted or unknown one answers
`404`.

Beside each entrypoint the page shows when its last event arrived, and how many
events arrived on it within the retention period.

### Targets

**Add** under **Targets** asks for the type first, then that type's settings.

- An `http` target posts each event to a URL. It takes optional request headers,
  a timeout in seconds, and a switch to pass the event's query string on.
- A `slack` target posts each event as a message to a Slack or Mattermost
  incoming webhook URL.
- An archive target writes each event as a row in a SQLite file of its own. Its
  type is `database` in the data. It takes an expiry and a rotation.
- A `log` target writes each event to the application log. It has no settings
  beyond its name.

`http` and `slack` targets also take **Delivery attempts**: how many times each
delivery is tried in all. The default is 1. The most is 20.

The target list shows each target's settings. A destination URL is shown as its
scheme and host, with `/...` standing for its path. Headers are shown as a
count. The target edit page is the one place that shows a URL and its headers in
full.

The list also shows, for each target, how many deliveries have ever succeeded
and failed, and how many did so in the last 24 hours. For an archive target it
names the current archive file, the size of all its files together, and when the
last write happened.

**Deactivate** stops new events reaching the target. Deliveries already created
are unaffected. **Delete** removes the target. An archive target's files stay on
disk.

A target's type cannot be changed. Create a new target to deliver another way.

### Statistics

The top of the webhook page shows counts of entrypoints and targets, deliveries
in progress, the time of the last event, and the retention setting.

Below that are events, deliveries and failures, both lifetime and within
retention. Then come the last 10 minutes and the last 24 hours: events,
failures, and the failure percentage. The percentage is failed deliveries out of
all deliveries that finished in the window.

The webhook list shows three of these figures per webhook: events within
retention, the last event, and failed deliveries in the last 24 hours.

### Recent events and the event log

The webhook page shows the 50 newest events. Each expands to show its body. A
body is shown as text up to 32 KiB. A longer or binary body has a **Download the
body** link to the stored bytes.

**Full Event Log** at `/hook/{id}/events` shows the 50 newest events with their
deliveries and every attempt. **Failed** narrows it to events with a failed
delivery. **Pending** narrows it to events with a delivery still pending or
retrying. Each attempt shows its time, status code, duration, error, and the
first 4 KiB of the response.

Each event has a page of its own at `/hook/{id}/events/{eventID}`. It shows the
whole request: query string, headers, body and every delivery.

### Replay and resubmit

**Replay** sits beside each finished delivery. It creates a new delivery of the
same event to the same target and sends it on the ordinary path. The target's
settings as they stand now are used, so a replay is how you resend after fixing
the destination. The original delivery is left as it was. The new one is marked
as a replay.

A replay is refused in three cases:

- the delivery has not finished;
- its target is deactivated or deleted;
- another delivery of that event to that target is already in flight.

A delivery whose target was deleted shows no **Replay** button.

**Resubmit** sits beside each event. It stores a copy of the event as a new
event and delivers it to every target that is active now. That includes targets
created after the original arrived. The copy's row names the event it came from.
The original's row shows how many copies were made. Resubmitting the same event
many times is allowed.

Both actions are limited to 30 per minute per client.

### Profile and settings

`/user/{username}` shows your account and lets you change your password. The
current password is checked first. The new password must not be empty and must
match its confirmation. Password changes are limited to 5 per minute per client.

`/settings` lists every configuration variable with the value the running server
loaded. It is read only. `METRICS_PASSWORD` and `SENTRY_DSN` show only as set or
not set.

## Deploying

### Running with Docker

No image is published. `make docker` builds one from the clone and tags it
`webhooker`. Then:

```bash
docker run -d \
  -p 127.0.0.1:8080:8080 \
  -v /path/to/data:/var/lib/webhooker \
  webhooker
```

The image listens on every address inside the container: it ships
`BIND_ADDRESS=0.0.0.0`. A container has its own network namespace, which means
its own addresses, separate from the host's. That is the boundary, and the `-p`
flag decides what the host exposes. Publish to `127.0.0.1:8080`, the address
your reverse proxy connects from, and nothing wider. A bare `-p 8080:8080` opens
the port on every interface of the host, and Docker's forwarding rules sit ahead
of most host firewalls.

Do not set `BIND_ADDRESS` to an empty value in a container. An empty value means
unset, and unset falls back to the binary's default of `127.0.0.1`. The
container then answers nothing on its published port and its health check fails.
A templated Compose file with an empty value is the usual way this happens.

A container on an IPv6-only Docker network needs `BIND_ADDRESS=::`.

The app runs as user `webhooker` (UID 1000). The image's entrypoint script
starts as root. It creates `DATA_DIR` if it is missing, gives it and its
contents to `webhooker`, and sets the directory to mode `0750`. Then it runs the
app as `webhooker`. So a host directory can be mounted as it is, whoever owns
it. With `--user`, the script changes nothing and runs the app as that user.

webhooker creates every SQLite file with mode `0600`, and resets a file found
with another mode when it opens it. `webhooker.db` holds target settings in
plain text, including bearer tokens and Slack URLs, so the file mode is what
protects them.

The image's health check requests `/.well-known/healthcheck` on the container's
own address every 30 seconds.

The same binary runs the password reset. The image's `CMD` is `/app/webhooker`,
and a command given to `docker run` replaces all of it:

```bash
docker run --rm -v webhooker-data:/var/lib/webhooker \
    webhooker /app/webhooker resetpw -generate admin
```

### Running under upaas

[upaas](https://git.eeqj.de/sneak/upaas) builds the image from this repository's
`Dockerfile` and runs it.

- Add no port mapping. upaas publishes every mapped port on all interfaces of
  the host. Instead, set the app's Docker network to your reverse proxy's
  network. The proxy then reaches the app at `upaas-` followed by the app name,
  on port `8080`.
- Leave `PORT`, `BIND_ADDRESS` and `DATA_DIR` unset. The image's health check
  probes port `8080`.
- Mount one host directory at `/var/lib/webhooker`.
- Set `TRUSTED_PROXIES` to your reverse proxy's address if clients can reach the
  proxy from a private address. See [`TRUSTED_PROXIES`](#trusted_proxies).

The first start prints the `admin` password to the container's log. For an app
named `webhooker`:

```bash
docker logs upaas-webhooker
```

If the password is lost, stop the container, reset the password with the app's
own image and volume, and start it again:

```bash
docker stop upaas-webhooker
docker run --rm --volumes-from upaas-webhooker \
    "$(docker inspect -f '{{.Image}}' upaas-webhooker)" \
    /app/webhooker resetpw -generate admin
docker start upaas-webhooker
```

### Behind a reverse proxy

webhooker speaks plain HTTP and terminates no TLS. Put a reverse proxy in front
of it. The proxy holds the certificate and reaches webhooker over loopback.

Five things must be right.

1. **Keep the app port private.** The binary binds `127.0.0.1` by default. The
   image binds every address inside its container, so publish its port to
   `127.0.0.1` only. The plain port serves the sign-in form and the receiver
   without TLS.
2. **Send `X-Forwarded-Proto: https`.** This header tells webhooker the request
   arrived over TLS. It decides the `Secure` flag on cookies and the strict
   origin check on forms. Without it, cookies are sent without `Secure`.
   webhooker reads this header from any peer, so the proxy must overwrite
   whatever the client sent.
3. **Cover the proxy in `TRUSTED_PROXIES`.** Otherwise every client shares one
   rate-limit bucket, the count kept per client, keyed on the proxy's address.
   The default list is the private ranges, which does not cover a proxy that
   connects over loopback.
4. **Send `Host` as `$http_host`, not `$host`.** `$host` drops the port. The
   form origin check compares against the host it was given. On any port other
   than 443 every form submission then fails with `403`.
5. **Keep the proxy's access log.** It records the client address whatever
   `TRUSTED_PROXIES` says. It also records the full request path, which holds
   the entrypoint UUID of every event. Protect that log as you would the
   entrypoint URLs.

`WEBHOOKER_ENVIRONMENT` decides none of this. It controls only CORS, the headers
that let a page on another site call this one from a browser. Leave it unset in
production.

#### nginx

A complete server block. Replace the `server_name` and the certificate paths.

```nginx
server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;  # nginx 1.25.1+; older: listen 443 ssl http2;

    server_name webhooker.example.com;

    ssl_certificate     /etc/ssl/certs/webhooker.example.com.crt;
    ssl_certificate_key /etc/ssl/private/webhooker.example.com.key;
    ssl_protocols       TLSv1.2 TLSv1.3;

    # webhooker caps form and receiver bodies at 1 MB.
    client_max_body_size 1m;

    # The request path holds the entrypoint UUID. Protect this
    # file like the entrypoint URL itself.
    access_log /var/log/nginx/webhooker.access.log combined;

    location / {
        # A literal address, not localhost: localhost may resolve
        # to ::1 first, where webhooker is not listening.
        proxy_pass http://127.0.0.1:8080;

        # $http_host, not $host: $host drops the port.
        proxy_set_header Host              $http_host;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Above webhooker's own 60 second request timeout, so its
        # 503 reaches the client instead of nginx's 504.
        proxy_read_timeout 70s;
    }
}

server {
    listen 80;
    listen [::]:80;
    server_name webhooker.example.com;
    return 308 https://$host$request_uri;
}
```

With that block, webhooker's environment is:

```sh
BIND_ADDRESS=127.0.0.1   # the default
TRUSTED_PROXIES=127.0.0.1
```

If nginx runs on another host, `BIND_ADDRESS` becomes the address nginx connects
to, and `TRUSTED_PROXIES` becomes nginx's address. Firewall the port to that
address: the traffic between them is plain text.

### HSTS

HSTS is a response header that tells browsers to use HTTPS only for this
hostname. Every response carries it unless `HSTS_MAX_AGE` is `0`. With the
default of ten years it reads:

```
Strict-Transport-Security: max-age=315360000; includeSubDomains; preload
```

A browser that receives this header over HTTPS refuses plain HTTP to the
hostname until `max-age` runs out. Browsers ignore the header over plain HTTP.
So serve webhooker on a hostname you will keep on HTTPS.

`includeSubDomains` extends the rule to every name under the hostname. `preload`
invites browsers to put the domain on their built-in HSTS lists. The header
alone lists nothing. A listing happens when someone submits a registered domain
at `hstspreload.org`, and removal takes months.

Setting `HSTS_MAX_AGE=0` stops the header. Browsers that already received it
keep the rule until their `max-age` runs out. To end the rule sooner, send a
short `max-age` such as `1s` until returning browsers have picked it up.

## Recovering a lost admin password

`webhooker resetpw` sets an existing account's password from the command line.
The deployment must be stopped first: the command takes the same `DATA_DIR` lock
the server does and refuses while a running instance holds it.

```bash
# Generate a new password and print it.
DATA_DIR=/var/lib/webhooker webhooker resetpw -generate admin

# Or supply one on standard input (at least 8 characters).
printf '%s' "$NEW_PASSWORD" | \
    DATA_DIR=/var/lib/webhooker webhooker resetpw admin
```

The password is never taken as a command-line argument, because on Linux every
account on the host can read a running process's arguments. Standard input is
echoed when it is a terminal, and the prompt says so.

The command creates nothing. A `DATA_DIR` that does not exist, or that holds no
`webhooker.db`, is an error. An unknown username is an error. The new password
takes effect at the next sign-in. Sessions already signed in are unaffected.

`webhooker help` lists the subcommands.

## Backup and restore

### What to back up

All state lives in `DATA_DIR`. Back up the whole directory. It holds:

- `webhooker.db`, one per install. Settings, the session key, users, webhooks,
  entrypoints and targets.
- `events-{webhook_uuid}.db`, one per webhook. Events, deliveries and delivery
  attempts.
- `archive-{webhook_name}-{target_name}-{target_uuid}.db`, one per archive
  target, or one per period for a target that rotates. The period goes before
  `.db`.
- `webhooker.lock`, which is always empty and holds no state.

Every database runs in WAL mode. WAL is SQLite's write-ahead log: committed
changes land in a `-wal` file beside the database before they reach the `.db`
itself. So a running service has `{name}.db-wal` and `{name}.db-shm` beside each
`{name}.db`. The `-wal` file is part of the database. A copy of the `.db`
without it is missing data and may have no readable tables at all.

A clean shutdown closes every database and removes its `-wal` and `-shm` files.
A crash leaves them, and they must be carried with the `.db`.

`DATA_DIR` must be on a local filesystem. SQLite maps the `-shm` file into
memory, which network filesystems do not support reliably. webhooker does not
check this. An error reading `disk I/O error (522)` points at the storage under
`DATA_DIR`.

Configuration is not in `DATA_DIR`. It comes from the environment and from a
`.env` file in the working directory. Back those up with your deployment.

### A running copy is not safe

Databases are held open for the life of the process. `cp`, `rsync`, `tar` or a
filesystem snapshot against a running service can catch a `.db` and its `-wal`
at two different instants. Use one of these two procedures.

Stop, copy, start. This is the only way to get one point in time across every
file:

```bash
docker stop webhooker
cp -a /path/to/data /path/to/backup-$(date -u +%Y%m%dT%H%M%SZ)
docker start webhooker
```

SQLite online backup, one file at a time, with no downtime:

```bash
for db in /path/to/data/*.db; do
    sqlite3 "$db" ".backup '/path/to/backup/$(basename "$db")'"
done
```

`.backup` reads through the WAL and writes one complete file. The runtime image
has no `sqlite3`, so run this on the host or from another container with the
volume mounted. Each file is captured at its own instant, so an event stored
between two copies lands in one file and not the other.

`sqlite3 <db> .dump` is an export, not a backup. It is safe to run but holds a
read transaction open for its whole duration.

Archive files are the exception. The archive writer closes and reopens its file
after a write, at most once per second. So you can move a finished archive file
away while the service runs. Move its `-wal` and `-shm` with it, or wait until
there are none. The next write recreates a missing archive file.

### Restore

1. Stop the service.
2. Restore `webhooker.db`, every `events-*.db` and every `archive-*.db`
   together. A missing `events-{uuid}.db` is created empty at the next start,
   and the webhook comes back with its settings but no history. That start logs
   `created a new, empty database` at `WARN` with the file's path. A missing
   archive file is recreated silently at the next write.
3. Carry any `-wal` and `-shm` files that are in the backup.
4. Start the service. The container fixes the owner of the restored files.

### Backups contain secrets

Treat a backup like the credentials inside it. Encrypt it at rest and restrict
who can read it.

- The event and archive databases hold the full headers and body of every event,
  including whatever the sender put in them.
- `webhooker.db` holds target settings unencrypted, next to the session key and
  the password hashes.

## Security

### The entrypoint URL is the credential

Each entrypoint is a random version 4 UUID served at `/h/{uuid}`. Holding that
URL is what authorises a sender. The receiver checks nothing else: no shared
secret, no HMAC signature (a hash of the body made with a shared key), no token
header. None will be added. Signature headers a sender adds anyway, such as
`X-Hub-Signature`, are stored and forwarded as ordinary headers.

What that means in practice:

- Keep the URL out of tickets, chat and screenshots. webhooker's own log never
  writes it, except in the line a `log` target writes. The reverse proxy's
  access log does hold it, in the request path.
- To rotate, create a new entrypoint and point the sender at it. Then deactivate
  or delete the old one.
- A sender that can only sign payloads to a fixed URL is a limit of that sender.
  Pick another integration path for it.

### The web interface

- Passwords are hashed with Argon2id, a memory-hard password hashing function,
  using 64 MB of memory, 1 pass, 4 threads, a 32-byte key and a 16-byte salt.
- Session cookies are signed, `HttpOnly` and `SameSite=Lax`. They carry `Secure`
  when the request arrived over TLS, judged per request from a direct TLS
  connection or `X-Forwarded-Proto: https`.
- The session is regenerated at sign-in, which prevents session fixation.
- The key that signs the cookies is 32 random bytes made on first start and
  stored in `webhooker.db`.
- Every form is protected against cross-site request forgery with a token tied
  to a cookie. Over TLS the form's `Origin` or `Referer` must also match the
  host.
- Form bodies are capped at 1 MB. A larger declared body is answered `413`
  before anything reads it.
- Every response carries these headers:
    - `Strict-Transport-Security`, unless `HSTS_MAX_AGE` is `0`;
    - `X-Content-Type-Options: nosniff`;
    - `X-Frame-Options: DENY`;
    - `Content-Security-Policy`, which tells the browser where scripts and
      styles may come from, set to
      `default-src 'self'; script-src 'self'; style-src 'self'`;
    - `Referrer-Policy: strict-origin-when-cross-origin`;
    - a `Permissions-Policy` that denies camera, microphone and location.
- Authenticated pages are sent with `Cache-Control: no-store`.
- `/metrics` sits behind HTTP basic authentication, a username and password sent
  with each request.

### Sign-in limits

The sign-in form has no rate limiter in front of it, on purpose. Behind a proxy
that `TRUSTED_PROXIES` does not cover, every client shares one bucket. A limit
that counted every request would let any stranger lock the operator out with a
few requests a minute.

Instead, the password is checked first, and only a failure is counted. A correct
password is never throttled. Failures are counted per client and submitted
username. From the fifth failure within a minute, each failure is answered `429`
with `Retry-After`. A successful sign-in clears the count. At most 1024 username
counters and 1024 client counters are kept.

Two password checks can run at once; each running check holds one of two slots.
Each check takes 64 MB of memory, so hashing is capped at 128 MB. A request that
gets no slot within 5 seconds is answered `503` without hashing anything. So is
one that arrives when 16 requests are already waiting for a slot. The password
change form shares these slots.

The check runs before the count, so the two slots, not the failure count, bound
how fast passwords can be guessed. A flood of wrong passwords is tried as fast
as two Argon2id checks allow. Choose a long random admin password. To put a
cheaper limit on top, rate-limit `POST /pages/login` at the reverse proxy, where
the real client address is visible.

An unknown username is checked against a placeholder hash, so it costs the same
time as a real one.

A flood can hold both slots and make sign-in answer `503` for as long as it
runs. Nothing accumulates: the operator's correct password works on the first
try once the flood stops. Restarting webhooker is no remedy. Block the source at
the proxy.

### Delivery destinations

A delivery target could be used to make webhooker send requests into the network
it sits in and read the answers back. This is called server-side request
forgery. webhooker refuses it by default.

Every target URL must use `http` or `https` and resolve to a public address.
These private and reserved ranges are refused:

- loopback, `127.0.0.0/8` and `::1`;
- the RFC 1918 private ranges, `10.0.0.0/8`, `172.16.0.0/12` and
  `192.168.0.0/16`;
- carrier-grade NAT space, `100.64.0.0/10`, which internet providers use inside
  their own networks;
- link-local space, `169.254.0.0/16` and `fe80::/10`, which is valid only on one
  network segment;
- the IPv6 unique local range, `fc00::/7`;
- multicast, documentation space and the other reserved ranges.

One public address is refused too: `168.63.129.16`, the Azure service that hands
a VM its credentials.

The check runs when a target is saved and again when each delivery connects.
Both use the same function. The address is resolved again at connect time, so a
hostname that changes its address after validation is still checked.

Redirects are followed up to ten hops. Each hop is checked the same way. A hop
that leaves the target's origin carries none of the target's configured headers
and none of the forwarded event headers. Leaving the origin means a different
host, a different port, or a step from `https` to `http`. This stops an open
redirect from handing a credential to another host.

`ALLOWED_EGRESS_CIDRS` opens named networks for the common case of forwarding to
your own network. Read
[Allowing egress to your own network](#allowing-egress-to-your-own-network)
before setting it.

### What the logs contain

webhooker writes structured logs to standard output: plain text on a terminal,
JSON otherwise. Each request writes one access log line, `http request`, at
`INFO`. A request to a known entrypoint writes a second line,
`webhook request received`. A stored event writes a third,
`webhook event created`.

Fields a client can influence are cut to a fixed size, so one access log line is
at most 2,560 bytes. The query string is never logged. On the receiver route the
URL is logged as `/h/{uuid}`, never the real path, so entrypoint UUIDs stay out
of the log. On other routes a `3xx` or `4xx` response logs the route pattern,
which is the route as registered, such as `/user/{username}`. A `2xx` or `5xx`
logs the real path. The `Referer` is logged as its scheme and host only.

Every line that names a client carries two fields. `remoteIP` is the connecting
peer, which behind a proxy is the proxy. `clientIP` is the client as decided by
`TRUSTED_PROXIES`.

SQL statements are logged with `?` in place of each value, never with the values
themselves. Session cookies, API keys and target credentials are never logged.

Two things are logged in the clear. The first is the `admin` password, once, on
the first boot, and again when `webhooker resetpw -generate` is run. The second
is the operator's own configuration echoed back: webhook names and target
hostnames.

A `log` target writes the whole event, headers and body, to the log. That is its
purpose.

A recovered panic writes one `ERROR` record of at most 10,240 bytes, with the
request id of the request that caused it.

When `SENTRY_DSN` is set, error reports go to Sentry, an error reporting
service, with the request reduced to this:

- the URL with its path replaced by the route pattern;
- the query string and the body replaced by `(redacted)`;
- no cookies;
- only these headers: `Accept`, `Content-Length`, `Content-Type`, `Host`,
  `Origin`, `Referer` (scheme and host only), `User-Agent` and `X-Request-Id`.

## Configuration reference

All configuration comes from environment variables. For development, variables
can also go in a `.env` file in the working directory. It is read once at
startup. A variable set in the real environment wins over the file. A `.env`
file that cannot be parsed aborts startup, because a single bad line would
otherwise silently drop every variable in the file.

A variable that is unset or empty takes its default. A variable that is set but
cannot be parsed aborts startup with a message naming it. `PORT=eighty`,
`DEBUG=ture` and `RETENTION_SWEEP_INTERVAL=1 hour` all abort. Each section below
says what range its variable accepts.

A Go duration is a number with a unit, such as `30s`, `15m`, `24h` or `1h30m`.

The startup log line `Configuration loaded` shows the loaded values, with three
differences. `TRUSTED_PROXIES` and `ALLOWED_EGRESS_CIDRS` appear as counts.
`SENTRY_DSN` and the metrics credentials appear as on or off. `METRICS_USERNAME`
does not appear. The `/settings` page shows every value; there,
`METRICS_PASSWORD` and `SENTRY_DSN` show only as set or not set.

### `WEBHOOKER_ENVIRONMENT`

`dev` or `prod`. Default `prod`.

This controls one thing: CORS, the headers that let a page on another site call
this one from a browser. In `dev`, every origin is allowed with
`Access-Control-Allow-Origin: *`, for local testing. In `prod`, no CORS headers
are sent. Cookie security and log format do not depend on it.

### `PORT`

The TCP port the listener binds. Default `8080`. It must be a number from 1
to 65535.

### `BIND_ADDRESS`

The IP address the listener binds. Default `127.0.0.1`. The Docker image sets
`0.0.0.0`.

The value must be an IP address literal. `localhost` is refused, because it may
mean either `127.0.0.1` or `::1`. A value with a port, such as `127.0.0.1:8080`,
is refused; the port belongs to `PORT`.

- `127.0.0.1` answers only on this host. Use it with a reverse proxy on the same
  host.
- `0.0.0.0` answers on every IPv4 address and no IPv6 address.
- `::` answers on every IPv6 and IPv4 address.
- A specific address such as `10.0.0.5` answers on that interface only.

An address that parses but is not on this host passes configuration and fails at
bind time. The process logs the error and exits non-zero. The `bindAddress`
field of the startup log shows what was bound.

### `DATA_DIR`

The directory for every SQLite database and the lock file. Default
`/var/lib/webhooker`. It is created if it is missing.

Exactly one webhooker process may use a `DATA_DIR` at a time. Two processes on
one directory would both run delivery recovery and deliver every pending event
twice. At startup the process locks `{DATA_DIR}/webhooker.lock` with `flock`,
the kernel's file lock, and holds the lock until it exits. A second process
pointed at the same directory prints a message naming it and exits non-zero.

The kernel releases the lock when the process exits, even on `kill -9`. A
leftover `webhooker.lock` blocks nothing and must not be deleted by hand. To run
two webhookers on one host, give each its own `DATA_DIR`.

`flock` works between processes and containers that share a directory on one
host. It does not work across hosts on a network filesystem.

### `DEBUG`

`true` or `false`. Default `false`. It accepts the spellings Go's
`strconv.ParseBool` accepts: `1`, `t`, `T`, `TRUE`, `true`, `True`, `0`, `f`,
`F`, `FALSE`, `false`, `False`. `yes` and `on` are refused.

`DEBUG=true` lowers the log level to `DEBUG`. Lines at that level include:

- every SQL statement, with `?` in place of each value;
- the lookup misses on the receiver and sign-in routes;
- signed-out requests sent to the sign-in page;
- retries being scheduled, and retries skipped because the delivery had already
  been handled;
- the receiver's whole-route rate limit refusing a request;
- a target deleted while the archive sweep was running;
- the records of how the components were wired at startup.

Debug lines never carry SQL values, session cookies or target credentials. The
first-boot password is printed whatever the level, and so is every event a `log`
target writes.

### `METRICS_USERNAME` and `METRICS_PASSWORD`

The basic authentication credentials for `/metrics`. Default empty. Set both or
neither. With both set, `/metrics` is served behind basic authentication. With
neither set, the route does not exist and answers `404`. With only one set,
startup fails with an error naming both variables.

### `SENTRY_DSN`

A Sentry DSN for error reporting. Default empty, which leaves reporting off. A
set value is parsed with the Sentry SDK's own parser, and one it cannot parse
aborts startup. The release name sent to Sentry is `webhooker-{version}`.

### `RETENTION_SWEEP_INTERVAL`

How often the retention sweep and the archive sweep run. A Go duration greater
than zero. Default `1h`.

The retention sweep deletes events older than each webhook's retention period,
with their deliveries and attempts, in batches of 1000. The archive sweep prunes
rows past each archive target's expiry.

### `SESSION_IDLE_TIMEOUT`

How long a session may go unused before it ends. A Go duration. Default `24h`. A
zero or negative value disables idle expiry. The 7 day absolute limit always
applies.

Only authenticated requests count as use. The idle timestamp in the cookie is
rewritten at most once per tenth of the window, so a session can end up to 10%
early, never late.

### `HSTS_MAX_AGE`

The `max-age` of the `Strict-Transport-Security` header, as a Go duration in
whole seconds. Default `87600h` (ten years). `0` sends no header. `1.5s` and
`-1h` are refused. Read [HSTS](#hsts) before changing it.

### `RECEIVER_RATE_LIMIT`

How many requests per minute one client may send to one entrypoint. Default
`120`. It must be at least 1. A client over the limit is answered `429` with
`Retry-After`.

A second limit covers the whole `/h/` route per client at ten times this value,
1200 by default. It exists because the route matches any single path segment. A
client inventing a new path per request would otherwise get a fresh bucket each
time. Each such request still costs a database lookup. The second limit is not
configurable on its own.

### `TRUSTED_PROXIES`

A comma-separated list of networks in CIDR notation, which names a network as an
address and a prefix length, such as `10.0.0.0/8`. A bare address such as
`192.168.1.7` counts as a single host. Default
`10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`, the RFC 1918 private ranges. A set
value replaces the default.

The list decides whose `X-Forwarded-For` header is believed. When the connecting
peer is inside the list, the client address is read from that header. For any
other peer, the client is the peer itself and the header is ignored. `X-Real-IP`
and `True-Client-IP` are never read.

The header is read from right to left. The first hop that is not itself a
trusted proxy is the client. A hop that is not a bare IP address ends the walk,
and the peer address is used instead. At most 64 hops are examined. Your proxy
must therefore append the peer address to `X-Forwarded-For`, as nginx's
`$proxy_add_x_forwarded_for` does.

If any client can reach webhooker, or the proxy in front of it, from a private
address, set the list to the proxy's address alone. Otherwise those clients can
choose their own `X-Forwarded-For` and bypass every rate limit. The address to
set is the `remoteIP` field of an `http request` log line for a request that
came through the proxy.

Behind a proxy the list does not cover, every client shares one bucket per
limit. The receiver limits become service-wide ceilings. Sign-in failures from
every client land in one count, though a correct password is still never
throttled.

Rate-limit buckets are per IPv4 address and per IPv6 `/64` prefix.

### Allowing egress to your own network

`ALLOWED_EGRESS_CIDRS` is a comma-separated list of networks in CIDR notation
that delivery targets may reach although the default rules refuse them. Default
empty. A bare address counts as a single host. Example:
`10.0.0.0/8, 172.17.0.0/16`.

This only ever adds destinations. The default refusals stay in force for every
address not listed. When the list is set, startup logs the whole list at `WARN`.

Anyone with the admin account can make webhooker send requests to every network
you list. They can read the responses in the delivery log. List the smallest
blocks that cover your destinations. Prefer `10.1.2.3/32` over `10.0.0.0/8`.
`0.0.0.0/0` or `::/0` opens every private and reserved range at once, including
this host's own loopback. Do not list them.

Some addresses stay blocked whatever you list. They are link-local space, the
unspecified addresses, and cloud metadata endpoints at fixed non-public
addresses that hand out credentials:

- `169.254.0.0/16` and `fe80::/10`, the link-local ranges. `169.254.169.254` is
  the metadata address on AWS, Azure, DigitalOcean, Hetzner, OpenStack and
  others.
- `fd00:ec2::254/128` and `fd00:ec2::23/128`, AWS metadata and EKS Pod Identity
  over IPv6.
- `fd20:ce::254/128`, Google Cloud metadata over IPv6.
- `fd00:c1::a9fe:a9fe/128`, Oracle Cloud metadata over IPv6.
- `fd00:42::42/128`, Scaleway metadata over IPv6.
- `fd00:a9fe:a9fe::1/128`, Linode and Akamai metadata over IPv6.
- `100.100.100.200/32`, Alibaba Cloud metadata, inside the carrier-grade NAT
  range.
- `192.0.0.192/32`, Oracle Cloud Classic metadata.
- `0.0.0.0/32` and `::/128`, the unspecified addresses, which reach this host's
  loopback on Linux.
- `::a9fe:a9fe/128` and `64:ff9b::a9fe:a9fe/128`, two other spellings of
  `169.254.169.254`.

The six `fd00::/8` entries matter because `fd00::/8` is an ordinary block to
list for your own IPv6 network. Without them, that one line would open cloud
credentials on five providers.

This list is not every cloud's metadata address. If yours is not here, do not
list the block that contains it. Azure's public `168.63.129.16` is refused by
default but is reopened by a list that covers it.

## Delivery reference

### Target types

An `http` target posts the event body to its URL. The request carries the
event's `Content-Type`, the event's headers, then the target's configured
headers, then `User-Agent: webhooker/{version}`. A configured header replaces an
event header of the same name. A `2xx` response is a success. The first 4 KiB of
the response body is stored with the attempt.

These event headers are never forwarded:

- `Host`
- `Connection` and `Keep-Alive`
- `Transfer-Encoding` and `Content-Length`
- `Te` and `Trailer`
- `Upgrade`
- `Proxy-Authorization` and `Proxy-Connection`

An `http` target's settings:

- `url`, the destination. Only `http` and `https` are accepted.
- `headers`, one `Name: value` per line. `Host`, `Content-Length`,
  `Transfer-Encoding`, `Connection`, `Trailer` and `User-Agent` are refused,
  because the delivery sets them itself. `Content-Type` is allowed and replaces
  the event's.
- `timeout_seconds`, the per-request timeout, at most 300. Blank means the
  shared 30 second default. A long timeout holds one of the ten delivery workers
  for its whole duration.
- `forward_query`, off by default. On, the event's query string is appended to
  the URL, joined with `&` if the URL already has one.

A `slack` target posts a JSON message with one `text` field to its URL. The
message names the method, content type, time received and body size, then shows
the body in a code block. A JSON body is indented. A body over 3500 bytes is
cut. The URL is any Slack-compatible incoming webhook URL, which includes
Mattermost.

A `database` target, called an archive in the web interface, appends the event
as a row to its own SQLite file. See [Archives](#archives). It makes no network
request and never retries. A write failure fails the delivery.

A `log` target writes the event, headers and body included, as one `INFO` line.
It never retries.

### Attempts and retries

`http` and `slack` targets take `max_attempts`, from 1 to 20. With 1, a failed
delivery is marked `failed` after its one attempt. With more, a failed attempt
marks the delivery `retrying` and schedules the next attempt. The wait doubles
each time, starting at one second. After the last allowed attempt fails, the
delivery is marked `failed`.

A delivery has one of four statuses. `pending` means created but not yet
attempted. `retrying` means at least one attempt failed and more remain.
`delivered` means an attempt succeeded. `failed` means every allowed attempt
failed.

Every attempt is recorded with its number, success, status code, response body,
error and duration. One case is not recorded: an attempt cut off by shutdown.

### Circuit breaker

An `http` or `slack` target with `max_attempts` above 1 has a circuit breaker. A
circuit breaker stops hammering a destination that is down. It is in memory only
and resets when the service restarts.

The breaker is closed in normal operation. After 5 failures in a row it opens
for 30 seconds. While it is open, each delivery to the target is marked
`retrying` and scheduled for when the cooldown ends. When the cooldown ends, one
delivery is sent as a probe. The others wait one more cooldown. If the probe
succeeds, the breaker closes. If it fails, the breaker opens again.

The webhook page says when a target's deliveries are paused and until when. Each
waiting delivery in the event log shows the earliest time it can be tried.

### Workers and queues

Ten workers carry out deliveries. New deliveries wait in one queue and retries
in another. Each queue holds 10,000 tasks. At most ten deliveries are in flight
at once, however deep the queues get.

When a retry falls due and its queue is full, the retry is dropped and the
delivery stays `retrying` in the database. The recovery sweep picks it up.

Each task carries the target's settings and the event. A body under 16 KiB
travels with the task. A larger one is read from the database when the attempt
runs.

### Recovery

A recovery routine runs once at start and then every 60 seconds. For every
webhook it reads the deliveries still `pending` or `retrying`. One that already
has a successful attempt recorded is marked `delivered` and not sent again. A
`retrying` delivery is scheduled for whatever remains of its wait. A `pending`
one is queued. A delivery the engine is already working on is left alone.

At start, every `pending` delivery is taken, however recent. So a delivery cut
off by a stop moments earlier is sent again at once. The periodic sweep takes
only `pending` deliveries untouched for 15 minutes, at most 500 per webhook each
time.

If a target was deleted while a delivery to it was still open, recovery marks
the delivery `failed` and records the reason.

### Shutdown

On `SIGINT` or `SIGTERM`, components stop in reverse start order under a 5
second budget for the whole sequence. The HTTP server drains open requests for
at most 3 seconds, leaving 2 seconds for the components after it. A Sentry
flush, if enabled, fits inside the same budget and is skipped when little is
left.

The delivery engine cancels any `http` or `slack` request in flight. That
attempt is not recorded, and the delivery keeps its status, so it is sent again
at the next start. A `database` or `log` task finishes as normal.

A component that does not stop in time is logged at `ERROR`, and the components
after it are skipped. SQLite is crash-safe, so this loses no data, but the
database is not closed cleanly.

Keep the container's stop grace period above 5 seconds. Docker's default is 10.
A shorter grace kills the process before the shutdown can run.

### Archives

An archive target writes to
`archive-{webhook_name}-{target_name}-{target_uuid}.db` in `DATA_DIR`. The two
names are made safe for a file name:

- lowercased;
- only the ASCII letters `a` to `z` and the digits `0` to `9` kept, and every
  other run of characters turned into one `-`;
- cut to 40 characters;
- `unnamed` when nothing is left.

A webhook named `Orders (EU)` with a target named `Long-term archive` writes
`archive-orders-eu-long-term-archive-{target_uuid}.db`.

Each row holds:

- the event's id, and the ids of its webhook and entrypoint;
- the method, query string, headers, body and content type;
- the time the row was archived.

`archive_rotation` is `none` by default, which keeps one file. `monthly`,
`daily` and `hourly` start a new file for each period of the event's receive
time, in UTC, with the period before `.db`: `-2026-10.db`, `-2026-10-01.db` or
`-2026-10-01-19.db`. A changed rotation applies from the next event. Files
already written keep their names and are still shown, pruned and downloaded.

`archive_expiry` is `never` by default. The forms offer `1h`, `12h`, `24h`,
`30d`, `90d` and `365d`. Rows older than the expiry are deleted each time the
file is opened for a write, and by the archive sweep every
`RETENTION_SWEEP_INTERVAL`. The sweep deletes a period file it has emptied. It
never creates a file.

Renaming the webhook or the target renames every file of the target. If a file
with the new name already exists, the edit is refused and nothing moves. A file
the operator has moved away is skipped.

Deleting a webhook or a target leaves its archive files on disk. Removing them
is the operator's decision.

**Download** on the webhook page returns the target's whole archive as one
gzipped JSON file named
`archive-{webhook_name}-{target_name}-{YYYYMMDDTHHMMSSZ}.json.gz`. It holds
`webhook` and `target` (each an `id` and a `name`), `exported_at`, and
`archived_events`, one object per row with every column. Rows from a period file
carry a `period` field. A body that is not valid UTF-8 is written in base64 with
`"body_encoding": "base64"` beside it. The download streams one file at a time
and creates nothing.

## Data reference

### Files

`webhooker.db` holds the application tier: settings, users, API keys, webhooks,
entrypoints and targets. Each `events-{webhook_uuid}.db` holds that webhook's
event tier: events, deliveries, delivery attempts and running totals. Each
`archive-*.db` holds one archive target's rows.

Every database is opened through one code path, in WAL mode, with a 10 second
busy timeout and at most 4 connections. Under WAL a reader never blocks a
writer.

An event database is created with its webhook. Every webhook's database is
opened at start. One that is missing or empty is created again and logged with
`created a new, empty database` at `WARN`. On a deployment that has run before,
that line means the file was lost, most often because the volume holding
`DATA_DIR` is not mounted. A damaged file fails that webhook alone, with an
`ERROR` naming it and a `500` to its senders.

Deleting a webhook soft-deletes its row, entrypoints and targets in
`webhooker.db` and deletes its event database file. Deleting an entrypoint or a
target soft-deletes it.

### Schema

Each kind of database file has its schema in SQL files built into the binary,
under `internal/db/migrations/`: `main/` for `webhooker.db`, `events/` for event
databases and `archive/` for archive files. In each, `000_migration.sql` creates
the `schema_migrations` table and `001_schema.sql` holds the schema. Opening a
database applies every file its `schema_migrations` does not list, in one
transaction.

### Application tier

Every entity below except `Setting` has `id` (a version 4 UUID), `created_at`,
`updated_at` and `deleted_at`. A row with `deleted_at` set is soft-deleted and
hidden from the application.

`Setting` is a key-value row with no other fields. The one key stored is
`session_key`, the base64 session key made on first start.

`User` has `username` (unique, at most 1024 bytes) and `password` (the Argon2id
hash). The first start creates `admin` with a random 16-character password.

`Webhook` has `user_id`, `name`, `description` and `retention_days`.
`retention_days` is `0` for forever, or a whole number from 1 to 106751. The cap
keeps the retention cutoff inside the range of a Go duration.

`Entrypoint` has `webhook_id`, `path`, `description` and `active`. `path` is the
bare UUID. The `/h/` prefix is route only and is not stored.

`Target` has these fields. Each type fills only its own settings.

- `webhook_id`, `name`, `type` and `active`. `type` is one of `http`, `slack`,
  `database` or `log`.
- `url`, `headers`, `timeout_seconds` and `forward_query`, the `http` settings.
  A `slack` target uses `url` alone.
- `archive_expiry` and `archive_rotation`, the `database` settings.
  `archive_expiry` is stored in nanoseconds, `0` for never.
- `max_attempts`, for `http` and `slack`.

`APIKey` has `user_id`, `key`, `description` and `last_used_at`. Its table,
`api_keys`, exists, but nothing creates a key and nothing checks one.

### Event tier

`Event` has these fields:

- `webhook_id` and `entrypoint_id`;
- `method`, which is always `POST`;
- `raw_query`, `headers` (as JSON), `body`, `body_bytes` and `content_type`;
- `resubmitted_from_id`, the event a resubmit copied. It is empty for an event
  that arrived on the receiver.

`Delivery` has `event_id`, `target_id`, `status`, `finished_at` and `replay`.
`finished_at` is set when the status becomes `delivered` or `failed`.

`DeliveryResult` is one attempt. It has `delivery_id`, `attempt_num`, `success`,
`status_code`, `response_body` (the first 4 KiB), `error` and `duration` in
milliseconds.

Three tables hold running totals so the pages need not count rows. `EventTotals`
is one row: `events`, `events_removed` and `last_event_at`. `TargetTotals` is
one row per target: `deliveries`, `delivered`, `failed`, `deliveries_removed`
and `failed_removed`. `EntrypointTotals` is one row per entrypoint:
`last_event_at`. Each count changes in the transaction that writes or deletes
the rows it counts. Retention raises the `_removed` counts and leaves
`last_event_at` alone, so a webhook's last event is still shown after its events
have expired.

## HTTP routes reference

Public routes:

- `GET /` redirects to `/hooks` when signed in and to `/pages/login` otherwise.
- `GET /.well-known/healthcheck` returns JSON with `status`, `now`,
  `uptimeSeconds`, `uptimeHuman`, `version` and `appname`.
- `GET /s/*` and `HEAD /s/*` serve the embedded CSS and JavaScript. Other
  methods get `405` with `Allow: GET, HEAD`.
- `POST /h/{uuid}` is the receiver. Any other method gets `405` with
  `Allow: POST`. An unknown UUID gets `404`, a deactivated entrypoint `410`, a
  body over 1 MB `413`, and a stored event `200` with `{"status":"ok"}`.

Sign-in routes:

- `GET /pages/login` shows the sign-in form. Its `next` parameter names the page
  to return to. Anything but a path on this site is replaced with `/`.
- `POST /pages/login` signs in. See [Sign-in limits](#sign-in-limits).
- `POST /pages/logout` signs out.

Routes behind the sign-in. A `GET` without a session is redirected to
`/pages/login` with the path as `next`:

- `GET /user/{username}` is the profile page. Another user's name gets `403`.
- `POST /user/{username}/password` changes the password, 5 per minute per
  client.
- `GET /settings` lists the loaded configuration.
- `GET /hooks` lists webhooks. `GET` and `POST /hooks/new` create one.
- `GET /hook/{id}` is the webhook page. `GET` and `POST /hook/{id}/edit` edit
  it. `POST /hook/{id}/delete` deletes it.
- `GET /hook/{id}/events` is the event log. `?show=failed` and `?show=pending`
  narrow it.
- `GET /hook/{id}/events/{eventID}` is one event's page, and
  `GET /hook/{id}/events/{eventID}/body` downloads its stored body.
- `POST /hook/{id}/deliveries/{deliveryID}/replay` replays a delivery, and
  `POST /hook/{id}/events/{eventID}/resubmit` resubmits an event, each 30 per
  minute per client.
- `POST /hook/{id}/entrypoints` adds an entrypoint.
- `POST /hook/{id}/entrypoints/{entrypointID}/edit` changes its description.
- `POST /hook/{id}/entrypoints/{entrypointID}/delete` deletes it.
- `POST /hook/{id}/entrypoints/{entrypointID}/toggle` activates or deactivates
  it.
- `POST /hook/{id}/targets` adds a target.
- `GET /hook/{id}/targets/{targetID}/edit` shows the target edit form, and
  `POST` to the same path saves it.
- `POST /hook/{id}/targets/{targetID}/delete` deletes the target.
- `POST /hook/{id}/targets/{targetID}/toggle` activates or deactivates it.
- `GET /hook/{id}/targets/{targetID}/download` downloads an archive.

`GET /metrics` exists only when `METRICS_USERNAME` and `METRICS_PASSWORD` are
set, behind basic authentication.

`/api/v1` is mounted with no routes. Every path under it answers `404`.

## Metrics reference

`/metrics` serves one Prometheus registry. It carries the Go runtime and process
collectors, the delivery metrics below, and three inbound HTTP metrics.

Nine of the eleven metrics below carry one label, `target_type`, with one of
`http`, `slack`, `database`, `log` or `unknown`. The two event counters carry
none. Target ids and event ids are never labels, because a label value that
grows with traffic would grow the registry without bound.

- `webhooker_events_received_total` (counter, no label): events stored,
  resubmitted copies included.
- `webhooker_delivery_attempts_total` (counter): attempts sent to a target. A
  delivery the circuit breaker turned away is not an attempt.
- `webhooker_deliveries_succeeded_total` (counter): deliveries that became
  `delivered`.
- `webhooker_deliveries_failed_total` (counter): deliveries that became
  `failed`.
- `webhooker_delivery_retries_total` (counter): deliveries put into `retrying`.
- `webhooker_delivery_replays_total` (counter): deliveries created by
  **Replay**.
- `webhooker_events_resubmitted_total` (counter, no label): events created by
  **Resubmit**.
- `webhooker_delivery_duration_seconds` (histogram): wall time of one attempt.
- `webhooker_deliveries_pending` and `webhooker_deliveries_retrying` (gauges):
  deliveries in those statuses, counted from the databases every 30 seconds.
  Deliveries to a deleted target count under `unknown`.
- `webhooker_circuit_breakers_open` (gauge): breakers currently open.

The success and failure counters move only after the new status has been
written.

The inbound HTTP metrics are `http_request_duration_seconds`,
`http_response_size_bytes` and `http_requests_inflight`. Their `handler` label
is the route pattern, such as `/h/{uuid}`, never the real path. A request that
matched no route carries `(unmatched)`. `http_requests_inflight` is one series
for the whole service, with `handler` set to `(all)`.

## Development

### Prerequisites

- The Go version named in `go.mod`.
- Docker. The tests, the linters, prettier and the stylesheet build run in
  Docker.

`gofmt` and `goimports` run on the host. These tools run only inside Docker, at
the versions pinned in the `Dockerfile`, `package.json` and `yarn.lock`:

- `golangci-lint`;
- ESLint;
- prettier;
- node and yarn;
- the Tailwind CLI.

### Commands

```bash
make bootstrap    # Install git, make, Go and the Go modules (safe to rerun)
make setup        # bootstrap, then install the git pre-commit hook
make assets       # Extract Alpine.js from 3p/ to static/js/alpine.min.js
make fmt          # Format Go (gofmt, goimports) and Markdown (prettier, in Docker)
make fmt-check    # Fail if gofmt or prettier would change anything
make lint         # Run gofmt, golangci-lint and ESLint in Docker
make test         # Run the tests in Docker, with cgo off
make test-browser # Run the browser test in Docker (Dockerfile.browser)
make check        # test, lint, fmt-check and css-check: what CI runs
make build        # Build bin/webhooker, version-stamped
make version      # Print the version this checkout would stamp
make run          # build, then run ./bin/webhooker
make dev          # go run ./cmd/webhooker
make deps         # go mod download, then go mod tidy
make docker       # Build the Docker image
make hooks        # Install the git pre-commit hook
make css          # Regenerate static/css/tailwind.css in Docker
make css-check    # Fail if static/css/tailwind.css is out of date
make clean        # Remove bin/
```

The repository follows
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all).
Each `make` target above, except `build`, `run`, `dev`, `deps`, `version` and
`clean`, is a thin shim over a script in `script/`. `make hooks` calls
`script/install-precommit`; every other shim calls the script of the same name.
`script/cibuild` is what CI runs: `script/bootstrap`, then `script/check`, then
the image build. `script/precommit` checks that `go mod tidy` changes nothing,
then runs `script/check`.

The tests run with cgo, Go's bridge to C code, turned off. Go's race detector
needs cgo, so the tests run without it.

### How the checks run

Each check is a stage of the `Dockerfile`. `script/test` builds the `test`
stage, `script/lint` the `lint` stage and `script/css-check` the `css-check`
stage. `script/fmt-check` runs `gofmt` on the host and builds the
`markdown-check` stage. Every `docker build` in `script/` passes `--no-cache`,
so a check that reports success ran. A bare `docker build .` carries no such
guarantee.

The `lint` stage runs the `gofmt` check, `golangci-lint config verify` and
`golangci-lint run` with `--network=none`, and ESLint over `static/js/`. The
`test` stage runs `go test -cover` with at most four packages and eight tests at
a time and a 90 second timeout per package. On a failure it runs only the failed
tests again with `-v`.

The image build depends on the `lint`, `test`, `css-check` and `markdown-check`
stages, so an image that built is a tree that passed every check. The runtime
stage is Alpine with `ca-certificates` and `su-exec` added. It has no `sqlite3`.

The `check` workflow in `.gitea/workflows/check.yml` checks out the repository
and runs `script/cibuild`, nothing else.

### Version stamping

The binary reports its version at `/.well-known/healthcheck`, in the page
footer, in the `starting` log line and as the Sentry release. The value is
stamped by the linker at build time.

`script/version` produces it. With `VERSION` set in the environment, that value
is used. Otherwise it is `git describe --tags --always --dirty` on this
checkout, which gives:

- the tag at a clean tagged commit, such as `v1.0.0`;
- the tag, the commit count and the short hash past a tag, such as
  `v1.0.0-3-g1a2b3c4`;
- the short hash alone when no tag is reachable;
- a `-dirty` suffix on any of these for uncommitted changes.

With no git metadata it is `unknown`.

`make build VERSION=v1.2.3` overrides it. `script/docker` and `script/cibuild`
run the same `git describe` on the host and pass the result in as the `VERSION`
build argument. Without that argument the image build reads the `.git` in the
build context, and fails if `.git` is present but the version comes out empty,
`dev` or `unknown`. `.dockerignore` leaves out every git `config` file, which
can carry a credential, and keeps the rest of `.git`.

`make build` passes `-trimpath` and `-s -w`, so the binary carries no build
paths, symbol table or debug information. Nothing that differs between two
builds of one commit is stamped in.

### Third-party browser assets

The pages use one third-party script, Alpine.js, a small JavaScript library, in
its CSP build (`@alpinejs/csp`). The Content-Security-Policy forbids `eval`,
which the standard build needs. With the CSP build, every Alpine directive in
`templates/` names a component registered in `static/js/app.js`, such as
`x-data="collapsible"`, and never an inline expression.

The npm tarball is committed as `3p/alpinejs-csp-3.14.9.tgz`, byte for byte as
the registry publishes it. `script/assets` extracts `package/dist/cdn.min.js`
from it to `static/js/alpine.min.js`, which the binary embeds. `make build`,
`make dev` and the Docker stages run it first. The extracted file is not
committed, and a build that skips the extraction fails with an error naming
`js/alpine.min.js`.

To move to a new version:

1. Download `https://registry.npmjs.org/@alpinejs/csp/-/csp-<version>.tgz`.
2. Check it against the `dist.integrity` hash at
   `https://registry.npmjs.org/@alpinejs/csp/<version>`.
3. Replace the tarball in `3p/` and update the file name in `script/assets`.
4. Run `make check` and `make test-browser`.

### Browser test

`internal/server` has a test that loads the pages in a headless browser, one
without a window, under the real Content-Security-Policy and drives the
interface. It is built only with the `browser` build tag, so `make test` leaves
it out. `make test-browser` runs it in Docker with a pinned headless browser
image. Run it after changing `templates/` or `static/js/`.

### Stylesheet

`static/css/tailwind.css` is generated by Tailwind and committed. To change the
styles, edit the templates, `static/js/app.js`,
`internal/handlers/recent_events.go` or `static/css/input.css`, then run
`make css` and commit the result. Tailwind reads classes only from the files
that `input.css` names in its `@source` lines. `make check` fails when the
committed file differs from what `make css` generates. `static/css/style.css` is
hand-written. The Content-Security-Policy refuses inline styles, so templates
style elements with classes only.

### Libraries

- [Uber fx](https://go.uber.org/fx) wires the components and runs their start
  and stop hooks.
- [chi](https://github.com/go-chi/chi) routes HTTP.
- [GORM](https://gorm.io) with
  [modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite), a pure Go SQLite
  driver, handles the databases.
- `log/slog` from the standard library writes the logs.
- [gorilla/sessions](https://github.com/gorilla/sessions) and
  [gorilla/csrf](https://github.com/gorilla/csrf) handle sessions and form
  protection.
- [go-chi/httprate](https://github.com/go-chi/httprate) rate-limits the
  receiver, the password change, replay and resubmit.
- [Prometheus](https://prometheus.io) client and
  [go-http-metrics](https://github.com/slok/go-http-metrics) serve the metrics.
- [Sentry](https://sentry.io) reports errors when configured.

### Package layout

All application code is under `internal/`. The entry point is
`cmd/webhooker/main.go`, which dispatches subcommands, takes the `DATA_DIR` lock
and builds the fx graph.

- `internal/archive`: archive files, rotation, pruning, renaming and download.
- `internal/banner`: the ruled box that prints a credential.
- `internal/config`: configuration from the environment and `.env`.
- `internal/database`: the main database, the models, Argon2id, the retention
  sweep and the per-webhook database manager.
- `internal/datadir`: the exclusive lock on `DATA_DIR`.
- `internal/db/migrations`: the SQL schema files and their application.
- `internal/delivery`: the engine, the four target types, retries, the circuit
  breaker and the destination guard.
- `internal/globals`: build-time variables.
- `internal/gormlog`: GORM's logger on top of `slog`, with values withheld.
- `internal/handlers`: every HTTP handler and page.
- `internal/healthcheck`: the health check response.
- `internal/lifecycle`: the bounded wait used by stop hooks.
- `internal/logfield`: the size budget for client-supplied log values.
- `internal/logger`: `slog` setup and the fx event logger.
- `internal/metrics`: the delivery metrics.
- `internal/middleware`: logging, security headers, CORS, sessions, CSRF, rate
  limits, body caps, metrics and panic recovery.
- `internal/referer`: cuts a `Referer` to its scheme and host.
- `internal/reqtls`: the one predicate that decides whether a request is TLS.
- `internal/resetpw`: the `webhooker resetpw` subcommand.
- `internal/server`: the HTTP server, routes, middleware order and Sentry.
- `internal/session`: the cookie session.
- `internal/versionscript`: tests for `script/version` and the build files.
- `static/`: embedded CSS and JavaScript. `templates/`: the HTML templates.

### Components and middleware

fx builds each component when another needs it. It runs their start hooks in
that order and their stop hooks in reverse. The components are:

- the globals, the logger and the config;
- the main database and the per-webhook database manager;
- the retention sweep and the health check;
- the session and the handlers;
- the metrics registry and collectors;
- the middleware and the destination guard;
- the delivery engine, the archive writers and the archive sweep;
- the server.

The global middleware runs in this order:

1. request id;
2. security headers;
3. access log;
4. metrics, when `/metrics` is served;
5. CORS;
6. a 60 second request timeout;
7. panic recovery;
8. Sentry, when configured.

Each page route group then adds its own, in this order:

1. panic recovery;
2. the 1 MB body cap;
3. CSRF;
4. `Cache-Control: no-store`;
5. the sign-in check, in every group except `/pages`, which serves the sign-in
   form.

The body cap comes before CSRF because CSRF parses the form.

## License

MIT.

## Author

[@sneak](https://sneak.berlin)