1 Commits
Author SHA1 Message Date
sneak 98af5afb9d Rewrite README.md sentence by sentence
check / check (push) Canceled after 0s
Rewritten from scratch against the code on next. The old text was
the list of what a reader needs covered.

Model: fable-5-1
2026-10-25 03:13:30 +00:00
+243 -160
View File
@@ -8,9 +8,10 @@ receiver is down at that moment, the event is lost.
webhooker sits between the sender and your receiver. It is a single Go binary webhooker sits between the sender and your receiver. It is a single Go binary
with a web interface and SQLite storage. It accepts every event the sender posts with a web interface and SQLite storage. It accepts every event the sender posts
and writes it to disk before answering. It then delivers the event to each and writes it to disk before answering. It then delivers the event to each
target you configured, with retries. You can read every event and every delivery target you configured, with retries. The web interface shows each webhook's
attempt in the web interface, send a stored event again, and watch the whole newest events, with every delivery and attempt they had. From there you can send
thing through Prometheus metrics. a stored event again. It also exports metrics for Prometheus, the monitoring
system, so you can watch the whole flow.
webhooker is written in [Go](https://golang.org) by webhooker is written in [Go](https://golang.org) by
[@sneak](https://sneak.berlin). The license is MIT. [@sneak](https://sneak.berlin). The license is MIT.
@@ -22,7 +23,8 @@ Five words name the moving parts.
- A **webhook** is a named configuration. It groups entrypoints and targets, and - A **webhook** is a named configuration. It groups entrypoints and targets, and
it owns the events it receives. it owns the events it receives.
- An **entrypoint** is a receiving URL of the form `/h/{uuid}`. Senders post to - An **entrypoint** is a receiving URL of the form `/h/{uuid}`. Senders post to
it. The UUID is random, and it is the only thing that authorises a sender. it. The UUID is a random 36-character identifier, and it is the only thing
that authorises a sender.
- A **target** is a place events are delivered to. There are four kinds: an HTTP - A **target** is a place events are delivered to. There are four kinds: an HTTP
URL, a Slack or Mattermost incoming webhook, an archive database file, and the URL, a Slack or Mattermost incoming webhook, an archive database file, and the
application log. application log.
@@ -36,11 +38,11 @@ webhook. The sender gets `200` once that transaction has committed. Workers then
carry out the deliveries. A delivery to an HTTP or Slack target that fails is carry out the deliveries. A delivery to an HTTP or Slack target that fails is
retried, up to the number of attempts the target allows. retried, up to the number of attempts the target allows.
Delivery is at least once, not exactly once. If a send reaches the target but Delivery is at least once, not exactly once. If a delivery reaches the target
the record of that outcome cannot be written, the delivery is sent again later. but webhooker cannot record that outcome, it sends the delivery again later. A
A send that is in flight when the service stops is also sent again at the next delivery in flight when the service stops is also sent again at the next start.
start. webhooker adds no delivery identifier of its own to the request, so make webhooker adds no delivery identifier of its own to the request, so make your
your receiver tolerate a duplicate. receiver tolerate a duplicate.
## Quick start ## Quick start
@@ -75,13 +77,14 @@ If it is lost, run `webhooker resetpw admin` on a stopped deployment.
Open `http://127.0.0.1:8080/` and sign in. The server listens on `127.0.0.1` Open `http://127.0.0.1:8080/` and sign in. The server listens on `127.0.0.1`
port `8080` by default. port `8080` by default.
To run the published image instead of a clone: To run it in Docker instead, build the image from the clone with `make docker`,
which tags it `webhooker`. Then:
```bash ```bash
docker run -d \ docker run -d \
-p 127.0.0.1:8080:8080 \ -p 127.0.0.1:8080:8080 \
-v /path/to/data:/var/lib/webhooker \ -v /path/to/data:/var/lib/webhooker \
webhooker:latest webhooker
``` ```
Read [Running with Docker](#running-with-docker) before publishing the port Read [Running with Docker](#running-with-docker) before publishing the port
@@ -207,9 +210,9 @@ A delivery whose target was deleted shows no **Replay** button.
**Resubmit** sits beside each event. It stores a copy of the event as a new **Resubmit** sits beside each event. It stores a copy of the event as a new
event and delivers it to every target that is active now. That includes targets event and delivers it to every target that is active now. That includes targets
created after the original arrived. The copy records which event it came from, created after the original arrived. The copy's row names the event it came from.
and the log shows the link both ways. Resubmitting the same event many times is The original's row shows how many copies were made. Resubmitting the same event
allowed. many times is allowed.
Both actions are limited to 30 per minute per client. Both actions are limited to 30 per minute per client.
@@ -227,19 +230,23 @@ not set.
### Running with Docker ### Running with Docker
No image is published. `make docker` builds one from the clone and tags it
`webhooker`. Then:
```bash ```bash
docker run -d \ docker run -d \
-p 127.0.0.1:8080:8080 \ -p 127.0.0.1:8080:8080 \
-v /path/to/data:/var/lib/webhooker \ -v /path/to/data:/var/lib/webhooker \
webhooker:latest webhooker
``` ```
The image listens on every address inside the container: it ships The image listens on every address inside the container: it ships
`BIND_ADDRESS=0.0.0.0`. The container's network namespace is the boundary, and `BIND_ADDRESS=0.0.0.0`. A container has its own network namespace, which means
the `-p` flag decides what the host exposes. Publish to `127.0.0.1:8080`, the its own addresses, separate from the host's. That is the boundary, and the `-p`
address your reverse proxy connects from, and nothing wider. A bare flag decides what the host exposes. Publish to `127.0.0.1:8080`, the address
`-p 8080:8080` opens the port on every interface of the host, and Docker's your reverse proxy connects from, and nothing wider. A bare `-p 8080:8080` opens
forwarding rules sit ahead of most host firewalls. the port on every interface of the host, and Docker's forwarding rules sit ahead
of most host firewalls.
Do not set `BIND_ADDRESS` to an empty value in a container. An empty value means Do not set `BIND_ADDRESS` to an empty value in a container. An empty value means
unset, and unset falls back to the binary's default of `127.0.0.1`. The unset, and unset falls back to the binary's default of `127.0.0.1`. The
@@ -283,7 +290,7 @@ docker run --rm -v webhooker-data:/var/lib/webhooker \
probes port `8080`. probes port `8080`.
- Mount one host directory at `/var/lib/webhooker`. - Mount one host directory at `/var/lib/webhooker`.
- Set `TRUSTED_PROXIES` to your reverse proxy's address if clients can reach the - Set `TRUSTED_PROXIES` to your reverse proxy's address if clients can reach the
proxy from a private address. See [Trusted proxies](#trusted-proxies). proxy from a private address. See [`TRUSTED_PROXIES`](#trusted_proxies).
The first start prints the `admin` password to the container's log. For an app The first start prints the `admin` password to the container's log. For an app
named `webhooker`: named `webhooker`:
@@ -308,7 +315,7 @@ docker start upaas-webhooker
webhooker speaks plain HTTP and terminates no TLS. Put a reverse proxy in front webhooker speaks plain HTTP and terminates no TLS. Put a reverse proxy in front
of it. The proxy holds the certificate and reaches webhooker over loopback. of it. The proxy holds the certificate and reaches webhooker over loopback.
Five settings must be right. Each one fails silently when wrong. Five things must be right.
1. **Keep the app port private.** The binary binds `127.0.0.1` by default. The 1. **Keep the app port private.** The binary binds `127.0.0.1` by default. The
image binds every address inside its container, so publish its port to image binds every address inside its container, so publish its port to
@@ -320,8 +327,9 @@ Five settings must be right. Each one fails silently when wrong.
webhooker reads this header from any peer, so the proxy must overwrite webhooker reads this header from any peer, so the proxy must overwrite
whatever the client sent. whatever the client sent.
3. **Cover the proxy in `TRUSTED_PROXIES`.** Otherwise every client shares one 3. **Cover the proxy in `TRUSTED_PROXIES`.** Otherwise every client shares one
rate-limit bucket keyed on the proxy's address. The default list is the rate-limit bucket, the count kept per client, keyed on the proxy's address.
private ranges, which does not cover a proxy that connects over loopback. The default list is the private ranges, which does not cover a proxy that
connects over loopback.
4. **Send `Host` as `$http_host`, not `$host`.** `$host` drops the port. The 4. **Send `Host` as `$http_host`, not `$host`.** `$host` drops the port. The
form origin check compares against the host it was given. On any port other form origin check compares against the host it was given. On any port other
than 443 every form submission then fails with `403`. than 443 every form submission then fails with `403`.
@@ -330,8 +338,9 @@ Five settings must be right. Each one fails silently when wrong.
the entrypoint UUID of every event. Protect that log as you would the the entrypoint UUID of every event. Protect that log as you would the
entrypoint URLs. entrypoint URLs.
`WEBHOOKER_ENVIRONMENT` decides none of this. It controls CORS only. Leave it `WEBHOOKER_ENVIRONMENT` decides none of this. It controls only CORS, the headers
unset in production. that let a page on another site call this one from a browser. Leave it unset in
production.
#### nginx #### nginx
@@ -408,7 +417,7 @@ So serve webhooker on a hostname you will keep on HTTPS.
`includeSubDomains` extends the rule to every name under the hostname. `preload` `includeSubDomains` extends the rule to every name under the hostname. `preload`
invites browsers to put the domain on their built-in HSTS lists. The header invites browsers to put the domain on their built-in HSTS lists. The header
alone lists nothing. A listing happens when someone submits a registered domain alone lists nothing. A listing happens when someone submits a registered domain
at hstspreload.org, and removal takes months. at `hstspreload.org`, and removal takes months.
Setting `HSTS_MAX_AGE=0` stops the header. Browsers that already received it Setting `HSTS_MAX_AGE=0` stops the header. Browsers that already received it
keep the rule until their `max-age` runs out. To end the rule sooner, send a keep the rule until their `max-age` runs out. To end the rule sooner, send a
@@ -534,9 +543,9 @@ who can read it.
Each entrypoint is a random version 4 UUID served at `/h/{uuid}`. Holding that Each entrypoint is a random version 4 UUID served at `/h/{uuid}`. Holding that
URL is what authorises a sender. The receiver checks nothing else: no shared URL is what authorises a sender. The receiver checks nothing else: no shared
secret, no HMAC signature, no token header. None will be added. Signature secret, no HMAC signature (a hash of the body made with a shared key), no token
headers a sender adds anyway, such as `X-Hub-Signature`, are stored and header. None will be added. Signature headers a sender adds anyway, such as
forwarded as ordinary headers. `X-Hub-Signature`, are stored and forwarded as ordinary headers.
What that means in practice: What that means in practice:
@@ -550,54 +559,59 @@ What that means in practice:
### The web interface ### The web interface
- Passwords are hashed with Argon2id, using 64 MB of memory, 1 pass, 4 threads, - Passwords are hashed with Argon2id, a memory-hard password hashing function,
a 32-byte key and a 16-byte salt. using 64 MB of memory, 1 pass, 4 threads, a 32-byte key and a 16-byte salt.
- Session cookies are encrypted, `HttpOnly` and `SameSite=Lax`. They carry - Session cookies are signed, `HttpOnly` and `SameSite=Lax`. They carry `Secure`
`Secure` when the request arrived over TLS, judged per request from a direct when the request arrived over TLS, judged per request from a direct TLS
TLS connection or `X-Forwarded-Proto: https`. connection or `X-Forwarded-Proto: https`.
- The session is regenerated at sign-in, which prevents session fixation. - The session is regenerated at sign-in, which prevents session fixation.
- The session key is 32 random bytes made on first start and stored in - The key that signs the cookies is 32 random bytes made on first start and
`webhooker.db`. stored in `webhooker.db`.
- Every form is protected against cross-site request forgery with a token tied - Every form is protected against cross-site request forgery with a token tied
to a cookie. Over TLS the form's `Origin` or `Referer` must also match the to a cookie. Over TLS the form's `Origin` or `Referer` must also match the
host. host.
- Form bodies are capped at 1 MB. A larger declared body is answered `413` - Form bodies are capped at 1 MB. A larger declared body is answered `413`
before anything reads it. before anything reads it.
- Every response carries `Strict-Transport-Security` (unless `HSTS_MAX_AGE` is - Every response carries these headers:
`0`), `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, a - `Strict-Transport-Security`, unless `HSTS_MAX_AGE` is `0`;
`Content-Security-Policy` of - `X-Content-Type-Options: nosniff`;
`default-src 'self'; script-src 'self'; style-src 'self'`, - `X-Frame-Options: DENY`;
`Referrer-Policy: strict-origin-when-cross-origin` and a `Permissions-Policy` - `Content-Security-Policy`, which tells the browser where scripts and
that denies camera, microphone and location. styles may come from, set to
`default-src 'self'; script-src 'self'; style-src 'self'`;
- `Referrer-Policy: strict-origin-when-cross-origin`;
- a `Permissions-Policy` that denies camera, microphone and location.
- Authenticated pages are sent with `Cache-Control: no-store`. - Authenticated pages are sent with `Cache-Control: no-store`.
- `/metrics` sits behind HTTP basic authentication. - `/metrics` sits behind HTTP basic authentication, a username and password sent
with each request.
### Sign-in limits ### Sign-in limits
The sign-in form has no rate limiter in front of it, on purpose. Behind a proxy The sign-in form has no rate limiter in front of it, on purpose. Behind a proxy
that `TRUSTED_PROXIES` does not cover, every client shares one bucket. A limiter that `TRUSTED_PROXIES` does not cover, every client shares one bucket. A limit
spent on arrival would let any stranger lock the operator out with a few that counted every request would let any stranger lock the operator out with a
requests a minute. few requests a minute.
Instead, the password is checked first, and only a failure is counted. A correct Instead, the password is checked first, and only a failure is counted. A correct
password is never throttled. Failures are counted per client and submitted password is never throttled. Failures are counted per client and submitted
username, 5 per minute, after which further failures are answered `429` with username. From the fifth failure within a minute, each failure is answered `429`
`Retry-After`. A successful sign-in clears the count. At most 1024 username with `Retry-After`. A successful sign-in clears the count. At most 1024 username
counters and 1024 client counters are kept. counters and 1024 client counters are kept.
Two password checks can run at once. Each takes 64 MB of memory, so this caps Two password checks can run at once; each running check holds one of two slots.
hashing at 128 MB. A request that gets no slot within 5 seconds, or arrives when Each check takes 64 MB of memory, so hashing is capped at 128 MB. A request that
16 requests are already waiting, is answered `503` without hashing anything. The gets no slot within 5 seconds is answered `503` without hashing anything. So is
password change form shares these slots. one that arrives when 16 requests are already waiting for a slot. The password
change form shares these slots.
Because the check runs before the count, what bounds online guessing is the two The check runs before the count, so the two slots, not the failure count, bound
slots, not the counter. A flood of wrong passwords is tried at the speed two how fast passwords can be guessed. A flood of wrong passwords is tried as fast
Argon2id checks allow. Choose a long random admin password. To put a cheaper as two Argon2id checks allow. Choose a long random admin password. To put a
limit on top, rate-limit `POST /pages/login` at the reverse proxy, where the cheaper limit on top, rate-limit `POST /pages/login` at the reverse proxy, where
real client address is visible. the real client address is visible.
An unknown username is checked against a dummy hash, so it costs the same time An unknown username is checked against a placeholder hash, so it costs the same
as a real one. time as a real one.
A flood can hold both slots and make sign-in answer `503` for as long as it A flood can hold both slots and make sign-in answer `503` for as long as it
runs. Nothing accumulates: the operator's correct password works on the first runs. Nothing accumulates: the operator's correct password works on the first
@@ -611,10 +625,20 @@ it sits in and read the answers back. This is called server-side request
forgery. webhooker refuses it by default. forgery. webhooker refuses it by default.
Every target URL must use `http` or `https` and resolve to a public address. Every target URL must use `http` or `https` and resolve to a public address.
Private and reserved ranges are refused: loopback, the RFC 1918 ranges, carrier These private and reserved ranges are refused:
NAT, link-local, multicast, documentation space, and the IPv6 unique local
range. One public address is refused too: `168.63.129.16`, the Azure service - loopback, `127.0.0.0/8` and `::1`;
that hands a VM its credentials. - the RFC 1918 private ranges, `10.0.0.0/8`, `172.16.0.0/12` and
`192.168.0.0/16`;
- carrier-grade NAT space, `100.64.0.0/10`, which internet providers use inside
their own networks;
- link-local space, `169.254.0.0/16` and `fe80::/10`, which is valid only on one
network segment;
- the IPv6 unique local range, `fc00::/7`;
- multicast, documentation space and the other reserved ranges.
One public address is refused too: `168.63.129.16`, the Azure service that hands
a VM its credentials.
The check runs when a target is saved and again when each delivery connects. The check runs when a target is saved and again when each delivery connects.
Both use the same function. The address is resolved again at connect time, so a Both use the same function. The address is resolved again at connect time, so a
@@ -627,27 +651,31 @@ host, a different port, or a step from `https` to `http`. This stops an open
redirect from handing a credential to another host. redirect from handing a credential to another host.
`ALLOWED_EGRESS_CIDRS` opens named networks for the common case of forwarding to `ALLOWED_EGRESS_CIDRS` opens named networks for the common case of forwarding to
your own network. Read [`ALLOWED_EGRESS_CIDRS`](#allowed_egress_cidrs) before your own network. Read
setting it. [Allowing egress to your own network](#allowing-egress-to-your-own-network)
before setting it.
### What the logs contain ### What the logs contain
webhooker writes structured logs to standard output: plain text on a terminal, webhooker writes structured logs to standard output: plain text on a terminal,
JSON otherwise. Each request writes one `INFO` line. JSON otherwise. Each request writes one access log line, `http request`, at
`INFO`. A request to a known entrypoint writes a second line,
`webhook request received`. A stored event writes a third,
`webhook event created`.
Fields a client can influence are cut to a fixed size, so one access log line is Fields a client can influence are cut to a fixed size, so one access log line is
at most 2,560 bytes. The query string is never logged. On the receiver route the at most 2,560 bytes. The query string is never logged. On the receiver route the
URL is logged as `/h/{uuid}`, never the real path, so entrypoint UUIDs stay out URL is logged as `/h/{uuid}`, never the real path, so entrypoint UUIDs stay out
of the log. On other routes a `3xx` or `4xx` response logs the route pattern, of the log. On other routes a `3xx` or `4xx` response logs the route pattern,
and a `2xx` or `5xx` logs the real path. The `Referer` is logged as its scheme which is the route as registered, such as `/user/{username}`. A `2xx` or `5xx`
and host only. logs the real path. The `Referer` is logged as its scheme and host only.
Every line that names a client carries two fields. `remoteIP` is the connecting Every line that names a client carries two fields. `remoteIP` is the connecting
peer, which behind a proxy is the proxy. `clientIP` is the client as decided by peer, which behind a proxy is the proxy. `clientIP` is the client as decided by
`TRUSTED_PROXIES`. `TRUSTED_PROXIES`.
SQL statements are logged with placeholders, never with the values bound to SQL statements are logged with `?` in place of each value, never with the values
them. Session cookies, API keys and target credentials are never logged. themselves. Session cookies, API keys and target credentials are never logged.
Two things are logged in the clear. The first is the `admin` password, once, on Two things are logged in the clear. The first is the `admin` password, once, on
the first boot, and again when `webhooker resetpw -generate` is run. The second the first boot, and again when `webhooker resetpw -generate` is run. The second
@@ -660,8 +688,8 @@ purpose.
A recovered panic writes one `ERROR` record of at most 10,240 bytes, with the A recovered panic writes one `ERROR` record of at most 10,240 bytes, with the
request id of the request that caused it. request id of the request that caused it.
When `SENTRY_DSN` is set, error reports go to Sentry with the request reduced to When `SENTRY_DSN` is set, error reports go to Sentry, an error reporting
this: service, with the request reduced to this:
- the URL with its path replaced by the route pattern; - the URL with its path replaced by the route pattern;
- the query string and the body replaced by `(redacted)`; - the query string and the body replaced by `(redacted)`;
@@ -684,14 +712,18 @@ says what range its variable accepts.
A Go duration is a number with a unit, such as `30s`, `15m`, `24h` or `1h30m`. A Go duration is a number with a unit, such as `30s`, `15m`, `24h` or `1h30m`.
The startup log line `Configuration loaded` shows every value except the The startup log line `Configuration loaded` shows the loaded values, with three
secrets. The `/settings` page shows the same. differences. `TRUSTED_PROXIES` and `ALLOWED_EGRESS_CIDRS` appear as counts.
`SENTRY_DSN` and the metrics credentials appear as on or off. `METRICS_USERNAME`
does not appear. The `/settings` page shows every value; there,
`METRICS_PASSWORD` and `SENTRY_DSN` show only as set or not set.
### `WEBHOOKER_ENVIRONMENT` ### `WEBHOOKER_ENVIRONMENT`
`dev` or `prod`. Default `prod`. `dev` or `prod`. Default `prod`.
This controls one thing: CORS. In `dev`, every origin is allowed with This controls one thing: CORS, the headers that let a page on another site call
this one from a browser. In `dev`, every origin is allowed with
`Access-Control-Allow-Origin: *`, for local testing. In `prod`, no CORS headers `Access-Control-Allow-Origin: *`, for local testing. In `prod`, no CORS headers
are sent. Cookie security and log format do not depend on it. are sent. Cookie security and log format do not depend on it.
@@ -726,8 +758,8 @@ The directory for every SQLite database and the lock file. Default
Exactly one webhooker process may use a `DATA_DIR` at a time. Two processes on Exactly one webhooker process may use a `DATA_DIR` at a time. Two processes on
one directory would both run delivery recovery and deliver every pending event one directory would both run delivery recovery and deliver every pending event
twice. At startup the process takes an exclusive `flock` on twice. At startup the process locks `{DATA_DIR}/webhooker.lock` with `flock`,
`{DATA_DIR}/webhooker.lock` and holds it until it exits. A second process the kernel's file lock, and holds the lock until it exits. A second process
pointed at the same directory prints a message naming it and exits non-zero. pointed at the same directory prints a message naming it and exits non-zero.
The kernel releases the lock when the process exits, even on `kill -9`. A The kernel releases the lock when the process exits, even on `kill -9`. A
@@ -743,15 +775,20 @@ host. It does not work across hosts on a network filesystem.
`strconv.ParseBool` accepts: `1`, `t`, `T`, `TRUE`, `true`, `True`, `0`, `f`, `strconv.ParseBool` accepts: `1`, `t`, `T`, `TRUE`, `true`, `True`, `0`, `f`,
`F`, `FALSE`, `false`, `False`. `yes` and `on` are refused. `F`, `FALSE`, `false`, `False`. `yes` and `on` are refused.
`DEBUG=true` lowers the log level to `DEBUG`. That adds four kinds of line: `DEBUG=true` lowers the log level to `DEBUG`. Lines at that level include:
- every SQL statement, with placeholders rather than values; - every SQL statement, with `?` in place of each value;
- the lookup misses on the receiver and sign-in routes; - the lookup misses on the receiver and sign-in routes;
- the rate limiters' rejections; - signed-out requests sent to the sign-in page;
- the records of building the dependency graph at startup. - retries being scheduled, and retries skipped because the delivery had already
been handled;
- the receiver's whole-route rate limit refusing a request;
- a target deleted while the archive sweep was running;
- the records of how the components were wired at startup.
The output is safe to paste into a bug report once the first-boot password has Debug lines never carry SQL values, session cookies or target credentials. The
been changed. first-boot password is printed whatever the level, and so is every event a `log`
target writes.
### `METRICS_USERNAME` and `METRICS_PASSWORD` ### `METRICS_USERNAME` and `METRICS_PASSWORD`
@@ -805,8 +842,9 @@ configurable on its own.
### `TRUSTED_PROXIES` ### `TRUSTED_PROXIES`
A comma-separated list of networks in CIDR notation, such as A comma-separated list of networks in CIDR notation, which names a network as an
`192.168.1.7, 2001:db8::5`. A bare address counts as a single host. Default address and a prefix length, such as `10.0.0.0/8`. A bare address such as
`192.168.1.7` counts as a single host. Default
`10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`, the RFC 1918 private ranges. A set `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`, the RFC 1918 private ranges. A set
value replaces the default. value replaces the default.
@@ -834,21 +872,21 @@ throttled.
Rate-limit buckets are per IPv4 address and per IPv6 `/64` prefix. Rate-limit buckets are per IPv4 address and per IPv6 `/64` prefix.
### `ALLOWED_EGRESS_CIDRS` ### Allowing egress to your own network
A comma-separated list of networks in CIDR notation that delivery targets may `ALLOWED_EGRESS_CIDRS` is a comma-separated list of networks in CIDR notation
reach although the default rules refuse them. Default empty. A bare address that delivery targets may reach although the default rules refuse them. Default
counts as a single host. Example: `10.0.0.0/8, 172.17.0.0/16`. empty. A bare address counts as a single host. Example:
`10.0.0.0/8, 172.17.0.0/16`.
This only ever adds destinations. The guard stays on, and nothing in the list This only ever adds destinations. The default refusals stay in force for every
removes a destination that was already allowed. When the list is set, startup address not listed. When the list is set, startup logs the whole list at `WARN`.
logs it at `WARN`, blocks and all.
Each network you list is one that anyone with an admin account can make this Anyone with the admin account can make webhooker send requests to every network
process send requests into, and read the responses from in the delivery log. you list. They can read the responses in the delivery log. List the smallest
List the smallest blocks that cover your destinations. Prefer `10.1.2.3/32` over blocks that cover your destinations. Prefer `10.1.2.3/32` over `10.0.0.0/8`.
`10.0.0.0/8`. `0.0.0.0/0` or `::/0` opens every private and reserved range at `0.0.0.0/0` or `::/0` opens every private and reserved range at once, including
once, including this host's own loopback. Do not list them. this host's own loopback. Do not list them.
Some addresses stay blocked whatever you list. They are link-local space, the Some addresses stay blocked whatever you list. They are link-local space, the
unspecified addresses, and cloud metadata endpoints at fixed non-public unspecified addresses, and cloud metadata endpoints at fixed non-public
@@ -863,7 +901,8 @@ addresses that hand out credentials:
- `fd00:c1::a9fe:a9fe/128`, Oracle Cloud metadata over IPv6. - `fd00:c1::a9fe:a9fe/128`, Oracle Cloud metadata over IPv6.
- `fd00:42::42/128`, Scaleway metadata over IPv6. - `fd00:42::42/128`, Scaleway metadata over IPv6.
- `fd00:a9fe:a9fe::1/128`, Linode and Akamai metadata over IPv6. - `fd00:a9fe:a9fe::1/128`, Linode and Akamai metadata over IPv6.
- `100.100.100.200/32`, Alibaba Cloud metadata, inside the carrier NAT range. - `100.100.100.200/32`, Alibaba Cloud metadata, inside the carrier-grade NAT
range.
- `192.0.0.192/32`, Oracle Cloud Classic metadata. - `192.0.0.192/32`, Oracle Cloud Classic metadata.
- `0.0.0.0/32` and `::/128`, the unspecified addresses, which reach this host's - `0.0.0.0/32` and `::/128`, the unspecified addresses, which reach this host's
loopback on Linux. loopback on Linux.
@@ -885,10 +924,17 @@ default but is reopened by a list that covers it.
An `http` target posts the event body to its URL. The request carries the An `http` target posts the event body to its URL. The request carries the
event's `Content-Type`, the event's headers, then the target's configured event's `Content-Type`, the event's headers, then the target's configured
headers, then `User-Agent: webhooker/{version}`. A configured header replaces an headers, then `User-Agent: webhooker/{version}`. A configured header replaces an
event header of the same name. These event headers are never forwarded: `Host`, event header of the same name. A `2xx` response is a success. The first 4 KiB of
`Connection`, `Keep-Alive`, `Transfer-Encoding`, `Te`, `Trailer`, `Upgrade`, the response body is stored with the attempt.
`Proxy-Authorization`, `Proxy-Connection` and `Content-Length`. A `2xx` response
is a success. The first 4 KiB of the response body is stored with the attempt. These event headers are never forwarded:
- `Host`
- `Connection` and `Keep-Alive`
- `Transfer-Encoding` and `Content-Length`
- `Te` and `Trailer`
- `Upgrade`
- `Proxy-Authorization` and `Proxy-Connection`
An `http` target's settings: An `http` target's settings:
@@ -905,8 +951,8 @@ An `http` target's settings:
A `slack` target posts a JSON message with one `text` field to its URL. The A `slack` target posts a JSON message with one `text` field to its URL. The
message names the method, content type, time received and body size, then shows message names the method, content type, time received and body size, then shows
the body in a code block. A JSON body is indented. A body over 3500 characters the body in a code block. A JSON body is indented. A body over 3500 bytes is
is cut. The URL is any Slack-compatible incoming webhook URL, which includes cut. The URL is any Slack-compatible incoming webhook URL, which includes
Mattermost. Mattermost.
A `database` target, called an archive in the web interface, appends the event A `database` target, called an archive in the web interface, appends the event
@@ -1002,16 +1048,19 @@ An archive target writes to
names are made safe for a file name: names are made safe for a file name:
- lowercased; - lowercased;
- letters and digits kept, and every other run of characters turned into one - only the ASCII letters `a` to `z` and the digits `0` to `9` kept, and every
`-`; other run of characters turned into one `-`;
- cut to 40 characters; - cut to 40 characters;
- `unnamed` when nothing is left. - `unnamed` when nothing is left.
A webhook named `Orders (EU)` with a target named `Long-term archive` writes A webhook named `Orders (EU)` with a target named `Long-term archive` writes
`archive-orders-eu-long-term-archive-{target_uuid}.db`. `archive-orders-eu-long-term-archive-{target_uuid}.db`.
Each row holds the event's id, webhook id, entrypoint id, method, query string, Each row holds:
headers, body, content type and the time it was archived.
- the event's id, and the ids of its webhook and entrypoint;
- the method, query string, headers, body and content type;
- the time the row was archived.
`archive_rotation` is `none` by default, which keeps one file. `monthly`, `archive_rotation` is `none` by default, which keeps one file. `monthly`,
`daily` and `hourly` start a new file for each period of the event's receive `daily` and `hourly` start a new file for each period of the event's receive
@@ -1093,21 +1142,28 @@ keeps the retention cutoff inside the range of a Go duration.
`Entrypoint` has `webhook_id`, `path`, `description` and `active`. `path` is the `Entrypoint` has `webhook_id`, `path`, `description` and `active`. `path` is the
bare UUID. The `/h/` prefix is route only and is not stored. bare UUID. The `/h/` prefix is route only and is not stored.
`Target` has `webhook_id`, `name`, `type`, `active`, `url`, `headers`, `Target` has these fields. Each type fills only its own settings.
`timeout_seconds`, `forward_query`, `archive_expiry`, `archive_rotation` and
`max_attempts`. `type` is one of `http`, `slack`, `database` or `log`. Each type
fills only its own settings. `archive_expiry` is stored in nanoseconds, `0` for
never.
`APIKey` has `user_id`, `key`, `description` and `last_used_at`. The table - `webhook_id`, `name`, `type` and `active`. `type` is one of `http`, `slack`,
exists, but nothing creates a key and nothing checks one. `database` or `log`.
- `url`, `headers`, `timeout_seconds` and `forward_query`, the `http` settings.
A `slack` target uses `url` alone.
- `archive_expiry` and `archive_rotation`, the `database` settings.
`archive_expiry` is stored in nanoseconds, `0` for never.
- `max_attempts`, for `http` and `slack`.
`APIKey` has `user_id`, `key`, `description` and `last_used_at`. Its table,
`api_keys`, exists, but nothing creates a key and nothing checks one.
### Event tier ### Event tier
`Event` has `webhook_id`, `entrypoint_id`, `method`, `raw_query`, `headers` (as `Event` has these fields:
JSON), `body`, `content_type`, `body_bytes` and `resubmitted_from_id`. `method`
is always `POST`. `resubmitted_from_id` names the event a resubmit copied; it is - `webhook_id` and `entrypoint_id`;
empty for an event that arrived on the receiver. - `method`, which is always `POST`;
- `raw_query`, `headers` (as JSON), `body`, `body_bytes` and `content_type`;
- `resubmitted_from_id`, the event a resubmit copied. It is empty for an event
that arrived on the receiver.
`Delivery` has `event_id`, `target_id`, `status`, `finished_at` and `replay`. `Delivery` has `event_id`, `target_id`, `status`, `finished_at` and `replay`.
`finished_at` is set when the status becomes `delivered` or `failed`. `finished_at` is set when the status becomes `delivered` or `failed`.
@@ -1163,12 +1219,16 @@ Routes behind the sign-in. A `GET` without a session is redirected to
`POST /hook/{id}/events/{eventID}/resubmit` resubmits an event, each 30 per `POST /hook/{id}/events/{eventID}/resubmit` resubmits an event, each 30 per
minute per client. minute per client.
- `POST /hook/{id}/entrypoints` adds an entrypoint. - `POST /hook/{id}/entrypoints` adds an entrypoint.
`POST /hook/{id}/entrypoints/{entrypointID}/edit`, `/delete` and `/toggle` - `POST /hook/{id}/entrypoints/{entrypointID}/edit` changes its description.
edit, delete, and activate or deactivate one. - `POST /hook/{id}/entrypoints/{entrypointID}/delete` deletes it.
- `POST /hook/{id}/targets` adds a target. `GET` and - `POST /hook/{id}/entrypoints/{entrypointID}/toggle` activates or deactivates
`POST /hook/{id}/targets/{targetID}/edit` edit one. `POST .../delete` and it.
`POST .../toggle` delete, and activate or deactivate one. - `POST /hook/{id}/targets` adds a target.
`GET /hook/{id}/targets/{targetID}/download` downloads an archive. - `GET /hook/{id}/targets/{targetID}/edit` shows the target edit form, and
`POST` to the same path saves it.
- `POST /hook/{id}/targets/{targetID}/delete` deletes the target.
- `POST /hook/{id}/targets/{targetID}/toggle` activates or deactivates it.
- `GET /hook/{id}/targets/{targetID}/download` downloads an archive.
`GET /metrics` exists only when `METRICS_USERNAME` and `METRICS_PASSWORD` are `GET /metrics` exists only when `METRICS_USERNAME` and `METRICS_PASSWORD` are
set, behind basic authentication. set, behind basic authentication.
@@ -1180,10 +1240,10 @@ set, behind basic authentication.
`/metrics` serves one Prometheus registry. It carries the Go runtime and process `/metrics` serves one Prometheus registry. It carries the Go runtime and process
collectors, the delivery metrics below, and three inbound HTTP metrics. collectors, the delivery metrics below, and three inbound HTTP metrics.
Every delivery metric carries one label, `target_type`, with one of `http`, Nine of the eleven metrics below carry one label, `target_type`, with one of
`slack`, `database`, `log` or `unknown`. Target ids and event ids are never `http`, `slack`, `database`, `log` or `unknown`. The two event counters carry
labels, because a label value that grows with traffic would grow the registry none. Target ids and event ids are never labels, because a label value that
without bound. grows with traffic would grow the registry without bound.
- `webhooker_events_received_total` (counter, no label): events stored, - `webhooker_events_received_total` (counter, no label): events stored,
resubmitted copies included. resubmitted copies included.
@@ -1204,7 +1264,8 @@ without bound.
Deliveries to a deleted target count under `unknown`. Deliveries to a deleted target count under `unknown`.
- `webhooker_circuit_breakers_open` (gauge): breakers currently open. - `webhooker_circuit_breakers_open` (gauge): breakers currently open.
The outcome counters move only after the status change has been written. The success and failure counters move only after the new status has been
written.
The inbound HTTP metrics are `http_request_duration_seconds`, The inbound HTTP metrics are `http_request_duration_seconds`,
`http_response_size_bytes` and `http_requests_inflight`. Their `handler` label `http_response_size_bytes` and `http_requests_inflight`. Their `handler` label
@@ -1217,17 +1278,22 @@ for the whole service, with `handler` set to `(all)`.
### Prerequisites ### Prerequisites
- The Go version named in `go.mod`. - The Go version named in `go.mod`.
- Docker. The tests, the linters, the formatters and the stylesheet build all - Docker. The tests, the linters, prettier and the stylesheet build run in
run in Docker. Docker.
Nothing else is installed on the host. `golangci-lint`, ESLint, prettier, node, `gofmt` and `goimports` run on the host. These tools run only inside Docker, at
yarn and the Tailwind CLI run only inside Docker, at the versions pinned in the the versions pinned in the `Dockerfile`, `package.json` and `yarn.lock`:
`Dockerfile`, `package.json` and `yarn.lock`.
- `golangci-lint`;
- ESLint;
- prettier;
- node and yarn;
- the Tailwind CLI.
### Commands ### Commands
```bash ```bash
make bootstrap # Install git, make, Go and the Go modules (idempotent) make bootstrap # Install git, make, Go and the Go modules (safe to rerun)
make setup # bootstrap, then install the git pre-commit hook make setup # bootstrap, then install the git pre-commit hook
make assets # Extract Alpine.js from 3p/ to static/js/alpine.min.js make assets # Extract Alpine.js from 3p/ to static/js/alpine.min.js
make fmt # Format Go (gofmt, goimports) and Markdown (prettier, in Docker) make fmt # Format Go (gofmt, goimports) and Markdown (prettier, in Docker)
@@ -1235,7 +1301,7 @@ make fmt-check # Fail if gofmt or prettier would change anything
make lint # Run gofmt, golangci-lint and ESLint in Docker make lint # Run gofmt, golangci-lint and ESLint in Docker
make test # Run the tests in Docker, with cgo off make test # Run the tests in Docker, with cgo off
make test-browser # Run the browser test in Docker (Dockerfile.browser) make test-browser # Run the browser test in Docker (Dockerfile.browser)
make check # test, lint, fmt-check and css-check: the CI gate make check # test, lint, fmt-check and css-check: what CI runs
make build # Build bin/webhooker, version-stamped make build # Build bin/webhooker, version-stamped
make version # Print the version this checkout would stamp make version # Print the version this checkout would stamp
make run # build, then run ./bin/webhooker make run # build, then run ./bin/webhooker
@@ -1244,19 +1310,21 @@ make deps # go mod download, then go mod tidy
make docker # Build the Docker image make docker # Build the Docker image
make hooks # Install the git pre-commit hook make hooks # Install the git pre-commit hook
make css # Regenerate static/css/tailwind.css in Docker make css # Regenerate static/css/tailwind.css in Docker
make css-check # Fail if static/css/tailwind.css is stale make css-check # Fail if static/css/tailwind.css is out of date
make clean # Remove bin/ make clean # Remove bin/
``` ```
The repository follows The repository follows
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all). [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all).
Each `make` target above, except `build`, `run`, `dev`, `deps`, `version` and Each `make` target above, except `build`, `run`, `dev`, `deps`, `version` and
`clean`, is a thin shim over a script of the same name in `script/`. `clean`, is a thin shim over a script in `script/`. `make hooks` calls
`script/cibuild` is the CI entrypoint: it runs `script/bootstrap`, then `script/install-precommit`; every other shim calls the script of the same name.
`script/check`, then builds the image. `script/precommit` checks that `script/cibuild` is what CI runs: `script/bootstrap`, then `script/check`, then
`go mod tidy` changes nothing, then runs `script/check`. the image build. `script/precommit` checks that `go mod tidy` changes nothing,
then runs `script/check`.
The tests run with cgo off and so without Go's race detector. The tests run with cgo, Go's bridge to C code, turned off. Go's race detector
needs cgo, so the tests run without it.
### How the checks run ### How the checks run
@@ -1311,15 +1379,15 @@ builds of one commit is stamped in.
### Third-party browser assets ### Third-party browser assets
The pages use one third-party script, Alpine.js, in its CSP build The pages use one third-party script, Alpine.js, a small JavaScript library, in
(`@alpinejs/csp`). The Content-Security-Policy forbids `eval`, which the its CSP build (`@alpinejs/csp`). The Content-Security-Policy forbids `eval`,
standard build needs. With the CSP build, every Alpine directive in `templates/` which the standard build needs. With the CSP build, every Alpine directive in
names a component registered in `static/js/app.js`, such as `templates/` names a component registered in `static/js/app.js`, such as
`x-data="collapsible"`, and never an inline expression. `x-data="collapsible"`, and never an inline expression.
The npm tarball is committed as `3p/alpinejs-csp-3.14.9.tgz`, byte for byte as The npm tarball is committed as `3p/alpinejs-csp-3.14.9.tgz`, byte for byte as
the registry publishes it. `script/assets` extracts `package/dist/cdn.min.js` the registry publishes it. `script/assets` extracts `package/dist/cdn.min.js`
from it to `static/js/alpine.min.js`, which `go:embed` picks up. `make build`, from it to `static/js/alpine.min.js`, which the binary embeds. `make build`,
`make dev` and the Docker stages run it first. The extracted file is not `make dev` and the Docker stages run it first. The extracted file is not
committed, and a build that skips the extraction fails with an error naming committed, and a build that skips the extraction fails with an error naming
`js/alpine.min.js`. `js/alpine.min.js`.
@@ -1334,11 +1402,11 @@ To move to a new version:
### Browser test ### Browser test
`internal/server` has a test that loads the pages in a headless browser under `internal/server` has a test that loads the pages in a headless browser, one
the real Content-Security-Policy and drives the interface. It is built only with without a window, under the real Content-Security-Policy and drives the
the `browser` build tag, so `make test` leaves it out. `make test-browser` runs interface. It is built only with the `browser` build tag, so `make test` leaves
it in Docker with a pinned headless browser image. Run it after changing it out. `make test-browser` runs it in Docker with a pinned headless browser
`templates/` or `static/js/`. image. Run it after changing `templates/` or `static/js/`.
### Stylesheet ### Stylesheet
@@ -1416,12 +1484,27 @@ that order and their stop hooks in reverse. The components are:
- the delivery engine, the archive writers and the archive sweep; - the delivery engine, the archive writers and the archive sweep;
- the server. - the server.
The global middleware runs in this order: request id, security headers, access The global middleware runs in this order:
log, metrics (when `/metrics` is served), CORS, a 60 second request timeout,
panic recovery, and Sentry when configured. Each page route group adds, in 1. request id;
order, its own panic recovery, the 1 MB body cap, CSRF, 2. security headers;
`Cache-Control: no-store`, and the sign-in check. The body cap comes before CSRF 3. access log;
because CSRF parses the form. 4. metrics, when `/metrics` is served;
5. CORS;
6. a 60 second request timeout;
7. panic recovery;
8. Sentry, when configured.
Each page route group then adds its own, in this order:
1. panic recovery;
2. the 1 MB body cap;
3. CSRF;
4. `Cache-Control: no-store`;
5. the sign-in check, in every group except `/pages`, which serves the sign-in
form.
The body cap comes before CSRF because CSRF parses the form.
## License ## License