1 Commits
Author SHA1 Message Date
sneak 542189f58e Rewrite README.md sentence by sentence
check / check (push) Canceled after 0s
Rewritten from scratch against the code on next. The old text was
the list of what a reader needs covered.

Model: fable-5-1
2026-10-25 02:23:08 +00:00
+160 -243
View File
@@ -8,10 +8,9 @@ receiver is down at that moment, the event is lost.
webhooker sits between the sender and your receiver. It is a single Go binary webhooker sits between the sender and your receiver. It is a single Go binary
with a web interface and SQLite storage. It accepts every event the sender posts with a web interface and SQLite storage. It accepts every event the sender posts
and writes it to disk before answering. It then delivers the event to each and writes it to disk before answering. It then delivers the event to each
target you configured, with retries. The web interface shows each webhook's target you configured, with retries. You can read every event and every delivery
newest events, with every delivery and attempt they had. From there you can send attempt in the web interface, send a stored event again, and watch the whole
a stored event again. It also exports metrics for Prometheus, the monitoring thing through Prometheus metrics.
system, so you can watch the whole flow.
webhooker is written in [Go](https://golang.org) by webhooker is written in [Go](https://golang.org) by
[@sneak](https://sneak.berlin). The license is MIT. [@sneak](https://sneak.berlin). The license is MIT.
@@ -23,8 +22,7 @@ Five words name the moving parts.
- A **webhook** is a named configuration. It groups entrypoints and targets, and - A **webhook** is a named configuration. It groups entrypoints and targets, and
it owns the events it receives. it owns the events it receives.
- An **entrypoint** is a receiving URL of the form `/h/{uuid}`. Senders post to - An **entrypoint** is a receiving URL of the form `/h/{uuid}`. Senders post to
it. The UUID is a random 36-character identifier, and it is the only thing it. The UUID is random, and it is the only thing that authorises a sender.
that authorises a sender.
- A **target** is a place events are delivered to. There are four kinds: an HTTP - A **target** is a place events are delivered to. There are four kinds: an HTTP
URL, a Slack or Mattermost incoming webhook, an archive database file, and the URL, a Slack or Mattermost incoming webhook, an archive database file, and the
application log. application log.
@@ -38,11 +36,11 @@ webhook. The sender gets `200` once that transaction has committed. Workers then
carry out the deliveries. A delivery to an HTTP or Slack target that fails is carry out the deliveries. A delivery to an HTTP or Slack target that fails is
retried, up to the number of attempts the target allows. retried, up to the number of attempts the target allows.
Delivery is at least once, not exactly once. If a delivery reaches the target Delivery is at least once, not exactly once. If a send reaches the target but
but webhooker cannot record that outcome, it sends the delivery again later. A the record of that outcome cannot be written, the delivery is sent again later.
delivery in flight when the service stops is also sent again at the next start. A send that is in flight when the service stops is also sent again at the next
webhooker adds no delivery identifier of its own to the request, so make your start. webhooker adds no delivery identifier of its own to the request, so make
receiver tolerate a duplicate. your receiver tolerate a duplicate.
## Quick start ## Quick start
@@ -77,14 +75,13 @@ If it is lost, run `webhooker resetpw admin` on a stopped deployment.
Open `http://127.0.0.1:8080/` and sign in. The server listens on `127.0.0.1` Open `http://127.0.0.1:8080/` and sign in. The server listens on `127.0.0.1`
port `8080` by default. port `8080` by default.
To run it in Docker instead, build the image from the clone with `make docker`, To run the published image instead of a clone:
which tags it `webhooker`. Then:
```bash ```bash
docker run -d \ docker run -d \
-p 127.0.0.1:8080:8080 \ -p 127.0.0.1:8080:8080 \
-v /path/to/data:/var/lib/webhooker \ -v /path/to/data:/var/lib/webhooker \
webhooker webhooker:latest
``` ```
Read [Running with Docker](#running-with-docker) before publishing the port Read [Running with Docker](#running-with-docker) before publishing the port
@@ -210,9 +207,9 @@ A delivery whose target was deleted shows no **Replay** button.
**Resubmit** sits beside each event. It stores a copy of the event as a new **Resubmit** sits beside each event. It stores a copy of the event as a new
event and delivers it to every target that is active now. That includes targets event and delivers it to every target that is active now. That includes targets
created after the original arrived. The copy's row names the event it came from. created after the original arrived. The copy records which event it came from,
The original's row shows how many copies were made. Resubmitting the same event and the log shows the link both ways. Resubmitting the same event many times is
many times is allowed. allowed.
Both actions are limited to 30 per minute per client. Both actions are limited to 30 per minute per client.
@@ -230,23 +227,19 @@ not set.
### Running with Docker ### Running with Docker
No image is published. `make docker` builds one from the clone and tags it
`webhooker`. Then:
```bash ```bash
docker run -d \ docker run -d \
-p 127.0.0.1:8080:8080 \ -p 127.0.0.1:8080:8080 \
-v /path/to/data:/var/lib/webhooker \ -v /path/to/data:/var/lib/webhooker \
webhooker webhooker:latest
``` ```
The image listens on every address inside the container: it ships The image listens on every address inside the container: it ships
`BIND_ADDRESS=0.0.0.0`. A container has its own network namespace, which means `BIND_ADDRESS=0.0.0.0`. The container's network namespace is the boundary, and
its own addresses, separate from the host's. That is the boundary, and the `-p` the `-p` flag decides what the host exposes. Publish to `127.0.0.1:8080`, the
flag decides what the host exposes. Publish to `127.0.0.1:8080`, the address address your reverse proxy connects from, and nothing wider. A bare
your reverse proxy connects from, and nothing wider. A bare `-p 8080:8080` opens `-p 8080:8080` opens the port on every interface of the host, and Docker's
the port on every interface of the host, and Docker's forwarding rules sit ahead forwarding rules sit ahead of most host firewalls.
of most host firewalls.
Do not set `BIND_ADDRESS` to an empty value in a container. An empty value means Do not set `BIND_ADDRESS` to an empty value in a container. An empty value means
unset, and unset falls back to the binary's default of `127.0.0.1`. The unset, and unset falls back to the binary's default of `127.0.0.1`. The
@@ -290,7 +283,7 @@ docker run --rm -v webhooker-data:/var/lib/webhooker \
probes port `8080`. probes port `8080`.
- Mount one host directory at `/var/lib/webhooker`. - Mount one host directory at `/var/lib/webhooker`.
- Set `TRUSTED_PROXIES` to your reverse proxy's address if clients can reach the - Set `TRUSTED_PROXIES` to your reverse proxy's address if clients can reach the
proxy from a private address. See [`TRUSTED_PROXIES`](#trusted_proxies). proxy from a private address. See [Trusted proxies](#trusted-proxies).
The first start prints the `admin` password to the container's log. For an app The first start prints the `admin` password to the container's log. For an app
named `webhooker`: named `webhooker`:
@@ -315,7 +308,7 @@ docker start upaas-webhooker
webhooker speaks plain HTTP and terminates no TLS. Put a reverse proxy in front webhooker speaks plain HTTP and terminates no TLS. Put a reverse proxy in front
of it. The proxy holds the certificate and reaches webhooker over loopback. of it. The proxy holds the certificate and reaches webhooker over loopback.
Five things must be right. Five settings must be right. Each one fails silently when wrong.
1. **Keep the app port private.** The binary binds `127.0.0.1` by default. The 1. **Keep the app port private.** The binary binds `127.0.0.1` by default. The
image binds every address inside its container, so publish its port to image binds every address inside its container, so publish its port to
@@ -327,9 +320,8 @@ Five things must be right.
webhooker reads this header from any peer, so the proxy must overwrite webhooker reads this header from any peer, so the proxy must overwrite
whatever the client sent. whatever the client sent.
3. **Cover the proxy in `TRUSTED_PROXIES`.** Otherwise every client shares one 3. **Cover the proxy in `TRUSTED_PROXIES`.** Otherwise every client shares one
rate-limit bucket, the count kept per client, keyed on the proxy's address. rate-limit bucket keyed on the proxy's address. The default list is the
The default list is the private ranges, which does not cover a proxy that private ranges, which does not cover a proxy that connects over loopback.
connects over loopback.
4. **Send `Host` as `$http_host`, not `$host`.** `$host` drops the port. The 4. **Send `Host` as `$http_host`, not `$host`.** `$host` drops the port. The
form origin check compares against the host it was given. On any port other form origin check compares against the host it was given. On any port other
than 443 every form submission then fails with `403`. than 443 every form submission then fails with `403`.
@@ -338,9 +330,8 @@ Five things must be right.
the entrypoint UUID of every event. Protect that log as you would the the entrypoint UUID of every event. Protect that log as you would the
entrypoint URLs. entrypoint URLs.
`WEBHOOKER_ENVIRONMENT` decides none of this. It controls only CORS, the headers `WEBHOOKER_ENVIRONMENT` decides none of this. It controls CORS only. Leave it
that let a page on another site call this one from a browser. Leave it unset in unset in production.
production.
#### nginx #### nginx
@@ -417,7 +408,7 @@ So serve webhooker on a hostname you will keep on HTTPS.
`includeSubDomains` extends the rule to every name under the hostname. `preload` `includeSubDomains` extends the rule to every name under the hostname. `preload`
invites browsers to put the domain on their built-in HSTS lists. The header invites browsers to put the domain on their built-in HSTS lists. The header
alone lists nothing. A listing happens when someone submits a registered domain alone lists nothing. A listing happens when someone submits a registered domain
at `hstspreload.org`, and removal takes months. at hstspreload.org, and removal takes months.
Setting `HSTS_MAX_AGE=0` stops the header. Browsers that already received it Setting `HSTS_MAX_AGE=0` stops the header. Browsers that already received it
keep the rule until their `max-age` runs out. To end the rule sooner, send a keep the rule until their `max-age` runs out. To end the rule sooner, send a
@@ -543,9 +534,9 @@ who can read it.
Each entrypoint is a random version 4 UUID served at `/h/{uuid}`. Holding that Each entrypoint is a random version 4 UUID served at `/h/{uuid}`. Holding that
URL is what authorises a sender. The receiver checks nothing else: no shared URL is what authorises a sender. The receiver checks nothing else: no shared
secret, no HMAC signature (a hash of the body made with a shared key), no token secret, no HMAC signature, no token header. None will be added. Signature
header. None will be added. Signature headers a sender adds anyway, such as headers a sender adds anyway, such as `X-Hub-Signature`, are stored and
`X-Hub-Signature`, are stored and forwarded as ordinary headers. forwarded as ordinary headers.
What that means in practice: What that means in practice:
@@ -559,59 +550,54 @@ What that means in practice:
### The web interface ### The web interface
- Passwords are hashed with Argon2id, a memory-hard password hashing function, - Passwords are hashed with Argon2id, using 64 MB of memory, 1 pass, 4 threads,
using 64 MB of memory, 1 pass, 4 threads, a 32-byte key and a 16-byte salt. a 32-byte key and a 16-byte salt.
- Session cookies are signed, `HttpOnly` and `SameSite=Lax`. They carry `Secure` - Session cookies are encrypted, `HttpOnly` and `SameSite=Lax`. They carry
when the request arrived over TLS, judged per request from a direct TLS `Secure` when the request arrived over TLS, judged per request from a direct
connection or `X-Forwarded-Proto: https`. TLS connection or `X-Forwarded-Proto: https`.
- The session is regenerated at sign-in, which prevents session fixation. - The session is regenerated at sign-in, which prevents session fixation.
- The key that signs the cookies is 32 random bytes made on first start and - The session key is 32 random bytes made on first start and stored in
stored in `webhooker.db`. `webhooker.db`.
- Every form is protected against cross-site request forgery with a token tied - Every form is protected against cross-site request forgery with a token tied
to a cookie. Over TLS the form's `Origin` or `Referer` must also match the to a cookie. Over TLS the form's `Origin` or `Referer` must also match the
host. host.
- Form bodies are capped at 1 MB. A larger declared body is answered `413` - Form bodies are capped at 1 MB. A larger declared body is answered `413`
before anything reads it. before anything reads it.
- Every response carries these headers: - Every response carries `Strict-Transport-Security` (unless `HSTS_MAX_AGE` is
- `Strict-Transport-Security`, unless `HSTS_MAX_AGE` is `0`; `0`), `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, a
- `X-Content-Type-Options: nosniff`; `Content-Security-Policy` of
- `X-Frame-Options: DENY`; `default-src 'self'; script-src 'self'; style-src 'self'`,
- `Content-Security-Policy`, which tells the browser where scripts and `Referrer-Policy: strict-origin-when-cross-origin` and a `Permissions-Policy`
styles may come from, set to that denies camera, microphone and location.
`default-src 'self'; script-src 'self'; style-src 'self'`;
- `Referrer-Policy: strict-origin-when-cross-origin`;
- a `Permissions-Policy` that denies camera, microphone and location.
- Authenticated pages are sent with `Cache-Control: no-store`. - Authenticated pages are sent with `Cache-Control: no-store`.
- `/metrics` sits behind HTTP basic authentication, a username and password sent - `/metrics` sits behind HTTP basic authentication.
with each request.
### Sign-in limits ### Sign-in limits
The sign-in form has no rate limiter in front of it, on purpose. Behind a proxy The sign-in form has no rate limiter in front of it, on purpose. Behind a proxy
that `TRUSTED_PROXIES` does not cover, every client shares one bucket. A limit that `TRUSTED_PROXIES` does not cover, every client shares one bucket. A limiter
that counted every request would let any stranger lock the operator out with a spent on arrival would let any stranger lock the operator out with a few
few requests a minute. requests a minute.
Instead, the password is checked first, and only a failure is counted. A correct Instead, the password is checked first, and only a failure is counted. A correct
password is never throttled. Failures are counted per client and submitted password is never throttled. Failures are counted per client and submitted
username. From the fifth failure within a minute, each failure is answered `429` username, 5 per minute, after which further failures are answered `429` with
with `Retry-After`. A successful sign-in clears the count. At most 1024 username `Retry-After`. A successful sign-in clears the count. At most 1024 username
counters and 1024 client counters are kept. counters and 1024 client counters are kept.
Two password checks can run at once; each running check holds one of two slots. Two password checks can run at once. Each takes 64 MB of memory, so this caps
Each check takes 64 MB of memory, so hashing is capped at 128 MB. A request that hashing at 128 MB. A request that gets no slot within 5 seconds, or arrives when
gets no slot within 5 seconds is answered `503` without hashing anything. So is 16 requests are already waiting, is answered `503` without hashing anything. The
one that arrives when 16 requests are already waiting for a slot. The password password change form shares these slots.
change form shares these slots.
The check runs before the count, so the two slots, not the failure count, bound Because the check runs before the count, what bounds online guessing is the two
how fast passwords can be guessed. A flood of wrong passwords is tried as fast slots, not the counter. A flood of wrong passwords is tried at the speed two
as two Argon2id checks allow. Choose a long random admin password. To put a Argon2id checks allow. Choose a long random admin password. To put a cheaper
cheaper limit on top, rate-limit `POST /pages/login` at the reverse proxy, where limit on top, rate-limit `POST /pages/login` at the reverse proxy, where the
the real client address is visible. real client address is visible.
An unknown username is checked against a placeholder hash, so it costs the same An unknown username is checked against a dummy hash, so it costs the same time
time as a real one. as a real one.
A flood can hold both slots and make sign-in answer `503` for as long as it A flood can hold both slots and make sign-in answer `503` for as long as it
runs. Nothing accumulates: the operator's correct password works on the first runs. Nothing accumulates: the operator's correct password works on the first
@@ -625,20 +611,10 @@ it sits in and read the answers back. This is called server-side request
forgery. webhooker refuses it by default. forgery. webhooker refuses it by default.
Every target URL must use `http` or `https` and resolve to a public address. Every target URL must use `http` or `https` and resolve to a public address.
These private and reserved ranges are refused: Private and reserved ranges are refused: loopback, the RFC 1918 ranges, carrier
NAT, link-local, multicast, documentation space, and the IPv6 unique local
- loopback, `127.0.0.0/8` and `::1`; range. One public address is refused too: `168.63.129.16`, the Azure service
- the RFC 1918 private ranges, `10.0.0.0/8`, `172.16.0.0/12` and that hands a VM its credentials.
`192.168.0.0/16`;
- carrier-grade NAT space, `100.64.0.0/10`, which internet providers use inside
their own networks;
- link-local space, `169.254.0.0/16` and `fe80::/10`, which is valid only on one
network segment;
- the IPv6 unique local range, `fc00::/7`;
- multicast, documentation space and the other reserved ranges.
One public address is refused too: `168.63.129.16`, the Azure service that hands
a VM its credentials.
The check runs when a target is saved and again when each delivery connects. The check runs when a target is saved and again when each delivery connects.
Both use the same function. The address is resolved again at connect time, so a Both use the same function. The address is resolved again at connect time, so a
@@ -651,31 +627,27 @@ host, a different port, or a step from `https` to `http`. This stops an open
redirect from handing a credential to another host. redirect from handing a credential to another host.
`ALLOWED_EGRESS_CIDRS` opens named networks for the common case of forwarding to `ALLOWED_EGRESS_CIDRS` opens named networks for the common case of forwarding to
your own network. Read your own network. Read [`ALLOWED_EGRESS_CIDRS`](#allowed_egress_cidrs) before
[Allowing egress to your own network](#allowing-egress-to-your-own-network) setting it.
before setting it.
### What the logs contain ### What the logs contain
webhooker writes structured logs to standard output: plain text on a terminal, webhooker writes structured logs to standard output: plain text on a terminal,
JSON otherwise. Each request writes one access log line, `http request`, at JSON otherwise. Each request writes one `INFO` line.
`INFO`. A request to a known entrypoint writes a second line,
`webhook request received`. A stored event writes a third,
`webhook event created`.
Fields a client can influence are cut to a fixed size, so one access log line is Fields a client can influence are cut to a fixed size, so one access log line is
at most 2,560 bytes. The query string is never logged. On the receiver route the at most 2,560 bytes. The query string is never logged. On the receiver route the
URL is logged as `/h/{uuid}`, never the real path, so entrypoint UUIDs stay out URL is logged as `/h/{uuid}`, never the real path, so entrypoint UUIDs stay out
of the log. On other routes a `3xx` or `4xx` response logs the route pattern, of the log. On other routes a `3xx` or `4xx` response logs the route pattern,
which is the route as registered, such as `/user/{username}`. A `2xx` or `5xx` and a `2xx` or `5xx` logs the real path. The `Referer` is logged as its scheme
logs the real path. The `Referer` is logged as its scheme and host only. and host only.
Every line that names a client carries two fields. `remoteIP` is the connecting Every line that names a client carries two fields. `remoteIP` is the connecting
peer, which behind a proxy is the proxy. `clientIP` is the client as decided by peer, which behind a proxy is the proxy. `clientIP` is the client as decided by
`TRUSTED_PROXIES`. `TRUSTED_PROXIES`.
SQL statements are logged with `?` in place of each value, never with the values SQL statements are logged with placeholders, never with the values bound to
themselves. Session cookies, API keys and target credentials are never logged. them. Session cookies, API keys and target credentials are never logged.
Two things are logged in the clear. The first is the `admin` password, once, on Two things are logged in the clear. The first is the `admin` password, once, on
the first boot, and again when `webhooker resetpw -generate` is run. The second the first boot, and again when `webhooker resetpw -generate` is run. The second
@@ -688,8 +660,8 @@ purpose.
A recovered panic writes one `ERROR` record of at most 10,240 bytes, with the A recovered panic writes one `ERROR` record of at most 10,240 bytes, with the
request id of the request that caused it. request id of the request that caused it.
When `SENTRY_DSN` is set, error reports go to Sentry, an error reporting When `SENTRY_DSN` is set, error reports go to Sentry with the request reduced to
service, with the request reduced to this: this:
- the URL with its path replaced by the route pattern; - the URL with its path replaced by the route pattern;
- the query string and the body replaced by `(redacted)`; - the query string and the body replaced by `(redacted)`;
@@ -712,18 +684,14 @@ says what range its variable accepts.
A Go duration is a number with a unit, such as `30s`, `15m`, `24h` or `1h30m`. A Go duration is a number with a unit, such as `30s`, `15m`, `24h` or `1h30m`.
The startup log line `Configuration loaded` shows the loaded values, with three The startup log line `Configuration loaded` shows every value except the
differences. `TRUSTED_PROXIES` and `ALLOWED_EGRESS_CIDRS` appear as counts. secrets. The `/settings` page shows the same.
`SENTRY_DSN` and the metrics credentials appear as on or off. `METRICS_USERNAME`
does not appear. The `/settings` page shows every value; there,
`METRICS_PASSWORD` and `SENTRY_DSN` show only as set or not set.
### `WEBHOOKER_ENVIRONMENT` ### `WEBHOOKER_ENVIRONMENT`
`dev` or `prod`. Default `prod`. `dev` or `prod`. Default `prod`.
This controls one thing: CORS, the headers that let a page on another site call This controls one thing: CORS. In `dev`, every origin is allowed with
this one from a browser. In `dev`, every origin is allowed with
`Access-Control-Allow-Origin: *`, for local testing. In `prod`, no CORS headers `Access-Control-Allow-Origin: *`, for local testing. In `prod`, no CORS headers
are sent. Cookie security and log format do not depend on it. are sent. Cookie security and log format do not depend on it.
@@ -758,8 +726,8 @@ The directory for every SQLite database and the lock file. Default
Exactly one webhooker process may use a `DATA_DIR` at a time. Two processes on Exactly one webhooker process may use a `DATA_DIR` at a time. Two processes on
one directory would both run delivery recovery and deliver every pending event one directory would both run delivery recovery and deliver every pending event
twice. At startup the process locks `{DATA_DIR}/webhooker.lock` with `flock`, twice. At startup the process takes an exclusive `flock` on
the kernel's file lock, and holds the lock until it exits. A second process `{DATA_DIR}/webhooker.lock` and holds it until it exits. A second process
pointed at the same directory prints a message naming it and exits non-zero. pointed at the same directory prints a message naming it and exits non-zero.
The kernel releases the lock when the process exits, even on `kill -9`. A The kernel releases the lock when the process exits, even on `kill -9`. A
@@ -775,20 +743,15 @@ host. It does not work across hosts on a network filesystem.
`strconv.ParseBool` accepts: `1`, `t`, `T`, `TRUE`, `true`, `True`, `0`, `f`, `strconv.ParseBool` accepts: `1`, `t`, `T`, `TRUE`, `true`, `True`, `0`, `f`,
`F`, `FALSE`, `false`, `False`. `yes` and `on` are refused. `F`, `FALSE`, `false`, `False`. `yes` and `on` are refused.
`DEBUG=true` lowers the log level to `DEBUG`. Lines at that level include: `DEBUG=true` lowers the log level to `DEBUG`. That adds four kinds of line:
- every SQL statement, with `?` in place of each value; - every SQL statement, with placeholders rather than values;
- the lookup misses on the receiver and sign-in routes; - the lookup misses on the receiver and sign-in routes;
- signed-out requests sent to the sign-in page; - the rate limiters' rejections;
- retries being scheduled, and retries skipped because the delivery had already - the records of building the dependency graph at startup.
been handled;
- the receiver's whole-route rate limit refusing a request;
- a target deleted while the archive sweep was running;
- the records of how the components were wired at startup.
Debug lines never carry SQL values, session cookies or target credentials. The The output is safe to paste into a bug report once the first-boot password has
first-boot password is printed whatever the level, and so is every event a `log` been changed.
target writes.
### `METRICS_USERNAME` and `METRICS_PASSWORD` ### `METRICS_USERNAME` and `METRICS_PASSWORD`
@@ -842,9 +805,8 @@ configurable on its own.
### `TRUSTED_PROXIES` ### `TRUSTED_PROXIES`
A comma-separated list of networks in CIDR notation, which names a network as an A comma-separated list of networks in CIDR notation, such as
address and a prefix length, such as `10.0.0.0/8`. A bare address such as `192.168.1.7, 2001:db8::5`. A bare address counts as a single host. Default
`192.168.1.7` counts as a single host. Default
`10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`, the RFC 1918 private ranges. A set `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`, the RFC 1918 private ranges. A set
value replaces the default. value replaces the default.
@@ -872,21 +834,21 @@ throttled.
Rate-limit buckets are per IPv4 address and per IPv6 `/64` prefix. Rate-limit buckets are per IPv4 address and per IPv6 `/64` prefix.
### Allowing egress to your own network ### `ALLOWED_EGRESS_CIDRS`
`ALLOWED_EGRESS_CIDRS` is a comma-separated list of networks in CIDR notation A comma-separated list of networks in CIDR notation that delivery targets may
that delivery targets may reach although the default rules refuse them. Default reach although the default rules refuse them. Default empty. A bare address
empty. A bare address counts as a single host. Example: counts as a single host. Example: `10.0.0.0/8, 172.17.0.0/16`.
`10.0.0.0/8, 172.17.0.0/16`.
This only ever adds destinations. The default refusals stay in force for every This only ever adds destinations. The guard stays on, and nothing in the list
address not listed. When the list is set, startup logs the whole list at `WARN`. removes a destination that was already allowed. When the list is set, startup
logs it at `WARN`, blocks and all.
Anyone with the admin account can make webhooker send requests to every network Each network you list is one that anyone with an admin account can make this
you list. They can read the responses in the delivery log. List the smallest process send requests into, and read the responses from in the delivery log.
blocks that cover your destinations. Prefer `10.1.2.3/32` over `10.0.0.0/8`. List the smallest blocks that cover your destinations. Prefer `10.1.2.3/32` over
`0.0.0.0/0` or `::/0` opens every private and reserved range at once, including `10.0.0.0/8`. `0.0.0.0/0` or `::/0` opens every private and reserved range at
this host's own loopback. Do not list them. once, including this host's own loopback. Do not list them.
Some addresses stay blocked whatever you list. They are link-local space, the Some addresses stay blocked whatever you list. They are link-local space, the
unspecified addresses, and cloud metadata endpoints at fixed non-public unspecified addresses, and cloud metadata endpoints at fixed non-public
@@ -901,8 +863,7 @@ addresses that hand out credentials:
- `fd00:c1::a9fe:a9fe/128`, Oracle Cloud metadata over IPv6. - `fd00:c1::a9fe:a9fe/128`, Oracle Cloud metadata over IPv6.
- `fd00:42::42/128`, Scaleway metadata over IPv6. - `fd00:42::42/128`, Scaleway metadata over IPv6.
- `fd00:a9fe:a9fe::1/128`, Linode and Akamai metadata over IPv6. - `fd00:a9fe:a9fe::1/128`, Linode and Akamai metadata over IPv6.
- `100.100.100.200/32`, Alibaba Cloud metadata, inside the carrier-grade NAT - `100.100.100.200/32`, Alibaba Cloud metadata, inside the carrier NAT range.
range.
- `192.0.0.192/32`, Oracle Cloud Classic metadata. - `192.0.0.192/32`, Oracle Cloud Classic metadata.
- `0.0.0.0/32` and `::/128`, the unspecified addresses, which reach this host's - `0.0.0.0/32` and `::/128`, the unspecified addresses, which reach this host's
loopback on Linux. loopback on Linux.
@@ -924,17 +885,10 @@ default but is reopened by a list that covers it.
An `http` target posts the event body to its URL. The request carries the An `http` target posts the event body to its URL. The request carries the
event's `Content-Type`, the event's headers, then the target's configured event's `Content-Type`, the event's headers, then the target's configured
headers, then `User-Agent: webhooker/{version}`. A configured header replaces an headers, then `User-Agent: webhooker/{version}`. A configured header replaces an
event header of the same name. A `2xx` response is a success. The first 4 KiB of event header of the same name. These event headers are never forwarded: `Host`,
the response body is stored with the attempt. `Connection`, `Keep-Alive`, `Transfer-Encoding`, `Te`, `Trailer`, `Upgrade`,
`Proxy-Authorization`, `Proxy-Connection` and `Content-Length`. A `2xx` response
These event headers are never forwarded: is a success. The first 4 KiB of the response body is stored with the attempt.
- `Host`
- `Connection` and `Keep-Alive`
- `Transfer-Encoding` and `Content-Length`
- `Te` and `Trailer`
- `Upgrade`
- `Proxy-Authorization` and `Proxy-Connection`
An `http` target's settings: An `http` target's settings:
@@ -951,8 +905,8 @@ An `http` target's settings:
A `slack` target posts a JSON message with one `text` field to its URL. The A `slack` target posts a JSON message with one `text` field to its URL. The
message names the method, content type, time received and body size, then shows message names the method, content type, time received and body size, then shows
the body in a code block. A JSON body is indented. A body over 3500 bytes is the body in a code block. A JSON body is indented. A body over 3500 characters
cut. The URL is any Slack-compatible incoming webhook URL, which includes is cut. The URL is any Slack-compatible incoming webhook URL, which includes
Mattermost. Mattermost.
A `database` target, called an archive in the web interface, appends the event A `database` target, called an archive in the web interface, appends the event
@@ -1048,19 +1002,16 @@ An archive target writes to
names are made safe for a file name: names are made safe for a file name:
- lowercased; - lowercased;
- only the ASCII letters `a` to `z` and the digits `0` to `9` kept, and every - letters and digits kept, and every other run of characters turned into one
other run of characters turned into one `-`; `-`;
- cut to 40 characters; - cut to 40 characters;
- `unnamed` when nothing is left. - `unnamed` when nothing is left.
A webhook named `Orders (EU)` with a target named `Long-term archive` writes A webhook named `Orders (EU)` with a target named `Long-term archive` writes
`archive-orders-eu-long-term-archive-{target_uuid}.db`. `archive-orders-eu-long-term-archive-{target_uuid}.db`.
Each row holds: Each row holds the event's id, webhook id, entrypoint id, method, query string,
headers, body, content type and the time it was archived.
- the event's id, and the ids of its webhook and entrypoint;
- the method, query string, headers, body and content type;
- the time the row was archived.
`archive_rotation` is `none` by default, which keeps one file. `monthly`, `archive_rotation` is `none` by default, which keeps one file. `monthly`,
`daily` and `hourly` start a new file for each period of the event's receive `daily` and `hourly` start a new file for each period of the event's receive
@@ -1142,28 +1093,21 @@ keeps the retention cutoff inside the range of a Go duration.
`Entrypoint` has `webhook_id`, `path`, `description` and `active`. `path` is the `Entrypoint` has `webhook_id`, `path`, `description` and `active`. `path` is the
bare UUID. The `/h/` prefix is route only and is not stored. bare UUID. The `/h/` prefix is route only and is not stored.
`Target` has these fields. Each type fills only its own settings. `Target` has `webhook_id`, `name`, `type`, `active`, `url`, `headers`,
`timeout_seconds`, `forward_query`, `archive_expiry`, `archive_rotation` and
`max_attempts`. `type` is one of `http`, `slack`, `database` or `log`. Each type
fills only its own settings. `archive_expiry` is stored in nanoseconds, `0` for
never.
- `webhook_id`, `name`, `type` and `active`. `type` is one of `http`, `slack`, `APIKey` has `user_id`, `key`, `description` and `last_used_at`. The table
`database` or `log`. exists, but nothing creates a key and nothing checks one.
- `url`, `headers`, `timeout_seconds` and `forward_query`, the `http` settings.
A `slack` target uses `url` alone.
- `archive_expiry` and `archive_rotation`, the `database` settings.
`archive_expiry` is stored in nanoseconds, `0` for never.
- `max_attempts`, for `http` and `slack`.
`APIKey` has `user_id`, `key`, `description` and `last_used_at`. Its table,
`api_keys`, exists, but nothing creates a key and nothing checks one.
### Event tier ### Event tier
`Event` has these fields: `Event` has `webhook_id`, `entrypoint_id`, `method`, `raw_query`, `headers` (as
JSON), `body`, `content_type`, `body_bytes` and `resubmitted_from_id`. `method`
- `webhook_id` and `entrypoint_id`; is always `POST`. `resubmitted_from_id` names the event a resubmit copied; it is
- `method`, which is always `POST`; empty for an event that arrived on the receiver.
- `raw_query`, `headers` (as JSON), `body`, `body_bytes` and `content_type`;
- `resubmitted_from_id`, the event a resubmit copied. It is empty for an event
that arrived on the receiver.
`Delivery` has `event_id`, `target_id`, `status`, `finished_at` and `replay`. `Delivery` has `event_id`, `target_id`, `status`, `finished_at` and `replay`.
`finished_at` is set when the status becomes `delivered` or `failed`. `finished_at` is set when the status becomes `delivered` or `failed`.
@@ -1219,16 +1163,12 @@ Routes behind the sign-in. A `GET` without a session is redirected to
`POST /hook/{id}/events/{eventID}/resubmit` resubmits an event, each 30 per `POST /hook/{id}/events/{eventID}/resubmit` resubmits an event, each 30 per
minute per client. minute per client.
- `POST /hook/{id}/entrypoints` adds an entrypoint. - `POST /hook/{id}/entrypoints` adds an entrypoint.
- `POST /hook/{id}/entrypoints/{entrypointID}/edit` changes its description. `POST /hook/{id}/entrypoints/{entrypointID}/edit`, `/delete` and `/toggle`
- `POST /hook/{id}/entrypoints/{entrypointID}/delete` deletes it. edit, delete, and activate or deactivate one.
- `POST /hook/{id}/entrypoints/{entrypointID}/toggle` activates or deactivates - `POST /hook/{id}/targets` adds a target. `GET` and
it. `POST /hook/{id}/targets/{targetID}/edit` edit one. `POST .../delete` and
- `POST /hook/{id}/targets` adds a target. `POST .../toggle` delete, and activate or deactivate one.
- `GET /hook/{id}/targets/{targetID}/edit` shows the target edit form, and `GET /hook/{id}/targets/{targetID}/download` downloads an archive.
`POST` to the same path saves it.
- `POST /hook/{id}/targets/{targetID}/delete` deletes the target.
- `POST /hook/{id}/targets/{targetID}/toggle` activates or deactivates it.
- `GET /hook/{id}/targets/{targetID}/download` downloads an archive.
`GET /metrics` exists only when `METRICS_USERNAME` and `METRICS_PASSWORD` are `GET /metrics` exists only when `METRICS_USERNAME` and `METRICS_PASSWORD` are
set, behind basic authentication. set, behind basic authentication.
@@ -1240,10 +1180,10 @@ set, behind basic authentication.
`/metrics` serves one Prometheus registry. It carries the Go runtime and process `/metrics` serves one Prometheus registry. It carries the Go runtime and process
collectors, the delivery metrics below, and three inbound HTTP metrics. collectors, the delivery metrics below, and three inbound HTTP metrics.
Nine of the eleven metrics below carry one label, `target_type`, with one of Every delivery metric carries one label, `target_type`, with one of `http`,
`http`, `slack`, `database`, `log` or `unknown`. The two event counters carry `slack`, `database`, `log` or `unknown`. Target ids and event ids are never
none. Target ids and event ids are never labels, because a label value that labels, because a label value that grows with traffic would grow the registry
grows with traffic would grow the registry without bound. without bound.
- `webhooker_events_received_total` (counter, no label): events stored, - `webhooker_events_received_total` (counter, no label): events stored,
resubmitted copies included. resubmitted copies included.
@@ -1264,8 +1204,7 @@ grows with traffic would grow the registry without bound.
Deliveries to a deleted target count under `unknown`. Deliveries to a deleted target count under `unknown`.
- `webhooker_circuit_breakers_open` (gauge): breakers currently open. - `webhooker_circuit_breakers_open` (gauge): breakers currently open.
The success and failure counters move only after the new status has been The outcome counters move only after the status change has been written.
written.
The inbound HTTP metrics are `http_request_duration_seconds`, The inbound HTTP metrics are `http_request_duration_seconds`,
`http_response_size_bytes` and `http_requests_inflight`. Their `handler` label `http_response_size_bytes` and `http_requests_inflight`. Their `handler` label
@@ -1278,22 +1217,17 @@ for the whole service, with `handler` set to `(all)`.
### Prerequisites ### Prerequisites
- The Go version named in `go.mod`. - The Go version named in `go.mod`.
- Docker. The tests, the linters, prettier and the stylesheet build run in - Docker. The tests, the linters, the formatters and the stylesheet build all
Docker. run in Docker.
`gofmt` and `goimports` run on the host. These tools run only inside Docker, at Nothing else is installed on the host. `golangci-lint`, ESLint, prettier, node,
the versions pinned in the `Dockerfile`, `package.json` and `yarn.lock`: yarn and the Tailwind CLI run only inside Docker, at the versions pinned in the
`Dockerfile`, `package.json` and `yarn.lock`.
- `golangci-lint`;
- ESLint;
- prettier;
- node and yarn;
- the Tailwind CLI.
### Commands ### Commands
```bash ```bash
make bootstrap # Install git, make, Go and the Go modules (safe to rerun) make bootstrap # Install git, make, Go and the Go modules (idempotent)
make setup # bootstrap, then install the git pre-commit hook make setup # bootstrap, then install the git pre-commit hook
make assets # Extract Alpine.js from 3p/ to static/js/alpine.min.js make assets # Extract Alpine.js from 3p/ to static/js/alpine.min.js
make fmt # Format Go (gofmt, goimports) and Markdown (prettier, in Docker) make fmt # Format Go (gofmt, goimports) and Markdown (prettier, in Docker)
@@ -1301,7 +1235,7 @@ make fmt-check # Fail if gofmt or prettier would change anything
make lint # Run gofmt, golangci-lint and ESLint in Docker make lint # Run gofmt, golangci-lint and ESLint in Docker
make test # Run the tests in Docker, with cgo off make test # Run the tests in Docker, with cgo off
make test-browser # Run the browser test in Docker (Dockerfile.browser) make test-browser # Run the browser test in Docker (Dockerfile.browser)
make check # test, lint, fmt-check and css-check: what CI runs make check # test, lint, fmt-check and css-check: the CI gate
make build # Build bin/webhooker, version-stamped make build # Build bin/webhooker, version-stamped
make version # Print the version this checkout would stamp make version # Print the version this checkout would stamp
make run # build, then run ./bin/webhooker make run # build, then run ./bin/webhooker
@@ -1310,21 +1244,19 @@ make deps # go mod download, then go mod tidy
make docker # Build the Docker image make docker # Build the Docker image
make hooks # Install the git pre-commit hook make hooks # Install the git pre-commit hook
make css # Regenerate static/css/tailwind.css in Docker make css # Regenerate static/css/tailwind.css in Docker
make css-check # Fail if static/css/tailwind.css is out of date make css-check # Fail if static/css/tailwind.css is stale
make clean # Remove bin/ make clean # Remove bin/
``` ```
The repository follows The repository follows
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all). [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all).
Each `make` target above, except `build`, `run`, `dev`, `deps`, `version` and Each `make` target above, except `build`, `run`, `dev`, `deps`, `version` and
`clean`, is a thin shim over a script in `script/`. `make hooks` calls `clean`, is a thin shim over a script of the same name in `script/`.
`script/install-precommit`; every other shim calls the script of the same name. `script/cibuild` is the CI entrypoint: it runs `script/bootstrap`, then
`script/cibuild` is what CI runs: `script/bootstrap`, then `script/check`, then `script/check`, then builds the image. `script/precommit` checks that
the image build. `script/precommit` checks that `go mod tidy` changes nothing, `go mod tidy` changes nothing, then runs `script/check`.
then runs `script/check`.
The tests run with cgo, Go's bridge to C code, turned off. Go's race detector The tests run with cgo off and so without Go's race detector.
needs cgo, so the tests run without it.
### How the checks run ### How the checks run
@@ -1379,15 +1311,15 @@ builds of one commit is stamped in.
### Third-party browser assets ### Third-party browser assets
The pages use one third-party script, Alpine.js, a small JavaScript library, in The pages use one third-party script, Alpine.js, in its CSP build
its CSP build (`@alpinejs/csp`). The Content-Security-Policy forbids `eval`, (`@alpinejs/csp`). The Content-Security-Policy forbids `eval`, which the
which the standard build needs. With the CSP build, every Alpine directive in standard build needs. With the CSP build, every Alpine directive in `templates/`
`templates/` names a component registered in `static/js/app.js`, such as names a component registered in `static/js/app.js`, such as
`x-data="collapsible"`, and never an inline expression. `x-data="collapsible"`, and never an inline expression.
The npm tarball is committed as `3p/alpinejs-csp-3.14.9.tgz`, byte for byte as The npm tarball is committed as `3p/alpinejs-csp-3.14.9.tgz`, byte for byte as
the registry publishes it. `script/assets` extracts `package/dist/cdn.min.js` the registry publishes it. `script/assets` extracts `package/dist/cdn.min.js`
from it to `static/js/alpine.min.js`, which the binary embeds. `make build`, from it to `static/js/alpine.min.js`, which `go:embed` picks up. `make build`,
`make dev` and the Docker stages run it first. The extracted file is not `make dev` and the Docker stages run it first. The extracted file is not
committed, and a build that skips the extraction fails with an error naming committed, and a build that skips the extraction fails with an error naming
`js/alpine.min.js`. `js/alpine.min.js`.
@@ -1402,11 +1334,11 @@ To move to a new version:
### Browser test ### Browser test
`internal/server` has a test that loads the pages in a headless browser, one `internal/server` has a test that loads the pages in a headless browser under
without a window, under the real Content-Security-Policy and drives the the real Content-Security-Policy and drives the interface. It is built only with
interface. It is built only with the `browser` build tag, so `make test` leaves the `browser` build tag, so `make test` leaves it out. `make test-browser` runs
it out. `make test-browser` runs it in Docker with a pinned headless browser it in Docker with a pinned headless browser image. Run it after changing
image. Run it after changing `templates/` or `static/js/`. `templates/` or `static/js/`.
### Stylesheet ### Stylesheet
@@ -1484,27 +1416,12 @@ that order and their stop hooks in reverse. The components are:
- the delivery engine, the archive writers and the archive sweep; - the delivery engine, the archive writers and the archive sweep;
- the server. - the server.
The global middleware runs in this order: The global middleware runs in this order: request id, security headers, access
log, metrics (when `/metrics` is served), CORS, a 60 second request timeout,
1. request id; panic recovery, and Sentry when configured. Each page route group adds, in
2. security headers; order, its own panic recovery, the 1 MB body cap, CSRF,
3. access log; `Cache-Control: no-store`, and the sign-in check. The body cap comes before CSRF
4. metrics, when `/metrics` is served; because CSRF parses the form.
5. CORS;
6. a 60 second request timeout;
7. panic recovery;
8. Sentry, when configured.
Each page route group then adds its own, in this order:
1. panic recovery;
2. the 1 MB body cap;
3. CSRF;
4. `Cache-Control: no-store`;
5. the sign-in check, in every group except `/pages`, which serves the sign-in
form.
The body cap comes before CSRF because CSRF parses the form.
## License ## License