1 Commits
Author SHA1 Message Date
sneak 98af5afb9d Rewrite README.md sentence by sentence
check / check (push) Canceled after 0s
Rewritten from scratch against the code on next. The old text was
the list of what a reader needs covered.

Model: fable-5-1
2026-10-25 03:13:30 +00:00
+243 -160
View File
@@ -8,9 +8,10 @@ receiver is down at that moment, the event is lost.
webhooker sits between the sender and your receiver. It is a single Go binary
with a web interface and SQLite storage. It accepts every event the sender posts
and writes it to disk before answering. It then delivers the event to each
target you configured, with retries. You can read every event and every delivery
attempt in the web interface, send a stored event again, and watch the whole
thing through Prometheus metrics.
target you configured, with retries. The web interface shows each webhook's
newest events, with every delivery and attempt they had. From there you can send
a stored event again. It also exports metrics for Prometheus, the monitoring
system, so you can watch the whole flow.
webhooker is written in [Go](https://golang.org) by
[@sneak](https://sneak.berlin). The license is MIT.
@@ -22,7 +23,8 @@ Five words name the moving parts.
- A **webhook** is a named configuration. It groups entrypoints and targets, and
it owns the events it receives.
- An **entrypoint** is a receiving URL of the form `/h/{uuid}`. Senders post to
it. The UUID is random, and it is the only thing that authorises a sender.
it. The UUID is a random 36-character identifier, and it is the only thing
that authorises a sender.
- A **target** is a place events are delivered to. There are four kinds: an HTTP
URL, a Slack or Mattermost incoming webhook, an archive database file, and the
application log.
@@ -36,11 +38,11 @@ webhook. The sender gets `200` once that transaction has committed. Workers then
carry out the deliveries. A delivery to an HTTP or Slack target that fails is
retried, up to the number of attempts the target allows.
Delivery is at least once, not exactly once. If a send reaches the target but
the record of that outcome cannot be written, the delivery is sent again later.
A send that is in flight when the service stops is also sent again at the next
start. webhooker adds no delivery identifier of its own to the request, so make
your receiver tolerate a duplicate.
Delivery is at least once, not exactly once. If a delivery reaches the target
but webhooker cannot record that outcome, it sends the delivery again later. A
delivery in flight when the service stops is also sent again at the next start.
webhooker adds no delivery identifier of its own to the request, so make your
receiver tolerate a duplicate.
## Quick start
@@ -75,13 +77,14 @@ If it is lost, run `webhooker resetpw admin` on a stopped deployment.
Open `http://127.0.0.1:8080/` and sign in. The server listens on `127.0.0.1`
port `8080` by default.
To run the published image instead of a clone:
To run it in Docker instead, build the image from the clone with `make docker`,
which tags it `webhooker`. Then:
```bash
docker run -d \
-p 127.0.0.1:8080:8080 \
-v /path/to/data:/var/lib/webhooker \
webhooker:latest
webhooker
```
Read [Running with Docker](#running-with-docker) before publishing the port
@@ -207,9 +210,9 @@ A delivery whose target was deleted shows no **Replay** button.
**Resubmit** sits beside each event. It stores a copy of the event as a new
event and delivers it to every target that is active now. That includes targets
created after the original arrived. The copy records which event it came from,
and the log shows the link both ways. Resubmitting the same event many times is
allowed.
created after the original arrived. The copy's row names the event it came from.
The original's row shows how many copies were made. Resubmitting the same event
many times is allowed.
Both actions are limited to 30 per minute per client.
@@ -227,19 +230,23 @@ not set.
### Running with Docker
No image is published. `make docker` builds one from the clone and tags it
`webhooker`. Then:
```bash
docker run -d \
-p 127.0.0.1:8080:8080 \
-v /path/to/data:/var/lib/webhooker \
webhooker:latest
webhooker
```
The image listens on every address inside the container: it ships
`BIND_ADDRESS=0.0.0.0`. The container's network namespace is the boundary, and
the `-p` flag decides what the host exposes. Publish to `127.0.0.1:8080`, the
address your reverse proxy connects from, and nothing wider. A bare
`-p 8080:8080` opens the port on every interface of the host, and Docker's
forwarding rules sit ahead of most host firewalls.
`BIND_ADDRESS=0.0.0.0`. A container has its own network namespace, which means
its own addresses, separate from the host's. That is the boundary, and the `-p`
flag decides what the host exposes. Publish to `127.0.0.1:8080`, the address
your reverse proxy connects from, and nothing wider. A bare `-p 8080:8080` opens
the port on every interface of the host, and Docker's forwarding rules sit ahead
of most host firewalls.
Do not set `BIND_ADDRESS` to an empty value in a container. An empty value means
unset, and unset falls back to the binary's default of `127.0.0.1`. The
@@ -283,7 +290,7 @@ docker run --rm -v webhooker-data:/var/lib/webhooker \
probes port `8080`.
- Mount one host directory at `/var/lib/webhooker`.
- Set `TRUSTED_PROXIES` to your reverse proxy's address if clients can reach the
proxy from a private address. See [Trusted proxies](#trusted-proxies).
proxy from a private address. See [`TRUSTED_PROXIES`](#trusted_proxies).
The first start prints the `admin` password to the container's log. For an app
named `webhooker`:
@@ -308,7 +315,7 @@ docker start upaas-webhooker
webhooker speaks plain HTTP and terminates no TLS. Put a reverse proxy in front
of it. The proxy holds the certificate and reaches webhooker over loopback.
Five settings must be right. Each one fails silently when wrong.
Five things must be right.
1. **Keep the app port private.** The binary binds `127.0.0.1` by default. The
image binds every address inside its container, so publish its port to
@@ -320,8 +327,9 @@ Five settings must be right. Each one fails silently when wrong.
webhooker reads this header from any peer, so the proxy must overwrite
whatever the client sent.
3. **Cover the proxy in `TRUSTED_PROXIES`.** Otherwise every client shares one
rate-limit bucket keyed on the proxy's address. The default list is the
private ranges, which does not cover a proxy that connects over loopback.
rate-limit bucket, the count kept per client, keyed on the proxy's address.
The default list is the private ranges, which does not cover a proxy that
connects over loopback.
4. **Send `Host` as `$http_host`, not `$host`.** `$host` drops the port. The
form origin check compares against the host it was given. On any port other
than 443 every form submission then fails with `403`.
@@ -330,8 +338,9 @@ Five settings must be right. Each one fails silently when wrong.
the entrypoint UUID of every event. Protect that log as you would the
entrypoint URLs.
`WEBHOOKER_ENVIRONMENT` decides none of this. It controls CORS only. Leave it
unset in production.
`WEBHOOKER_ENVIRONMENT` decides none of this. It controls only CORS, the headers
that let a page on another site call this one from a browser. Leave it unset in
production.
#### nginx
@@ -408,7 +417,7 @@ So serve webhooker on a hostname you will keep on HTTPS.
`includeSubDomains` extends the rule to every name under the hostname. `preload`
invites browsers to put the domain on their built-in HSTS lists. The header
alone lists nothing. A listing happens when someone submits a registered domain
at hstspreload.org, and removal takes months.
at `hstspreload.org`, and removal takes months.
Setting `HSTS_MAX_AGE=0` stops the header. Browsers that already received it
keep the rule until their `max-age` runs out. To end the rule sooner, send a
@@ -534,9 +543,9 @@ who can read it.
Each entrypoint is a random version 4 UUID served at `/h/{uuid}`. Holding that
URL is what authorises a sender. The receiver checks nothing else: no shared
secret, no HMAC signature, no token header. None will be added. Signature
headers a sender adds anyway, such as `X-Hub-Signature`, are stored and
forwarded as ordinary headers.
secret, no HMAC signature (a hash of the body made with a shared key), no token
header. None will be added. Signature headers a sender adds anyway, such as
`X-Hub-Signature`, are stored and forwarded as ordinary headers.
What that means in practice:
@@ -550,54 +559,59 @@ What that means in practice:
### The web interface
- Passwords are hashed with Argon2id, using 64 MB of memory, 1 pass, 4 threads,
a 32-byte key and a 16-byte salt.
- Session cookies are encrypted, `HttpOnly` and `SameSite=Lax`. They carry
`Secure` when the request arrived over TLS, judged per request from a direct
TLS connection or `X-Forwarded-Proto: https`.
- Passwords are hashed with Argon2id, a memory-hard password hashing function,
using 64 MB of memory, 1 pass, 4 threads, a 32-byte key and a 16-byte salt.
- Session cookies are signed, `HttpOnly` and `SameSite=Lax`. They carry `Secure`
when the request arrived over TLS, judged per request from a direct TLS
connection or `X-Forwarded-Proto: https`.
- The session is regenerated at sign-in, which prevents session fixation.
- The session key is 32 random bytes made on first start and stored in
`webhooker.db`.
- The key that signs the cookies is 32 random bytes made on first start and
stored in `webhooker.db`.
- Every form is protected against cross-site request forgery with a token tied
to a cookie. Over TLS the form's `Origin` or `Referer` must also match the
host.
- Form bodies are capped at 1 MB. A larger declared body is answered `413`
before anything reads it.
- Every response carries `Strict-Transport-Security` (unless `HSTS_MAX_AGE` is
`0`), `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, a
`Content-Security-Policy` of
`default-src 'self'; script-src 'self'; style-src 'self'`,
`Referrer-Policy: strict-origin-when-cross-origin` and a `Permissions-Policy`
that denies camera, microphone and location.
- Every response carries these headers:
- `Strict-Transport-Security`, unless `HSTS_MAX_AGE` is `0`;
- `X-Content-Type-Options: nosniff`;
- `X-Frame-Options: DENY`;
- `Content-Security-Policy`, which tells the browser where scripts and
styles may come from, set to
`default-src 'self'; script-src 'self'; style-src 'self'`;
- `Referrer-Policy: strict-origin-when-cross-origin`;
- a `Permissions-Policy` that denies camera, microphone and location.
- Authenticated pages are sent with `Cache-Control: no-store`.
- `/metrics` sits behind HTTP basic authentication.
- `/metrics` sits behind HTTP basic authentication, a username and password sent
with each request.
### Sign-in limits
The sign-in form has no rate limiter in front of it, on purpose. Behind a proxy
that `TRUSTED_PROXIES` does not cover, every client shares one bucket. A limiter
spent on arrival would let any stranger lock the operator out with a few
requests a minute.
that `TRUSTED_PROXIES` does not cover, every client shares one bucket. A limit
that counted every request would let any stranger lock the operator out with a
few requests a minute.
Instead, the password is checked first, and only a failure is counted. A correct
password is never throttled. Failures are counted per client and submitted
username, 5 per minute, after which further failures are answered `429` with
`Retry-After`. A successful sign-in clears the count. At most 1024 username
username. From the fifth failure within a minute, each failure is answered `429`
with `Retry-After`. A successful sign-in clears the count. At most 1024 username
counters and 1024 client counters are kept.
Two password checks can run at once. Each takes 64 MB of memory, so this caps
hashing at 128 MB. A request that gets no slot within 5 seconds, or arrives when
16 requests are already waiting, is answered `503` without hashing anything. The
password change form shares these slots.
Two password checks can run at once; each running check holds one of two slots.
Each check takes 64 MB of memory, so hashing is capped at 128 MB. A request that
gets no slot within 5 seconds is answered `503` without hashing anything. So is
one that arrives when 16 requests are already waiting for a slot. The password
change form shares these slots.
Because the check runs before the count, what bounds online guessing is the two
slots, not the counter. A flood of wrong passwords is tried at the speed two
Argon2id checks allow. Choose a long random admin password. To put a cheaper
limit on top, rate-limit `POST /pages/login` at the reverse proxy, where the
real client address is visible.
The check runs before the count, so the two slots, not the failure count, bound
how fast passwords can be guessed. A flood of wrong passwords is tried as fast
as two Argon2id checks allow. Choose a long random admin password. To put a
cheaper limit on top, rate-limit `POST /pages/login` at the reverse proxy, where
the real client address is visible.
An unknown username is checked against a dummy hash, so it costs the same time
as a real one.
An unknown username is checked against a placeholder hash, so it costs the same
time as a real one.
A flood can hold both slots and make sign-in answer `503` for as long as it
runs. Nothing accumulates: the operator's correct password works on the first
@@ -611,10 +625,20 @@ it sits in and read the answers back. This is called server-side request
forgery. webhooker refuses it by default.
Every target URL must use `http` or `https` and resolve to a public address.
Private and reserved ranges are refused: loopback, the RFC 1918 ranges, carrier
NAT, link-local, multicast, documentation space, and the IPv6 unique local
range. One public address is refused too: `168.63.129.16`, the Azure service
that hands a VM its credentials.
These private and reserved ranges are refused:
- loopback, `127.0.0.0/8` and `::1`;
- the RFC 1918 private ranges, `10.0.0.0/8`, `172.16.0.0/12` and
`192.168.0.0/16`;
- carrier-grade NAT space, `100.64.0.0/10`, which internet providers use inside
their own networks;
- link-local space, `169.254.0.0/16` and `fe80::/10`, which is valid only on one
network segment;
- the IPv6 unique local range, `fc00::/7`;
- multicast, documentation space and the other reserved ranges.
One public address is refused too: `168.63.129.16`, the Azure service that hands
a VM its credentials.
The check runs when a target is saved and again when each delivery connects.
Both use the same function. The address is resolved again at connect time, so a
@@ -627,27 +651,31 @@ host, a different port, or a step from `https` to `http`. This stops an open
redirect from handing a credential to another host.
`ALLOWED_EGRESS_CIDRS` opens named networks for the common case of forwarding to
your own network. Read [`ALLOWED_EGRESS_CIDRS`](#allowed_egress_cidrs) before
setting it.
your own network. Read
[Allowing egress to your own network](#allowing-egress-to-your-own-network)
before setting it.
### What the logs contain
webhooker writes structured logs to standard output: plain text on a terminal,
JSON otherwise. Each request writes one `INFO` line.
JSON otherwise. Each request writes one access log line, `http request`, at
`INFO`. A request to a known entrypoint writes a second line,
`webhook request received`. A stored event writes a third,
`webhook event created`.
Fields a client can influence are cut to a fixed size, so one access log line is
at most 2,560 bytes. The query string is never logged. On the receiver route the
URL is logged as `/h/{uuid}`, never the real path, so entrypoint UUIDs stay out
of the log. On other routes a `3xx` or `4xx` response logs the route pattern,
and a `2xx` or `5xx` logs the real path. The `Referer` is logged as its scheme
and host only.
which is the route as registered, such as `/user/{username}`. A `2xx` or `5xx`
logs the real path. The `Referer` is logged as its scheme and host only.
Every line that names a client carries two fields. `remoteIP` is the connecting
peer, which behind a proxy is the proxy. `clientIP` is the client as decided by
`TRUSTED_PROXIES`.
SQL statements are logged with placeholders, never with the values bound to
them. Session cookies, API keys and target credentials are never logged.
SQL statements are logged with `?` in place of each value, never with the values
themselves. Session cookies, API keys and target credentials are never logged.
Two things are logged in the clear. The first is the `admin` password, once, on
the first boot, and again when `webhooker resetpw -generate` is run. The second
@@ -660,8 +688,8 @@ purpose.
A recovered panic writes one `ERROR` record of at most 10,240 bytes, with the
request id of the request that caused it.
When `SENTRY_DSN` is set, error reports go to Sentry with the request reduced to
this:
When `SENTRY_DSN` is set, error reports go to Sentry, an error reporting
service, with the request reduced to this:
- the URL with its path replaced by the route pattern;
- the query string and the body replaced by `(redacted)`;
@@ -684,14 +712,18 @@ says what range its variable accepts.
A Go duration is a number with a unit, such as `30s`, `15m`, `24h` or `1h30m`.
The startup log line `Configuration loaded` shows every value except the
secrets. The `/settings` page shows the same.
The startup log line `Configuration loaded` shows the loaded values, with three
differences. `TRUSTED_PROXIES` and `ALLOWED_EGRESS_CIDRS` appear as counts.
`SENTRY_DSN` and the metrics credentials appear as on or off. `METRICS_USERNAME`
does not appear. The `/settings` page shows every value; there,
`METRICS_PASSWORD` and `SENTRY_DSN` show only as set or not set.
### `WEBHOOKER_ENVIRONMENT`
`dev` or `prod`. Default `prod`.
This controls one thing: CORS. In `dev`, every origin is allowed with
This controls one thing: CORS, the headers that let a page on another site call
this one from a browser. In `dev`, every origin is allowed with
`Access-Control-Allow-Origin: *`, for local testing. In `prod`, no CORS headers
are sent. Cookie security and log format do not depend on it.
@@ -726,8 +758,8 @@ The directory for every SQLite database and the lock file. Default
Exactly one webhooker process may use a `DATA_DIR` at a time. Two processes on
one directory would both run delivery recovery and deliver every pending event
twice. At startup the process takes an exclusive `flock` on
`{DATA_DIR}/webhooker.lock` and holds it until it exits. A second process
twice. At startup the process locks `{DATA_DIR}/webhooker.lock` with `flock`,
the kernel's file lock, and holds the lock until it exits. A second process
pointed at the same directory prints a message naming it and exits non-zero.
The kernel releases the lock when the process exits, even on `kill -9`. A
@@ -743,15 +775,20 @@ host. It does not work across hosts on a network filesystem.
`strconv.ParseBool` accepts: `1`, `t`, `T`, `TRUE`, `true`, `True`, `0`, `f`,
`F`, `FALSE`, `false`, `False`. `yes` and `on` are refused.
`DEBUG=true` lowers the log level to `DEBUG`. That adds four kinds of line:
`DEBUG=true` lowers the log level to `DEBUG`. Lines at that level include:
- every SQL statement, with placeholders rather than values;
- every SQL statement, with `?` in place of each value;
- the lookup misses on the receiver and sign-in routes;
- the rate limiters' rejections;
- the records of building the dependency graph at startup.
- signed-out requests sent to the sign-in page;
- retries being scheduled, and retries skipped because the delivery had already
been handled;
- the receiver's whole-route rate limit refusing a request;
- a target deleted while the archive sweep was running;
- the records of how the components were wired at startup.
The output is safe to paste into a bug report once the first-boot password has
been changed.
Debug lines never carry SQL values, session cookies or target credentials. The
first-boot password is printed whatever the level, and so is every event a `log`
target writes.
### `METRICS_USERNAME` and `METRICS_PASSWORD`
@@ -805,8 +842,9 @@ configurable on its own.
### `TRUSTED_PROXIES`
A comma-separated list of networks in CIDR notation, such as
`192.168.1.7, 2001:db8::5`. A bare address counts as a single host. Default
A comma-separated list of networks in CIDR notation, which names a network as an
address and a prefix length, such as `10.0.0.0/8`. A bare address such as
`192.168.1.7` counts as a single host. Default
`10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`, the RFC 1918 private ranges. A set
value replaces the default.
@@ -834,21 +872,21 @@ throttled.
Rate-limit buckets are per IPv4 address and per IPv6 `/64` prefix.
### `ALLOWED_EGRESS_CIDRS`
### Allowing egress to your own network
A comma-separated list of networks in CIDR notation that delivery targets may
reach although the default rules refuse them. Default empty. A bare address
counts as a single host. Example: `10.0.0.0/8, 172.17.0.0/16`.
`ALLOWED_EGRESS_CIDRS` is a comma-separated list of networks in CIDR notation
that delivery targets may reach although the default rules refuse them. Default
empty. A bare address counts as a single host. Example:
`10.0.0.0/8, 172.17.0.0/16`.
This only ever adds destinations. The guard stays on, and nothing in the list
removes a destination that was already allowed. When the list is set, startup
logs it at `WARN`, blocks and all.
This only ever adds destinations. The default refusals stay in force for every
address not listed. When the list is set, startup logs the whole list at `WARN`.
Each network you list is one that anyone with an admin account can make this
process send requests into, and read the responses from in the delivery log.
List the smallest blocks that cover your destinations. Prefer `10.1.2.3/32` over
`10.0.0.0/8`. `0.0.0.0/0` or `::/0` opens every private and reserved range at
once, including this host's own loopback. Do not list them.
Anyone with the admin account can make webhooker send requests to every network
you list. They can read the responses in the delivery log. List the smallest
blocks that cover your destinations. Prefer `10.1.2.3/32` over `10.0.0.0/8`.
`0.0.0.0/0` or `::/0` opens every private and reserved range at once, including
this host's own loopback. Do not list them.
Some addresses stay blocked whatever you list. They are link-local space, the
unspecified addresses, and cloud metadata endpoints at fixed non-public
@@ -863,7 +901,8 @@ addresses that hand out credentials:
- `fd00:c1::a9fe:a9fe/128`, Oracle Cloud metadata over IPv6.
- `fd00:42::42/128`, Scaleway metadata over IPv6.
- `fd00:a9fe:a9fe::1/128`, Linode and Akamai metadata over IPv6.
- `100.100.100.200/32`, Alibaba Cloud metadata, inside the carrier NAT range.
- `100.100.100.200/32`, Alibaba Cloud metadata, inside the carrier-grade NAT
range.
- `192.0.0.192/32`, Oracle Cloud Classic metadata.
- `0.0.0.0/32` and `::/128`, the unspecified addresses, which reach this host's
loopback on Linux.
@@ -885,10 +924,17 @@ default but is reopened by a list that covers it.
An `http` target posts the event body to its URL. The request carries the
event's `Content-Type`, the event's headers, then the target's configured
headers, then `User-Agent: webhooker/{version}`. A configured header replaces an
event header of the same name. These event headers are never forwarded: `Host`,
`Connection`, `Keep-Alive`, `Transfer-Encoding`, `Te`, `Trailer`, `Upgrade`,
`Proxy-Authorization`, `Proxy-Connection` and `Content-Length`. A `2xx` response
is a success. The first 4 KiB of the response body is stored with the attempt.
event header of the same name. A `2xx` response is a success. The first 4 KiB of
the response body is stored with the attempt.
These event headers are never forwarded:
- `Host`
- `Connection` and `Keep-Alive`
- `Transfer-Encoding` and `Content-Length`
- `Te` and `Trailer`
- `Upgrade`
- `Proxy-Authorization` and `Proxy-Connection`
An `http` target's settings:
@@ -905,8 +951,8 @@ An `http` target's settings:
A `slack` target posts a JSON message with one `text` field to its URL. The
message names the method, content type, time received and body size, then shows
the body in a code block. A JSON body is indented. A body over 3500 characters
is cut. The URL is any Slack-compatible incoming webhook URL, which includes
the body in a code block. A JSON body is indented. A body over 3500 bytes is
cut. The URL is any Slack-compatible incoming webhook URL, which includes
Mattermost.
A `database` target, called an archive in the web interface, appends the event
@@ -1002,16 +1048,19 @@ An archive target writes to
names are made safe for a file name:
- lowercased;
- letters and digits kept, and every other run of characters turned into one
`-`;
- only the ASCII letters `a` to `z` and the digits `0` to `9` kept, and every
other run of characters turned into one `-`;
- cut to 40 characters;
- `unnamed` when nothing is left.
A webhook named `Orders (EU)` with a target named `Long-term archive` writes
`archive-orders-eu-long-term-archive-{target_uuid}.db`.
Each row holds the event's id, webhook id, entrypoint id, method, query string,
headers, body, content type and the time it was archived.
Each row holds:
- the event's id, and the ids of its webhook and entrypoint;
- the method, query string, headers, body and content type;
- the time the row was archived.
`archive_rotation` is `none` by default, which keeps one file. `monthly`,
`daily` and `hourly` start a new file for each period of the event's receive
@@ -1093,21 +1142,28 @@ keeps the retention cutoff inside the range of a Go duration.
`Entrypoint` has `webhook_id`, `path`, `description` and `active`. `path` is the
bare UUID. The `/h/` prefix is route only and is not stored.
`Target` has `webhook_id`, `name`, `type`, `active`, `url`, `headers`,
`timeout_seconds`, `forward_query`, `archive_expiry`, `archive_rotation` and
`max_attempts`. `type` is one of `http`, `slack`, `database` or `log`. Each type
fills only its own settings. `archive_expiry` is stored in nanoseconds, `0` for
never.
`Target` has these fields. Each type fills only its own settings.
`APIKey` has `user_id`, `key`, `description` and `last_used_at`. The table
exists, but nothing creates a key and nothing checks one.
- `webhook_id`, `name`, `type` and `active`. `type` is one of `http`, `slack`,
`database` or `log`.
- `url`, `headers`, `timeout_seconds` and `forward_query`, the `http` settings.
A `slack` target uses `url` alone.
- `archive_expiry` and `archive_rotation`, the `database` settings.
`archive_expiry` is stored in nanoseconds, `0` for never.
- `max_attempts`, for `http` and `slack`.
`APIKey` has `user_id`, `key`, `description` and `last_used_at`. Its table,
`api_keys`, exists, but nothing creates a key and nothing checks one.
### Event tier
`Event` has `webhook_id`, `entrypoint_id`, `method`, `raw_query`, `headers` (as
JSON), `body`, `content_type`, `body_bytes` and `resubmitted_from_id`. `method`
is always `POST`. `resubmitted_from_id` names the event a resubmit copied; it is
empty for an event that arrived on the receiver.
`Event` has these fields:
- `webhook_id` and `entrypoint_id`;
- `method`, which is always `POST`;
- `raw_query`, `headers` (as JSON), `body`, `body_bytes` and `content_type`;
- `resubmitted_from_id`, the event a resubmit copied. It is empty for an event
that arrived on the receiver.
`Delivery` has `event_id`, `target_id`, `status`, `finished_at` and `replay`.
`finished_at` is set when the status becomes `delivered` or `failed`.
@@ -1163,12 +1219,16 @@ Routes behind the sign-in. A `GET` without a session is redirected to
`POST /hook/{id}/events/{eventID}/resubmit` resubmits an event, each 30 per
minute per client.
- `POST /hook/{id}/entrypoints` adds an entrypoint.
`POST /hook/{id}/entrypoints/{entrypointID}/edit`, `/delete` and `/toggle`
edit, delete, and activate or deactivate one.
- `POST /hook/{id}/targets` adds a target. `GET` and
`POST /hook/{id}/targets/{targetID}/edit` edit one. `POST .../delete` and
`POST .../toggle` delete, and activate or deactivate one.
`GET /hook/{id}/targets/{targetID}/download` downloads an archive.
- `POST /hook/{id}/entrypoints/{entrypointID}/edit` changes its description.
- `POST /hook/{id}/entrypoints/{entrypointID}/delete` deletes it.
- `POST /hook/{id}/entrypoints/{entrypointID}/toggle` activates or deactivates
it.
- `POST /hook/{id}/targets` adds a target.
- `GET /hook/{id}/targets/{targetID}/edit` shows the target edit form, and
`POST` to the same path saves it.
- `POST /hook/{id}/targets/{targetID}/delete` deletes the target.
- `POST /hook/{id}/targets/{targetID}/toggle` activates or deactivates it.
- `GET /hook/{id}/targets/{targetID}/download` downloads an archive.
`GET /metrics` exists only when `METRICS_USERNAME` and `METRICS_PASSWORD` are
set, behind basic authentication.
@@ -1180,10 +1240,10 @@ set, behind basic authentication.
`/metrics` serves one Prometheus registry. It carries the Go runtime and process
collectors, the delivery metrics below, and three inbound HTTP metrics.
Every delivery metric carries one label, `target_type`, with one of `http`,
`slack`, `database`, `log` or `unknown`. Target ids and event ids are never
labels, because a label value that grows with traffic would grow the registry
without bound.
Nine of the eleven metrics below carry one label, `target_type`, with one of
`http`, `slack`, `database`, `log` or `unknown`. The two event counters carry
none. Target ids and event ids are never labels, because a label value that
grows with traffic would grow the registry without bound.
- `webhooker_events_received_total` (counter, no label): events stored,
resubmitted copies included.
@@ -1204,7 +1264,8 @@ without bound.
Deliveries to a deleted target count under `unknown`.
- `webhooker_circuit_breakers_open` (gauge): breakers currently open.
The outcome counters move only after the status change has been written.
The success and failure counters move only after the new status has been
written.
The inbound HTTP metrics are `http_request_duration_seconds`,
`http_response_size_bytes` and `http_requests_inflight`. Their `handler` label
@@ -1217,17 +1278,22 @@ for the whole service, with `handler` set to `(all)`.
### Prerequisites
- The Go version named in `go.mod`.
- Docker. The tests, the linters, the formatters and the stylesheet build all
run in Docker.
- Docker. The tests, the linters, prettier and the stylesheet build run in
Docker.
Nothing else is installed on the host. `golangci-lint`, ESLint, prettier, node,
yarn and the Tailwind CLI run only inside Docker, at the versions pinned in the
`Dockerfile`, `package.json` and `yarn.lock`.
`gofmt` and `goimports` run on the host. These tools run only inside Docker, at
the versions pinned in the `Dockerfile`, `package.json` and `yarn.lock`:
- `golangci-lint`;
- ESLint;
- prettier;
- node and yarn;
- the Tailwind CLI.
### Commands
```bash
make bootstrap # Install git, make, Go and the Go modules (idempotent)
make bootstrap # Install git, make, Go and the Go modules (safe to rerun)
make setup # bootstrap, then install the git pre-commit hook
make assets # Extract Alpine.js from 3p/ to static/js/alpine.min.js
make fmt # Format Go (gofmt, goimports) and Markdown (prettier, in Docker)
@@ -1235,7 +1301,7 @@ make fmt-check # Fail if gofmt or prettier would change anything
make lint # Run gofmt, golangci-lint and ESLint in Docker
make test # Run the tests in Docker, with cgo off
make test-browser # Run the browser test in Docker (Dockerfile.browser)
make check # test, lint, fmt-check and css-check: the CI gate
make check # test, lint, fmt-check and css-check: what CI runs
make build # Build bin/webhooker, version-stamped
make version # Print the version this checkout would stamp
make run # build, then run ./bin/webhooker
@@ -1244,19 +1310,21 @@ make deps # go mod download, then go mod tidy
make docker # Build the Docker image
make hooks # Install the git pre-commit hook
make css # Regenerate static/css/tailwind.css in Docker
make css-check # Fail if static/css/tailwind.css is stale
make css-check # Fail if static/css/tailwind.css is out of date
make clean # Remove bin/
```
The repository follows
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all).
Each `make` target above, except `build`, `run`, `dev`, `deps`, `version` and
`clean`, is a thin shim over a script of the same name in `script/`.
`script/cibuild` is the CI entrypoint: it runs `script/bootstrap`, then
`script/check`, then builds the image. `script/precommit` checks that
`go mod tidy` changes nothing, then runs `script/check`.
`clean`, is a thin shim over a script in `script/`. `make hooks` calls
`script/install-precommit`; every other shim calls the script of the same name.
`script/cibuild` is what CI runs: `script/bootstrap`, then `script/check`, then
the image build. `script/precommit` checks that `go mod tidy` changes nothing,
then runs `script/check`.
The tests run with cgo off and so without Go's race detector.
The tests run with cgo, Go's bridge to C code, turned off. Go's race detector
needs cgo, so the tests run without it.
### How the checks run
@@ -1311,15 +1379,15 @@ builds of one commit is stamped in.
### Third-party browser assets
The pages use one third-party script, Alpine.js, in its CSP build
(`@alpinejs/csp`). The Content-Security-Policy forbids `eval`, which the
standard build needs. With the CSP build, every Alpine directive in `templates/`
names a component registered in `static/js/app.js`, such as
The pages use one third-party script, Alpine.js, a small JavaScript library, in
its CSP build (`@alpinejs/csp`). The Content-Security-Policy forbids `eval`,
which the standard build needs. With the CSP build, every Alpine directive in
`templates/` names a component registered in `static/js/app.js`, such as
`x-data="collapsible"`, and never an inline expression.
The npm tarball is committed as `3p/alpinejs-csp-3.14.9.tgz`, byte for byte as
the registry publishes it. `script/assets` extracts `package/dist/cdn.min.js`
from it to `static/js/alpine.min.js`, which `go:embed` picks up. `make build`,
from it to `static/js/alpine.min.js`, which the binary embeds. `make build`,
`make dev` and the Docker stages run it first. The extracted file is not
committed, and a build that skips the extraction fails with an error naming
`js/alpine.min.js`.
@@ -1334,11 +1402,11 @@ To move to a new version:
### Browser test
`internal/server` has a test that loads the pages in a headless browser under
the real Content-Security-Policy and drives the interface. It is built only with
the `browser` build tag, so `make test` leaves it out. `make test-browser` runs
it in Docker with a pinned headless browser image. Run it after changing
`templates/` or `static/js/`.
`internal/server` has a test that loads the pages in a headless browser, one
without a window, under the real Content-Security-Policy and drives the
interface. It is built only with the `browser` build tag, so `make test` leaves
it out. `make test-browser` runs it in Docker with a pinned headless browser
image. Run it after changing `templates/` or `static/js/`.
### Stylesheet
@@ -1416,12 +1484,27 @@ that order and their stop hooks in reverse. The components are:
- the delivery engine, the archive writers and the archive sweep;
- the server.
The global middleware runs in this order: request id, security headers, access
log, metrics (when `/metrics` is served), CORS, a 60 second request timeout,
panic recovery, and Sentry when configured. Each page route group adds, in
order, its own panic recovery, the 1 MB body cap, CSRF,
`Cache-Control: no-store`, and the sign-in check. The body cap comes before CSRF
because CSRF parses the form.
The global middleware runs in this order:
1. request id;
2. security headers;
3. access log;
4. metrics, when `/metrics` is served;
5. CORS;
6. a 60 second request timeout;
7. panic recovery;
8. Sentry, when configured.
Each page route group then adds its own, in this order:
1. panic recovery;
2. the 1 MB body cap;
3. CSRF;
4. `Cache-Control: no-store`;
5. the sign-in check, in every group except `/pages`, which serves the sign-in
form.
The body cap comes before CSRF because CSRF parses the form.
## License