1 Commits
Author SHA1 Message Date
sneak 5471054adb Rewrite README.md sentence by sentence
check / check (push) Canceled after 0s
Rewritten from scratch against the code on next. The old text was
the list of what a reader needs covered.

Model: fable-5-1
2026-10-25 06:43:00 +00:00
+175 -316
View File
@@ -48,9 +48,9 @@ The part of webhooker that answers requests to `/h/{uuid}` is the receiver.
A request to an entrypoint is stored as an event. One pending delivery is A request to an entrypoint is stored as an event. One pending delivery is
created for each active target of the webhook, in the same database write as the created for each active target of the webhook, in the same database write as the
event: either all of them are stored or none. The sender gets `200` once they event: either all of them are stored or none. The sender gets `200` once they
are stored. Workers then carry out the deliveries. A delivery to an HTTP or are stored. Background workers, the part of webhooker that sends deliveries,
Slack target that fails is retried, up to the number of attempts the target then carry them out. A delivery to an HTTP or Slack target that fails is
allows. retried, up to the number of attempts the target allows.
Delivery is at least once, not exactly once. If a delivery reaches the target Delivery is at least once, not exactly once. If a delivery reaches the target
but webhooker cannot record that outcome, it sends the delivery again later. A but webhooker cannot record that outcome, it sends the delivery again later. A
@@ -123,7 +123,9 @@ keeps events forever.
The form can also create the first targets. An HTTP target URL creates an `http` The form can also create the first targets. An HTTP target URL creates an `http`
target named `HTTP`. The **Archive** checkbox creates an archive target named target named `HTTP`. The **Archive** checkbox creates an archive target named
`Archive`, with the expiry and rotation chosen beside it. `Archive`. Beside it you choose the archive's expiry, which is how long archived
events are kept, and its rotation, which is how often a new archive file is
started.
Every new webhook gets one entrypoint, described as `Default entrypoint`. Every new webhook gets one entrypoint, described as `Default entrypoint`.
@@ -137,8 +139,8 @@ like a password. Read
**Add** creates another entrypoint with its own URL. Several entrypoints on one **Add** creates another entrypoint with its own URL. Several entrypoints on one
webhook let you tell senders apart while delivering to the same targets. webhook let you tell senders apart while delivering to the same targets.
**Edit** changes an entrypoint's description. Its URL never changes. To retire a **Edit** changes an entrypoint's description. Its URL never changes. To replace
URL, deactivate the entrypoint or delete it, then create a new one. a URL, deactivate the entrypoint or delete it, then create a new one.
A deactivated entrypoint answers `410 Gone`. A deleted or unknown one answers A deactivated entrypoint answers `410 Gone`. A deleted or unknown one answers
`404`. `404`.
@@ -194,8 +196,13 @@ retention. Next are figures for the last 10 minutes and the last 24 hours:
events, failures and the failure percentage. The percentage is failed deliveries events, failures and the failure percentage. The percentage is failed deliveries
out of all deliveries that finished in the window. out of all deliveries that finished in the window.
The webhook list shows three of these figures per webhook: events within The webhook list shows, for each webhook:
retention, the last event, and failed deliveries in the last 24 hours.
- its entrypoint and target counts, with how many are inactive;
- its retention setting;
- its events within retention;
- its last event;
- its failed deliveries in the last 24 hours.
### Recent events and the event log ### Recent events and the event log
@@ -206,8 +213,13 @@ body** link to the stored bytes.
**Full Event Log** at `/hook/{id}/events` shows the 50 newest events with their **Full Event Log** at `/hook/{id}/events` shows the 50 newest events with their
deliveries and every attempt. **Failed** narrows it to events with a failed deliveries and every attempt. **Failed** narrows it to events with a failed
delivery. **Pending** narrows it to events with a delivery still pending or delivery. **Pending** narrows it to events with a delivery still pending or
retrying. Each attempt shows its time, status code, duration, error, and the retrying. Each attempt shows:
first 4 KiB of the response.
- when it ran;
- its status code;
- its duration;
- its error, if any;
- the first 4 KiB of the response.
Each event has a page of its own at `/hook/{id}/events/{eventID}`. It shows the Each event has a page of its own at `/hook/{id}/events/{eventID}`. It shows the
whole request: query string, headers, body and every delivery. whole request: query string, headers, body and every delivery.
@@ -262,10 +274,11 @@ docker run -d \
The image listens on every IPv4 address inside the container: it ships The image listens on every IPv4 address inside the container: it ships
`BIND_ADDRESS=0.0.0.0`. A container has its own network namespace, which means `BIND_ADDRESS=0.0.0.0`. A container has its own network namespace, which means
its own addresses, separate from the host's. That is the boundary, and the `-p` its own addresses, separate from the host's. The namespace is what keeps the
flag decides what the host exposes. Publish to `127.0.0.1:8080`, the address container's ports apart from the host's, and the `-p` flag decides what the host
your reverse proxy connects to, and nothing wider. A bare `-p 8080:8080` exposes. Publish to `127.0.0.1:8080`, the address your reverse proxy connects
publishes the port on every interface of the host. to, and nothing wider. A bare `-p 8080:8080` publishes the port on every
interface of the host.
Do not set `BIND_ADDRESS` to an empty value in a container. An empty value means Do not set `BIND_ADDRESS` to an empty value in a container. An empty value means
unset, and unset falls back to the binary's default of `127.0.0.1`. The unset, and unset falls back to the binary's default of `127.0.0.1`. The
@@ -344,17 +357,19 @@ Five things must be right.
`127.0.0.1` only. The plain port serves the sign-in form and the receiver `127.0.0.1` only. The plain port serves the sign-in form and the receiver
without TLS. without TLS.
2. **Send `X-Forwarded-Proto: https`.** This header tells webhooker the request 2. **Send `X-Forwarded-Proto: https`.** This header tells webhooker the request
arrived over TLS. It decides the `Secure` flag on cookies and the strict arrived over TLS. It decides the `Secure` flag on cookies, and whether a form
origin check on forms. Without it, cookies are sent without `Secure`. submission must carry a matching `Origin` or `Referer` (see
webhooker reads this header from any peer, so the proxy must overwrite [The web interface](#the-web-interface) under Security). Without it, cookies
whatever the client sent. are sent without `Secure`. webhooker reads this header from any connecting
machine, so the proxy must overwrite whatever the client sent.
3. **Cover the proxy in `TRUSTED_PROXIES`.** Otherwise every client shares one 3. **Cover the proxy in `TRUSTED_PROXIES`.** Otherwise every client shares one
rate-limit bucket. A bucket is the count a limit keeps per client, and here rate-limit bucket. A bucket is the count a limit keeps per client, and here
it would be keyed on the proxy's address. The default list is the private it would be keyed on the proxy's address. The default list is the private
ranges, which does not cover a proxy that connects over loopback. ranges, which does not cover a proxy that connects over loopback.
4. **Send `Host` as `$http_host`, not `$host`.** `$host` drops the port. The 4. **Send `Host` as `$http_host`, not `$host`.** webhooker checks a form's
form origin check compares against the host it was given. On any port other `Origin` against the `Host` header it receives. `$host` drops the port, so on
than 443 every form submission then fails with `403`. any port other than 443 the two differ and every form submission fails with
`403`.
5. **Keep the proxy's access log.** It records the client address whatever 5. **Keep the proxy's access log.** It records the client address whatever
`TRUSTED_PROXIES` says. It also records the full request path, which holds `TRUSTED_PROXIES` says. It also records the full request path, which holds
the entrypoint UUID of every event. Protect that log as you would the the entrypoint UUID of every event. Protect that log as you would the
@@ -476,21 +491,24 @@ takes effect at the next sign-in. Sessions already signed in are unaffected.
All state lives in `DATA_DIR`. Back up the whole directory. It holds: All state lives in `DATA_DIR`. Back up the whole directory. It holds:
- `webhooker.db`, one per install. Settings, the session key, users, webhooks, - `webhooker.db`, the main database, one per install;
entrypoints and targets. - `events-{webhook_uuid}.db`, one event database per webhook;
- `events-{webhook_uuid}.db`, one per webhook. Events, deliveries and delivery
attempts.
- `archive-{webhook_name}-{target_name}-{target_uuid}.db`, one per archive - `archive-{webhook_name}-{target_name}-{target_uuid}.db`, one per archive
target, or one per period for a target that rotates. The period goes before target, or one per period for a target that rotates, with the period before
`.db`. `.db`;
- `webhooker.lock`, which is always empty and holds no state. - `webhooker.lock`, which is always empty and holds no state.
Every database runs in WAL mode. WAL is SQLite's write-ahead log: committed [Data files](#data-files) says what each database holds.
changes land in a `-wal` file beside the database before they reach the `.db`
Every database runs in WAL mode. WAL is SQLite's write-ahead log: finished
writes land in a `-wal` file beside the database before they reach the `.db`
itself. So a running service has two more files beside each `{name}.db`: itself. So a running service has two more files beside each `{name}.db`:
`{name}.db-wal`, the log itself, and `{name}.db-shm`, the shared index SQLite
keeps for that log. The `-wal` file is part of the database. A copy of the `.db` - `{name}.db-wal`, the log itself;
without it is missing data and may have no readable tables at all. - `{name}.db-shm`, the shared index SQLite keeps for that log.
The `-wal` file is part of the database. A copy of the `.db` without it is
missing data and may have no readable tables at all.
A clean shutdown closes every database and removes its `-wal` and `-shm` files. A clean shutdown closes every database and removes its `-wal` and `-shm` files.
A crash leaves them, and they must be carried with the `.db`. A crash leaves them, and they must be carried with the `.db`.
@@ -526,8 +544,9 @@ done
`.backup` writes one complete file, with no `-wal` beside it. The runtime image `.backup` writes one complete file, with no `-wal` beside it. The runtime image
has no `sqlite3`, so run this on the host or from another container with the has no `sqlite3`, so run this on the host or from another container with the
volume mounted. Each file is captured at its own instant, so an event stored volume mounted. Each file is captured at its own instant. So `webhooker.db` and
between two copies lands in one file and not the other. an event database copied at different moments can disagree: a webhook created
between the two copies is in one and has no event database in the other.
Archive files are the exception. The archive writer closes and reopens its file Archive files are the exception. The archive writer closes and reopens its file
after a write, at most once per second. So you can move a finished archive file after a write, at most once per second. So you can move a finished archive file
@@ -553,7 +572,7 @@ who can read it.
- The event and archive databases hold the full headers and body of every event, - The event and archive databases hold the full headers and body of every event,
including whatever the sender put in them. including whatever the sender put in them.
- `webhooker.db` holds target settings unencrypted, next to the session key and - `webhooker.db` holds target settings unencrypted, next to the session key and
the password hashes. the password hash.
## Security ## Security
@@ -568,10 +587,9 @@ stored and forwarded as ordinary headers.
What that means in practice: What that means in practice:
- Keep the URL out of tickets, chat and screenshots. webhooker's own log never - Keep the URL out of tickets, chat and screenshots. webhooker's own log never
writes it, except in the line a `log` target writes. The reverse proxy's writes it. The reverse proxy's access log does hold it, in the request path.
access log does hold it, in the request path. - To replace a URL, create a new entrypoint and point the sender at it. Then
- To rotate, create a new entrypoint and point the sender at it. Then deactivate deactivate or delete the old one.
or delete the old one.
- Some senders cannot be given a secret URL: they post to one fixed address and - Some senders cannot be given a secret URL: they post to one fixed address and
prove themselves with a signature instead. webhooker checks no signatures, so prove themselves with a signature instead. webhooker checks no signatures, so
such a sender cannot use it. Pick another integration path for it. such a sender cannot use it. Pick another integration path for it.
@@ -677,11 +695,12 @@ Redirects are followed up to ten hops. Each hop is checked the same way. A hop
that leaves the target's origin carries none of the target's configured headers that leaves the target's origin carries none of the target's configured headers
and none of the forwarded event headers. Leaving the origin means a different and none of the forwarded event headers. Leaving the origin means a different
host, a different port, or a step from `https` to `http`. An open redirect is a host, a different port, or a step from `https` to `http`. An open redirect is a
destination that forwards requests to any URL a caller names. This rule stops destination that forwards requests to any URL a caller names. Because of this
one from handing a credential to another host. rule, an open redirect cannot pass those headers to another host.
`ALLOWED_EGRESS_CIDRS` opens named networks for the common case of forwarding to Egress means the connections webhooker opens outward to deliver.
your own network. Read `ALLOWED_EGRESS_CIDRS` opens named networks to it, for the common case of
forwarding to your own network. Read
[Allowing egress to your own network](#allowing-egress-to-your-own-network) [Allowing egress to your own network](#allowing-egress-to-your-own-network)
before setting it. before setting it.
@@ -702,8 +721,8 @@ logs the route pattern, which is the route as registered, such as
as its scheme and host only. as its scheme and host only.
Every line that names a client carries two fields. `remoteIP` is the connecting Every line that names a client carries two fields. `remoteIP` is the connecting
peer, which behind a proxy is the proxy. `clientIP` is the client as decided by machine, which behind a proxy is the proxy. `clientIP` is the client as decided
`TRUSTED_PROXIES`. by `TRUSTED_PROXIES`.
SQL statements are logged with `?` in place of each value, never with the values SQL statements are logged with `?` in place of each value, never with the values
themselves. Outside a `log` target's line, no line carries session cookies or themselves. Outside a `log` target's line, no line carries session cookies or
@@ -731,64 +750,31 @@ service, with the request reduced to this:
## Design ## Design
### Request flow webhooker is one process. An HTTP server receives requests, a delivery engine
sends them on, and SQLite holds everything in between.
One request to an entrypoint goes through these steps: The HTTP server runs every request through middleware, code that runs around
each request. The middleware writes the access log, sets the security headers,
applies a 60 second timeout that answers `504`, and recovers from a crash inside
the request. On the page routes it also caps the body at 1 MB, applies the CSRF
protection and checks the sign-in. On the receiver route it applies the two rate
limits. The handler for the route then does the work.
1. The rate limits run: one per client per entrypoint, one per client across the The receiver's handler stores the event, one pending delivery per active target,
whole route. and the running totals in one database write. It then queues one task per
2. The handler answers any method but `POST` with `405`. delivery. The delivery engine runs ten background workers that take tasks from
3. The handler looks up the entrypoint. Unknown gets `404`, deactivated `410`. two queues, one for new deliveries and one for retries, and run the target's
4. The handler reads the body, up to 1 MB. A larger body gets `413`. delivery. Each attempt is recorded, and the delivery's status is updated. A
5. The event, one pending delivery per active target, and the running totals are recovery routine re-queues deliveries the engine lost track of.
stored together, or not at all. The sender gets `200`.
6. One task per delivery goes onto the delivery queue.
7. A worker takes the task and runs the target's delivery. The outcome is
recorded as an attempt, and the delivery's status is updated.
### Components and middleware The main database holds configuration. Each webhook has its own event database,
so a busy webhook does not slow the others. Archive targets write to files of
their own. [Data files](#data-files) names each file.
The components are wired by Uber fx, a library that builds each component when The components are wired by [Uber fx](https://go.uber.org/fx), a library that
another needs it. fx starts the components in the order their dependencies need, builds each component when another needs it and runs their start and stop hooks.
and stops them in reverse. The components are: The other libraries are:
- the globals, the logger and the config;
- the main database and the per-webhook database manager;
- the retention sweep and the health check;
- the session and the handlers;
- the metrics registry and collectors;
- the middleware and the SSRF guard;
- the delivery engine, the archive writers and the archive sweep;
- the server.
Middleware is code that runs around every request. The global middleware runs in
this order:
1. request id;
2. security headers;
3. access log;
4. metrics, when `/metrics` is served;
5. CORS;
6. a 60 second request timeout, which answers `504` when it fires;
7. panic recovery;
8. Sentry, when configured.
A page route group is a set of routes under one prefix, such as `/hook/`. Each
page route group then adds its own middleware, in this order:
1. panic recovery;
2. Sentry, when configured;
3. the 1 MB body cap;
4. CSRF;
5. `Cache-Control: no-store`;
6. the sign-in check, in every group except `/pages`, which serves the sign-in
form.
The body cap comes before CSRF because CSRF parses the form.
### Libraries
- [Uber fx](https://go.uber.org/fx) wires the components, as above.
- [chi](https://github.com/go-chi/chi), an HTTP router, matches requests to - [chi](https://github.com/go-chi/chi), an HTTP router, matches requests to
handlers. handlers.
- [GORM](https://gorm.io), a Go database library, with - [GORM](https://gorm.io), a Go database library, with
@@ -804,38 +790,9 @@ The body cap comes before CSRF because CSRF parses the form.
[go-http-metrics](https://github.com/slok/go-http-metrics) serve the metrics. [go-http-metrics](https://github.com/slok/go-http-metrics) serve the metrics.
- The [Sentry](https://sentry.io) library reports errors when configured. - The [Sentry](https://sentry.io) library reports errors when configured.
### Package layout Most of the Go code is under `internal/`, one package per concern. The program
starts in `cmd/webhooker/main.go`. `templates/` holds the pages' HTML, and
Most of the Go code is under `internal/`. The program starts in `static/` their CSS and JavaScript, both embedded in the binary.
`cmd/webhooker/main.go`, which dispatches subcommands, takes the `DATA_DIR` lock
and wires the components.
- `internal/archive`: archive files, rotation, pruning, renaming and download.
- `internal/banner`: the ruled box that prints a credential.
- `internal/config`: configuration from the environment and `.env`.
- `internal/database`: the main database, the models, Argon2id, the retention
sweep and the per-webhook database manager.
- `internal/datadir`: the exclusive lock on `DATA_DIR`.
- `internal/db/migrations`: the SQL schema files and their application.
- `internal/delivery`: the engine, the four target types, retries, the circuit
breaker and the SSRF guard.
- `internal/globals`: build-time variables.
- `internal/gormlog`: GORM's logger on top of `slog`, with values withheld.
- `internal/handlers`: every HTTP handler and page.
- `internal/healthcheck`: the health check response.
- `internal/lifecycle`: the bounded wait used by stop hooks.
- `internal/logfield`: the size budget for client-supplied log values.
- `internal/logger`: `slog` setup and the fx event logger.
- `internal/metrics`: the delivery metrics.
- `internal/middleware`: logging, security headers, CORS, sessions, CSRF, rate
limits, body caps, metrics and panic recovery.
- `internal/referer`: cuts a `Referer` to its scheme and host.
- `internal/reqtls`: the one function that decides whether a request is TLS.
- `internal/resetpw`: the `webhooker resetpw` subcommand.
- `internal/server`: the HTTP server, routes, middleware order and Sentry.
- `internal/session`: the cookie session.
- `internal/versionscript`: tests for `script/version` and the build files.
- `static/`: embedded CSS and JavaScript. `templates/`: the HTML templates.
## Configuration reference ## Configuration reference
@@ -862,8 +819,7 @@ on-or-off field. The `/settings` page shows every value; there,
`dev` or `prod`. Default `prod`. `dev` or `prod`. Default `prod`.
This controls one thing: CORS, the headers that let a page on another site call This controls one thing: CORS. In `dev`, every origin is allowed with
this one from a browser. In `dev`, every origin is allowed with
`Access-Control-Allow-Origin: *`, for local testing. In `prod`, no CORS headers `Access-Control-Allow-Origin: *`, for local testing. In `prod`, no CORS headers
are sent. Cookie security and log format do not depend on it. are sent. Cookie security and log format do not depend on it.
@@ -897,10 +853,11 @@ The directory for every SQLite database and the lock file. Default
`/var/lib/webhooker`. It is created if it is missing. `/var/lib/webhooker`. It is created if it is missing.
Exactly one webhooker process may use a `DATA_DIR` at a time. Two processes on Exactly one webhooker process may use a `DATA_DIR` at a time. Two processes on
one directory would both run delivery recovery and deliver every pending event one directory would both run the recovery routine and deliver every pending
twice. At startup the process locks `{DATA_DIR}/webhooker.lock` with `flock`, event twice. At startup the process locks `{DATA_DIR}/webhooker.lock` with
the kernel's file lock, and holds the lock until it exits. A second process `flock`, the kernel's file lock, and holds the lock until it exits. A second
pointed at the same directory prints a message naming it and exits non-zero. process pointed at the same directory prints a message naming it and exits
non-zero.
The kernel releases the lock when the process exits, even on `kill -9`. A The kernel releases the lock when the process exits, even on `kill -9`. A
leftover `webhooker.lock` blocks nothing and must not be deleted by hand. To run leftover `webhooker.lock` blocks nothing and must not be deleted by hand. To run
@@ -911,9 +868,9 @@ host. It does not work across hosts on a network filesystem.
### `DEBUG` ### `DEBUG`
`true` or `false`. Default `false`. It accepts the spellings Go's `true` or `false`. Default `false`. It accepts the twelve spellings Go's
`strconv.ParseBool` accepts: `1`, `t`, `T`, `TRUE`, `true`, `True`, `0`, `f`, `strconv.ParseBool` accepts, such as `true`, `1` and `false`. `yes` and `on` are
`F`, `FALSE`, `false`, `False`. `yes` and `on` are refused. refused.
`DEBUG=true` lowers the log level to `DEBUG`. Lines at that level include: `DEBUG=true` lowers the log level to `DEBUG`. Lines at that level include:
@@ -991,15 +948,15 @@ value replaces the default.
The list decides whose `X-Forwarded-For` header is believed. `X-Forwarded-For` The list decides whose `X-Forwarded-For` header is believed. `X-Forwarded-For`
is the header a proxy uses to pass on the address of the client it is serving. is the header a proxy uses to pass on the address of the client it is serving.
When the connecting peer is inside the list, the client address is read from When the connecting machine is inside the list, the client address is read from
that header. For any other peer, the client is the peer itself and the header is that header. For any other connecting machine, the client is that machine itself
ignored. `X-Real-IP` and `True-Client-IP` are never read. and the header is ignored. `X-Real-IP` and `True-Client-IP` are never read.
The header is read from right to left. The first entry that is not itself a The header is read from right to left. The first entry that is not itself a
trusted proxy is the client. An entry that is not a bare IP address ends the trusted proxy is the client. An entry that is not a bare IP address ends the
walk, and the peer address is used instead. At most 64 entries are examined. walk, and the connecting machine's address is used instead. At most 64 entries
Your proxy must therefore append the peer address to `X-Forwarded-For`, as are examined. Your proxy must therefore append the connecting address to
nginx's `$proxy_add_x_forwarded_for` does. `X-Forwarded-For`, as nginx's `$proxy_add_x_forwarded_for` does.
If any client can reach webhooker, or the proxy in front of it, from a private If any client can reach webhooker, or the proxy in front of it, from a private
address, set the list to the proxy's address alone. Otherwise those clients can address, set the list to the proxy's address alone. Otherwise those clients can
@@ -1016,6 +973,7 @@ Rate-limit buckets are per IPv4 address and per IPv6 `/64` prefix.
### Allowing egress to your own network ### Allowing egress to your own network
Egress means the connections webhooker opens outward to deliver.
`ALLOWED_EGRESS_CIDRS` is a comma-separated list of networks in CIDR notation `ALLOWED_EGRESS_CIDRS` is a comma-separated list of networks in CIDR notation
that delivery targets may reach although the default rules refuse them. Default that delivery targets may reach although the default rules refuse them. Default
empty. A bare address counts as a single host. Example: empty. A bare address counts as a single host. Example:
@@ -1064,11 +1022,17 @@ default but is reopened by a list that covers it.
### Target types ### Target types
An `http` target posts the event body to its URL. The request carries the An `http` target posts the event body to its URL. The request carries, in this
event's `Content-Type`, the event's headers, then the target's configured order:
headers, then `User-Agent: webhooker/{version}`. A configured header replaces an
event header of the same name. A `2xx` response is a success. The first 4 KiB of 1. the event's `Content-Type`;
the response body is stored with the attempt. 2. the event's headers;
3. the target's configured headers, each replacing an event header of the same
name;
4. `User-Agent: webhooker/{version}`.
A `2xx` response is a success. The first 4 KiB of the response body is stored
with the attempt.
These event headers are never forwarded: These event headers are never forwarded:
@@ -1091,8 +1055,8 @@ An `http` target's settings:
- `Trailer` - `Trailer`
- `User-Agent` - `User-Agent`
- `timeout_seconds`, the per-request timeout, at most 300. Blank means the - `timeout_seconds`, the per-request timeout, at most 300. Blank means the
shared 30 second default. A long timeout holds one of the ten delivery workers shared 30 second default. A long timeout holds one of the ten background
for its whole duration. workers for its whole duration.
- `forward_query`, off by default. On, the event's query string is appended to - `forward_query`, off by default. On, the event's query string is appended to
the URL, joined with `&` if the URL already has one. the URL, joined with `&` if the URL already has one.
@@ -1122,8 +1086,7 @@ attempted. `retrying` means at least one attempt failed and more remain.
`delivered` means an attempt succeeded. `failed` means every allowed attempt `delivered` means an attempt succeeded. `failed` means every allowed attempt
failed. failed.
Every attempt is recorded with its number, success, status code, response body, Every attempt is recorded, except one cut off by shutdown.
error and duration. One case is not recorded: an attempt cut off by shutdown.
### Circuit breaker ### Circuit breaker
@@ -1141,42 +1104,30 @@ breaker opens again.
The webhook page says when a target's deliveries are paused and until when. Each The webhook page says when a target's deliveries are paused and until when. Each
waiting delivery in the event log shows the earliest time it can be tried. waiting delivery in the event log shows the earliest time it can be tried.
### Workers and queues
Ten workers carry out deliveries. New deliveries wait in one queue and retries
in another. Each queue holds 10,000 tasks. At most ten deliveries are in flight
at once, however deep the queues get.
When a retry falls due and its queue is full, the retry is dropped and the
delivery stays `retrying` in the database. The recovery sweep picks it up.
Each task carries the target's settings and the event. A body under 16 KiB
travels with the task. A larger one is read from the database when the attempt
runs.
### Recovery ### Recovery
A recovery routine runs once at start and then every 60 seconds. For every The recovery routine finds deliveries the delivery engine is not working on and
webhook it reads the deliveries still `pending` or `retrying`. One that already queues them again. It runs once at start and then every 60 seconds; each
has a successful attempt recorded is marked `delivered` and not sent again. A periodic run is a sweep. It reads the deliveries still `pending` or `retrying`
`retrying` delivery is scheduled for whatever remains of its wait. A `pending` in every event database. One that already has a successful attempt recorded is
one is queued. A delivery the engine is already working on is left alone. marked `delivered` and not sent again. A `retrying` delivery is scheduled for
whatever remains of its wait. A `pending` one is queued. A delivery the engine
is already working on is left alone.
At start, every `pending` delivery is taken, however recent. So a delivery cut At start, every `pending` delivery is taken, however recent. So a delivery cut
off by a stop moments earlier is sent again at once. The periodic sweep takes off by a stop moments earlier is sent again at once. A sweep takes only
only `pending` deliveries untouched for 15 minutes, at most 500 per webhook each `pending` deliveries untouched for 15 minutes, at most 500 per webhook each
time. time.
If a target was deleted while a delivery to it was still open, recovery marks If a target was deleted while a delivery to it was still open, the recovery
the delivery `failed` and records the reason. routine marks the delivery `failed` and records the reason.
### Shutdown ### Shutdown
On `SIGINT` or `SIGTERM`, components stop in reverse start order under a 5 On `SIGINT` or `SIGTERM`, the whole shutdown has a 5 second budget. The HTTP
second budget for the whole sequence. The HTTP server drains open requests, server first waits for open requests to finish, for at most 3 seconds. That
which means it waits for them to finish, for at most 3 seconds. That leaves 2 leaves 2 seconds for the rest. A Sentry flush, if enabled, fits inside the same
seconds for the components after it. A Sentry flush, if enabled, fits inside the budget and is skipped when little is left.
same budget and is skipped when little is left.
The delivery engine cancels any `http` or `slack` request in flight. That The delivery engine cancels any `http` or `slack` request in flight. That
attempt is not recorded, and the delivery keeps its status, so it is sent again attempt is not recorded, and the delivery keeps its status, so it is sent again
@@ -1187,8 +1138,9 @@ after it are skipped, the database close included. The database files are then
left as a crash leaves them, with their `-wal` and `-shm` files beside them. left as a crash leaves them, with their `-wal` and `-shm` files beside them.
Keep the container's stop grace period above 5 seconds. That is the time Docker Keep the container's stop grace period above 5 seconds. That is the time Docker
waits after `docker stop` before it kills the process. Docker's default is 10. A waits after `docker stop` before it kills the process. Docker's default is 10.
shorter grace kills the process before the shutdown can run. The shutdown starts at the signal, and a shorter grace cuts it off before it
finishes.
### Archives ### Archives
@@ -1244,20 +1196,29 @@ Rows from a period file carry a `period` field. A body that is not valid UTF-8
is written in base64 with `"body_encoding": "base64"` beside it. The download is written in base64 with `"body_encoding": "base64"` beside it. The download
streams one file at a time and creates nothing. streams one file at a time and creates nothing.
## Data reference ## Data files
### Files The main database, `webhooker.db`, holds:
`webhooker.db` holds the application tier: settings, users, API keys, webhooks, - the settings, which today are only the session key;
entrypoints and targets. Each `events-{webhook_uuid}.db` holds that webhook's - the `admin` account and its password hash;
event tier: events, deliveries, delivery attempts and running totals. Each - the webhooks;
`archive-*.db` holds one archive target's rows. - the entrypoints;
- the targets.
Every database is opened through one code path, in WAL mode, with at most 4 Each event database, `events-{webhook_uuid}.db`, holds that webhook's:
connections. A connection waits up to 10 seconds for a locked database before
giving up. Under WAL a reader never blocks a writer.
An event database is created with its webhook. Every webhook's database is - events;
- deliveries;
- delivery attempts;
- running totals, which the pages read instead of counting rows.
Each archive file, `archive-*.db`, holds one archive target's rows.
Every database is opened in WAL mode, with at most 4 connections. A connection
waits up to 10 seconds for a locked database before giving up.
An event database is created with its webhook. Every webhook's event database is
opened at start. One that is missing or empty is created again and logged with opened at start. One that is missing or empty is created again and logged with
`created a new, empty database` at `WARN`. On a deployment that has run before, `created a new, empty database` at `WARN`. On a deployment that has run before,
that line means the file was lost, most often because the volume holding that line means the file was lost, most often because the volume holding
@@ -1269,82 +1230,10 @@ Deleting a webhook soft-deletes its row, entrypoints and targets in
with a deletion time and hides it from the application; the row stays in the with a deletion time and hides it from the application; the row stays in the
file. Deleting an entrypoint or a target soft-deletes it. file. Deleting an entrypoint or a target soft-deletes it.
### Schema
Each kind of database file has its schema in SQL files built into the binary, Each kind of database file has its schema in SQL files built into the binary,
under `internal/db/migrations/`: `main/` for `webhooker.db`, `events/` for event under `internal/db/migrations/`: `main/` for `webhooker.db`, `events/` for event
databases and `archive/` for archive files. In each, `000_migration.sql` creates databases and `archive/` for archive files. Opening a database applies the files
the `schema_migrations` table and `001_schema.sql` holds the schema. Opening a it has not had yet.
database applies every file its `schema_migrations` does not list, in one
transaction.
### Application tier
Every entity below except `Setting` has `id` (a version 4 UUID), `created_at`,
`updated_at` and `deleted_at`. A row with `deleted_at` set is soft-deleted.
`Setting` is a key-value row with no other fields. The one key stored is
`session_key`, the base64 session key made on first start.
`User` has `username` (unique, at most 1024 bytes) and `password` (the Argon2id
hash). The first start creates `admin` with a random 16-character password.
`Webhook` has `user_id`, `name`, `description` and `retention_days`.
`retention_days` is `0` for forever, or a whole number from 1 to 106751. The cap
keeps the retention cutoff inside the range of a Go duration.
`Entrypoint` has `webhook_id`, `path`, `description` and `active`. `path` is the
bare UUID. The `/h/` prefix is route only and is not stored.
`Target` has these fields. Each type fills only its own settings.
- `webhook_id`, `name`, `type` and `active`. `type` is one of `http`, `slack`,
`database` or `log`.
- `url`, `headers`, `timeout_seconds` and `forward_query`, the `http` settings.
A `slack` target uses `url` alone.
- `archive_expiry` and `archive_rotation`, the archive target's settings.
`archive_expiry` is stored in nanoseconds, `0` for never.
- `max_attempts`, for `http` and `slack`.
`APIKey` has `user_id`, `key`, `description` and `last_used_at`. Its table,
`api_keys`, exists, but nothing creates a key and nothing checks one.
### Event tier
`Event` has these fields:
- `webhook_id` and `entrypoint_id`;
- `method`, which is always `POST`;
- `raw_query`, `headers` (as JSON), `body`, `body_bytes` and `content_type`;
- `resubmitted_from_id`, the event a resubmit copied. It is empty for an event
that arrived on the receiver.
`Delivery` has these fields:
- `event_id` and `target_id`;
- `status`;
- `finished_at`, set when the status becomes `delivered` or `failed`;
- `replay`.
`DeliveryResult` is one attempt. It has these fields:
- `delivery_id` and `attempt_num`;
- `success` and `status_code`;
- `response_body`, the first 4 KiB;
- `error`;
- `duration`, in milliseconds.
Three tables hold running totals so the pages need not count rows. Each count
changes in the transaction that writes or deletes the rows it counts.
- `EventTotals` is one row: `events`, `events_removed` and `last_event_at`.
- `TargetTotals` is one row per target, with:
- `deliveries`, `delivered` and `failed`;
- `deliveries_removed` and `failed_removed`.
- `EntrypointTotals` is one row per entrypoint: `last_event_at`.
Retention raises the `_removed` counts and leaves `last_event_at` alone, so a
webhook's last event is still shown after its events have expired.
## HTTP routes reference ## HTTP routes reference
@@ -1372,34 +1261,10 @@ Sign-in routes:
- `POST /pages/login` signs in. See [Sign-in limits](#sign-in-limits). - `POST /pages/login` signs in. See [Sign-in limits](#sign-in-limits).
- `POST /pages/logout` signs out. - `POST /pages/logout` signs out.
Routes behind the sign-in. A `GET` without a session is redirected to The pages behind the sign-in live under `/user/`, `/settings`, `/hooks` and
`/pages/login` with the path as `next`: `/hook/`. They are reached from the web interface. A `GET` to one of them
without a session is redirected to `/pages/login`, with the page's path as
- `GET /user/{username}` is the profile page. Another user's name gets `403`. `next`.
- `POST /user/{username}/password` changes the password, 5 per minute per
client.
- `GET /settings` lists the loaded configuration.
- `GET /hooks` lists webhooks. `GET` and `POST /hooks/new` create one.
- `GET /hook/{id}` is the webhook page. `GET` and `POST /hook/{id}/edit` edit
it. `POST /hook/{id}/delete` deletes it.
- `GET /hook/{id}/events` is the event log. `?show=failed` and `?show=pending`
narrow it.
- `GET /hook/{id}/events/{eventID}` is one event's page, and
`GET /hook/{id}/events/{eventID}/body` downloads its stored body.
- `POST /hook/{id}/deliveries/{deliveryID}/replay` replays a delivery, and
`POST /hook/{id}/events/{eventID}/resubmit` resubmits an event, each 30 per
minute per client.
- `POST /hook/{id}/entrypoints` adds an entrypoint.
- `POST /hook/{id}/entrypoints/{entrypointID}/edit` changes its description.
- `POST /hook/{id}/entrypoints/{entrypointID}/delete` deletes it.
- `POST /hook/{id}/entrypoints/{entrypointID}/toggle` activates or deactivates
it.
- `POST /hook/{id}/targets` adds a target.
- `GET /hook/{id}/targets/{targetID}/edit` shows the target edit form, and
`POST` to the same path saves it.
- `POST /hook/{id}/targets/{targetID}/delete` deletes the target.
- `POST /hook/{id}/targets/{targetID}/toggle` activates or deactivates it.
- `GET /hook/{id}/targets/{targetID}/download` downloads an archive.
`GET /metrics` exists only when `METRICS_USERNAME` and `METRICS_PASSWORD` are `GET /metrics` exists only when `METRICS_USERNAME` and `METRICS_PASSWORD` are
set, behind basic authentication. set, behind basic authentication.
@@ -1408,18 +1273,13 @@ set, behind basic authentication.
## Metrics reference ## Metrics reference
`/metrics` serves one metrics registry, the set of metrics it exposes. The `/metrics` serves the delivery metrics below, three inbound HTTP metrics, and
registry carries: the Go runtime and process metrics the Prometheus library provides.
- the Prometheus library's Go runtime collector;
- its process collector;
- the delivery metrics below;
- three inbound HTTP metrics.
Nine of the eleven metrics below carry one label, `target_type`, with one of Nine of the eleven metrics below carry one label, `target_type`, with one of
`http`, `slack`, `database`, `log` or `unknown`. The two event counters carry `http`, `slack`, `database`, `log` or `unknown`. The two event counters carry
none. Target ids and event ids are never labels, because a label value that none. Target ids and event ids are never labels, because a label value that
grows with traffic would grow the registry without bound. grows with traffic would grow the set of metrics without bound.
- `webhooker_events_received_total` (counter, no label): events stored, - `webhooker_events_received_total` (counter, no label): events stored,
resubmitted copies included. resubmitted copies included.
@@ -1436,8 +1296,8 @@ grows with traffic would grow the registry without bound.
**Resubmit**. **Resubmit**.
- `webhooker_delivery_duration_seconds` (histogram): wall time of one attempt. - `webhooker_delivery_duration_seconds` (histogram): wall time of one attempt.
- `webhooker_deliveries_pending` and `webhooker_deliveries_retrying` (gauges): - `webhooker_deliveries_pending` and `webhooker_deliveries_retrying` (gauges):
deliveries in those statuses, counted from the databases every 30 seconds. deliveries in those statuses, counted from the event databases every 30
Deliveries to a deleted target count under `unknown`. seconds. Deliveries to a deleted target count under `unknown`.
- `webhooker_circuit_breakers_open` (gauge): breakers currently open. - `webhooker_circuit_breakers_open` (gauge): breakers currently open.
The success and failure counters move only after the new status has been The success and failure counters move only after the new status has been
@@ -1458,8 +1318,8 @@ set to `(all)`.
- Docker. The tests, the linters, prettier and the stylesheet build run in - Docker. The tests, the linters, prettier and the stylesheet build run in
Docker. Docker.
`gofmt` and `goimports` run on the host. These tools run only inside Docker, at `gofmt` and `goimports` run on the host. The tools below run only inside Docker,
the versions pinned (fixed to one exact release) in the `Dockerfile`, at the versions pinned (fixed to one exact release) in the `Dockerfile`,
`package.json` and `yarn.lock`: `package.json` and `yarn.lock`:
- `golangci-lint`, the Go linter; - `golangci-lint`, the Go linter;
@@ -1519,9 +1379,8 @@ The `lint` stage runs these steps:
3. `golangci-lint run`, with `--network=none`; 3. `golangci-lint run`, with `--network=none`;
4. ESLint over `static/js/`, in the `js-lint` stage, with `--network=none`. 4. ESLint over `static/js/`, in the `js-lint` stage, with `--network=none`.
The `test` stage runs `go test -cover` on four packages at a time, each running The `test` stage runs `go test -cover`. On a failure it runs only the failed
at most eight tests in parallel, with a 90 second timeout per package. On a tests again with `-v`.
failure it runs only the failed tests again with `-v`.
The image build depends on the `lint`, `test`, `css-check` and `markdown-check` The image build depends on the `lint`, `test`, `css-check` and `markdown-check`
stages, so an image that built is a tree that passed every check. The runtime stages, so an image that built is a tree that passed every check. The runtime
@@ -1569,12 +1428,12 @@ With the CSP build, every Alpine.js directive in `templates/` names a component
registered in `static/js/app.js`, such as `x-data="collapsible"`, and never an registered in `static/js/app.js`, such as `x-data="collapsible"`, and never an
inline expression. inline expression.
The package file is committed as `3p/alpinejs-csp-3.14.9.tgz`, byte for byte as The package file is committed as `3p/alpinejs-csp-3.14.9.tgz`. It is byte for
the npm registry, the JavaScript package registry, publishes it. `script/assets` byte the file the npm registry, the JavaScript package registry, publishes.
extracts `package/dist/cdn.min.js` from it to `static/js/alpine.min.js`, which `script/assets` extracts `package/dist/cdn.min.js` from it to
the binary embeds. `make build`, `make dev` and the Docker stages run it first. `static/js/alpine.min.js`, which the binary embeds. `make build`, `make dev` and
The extracted file is not committed, and a build that skips the extraction fails the Docker stages run it first. The extracted file is not committed, and a build
with an error naming `js/alpine.min.js`. that skips the extraction fails with an error naming `js/alpine.min.js`.
To move to a new version: To move to a new version: