1 Commits
Author SHA1 Message Date
sneak d9ededcb08 Rewrite README.md sentence by sentence
check / check (push) Canceled after 0s
Rewritten from scratch against the code on next. The old text was
the list of what a reader needs covered.

Model: fable-5-1
2026-10-25 04:16:55 +00:00
+142 -166
View File
@@ -1,31 +1,32 @@
# webhooker # webhooker
webhooker is an MIT-licensed web service, written in [Go](https://golang.org) by webhooker is an MIT-licensed web service, written in [Go](https://golang.org) by
[@sneak](https://sneak.berlin), that receives the HTTP requests other services [@sneak](https://sneak.berlin), that receives the HTTP notifications other
send when something happens and forwards them to your own systems. services send and forwards them to your own systems.
Many services send an HTTP request to a URL you give them when something Many services send an HTTP request to a URL you give them when something
happens: a code push, a payment, a new ticket. webhooker receives those happens: a code push, a payment, a new ticket. webhooker receives those
requests, stores each one as an event, and forwards it to the places you choose. requests, stores them, and forwards them to the places you choose.
webhooker is a single binary with a web interface. It stores its data in SQLite, webhooker is a single binary with a web interface. It stores its data in SQLite,
a database engine that keeps each database in one ordinary file. It writes each a database engine that keeps each database in one ordinary file. It writes each
event to disk before answering the sender. It then delivers the event to each notification to disk before answering the sender. It then delivers it to each
target you configured. The web interface shows the newest events, with every target you configured. The web interface shows the newest notifications, with
delivery and attempt they had. From there you can send a stored event again. every delivery and attempt they had. From there you can send a stored
webhooker also exports metrics for Prometheus, the monitoring system, so you can notification again. webhooker also exports metrics for Prometheus, the
watch the whole flow. monitoring system, so you can watch the whole flow.
## Rationale ## Rationale
webhooker exists so that a request from another service is not lost while your Senders of these notifications usually make one attempt and keep no record. If
service is down. It also exists so that you can see what was sent and what your service is down at that moment, the notification is lost. Few senders let
happened to it. you see what they sent, or when, or whether it arrived.
It stores each event before it forwards it, so an event that arrives while your webhooker sits between the sender and your service and fixes those three things.
service is down is not lost. It keeps events and their delivery attempts for the It is always ready to receive, and it stores before it forwards, so nothing is
retention period you set. It can forward one event to several targets, and it lost while your service is down. It keeps every notification and every delivery
can send a stored event again. attempt, so you can read what was sent and what happened to it. It can forward
one notification to several targets, and it can send a stored one again.
## How it works ## How it works
@@ -39,18 +40,15 @@ Five words name the moving parts.
- A **target** is a place events are delivered to. There are four kinds: an HTTP - A **target** is a place events are delivered to. There are four kinds: an HTTP
URL, a Slack or Mattermost message-posting URL, an archive database file, and URL, a Slack or Mattermost message-posting URL, an archive database file, and
the application log. the application log.
- An **event** is one stored request: its method, query string, headers and - An **event** is one stored notification: its method, query string, headers and
body. body.
- A **delivery** is one event paired with one target. It records every attempt. - A **delivery** is one event paired with one target. It records every attempt.
The part of webhooker that answers requests to `/h/{uuid}` is the receiver. A request to an entrypoint is stored as an event. In the same database
transaction, one pending delivery is created for each active target of the
A request to an entrypoint is stored as an event. One pending delivery is webhook. The sender gets `200` once that transaction has committed. Workers then
created for each active target of the webhook, in the same database write as the carry out the deliveries. A delivery to an HTTP or Slack target that fails is
event: either all of them are stored or none. The sender gets `200` once they retried, up to the number of attempts the target allows.
are stored. Workers then carry out the deliveries. A delivery to an HTTP or
Slack target that fails is retried, up to the number of attempts the target
allows.
Delivery is at least once, not exactly once. If a delivery reaches the target Delivery is at least once, not exactly once. If a delivery reaches the target
but webhooker cannot record that outcome, it sends the delivery again later. A but webhooker cannot record that outcome, it sends the delivery again later. A
@@ -181,13 +179,8 @@ A target's type cannot be changed. Create a new target to deliver another way.
### Statistics ### Statistics
The top of the webhook page shows: The top of the webhook page shows counts of entrypoints and targets, deliveries
in progress, the time of the last event, and the retention setting.
- the number of entrypoints, and how many are active;
- the number of targets, and how many are active;
- deliveries in progress;
- the time of the last event;
- the retention setting.
Below that are events, deliveries and failures, both lifetime and within Below that are events, deliveries and failures, both lifetime and within
retention. Next are figures for the last 10 minutes and the last 24 hours: retention. Next are figures for the last 10 minutes and the last 24 hours:
@@ -264,21 +257,22 @@ The image listens on every IPv4 address inside the container: it ships
`BIND_ADDRESS=0.0.0.0`. A container has its own network namespace, which means `BIND_ADDRESS=0.0.0.0`. A container has its own network namespace, which means
its own addresses, separate from the host's. That is the boundary, and the `-p` its own addresses, separate from the host's. That is the boundary, and the `-p`
flag decides what the host exposes. Publish to `127.0.0.1:8080`, the address flag decides what the host exposes. Publish to `127.0.0.1:8080`, the address
your reverse proxy connects to, and nothing wider. A bare `-p 8080:8080` your reverse proxy connects from, and nothing wider. A bare `-p 8080:8080` opens
publishes the port on every interface of the host. the port on every interface of the host, and Docker's forwarding rules sit ahead
of most host firewalls.
Do not set `BIND_ADDRESS` to an empty value in a container. An empty value means Do not set `BIND_ADDRESS` to an empty value in a container. An empty value means
unset, and unset falls back to the binary's default of `127.0.0.1`. The unset, and unset falls back to the binary's default of `127.0.0.1`. The
container then answers nothing on its published port and its health check fails. container then answers nothing on its published port and its health check fails.
A templated Compose file with an empty value is the usual way this happens.
A container on an IPv6-only Docker network needs `BIND_ADDRESS=::`. A container on an IPv6-only Docker network needs `BIND_ADDRESS=::`.
The app runs as user `webhooker` (UID 1000). The image's start-up script, The app runs as user `webhooker` (UID 1000). The image's entrypoint script
`deploy/docker-entrypoint.sh`, starts as root. It creates `DATA_DIR` if it is starts as root. It creates `DATA_DIR` if it is missing, gives it and its
missing, gives it and its contents to `webhooker`, and sets the directory to contents to `webhooker`, and sets the directory to mode `0750`. Then it runs the
mode `0750`. Then it runs the app as `webhooker`. So a host directory can be app as `webhooker`. So a host directory can be mounted as it is, whoever owns
mounted as it is, whoever owns it. With `--user`, the script changes nothing and it. With `--user`, the script changes nothing and runs the app as that user.
runs the app as that user.
webhooker creates every SQLite file with mode `0600`, and resets a file found webhooker creates every SQLite file with mode `0600`, and resets a file found
with another mode when it opens it. `webhooker.db` holds target settings in with another mode when it opens it. `webhooker.db` holds target settings in
@@ -331,11 +325,10 @@ docker start upaas-webhooker
### Behind a reverse proxy ### Behind a reverse proxy
webhooker speaks plain HTTP. It does not handle TLS, the encryption behind webhooker speaks plain HTTP. It does not terminate TLS, the encryption behind
HTTPS: it holds no certificate and accepts no HTTPS connections. Put a reverse HTTPS. Put a reverse proxy in front of it: a server that accepts the public
proxy in front of it: a server that accepts the public connections and passes connections and passes them on. The proxy holds the certificate and reaches
them on. The proxy holds the certificate and reaches webhooker over loopback, webhooker over loopback, the address only this host can reach.
the address only this host can reach.
Five things must be right. Five things must be right.
@@ -495,8 +488,10 @@ without it is missing data and may have no readable tables at all.
A clean shutdown closes every database and removes its `-wal` and `-shm` files. A clean shutdown closes every database and removes its `-wal` and `-shm` files.
A crash leaves them, and they must be carried with the `.db`. A crash leaves them, and they must be carried with the `.db`.
Keep `DATA_DIR` on a local disk. WAL mode is not reliable on network `DATA_DIR` must be on a local filesystem. SQLite maps the `-shm` file into
filesystems, and webhooker does not check for one. memory, which network filesystems do not support reliably. webhooker does not
check this. An error reading `disk I/O error (522)` points at the storage under
`DATA_DIR`.
Configuration is not in `DATA_DIR`. It comes from the environment and from a Configuration is not in `DATA_DIR`. It comes from the environment and from a
`.env` file in the working directory. Back those up with your deployment. `.env` file in the working directory. Back those up with your deployment.
@@ -524,11 +519,14 @@ for db in /path/to/data/*.db; do
done done
``` ```
`.backup` writes one complete file, with no `-wal` beside it. The runtime image `.backup` reads through the WAL and writes one complete file. The runtime image
has no `sqlite3`, so run this on the host or from another container with the has no `sqlite3`, so run this on the host or from another container with the
volume mounted. Each file is captured at its own instant, so an event stored volume mounted. Each file is captured at its own instant, so an event stored
between two copies lands in one file and not the other. between two copies lands in one file and not the other.
`sqlite3 <db> .dump` is an export, not a backup. It is safe to run but holds a
read transaction open for its whole duration.
Archive files are the exception. The archive writer closes and reopens its file Archive files are the exception. The archive writer closes and reopens its file
after a write, at most once per second. So you can move a finished archive file after a write, at most once per second. So you can move a finished archive file
away while the service runs. Move its `-wal` and `-shm` with it, or wait until away while the service runs. Move its `-wal` and `-shm` with it, or wait until
@@ -562,8 +560,8 @@ who can read it.
Each entrypoint is a random version 4 UUID served at `/h/{uuid}`. Holding that Each entrypoint is a random version 4 UUID served at `/h/{uuid}`. Holding that
URL is what authorises a sender. The receiver checks nothing else: no shared URL is what authorises a sender. The receiver checks nothing else: no shared
secret, no HMAC signature (a hash of the body made with a shared key), no token secret, no HMAC signature (a hash of the body made with a shared key), no token
header. Signature headers a sender adds anyway, such as `X-Hub-Signature`, are header. None will be added. Signature headers a sender adds anyway, such as
stored and forwarded as ordinary headers. `X-Hub-Signature`, are stored and forwarded as ordinary headers.
What that means in practice: What that means in practice:
@@ -649,8 +647,9 @@ the proxy.
### Delivery destinations ### Delivery destinations
A delivery target could be used to make webhooker send requests into the network A delivery target could be used to make webhooker send requests into the network
it sits in and read the answers back. This is called server-side request forgery it sits in and read the answers back. This is called server-side request
(SSRF). webhooker refuses it with the SSRF guard, described here. forgery. webhooker refuses it by default, with a check called the destination
check below.
Every target URL must use `http` or `https` and resolve to a public address. Every target URL must use `http` or `https` and resolve to a public address.
These private and reserved ranges are refused: These private and reserved ranges are refused:
@@ -669,16 +668,16 @@ These private and reserved ranges are refused:
One public address is refused too: `168.63.129.16`, the Azure service that hands One public address is refused too: `168.63.129.16`, the Azure service that hands
a virtual machine its credentials. a virtual machine its credentials.
The SSRF guard runs when a target is saved and again when each delivery The destination check runs when a target is saved and again when each delivery
connects. Both use the same function. The address is resolved again at connect connects. Both use the same function. The address is resolved again at connect
time, so a hostname that changes its address after validation is still checked. time, so a hostname that changes its address after validation is still checked.
Redirects are followed up to ten hops. Each hop is checked the same way. A hop Redirects are followed up to ten hops. Each hop is checked the same way. A hop
that leaves the target's origin carries none of the target's configured headers that leaves the target's origin carries none of the target's configured headers
and none of the forwarded event headers. Leaving the origin means a different and none of the forwarded event headers. Leaving the origin means a different
host, a different port, or a step from `https` to `http`. An open redirect is a host, a different port, or a step from `https` to `http`. This stops an open
destination that forwards requests to any URL a caller names. This rule stops redirect, a destination that forwards requests to any URL a caller names, from
one from handing a credential to another host. handing a credential to another host.
`ALLOWED_EGRESS_CIDRS` opens named networks for the common case of forwarding to `ALLOWED_EGRESS_CIDRS` opens named networks for the common case of forwarding to
your own network. Read your own network. Read
@@ -706,19 +705,20 @@ peer, which behind a proxy is the proxy. `clientIP` is the client as decided by
`TRUSTED_PROXIES`. `TRUSTED_PROXIES`.
SQL statements are logged with `?` in place of each value, never with the values SQL statements are logged with `?` in place of each value, never with the values
themselves. Outside a `log` target's line, no line carries session cookies or themselves. Session cookies, API keys and target credentials are never logged.
target credentials.
The `admin` password is logged in the clear once, on the first boot, and again Two things are logged in the clear. The first is the `admin` password, once, on
when `webhooker resetpw -generate` is run. The operator's own configuration is the first boot, and again when `webhooker resetpw -generate` is run. The second
echoed back in the clear: webhook names and target hostnames. is the operator's own configuration echoed back: webhook names and target
hostnames.
A `log` target writes the whole event to the log: query string, headers and A `log` target writes the whole event to the log: query string, headers and
body. That is its purpose. body. That is its purpose.
A recovered panic is a crash inside one request that the server caught. It A recovered panic is a crash inside one request that the server caught. It
writes one `ERROR` record of at most 10,240 bytes. The record carries the same writes one `ERROR` record of at most 10,240 bytes. The record carries the
`request_id` as the request's access log line, which ties the two together. request id of the request that caused it. A request id is a random token given
to each request and written in its log lines.
When `SENTRY_DSN` is set, error reports go to Sentry, an error reporting When `SENTRY_DSN` is set, error reports go to Sentry, an error reporting
service, with the request reduced to this: service, with the request reduced to this:
@@ -735,13 +735,13 @@ service, with the request reduced to this:
One request to an entrypoint goes through these steps: One request to an entrypoint goes through these steps:
1. The rate limits run: one per client per entrypoint, one per client across the 1. The router matches `POST /h/{uuid}`. Any other method gets `405`.
2. The rate limits run: one per client per entrypoint, one per client across the
whole route. whole route.
2. The handler answers any method but `POST` with `405`.
3. The handler looks up the entrypoint. Unknown gets `404`, deactivated `410`. 3. The handler looks up the entrypoint. Unknown gets `404`, deactivated `410`.
4. The handler reads the body, up to 1 MB. A larger body gets `413`. 4. The handler reads the body, up to 1 MB. A larger body gets `413`.
5. The event, one pending delivery per active target, and the running totals are 5. One transaction stores the event, one pending delivery per active target, and
stored together, or not at all. The sender gets `200`. the running totals. The sender gets `200`.
6. One task per delivery goes onto the delivery queue. 6. One task per delivery goes onto the delivery queue.
7. A worker takes the task and runs the target's delivery. The outcome is 7. A worker takes the task and runs the target's delivery. The outcome is
recorded as an attempt, and the delivery's status is updated. recorded as an attempt, and the delivery's status is updated.
@@ -749,15 +749,15 @@ One request to an entrypoint goes through these steps:
### Components and middleware ### Components and middleware
The components are wired by Uber fx, a library that builds each component when The components are wired by Uber fx, a library that builds each component when
another needs it. fx starts the components in the order their dependencies need, another needs it. It runs their start hooks (the code each component runs at
and stops them in reverse. The components are: start) in that order, and their stop hooks in reverse. The components are:
- the globals, the logger and the config; - the globals, the logger and the config;
- the main database and the per-webhook database manager; - the main database and the per-webhook database manager;
- the retention sweep and the health check; - the retention sweep and the health check;
- the session and the handlers; - the session and the handlers;
- the metrics registry and collectors; - the metrics registry and collectors;
- the middleware and the SSRF guard; - the middleware and the destination check;
- the delivery engine, the archive writers and the archive sweep; - the delivery engine, the archive writers and the archive sweep;
- the server. - the server.
@@ -800,13 +800,13 @@ The body cap comes before CSRF because CSRF parses the form.
protection. protection.
- [go-chi/httprate](https://github.com/go-chi/httprate) rate-limits the - [go-chi/httprate](https://github.com/go-chi/httprate) rate-limits the
receiver, the password change, replay and resubmit. receiver, the password change, replay and resubmit.
- The [Prometheus](https://prometheus.io) library and - The [Prometheus](https://prometheus.io) client and
[go-http-metrics](https://github.com/slok/go-http-metrics) serve the metrics. [go-http-metrics](https://github.com/slok/go-http-metrics) serve the metrics.
- The [Sentry](https://sentry.io) library reports errors when configured. - The [Sentry](https://sentry.io) client reports errors when configured.
### Package layout ### Package layout
Most of the Go code is under `internal/`. The program starts in All application code is under `internal/`. The program starts in
`cmd/webhooker/main.go`, which dispatches subcommands, takes the `DATA_DIR` lock `cmd/webhooker/main.go`, which dispatches subcommands, takes the `DATA_DIR` lock
and wires the components. and wires the components.
@@ -818,7 +818,7 @@ and wires the components.
- `internal/datadir`: the exclusive lock on `DATA_DIR`. - `internal/datadir`: the exclusive lock on `DATA_DIR`.
- `internal/db/migrations`: the SQL schema files and their application. - `internal/db/migrations`: the SQL schema files and their application.
- `internal/delivery`: the engine, the four target types, retries, the circuit - `internal/delivery`: the engine, the four target types, retries, the circuit
breaker and the SSRF guard. breaker and the destination check.
- `internal/globals`: build-time variables. - `internal/globals`: build-time variables.
- `internal/gormlog`: GORM's logger on top of `slog`, with values withheld. - `internal/gormlog`: GORM's logger on top of `slog`, with values withheld.
- `internal/handlers`: every HTTP handler and page. - `internal/handlers`: every HTTP handler and page.
@@ -889,7 +889,7 @@ is refused; the port belongs to `PORT`.
An address that parses but is not on this host passes configuration and fails at An address that parses but is not on this host passes configuration and fails at
bind time. The process logs the error and exits non-zero. The `bindAddress` bind time. The process logs the error and exits non-zero. The `bindAddress`
field of the `Configuration loaded` line shows the configured address. field of the startup log shows what was bound.
### `DATA_DIR` ### `DATA_DIR`
@@ -939,9 +939,9 @@ startup fails with an error naming both variables.
### `SENTRY_DSN` ### `SENTRY_DSN`
The DSN for Sentry error reporting: the URL that tells the Sentry library where The DSN for Sentry error reporting: the URL that tells the Sentry client where
to send reports. Default empty, which leaves reporting off. A set value is to send reports. Default empty, which leaves reporting off. A set value is
parsed with the Sentry library's own parser, and one it cannot parse aborts parsed with the Sentry client's own parser, and one it cannot parse aborts
startup. The release name sent to Sentry is `webhooker-{version}`. startup. The release name sent to Sentry is `webhooker-{version}`.
### `RETENTION_SWEEP_INTERVAL` ### `RETENTION_SWEEP_INTERVAL`
@@ -995,11 +995,11 @@ When the connecting peer is inside the list, the client address is read from
that header. For any other peer, the client is the peer itself and the header is that header. For any other peer, the client is the peer itself and the header is
ignored. `X-Real-IP` and `True-Client-IP` are never read. ignored. `X-Real-IP` and `True-Client-IP` are never read.
The header is read from right to left. The first entry that is not itself a The header is read from right to left. The first hop that is not itself a
trusted proxy is the client. An entry that is not a bare IP address ends the trusted proxy is the client. A hop that is not a bare IP address ends the walk,
walk, and the peer address is used instead. At most 64 entries are examined. and the peer address is used instead. At most 64 hops are examined. Your proxy
Your proxy must therefore append the peer address to `X-Forwarded-For`, as must therefore append the peer address to `X-Forwarded-For`, as nginx's
nginx's `$proxy_add_x_forwarded_for` does. `$proxy_add_x_forwarded_for` does.
If any client can reach webhooker, or the proxy in front of it, from a private If any client can reach webhooker, or the proxy in front of it, from a private
address, set the list to the proxy's address alone. Otherwise those clients can address, set the list to the proxy's address alone. Otherwise those clients can
@@ -1025,8 +1025,8 @@ This only ever adds destinations. The default refusals stay in force for every
address not listed. When the list is set, startup logs the whole list at `WARN`. address not listed. When the list is set, startup logs the whole list at `WARN`.
Anyone with the admin account can make webhooker send requests to every network Anyone with the admin account can make webhooker send requests to every network
you list. They can read the responses in the event log. List the smallest blocks you list. They can read the responses in the delivery log. List the smallest
that cover your destinations. Prefer `10.1.2.3/32` over `10.0.0.0/8`. blocks that cover your destinations. Prefer `10.1.2.3/32` over `10.0.0.0/8`.
`0.0.0.0/0` or `::/0` opens every private and reserved range at once, including `0.0.0.0/0` or `::/0` opens every private and reserved range at once, including
this host's own loopback. Do not list them. this host's own loopback. Do not list them.
@@ -1102,9 +1102,9 @@ the body in a code block. A JSON body is indented. A body over 3500 bytes is
cut. The URL is any message-posting URL that accepts Slack's format, which cut. The URL is any message-posting URL that accepts Slack's format, which
includes Mattermost's. includes Mattermost's.
An archive target, whose stored type is `database`, appends the event as a row A `database` target, called an archive in the web interface, appends the event
to its own SQLite file. See [Archives](#archives). It makes no network request as a row to its own SQLite file. See [Archives](#archives). It makes no network
and never retries. A write failure fails the delivery. request and never retries. A write failure fails the delivery.
A `log` target writes the event, headers and body included, as one `INFO` line. A `log` target writes the event, headers and body included, as one `INFO` line.
It never retries. It never retries.
@@ -1180,11 +1180,11 @@ same budget and is skipped when little is left.
The delivery engine cancels any `http` or `slack` request in flight. That The delivery engine cancels any `http` or `slack` request in flight. That
attempt is not recorded, and the delivery keeps its status, so it is sent again attempt is not recorded, and the delivery keeps its status, so it is sent again
at the next start. An archive or `log` task finishes as normal. at the next start. A `database` or `log` task finishes as normal.
A component that does not stop in time is logged at `ERROR`, and the components A component that does not stop in time is logged at `ERROR`, and the components
after it are skipped, the database close included. The database files are then after it are skipped. SQLite is crash-safe, so this loses no data, but the
left as a crash leaves them, with their `-wal` and `-shm` files beside them. database is not closed cleanly.
Keep the container's stop grace period above 5 seconds. That is the time Docker Keep the container's stop grace period above 5 seconds. That is the time Docker
waits after `docker stop` before it kills the process. Docker's default is 10. A waits after `docker stop` before it kills the process. Docker's default is 10. A
@@ -1232,17 +1232,12 @@ is the operator's decision.
**Download** on the webhook page returns the target's whole archive as one **Download** on the webhook page returns the target's whole archive as one
gzipped JSON file named gzipped JSON file named
`archive-{webhook_name}-{target_name}-{YYYYMMDDTHHMMSSZ}.json.gz`. It holds one `archive-{webhook_name}-{target_name}-{YYYYMMDDTHHMMSSZ}.json.gz`. It holds
object with these fields: `webhook` and `target` (each an `id` and a `name`), `exported_at`, and
`archived_events`, one object per row with every column. Rows from a period file
- `webhook`, with the webhook's `id` and `name`; carry a `period` field. A body that is not valid UTF-8 is written in base64 with
- `target`, with the target's `id` and `name`; `"body_encoding": "base64"` beside it. The download streams one file at a time
- `exported_at`; and creates nothing.
- `archived_events`, one object per row with every column.
Rows from a period file carry a `period` field. A body that is not valid UTF-8
is written in base64 with `"body_encoding": "base64"` beside it. The download
streams one file at a time and creates nothing.
## Data reference ## Data reference
@@ -1302,7 +1297,7 @@ bare UUID. The `/h/` prefix is route only and is not stored.
`database` or `log`. `database` or `log`.
- `url`, `headers`, `timeout_seconds` and `forward_query`, the `http` settings. - `url`, `headers`, `timeout_seconds` and `forward_query`, the `http` settings.
A `slack` target uses `url` alone. A `slack` target uses `url` alone.
- `archive_expiry` and `archive_rotation`, the archive target's settings. - `archive_expiry` and `archive_rotation`, the `database` settings.
`archive_expiry` is stored in nanoseconds, `0` for never. `archive_expiry` is stored in nanoseconds, `0` for never.
- `max_attempts`, for `http` and `slack`. - `max_attempts`, for `http` and `slack`.
@@ -1319,12 +1314,8 @@ bare UUID. The `/h/` prefix is route only and is not stored.
- `resubmitted_from_id`, the event a resubmit copied. It is empty for an event - `resubmitted_from_id`, the event a resubmit copied. It is empty for an event
that arrived on the receiver. that arrived on the receiver.
`Delivery` has these fields: `Delivery` has `event_id`, `target_id`, `status`, `finished_at` and `replay`.
`finished_at` is set when the status becomes `delivered` or `failed`.
- `event_id` and `target_id`;
- `status`;
- `finished_at`, set when the status becomes `delivered` or `failed`;
- `replay`.
`DeliveryResult` is one attempt. It has these fields: `DeliveryResult` is one attempt. It has these fields:
@@ -1351,19 +1342,13 @@ webhook's last event is still shown after its events have expired.
Public routes: Public routes:
- `GET /` redirects to `/hooks` when signed in and to `/pages/login` otherwise. - `GET /` redirects to `/hooks` when signed in and to `/pages/login` otherwise.
- `GET /.well-known/healthcheck` returns a JSON object with these fields: - `GET /.well-known/healthcheck` returns JSON with `status`, `now`,
- `status`, always `ok`; `uptimeSeconds`, `uptimeHuman`, `version` and `appname`.
- `now`, the current time;
- `uptimeSeconds` and `uptimeHuman`;
- `version` and `appname`.
- `GET /s/*` and `HEAD /s/*` serve the embedded CSS and JavaScript. Other - `GET /s/*` and `HEAD /s/*` serve the embedded CSS and JavaScript. Other
methods get `405` with `Allow: GET, HEAD`. methods get `405` with `Allow: GET, HEAD`.
- `POST /h/{uuid}` is the receiver. Its answers: - `POST /h/{uuid}` is the receiver. Any other method gets `405` with
- any method but `POST` gets `405` with `Allow: POST`; `Allow: POST`. An unknown UUID gets `404`, a deactivated entrypoint `410`, a
- an unknown UUID gets `404`; body over 1 MB `413`, and a stored event `200` with `{"status":"ok"}`.
- a deactivated entrypoint gets `410`;
- a body over 1 MB gets `413`;
- a stored event gets `200` with `{"status":"ok"}`.
Sign-in routes: Sign-in routes:
@@ -1408,13 +1393,10 @@ set, behind basic authentication.
## Metrics reference ## Metrics reference
`/metrics` serves one metrics registry, the set of metrics it exposes. The `/metrics` serves one Prometheus registry, the set of metrics it exposes. It
registry carries: carries the Go runtime and process collectors, which report the process's
memory, threads and CPU use, the delivery metrics below, and three inbound HTTP
- the Prometheus library's Go runtime collector; metrics.
- its process collector;
- the delivery metrics below;
- three inbound HTTP metrics.
Nine of the eleven metrics below carry one label, `target_type`, with one of Nine of the eleven metrics below carry one label, `target_type`, with one of
`http`, `slack`, `database`, `log` or `unknown`. The two event counters carry `http`, `slack`, `database`, `log` or `unknown`. The two event counters carry
@@ -1479,7 +1461,7 @@ make fmt-check # Fail if gofmt or prettier would change anything
make lint # Run gofmt, golangci-lint and ESLint in Docker make lint # Run gofmt, golangci-lint and ESLint in Docker
make test # Run the tests in Docker, with cgo off make test # Run the tests in Docker, with cgo off
make test-browser # Run the browser test in Docker (Dockerfile.browser) make test-browser # Run the browser test in Docker (Dockerfile.browser)
make check # test, lint, fmt-check and css-check make check # test, lint, fmt-check and css-check: what CI runs
make build # Build bin/webhooker, version-stamped make build # Build bin/webhooker, version-stamped
make version # Print the version this checkout would stamp make version # Print the version this checkout would stamp
make run # build, then run ./bin/webhooker make run # build, then run ./bin/webhooker
@@ -1492,11 +1474,10 @@ make css-check # Fail if static/css/tailwind.css is out of date
make clean # Remove bin/ make clean # Remove bin/
``` ```
Most `make` targets above are one-line wrappers: each runs one script in Each `make` target above, except `build`, `run`, `dev`, `deps`, `version` and
`script/`. The exceptions are `build`, `run`, `dev`, `deps`, `version` and `clean`, is a thin shim over a script in `script/`. `make hooks` calls
`clean`, which have no script. `make hooks` runs `script/install-precommit`; `script/install-precommit`; every other shim calls the script of the same name.
every other wrapper runs the script of the same name. The scripts are listed The scripts are listed under [Entrypoints](#entrypoints).
under [Entrypoints](#entrypoints).
The tests run with cgo, Go's bridge to C code, turned off. Go's race detector, The tests run with cgo, Go's bridge to C code, turned off. Go's race detector,
which finds unsynchronised access to shared memory, needs cgo, so the tests run which finds unsynchronised access to shared memory, needs cgo, so the tests run
@@ -1504,33 +1485,27 @@ without it.
### How the checks run ### How the checks run
Each check is a stage of the `Dockerfile`. A stage is one named step of the Each check is a stage of the `Dockerfile`, which is one named step of the image
image build. `script/test` builds the `test` stage, `script/lint` the `lint` build. `script/test` builds the `test` stage, `script/lint` the `lint` stage and
stage and `script/css-check` the `css-check` stage. `script/fmt-check` runs `script/css-check` the `css-check` stage. `script/fmt-check` runs `gofmt` on the
`gofmt` on the host and builds the `markdown-check` stage. Every `docker build` host and builds the `markdown-check` stage. Every `docker build` in `script/`
in `script/` passes `--no-cache`, so Docker reruns every step instead of reusing passes `--no-cache`, so Docker reruns every step instead of reusing an earlier
an earlier result. So a check that passes has really run. A bare result, and a check that reports success ran. A bare `docker build .` carries no
`docker build .` carries no such guarantee. such guarantee.
The `lint` stage runs these steps: The `lint` stage runs the `gofmt` check, `golangci-lint config verify` and
`golangci-lint run` with `--network=none`, and ESLint over `static/js/`. The
1. the `gofmt` check; `test` stage runs `go test -cover` on four packages at a time, each running at
2. `golangci-lint config verify`, with `--network=none`; most eight tests in parallel, with a 90 second timeout per package. On a failure
3. `golangci-lint run`, with `--network=none`; it runs only the failed tests again with `-v`.
4. ESLint over `static/js/`, in the `js-lint` stage, with `--network=none`.
The `test` stage runs `go test -cover` on four packages at a time, each running
at most eight tests in parallel, with a 90 second timeout per package. On a
failure it runs only the failed tests again with `-v`.
The image build depends on the `lint`, `test`, `css-check` and `markdown-check` The image build depends on the `lint`, `test`, `css-check` and `markdown-check`
stages, so an image that built is a tree that passed every check. The runtime stages, so an image that built is a tree that passed every check. The runtime
stage is Alpine Linux. It adds `ca-certificates` and `su-exec`, a tool that stage is Alpine with `ca-certificates` and `su-exec`, a tool that switches user,
switches user. It has no `sqlite3`. added. It has no `sqlite3`.
CI, the check run on every push, is the `check` workflow in The `check` workflow in `.gitea/workflows/check.yml` checks out the repository
`.gitea/workflows/check.yml`. It checks out the repository and runs and runs `script/cibuild`, nothing else.
`script/cibuild`, nothing else.
### Version stamping ### Version stamping
@@ -1552,29 +1527,30 @@ With no git metadata it is `unknown`.
`make build VERSION=v1.2.3` overrides it. `script/docker` and `script/cibuild` `make build VERSION=v1.2.3` overrides it. `script/docker` and `script/cibuild`
run the same `git describe` on the host and pass the result in as the `VERSION` run the same `git describe` on the host and pass the result in as the `VERSION`
build argument. Without that argument the image build reads the `.git` among the build argument. Without that argument the image build reads the `.git` in the
files sent to it, and fails if `.git` is present but the version comes out build context, and fails if `.git` is present but the version comes out empty,
empty, `dev` or `unknown`. `.dockerignore` leaves out every git `config` file, `dev` or `unknown`. `.dockerignore` leaves out every git `config` file, which
which can carry a credential, and keeps the rest of `.git`. can carry a credential, and keeps the rest of `.git`.
`make build` passes `-trimpath` and `-s -w`, so the binary carries no build `make build` passes `-trimpath` and `-s -w`, so the binary carries no build
paths, symbol table or debug information. paths, symbol table or debug information. Nothing that differs between two
builds of one commit is stamped in.
### Third-party browser assets ### Third-party browser assets
The pages use one third-party script: Alpine.js, a small JavaScript library. The pages use one third-party script: Alpine.js, a small JavaScript library.
They use its CSP build, `@alpinejs/csp`, which runs under a They use its CSP build, `@alpinejs/csp`, which runs under a
Content-Security-Policy that forbids `eval`. The standard build needs `eval`. Content-Security-Policy that forbids `eval`. The standard build needs `eval`.
With the CSP build, every Alpine.js directive in `templates/` names a component With the CSP build, every Alpine directive in `templates/` names a component
registered in `static/js/app.js`, such as `x-data="collapsible"`, and never an registered in `static/js/app.js`, such as `x-data="collapsible"`, and never an
inline expression. inline expression.
The package file is committed as `3p/alpinejs-csp-3.14.9.tgz`, byte for byte as The npm package file is committed as `3p/alpinejs-csp-3.14.9.tgz`, byte for byte
the npm registry, the JavaScript package registry, publishes it. `script/assets` as the registry publishes it. `script/assets` extracts `package/dist/cdn.min.js`
extracts `package/dist/cdn.min.js` from it to `static/js/alpine.min.js`, which from it to `static/js/alpine.min.js`, which the binary embeds. `make build`,
the binary embeds. `make build`, `make dev` and the Docker stages run it first. `make dev` and the Docker stages run it first. The extracted file is not
The extracted file is not committed, and a build that skips the extraction fails committed, and a build that skips the extraction fails with an error naming
with an error naming `js/alpine.min.js`. `js/alpine.min.js`.
To move to a new version: To move to a new version:
@@ -1625,7 +1601,7 @@ application are under [Entrypoint URLs](#entrypoint-urls).
ESLint. ESLint.
- `script/fmt` formats the Go code with `gofmt` and `goimports` on the host, and - `script/fmt` formats the Go code with `gofmt` and `goimports` on the host, and
the Markdown with prettier in Docker. It writes. the Markdown with prettier in Docker. It writes.
- `script/fmt-check` checks `gofmt` and prettier and fails instead of writing. - `script/fmt-check` runs the same checks and fails instead of writing.
- `script/css` regenerates `static/css/tailwind.css` in Docker. It writes. - `script/css` regenerates `static/css/tailwind.css` in Docker. It writes.
- `script/css-check` fails when `static/css/tailwind.css` differs from what - `script/css-check` fails when `static/css/tailwind.css` differs from what
`script/css` would generate. `script/css` would generate.