Author SHA1 Message Date
sneak c5812c9964 Show the last event as a relative time with a UTC tooltip (closes #567)
check / check (push) Waiting to run
The Statistics block's "Last event" figure now shows how long ago the
newest event arrived, as its two largest units ("59s", "1m1s", "3d4h"),
worked out when the page is served. Its tooltip is the full time in
UTC. With no event it still shows "none", without a tooltip.

Model: opus-5-5
2026-10-26 13:09:05 +00:00
clawbot eda8e80844 Fix six README sentences (closes #565)
check / check (push) Waiting to run
Rewrite the six sentences the documents check failed, each checked
against the code. The Sign-in limits section now opens with its rule.
The recovery routine's sentences say it resumes a delivery: it queues a
pending one at once, but gives a retrying one the rest of its wait.

Model: opus-5-5
2026-10-26 06:53:59 +01:00
clawbot e584718eb5 Commit Alpine.js to the repo instead of unpacking it at build time (closes #163)
check / check (push) Waiting to run
static/js/alpine.min.js is now committed: package/dist/cdn.min.js from
the @alpinejs/csp 3.14.9 npm package, byte for byte, taken once from the
package file that was in 3p/. The build no longer unpacks it, so
script/assets, the assets make target, its Dockerfile steps and the
ignore entries for the file are gone, and so is the package file.
Prettier and ESLint both ignore the file, so it stays as published.
Comments and the README now describe the committed file.

Model: opus-5-5
2026-10-26 05:16:05 +01:00
clawbot a8d226942e Rewrite TODO.md sentence by sentence (closes #558)
check / check (push) Canceled after 0s
Rewritten from scratch, sections kept, every statement checked against the code. Passed independent review in round 4.

Model: fable-5-1 (writing); opus-5-5 (review, merge)
2026-10-25 12:05:48 +01:00
clawbot 6df1fc7875 Rewrite README.md sentence by sentence (#561)
check / check (push) Canceled after 0s
Rewritten from scratch, about 4,600 words, every statement checked against the code. Part of #558; passed independent review in round 10.

Model: fable-5-1 (writing); opus-5-5 (review, merge)
2026-10-25 12:05:41 +01:00
19 changed files with 483 additions and 302 deletions
+1 -3
View File
@@ -72,10 +72,8 @@
**/*.sublime-*
# This repository's own host-built artifacts: the binary `make build`
# writes, and the Alpine.js file `make assets` extracts from 3p/ (the
# build extracts its own).
# writes.
/bin
/static/js/alpine.min.js
# SQLite databases, which hold the session key and webhook payloads, at
# any depth.
-4
View File
@@ -77,7 +77,3 @@ data/
# Temporary files
tmp/
temp/
# Alpine.js, extracted by `make assets` from its tarball in 3p/, which is
# what is committed.
/static/js/alpine.min.js
+1
View File
@@ -1,2 +1,3 @@
node_modules/
yarn.lock
static/js/alpine.min.js
Binary file not shown.
+2 -11
View File
@@ -92,10 +92,6 @@ COPY . .
# fmt-check` is the markdown-check stage above.
RUN if [ -n "$(gofmt -s -l .)" ]; then echo "gofmt needed on:"; gofmt -s -l .; exit 1; fi
# static/static.go embeds the Alpine.js file this extracts from 3p/; without
# it the static package does not compile and cannot be linted.
RUN script/assets
# The golangci-lint steps run with --network=none. `golangci-lint config
# verify` is documented as fetching its JSON schema over HTTPS; this pinned
# image resolves the schema without any network, and --network=none enforces
@@ -130,9 +126,6 @@ RUN go mod download
COPY . .
# static/static.go embeds the Alpine.js file this extracts from 3p/.
RUN script/assets
# -timeout applies to each package on its own, so 90s has only to clear the
# slowest one. -p 4 -parallel 8 keep the run under 2 GB of memory: at most
# four test binaries build or run at once, each with at most eight parallel
@@ -211,10 +204,8 @@ RUN version="$(make version VERSION="$VERSION")"; \
esac; \
fi
# Builds through the Makefile's build target, which runs script/assets
# (Alpine.js, extracted from its tarball in 3p/) first. The target builds
# with cgo off, so the binary is statically linked and runs on the Alpine
# runtime image below as it is.
# The Makefile's build target builds with cgo off, so the binary is
# statically linked and runs on the Alpine runtime image below.
RUN make build VERSION="$VERSION"
# Runtime stage
+1 -1
View File
@@ -17,7 +17,7 @@ COPY . .
# The test binary embeds the templates and static files, so the browser
# stage needs nothing else. It is built with cgo off and -p 4 keeps the
# compile's memory down, both as in the test phase of Dockerfile.
RUN make assets && CGO_ENABLED=0 go test -c -p 4 -tags browser -o /browser.test ./internal/server
RUN CGO_ENABLED=0 go test -c -p 4 -tags browser -o /browser.test ./internal/server
# chromedp/headless-shell:151.0.7922.109 (Debian trixie), 2026-08-11. The
# browser is on PATH as headless-shell, where the test's browser library
+3 -6
View File
@@ -1,4 +1,4 @@
.PHONY: bootstrap setup assets test test-browser lint fmt fmt-check check build run dev deps docker clean hooks css css-check version
.PHONY: bootstrap setup test test-browser lint fmt fmt-check check build run dev deps docker clean hooks css css-check version
# Default target
.DEFAULT_GOAL := check
@@ -22,9 +22,6 @@ bootstrap:
setup:
@script/setup
assets:
@script/assets
test:
@script/test
@@ -47,13 +44,13 @@ check:
# REPO_POLICIES.md does: the binary carries no build paths, symbol table or
# debug information, and on Linux, the image's included, it is statically
# linked.
build: assets
build:
CGO_ENABLED=0 go build -trimpath -ldflags '-s -w -X main.version=$(VERSION)' -o bin/webhooker ./cmd/webhooker
run: build
./bin/webhooker
dev: assets
dev:
go run ./cmd/webhooker
deps:
+241 -182
View File
@@ -1,8 +1,8 @@
# webhooker
webhooker is an MIT-licensed web service, written in [Go](https://golang.org) by
[@sneak](https://sneak.berlin), that receives the HTTP requests other services
send when something happens and forwards them to your own systems.
webhooker is an MIT-licensed [Go](https://golang.org) web service by
[@sneak](https://sneak.berlin) that receives the HTTP requests other services
send and forwards them to your own systems.
Many services send an HTTP request to a URL you give them when something
happens: a code push, a payment, a new ticket. webhooker receives those
@@ -30,8 +30,8 @@ Five words name the moving parts.
it. The UUID is a random 36-character identifier, and it is the only thing
that authorises a sender.
- A **target** is a place events are delivered to. There are four kinds: an HTTP
URL, a Slack or Mattermost message-posting URL, an archive file, and the
application log.
URL, the message-posting URL of a chat service (Slack or Mattermost), an
archive file, and the application log.
- An **event** is one stored request: its method, query string, headers and
body.
- A **delivery** is one event paired with one target. It records every attempt.
@@ -39,15 +39,15 @@ Five words name the moving parts.
The part of webhooker that answers requests to `/h/{uuid}` is the receiver. It
stores the event and one pending delivery per active target, then answers `200`.
Delivery is at least once, not exactly once. If a delivery reaches the target
but webhooker cannot record that outcome, it sends the delivery again later. A
delivery in flight when the service stops is also sent again at the next start.
webhooker adds no delivery identifier of its own to the request, so make your
targets tolerate a duplicate.
Delivery is at least once, not exactly once. A delivery can reach its target
more than once. If a delivery reaches the target but webhooker cannot record
that outcome, it sends the delivery again later. A delivery in flight when the
service stops is also sent again at the next start. webhooker adds no delivery
identifier of its own to the request, so make your targets tolerate a duplicate.
## Getting Started
You need the Go version named in `go.mod`, and Docker for the checks.
You need the Go version named in `go.mod`.
```bash
git clone https://git.eeqj.de/sneak/webhooker.git
@@ -57,7 +57,8 @@ DATA_DIR=../webhooker-data make dev
```
`DATA_DIR` is where webhooker keeps its databases. It defaults to
`/var/lib/webhooker`, so point it at a directory you can write to.
`/var/lib/webhooker`, so point it at a directory you can write to. `make build`
writes the program to `bin/webhooker`. Running that file starts webhooker.
The first start creates the `admin` account and prints its password once, in a
ruled box on standard output:
@@ -87,6 +88,9 @@ docker run -d \
webhooker
```
The first-boot password goes to the container's log. Read it with `docker logs`
and the container's name.
In production, put a reverse proxy in front of webhooker: a server that accepts
the public connections and passes them on. Read
[Behind a reverse proxy](#behind-a-reverse-proxy) first.
@@ -99,11 +103,12 @@ sooner after 24 hours without use, unless `SESSION_IDLE_TIMEOUT` says otherwise.
### Webhooks
**New Webhook** on the webhook list at `/hooks` asks for a name, an optional
description and a retention period in days. Events older than the retention
period are deleted, along with their deliveries. The default is 30 days. `0`
keeps events forever. Every new webhook gets one entrypoint, described as
`Default entrypoint`.
On the webhook list at `/hooks`, **New Webhook** asks for **Name**, an optional
**Description** and **Retention (days)**. Events older than the retention period
are deleted, along with their deliveries. The default is 30 days. `0` keeps
events forever. Filling in **HTTP target URL** creates an `http` target with the
webhook. Ticking **Archive** creates an archive target. Every new webhook gets
one entrypoint, described as `Default entrypoint`.
### Entrypoint URLs
@@ -119,21 +124,23 @@ deactivated entrypoint answers `410 Gone`. A deleted or unknown one answers
### Targets
**Add** under **Targets** asks for the type first, then that type's settings.
Under **Targets**, **Add** asks for the type first, then that type's settings.
- An `http` target posts each event to a URL, with optional request headers and
a timeout.
- A `slack` target posts each event as a message to the message-posting URL
Slack or Mattermost give out, which they call an incoming webhook.
- An `http` target posts each event to a URL, with optional **Headers** and a
**Timeout**.
- A `slack` target posts each event as a message. Its **Webhook URL** is the
message-posting URL Slack or Mattermost give out, which they call an incoming
webhook.
- An archive target writes each event as a row in a SQLite file of its own. Its
stored type is `database`. It takes an expiry, which is how long archived
events are kept, and a rotation, which is how often a new archive file is
started.
stored type is `database`. **Archive expiry** is how long archived events are
kept, and **Archive rotation** is how often a new archive file is started.
- A `log` target writes each event to the application log.
`http` and `slack` targets also take **Delivery attempts**: how many times each
delivery is tried in all. The default is 1. The most is 20. A target's type
cannot be changed. Deleting an archive target leaves its files on disk.
**Deactivate** and **Activate** on the webhook page switch a target off and on,
and new events skip a target that is off.
### Events
@@ -143,11 +150,11 @@ body** link. **Full Event Log** shows the 50 newest events with their deliveries
and every attempt. **Failed** and **Pending** narrow it to events with a failed
delivery, or with one still pending or retrying.
**Replay** sits beside each finished delivery. It creates a new delivery of the
same event to the same target, with the target's settings as they stand now.
**Resubmit** sits beside each event. It stores a copy of the event as a new
event and delivers it to every target that is active now. Both actions are
limited to 30 per minute per client.
A finished delivery has a **Replay** button while its target exists. **Replay**
creates a new delivery of the same event to the same target, with the target's
settings as they stand now. **Resubmit** sits beside each event. It stores a
copy of the event as a new event and delivers it to every target that is active
now. Both actions are limited to 30 per minute per client.
### Profile and settings
@@ -170,46 +177,55 @@ Do not set `BIND_ADDRESS` to an empty value in a container. An empty value means
unset, and unset falls back to `127.0.0.1`, so the container answers nothing on
its published port.
The app runs as user `webhooker`. The image's start-up script,
`deploy/docker-entrypoint.sh`, gives `DATA_DIR` and its contents to that user
The app runs as user `webhooker`. The image's start-up script is
`deploy/docker-entrypoint.sh`. It gives `DATA_DIR` and its contents to that user
before the app starts. So a host directory can be mounted as it is. webhooker
creates every SQLite file with mode `0600`.
creates every SQLite file readable and writable by that user alone (mode
`0600`).
### Running under upaas
[upaas](https://git.eeqj.de/sneak/upaas), a deployment tool, builds the image
from this repository's `Dockerfile` and runs it. Add no port mapping, because
upaas publishes every mapped port on all interfaces of the host. Put the app on
your reverse proxy's Docker network instead, and mount one host directory at
`/var/lib/webhooker`.
[upaas](https://git.eeqj.de/sneak/upaas) is a deployment tool. It builds the
image from this repository's `Dockerfile` and runs it. Add no port mapping,
because upaas publishes every mapped port on all interfaces of the host. Put the
app on your reverse proxy's Docker network instead, and mount one host directory
at `/var/lib/webhooker`. The first-boot password goes to the container's log.
Read it with `docker logs` and the name upaas gave the container.
### Behind a reverse proxy
webhooker speaks plain HTTP. It does not handle TLS, the encryption behind
HTTPS: it holds no certificate and accepts no HTTPS connections. The reverse
proxy holds the certificate and reaches webhooker over loopback, the address
only this host can reach.
proxy holds the certificate and reaches webhooker over a private address. A bare
host is a host without Docker. On a bare host the private address is loopback
(`127.0.0.1`), which only this host can reach. For the image it is the Docker
bridge, the private network Docker puts containers on.
Four things must be right.
1. **Keep the app port private.** Only the proxy may reach it: `127.0.0.1` on a
bare host, or a port published to `127.0.0.1` for the image.
2. **Send `X-Forwarded-Proto: https`.** This header tells webhooker the request
arrived over TLS. It decides the `Secure` flag on cookies, and whether a form
submission must carry a matching `Origin` or `Referer`. webhooker reads it
from any connecting machine, so the proxy must overwrite whatever the client
sent.
arrived over TLS. It decides the `Secure` flag on cookies, the flag that
makes a browser send them only over HTTPS. It also decides whether a form
submission must carry a matching `Origin` or `Referer` header. A browser
sends those two headers to say which site or page a request came from.
webhooker reads `X-Forwarded-Proto` from any connecting machine, so the proxy
must overwrite whatever the client sent.
3. **Cover the proxy in `TRUSTED_PROXIES`.** Otherwise every client shares one
rate-limit bucket, the count a limit keeps per client. The default list does
not cover a proxy that connects over loopback.
4. **Send `Host` as `$http_host`, not `$host`.** webhooker checks a form's
`Origin` against the `Host` header it receives. `$host` drops the port, so on
any port other than 443 every form submission fails with `403`.
rate-limit bucket, the count a limit keeps per client. A proxy on the host
reaches the container from the Docker bridge address, such as `172.17.0.1`.
The default list covers it. The list does not cover loopback, so on a bare
host set `TRUSTED_PROXIES=127.0.0.1`.
4. **Pass on the client's `Host` header with its port.** `Host` is the header
that names the site the client asked for. webhooker checks a form's `Origin`
against the `Host` header it receives. A `Host` without the port makes every
form submission fail with `403` on any port other than `443`. For the nginx
reverse proxy that means `$http_host`, not `$host`.
Protect the proxy's access log: its request paths hold the entrypoint UUIDs.
An nginx server block for this, with `TRUSTED_PROXIES=127.0.0.1` on webhooker's
side:
An nginx server block for a bare host:
```nginx
server {
@@ -229,14 +245,19 @@ server {
}
```
The block matches webhooker's 1 MB body cap and its 60 second request timeout.
The block matches webhooker's 1 MB body cap. Its `proxy_read_timeout` is longer
than webhooker's 60 second request timeout, so webhooker's own `504` reaches the
client first.
### HSTS
HSTS is a response header that tells browsers to use HTTPS only for this
hostname and every name under it, for `max-age` seconds. Every response carries
it unless `HSTS_MAX_AGE` is `0`. A browser keeps the rule until `max-age` runs
out, so serve webhooker on a hostname you will keep on HTTPS.
out, so serve webhooker on a hostname you will keep on HTTPS. The header always
carries `preload`. On a whole domain such as `example.com`, that lets the domain
be added to the HTTPS-only list that browsers ship with. That listing is hard to
undo.
## Recovering a lost admin password
@@ -248,30 +269,39 @@ the server does and refuses while a running instance holds it.
DATA_DIR=/var/lib/webhooker webhooker resetpw -generate admin
```
Without `-generate`, the new password is read from standard input, at least 8
characters. It is never taken as a command-line argument, because other accounts
on the host can read those. In a container, run the same command with
`docker run --rm` and the stopped container's volume mounted. The command
creates nothing: a missing `DATA_DIR`, a missing `webhooker.db` or an unknown
username is an error.
`-generate` makes up a random password and prints it. Without `-generate`, the
new password is read from standard input. It must be at least 8 bytes long. It
is never taken as a command-line argument, because other accounts on the host
can read those. The command creates nothing: a missing `DATA_DIR`, a missing
`webhooker.db` or an unknown username is an error.
In a container, run the binary at `/app/webhooker` with the stopped container's
volume mounted:
```bash
docker run --rm -v /path/to/data:/var/lib/webhooker \
webhooker /app/webhooker resetpw -generate admin
```
## Backup and restore
All state lives in `DATA_DIR`. Back up the whole directory. It holds:
- the main database, `webhooker.db`: the session key, the `admin` account's
password hash, and the webhooks, entrypoints and targets;
- the main database, `webhooker.db`: the session key (which signs the session
cookies), the `admin` account's password hash, and the webhooks, entrypoints
and targets;
- one event database per webhook, `events-{webhook_uuid}.db`: its events,
deliveries and attempts;
- the archive files, one per archive target;
- the files of each archive target;
- the lock file, `webhooker.lock`, which is always empty.
Every database runs in WAL mode. WAL is SQLite's write-ahead log: finished
Every database runs in WAL mode. WAL is SQLite's write-ahead log. Finished
writes land in a `-wal` file beside the database before they reach the `.db`
itself. A copy of the `.db` without its `-wal` is missing data. A clean shutdown
removes the `-wal` and `-shm` files; a crash leaves them, and they must be
carried with the `.db`. Keep `DATA_DIR` on a local disk: WAL mode is not
reliable on network filesystems, and webhooker does not check for one.
itself. A `-shm` file beside it indexes that log. A copy of the `.db` without
its `-wal` is missing data. A clean shutdown removes the `-wal` and `-shm`
files; a crash leaves them, and they must be carried with the `.db`. Keep
`DATA_DIR` on a local disk: WAL mode is not reliable on network filesystems, and
webhooker does not check for one.
A copy taken while the service runs can catch a `.db` and its `-wal` at two
different instants. Stop the service, copy the directory, and start it again.
@@ -285,7 +315,7 @@ done
```
Each file is captured at its own instant. A webhook created during the backup
can be in the copy of `webhooker.db` with no event database in the backup; a
can be in the copy of `webhooker.db` with no event database in the backup. A
restore then creates that event database empty.
To restore, stop the service and put back every file together, with any `-wal`
@@ -303,11 +333,11 @@ plain text. Protect it accordingly.
### The entrypoint URL is the credential
Each entrypoint is a random version 4 UUID served at `/h/{uuid}`. Holding that
URL is what authorises a sender. The receiver checks nothing else: no shared
secret, no HMAC signature (a hash of the body made with a shared key), no token
header. Signature headers a sender adds anyway are stored and forwarded as
ordinary headers.
Each entrypoint is a random UUID served at `/h/{uuid}`. Holding that URL is what
authorises a sender. The receiver checks nothing else: no shared secret, no HMAC
signature (a hash of the body made with a shared key), no token header. If a
sender adds signature headers anyway, they are stored and forwarded as ordinary
headers.
Keep the URL out of tickets, chat and screenshots. webhooker's own log never
writes it. The reverse proxy's access log does hold it, in the request path.
@@ -316,31 +346,37 @@ writes it. The reverse proxy's access log does hold it, in the request path.
- Passwords are hashed with Argon2id, a slow password hashing function.
- Session cookies are signed, `HttpOnly` (hidden from page scripts) and
`SameSite=Lax` (not sent with cross-site form posts). They carry `Secure` when
the request carries `X-Forwarded-Proto: https`.
- Every form is protected against cross-site request forgery (CSRF), where
another site makes your browser submit a form here: each form carries a token
tied to a cookie, and a submission's `Origin` must name this host.
`SameSite=Lax` (not sent with cross-site form posts). They carry `Secure`
(sent only over HTTPS) when the request carries `X-Forwarded-Proto: https`.
- Every form is protected against cross-site request forgery (CSRF). CSRF is
when another site makes your browser submit a form here. Each form carries a
token tied to a cookie, and a submission's `Origin` must name this host.
- Form bodies are capped at 1 MB.
- Every response carries these headers:
- `Strict-Transport-Security`, unless `HSTS_MAX_AGE` is `0`;
- `X-Content-Type-Options: nosniff`;
- `X-Frame-Options: DENY`;
- a Content-Security-Policy (CSP), the header that tells the browser where
scripts and styles may come from, set to
`default-src 'self'; script-src 'self'; style-src 'self'`.
- `Strict-Transport-Security`, the HSTS header, unless `HSTS_MAX_AGE` is
`0`;
- `X-Content-Type-Options: nosniff`, which stops the browser guessing a
file's type;
- `X-Frame-Options: DENY`, which stops other sites showing these pages in a
frame;
- `Content-Security-Policy: default-src 'self'; script-src 'self'; style-src 'self'`,
which tells the browser where scripts and styles may come from;
- `Referrer-Policy: strict-origin-when-cross-origin`, which tells another
site only which site a visitor came from, not which page;
- `Permissions-Policy: camera=(), microphone=(), geolocation=()`, which
denies the pages the camera, microphone and location.
- `/metrics` sits behind HTTP basic authentication, a username and password sent
with each request.
### Sign-in limits
The password is checked first, and only a failed sign-in is counted. A correct
password is never throttled. Failures are counted per client and submitted
username. From the fifth failure within a minute, each failure is answered
`429`. At most two password checks run at once; a request that gets no turn
within 5 seconds is answered `503`. So those two checks, not the failure count,
bound how fast passwords can be guessed. Choose a long random admin password, or
rate-limit `POST /pages/login` at the reverse proxy.
From the fifth failure within a minute, each failure is answered `429`. Only
failures are counted. A correct password is never answered `429`. Failures are
counted per client and submitted username. At most two password checks run at
once; a request that gets no turn within 5 seconds is answered `503`. So
guessing speed is bounded by the limit of two checks at a time, not by the
failure count. Choose a long random `admin` password, or rate-limit
`POST /pages/login` at the reverse proxy.
### Delivery destinations
@@ -348,8 +384,9 @@ A delivery target could be used to make webhooker send requests into the network
it sits in and read the answers back. This is called server-side request forgery
(SSRF). webhooker refuses it with the SSRF guard.
Every target URL must use `http` or `https` and resolve to a public address. The
SSRF guard refuses these ranges:
Every target URL must use `http` or `https`. By default its hostname must also
point at a public address, one reachable from the internet. The SSRF guard
refuses these ranges:
- loopback, `127.0.0.0/8` and `::1`;
- the RFC 1918 private ranges, `10.0.0.0/8`, `172.16.0.0/12` and
@@ -357,17 +394,20 @@ SSRF guard refuses these ranges:
- link-local space, which is valid only on one network segment;
- carrier-grade NAT space, which internet providers use inside their own
networks;
- multicast, the IPv6 unique local range, and the other reserved ranges.
- multicast, the addresses that reach a group of hosts at once;
- the IPv6 unique local range, `fc00::/7`, the private space of IPv6;
- the other reserved ranges.
The SSRF guard runs when a target is saved and again when each delivery
connects, so a hostname that changes its address later is still checked.
connects. So a hostname that changes its address later is still checked.
Redirects are followed up to ten hops, and each hop is checked. A hop to another
host or port, or from `https` to `http`, carries none of the configured or
forwarded headers, so a redirect cannot pass a credential to another host.
host or port, or from `https` to `http`, drops every configured or forwarded
header except `Content-Type`, which names the body's format. So a redirect
cannot pass a credential to another host.
Egress means the connections webhooker opens outward to deliver.
[`ALLOWED_EGRESS_CIDRS`](#allowing-egress-to-your-own-network) opens named
networks to it.
[`ALLOWED_EGRESS_CIDRS`](#allowing-egress-to-your-own-network) lets deliveries
reach networks you name.
### What the logs contain
@@ -381,8 +421,9 @@ The `admin` password is printed in the clear on the first boot and by
query string, headers and body.
When `SENTRY_DSN` is set, error reports go to Sentry, an error reporting
service. A report carries no query string, body or cookies, and the route
pattern (the route as registered, such as `/h/{uuid}`) in place of the path.
service. A report carries no query string, body or cookies. It carries the route
pattern in place of the path. A route pattern is the path with its variable part
left as a name, such as `/h/{uuid}`.
## Design
@@ -391,21 +432,23 @@ sends them on in the background, and SQLite holds everything in between.
The receiver stores the event and one pending delivery per active target in one
database write, then queues the deliveries. Background workers run them and
record each attempt. At start and then every minute, a recovery routine queues
again any delivery left `pending` or `retrying`, so a stop or a crash loses no
delivery.
record each attempt. At start, a recovery routine resumes every delivery left
`pending` or `retrying`. It then runs every minute. Each run resumes every
`retrying` delivery that the engine is not already handling. It also resumes a
`pending` delivery that has sat untouched for 15 minutes. So a stop or a crash
loses no delivery.
The main database holds configuration. Each webhook has its own event database,
so writes for one webhook never wait on another webhook's database. The Go code
is under `internal/`, one package per concern, and the pages' HTML, CSS and
JavaScript are embedded in the binary.
so writes for one webhook never wait on another webhook's database. The program
starts in `cmd/webhooker/main.go`, and its logic is under `internal/`. The
`static` and `templates` packages embed the pages' HTML, CSS and JavaScript in
the binary.
## Configuration reference
All configuration comes from environment variables. For development, variables
can also go in a `.env` file in the working directory. A variable set in the
real environment wins over the file. A `.env` file that cannot be parsed aborts
startup.
All configuration comes from environment variables. Variables can also go in a
`.env` file in the working directory. A variable set in the real environment
wins over the file. A `.env` file that cannot be parsed aborts startup.
A variable that is unset or empty takes its default. A variable that is set but
cannot be parsed aborts startup with a message naming it. A Go duration is a
@@ -414,16 +457,20 @@ number with a unit, such as `30s`, `15m` or `24h`.
### `WEBHOOKER_ENVIRONMENT`
`dev` or `prod`. Default `prod`. This controls one thing: CORS, the headers that
let a page on another site call this one from a browser. In `dev`, every origin
is allowed. In `prod`, no CORS headers are sent.
let a page on another site call this one from a browser. An origin is a site as
a browser names it: scheme, host and port. In `dev`, every origin is allowed. In
`prod`, no CORS headers are sent.
### `PORT`
The TCP port the listener binds. Default `8080`.
The port webhooker listens on. Default `8080`. Docker marks the image's
container healthy by a check every 30 seconds. The check requests port `8080`
whatever `PORT` says, so a container started with another `PORT` reports
unhealthy.
### `BIND_ADDRESS`
The IP address the listener binds. Default `127.0.0.1`. The Docker image sets
The IP address webhooker listens on. Default `127.0.0.1`. The Docker image sets
`0.0.0.0`. It must be an IP address literal: `localhost` and a value with a port
are refused. `0.0.0.0` listens on IPv4 only; `::` listens on IPv6 and IPv4.
@@ -432,17 +479,17 @@ are refused. `0.0.0.0` listens on IPv4 only; `::` listens on IPv6 and IPv4.
The directory for every SQLite database and the lock file. Default
`/var/lib/webhooker`. It is created if it is missing.
Exactly one webhooker process may use a `DATA_DIR` at a time; two would both run
the recovery routine and send every pending delivery twice. At startup the
process locks `{DATA_DIR}/webhooker.lock` with `flock`, the kernel's file lock,
and a second process pointed at the same directory exits with a message naming
it.
Exactly one webhooker process may use a `DATA_DIR` at a time. Two would both run
the recovery routine and could send one delivery twice. At startup the process
locks `{DATA_DIR}/webhooker.lock` with `flock`, the kernel's file lock. A second
process pointed at the same directory exits with a message that names the
directory and the lock file.
### `DEBUG`
`true` or `false`. Default `false`. It accepts the spellings Go's
`strconv.ParseBool` accepts; `yes` and `on` are refused. `DEBUG=true` lowers the
log level to `DEBUG`, which adds every SQL statement, with `?` in place of each
log level to `DEBUG`, which adds SQL statements, with `?` in place of each
value, and other diagnostic lines.
### `METRICS_USERNAME` and `METRICS_PASSWORD`
@@ -491,11 +538,11 @@ private ranges. A set value replaces the default.
The list decides whose `X-Forwarded-For` header is believed. `X-Forwarded-For`
is the header a proxy uses to pass on the address of the client it is serving.
It is read only when the connecting machine is inside the list. `X-Real-IP` and
`True-Client-IP` are never read. If any client can reach webhooker, or the proxy
in front of it, from a private address, set the list to the proxy's address
alone. Otherwise those clients can choose their own `X-Forwarded-For` and bypass
every rate limit.
It is read only when the connecting machine is inside the list. webhooker never
reads `X-Real-IP` or `True-Client-IP`, two other headers with the same job. Some
clients may reach webhooker, or the proxy in front of it, from a private
address. If so, set the list to the proxy's address alone. Otherwise those
clients can choose their own `X-Forwarded-For` and bypass every rate limit.
### Allowing egress to your own network
@@ -504,13 +551,15 @@ that delivery targets may reach although the SSRF guard refuses them by default.
Default empty. Example: `10.0.0.0/8, 172.17.0.0/16`. When the list is set,
startup logs it at `WARN`.
Anyone with the admin account can make webhooker send requests to every network
you list, and read the responses in the event log. List the smallest blocks that
cover your destinations, and never `0.0.0.0/0` or `::/0`.
Anyone with the `admin` account can make webhooker send requests to every
network you list, and read the responses in **Full Event Log**. List the
smallest networks that cover your destinations, and never `0.0.0.0/0` or `::/0`.
These addresses stay blocked whatever you list. They are link-local space, the
unspecified addresses, and cloud metadata endpoints, the addresses a cloud gives
each virtual machine for its own credentials:
unspecified addresses and cloud metadata endpoints. The unspecified addresses
are `0.0.0.0` and `::`, which name no host. A cloud metadata endpoint is the
address a cloud gives each virtual machine for its own credentials. The blocked
addresses are:
- `169.254.0.0/16` and `fe80::/10`;
- `fd00:ec2::254/128` and `fd00:ec2::23/128`;
@@ -523,30 +572,32 @@ each virtual machine for its own credentials:
- `0.0.0.0/32` and `::/128`;
- `::a9fe:a9fe/128` and `64:ff9b::a9fe:a9fe/128`.
This list is not every cloud's metadata address. If yours is not here, do not
list the block that contains it. Azure's public `168.63.129.16` is refused by
default but is reopened by a list that covers it.
Some clouds' metadata addresses are not on this list. If yours is not here, do
not list the network that contains it. Azure's public metadata address,
`168.63.129.16`, is refused by default but is reopened by a list that covers it.
## Delivery reference
An `http` target posts the event body to its URL with the event's `Content-Type`
and headers, then the target's configured headers, which replace event headers
of the same name. A `2xx` response is a success. The first 4 KiB of the response
body is stored with the attempt. `timeout_seconds` is at most 300, and blank
means the 30 second default. `forward_query` appends the event's query string to
the URL.
An `http` target posts the event body to its URL. The request carries the
event's `Content-Type` and headers. The target's configured **Headers** are
added next and replace event headers of the same name. Every `http` and `slack`
request carries `User-Agent: webhooker/{version}`. `User-Agent` is the header
that names the sending program. The **Headers** field refuses `User-Agent`. A
response with a status from `200` to `299` is a success. The first 4 KiB of the
response body is stored with the attempt. **Timeout** is at most 300 seconds,
and blank means the 30 second default. **Pass the query string on to this
target** appends the event's query string to the URL.
A `slack` target posts a message that describes the event and shows its body in
a code block, cut at 3500 bytes. An archive target, whose stored type is
`database`, appends the event as a row to its own SQLite file. A `log` target
writes the event as one `INFO` line.
a code block, cut at 3500 bytes. An archive target appends the event as a row to
its own SQLite file. A `log` target writes the event as one `INFO` line.
`http` and `slack` targets take `max_attempts`, from 1 to 20. A failed attempt
marks the delivery `retrying` and schedules the next attempt; the wait doubles
each time, starting at one second. After the last allowed attempt fails, the
delivery is marked `failed`.
**Delivery attempts** is from 1 to 20. A failed attempt marks the delivery
`retrying` and schedules the next attempt; the wait doubles each time, starting
at one second. After the last allowed attempt fails, the delivery is marked
`failed`.
A target with `max_attempts` above 1 has a circuit breaker, which stops
A target with more than one delivery attempt has a circuit breaker, which stops
hammering a destination that is down. After 5 failures in a row, deliveries to
the target pause for 30 seconds. Then one delivery is sent as a probe; the
others wait until a probe succeeds. The webhook page says when a target is
@@ -557,11 +608,12 @@ paused.
An archive target writes to
`archive-{webhook_name}-{target_name}-{target_uuid}.db` in `DATA_DIR`, the two
names made safe for a file name. A rotation of `monthly`, `daily` or `hourly`
starts a new file for each period, in UTC, with the period before `.db`, such as
`-2026-10-01.db`. Rows older than the expiry are deleted on each write and by
the archive sweep. Renaming the webhook or the target renames the files.
**Download** on the webhook page returns the target's whole archive as one
gzipped JSON file.
starts a new file for each period. Periods follow UTC, the universal time that
ignores local time zones. The period goes before `.db`, such as
`-2026-10-01.db`. Rows older than the expiry are deleted each time a file is
opened and by the archive sweep. Renaming the webhook or the target renames the
files. **Download** on the webhook page returns the target's whole archive as
one JSON file compressed with gzip.
## HTTP routes reference
@@ -572,6 +624,8 @@ gzipped JSON file.
- an unknown UUID gets `404`;
- a deactivated entrypoint gets `410`;
- a body over 1 MB gets `413`;
- a body that cannot be read in full gets `400`, for example one still
arriving after the server's 10 second read limit;
- a failure to store the event gets `500`;
- a stored event gets `200` with `{"status":"ok"}`.
- `GET /pages/login` shows the sign-in form, `POST /pages/login` signs in, and
@@ -585,9 +639,9 @@ gzipped JSON file.
## Metrics reference
`/metrics` serves these metrics for Prometheus, the monitoring system, plus the
Go runtime and process metrics the Prometheus library provides. Each delivery
metric carries one label, `target_type`, which is `http`, `slack`, `database`,
`log` or `unknown`.
Go runtime and process metrics the Prometheus library provides. A label is a
name and value attached to a metric. Each delivery metric carries one label,
`target_type`, which is `http`, `slack`, `database`, `log` or `unknown`.
- `webhooker_events_received_total` and `webhooker_events_resubmitted_total`,
two counters (totals that only grow) with no label;
@@ -595,21 +649,23 @@ metric carries one label, `target_type`, which is `http`, `slack`, `database`,
- `webhooker_deliveries_succeeded_total` and
`webhooker_deliveries_failed_total`;
- `webhooker_delivery_retries_total` and `webhooker_delivery_replays_total`;
- `webhooker_delivery_duration_seconds`, a histogram (a distribution) of one
attempt's wall time;
- `webhooker_delivery_duration_seconds`, a histogram (a distribution) of how
long one attempt took;
- `webhooker_deliveries_pending` and `webhooker_deliveries_retrying`, gauges
(current values) sampled every 30 seconds;
- `webhooker_circuit_breakers_open`.
Three inbound HTTP metrics, named `http_...`, carry the route pattern as their
`handler` label, never the real path.
`http_request_duration_seconds` and `http_response_size_bytes` measure inbound
requests. Their `handler` label is the route pattern, never the real path.
`http_requests_inflight` counts all routes together.
## Development
You need the Go version named in `go.mod`, and Docker. The tests, the linters,
prettier and the stylesheet build run in Docker, at versions pinned (fixed to
one exact release) in the `Dockerfile`, `package.json` and `yarn.lock`. Only
`gofmt` and `goimports` run on the host.
`prettier` and the stylesheet build run in Docker. Their versions are pinned
(fixed to one exact release) in the `Dockerfile`, `package.json` and
`yarn.lock`. `gofmt` runs on the host. `goimports` runs on the host too, when it
is installed. `script/bootstrap` does not install it.
```bash
make bootstrap # Install git, make, Go and the Go modules
@@ -622,25 +678,27 @@ make build # Build bin/webhooker, version-stamped
make docker # Build the Docker image
```
Each `make` target except `build`, `run`, `dev`, `deps` and `clean` runs a
script in `script/`, listed under [Entrypoints](#entrypoints). Every
`docker build` in `script/` passes `--no-cache`, so a check that passes has
really run.
Most `make` targets run a script in `script/`, listed under
[Entrypoints](#entrypoints). `build`, `run`, `dev`, `deps` and `clean` run their
own commands. Every `docker build` in `script/` passes `--no-cache`, so a check
that passes has really run.
The binary's version comes from `script/version`: `git describe` on the
checkout, or `VERSION` when that is set. It is shown at
`/.well-known/healthcheck` and in the page footer.
The pages use one third-party script, Alpine.js, in its CSP build
`@alpinejs/csp`, which runs under a Content-Security-Policy that forbids `eval`.
The package file in `3p/` is byte for byte the one the npm registry, the
JavaScript package registry, publishes.
The pages use one third-party script, Alpine.js. They load `@alpinejs/csp`, the
version of it that works when the Content-Security-Policy forbids running text
as code. The npm registry is where JavaScript packages are published.
`static/js/alpine.min.js` is committed byte for byte as the npm registry
publishes it.
A browser test in `internal/server` loads the pages in a headless browser, one
without a window, under the real Content-Security-Policy. `make test` leaves it
out; run `make test-browser` after changing `templates/` or `static/js/`.
`static/css/tailwind.css` is generated by Tailwind and committed; `make check`
fails when it is out of date, and `make css` regenerates it.
A headless browser is a browser without a window. A browser test in
`internal/server` loads the pages in one under the real Content-Security-Policy.
`make test` leaves it out; run `make test-browser` after changing `templates/`
or `static/js/`. `static/css/tailwind.css` is generated by Tailwind, a CSS
generator. The file is committed. `make check` fails when it is out of date, and
`make css` regenerates it.
## Entrypoints
@@ -653,24 +711,25 @@ application are under [Entrypoint URLs](#entrypoint-urls).
- `script/bootstrap` installs git, make, Go and the Go modules.
- `script/setup` runs `script/bootstrap`, then `script/install-precommit`.
- `script/projectname` prints `webhooker`.
- `script/assets` extracts Alpine.js from `3p/` to `static/js/alpine.min.js`.
- `script/test` runs the tests in Docker.
- `script/test-browser` runs the browser test in Docker.
- `script/lint` runs `gofmt`, `golangci-lint` and ESLint in Docker.
- `script/fmt` formats the Go code on the host and the Markdown in Docker.
- `script/fmt-check` checks the same and fails instead of writing.
- `script/fmt` formats the Go code with `gofmt` on the host and the Markdown
with `prettier` in Docker. It also runs `goimports` when that is installed.
- `script/fmt-check` fails when `gofmt` or `prettier` would change a file. It
does not run `goimports`, and it writes nothing.
- `script/css` regenerates `static/css/tailwind.css` in Docker.
- `script/css-check` fails when `static/css/tailwind.css` is out of date.
- `script/check` runs `script/test`, `script/lint`, `script/fmt-check` and
`script/css-check`.
- `script/version` prints the version to stamp into the binary.
- `script/docker` builds the image tagged `webhooker`.
- `script/cibuild` is what CI, the check run on every push, runs:
- CI is the check that runs on every push. It runs `script/cibuild`:
`script/bootstrap`, `script/check`, then the image build.
- `script/precommit` checks that `go mod tidy` changes nothing, then runs
`script/check`.
- `script/install-precommit` installs the git pre-commit hook that runs
`script/precommit`.
- `script/install-precommit` installs the git hook that runs `script/precommit`
before each commit.
## TODO
+51 -46
View File
@@ -1,72 +1,77 @@
# Workflow
One issue per unit of work, one branch and one PR per issue:
One issue per unit of work, and one branch and one PR per issue:
- ensure a tracked issue exists with a definition of done
- branch from `next` (never from `main`)
- do the work; open a PR based on `next` (never on `main`)
- pass an independent review, then the manager squash-merges into `next`
- push; nothing stays local-only
- make sure a tracked issue exists with a definition of done
- branch from `next`, never from `main`
- do the work and push the branch
- open a PR with base `next`, never `main`
- pass an independent review
- the repository manager squash-merges it into `next`
`next` is the branch for the next milestone and must stay green and mergeable to
`main` without notice. One `next` -> `main` PR accumulates the milestone;
releases are cut from `main` separately.
The repository manager is the role that merges each issue's PR into `next`.
Nothing stays local-only: every commit is pushed.
The tracker (https://git.eeqj.de/sneak/webhooker/issues) is the record of what
is done and what is under way. This file keeps no list of either.
`next` is the branch for the next milestone. It must stay green and mergeable
into `main` at all times. A milestone's work reaches `main` through one PR from
`next`. The owner, @sneak, merges that PR. Cutting a release from `main` is a
separate step after that.
The tracker (https://git.eeqj.de/sneak/webhooker/issues) records what is done
and what is under way. This file keeps no list of either.
# Status
What is still open for 1.0.0 is in its milestone
(https://git.eeqj.de/sneak/webhooker/milestone/9). `v1.0.0` is not tagged, and
only @sneak tags it.
(https://git.eeqj.de/sneak/webhooker/milestone/9). `v1.0.0` is not tagged. Only
@sneak tags it.
Delivery is at-least-once by design, not by accident: a send whose result row
does not land is attempted again, so a receiver can see a duplicate. That is
deliberate — the alternative is a silent lost delivery — and the README says so
under Rationale. It is not a defect to re-file.
Delivery is at least once by design. When webhooker cannot record the outcome of
a delivery, it sends that delivery again. The target can then receive the same
event twice. The alternative is a lost delivery that nothing reports. The README
says so. Such a duplicate is not a defect to re-file.
# Next Step
@sneak runs webhooker in limited production, then tags `v1.0.0`. Merging `next`
into `main` is a separate act from tagging and waits on neither of those: `next`
is kept mergeable at all times, which is the point of the branch.
into `main` is a separate act. It waits on neither the production run nor the
tag.
# Future Steps
- Per-webhook rate limits, set in the web UI and enforced in the receiver
handler, on top of the receiver's existing limits, which `RECEIVER_RATE_LIMIT`
sets for the whole server (see Rate Limiting in the README)
- API key authentication for programmatic access (the `APIKey` model exists, but
nothing creates a key and no middleware checks one)
- REST API v1
- CRUD for webhooks, entrypoints, targets
- Per-webhook rate limits, set in the web UI and enforced at the entrypoint.
Today `RECEIVER_RATE_LIMIT` is the only rate-limit setting. It is how many
requests per minute one client address may send to one entrypoint. The same
address is also capped at ten times that across all entrypoints together.
- API key authentication for programmatic access. The `APIKey` model and its
`api_keys` table exist. Nothing creates a key and nothing checks one.
- REST API v1. `/api/v1` is mounted with no routes today.
- CRUD for webhooks, entrypoints and targets
- event viewing and filtering endpoints
- event redelivery endpoint
- OpenAPI specification
- Analytics in the web UI: response times summarised per webhook or per target,
and figures for all webhooks together. The webhook page already shows event,
delivery and failure counts and failure percentages, the target list counts
each target's deliveries, the event log and each event's page show every
attempt's duration, and `/metrics` carries server-wide counts and delivery
durations
- A remember-me option at login. Every session already survives a browser
restart: it lasts up to 7 days, and ends sooner after `SESSION_IDLE_TIMEOUT`
(default 24 hours) without activity
- Password reset flow for a forgotten password over the web. The authenticated
password _change_ flow already landed, and a lost password is recoverable from
the console with `webhooker resetpw`
(https://git.eeqj.de/sneak/webhooker/issues/208)
and figures for all webhooks together. Today the webhook page shows event,
delivery and failure counts and failure percentages. The target list counts
each target's deliveries. The event log shows every attempt's duration.
`/metrics` carries server-wide counts and delivery durations.
- A remember-me option at login. Today every session survives a browser restart.
It lasts up to 7 days. It ends sooner after `SESSION_IDLE_TIMEOUT` (default 24
hours) without use.
- Password reset over the web for a forgotten password. Today a signed-in user
can change their password. A lost password is reset from the command line with
`webhooker resetpw` (https://git.eeqj.de/sneak/webhooker/issues/208).
- Later, nice to have
- email delivery target type
- SNS and S3 delivery targets
- data transformations (e.g. a configurable Slack message format; the Slack
target sends one fixed format today)
- JSONL file delivery with periodic S3 upload (the `database` target already
archives events to SQLite files, rotated hourly, daily or monthly if set,
and the webhook page offers them as one gzipped JSON download)
- data transformations, such as a configurable Slack message format. The
Slack target sends one fixed format today.
- JSONL file delivery with periodic S3 upload. Today an archive target
writes events to SQLite files. An archive target can rotate those files
hourly, daily or monthly. The webhook page offers them as one gzipped JSON
download.
- webhook event search, and filtering beyond the event log's failed and
pending views
- multi-user with role-based access control (each webhook already belongs to
one account, but there is only the account made at first boot and no way
to add another)
- multi-user with role-based access control. Today each webhook belongs to
one account. Only the account made at first boot exists. There is no way
to add another.
+1 -1
View File
@@ -5,7 +5,7 @@
// REPO_POLICIES.md states that a linter can check: const for everything,
// let only for a variable that is reassigned, never var.
export default [
// Alpine.js, extracted from 3p/ by make assets; not ours to lint.
// Alpine.js, committed as published; not ours to lint.
{ ignores: ["static/js/alpine.min.js"] },
{
// The pages load static/js/app.js as a classic script, not a module.
+6
View File
@@ -96,6 +96,12 @@ func (s *Handlers) WebhookStatsForTest(webhookID string) *WebhookStats {
return s.loadWebhookStats(webhookID, entrypoints, targets)
}
// RelativeTimeForTest exposes relativeTime for use in the
// handlers_test package.
func RelativeTimeForTest(at, now time.Time) string {
return relativeTime(at, now)
}
// FinishedByTargetForTest exposes finishedByTarget for use in the
// handlers_test package.
func FinishedByTargetForTest(
+55 -6
View File
@@ -15,6 +15,9 @@ const (
longWindow = 24 * time.Hour
)
// day is the largest unit relativeTime writes.
const day = 24 * time.Hour
// percent turns a fraction into a percentage.
const percent = 100
@@ -35,8 +38,12 @@ type WebhookStats struct {
InProgress int64
// LastEventAt is when the newest event arrived, or nil when none
// has. Retention does not change it.
LastEventAt *time.Time
// has. Retention does not change it. LastEvent is how long ago
// that was, as relativeTime writes it, and LastEventUTC the full
// time; both are empty when no event has arrived.
LastEventAt *time.Time
LastEvent string
LastEventUTC string
Last10Minutes RecentWindow
Last24Hours RecentWindow
@@ -81,6 +88,42 @@ func (w RecentWindow) FailurePercent() string {
)
}
// relativeTime is how long before now the time at was, in whole
// seconds, written as its two largest units out of d, h, m and s:
// "59s", "1m1s", "23h59m", "3d4h". A second unit of zero is left off,
// as in "5m". A time after now is "0s".
func relativeTime(at, now time.Time) string {
elapsed := max(now.Sub(at), 0)
days := int64(elapsed / day)
hours := int64(elapsed % day / time.Hour)
minutes := int64(elapsed % time.Hour / time.Minute)
seconds := int64(elapsed % time.Minute / time.Second)
switch {
case days > 0:
return twoUnits(days, "d", hours, "h")
case hours > 0:
return twoUnits(hours, "h", minutes, "m")
case minutes > 0:
return twoUnits(minutes, "m", seconds, "s")
default:
return fmt.Sprintf("%ds", seconds)
}
}
// twoUnits writes a count of a larger unit followed by a count of the
// next smaller one, leaving the smaller off when it is zero.
func twoUnits(
larger int64, largerUnit string, smaller int64, smallerUnit string,
) string {
if smaller == 0 {
return fmt.Sprintf("%d%s", larger, largerUnit)
}
return fmt.Sprintf("%d%s%d%s", larger, largerUnit, smaller, smallerUnit)
}
// loadWebhookStats gathers the figures for the statistics pane from the
// webhook's entrypoints and targets, as the page has already loaded
// them, and from its event database. It returns nil, and logs why, when
@@ -139,7 +182,7 @@ func (h *Handlers) loadWebhookStats(
func readEventStats(
db *gorm.DB, now time.Time, stats *WebhookStats,
) error {
err := readTotals(db, stats)
err := readTotals(db, now, stats)
if err != nil {
return err
}
@@ -169,9 +212,9 @@ func readEventStats(
}
// readTotals fills in the lifetime and within-retention figures, and
// when the last event arrived, from the running totals: the events'
// row, and the targets' rows summed.
func readTotals(db *gorm.DB, stats *WebhookStats) error {
// when the last event arrived and how long before now that was, from
// the running totals: the events' row, and the targets' rows summed.
func readTotals(db *gorm.DB, now time.Time, stats *WebhookStats) error {
var events database.EventTotals
err := db.Take(&events).Error
@@ -190,6 +233,12 @@ func readTotals(db *gorm.DB, stats *WebhookStats) error {
stats.WithinRetention.Events = events.Events - events.EventsRemoved
stats.LastEventAt = events.LastEventAt
if events.LastEventAt != nil {
stats.LastEvent = relativeTime(*events.LastEventAt, now)
stats.LastEventUTC =
events.LastEventAt.UTC().Format(time.DateTime) + " UTC"
}
for _, t := range targets {
stats.Lifetime.Deliveries += t.Deliveries
stats.Lifetime.Failures += t.Failed
+62 -17
View File
@@ -265,6 +265,13 @@ func statsPane(t *testing.T, page string) string {
return strings.Join(strings.Fields(pane), " ")
}
// statsLastEventShown matches the pane's last event for an event that
// arrived at at: a figure with the full time in UTC as its tooltip.
func statsLastEventShown(at time.Time) string {
return `Last event</span>\s*<span [^>]*title="` +
at.UTC().Format(time.DateTime) + ` UTC">[^<]+</span>`
}
// assertStatsTargets checks, for the history seedStatsHistory builds,
// each target's totals and its deliveries finished in the last 24
// hours. The first target has ten deliveries and the replay, the
@@ -300,14 +307,17 @@ func assertStatsPaneAfterPrune(
) {
t.Helper()
pane := statsPane(t, renderSourceDetailPage(t, h, sess, hist.webhook.ID))
lastEvent := hist.newest.CreatedAt.UTC().Format("2006-01-02 15:04:05 UTC")
page := renderSourceDetailPage(t, h, sess, hist.webhook.ID)
pane := statsPane(t, page)
assert.Contains(t, pane, "Entrypoints 12 (11 active) "+
"Targets 6 (2 active) "+
"Deliveries in progress 13 "+
"Last event "+lastEvent+" "+
"Retention 14 days")
// The newest event arrived moments ago, so the last event is a
// few seconds ago.
assert.Regexp(t, `Entrypoints 12 \(11 active\) `+
`Targets 6 \(2 active\) `+
`Deliveries in progress 13 `+
`Last event \d+s `+
`Retention 14 days`, pane)
assert.Regexp(t, statsLastEventShown(hist.newest.CreatedAt), page)
assert.Contains(t, pane, "Lifetime Within retention "+
"Events 10 9 "+
"Deliveries 21 18 "+
@@ -440,15 +450,14 @@ func TestWebhookStats_LastEventSurvivesPruningEveryEvent(t *testing.T) {
require.NotNil(t, stats.LastEventAt)
assert.True(t, arrived.Equal(*stats.LastEventAt))
pane := statsPane(t, renderSourceDetailPage(t, h, sess, wh.ID))
assert.Contains(t, pane,
"Last event "+arrived.UTC().Format("2006-01-02 15:04:05 UTC"))
assert.Regexp(t, statsLastEventShown(arrived),
renderSourceDetailPage(t, h, sess, wh.ID))
}
// TestWebhookStats_LastEventInUTC checks that the pane shows when the
// last event arrived in UTC, as the event list does, when the time was
// stored in another zone, as it is on a host whose local time is not
// UTC.
// TestWebhookStats_LastEventInUTC checks that the tooltip of the pane's
// last event gives the time in UTC, as the event list does, when the
// time was stored in another zone, as it is on a host whose local time
// is not UTC.
func TestWebhookStats_LastEventInUTC(t *testing.T) {
t.Parallel()
@@ -474,8 +483,9 @@ func TestWebhookStats_LastEventInUTC(t *testing.T) {
require.NoError(t, database.AddEventTotals(webhookDB,
database.EventTotals{Events: 1, LastEventAt: &arrived}))
pane := statsPane(t, renderSourceDetailPage(t, h, sess, wh.ID))
assert.Contains(t, pane, "Last event 2026-03-05 03:30:00 UTC")
assert.Regexp(t,
`Last event</span>\s*<span [^>]*title="2026-03-05 03:30:00 UTC">`,
renderSourceDetailPage(t, h, sess, wh.ID))
}
// TestWebhookStats_PaneShowsRetentionPeriod checks that the statistics
@@ -541,8 +551,11 @@ func TestWebhookStats_WebhookWithNoEvents(t *testing.T) {
assert.Equal(t, &handlers.WebhookStats{}, h.WebhookStatsForTest(wh.ID))
assert.Equal(t, "—", handlers.RecentWindow{}.FailurePercent())
pane := statsPane(t, renderSourceDetailPage(t, h, sess, wh.ID))
page := renderSourceDetailPage(t, h, sess, wh.ID)
pane := statsPane(t, page)
assert.Contains(t, pane, "Last event none")
assert.Regexp(t, `Last event</span>\s*<span class="[^"]*">none</span>`,
page, "a last event of none has a tooltip")
assert.Contains(t, pane, "Failure percentage — —")
assert.False(t, dbMgr.DBExists(wh.ID))
}
@@ -567,3 +580,35 @@ func TestRecentWindow_FailurePercent(t *testing.T) {
assert.Equal(t, tt.want, tt.window.FailurePercent(), tt.window)
}
}
// TestRelativeTime pins how the pane writes the time since the last
// event, at each boundary between its units.
func TestRelativeTime(t *testing.T) {
t.Parallel()
now := time.Date(2026, time.October, 26, 12, 0, 0, 0, time.UTC)
tests := []struct {
ago time.Duration
want string
}{
{0, "0s"},
{59900 * time.Millisecond, "59s"},
{60 * time.Second, "1m"},
{61 * time.Second, "1m1s"},
{3599 * time.Second, "59m59s"},
{3600 * time.Second, "1h"},
{3660 * time.Second, "1h1m"},
{86399 * time.Second, "23h59m"},
{86400 * time.Second, "1d"},
{90000 * time.Second, "1d1h"},
{400*24*time.Hour + 3*time.Hour, "400d3h"},
// An event time ahead of the server clock.
{-time.Hour, "0s"},
}
for _, tt := range tests {
assert.Equal(t, tt.want,
handlers.RelativeTimeForTest(now.Add(-tt.ago), now), tt.ago)
}
}
+38 -1
View File
@@ -61,7 +61,8 @@ const (
// event log in a headless browser, served by the real router and so
// under the real Content-Security-Policy, and checks that the pages'
// Alpine.js directives, the copy control and the delete prompts work,
// and that a target's row shows its delivery counts.
// that a target's row shows its delivery counts, and that the
// Statistics block shows how long ago the last event arrived.
func TestAlpineRunsUnderTheSecurityPolicy(t *testing.T) {
t.Parallel()
@@ -97,6 +98,7 @@ func TestAlpineRunsUnderTheSecurityPolicy(t *testing.T) {
checkCopy(ctx, t, page)
checkEntrypointEdit(ctx, t, page, page+"/events")
checkRecentEvents(ctx, t, page)
checkStatsLastEvent(ctx, t, env, webhook.ID, page)
checkArchiveChoice(ctx, t, srv.URL+"/hooks/new", page)
checkNewWebhookTargets(ctx, t, env, srv.URL+"/hooks/new")
checkRefusedNewWebhook(ctx, t, srv.URL+"/hooks/new")
@@ -831,6 +833,41 @@ func checkRecentEvents(ctx context.Context, t *testing.T, url string) {
"the event's own page does not show its body")
}
// checkStatsLastEvent records that the last event of the webhook
// webhookID arrived 3 days, 4 hours and 30 minutes ago, loads its page,
// and checks that the Statistics block shows the last event as "3d4h"
// with the full time in UTC as its tooltip.
func checkStatsLastEvent(
ctx context.Context, t *testing.T, env *testEnv, webhookID, url string,
) {
t.Helper()
webhookDB, err := env.dbMgr.GetDB(webhookID)
require.NoError(t, err)
ago := 3*24*time.Hour + 4*time.Hour + 30*time.Minute
arrived := time.Now().Add(-ago)
require.NoError(t, database.AddEventTotals(webhookDB,
database.EventTotals{LastEventAt: &arrived}))
figure := `//span[text()="Last event"]` +
`/following-sibling::span[text()="3d4h"]`
var (
title string
found bool
)
require.NoError(t, chromedp.Run(ctx, loadPage(url)))
require.True(t, shown(ctx, figure),
`the Statistics block does not show the last event as "3d4h"`)
require.NoError(t, chromedp.Run(ctx, chromedp.AttributeValue(
figure, "title", &title, &found, chromedp.BySearch,
)))
assert.Equal(t, arrived.UTC().Format(time.DateTime)+" UTC", title,
"the last event's tooltip is not the full time in UTC")
}
// checkEventLog loads the event log and checks that of its events only
// the newest, eventID, starts expanded: its row says so and its caret is
// turned up, and the log's last event, lastEventID, starts collapsed. In
-3
View File
@@ -13,9 +13,6 @@ import (
// TestBaseTemplateScriptsAreServed walks every /s/ script the base
// template loads on each page and fetches it through the real router.
// Alpine.js is extracted from its tarball in 3p/ at build time, so the
// file is not in the tree: this is the check that the page still gets
// the JavaScript it asks for.
func TestBaseTemplateScriptsAreServed(t *testing.T) {
t.Parallel()
-17
View File
@@ -1,17 +0,0 @@
#!/bin/sh
# script/assets: extract Alpine.js from its npm package tarball, committed
# in 3p/, to static/js/alpine.min.js, where go:embed reads it. The package
# is @alpinejs/csp, Alpine's build for pages whose Content-Security-Policy
# forbids eval. The extracted file is not committed. make build, make dev
# and the Dockerfile's lint, test and build stages run this first.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
tar -xzOf 3p/alpinejs-csp-3.14.9.tgz package/dist/cdn.min.js \
>static/js/alpine.min.js
}
main "$@"
+11
View File
File diff suppressed because one or more lines are too long
+5 -3
View File
@@ -6,9 +6,11 @@ import (
)
// Static holds the CSS and JavaScript files the web UI's pages load. They
// are named one by one so that a missing js/alpine.min.js, which make
// assets extracts and git does not track, fails the build instead of
// leaving the pages without Alpine.js.
// are named one by one, so a missing file fails the build instead of
// leaving the pages without it.
//
// js/alpine.min.js is Alpine.js, committed unchanged. It is
// package/dist/cdn.min.js from the npm package @alpinejs/csp 3.14.9.
//
//go:embed css/tailwind.css css/style.css js/app.js js/alpine.min.js
var Static embed.FS
+5 -1
View File
@@ -22,7 +22,11 @@
</div>
<div>
<span class="text-gray-500">Last event</span>
<span class="font-medium text-gray-900">{{with .LastEventAt}}{{.UTC.Format "2006-01-02 15:04:05 UTC"}}{{else}}none{{end}}</span>
{{if .LastEvent}}
<span class="font-medium text-gray-900" title="{{.LastEventUTC}}">{{.LastEvent}}</span>
{{else}}
<span class="font-medium text-gray-900">none</span>
{{end}}
</div>
<div>
<span class="text-gray-500">Retention</span>