Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
17bd4770dd |
@@ -99,11 +99,12 @@ sooner after 24 hours without use, unless `SESSION_IDLE_TIMEOUT` says otherwise.
|
||||
|
||||
### Webhooks
|
||||
|
||||
**New Webhook** on the webhook list at `/hooks` asks for a name, an optional
|
||||
description and a retention period in days. Events older than the retention
|
||||
period are deleted, along with their deliveries. The default is 30 days. `0`
|
||||
keeps events forever. Every new webhook gets one entrypoint, described as
|
||||
`Default entrypoint`.
|
||||
**New Webhook** on the webhook list at `/hooks` asks for **Name**, an optional
|
||||
**Description** and **Retention (days)**. Events older than the retention period
|
||||
are deleted, along with their deliveries. The default is 30 days. `0` keeps
|
||||
events forever. The form also has an **HTTP target URL** field and an
|
||||
**Archive** checkbox; each creates that target with the webhook. Every new
|
||||
webhook gets one entrypoint, described as `Default entrypoint`.
|
||||
|
||||
### Entrypoint URLs
|
||||
|
||||
@@ -121,14 +122,14 @@ deactivated entrypoint answers `410 Gone`. A deleted or unknown one answers
|
||||
|
||||
**Add** under **Targets** asks for the type first, then that type's settings.
|
||||
|
||||
- An `http` target posts each event to a URL, with optional request headers and
|
||||
a timeout.
|
||||
- A `slack` target posts each event as a message to the message-posting URL
|
||||
Slack or Mattermost give out, which they call an incoming webhook.
|
||||
- An `http` target posts each event to a URL, with optional **Headers** and a
|
||||
**Timeout**.
|
||||
- A `slack` target posts each event as a message. Its **Webhook URL** is the
|
||||
message-posting URL Slack or Mattermost give out, which they call an incoming
|
||||
webhook.
|
||||
- An archive target writes each event as a row in a SQLite file of its own. Its
|
||||
stored type is `database`. It takes an expiry, which is how long archived
|
||||
events are kept, and a rotation, which is how often a new archive file is
|
||||
started.
|
||||
stored type is `database`. **Archive expiry** is how long archived events are
|
||||
kept, and **Archive rotation** is how often a new archive file is started.
|
||||
- A `log` target writes each event to the application log.
|
||||
|
||||
`http` and `slack` targets also take **Delivery attempts**: how many times each
|
||||
@@ -187,8 +188,9 @@ your reverse proxy's Docker network instead, and mount one host directory at
|
||||
|
||||
webhooker speaks plain HTTP. It does not handle TLS, the encryption behind
|
||||
HTTPS: it holds no certificate and accepts no HTTPS connections. The reverse
|
||||
proxy holds the certificate and reaches webhooker over loopback, the address
|
||||
only this host can reach.
|
||||
proxy holds the certificate and reaches webhooker over a private address. On a
|
||||
bare host that is loopback, the address only this host can reach. For the image
|
||||
it is the Docker bridge.
|
||||
|
||||
Four things must be right.
|
||||
|
||||
@@ -200,16 +202,18 @@ Four things must be right.
|
||||
from any connecting machine, so the proxy must overwrite whatever the client
|
||||
sent.
|
||||
3. **Cover the proxy in `TRUSTED_PROXIES`.** Otherwise every client shares one
|
||||
rate-limit bucket, the count a limit keeps per client. The default list does
|
||||
not cover a proxy that connects over loopback.
|
||||
4. **Send `Host` as `$http_host`, not `$host`.** webhooker checks a form's
|
||||
`Origin` against the `Host` header it receives. `$host` drops the port, so on
|
||||
any port other than 443 every form submission fails with `403`.
|
||||
rate-limit bucket, the count a limit keeps per client. The default list
|
||||
covers the Docker bridge address, such as `172.17.0.1`, that the image sees a
|
||||
proxy at. It does not cover loopback, so on a bare host set
|
||||
`TRUSTED_PROXIES=127.0.0.1`.
|
||||
4. **Pass on the client's `Host` header with its port.** webhooker checks a
|
||||
form's `Origin` against the `Host` header it receives. A `Host` without the
|
||||
port makes every form submission fail with `403` on any port other than 443.
|
||||
In nginx that means `$http_host`, not `$host`.
|
||||
|
||||
Protect the proxy's access log: its request paths hold the entrypoint UUIDs.
|
||||
|
||||
An nginx server block for this, with `TRUSTED_PROXIES=127.0.0.1` on webhooker's
|
||||
side:
|
||||
An nginx server block for a bare host:
|
||||
|
||||
```nginx
|
||||
server {
|
||||
@@ -248,12 +252,18 @@ the server does and refuses while a running instance holds it.
|
||||
DATA_DIR=/var/lib/webhooker webhooker resetpw -generate admin
|
||||
```
|
||||
|
||||
Without `-generate`, the new password is read from standard input, at least 8
|
||||
characters. It is never taken as a command-line argument, because other accounts
|
||||
on the host can read those. In a container, run the same command with
|
||||
`docker run --rm` and the stopped container's volume mounted. The command
|
||||
creates nothing: a missing `DATA_DIR`, a missing `webhooker.db` or an unknown
|
||||
username is an error.
|
||||
Without `-generate`, the new password is read from standard input. It must be at
|
||||
least 8 characters. It is never taken as a command-line argument, because other
|
||||
accounts on the host can read those. The command creates nothing: a missing
|
||||
`DATA_DIR`, a missing `webhooker.db` or an unknown username is an error.
|
||||
|
||||
In a container, run the binary at `/app/webhooker` with the stopped container's
|
||||
volume mounted:
|
||||
|
||||
```bash
|
||||
docker run --rm -v /path/to/data:/var/lib/webhooker \
|
||||
webhooker /app/webhooker resetpw -generate admin
|
||||
```
|
||||
|
||||
## Backup and restore
|
||||
|
||||
@@ -263,15 +273,16 @@ All state lives in `DATA_DIR`. Back up the whole directory. It holds:
|
||||
password hash, and the webhooks, entrypoints and targets;
|
||||
- one event database per webhook, `events-{webhook_uuid}.db`: its events,
|
||||
deliveries and attempts;
|
||||
- the archive files, one per archive target;
|
||||
- the files of each archive target;
|
||||
- the lock file, `webhooker.lock`, which is always empty.
|
||||
|
||||
Every database runs in WAL mode. WAL is SQLite's write-ahead log: finished
|
||||
writes land in a `-wal` file beside the database before they reach the `.db`
|
||||
itself. A copy of the `.db` without its `-wal` is missing data. A clean shutdown
|
||||
removes the `-wal` and `-shm` files; a crash leaves them, and they must be
|
||||
carried with the `.db`. Keep `DATA_DIR` on a local disk: WAL mode is not
|
||||
reliable on network filesystems, and webhooker does not check for one.
|
||||
itself, and a `-shm` file beside it indexes that log. A copy of the `.db`
|
||||
without its `-wal` is missing data. A clean shutdown removes the `-wal` and
|
||||
`-shm` files; a crash leaves them, and they must be carried with the `.db`. Keep
|
||||
`DATA_DIR` on a local disk: WAL mode is not reliable on network filesystems, and
|
||||
webhooker does not check for one.
|
||||
|
||||
A copy taken while the service runs can catch a `.db` and its `-wal` at two
|
||||
different instants. Stop the service, copy the directory, and start it again.
|
||||
@@ -318,9 +329,9 @@ writes it. The reverse proxy's access log does hold it, in the request path.
|
||||
- Session cookies are signed, `HttpOnly` (hidden from page scripts) and
|
||||
`SameSite=Lax` (not sent with cross-site form posts). They carry `Secure` when
|
||||
the request carries `X-Forwarded-Proto: https`.
|
||||
- Every form is protected against cross-site request forgery (CSRF), where
|
||||
another site makes your browser submit a form here: each form carries a token
|
||||
tied to a cookie, and a submission's `Origin` must name this host.
|
||||
- Every form is protected against cross-site request forgery (CSRF). CSRF is
|
||||
when another site makes your browser submit a form here. Each form carries a
|
||||
token tied to a cookie, and a submission's `Origin` must name this host.
|
||||
- Form bodies are capped at 1 MB.
|
||||
- Every response carries these headers:
|
||||
- `Strict-Transport-Security`, unless `HSTS_MAX_AGE` is `0`;
|
||||
@@ -328,14 +339,16 @@ writes it. The reverse proxy's access log does hold it, in the request path.
|
||||
- `X-Frame-Options: DENY`;
|
||||
- a Content-Security-Policy (CSP), the header that tells the browser where
|
||||
scripts and styles may come from, set to
|
||||
`default-src 'self'; script-src 'self'; style-src 'self'`.
|
||||
`default-src 'self'; script-src 'self'; style-src 'self'`;
|
||||
- `Referrer-Policy: strict-origin-when-cross-origin`;
|
||||
- `Permissions-Policy: camera=(), microphone=(), geolocation=()`.
|
||||
- `/metrics` sits behind HTTP basic authentication, a username and password sent
|
||||
with each request.
|
||||
|
||||
### Sign-in limits
|
||||
|
||||
The password is checked first, and only a failed sign-in is counted. A correct
|
||||
password is never throttled. Failures are counted per client and submitted
|
||||
password is never answered `429`. Failures are counted per client and submitted
|
||||
username. From the fifth failure within a minute, each failure is answered
|
||||
`429`. At most two password checks run at once; a request that gets no turn
|
||||
within 5 seconds is answered `503`. So those two checks, not the failure count,
|
||||
@@ -396,9 +409,9 @@ again any delivery left `pending` or `retrying`, so a stop or a crash loses no
|
||||
delivery.
|
||||
|
||||
The main database holds configuration. Each webhook has its own event database,
|
||||
so writes for one webhook never wait on another webhook's database. The Go code
|
||||
is under `internal/`, one package per concern, and the pages' HTML, CSS and
|
||||
JavaScript are embedded in the binary.
|
||||
so writes for one webhook never wait on another webhook's database. The program
|
||||
starts in `cmd/webhooker/main.go`, and its packages are under `internal/`. The
|
||||
pages' HTML, CSS and JavaScript are embedded in the binary.
|
||||
|
||||
## Configuration reference
|
||||
|
||||
@@ -530,23 +543,25 @@ default but is reopened by a list that covers it.
|
||||
## Delivery reference
|
||||
|
||||
An `http` target posts the event body to its URL with the event's `Content-Type`
|
||||
and headers, then the target's configured headers, which replace event headers
|
||||
of the same name. A `2xx` response is a success. The first 4 KiB of the response
|
||||
body is stored with the attempt. `timeout_seconds` is at most 300, and blank
|
||||
means the 30 second default. `forward_query` appends the event's query string to
|
||||
the URL.
|
||||
and headers, then the target's configured **Headers**, which replace event
|
||||
headers of the same name. Every `http` and `slack` request carries
|
||||
`User-Agent: webhooker/{version}`, replacing the sender's `User-Agent` and any
|
||||
configured one. A `2xx` response is a success. The first 4 KiB of the response
|
||||
body is stored with the attempt. **Timeout** is at most 300 seconds, and blank
|
||||
means the 30 second default. **Pass the query string on to this target** appends
|
||||
the event's query string to the URL.
|
||||
|
||||
A `slack` target posts a message that describes the event and shows its body in
|
||||
a code block, cut at 3500 bytes. An archive target, whose stored type is
|
||||
`database`, appends the event as a row to its own SQLite file. A `log` target
|
||||
writes the event as one `INFO` line.
|
||||
|
||||
`http` and `slack` targets take `max_attempts`, from 1 to 20. A failed attempt
|
||||
marks the delivery `retrying` and schedules the next attempt; the wait doubles
|
||||
each time, starting at one second. After the last allowed attempt fails, the
|
||||
delivery is marked `failed`.
|
||||
**Delivery attempts** is from 1 to 20. A failed attempt marks the delivery
|
||||
`retrying` and schedules the next attempt; the wait doubles each time, starting
|
||||
at one second. After the last allowed attempt fails, the delivery is marked
|
||||
`failed`.
|
||||
|
||||
A target with `max_attempts` above 1 has a circuit breaker, which stops
|
||||
A target with more than one delivery attempt has a circuit breaker, which stops
|
||||
hammering a destination that is down. After 5 failures in a row, deliveries to
|
||||
the target pause for 30 seconds. Then one delivery is sent as a probe; the
|
||||
others wait until a probe succeeds. The webhook page says when a target is
|
||||
@@ -595,21 +610,23 @@ metric carries one label, `target_type`, which is `http`, `slack`, `database`,
|
||||
- `webhooker_deliveries_succeeded_total` and
|
||||
`webhooker_deliveries_failed_total`;
|
||||
- `webhooker_delivery_retries_total` and `webhooker_delivery_replays_total`;
|
||||
- `webhooker_delivery_duration_seconds`, a histogram (a distribution) of one
|
||||
attempt's wall time;
|
||||
- `webhooker_delivery_duration_seconds`, a histogram (a distribution) of how
|
||||
long one attempt took;
|
||||
- `webhooker_deliveries_pending` and `webhooker_deliveries_retrying`, gauges
|
||||
(current values) sampled every 30 seconds;
|
||||
- `webhooker_circuit_breakers_open`.
|
||||
|
||||
Three inbound HTTP metrics, named `http_...`, carry the route pattern as their
|
||||
`handler` label, never the real path.
|
||||
Two inbound HTTP metrics, `http_request_duration_seconds` and
|
||||
`http_response_size_bytes`, carry the route pattern as their `handler` label,
|
||||
never the real path. A third, `http_requests_inflight`, counts all routes
|
||||
together.
|
||||
|
||||
## Development
|
||||
|
||||
You need the Go version named in `go.mod`, and Docker. The tests, the linters,
|
||||
prettier and the stylesheet build run in Docker, at versions pinned (fixed to
|
||||
one exact release) in the `Dockerfile`, `package.json` and `yarn.lock`. Only
|
||||
`gofmt` and `goimports` run on the host.
|
||||
one exact release) in the `Dockerfile`, `package.json` and `yarn.lock`. `gofmt`
|
||||
and `goimports` run on the host.
|
||||
|
||||
```bash
|
||||
make bootstrap # Install git, make, Go and the Go modules
|
||||
@@ -633,8 +650,8 @@ checkout, or `VERSION` when that is set. It is shown at
|
||||
|
||||
The pages use one third-party script, Alpine.js, in its CSP build
|
||||
`@alpinejs/csp`, which runs under a Content-Security-Policy that forbids `eval`.
|
||||
The package file in `3p/` is byte for byte the one the npm registry, the
|
||||
JavaScript package registry, publishes.
|
||||
The npm registry is where JavaScript packages are published. The package file in
|
||||
`3p/` is byte for byte the one it publishes.
|
||||
|
||||
A browser test in `internal/server` loads the pages in a headless browser, one
|
||||
without a window, under the real Content-Security-Policy. `make test` leaves it
|
||||
@@ -657,8 +674,10 @@ application are under [Entrypoint URLs](#entrypoint-urls).
|
||||
- `script/test` runs the tests in Docker.
|
||||
- `script/test-browser` runs the browser test in Docker.
|
||||
- `script/lint` runs `gofmt`, `golangci-lint` and ESLint in Docker.
|
||||
- `script/fmt` formats the Go code on the host and the Markdown in Docker.
|
||||
- `script/fmt-check` checks the same and fails instead of writing.
|
||||
- `script/fmt` formats the Go code with `gofmt` and `goimports` on the host, and
|
||||
the Markdown with prettier in Docker.
|
||||
- `script/fmt-check` fails when `gofmt` or prettier would change a file. It does
|
||||
not run `goimports`, and it writes nothing.
|
||||
- `script/css` regenerates `static/css/tailwind.css` in Docker.
|
||||
- `script/css-check` fails when `static/css/tailwind.css` is out of date.
|
||||
- `script/check` runs `script/test`, `script/lint`, `script/fmt-check` and
|
||||
|
||||
Reference in New Issue
Block a user