Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5471054adb |
@@ -48,9 +48,9 @@ The part of webhooker that answers requests to `/h/{uuid}` is the receiver.
|
||||
A request to an entrypoint is stored as an event. One pending delivery is
|
||||
created for each active target of the webhook, in the same database write as the
|
||||
event: either all of them are stored or none. The sender gets `200` once they
|
||||
are stored. Workers then carry out the deliveries. A delivery to an HTTP or
|
||||
Slack target that fails is retried, up to the number of attempts the target
|
||||
allows.
|
||||
are stored. Background workers, the part of webhooker that sends deliveries,
|
||||
then carry them out. A delivery to an HTTP or Slack target that fails is
|
||||
retried, up to the number of attempts the target allows.
|
||||
|
||||
Delivery is at least once, not exactly once. If a delivery reaches the target
|
||||
but webhooker cannot record that outcome, it sends the delivery again later. A
|
||||
@@ -123,7 +123,9 @@ keeps events forever.
|
||||
|
||||
The form can also create the first targets. An HTTP target URL creates an `http`
|
||||
target named `HTTP`. The **Archive** checkbox creates an archive target named
|
||||
`Archive`, with the expiry and rotation chosen beside it.
|
||||
`Archive`. Beside it you choose the archive's expiry, which is how long archived
|
||||
events are kept, and its rotation, which is how often a new archive file is
|
||||
started.
|
||||
|
||||
Every new webhook gets one entrypoint, described as `Default entrypoint`.
|
||||
|
||||
@@ -137,8 +139,8 @@ like a password. Read
|
||||
**Add** creates another entrypoint with its own URL. Several entrypoints on one
|
||||
webhook let you tell senders apart while delivering to the same targets.
|
||||
|
||||
**Edit** changes an entrypoint's description. Its URL never changes. To retire a
|
||||
URL, deactivate the entrypoint or delete it, then create a new one.
|
||||
**Edit** changes an entrypoint's description. Its URL never changes. To replace
|
||||
a URL, deactivate the entrypoint or delete it, then create a new one.
|
||||
|
||||
A deactivated entrypoint answers `410 Gone`. A deleted or unknown one answers
|
||||
`404`.
|
||||
@@ -194,8 +196,13 @@ retention. Next are figures for the last 10 minutes and the last 24 hours:
|
||||
events, failures and the failure percentage. The percentage is failed deliveries
|
||||
out of all deliveries that finished in the window.
|
||||
|
||||
The webhook list shows three of these figures per webhook: events within
|
||||
retention, the last event, and failed deliveries in the last 24 hours.
|
||||
The webhook list shows, for each webhook:
|
||||
|
||||
- its entrypoint and target counts, with how many are inactive;
|
||||
- its retention setting;
|
||||
- its events within retention;
|
||||
- its last event;
|
||||
- its failed deliveries in the last 24 hours.
|
||||
|
||||
### Recent events and the event log
|
||||
|
||||
@@ -206,8 +213,13 @@ body** link to the stored bytes.
|
||||
**Full Event Log** at `/hook/{id}/events` shows the 50 newest events with their
|
||||
deliveries and every attempt. **Failed** narrows it to events with a failed
|
||||
delivery. **Pending** narrows it to events with a delivery still pending or
|
||||
retrying. Each attempt shows its time, status code, duration, error, and the
|
||||
first 4 KiB of the response.
|
||||
retrying. Each attempt shows:
|
||||
|
||||
- when it ran;
|
||||
- its status code;
|
||||
- its duration;
|
||||
- its error, if any;
|
||||
- the first 4 KiB of the response.
|
||||
|
||||
Each event has a page of its own at `/hook/{id}/events/{eventID}`. It shows the
|
||||
whole request: query string, headers, body and every delivery.
|
||||
@@ -262,10 +274,11 @@ docker run -d \
|
||||
|
||||
The image listens on every IPv4 address inside the container: it ships
|
||||
`BIND_ADDRESS=0.0.0.0`. A container has its own network namespace, which means
|
||||
its own addresses, separate from the host's. That is the boundary, and the `-p`
|
||||
flag decides what the host exposes. Publish to `127.0.0.1:8080`, the address
|
||||
your reverse proxy connects to, and nothing wider. A bare `-p 8080:8080`
|
||||
publishes the port on every interface of the host.
|
||||
its own addresses, separate from the host's. The namespace is what keeps the
|
||||
container's ports apart from the host's, and the `-p` flag decides what the host
|
||||
exposes. Publish to `127.0.0.1:8080`, the address your reverse proxy connects
|
||||
to, and nothing wider. A bare `-p 8080:8080` publishes the port on every
|
||||
interface of the host.
|
||||
|
||||
Do not set `BIND_ADDRESS` to an empty value in a container. An empty value means
|
||||
unset, and unset falls back to the binary's default of `127.0.0.1`. The
|
||||
@@ -344,17 +357,19 @@ Five things must be right.
|
||||
`127.0.0.1` only. The plain port serves the sign-in form and the receiver
|
||||
without TLS.
|
||||
2. **Send `X-Forwarded-Proto: https`.** This header tells webhooker the request
|
||||
arrived over TLS. It decides the `Secure` flag on cookies and the strict
|
||||
origin check on forms. Without it, cookies are sent without `Secure`.
|
||||
webhooker reads this header from any peer, so the proxy must overwrite
|
||||
whatever the client sent.
|
||||
arrived over TLS. It decides the `Secure` flag on cookies, and whether a form
|
||||
submission must carry a matching `Origin` or `Referer` (see
|
||||
[The web interface](#the-web-interface) under Security). Without it, cookies
|
||||
are sent without `Secure`. webhooker reads this header from any connecting
|
||||
machine, so the proxy must overwrite whatever the client sent.
|
||||
3. **Cover the proxy in `TRUSTED_PROXIES`.** Otherwise every client shares one
|
||||
rate-limit bucket. A bucket is the count a limit keeps per client, and here
|
||||
it would be keyed on the proxy's address. The default list is the private
|
||||
ranges, which does not cover a proxy that connects over loopback.
|
||||
4. **Send `Host` as `$http_host`, not `$host`.** `$host` drops the port. The
|
||||
form origin check compares against the host it was given. On any port other
|
||||
than 443 every form submission then fails with `403`.
|
||||
4. **Send `Host` as `$http_host`, not `$host`.** webhooker checks a form's
|
||||
`Origin` against the `Host` header it receives. `$host` drops the port, so on
|
||||
any port other than 443 the two differ and every form submission fails with
|
||||
`403`.
|
||||
5. **Keep the proxy's access log.** It records the client address whatever
|
||||
`TRUSTED_PROXIES` says. It also records the full request path, which holds
|
||||
the entrypoint UUID of every event. Protect that log as you would the
|
||||
@@ -476,21 +491,24 @@ takes effect at the next sign-in. Sessions already signed in are unaffected.
|
||||
|
||||
All state lives in `DATA_DIR`. Back up the whole directory. It holds:
|
||||
|
||||
- `webhooker.db`, one per install. Settings, the session key, users, webhooks,
|
||||
entrypoints and targets.
|
||||
- `events-{webhook_uuid}.db`, one per webhook. Events, deliveries and delivery
|
||||
attempts.
|
||||
- `webhooker.db`, the main database, one per install;
|
||||
- `events-{webhook_uuid}.db`, one event database per webhook;
|
||||
- `archive-{webhook_name}-{target_name}-{target_uuid}.db`, one per archive
|
||||
target, or one per period for a target that rotates. The period goes before
|
||||
`.db`.
|
||||
target, or one per period for a target that rotates, with the period before
|
||||
`.db`;
|
||||
- `webhooker.lock`, which is always empty and holds no state.
|
||||
|
||||
Every database runs in WAL mode. WAL is SQLite's write-ahead log: committed
|
||||
changes land in a `-wal` file beside the database before they reach the `.db`
|
||||
[Data files](#data-files) says what each database holds.
|
||||
|
||||
Every database runs in WAL mode. WAL is SQLite's write-ahead log: finished
|
||||
writes land in a `-wal` file beside the database before they reach the `.db`
|
||||
itself. So a running service has two more files beside each `{name}.db`:
|
||||
`{name}.db-wal`, the log itself, and `{name}.db-shm`, the shared index SQLite
|
||||
keeps for that log. The `-wal` file is part of the database. A copy of the `.db`
|
||||
without it is missing data and may have no readable tables at all.
|
||||
|
||||
- `{name}.db-wal`, the log itself;
|
||||
- `{name}.db-shm`, the shared index SQLite keeps for that log.
|
||||
|
||||
The `-wal` file is part of the database. A copy of the `.db` without it is
|
||||
missing data and may have no readable tables at all.
|
||||
|
||||
A clean shutdown closes every database and removes its `-wal` and `-shm` files.
|
||||
A crash leaves them, and they must be carried with the `.db`.
|
||||
@@ -526,8 +544,9 @@ done
|
||||
|
||||
`.backup` writes one complete file, with no `-wal` beside it. The runtime image
|
||||
has no `sqlite3`, so run this on the host or from another container with the
|
||||
volume mounted. Each file is captured at its own instant, so an event stored
|
||||
between two copies lands in one file and not the other.
|
||||
volume mounted. Each file is captured at its own instant. So `webhooker.db` and
|
||||
an event database copied at different moments can disagree: a webhook created
|
||||
between the two copies is in one and has no event database in the other.
|
||||
|
||||
Archive files are the exception. The archive writer closes and reopens its file
|
||||
after a write, at most once per second. So you can move a finished archive file
|
||||
@@ -553,7 +572,7 @@ who can read it.
|
||||
- The event and archive databases hold the full headers and body of every event,
|
||||
including whatever the sender put in them.
|
||||
- `webhooker.db` holds target settings unencrypted, next to the session key and
|
||||
the password hashes.
|
||||
the password hash.
|
||||
|
||||
## Security
|
||||
|
||||
@@ -568,10 +587,9 @@ stored and forwarded as ordinary headers.
|
||||
What that means in practice:
|
||||
|
||||
- Keep the URL out of tickets, chat and screenshots. webhooker's own log never
|
||||
writes it, except in the line a `log` target writes. The reverse proxy's
|
||||
access log does hold it, in the request path.
|
||||
- To rotate, create a new entrypoint and point the sender at it. Then deactivate
|
||||
or delete the old one.
|
||||
writes it. The reverse proxy's access log does hold it, in the request path.
|
||||
- To replace a URL, create a new entrypoint and point the sender at it. Then
|
||||
deactivate or delete the old one.
|
||||
- Some senders cannot be given a secret URL: they post to one fixed address and
|
||||
prove themselves with a signature instead. webhooker checks no signatures, so
|
||||
such a sender cannot use it. Pick another integration path for it.
|
||||
@@ -677,11 +695,12 @@ Redirects are followed up to ten hops. Each hop is checked the same way. A hop
|
||||
that leaves the target's origin carries none of the target's configured headers
|
||||
and none of the forwarded event headers. Leaving the origin means a different
|
||||
host, a different port, or a step from `https` to `http`. An open redirect is a
|
||||
destination that forwards requests to any URL a caller names. This rule stops
|
||||
one from handing a credential to another host.
|
||||
destination that forwards requests to any URL a caller names. Because of this
|
||||
rule, an open redirect cannot pass those headers to another host.
|
||||
|
||||
`ALLOWED_EGRESS_CIDRS` opens named networks for the common case of forwarding to
|
||||
your own network. Read
|
||||
Egress means the connections webhooker opens outward to deliver.
|
||||
`ALLOWED_EGRESS_CIDRS` opens named networks to it, for the common case of
|
||||
forwarding to your own network. Read
|
||||
[Allowing egress to your own network](#allowing-egress-to-your-own-network)
|
||||
before setting it.
|
||||
|
||||
@@ -702,8 +721,8 @@ logs the route pattern, which is the route as registered, such as
|
||||
as its scheme and host only.
|
||||
|
||||
Every line that names a client carries two fields. `remoteIP` is the connecting
|
||||
peer, which behind a proxy is the proxy. `clientIP` is the client as decided by
|
||||
`TRUSTED_PROXIES`.
|
||||
machine, which behind a proxy is the proxy. `clientIP` is the client as decided
|
||||
by `TRUSTED_PROXIES`.
|
||||
|
||||
SQL statements are logged with `?` in place of each value, never with the values
|
||||
themselves. Outside a `log` target's line, no line carries session cookies or
|
||||
@@ -731,64 +750,31 @@ service, with the request reduced to this:
|
||||
|
||||
## Design
|
||||
|
||||
### Request flow
|
||||
webhooker is one process. An HTTP server receives requests, a delivery engine
|
||||
sends them on, and SQLite holds everything in between.
|
||||
|
||||
One request to an entrypoint goes through these steps:
|
||||
The HTTP server runs every request through middleware, code that runs around
|
||||
each request. The middleware writes the access log, sets the security headers,
|
||||
applies a 60 second timeout that answers `504`, and recovers from a crash inside
|
||||
the request. On the page routes it also caps the body at 1 MB, applies the CSRF
|
||||
protection and checks the sign-in. On the receiver route it applies the two rate
|
||||
limits. The handler for the route then does the work.
|
||||
|
||||
1. The rate limits run: one per client per entrypoint, one per client across the
|
||||
whole route.
|
||||
2. The handler answers any method but `POST` with `405`.
|
||||
3. The handler looks up the entrypoint. Unknown gets `404`, deactivated `410`.
|
||||
4. The handler reads the body, up to 1 MB. A larger body gets `413`.
|
||||
5. The event, one pending delivery per active target, and the running totals are
|
||||
stored together, or not at all. The sender gets `200`.
|
||||
6. One task per delivery goes onto the delivery queue.
|
||||
7. A worker takes the task and runs the target's delivery. The outcome is
|
||||
recorded as an attempt, and the delivery's status is updated.
|
||||
The receiver's handler stores the event, one pending delivery per active target,
|
||||
and the running totals in one database write. It then queues one task per
|
||||
delivery. The delivery engine runs ten background workers that take tasks from
|
||||
two queues, one for new deliveries and one for retries, and run the target's
|
||||
delivery. Each attempt is recorded, and the delivery's status is updated. A
|
||||
recovery routine re-queues deliveries the engine lost track of.
|
||||
|
||||
### Components and middleware
|
||||
The main database holds configuration. Each webhook has its own event database,
|
||||
so a busy webhook does not slow the others. Archive targets write to files of
|
||||
their own. [Data files](#data-files) names each file.
|
||||
|
||||
The components are wired by Uber fx, a library that builds each component when
|
||||
another needs it. fx starts the components in the order their dependencies need,
|
||||
and stops them in reverse. The components are:
|
||||
The components are wired by [Uber fx](https://go.uber.org/fx), a library that
|
||||
builds each component when another needs it and runs their start and stop hooks.
|
||||
The other libraries are:
|
||||
|
||||
- the globals, the logger and the config;
|
||||
- the main database and the per-webhook database manager;
|
||||
- the retention sweep and the health check;
|
||||
- the session and the handlers;
|
||||
- the metrics registry and collectors;
|
||||
- the middleware and the SSRF guard;
|
||||
- the delivery engine, the archive writers and the archive sweep;
|
||||
- the server.
|
||||
|
||||
Middleware is code that runs around every request. The global middleware runs in
|
||||
this order:
|
||||
|
||||
1. request id;
|
||||
2. security headers;
|
||||
3. access log;
|
||||
4. metrics, when `/metrics` is served;
|
||||
5. CORS;
|
||||
6. a 60 second request timeout, which answers `504` when it fires;
|
||||
7. panic recovery;
|
||||
8. Sentry, when configured.
|
||||
|
||||
A page route group is a set of routes under one prefix, such as `/hook/`. Each
|
||||
page route group then adds its own middleware, in this order:
|
||||
|
||||
1. panic recovery;
|
||||
2. Sentry, when configured;
|
||||
3. the 1 MB body cap;
|
||||
4. CSRF;
|
||||
5. `Cache-Control: no-store`;
|
||||
6. the sign-in check, in every group except `/pages`, which serves the sign-in
|
||||
form.
|
||||
|
||||
The body cap comes before CSRF because CSRF parses the form.
|
||||
|
||||
### Libraries
|
||||
|
||||
- [Uber fx](https://go.uber.org/fx) wires the components, as above.
|
||||
- [chi](https://github.com/go-chi/chi), an HTTP router, matches requests to
|
||||
handlers.
|
||||
- [GORM](https://gorm.io), a Go database library, with
|
||||
@@ -804,38 +790,9 @@ The body cap comes before CSRF because CSRF parses the form.
|
||||
[go-http-metrics](https://github.com/slok/go-http-metrics) serve the metrics.
|
||||
- The [Sentry](https://sentry.io) library reports errors when configured.
|
||||
|
||||
### Package layout
|
||||
|
||||
Most of the Go code is under `internal/`. The program starts in
|
||||
`cmd/webhooker/main.go`, which dispatches subcommands, takes the `DATA_DIR` lock
|
||||
and wires the components.
|
||||
|
||||
- `internal/archive`: archive files, rotation, pruning, renaming and download.
|
||||
- `internal/banner`: the ruled box that prints a credential.
|
||||
- `internal/config`: configuration from the environment and `.env`.
|
||||
- `internal/database`: the main database, the models, Argon2id, the retention
|
||||
sweep and the per-webhook database manager.
|
||||
- `internal/datadir`: the exclusive lock on `DATA_DIR`.
|
||||
- `internal/db/migrations`: the SQL schema files and their application.
|
||||
- `internal/delivery`: the engine, the four target types, retries, the circuit
|
||||
breaker and the SSRF guard.
|
||||
- `internal/globals`: build-time variables.
|
||||
- `internal/gormlog`: GORM's logger on top of `slog`, with values withheld.
|
||||
- `internal/handlers`: every HTTP handler and page.
|
||||
- `internal/healthcheck`: the health check response.
|
||||
- `internal/lifecycle`: the bounded wait used by stop hooks.
|
||||
- `internal/logfield`: the size budget for client-supplied log values.
|
||||
- `internal/logger`: `slog` setup and the fx event logger.
|
||||
- `internal/metrics`: the delivery metrics.
|
||||
- `internal/middleware`: logging, security headers, CORS, sessions, CSRF, rate
|
||||
limits, body caps, metrics and panic recovery.
|
||||
- `internal/referer`: cuts a `Referer` to its scheme and host.
|
||||
- `internal/reqtls`: the one function that decides whether a request is TLS.
|
||||
- `internal/resetpw`: the `webhooker resetpw` subcommand.
|
||||
- `internal/server`: the HTTP server, routes, middleware order and Sentry.
|
||||
- `internal/session`: the cookie session.
|
||||
- `internal/versionscript`: tests for `script/version` and the build files.
|
||||
- `static/`: embedded CSS and JavaScript. `templates/`: the HTML templates.
|
||||
Most of the Go code is under `internal/`, one package per concern. The program
|
||||
starts in `cmd/webhooker/main.go`. `templates/` holds the pages' HTML, and
|
||||
`static/` their CSS and JavaScript, both embedded in the binary.
|
||||
|
||||
## Configuration reference
|
||||
|
||||
@@ -862,8 +819,7 @@ on-or-off field. The `/settings` page shows every value; there,
|
||||
|
||||
`dev` or `prod`. Default `prod`.
|
||||
|
||||
This controls one thing: CORS, the headers that let a page on another site call
|
||||
this one from a browser. In `dev`, every origin is allowed with
|
||||
This controls one thing: CORS. In `dev`, every origin is allowed with
|
||||
`Access-Control-Allow-Origin: *`, for local testing. In `prod`, no CORS headers
|
||||
are sent. Cookie security and log format do not depend on it.
|
||||
|
||||
@@ -897,10 +853,11 @@ The directory for every SQLite database and the lock file. Default
|
||||
`/var/lib/webhooker`. It is created if it is missing.
|
||||
|
||||
Exactly one webhooker process may use a `DATA_DIR` at a time. Two processes on
|
||||
one directory would both run delivery recovery and deliver every pending event
|
||||
twice. At startup the process locks `{DATA_DIR}/webhooker.lock` with `flock`,
|
||||
the kernel's file lock, and holds the lock until it exits. A second process
|
||||
pointed at the same directory prints a message naming it and exits non-zero.
|
||||
one directory would both run the recovery routine and deliver every pending
|
||||
event twice. At startup the process locks `{DATA_DIR}/webhooker.lock` with
|
||||
`flock`, the kernel's file lock, and holds the lock until it exits. A second
|
||||
process pointed at the same directory prints a message naming it and exits
|
||||
non-zero.
|
||||
|
||||
The kernel releases the lock when the process exits, even on `kill -9`. A
|
||||
leftover `webhooker.lock` blocks nothing and must not be deleted by hand. To run
|
||||
@@ -911,9 +868,9 @@ host. It does not work across hosts on a network filesystem.
|
||||
|
||||
### `DEBUG`
|
||||
|
||||
`true` or `false`. Default `false`. It accepts the spellings Go's
|
||||
`strconv.ParseBool` accepts: `1`, `t`, `T`, `TRUE`, `true`, `True`, `0`, `f`,
|
||||
`F`, `FALSE`, `false`, `False`. `yes` and `on` are refused.
|
||||
`true` or `false`. Default `false`. It accepts the twelve spellings Go's
|
||||
`strconv.ParseBool` accepts, such as `true`, `1` and `false`. `yes` and `on` are
|
||||
refused.
|
||||
|
||||
`DEBUG=true` lowers the log level to `DEBUG`. Lines at that level include:
|
||||
|
||||
@@ -991,15 +948,15 @@ value replaces the default.
|
||||
|
||||
The list decides whose `X-Forwarded-For` header is believed. `X-Forwarded-For`
|
||||
is the header a proxy uses to pass on the address of the client it is serving.
|
||||
When the connecting peer is inside the list, the client address is read from
|
||||
that header. For any other peer, the client is the peer itself and the header is
|
||||
ignored. `X-Real-IP` and `True-Client-IP` are never read.
|
||||
When the connecting machine is inside the list, the client address is read from
|
||||
that header. For any other connecting machine, the client is that machine itself
|
||||
and the header is ignored. `X-Real-IP` and `True-Client-IP` are never read.
|
||||
|
||||
The header is read from right to left. The first entry that is not itself a
|
||||
trusted proxy is the client. An entry that is not a bare IP address ends the
|
||||
walk, and the peer address is used instead. At most 64 entries are examined.
|
||||
Your proxy must therefore append the peer address to `X-Forwarded-For`, as
|
||||
nginx's `$proxy_add_x_forwarded_for` does.
|
||||
walk, and the connecting machine's address is used instead. At most 64 entries
|
||||
are examined. Your proxy must therefore append the connecting address to
|
||||
`X-Forwarded-For`, as nginx's `$proxy_add_x_forwarded_for` does.
|
||||
|
||||
If any client can reach webhooker, or the proxy in front of it, from a private
|
||||
address, set the list to the proxy's address alone. Otherwise those clients can
|
||||
@@ -1016,6 +973,7 @@ Rate-limit buckets are per IPv4 address and per IPv6 `/64` prefix.
|
||||
|
||||
### Allowing egress to your own network
|
||||
|
||||
Egress means the connections webhooker opens outward to deliver.
|
||||
`ALLOWED_EGRESS_CIDRS` is a comma-separated list of networks in CIDR notation
|
||||
that delivery targets may reach although the default rules refuse them. Default
|
||||
empty. A bare address counts as a single host. Example:
|
||||
@@ -1064,11 +1022,17 @@ default but is reopened by a list that covers it.
|
||||
|
||||
### Target types
|
||||
|
||||
An `http` target posts the event body to its URL. The request carries the
|
||||
event's `Content-Type`, the event's headers, then the target's configured
|
||||
headers, then `User-Agent: webhooker/{version}`. A configured header replaces an
|
||||
event header of the same name. A `2xx` response is a success. The first 4 KiB of
|
||||
the response body is stored with the attempt.
|
||||
An `http` target posts the event body to its URL. The request carries, in this
|
||||
order:
|
||||
|
||||
1. the event's `Content-Type`;
|
||||
2. the event's headers;
|
||||
3. the target's configured headers, each replacing an event header of the same
|
||||
name;
|
||||
4. `User-Agent: webhooker/{version}`.
|
||||
|
||||
A `2xx` response is a success. The first 4 KiB of the response body is stored
|
||||
with the attempt.
|
||||
|
||||
These event headers are never forwarded:
|
||||
|
||||
@@ -1091,8 +1055,8 @@ An `http` target's settings:
|
||||
- `Trailer`
|
||||
- `User-Agent`
|
||||
- `timeout_seconds`, the per-request timeout, at most 300. Blank means the
|
||||
shared 30 second default. A long timeout holds one of the ten delivery workers
|
||||
for its whole duration.
|
||||
shared 30 second default. A long timeout holds one of the ten background
|
||||
workers for its whole duration.
|
||||
- `forward_query`, off by default. On, the event's query string is appended to
|
||||
the URL, joined with `&` if the URL already has one.
|
||||
|
||||
@@ -1122,8 +1086,7 @@ attempted. `retrying` means at least one attempt failed and more remain.
|
||||
`delivered` means an attempt succeeded. `failed` means every allowed attempt
|
||||
failed.
|
||||
|
||||
Every attempt is recorded with its number, success, status code, response body,
|
||||
error and duration. One case is not recorded: an attempt cut off by shutdown.
|
||||
Every attempt is recorded, except one cut off by shutdown.
|
||||
|
||||
### Circuit breaker
|
||||
|
||||
@@ -1141,42 +1104,30 @@ breaker opens again.
|
||||
The webhook page says when a target's deliveries are paused and until when. Each
|
||||
waiting delivery in the event log shows the earliest time it can be tried.
|
||||
|
||||
### Workers and queues
|
||||
|
||||
Ten workers carry out deliveries. New deliveries wait in one queue and retries
|
||||
in another. Each queue holds 10,000 tasks. At most ten deliveries are in flight
|
||||
at once, however deep the queues get.
|
||||
|
||||
When a retry falls due and its queue is full, the retry is dropped and the
|
||||
delivery stays `retrying` in the database. The recovery sweep picks it up.
|
||||
|
||||
Each task carries the target's settings and the event. A body under 16 KiB
|
||||
travels with the task. A larger one is read from the database when the attempt
|
||||
runs.
|
||||
|
||||
### Recovery
|
||||
|
||||
A recovery routine runs once at start and then every 60 seconds. For every
|
||||
webhook it reads the deliveries still `pending` or `retrying`. One that already
|
||||
has a successful attempt recorded is marked `delivered` and not sent again. A
|
||||
`retrying` delivery is scheduled for whatever remains of its wait. A `pending`
|
||||
one is queued. A delivery the engine is already working on is left alone.
|
||||
The recovery routine finds deliveries the delivery engine is not working on and
|
||||
queues them again. It runs once at start and then every 60 seconds; each
|
||||
periodic run is a sweep. It reads the deliveries still `pending` or `retrying`
|
||||
in every event database. One that already has a successful attempt recorded is
|
||||
marked `delivered` and not sent again. A `retrying` delivery is scheduled for
|
||||
whatever remains of its wait. A `pending` one is queued. A delivery the engine
|
||||
is already working on is left alone.
|
||||
|
||||
At start, every `pending` delivery is taken, however recent. So a delivery cut
|
||||
off by a stop moments earlier is sent again at once. The periodic sweep takes
|
||||
only `pending` deliveries untouched for 15 minutes, at most 500 per webhook each
|
||||
off by a stop moments earlier is sent again at once. A sweep takes only
|
||||
`pending` deliveries untouched for 15 minutes, at most 500 per webhook each
|
||||
time.
|
||||
|
||||
If a target was deleted while a delivery to it was still open, recovery marks
|
||||
the delivery `failed` and records the reason.
|
||||
If a target was deleted while a delivery to it was still open, the recovery
|
||||
routine marks the delivery `failed` and records the reason.
|
||||
|
||||
### Shutdown
|
||||
|
||||
On `SIGINT` or `SIGTERM`, components stop in reverse start order under a 5
|
||||
second budget for the whole sequence. The HTTP server drains open requests,
|
||||
which means it waits for them to finish, for at most 3 seconds. That leaves 2
|
||||
seconds for the components after it. A Sentry flush, if enabled, fits inside the
|
||||
same budget and is skipped when little is left.
|
||||
On `SIGINT` or `SIGTERM`, the whole shutdown has a 5 second budget. The HTTP
|
||||
server first waits for open requests to finish, for at most 3 seconds. That
|
||||
leaves 2 seconds for the rest. A Sentry flush, if enabled, fits inside the same
|
||||
budget and is skipped when little is left.
|
||||
|
||||
The delivery engine cancels any `http` or `slack` request in flight. That
|
||||
attempt is not recorded, and the delivery keeps its status, so it is sent again
|
||||
@@ -1187,8 +1138,9 @@ after it are skipped, the database close included. The database files are then
|
||||
left as a crash leaves them, with their `-wal` and `-shm` files beside them.
|
||||
|
||||
Keep the container's stop grace period above 5 seconds. That is the time Docker
|
||||
waits after `docker stop` before it kills the process. Docker's default is 10. A
|
||||
shorter grace kills the process before the shutdown can run.
|
||||
waits after `docker stop` before it kills the process. Docker's default is 10.
|
||||
The shutdown starts at the signal, and a shorter grace cuts it off before it
|
||||
finishes.
|
||||
|
||||
### Archives
|
||||
|
||||
@@ -1244,20 +1196,29 @@ Rows from a period file carry a `period` field. A body that is not valid UTF-8
|
||||
is written in base64 with `"body_encoding": "base64"` beside it. The download
|
||||
streams one file at a time and creates nothing.
|
||||
|
||||
## Data reference
|
||||
## Data files
|
||||
|
||||
### Files
|
||||
The main database, `webhooker.db`, holds:
|
||||
|
||||
`webhooker.db` holds the application tier: settings, users, API keys, webhooks,
|
||||
entrypoints and targets. Each `events-{webhook_uuid}.db` holds that webhook's
|
||||
event tier: events, deliveries, delivery attempts and running totals. Each
|
||||
`archive-*.db` holds one archive target's rows.
|
||||
- the settings, which today are only the session key;
|
||||
- the `admin` account and its password hash;
|
||||
- the webhooks;
|
||||
- the entrypoints;
|
||||
- the targets.
|
||||
|
||||
Every database is opened through one code path, in WAL mode, with at most 4
|
||||
connections. A connection waits up to 10 seconds for a locked database before
|
||||
giving up. Under WAL a reader never blocks a writer.
|
||||
Each event database, `events-{webhook_uuid}.db`, holds that webhook's:
|
||||
|
||||
An event database is created with its webhook. Every webhook's database is
|
||||
- events;
|
||||
- deliveries;
|
||||
- delivery attempts;
|
||||
- running totals, which the pages read instead of counting rows.
|
||||
|
||||
Each archive file, `archive-*.db`, holds one archive target's rows.
|
||||
|
||||
Every database is opened in WAL mode, with at most 4 connections. A connection
|
||||
waits up to 10 seconds for a locked database before giving up.
|
||||
|
||||
An event database is created with its webhook. Every webhook's event database is
|
||||
opened at start. One that is missing or empty is created again and logged with
|
||||
`created a new, empty database` at `WARN`. On a deployment that has run before,
|
||||
that line means the file was lost, most often because the volume holding
|
||||
@@ -1269,82 +1230,10 @@ Deleting a webhook soft-deletes its row, entrypoints and targets in
|
||||
with a deletion time and hides it from the application; the row stays in the
|
||||
file. Deleting an entrypoint or a target soft-deletes it.
|
||||
|
||||
### Schema
|
||||
|
||||
Each kind of database file has its schema in SQL files built into the binary,
|
||||
under `internal/db/migrations/`: `main/` for `webhooker.db`, `events/` for event
|
||||
databases and `archive/` for archive files. In each, `000_migration.sql` creates
|
||||
the `schema_migrations` table and `001_schema.sql` holds the schema. Opening a
|
||||
database applies every file its `schema_migrations` does not list, in one
|
||||
transaction.
|
||||
|
||||
### Application tier
|
||||
|
||||
Every entity below except `Setting` has `id` (a version 4 UUID), `created_at`,
|
||||
`updated_at` and `deleted_at`. A row with `deleted_at` set is soft-deleted.
|
||||
|
||||
`Setting` is a key-value row with no other fields. The one key stored is
|
||||
`session_key`, the base64 session key made on first start.
|
||||
|
||||
`User` has `username` (unique, at most 1024 bytes) and `password` (the Argon2id
|
||||
hash). The first start creates `admin` with a random 16-character password.
|
||||
|
||||
`Webhook` has `user_id`, `name`, `description` and `retention_days`.
|
||||
`retention_days` is `0` for forever, or a whole number from 1 to 106751. The cap
|
||||
keeps the retention cutoff inside the range of a Go duration.
|
||||
|
||||
`Entrypoint` has `webhook_id`, `path`, `description` and `active`. `path` is the
|
||||
bare UUID. The `/h/` prefix is route only and is not stored.
|
||||
|
||||
`Target` has these fields. Each type fills only its own settings.
|
||||
|
||||
- `webhook_id`, `name`, `type` and `active`. `type` is one of `http`, `slack`,
|
||||
`database` or `log`.
|
||||
- `url`, `headers`, `timeout_seconds` and `forward_query`, the `http` settings.
|
||||
A `slack` target uses `url` alone.
|
||||
- `archive_expiry` and `archive_rotation`, the archive target's settings.
|
||||
`archive_expiry` is stored in nanoseconds, `0` for never.
|
||||
- `max_attempts`, for `http` and `slack`.
|
||||
|
||||
`APIKey` has `user_id`, `key`, `description` and `last_used_at`. Its table,
|
||||
`api_keys`, exists, but nothing creates a key and nothing checks one.
|
||||
|
||||
### Event tier
|
||||
|
||||
`Event` has these fields:
|
||||
|
||||
- `webhook_id` and `entrypoint_id`;
|
||||
- `method`, which is always `POST`;
|
||||
- `raw_query`, `headers` (as JSON), `body`, `body_bytes` and `content_type`;
|
||||
- `resubmitted_from_id`, the event a resubmit copied. It is empty for an event
|
||||
that arrived on the receiver.
|
||||
|
||||
`Delivery` has these fields:
|
||||
|
||||
- `event_id` and `target_id`;
|
||||
- `status`;
|
||||
- `finished_at`, set when the status becomes `delivered` or `failed`;
|
||||
- `replay`.
|
||||
|
||||
`DeliveryResult` is one attempt. It has these fields:
|
||||
|
||||
- `delivery_id` and `attempt_num`;
|
||||
- `success` and `status_code`;
|
||||
- `response_body`, the first 4 KiB;
|
||||
- `error`;
|
||||
- `duration`, in milliseconds.
|
||||
|
||||
Three tables hold running totals so the pages need not count rows. Each count
|
||||
changes in the transaction that writes or deletes the rows it counts.
|
||||
|
||||
- `EventTotals` is one row: `events`, `events_removed` and `last_event_at`.
|
||||
- `TargetTotals` is one row per target, with:
|
||||
- `deliveries`, `delivered` and `failed`;
|
||||
- `deliveries_removed` and `failed_removed`.
|
||||
- `EntrypointTotals` is one row per entrypoint: `last_event_at`.
|
||||
|
||||
Retention raises the `_removed` counts and leaves `last_event_at` alone, so a
|
||||
webhook's last event is still shown after its events have expired.
|
||||
databases and `archive/` for archive files. Opening a database applies the files
|
||||
it has not had yet.
|
||||
|
||||
## HTTP routes reference
|
||||
|
||||
@@ -1372,34 +1261,10 @@ Sign-in routes:
|
||||
- `POST /pages/login` signs in. See [Sign-in limits](#sign-in-limits).
|
||||
- `POST /pages/logout` signs out.
|
||||
|
||||
Routes behind the sign-in. A `GET` without a session is redirected to
|
||||
`/pages/login` with the path as `next`:
|
||||
|
||||
- `GET /user/{username}` is the profile page. Another user's name gets `403`.
|
||||
- `POST /user/{username}/password` changes the password, 5 per minute per
|
||||
client.
|
||||
- `GET /settings` lists the loaded configuration.
|
||||
- `GET /hooks` lists webhooks. `GET` and `POST /hooks/new` create one.
|
||||
- `GET /hook/{id}` is the webhook page. `GET` and `POST /hook/{id}/edit` edit
|
||||
it. `POST /hook/{id}/delete` deletes it.
|
||||
- `GET /hook/{id}/events` is the event log. `?show=failed` and `?show=pending`
|
||||
narrow it.
|
||||
- `GET /hook/{id}/events/{eventID}` is one event's page, and
|
||||
`GET /hook/{id}/events/{eventID}/body` downloads its stored body.
|
||||
- `POST /hook/{id}/deliveries/{deliveryID}/replay` replays a delivery, and
|
||||
`POST /hook/{id}/events/{eventID}/resubmit` resubmits an event, each 30 per
|
||||
minute per client.
|
||||
- `POST /hook/{id}/entrypoints` adds an entrypoint.
|
||||
- `POST /hook/{id}/entrypoints/{entrypointID}/edit` changes its description.
|
||||
- `POST /hook/{id}/entrypoints/{entrypointID}/delete` deletes it.
|
||||
- `POST /hook/{id}/entrypoints/{entrypointID}/toggle` activates or deactivates
|
||||
it.
|
||||
- `POST /hook/{id}/targets` adds a target.
|
||||
- `GET /hook/{id}/targets/{targetID}/edit` shows the target edit form, and
|
||||
`POST` to the same path saves it.
|
||||
- `POST /hook/{id}/targets/{targetID}/delete` deletes the target.
|
||||
- `POST /hook/{id}/targets/{targetID}/toggle` activates or deactivates it.
|
||||
- `GET /hook/{id}/targets/{targetID}/download` downloads an archive.
|
||||
The pages behind the sign-in live under `/user/`, `/settings`, `/hooks` and
|
||||
`/hook/`. They are reached from the web interface. A `GET` to one of them
|
||||
without a session is redirected to `/pages/login`, with the page's path as
|
||||
`next`.
|
||||
|
||||
`GET /metrics` exists only when `METRICS_USERNAME` and `METRICS_PASSWORD` are
|
||||
set, behind basic authentication.
|
||||
@@ -1408,18 +1273,13 @@ set, behind basic authentication.
|
||||
|
||||
## Metrics reference
|
||||
|
||||
`/metrics` serves one metrics registry, the set of metrics it exposes. The
|
||||
registry carries:
|
||||
|
||||
- the Prometheus library's Go runtime collector;
|
||||
- its process collector;
|
||||
- the delivery metrics below;
|
||||
- three inbound HTTP metrics.
|
||||
`/metrics` serves the delivery metrics below, three inbound HTTP metrics, and
|
||||
the Go runtime and process metrics the Prometheus library provides.
|
||||
|
||||
Nine of the eleven metrics below carry one label, `target_type`, with one of
|
||||
`http`, `slack`, `database`, `log` or `unknown`. The two event counters carry
|
||||
none. Target ids and event ids are never labels, because a label value that
|
||||
grows with traffic would grow the registry without bound.
|
||||
grows with traffic would grow the set of metrics without bound.
|
||||
|
||||
- `webhooker_events_received_total` (counter, no label): events stored,
|
||||
resubmitted copies included.
|
||||
@@ -1436,8 +1296,8 @@ grows with traffic would grow the registry without bound.
|
||||
**Resubmit**.
|
||||
- `webhooker_delivery_duration_seconds` (histogram): wall time of one attempt.
|
||||
- `webhooker_deliveries_pending` and `webhooker_deliveries_retrying` (gauges):
|
||||
deliveries in those statuses, counted from the databases every 30 seconds.
|
||||
Deliveries to a deleted target count under `unknown`.
|
||||
deliveries in those statuses, counted from the event databases every 30
|
||||
seconds. Deliveries to a deleted target count under `unknown`.
|
||||
- `webhooker_circuit_breakers_open` (gauge): breakers currently open.
|
||||
|
||||
The success and failure counters move only after the new status has been
|
||||
@@ -1458,8 +1318,8 @@ set to `(all)`.
|
||||
- Docker. The tests, the linters, prettier and the stylesheet build run in
|
||||
Docker.
|
||||
|
||||
`gofmt` and `goimports` run on the host. These tools run only inside Docker, at
|
||||
the versions pinned (fixed to one exact release) in the `Dockerfile`,
|
||||
`gofmt` and `goimports` run on the host. The tools below run only inside Docker,
|
||||
at the versions pinned (fixed to one exact release) in the `Dockerfile`,
|
||||
`package.json` and `yarn.lock`:
|
||||
|
||||
- `golangci-lint`, the Go linter;
|
||||
@@ -1519,9 +1379,8 @@ The `lint` stage runs these steps:
|
||||
3. `golangci-lint run`, with `--network=none`;
|
||||
4. ESLint over `static/js/`, in the `js-lint` stage, with `--network=none`.
|
||||
|
||||
The `test` stage runs `go test -cover` on four packages at a time, each running
|
||||
at most eight tests in parallel, with a 90 second timeout per package. On a
|
||||
failure it runs only the failed tests again with `-v`.
|
||||
The `test` stage runs `go test -cover`. On a failure it runs only the failed
|
||||
tests again with `-v`.
|
||||
|
||||
The image build depends on the `lint`, `test`, `css-check` and `markdown-check`
|
||||
stages, so an image that built is a tree that passed every check. The runtime
|
||||
@@ -1569,12 +1428,12 @@ With the CSP build, every Alpine.js directive in `templates/` names a component
|
||||
registered in `static/js/app.js`, such as `x-data="collapsible"`, and never an
|
||||
inline expression.
|
||||
|
||||
The package file is committed as `3p/alpinejs-csp-3.14.9.tgz`, byte for byte as
|
||||
the npm registry, the JavaScript package registry, publishes it. `script/assets`
|
||||
extracts `package/dist/cdn.min.js` from it to `static/js/alpine.min.js`, which
|
||||
the binary embeds. `make build`, `make dev` and the Docker stages run it first.
|
||||
The extracted file is not committed, and a build that skips the extraction fails
|
||||
with an error naming `js/alpine.min.js`.
|
||||
The package file is committed as `3p/alpinejs-csp-3.14.9.tgz`. It is byte for
|
||||
byte the file the npm registry, the JavaScript package registry, publishes.
|
||||
`script/assets` extracts `package/dist/cdn.min.js` from it to
|
||||
`static/js/alpine.min.js`, which the binary embeds. `make build`, `make dev` and
|
||||
the Docker stages run it first. The extracted file is not committed, and a build
|
||||
that skips the extraction fails with an error naming `js/alpine.min.js`.
|
||||
|
||||
To move to a new version:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user