Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b41d29cc96 |
@@ -1,32 +1,31 @@
|
||||
# webhooker
|
||||
|
||||
webhooker is an MIT-licensed web service, written in [Go](https://golang.org) by
|
||||
[@sneak](https://sneak.berlin), that receives the HTTP notifications other
|
||||
services send and forwards them to your own systems.
|
||||
[@sneak](https://sneak.berlin), that receives the HTTP requests other services
|
||||
send when something happens and forwards them to your own systems.
|
||||
|
||||
Many services send an HTTP request to a URL you give them when something
|
||||
happens: a code push, a payment, a new ticket. webhooker receives those
|
||||
requests, stores them, and forwards them to the places you choose.
|
||||
requests, stores each one as an event, and forwards it to the places you choose.
|
||||
|
||||
webhooker is a single binary with a web interface. It stores its data in SQLite,
|
||||
a database engine that keeps each database in one ordinary file. It writes each
|
||||
notification to disk before answering the sender. It then delivers it to each
|
||||
target you configured. The web interface shows the newest notifications, with
|
||||
every delivery and attempt they had. From there you can send a stored
|
||||
notification again. webhooker also exports metrics for Prometheus, the
|
||||
monitoring system, so you can watch the whole flow.
|
||||
event to disk before answering the sender. It then delivers the event to each
|
||||
target you configured. The web interface shows the newest events, with every
|
||||
delivery and attempt they had. From there you can send a stored event again.
|
||||
webhooker also exports metrics for Prometheus, the monitoring system, so you can
|
||||
watch the whole flow.
|
||||
|
||||
## Rationale
|
||||
|
||||
Senders of these notifications usually make one attempt and keep no record. If
|
||||
your service is down at that moment, the notification is lost. Few senders let
|
||||
you see what they sent, or when, or whether it arrived.
|
||||
webhooker exists so that a request from another service is not lost while your
|
||||
service is down. It also exists so that you can see what was sent and what
|
||||
happened to it.
|
||||
|
||||
webhooker sits between the sender and your service and fixes those three things.
|
||||
It is always ready to receive, and it stores before it forwards, so nothing is
|
||||
lost while your service is down. It keeps every notification and every delivery
|
||||
attempt, so you can read what was sent and what happened to it. It can forward
|
||||
one notification to several targets, and it can send a stored one again.
|
||||
It stores each event before it forwards it, so an event that arrives while your
|
||||
service is down is not lost. It keeps events and their delivery attempts for the
|
||||
retention period you set. It can forward one event to several targets, and it
|
||||
can send a stored event again.
|
||||
|
||||
## How it works
|
||||
|
||||
@@ -40,15 +39,18 @@ Five words name the moving parts.
|
||||
- A **target** is a place events are delivered to. There are four kinds: an HTTP
|
||||
URL, a Slack or Mattermost message-posting URL, an archive database file, and
|
||||
the application log.
|
||||
- An **event** is one stored notification: its method, query string, headers and
|
||||
- An **event** is one stored request: its method, query string, headers and
|
||||
body.
|
||||
- A **delivery** is one event paired with one target. It records every attempt.
|
||||
|
||||
A request to an entrypoint is stored as an event. In the same database
|
||||
transaction, one pending delivery is created for each active target of the
|
||||
webhook. The sender gets `200` once that transaction has committed. Workers then
|
||||
carry out the deliveries. A delivery to an HTTP or Slack target that fails is
|
||||
retried, up to the number of attempts the target allows.
|
||||
The part of webhooker that answers requests to `/h/{uuid}` is the receiver.
|
||||
|
||||
A request to an entrypoint is stored as an event. One pending delivery is
|
||||
created for each active target of the webhook, in the same database write as the
|
||||
event: either all of them are stored or none. The sender gets `200` once they
|
||||
are stored. Workers then carry out the deliveries. A delivery to an HTTP or
|
||||
Slack target that fails is retried, up to the number of attempts the target
|
||||
allows.
|
||||
|
||||
Delivery is at least once, not exactly once. If a delivery reaches the target
|
||||
but webhooker cannot record that outcome, it sends the delivery again later. A
|
||||
@@ -179,8 +181,13 @@ A target's type cannot be changed. Create a new target to deliver another way.
|
||||
|
||||
### Statistics
|
||||
|
||||
The top of the webhook page shows counts of entrypoints and targets, deliveries
|
||||
in progress, the time of the last event, and the retention setting.
|
||||
The top of the webhook page shows:
|
||||
|
||||
- the number of entrypoints, and how many are active;
|
||||
- the number of targets, and how many are active;
|
||||
- deliveries in progress;
|
||||
- the time of the last event;
|
||||
- the retention setting.
|
||||
|
||||
Below that are events, deliveries and failures, both lifetime and within
|
||||
retention. Next are figures for the last 10 minutes and the last 24 hours:
|
||||
@@ -257,22 +264,21 @@ The image listens on every IPv4 address inside the container: it ships
|
||||
`BIND_ADDRESS=0.0.0.0`. A container has its own network namespace, which means
|
||||
its own addresses, separate from the host's. That is the boundary, and the `-p`
|
||||
flag decides what the host exposes. Publish to `127.0.0.1:8080`, the address
|
||||
your reverse proxy connects from, and nothing wider. A bare `-p 8080:8080` opens
|
||||
the port on every interface of the host, and Docker's forwarding rules sit ahead
|
||||
of most host firewalls.
|
||||
your reverse proxy connects to, and nothing wider. A bare `-p 8080:8080`
|
||||
publishes the port on every interface of the host.
|
||||
|
||||
Do not set `BIND_ADDRESS` to an empty value in a container. An empty value means
|
||||
unset, and unset falls back to the binary's default of `127.0.0.1`. The
|
||||
container then answers nothing on its published port and its health check fails.
|
||||
A templated Compose file with an empty value is the usual way this happens.
|
||||
|
||||
A container on an IPv6-only Docker network needs `BIND_ADDRESS=::`.
|
||||
|
||||
The app runs as user `webhooker` (UID 1000). The image's entrypoint script
|
||||
starts as root. It creates `DATA_DIR` if it is missing, gives it and its
|
||||
contents to `webhooker`, and sets the directory to mode `0750`. Then it runs the
|
||||
app as `webhooker`. So a host directory can be mounted as it is, whoever owns
|
||||
it. With `--user`, the script changes nothing and runs the app as that user.
|
||||
The app runs as user `webhooker` (UID 1000). The image's start-up script,
|
||||
`deploy/docker-entrypoint.sh`, starts as root. It creates `DATA_DIR` if it is
|
||||
missing, gives it and its contents to `webhooker`, and sets the directory to
|
||||
mode `0750`. Then it runs the app as `webhooker`. So a host directory can be
|
||||
mounted as it is, whoever owns it. With `--user`, the script changes nothing and
|
||||
runs the app as that user.
|
||||
|
||||
webhooker creates every SQLite file with mode `0600`, and resets a file found
|
||||
with another mode when it opens it. `webhooker.db` holds target settings in
|
||||
@@ -325,10 +331,11 @@ docker start upaas-webhooker
|
||||
|
||||
### Behind a reverse proxy
|
||||
|
||||
webhooker speaks plain HTTP. It does not terminate TLS, the encryption behind
|
||||
HTTPS. Put a reverse proxy in front of it: a server that accepts the public
|
||||
connections and passes them on. The proxy holds the certificate and reaches
|
||||
webhooker over loopback, the address only this host can reach.
|
||||
webhooker speaks plain HTTP. It does not handle TLS, the encryption behind
|
||||
HTTPS: it holds no certificate and accepts no HTTPS connections. Put a reverse
|
||||
proxy in front of it: a server that accepts the public connections and passes
|
||||
them on. The proxy holds the certificate and reaches webhooker over loopback,
|
||||
the address only this host can reach.
|
||||
|
||||
Five things must be right.
|
||||
|
||||
@@ -488,10 +495,8 @@ without it is missing data and may have no readable tables at all.
|
||||
A clean shutdown closes every database and removes its `-wal` and `-shm` files.
|
||||
A crash leaves them, and they must be carried with the `.db`.
|
||||
|
||||
`DATA_DIR` must be on a local filesystem. SQLite maps the `-shm` file into
|
||||
memory, which network filesystems do not support reliably. webhooker does not
|
||||
check this. An error reading `disk I/O error (522)` points at the storage under
|
||||
`DATA_DIR`.
|
||||
Keep `DATA_DIR` on a local disk. WAL mode is not reliable on network
|
||||
filesystems, and webhooker does not check for one.
|
||||
|
||||
Configuration is not in `DATA_DIR`. It comes from the environment and from a
|
||||
`.env` file in the working directory. Back those up with your deployment.
|
||||
@@ -519,14 +524,11 @@ for db in /path/to/data/*.db; do
|
||||
done
|
||||
```
|
||||
|
||||
`.backup` reads through the WAL and writes one complete file. The runtime image
|
||||
`.backup` writes one complete file, with no `-wal` beside it. The runtime image
|
||||
has no `sqlite3`, so run this on the host or from another container with the
|
||||
volume mounted. Each file is captured at its own instant, so an event stored
|
||||
between two copies lands in one file and not the other.
|
||||
|
||||
`sqlite3 <db> .dump` is an export, not a backup. It is safe to run but holds a
|
||||
read transaction open for its whole duration.
|
||||
|
||||
Archive files are the exception. The archive writer closes and reopens its file
|
||||
after a write, at most once per second. So you can move a finished archive file
|
||||
away while the service runs. Move its `-wal` and `-shm` with it, or wait until
|
||||
@@ -560,8 +562,8 @@ who can read it.
|
||||
Each entrypoint is a random version 4 UUID served at `/h/{uuid}`. Holding that
|
||||
URL is what authorises a sender. The receiver checks nothing else: no shared
|
||||
secret, no HMAC signature (a hash of the body made with a shared key), no token
|
||||
header. None will be added. Signature headers a sender adds anyway, such as
|
||||
`X-Hub-Signature`, are stored and forwarded as ordinary headers.
|
||||
header. Signature headers a sender adds anyway, such as `X-Hub-Signature`, are
|
||||
stored and forwarded as ordinary headers.
|
||||
|
||||
What that means in practice:
|
||||
|
||||
@@ -647,9 +649,8 @@ the proxy.
|
||||
### Delivery destinations
|
||||
|
||||
A delivery target could be used to make webhooker send requests into the network
|
||||
it sits in and read the answers back. This is called server-side request
|
||||
forgery. webhooker refuses it by default, with a check called the destination
|
||||
check below.
|
||||
it sits in and read the answers back. This is called server-side request forgery
|
||||
(SSRF). webhooker refuses it with the SSRF guard, described here.
|
||||
|
||||
Every target URL must use `http` or `https` and resolve to a public address.
|
||||
These private and reserved ranges are refused:
|
||||
@@ -668,16 +669,16 @@ These private and reserved ranges are refused:
|
||||
One public address is refused too: `168.63.129.16`, the Azure service that hands
|
||||
a virtual machine its credentials.
|
||||
|
||||
The destination check runs when a target is saved and again when each delivery
|
||||
The SSRF guard runs when a target is saved and again when each delivery
|
||||
connects. Both use the same function. The address is resolved again at connect
|
||||
time, so a hostname that changes its address after validation is still checked.
|
||||
|
||||
Redirects are followed up to ten hops. Each hop is checked the same way. A hop
|
||||
that leaves the target's origin carries none of the target's configured headers
|
||||
and none of the forwarded event headers. Leaving the origin means a different
|
||||
host, a different port, or a step from `https` to `http`. This stops an open
|
||||
redirect, a destination that forwards requests to any URL a caller names, from
|
||||
handing a credential to another host.
|
||||
host, a different port, or a step from `https` to `http`. An open redirect is a
|
||||
destination that forwards requests to any URL a caller names. This rule stops
|
||||
one from handing a credential to another host.
|
||||
|
||||
`ALLOWED_EGRESS_CIDRS` opens named networks for the common case of forwarding to
|
||||
your own network. Read
|
||||
@@ -705,20 +706,19 @@ peer, which behind a proxy is the proxy. `clientIP` is the client as decided by
|
||||
`TRUSTED_PROXIES`.
|
||||
|
||||
SQL statements are logged with `?` in place of each value, never with the values
|
||||
themselves. Session cookies, API keys and target credentials are never logged.
|
||||
themselves. Outside a `log` target's line, no line carries session cookies or
|
||||
target credentials.
|
||||
|
||||
Two things are logged in the clear. The first is the `admin` password, once, on
|
||||
the first boot, and again when `webhooker resetpw -generate` is run. The second
|
||||
is the operator's own configuration echoed back: webhook names and target
|
||||
hostnames.
|
||||
The `admin` password is logged in the clear once, on the first boot, and again
|
||||
when `webhooker resetpw -generate` is run. The operator's own configuration is
|
||||
echoed back in the clear: webhook names and target hostnames.
|
||||
|
||||
A `log` target writes the whole event to the log: query string, headers and
|
||||
body. That is its purpose.
|
||||
|
||||
A recovered panic is a crash inside one request that the server caught. It
|
||||
writes one `ERROR` record of at most 10,240 bytes. The record carries the
|
||||
request id of the request that caused it. A request id is a random token given
|
||||
to each request and written in its log lines.
|
||||
writes one `ERROR` record of at most 10,240 bytes. The record carries the same
|
||||
`request_id` as the request's access log line, which ties the two together.
|
||||
|
||||
When `SENTRY_DSN` is set, error reports go to Sentry, an error reporting
|
||||
service, with the request reduced to this:
|
||||
@@ -735,13 +735,13 @@ service, with the request reduced to this:
|
||||
|
||||
One request to an entrypoint goes through these steps:
|
||||
|
||||
1. The router matches `POST /h/{uuid}`. Any other method gets `405`.
|
||||
2. The rate limits run: one per client per entrypoint, one per client across the
|
||||
1. The rate limits run: one per client per entrypoint, one per client across the
|
||||
whole route.
|
||||
2. The handler answers any method but `POST` with `405`.
|
||||
3. The handler looks up the entrypoint. Unknown gets `404`, deactivated `410`.
|
||||
4. The handler reads the body, up to 1 MB. A larger body gets `413`.
|
||||
5. One transaction stores the event, one pending delivery per active target, and
|
||||
the running totals. The sender gets `200`.
|
||||
5. The event, one pending delivery per active target, and the running totals are
|
||||
stored together, or not at all. The sender gets `200`.
|
||||
6. One task per delivery goes onto the delivery queue.
|
||||
7. A worker takes the task and runs the target's delivery. The outcome is
|
||||
recorded as an attempt, and the delivery's status is updated.
|
||||
@@ -749,15 +749,15 @@ One request to an entrypoint goes through these steps:
|
||||
### Components and middleware
|
||||
|
||||
The components are wired by Uber fx, a library that builds each component when
|
||||
another needs it. It runs their start hooks (the code each component runs at
|
||||
start) in that order, and their stop hooks in reverse. The components are:
|
||||
another needs it. fx starts the components in the order their dependencies need,
|
||||
and stops them in reverse. The components are:
|
||||
|
||||
- the globals, the logger and the config;
|
||||
- the main database and the per-webhook database manager;
|
||||
- the retention sweep and the health check;
|
||||
- the session and the handlers;
|
||||
- the metrics registry and collectors;
|
||||
- the middleware and the destination check;
|
||||
- the middleware and the SSRF guard;
|
||||
- the delivery engine, the archive writers and the archive sweep;
|
||||
- the server.
|
||||
|
||||
@@ -800,13 +800,13 @@ The body cap comes before CSRF because CSRF parses the form.
|
||||
protection.
|
||||
- [go-chi/httprate](https://github.com/go-chi/httprate) rate-limits the
|
||||
receiver, the password change, replay and resubmit.
|
||||
- The [Prometheus](https://prometheus.io) client and
|
||||
- The [Prometheus](https://prometheus.io) library and
|
||||
[go-http-metrics](https://github.com/slok/go-http-metrics) serve the metrics.
|
||||
- The [Sentry](https://sentry.io) client reports errors when configured.
|
||||
- The [Sentry](https://sentry.io) library reports errors when configured.
|
||||
|
||||
### Package layout
|
||||
|
||||
All application code is under `internal/`. The program starts in
|
||||
Most of the Go code is under `internal/`. The program starts in
|
||||
`cmd/webhooker/main.go`, which dispatches subcommands, takes the `DATA_DIR` lock
|
||||
and wires the components.
|
||||
|
||||
@@ -818,7 +818,7 @@ and wires the components.
|
||||
- `internal/datadir`: the exclusive lock on `DATA_DIR`.
|
||||
- `internal/db/migrations`: the SQL schema files and their application.
|
||||
- `internal/delivery`: the engine, the four target types, retries, the circuit
|
||||
breaker and the destination check.
|
||||
breaker and the SSRF guard.
|
||||
- `internal/globals`: build-time variables.
|
||||
- `internal/gormlog`: GORM's logger on top of `slog`, with values withheld.
|
||||
- `internal/handlers`: every HTTP handler and page.
|
||||
@@ -889,7 +889,7 @@ is refused; the port belongs to `PORT`.
|
||||
|
||||
An address that parses but is not on this host passes configuration and fails at
|
||||
bind time. The process logs the error and exits non-zero. The `bindAddress`
|
||||
field of the startup log shows what was bound.
|
||||
field of the `Configuration loaded` line shows the configured address.
|
||||
|
||||
### `DATA_DIR`
|
||||
|
||||
@@ -939,9 +939,9 @@ startup fails with an error naming both variables.
|
||||
|
||||
### `SENTRY_DSN`
|
||||
|
||||
The DSN for Sentry error reporting: the URL that tells the Sentry client where
|
||||
The DSN for Sentry error reporting: the URL that tells the Sentry library where
|
||||
to send reports. Default empty, which leaves reporting off. A set value is
|
||||
parsed with the Sentry client's own parser, and one it cannot parse aborts
|
||||
parsed with the Sentry library's own parser, and one it cannot parse aborts
|
||||
startup. The release name sent to Sentry is `webhooker-{version}`.
|
||||
|
||||
### `RETENTION_SWEEP_INTERVAL`
|
||||
@@ -995,11 +995,11 @@ When the connecting peer is inside the list, the client address is read from
|
||||
that header. For any other peer, the client is the peer itself and the header is
|
||||
ignored. `X-Real-IP` and `True-Client-IP` are never read.
|
||||
|
||||
The header is read from right to left. The first hop that is not itself a
|
||||
trusted proxy is the client. A hop that is not a bare IP address ends the walk,
|
||||
and the peer address is used instead. At most 64 hops are examined. Your proxy
|
||||
must therefore append the peer address to `X-Forwarded-For`, as nginx's
|
||||
`$proxy_add_x_forwarded_for` does.
|
||||
The header is read from right to left. The first entry that is not itself a
|
||||
trusted proxy is the client. An entry that is not a bare IP address ends the
|
||||
walk, and the peer address is used instead. At most 64 entries are examined.
|
||||
Your proxy must therefore append the peer address to `X-Forwarded-For`, as
|
||||
nginx's `$proxy_add_x_forwarded_for` does.
|
||||
|
||||
If any client can reach webhooker, or the proxy in front of it, from a private
|
||||
address, set the list to the proxy's address alone. Otherwise those clients can
|
||||
@@ -1025,8 +1025,8 @@ This only ever adds destinations. The default refusals stay in force for every
|
||||
address not listed. When the list is set, startup logs the whole list at `WARN`.
|
||||
|
||||
Anyone with the admin account can make webhooker send requests to every network
|
||||
you list. They can read the responses in the delivery log. List the smallest
|
||||
blocks that cover your destinations. Prefer `10.1.2.3/32` over `10.0.0.0/8`.
|
||||
you list. They can read the responses in the event log. List the smallest blocks
|
||||
that cover your destinations. Prefer `10.1.2.3/32` over `10.0.0.0/8`.
|
||||
`0.0.0.0/0` or `::/0` opens every private and reserved range at once, including
|
||||
this host's own loopback. Do not list them.
|
||||
|
||||
@@ -1102,9 +1102,9 @@ the body in a code block. A JSON body is indented. A body over 3500 bytes is
|
||||
cut. The URL is any message-posting URL that accepts Slack's format, which
|
||||
includes Mattermost's.
|
||||
|
||||
A `database` target, called an archive in the web interface, appends the event
|
||||
as a row to its own SQLite file. See [Archives](#archives). It makes no network
|
||||
request and never retries. A write failure fails the delivery.
|
||||
An archive target, whose stored type is `database`, appends the event as a row
|
||||
to its own SQLite file. See [Archives](#archives). It makes no network request
|
||||
and never retries. A write failure fails the delivery.
|
||||
|
||||
A `log` target writes the event, headers and body included, as one `INFO` line.
|
||||
It never retries.
|
||||
@@ -1180,11 +1180,11 @@ same budget and is skipped when little is left.
|
||||
|
||||
The delivery engine cancels any `http` or `slack` request in flight. That
|
||||
attempt is not recorded, and the delivery keeps its status, so it is sent again
|
||||
at the next start. A `database` or `log` task finishes as normal.
|
||||
at the next start. An archive or `log` task finishes as normal.
|
||||
|
||||
A component that does not stop in time is logged at `ERROR`, and the components
|
||||
after it are skipped. SQLite is crash-safe, so this loses no data, but the
|
||||
database is not closed cleanly.
|
||||
after it are skipped, the database close included. The database files are then
|
||||
left as a crash leaves them, with their `-wal` and `-shm` files beside them.
|
||||
|
||||
Keep the container's stop grace period above 5 seconds. That is the time Docker
|
||||
waits after `docker stop` before it kills the process. Docker's default is 10. A
|
||||
@@ -1232,12 +1232,17 @@ is the operator's decision.
|
||||
|
||||
**Download** on the webhook page returns the target's whole archive as one
|
||||
gzipped JSON file named
|
||||
`archive-{webhook_name}-{target_name}-{YYYYMMDDTHHMMSSZ}.json.gz`. It holds
|
||||
`webhook` and `target` (each an `id` and a `name`), `exported_at`, and
|
||||
`archived_events`, one object per row with every column. Rows from a period file
|
||||
carry a `period` field. A body that is not valid UTF-8 is written in base64 with
|
||||
`"body_encoding": "base64"` beside it. The download streams one file at a time
|
||||
and creates nothing.
|
||||
`archive-{webhook_name}-{target_name}-{YYYYMMDDTHHMMSSZ}.json.gz`. It holds one
|
||||
object with these fields:
|
||||
|
||||
- `webhook`, with the webhook's `id` and `name`;
|
||||
- `target`, with the target's `id` and `name`;
|
||||
- `exported_at`;
|
||||
- `archived_events`, one object per row with every column.
|
||||
|
||||
Rows from a period file carry a `period` field. A body that is not valid UTF-8
|
||||
is written in base64 with `"body_encoding": "base64"` beside it. The download
|
||||
streams one file at a time and creates nothing.
|
||||
|
||||
## Data reference
|
||||
|
||||
@@ -1297,7 +1302,7 @@ bare UUID. The `/h/` prefix is route only and is not stored.
|
||||
`database` or `log`.
|
||||
- `url`, `headers`, `timeout_seconds` and `forward_query`, the `http` settings.
|
||||
A `slack` target uses `url` alone.
|
||||
- `archive_expiry` and `archive_rotation`, the `database` settings.
|
||||
- `archive_expiry` and `archive_rotation`, the archive target's settings.
|
||||
`archive_expiry` is stored in nanoseconds, `0` for never.
|
||||
- `max_attempts`, for `http` and `slack`.
|
||||
|
||||
@@ -1314,8 +1319,12 @@ bare UUID. The `/h/` prefix is route only and is not stored.
|
||||
- `resubmitted_from_id`, the event a resubmit copied. It is empty for an event
|
||||
that arrived on the receiver.
|
||||
|
||||
`Delivery` has `event_id`, `target_id`, `status`, `finished_at` and `replay`.
|
||||
`finished_at` is set when the status becomes `delivered` or `failed`.
|
||||
`Delivery` has these fields:
|
||||
|
||||
- `event_id` and `target_id`;
|
||||
- `status`;
|
||||
- `finished_at`, set when the status becomes `delivered` or `failed`;
|
||||
- `replay`.
|
||||
|
||||
`DeliveryResult` is one attempt. It has these fields:
|
||||
|
||||
@@ -1342,13 +1351,19 @@ webhook's last event is still shown after its events have expired.
|
||||
Public routes:
|
||||
|
||||
- `GET /` redirects to `/hooks` when signed in and to `/pages/login` otherwise.
|
||||
- `GET /.well-known/healthcheck` returns JSON with `status`, `now`,
|
||||
`uptimeSeconds`, `uptimeHuman`, `version` and `appname`.
|
||||
- `GET /.well-known/healthcheck` returns a JSON object with these fields:
|
||||
- `status`, always `ok`;
|
||||
- `now`, the current time;
|
||||
- `uptimeSeconds` and `uptimeHuman`;
|
||||
- `version` and `appname`.
|
||||
- `GET /s/*` and `HEAD /s/*` serve the embedded CSS and JavaScript. Other
|
||||
methods get `405` with `Allow: GET, HEAD`.
|
||||
- `POST /h/{uuid}` is the receiver. Any other method gets `405` with
|
||||
`Allow: POST`. An unknown UUID gets `404`, a deactivated entrypoint `410`, a
|
||||
body over 1 MB `413`, and a stored event `200` with `{"status":"ok"}`.
|
||||
- `POST /h/{uuid}` is the receiver. Its answers:
|
||||
- any method but `POST` gets `405` with `Allow: POST`;
|
||||
- an unknown UUID gets `404`;
|
||||
- a deactivated entrypoint gets `410`;
|
||||
- a body over 1 MB gets `413`;
|
||||
- a stored event gets `200` with `{"status":"ok"}`.
|
||||
|
||||
Sign-in routes:
|
||||
|
||||
@@ -1393,10 +1408,13 @@ set, behind basic authentication.
|
||||
|
||||
## Metrics reference
|
||||
|
||||
`/metrics` serves one Prometheus registry, the set of metrics it exposes. It
|
||||
carries the Go runtime and process collectors, which report the process's
|
||||
memory, threads and CPU use, the delivery metrics below, and three inbound HTTP
|
||||
metrics.
|
||||
`/metrics` serves one metrics registry, the set of metrics it exposes. The
|
||||
registry carries:
|
||||
|
||||
- the Prometheus library's Go runtime collector;
|
||||
- its process collector;
|
||||
- the delivery metrics below;
|
||||
- three inbound HTTP metrics.
|
||||
|
||||
Nine of the eleven metrics below carry one label, `target_type`, with one of
|
||||
`http`, `slack`, `database`, `log` or `unknown`. The two event counters carry
|
||||
@@ -1461,7 +1479,7 @@ make fmt-check # Fail if gofmt or prettier would change anything
|
||||
make lint # Run gofmt, golangci-lint and ESLint in Docker
|
||||
make test # Run the tests in Docker, with cgo off
|
||||
make test-browser # Run the browser test in Docker (Dockerfile.browser)
|
||||
make check # test, lint, fmt-check and css-check: what CI runs
|
||||
make check # test, lint, fmt-check and css-check
|
||||
make build # Build bin/webhooker, version-stamped
|
||||
make version # Print the version this checkout would stamp
|
||||
make run # build, then run ./bin/webhooker
|
||||
@@ -1474,10 +1492,11 @@ make css-check # Fail if static/css/tailwind.css is out of date
|
||||
make clean # Remove bin/
|
||||
```
|
||||
|
||||
Each `make` target above, except `build`, `run`, `dev`, `deps`, `version` and
|
||||
`clean`, is a thin shim over a script in `script/`. `make hooks` calls
|
||||
`script/install-precommit`; every other shim calls the script of the same name.
|
||||
The scripts are listed under [Entrypoints](#entrypoints).
|
||||
Most `make` targets above are one-line wrappers: each runs one script in
|
||||
`script/`. The exceptions are `build`, `run`, `dev`, `deps`, `version` and
|
||||
`clean`, which have no script. `make hooks` runs `script/install-precommit`;
|
||||
every other wrapper runs the script of the same name. The scripts are listed
|
||||
under [Entrypoints](#entrypoints).
|
||||
|
||||
The tests run with cgo, Go's bridge to C code, turned off. Go's race detector,
|
||||
which finds unsynchronised access to shared memory, needs cgo, so the tests run
|
||||
@@ -1485,27 +1504,33 @@ without it.
|
||||
|
||||
### How the checks run
|
||||
|
||||
Each check is a stage of the `Dockerfile`, which is one named step of the image
|
||||
build. `script/test` builds the `test` stage, `script/lint` the `lint` stage and
|
||||
`script/css-check` the `css-check` stage. `script/fmt-check` runs `gofmt` on the
|
||||
host and builds the `markdown-check` stage. Every `docker build` in `script/`
|
||||
passes `--no-cache`, so Docker reruns every step instead of reusing an earlier
|
||||
result, and a check that reports success ran. A bare `docker build .` carries no
|
||||
such guarantee.
|
||||
Each check is a stage of the `Dockerfile`. A stage is one named step of the
|
||||
image build. `script/test` builds the `test` stage, `script/lint` the `lint`
|
||||
stage and `script/css-check` the `css-check` stage. `script/fmt-check` runs
|
||||
`gofmt` on the host and builds the `markdown-check` stage. Every `docker build`
|
||||
in `script/` passes `--no-cache`, so Docker reruns every step instead of reusing
|
||||
an earlier result. So a check that passes has really run. A bare
|
||||
`docker build .` carries no such guarantee.
|
||||
|
||||
The `lint` stage runs the `gofmt` check, `golangci-lint config verify` and
|
||||
`golangci-lint run` with `--network=none`, and ESLint over `static/js/`. The
|
||||
`test` stage runs `go test -cover` on four packages at a time, each running at
|
||||
most eight tests in parallel, with a 90 second timeout per package. On a failure
|
||||
it runs only the failed tests again with `-v`.
|
||||
The `lint` stage runs these steps:
|
||||
|
||||
1. the `gofmt` check;
|
||||
2. `golangci-lint config verify`, with `--network=none`;
|
||||
3. `golangci-lint run`, with `--network=none`;
|
||||
4. ESLint over `static/js/`, in the `js-lint` stage, with `--network=none`.
|
||||
|
||||
The `test` stage runs `go test -cover` on four packages at a time, each running
|
||||
at most eight tests in parallel, with a 90 second timeout per package. On a
|
||||
failure it runs only the failed tests again with `-v`.
|
||||
|
||||
The image build depends on the `lint`, `test`, `css-check` and `markdown-check`
|
||||
stages, so an image that built is a tree that passed every check. The runtime
|
||||
stage is Alpine with `ca-certificates` and `su-exec`, a tool that switches user,
|
||||
added. It has no `sqlite3`.
|
||||
stage is Alpine Linux. It adds `ca-certificates` and `su-exec`, a tool that
|
||||
switches user. It has no `sqlite3`.
|
||||
|
||||
The `check` workflow in `.gitea/workflows/check.yml` checks out the repository
|
||||
and runs `script/cibuild`, nothing else.
|
||||
CI, the check run on every push, is the `check` workflow in
|
||||
`.gitea/workflows/check.yml`. It checks out the repository and runs
|
||||
`script/cibuild`, nothing else.
|
||||
|
||||
### Version stamping
|
||||
|
||||
@@ -1527,30 +1552,29 @@ With no git metadata it is `unknown`.
|
||||
|
||||
`make build VERSION=v1.2.3` overrides it. `script/docker` and `script/cibuild`
|
||||
run the same `git describe` on the host and pass the result in as the `VERSION`
|
||||
build argument. Without that argument the image build reads the `.git` in the
|
||||
build context, and fails if `.git` is present but the version comes out empty,
|
||||
`dev` or `unknown`. `.dockerignore` leaves out every git `config` file, which
|
||||
can carry a credential, and keeps the rest of `.git`.
|
||||
build argument. Without that argument the image build reads the `.git` among the
|
||||
files sent to it, and fails if `.git` is present but the version comes out
|
||||
empty, `dev` or `unknown`. `.dockerignore` leaves out every git `config` file,
|
||||
which can carry a credential, and keeps the rest of `.git`.
|
||||
|
||||
`make build` passes `-trimpath` and `-s -w`, so the binary carries no build
|
||||
paths, symbol table or debug information. Nothing that differs between two
|
||||
builds of one commit is stamped in.
|
||||
paths, symbol table or debug information.
|
||||
|
||||
### Third-party browser assets
|
||||
|
||||
The pages use one third-party script: Alpine.js, a small JavaScript library.
|
||||
They use its CSP build, `@alpinejs/csp`, which runs under a
|
||||
Content-Security-Policy that forbids `eval`. The standard build needs `eval`.
|
||||
With the CSP build, every Alpine directive in `templates/` names a component
|
||||
With the CSP build, every Alpine.js directive in `templates/` names a component
|
||||
registered in `static/js/app.js`, such as `x-data="collapsible"`, and never an
|
||||
inline expression.
|
||||
|
||||
The npm package file is committed as `3p/alpinejs-csp-3.14.9.tgz`, byte for byte
|
||||
as the registry publishes it. `script/assets` extracts `package/dist/cdn.min.js`
|
||||
from it to `static/js/alpine.min.js`, which the binary embeds. `make build`,
|
||||
`make dev` and the Docker stages run it first. The extracted file is not
|
||||
committed, and a build that skips the extraction fails with an error naming
|
||||
`js/alpine.min.js`.
|
||||
The package file is committed as `3p/alpinejs-csp-3.14.9.tgz`, byte for byte as
|
||||
the npm registry, the JavaScript package registry, publishes it. `script/assets`
|
||||
extracts `package/dist/cdn.min.js` from it to `static/js/alpine.min.js`, which
|
||||
the binary embeds. `make build`, `make dev` and the Docker stages run it first.
|
||||
The extracted file is not committed, and a build that skips the extraction fails
|
||||
with an error naming `js/alpine.min.js`.
|
||||
|
||||
To move to a new version:
|
||||
|
||||
@@ -1601,7 +1625,7 @@ application are under [Entrypoint URLs](#entrypoint-urls).
|
||||
ESLint.
|
||||
- `script/fmt` formats the Go code with `gofmt` and `goimports` on the host, and
|
||||
the Markdown with prettier in Docker. It writes.
|
||||
- `script/fmt-check` runs the same checks and fails instead of writing.
|
||||
- `script/fmt-check` checks `gofmt` and prettier and fails instead of writing.
|
||||
- `script/css` regenerates `static/css/tailwind.css` in Docker. It writes.
|
||||
- `script/css-check` fails when `static/css/tailwind.css` differs from what
|
||||
`script/css` would generate.
|
||||
|
||||
Reference in New Issue
Block a user