sneak/webhooker
1
0
Fork 0
Code Issues 62 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Compare commits

base: sneak:next
Branches Tags
sneak:main sneak:clawbot/docs-uuid-is-the-secret sneak:next sneak:feat/receiver-rate-limit sneak:issue-70-log-target
..
compare: sneak:027f0898e703ead02ccc6b47935982f5afe07856
Branches Tags
sneak:clawbot/docs-uuid-is-the-secret sneak:next sneak:main sneak:feat/receiver-rate-limit sneak:issue-70-log-target

1 Commits
next ... 027f0898e7

Author SHA1 Message Date
clawbot
027f0898e7 Make SQLite durable under concurrent readers and stop re-delivering stranded webhooks (closes #256)
All checks were successful
check / check (push) Successful in 3m38s
Details 
An operator running `sqlite3 <db> .dump` against their own per-webhook
database wedged it: 60 of 60 inbound webhooks rejected with HTTP 500,
206 delivered webhooks stranded at `pending`, and every one of them
POSTed a second time on the next restart while the event log recorded a
single attempt.

Durability. Every SQLite file — main, per-webhook, and archive — now
opens through one path, `internal/database/sqlite_open.go`, in WAL
journal mode with a 10-second busy timeout, `BEGIN IMMEDIATE`
transactions, and a bounded connection pool. WAL is what stops a
reader blocking writers at all. `_txlock=immediate` is what stops a
`COMMIT` failing while its transaction stays open on a pooled
connection, which is how four `database is locked` errors became 593
`cannot start a transaction within a transaction`: a deferred
transaction that upgrades to a write lock mid-flight gets SQLITE_BUSY
without the busy handler being consulted. `cache=shared` is gone,
because under it an in-process conflict is SQLITE_LOCKED, which the
busy handler does not retry.

Delivery. `recordResult` and `updateDeliveryStatus` return their
errors instead of logging and dropping them, and a caller whose
bookkeeping write failed writes nothing at all — the delivery keeps
whichever non-terminal status it already held, and both sweeps recover
it. Recovery and the sweep now reconcile before re-sending: a pending
delivery that already holds a successful `DeliveryResult` is marked
delivered rather than sent again, which is the state that did not
previously exist. A delivery handed back out is claimed by
compare-and-set so successive sweeps cannot send it repeatedly, and it
continues its own attempt numbering instead of restarting at 1. The
sweep gains a `pending`-with-age-bound arm, so a stranded delivery no
longer waits for a restart.

Docs. WAL produces `-wal`/`-shm` sidecars, so the backup and restore
procedures in README.md are corrected: both documented procedures were
re-run against a live instance, and a `-wal` left by a crash carries
data the `.db` alone does not.

Verified by reproducing the failure on unmodified `next` first — 6
targets, 60 events at 5/s, a concurrent `.dump` reader — which gave 38
HTTP 500s and 112 duplicate POSTs at the sinks across a restart. Both
arms of the matched pair now show 0 inbound 500s, 0 engine write
errors, and 0 new requests at the sinks after a restart, counted by
payload.
 2026-08-23 23:40:03 +00:00
73 changed files with 2752 additions and 7456 deletions
Expand all files Collapse all files

30
Dockerfile
View File

@@ -61,28 +61,14 @@ RUN script/fetch-assets
# Run tests and build # Run tests and build
RUN make test RUN make test
RUN make build
# Version stamped into the binary. .dockerignore excludes .git/, so
# nothing in this stage can derive it: script/docker resolves it on the
# host and passes it in. The default is what a bare `docker build .`
# with no --build-arg gets, and it names no tag the tree may not be at.
#
# Declared here, below the test and asset steps, so a changed version
# does not invalidate their cached layers.
ARG VERSION=unknown
RUN make build VERSION="$VERSION"
# Rebuild with static linking for Alpine runtime. # Rebuild with static linking for Alpine runtime.
# make build already verified compilation. # make build already verified compilation.
# The CGO binary from `make build` is dynamically linked against glibc, # The CGO binary from `make build` is dynamically linked against glibc,
# which doesn't exist on Alpine (musl). Rebuild with static linking so # which doesn't exist on Alpine (musl). Rebuild with static linking so
# the binary runs on Alpine without glibc. # the binary runs on Alpine without glibc.
# RUN CGO_ENABLED=1 go build -ldflags '-extldflags "-static"' -o bin/webhooker ./cmd/webhooker
# The static flags go in through GO_LDFLAGS rather than a -ldflags of
# their own: the build target composes them with the -X that stamps the
# version, so this relink cannot silently drop the stamp.
RUN CGO_ENABLED=1 make build VERSION="$VERSION" GO_LDFLAGS='-extldflags "-static"'
# Runtime stage # Runtime stage
# alpine:3.21, 2026-03-17 # alpine:3.21, 2026-03-17
@@ -109,18 +95,6 @@ USER webhooker
EXPOSE 8080 EXPOSE 8080
# The binary defaults BIND_ADDRESS to 127.0.0.1, which is right for a
# bare host: the cleartext listener serves the admin UI and the
# unauthenticated receiver, so it must not appear on every interface
# of a machine that configured nothing. A container is the other case.
# Its network namespace is already the isolation boundary, so binding
# every address inside it exposes nothing; what decides exposure is
# the publish flag, and `-p 127.0.0.1:8080:8080` is the operator's
# control there. Shipping the image on loopback would buy no security
# and would make the process unreachable through its own published
# port.
ENV BIND_ADDRESS=0.0.0.0
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \ HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:8080/.well-known/healthcheck || exit 1 CMD wget --no-verbose --tries=1 --spider http://localhost:8080/.well-known/healthcheck || exit 1

25
Makefile
View File

@@ -1,26 +1,8 @@
.PHONY: bootstrap setup assets test lint fmt fmt-check check build run dev deps docker clean hooks css version .PHONY: bootstrap setup assets test lint fmt fmt-check check build run dev deps docker clean hooks css
# Default target # Default target
.DEFAULT_GOAL := check .DEFAULT_GOAL := check
# Version stamped into the binary. Derived from git by script/version;
# override it (`make build VERSION=v1.2.3`) where git metadata is
# unavailable, which is how the Dockerfile passes its build arg in.
VERSION ?= $(shell script/version)
# An empty override (`make build VERSION=`, or a `--build-arg VERSION=`
# landing on the Dockerfile's `make build VERSION="$VERSION"`) means unset,
# exactly as it does in script/version -- stamping "" would leave the binary
# reporting no version and the footer back on its "dev" fallback. `override`
# is required: a plain assignment loses to the command-line definition it
# exists to correct.
override VERSION := $(or $(strip $(VERSION)),$(shell script/version))
# Extra linker flags for the build target. The static relink in the
# Dockerfile adds -extldflags here rather than passing its own -ldflags,
# so composing flags cannot drop the version stamp.
GO_LDFLAGS ?=
bootstrap: bootstrap:
@script/bootstrap @script/bootstrap
@@ -46,7 +28,7 @@ check:
@script/check @script/check
build: build:
go build -ldflags '$(strip -X main.version=$(VERSION) $(GO_LDFLAGS))' -o bin/webhooker ./cmd/webhooker go build -o bin/webhooker ./cmd/webhooker
run: build run: build
./bin/webhooker ./bin/webhooker
@@ -58,9 +40,6 @@ deps:
go mod download go mod download
go mod tidy go mod tidy
version:
@echo $(VERSION)
docker: docker:
@script/docker @script/docker

741
README.md
View File

@@ -57,8 +57,7 @@ make fmt-check # Fail if gofmt would change anything (writes nothing)
make lint # Run golangci-lint in Docker (Dockerfile.lint) make lint # Run golangci-lint in Docker (Dockerfile.lint)
make test # Run tests with race detection make test # Run tests with race detection
make check # test + lint + fmt-check (CI gate) make check # test + lint + fmt-check (CI gate)
make build # Build binary to bin/webhooker (version-stamped) make build # Build binary to bin/webhooker
make version # Print the version this checkout would stamp
make run # build, then run ./bin/webhooker make run # build, then run ./bin/webhooker
make dev # go run ./cmd/webhooker make dev # go run ./cmd/webhooker
make deps # go mod download + go mod tidy make deps # go mod download + go mod tidy
@@ -71,55 +70,31 @@ make clean # Remove bin/
### Configuration ### Configuration
All configuration is via environment variables. For local development, All configuration is via environment variables. For local development,
you can place variables in a `.env` file in the process working you can place variables in a `.env` file in the project root (loaded
directory, read once at startup before anything else looks at the automatically via `godotenv/autoload`).
environment.
The file is optional and having none is the normal case for a
deployment. A file that is there but cannot be parsed aborts startup
with a message naming it, because a single malformed line makes none
of the file apply: every variable in it silently reverts to its
default, which is exactly the failure [Invalid values abort
startup](#invalid-values-abort-startup) exists to prevent, for all of
them at once. A variable already present in the real environment wins
over the file's value for the same name.
The environment is selected by setting `WEBHOOKER_ENVIRONMENT` to `dev` The environment is selected by setting `WEBHOOKER_ENVIRONMENT` to `dev`
or `prod` (default: `dev`). The setting controls exactly one behavior: or `prod` (default: `dev`). The setting controls several behaviors:
| Behavior | `dev` | `prod` | | Behavior | `dev` | `prod` |
| -------- | ----------------------- | ---------------- | | --------------------- | -------------------------------- | ------------------------------- |
| CORS | Allows any origin (`*`) | Disabled (no-op) | | CORS | Allows any origin (`*`) | Disabled (no-op) |
| Session cookie Secure | `false` (works over plain HTTP) | `true` (requires HTTPS) |
The environment setting does **not** control cookie security. Both the The CSRF cookie's `Secure` flag and Origin/Referer validation mode are
session cookie and the CSRF cookie get their `Secure` flag, and the determined per-request based on the actual transport protocol, not the
CSRF middleware its Origin/Referer validation mode, from the transport environment setting. The middleware checks `r.TLS` (direct TLS) and the
of each individual request, decided by one predicate — `X-Forwarded-Proto` header (TLS-terminating reverse proxy) to decide:
`internal/reqtls.IsTLS`. It reports TLS for a direct TLS connection
(`r.TLS`) or for a TLS-terminating reverse proxy that reports one in
`X-Forwarded-Proto`:
- **Direct TLS or `X-Forwarded-Proto: https`**: Secure cookies, strict - **Direct TLS or `X-Forwarded-Proto: https`**: Secure cookies, strict
Origin/Referer validation. Origin/Referer validation.
- **Plaintext HTTP**: Non-Secure cookies, relaxed Origin/Referer - **Plaintext HTTP**: Non-Secure cookies, relaxed Origin/Referer
checks (token validation still enforced). checks (token validation still enforced).
The `X-Forwarded-Proto` value is matched case-insensitively on its This means CSRF protection works correctly in all deployment scenarios:
first comma-separated element, trimmed, so `HTTPS` and the appended behind a TLS-terminating reverse proxy, with direct TLS, or over plain
chains a proxy behind another proxy emits (`https, http`) are all read HTTP during development. When running behind a reverse proxy, ensure it
as TLS. sets the `X-Forwarded-Proto: https` header.
This means both cookie security and CSRF protection work correctly in
all deployment scenarios: behind a TLS-terminating reverse proxy, with
direct TLS, or over plain HTTP during development — a plain-HTTP local
run gets non-`Secure` cookies and remains usable, and a proxied
deployment gets `Secure` ones without the operator setting anything.
When running behind a reverse proxy, ensure it sets the
`X-Forwarded-Proto: https` header. Unlike `X-Forwarded-For`, this
header is read from any peer and is **not** gated by
`TRUSTED_PROXIES`; a correctly configured proxy overwrites whatever a
client sent. On a listener exposed directly to clients, any client can
assert it, so do not run one without a proxy in front.
All other differences (log format, security headers, etc.) are All other differences (log format, security headers, etc.) are
independent of the environment setting — log format is determined by independent of the environment setting — log format is determined by
@@ -129,13 +104,12 @@ TTY detection, and security headers are always applied.
| ----------------------- | ----------------------------------- | -------- | | ----------------------- | ----------------------------------- | -------- |
| `WEBHOOKER_ENVIRONMENT` | `dev` or `prod` | `dev` | | `WEBHOOKER_ENVIRONMENT` | `dev` or `prod` | `dev` |
| `PORT` | HTTP listen port | `8080` | | `PORT` | HTTP listen port | `8080` |
| `BIND_ADDRESS` | IP address the HTTP listener binds. Loopback by default, so the cleartext listener is not published on every interface. The Docker image ships `0.0.0.0` instead. See [Bind address](#bind-address) | `127.0.0.1` (image: `0.0.0.0`) |
| `DATA_DIR` | Directory for all SQLite databases | `/var/lib/webhooker` | | `DATA_DIR` | Directory for all SQLite databases | `/var/lib/webhooker` |
| `DEBUG` | Enable debug logging | `false` | | `DEBUG` | Enable debug logging | `false` |
| `MAINTENANCE_MODE` | Report `maintenanceMode: true` in the healthcheck JSON. It does not change how any request is served — no maintenance page exists | `false` | | `MAINTENANCE_MODE` | Report `maintenanceMode: true` in the healthcheck JSON. It does not change how any request is served — no maintenance page exists | `false` |
| `METRICS_USERNAME` | Basic auth username for `/metrics`. Must be set together with `METRICS_PASSWORD`; one without the other fails startup | `""` | | `METRICS_USERNAME` | Basic auth username for `/metrics`. Must be set together with `METRICS_PASSWORD`; one without the other fails startup | `""` |
| `METRICS_PASSWORD` | Basic auth password for `/metrics`. Must be set together with `METRICS_USERNAME`; one without the other fails startup | `""` | | `METRICS_PASSWORD` | Basic auth password for `/metrics`. Must be set together with `METRICS_USERNAME`; one without the other fails startup | `""` |
| `SENTRY_DSN` | Sentry error reporting DSN. Unset leaves error reporting off; a value the Sentry SDK cannot parse fails startup rather than serving with reporting silently off | `""` | | `SENTRY_DSN` | Sentry error reporting DSN | `""` |
| `RETENTION_SWEEP_INTERVAL` | How often the retention reaper and archive sweeper run (Go duration, must be positive) | `1h` | | `RETENTION_SWEEP_INTERVAL` | How often the retention reaper and archive sweeper run (Go duration, must be positive) | `1h` |
| `SESSION_IDLE_TIMEOUT` | Idle session timeout (Go duration) | `24h` | | `SESSION_IDLE_TIMEOUT` | Idle session timeout (Go duration) | `24h` |
| `RECEIVER_RATE_LIMIT` | Receiver requests/minute per IP per entrypoint (10x that per IP across the route) | `120` | | `RECEIVER_RATE_LIMIT` | Receiver requests/minute per IP per entrypoint (10x that per IP across the route) | `120` |
@@ -251,68 +225,6 @@ A set but unparseable value aborts startup. When the list is non-empty
webhooker logs it at startup, blocks and all, so the hole is visible in webhooker logs it at startup, blocks and all, so the hole is visible in
the log of any deployment that has one. the log of any deployment that has one.
#### Bind address
`BIND_ADDRESS` is the IP address the HTTP listener binds. The binary
defaults to `127.0.0.1`, so a bare webhooker is reachable only from the
host it runs on. The Docker image ships `ENV BIND_ADDRESS=0.0.0.0`
instead — see below for why the two differ.
That listener speaks **cleartext**, and it serves both the admin UI and
the unauthenticated webhook receiver. webhooker terminates no TLS
itself; a production deployment puts a reverse proxy in front of it
(see
[Deployment behind a reverse proxy](#deployment-behind-a-reverse-proxy)),
and the proxy reaches it over loopback. A default that bound every
interface would leave that cleartext port answering the internet
alongside the proxy — the admin login form and the receiver, in the
clear, on a port nobody chose to publish. Reaching webhooker from
another host is therefore something you configure, not something you
get by default.
**In a container the answer is `0.0.0.0`, which is why the image ships
that.** A container's network namespace is already the boundary the
loopback default is reaching for: nothing outside the container gets to
`0.0.0.0:8080` because of the namespace, whatever the process bound.
Exposure is decided at the publish flag instead — `-p
127.0.0.1:8080:8080` rather than `-p 8080:8080` — which is the
operator's to choose and is what
[Running with Docker](#running-with-docker) shows. A loopback bind
inside a container buys nothing and makes the process unreachable
through its own published port.
The value must be an IP address literal:
- `127.0.0.1` — loopback only (the binary's default). Use this with a
reverse proxy on the same host.
- `0.0.0.0` — every IPv4 address. The image's default; on a bare host,
only behind a firewall on the port.
- `::` — every address, IPv6 and (on Linux, with the default
`net.ipv6.bindv6only=0`) IPv4 as well.
- A specific address such as `10.0.0.5` — that interface only.
An **empty** value is treated as unset, as everywhere else here, and
takes the default. In a container that matters: `BIND_ADDRESS=` throws
away the image's `0.0.0.0` and falls back to the binary's
`127.0.0.1`, which is the one quiet failure this setting has — see
[Running with Docker](#running-with-docker).
Hostnames are **not** accepted. `localhost` aborts startup rather than
being resolved: which of `127.0.0.1` and `::1` it means differs by
host, a name can resolve to several addresses of which only one could
be bound, and the answer can change under a running process. A value
carrying a port (`127.0.0.1:8080`) is likewise rejected — the port is
`PORT`'s business. Any unparseable value aborts startup; see
[Invalid values abort startup](#invalid-values-abort-startup).
An address that parses but is not assigned to this host — say
`10.0.0.5` on a machine that has no such interface — is a valid
literal, so it reaches the listener and fails there. The process logs
the bind error and exits non-zero rather than staying up with nothing
listening. The effective value is in the `bindAddress` field of the
startup log line, which is the way to check what a running deployment
actually bound.
#### Metrics credentials #### Metrics credentials
`METRICS_USERNAME` and `METRICS_PASSWORD` are set together or not at `METRICS_USERNAME` and `METRICS_PASSWORD` are set together or not at
@@ -470,24 +382,11 @@ additionally be a number in the range 1–65535,
`RECEIVER_RATE_LIMIT` must be at least 1, `RECEIVER_RATE_LIMIT` must be at least 1,
`RETENTION_SWEEP_INTERVAL` must be greater than zero (it is a ticker `RETENTION_SWEEP_INTERVAL` must be greater than zero (it is a ticker
period, so `0s` or a negative value would crash the reaper after period, so `0s` or a negative value would crash the reaper after
startup), every entry in `TRUSTED_PROXIES` and startup), and every entry in `TRUSTED_PROXIES` and
`ALLOWED_EGRESS_CIDRS` must be a CIDR block or a bare IP address, and `ALLOWED_EGRESS_CIDRS` must be a CIDR block or a bare IP address.
`BIND_ADDRESS` must be an IP address literal — `localhost`,
`127.0.0.1:8080` and `10.0.0.0/8` are each rejected rather than
resolved, split, or narrowed to something they do not say — and
`SENTRY_DSN` must parse as a Sentry DSN.
`SESSION_IDLE_TIMEOUT` is the exception: a `SESSION_IDLE_TIMEOUT` is the exception: a
non-positive value there means idle expiry is disabled, not invalid. non-positive value there means idle expiry is disabled, not invalid.
`SENTRY_DSN` is checked with the Sentry SDK's own parser, the same call
the SDK makes on the DSN it is later handed, so what configuration
accepts is exactly what will initialise. A typo in it is the one
configuration mistake nothing downstream can ever notice — the variable
is still set, so every later signal reports error reporting as on while
no report is being sent — which is why it aborts rather than starting
with reporting off. Leaving it unset is not a mistake and not affected:
error reporting is simply off and startup is normal.
Boolean variables (`DEBUG`, `MAINTENANCE_MODE`) accept exactly the Boolean variables (`DEBUG`, `MAINTENANCE_MODE`) accept exactly the
spellings Go's `strconv.ParseBool` accepts — `1`, `t`, `T`, `TRUE`, spellings Go's `strconv.ParseBool` accepts — `1`, `t`, `T`, `TRUE`,
`true`, `True`, `0`, `f`, `F`, `FALSE`, `false`, `False` — and nothing `true`, `True`, `0`, `f`, `F`, `FALSE`, `false`, `False` — and nothing
@@ -629,54 +528,12 @@ decision:
```bash ```bash
docker run -d \ docker run -d \
-p 127.0.0.1:8080:8080 \ -p 8080:8080 \
-v /path/to/data:/var/lib/webhooker \ -v /path/to/data:/var/lib/webhooker \
-e WEBHOOKER_ENVIRONMENT=prod \ -e WEBHOOKER_ENVIRONMENT=prod \
-e BIND_ADDRESS=0.0.0.0 \
webhooker:latest webhooker:latest
``` ```
**The image and the bare binary default `BIND_ADDRESS` differently, on
purpose.** The binary defaults to `127.0.0.1`; the image ships
`ENV BIND_ADDRESS=0.0.0.0`, so the `-e BIND_ADDRESS=0.0.0.0` above is
belt-and-braces and the command works without it.
The two cases are not the same question. On a bare host, `0.0.0.0`
puts the cleartext admin UI and the unauthenticated receiver on every
interface of the machine, which is what the loopback default exists to
prevent. In a container, the network namespace is already that
boundary: nothing outside reaches `0.0.0.0:8080` because of the
namespace, not because of the bind. What decides exposure there is the
**publish flag**, and that is the line to get right.
So publish to `127.0.0.1:8080` rather than `8080`. A bare
`-p 8080:8080` opens the port on every interface of the host — through
firewall rules too, since Docker's forwarding rules are inserted ahead
of most host firewalls. Publish to the host address your reverse proxy
connects from, and nothing wider.
**An empty `BIND_ADDRESS` is treated as unset**, like every other
variable here, so `-e BIND_ADDRESS=` does not mean "keep the image
default" — it discards the image's `0.0.0.0` and falls back to the
_binary's_ `127.0.0.1`. In a container that is the failure below, and
nothing in the logs names the variable. A templated Compose file or a
`.env` line with an empty value is the usual way in. Either set a
literal or leave the variable out entirely.
Overriding `BIND_ADDRESS` to a loopback address in a container — by
that route or deliberately — makes the container unreachable from
outside its namespace even with `-p`. The published port answers
nothing, and the health check fails too: it requests
`http://localhost:8080`, `localhost` resolves to `::1` first, and a
`127.0.0.1` bind is not listening there. The container then goes
`unhealthy` about **65 seconds** after start — from `HEALTHCHECK
--start-period=5s --interval=30s --retries=3`, so failing probes at
5s, 35s and 65s, and `unhealthy` on the third. (Docker's probe cadence
during the start period has changed between versions; re-derive from
those three values rather than trusting the figure. Measured at 65s on
Docker 29.7.2.) A container `unhealthy` with `connection refused` in
its health log, or a published port that resets connections, is this.
The container runs as a non-root user (`webhooker`, UID 1000), exposes The container runs as a non-root user (`webhooker`, UID 1000), exposes
port 8080, and includes a health check against port 8080, and includes a health check against
`/.well-known/healthcheck`. The `/var/lib/webhooker` volume holds all `/.well-known/healthcheck`. The `/var/lib/webhooker` volume holds all
@@ -686,189 +543,6 @@ databases written by `database` targets (`archive-{uuid}.db`). Mount
this as a persistent volume to preserve data across container this as a persistent volume to preserve data across container
restarts. restarts.
**The bind-mounted directory must be owned by UID 1000, or the
container does not start.** Docker creates a `-v` source path that
does not exist yet as `root:root`, and the process runs as UID 1000,
so it cannot take its `DATA_DIR` lock:
```
webhooker: locking data directory /var/lib/webhooker: open
/var/lib/webhooker/webhooker.lock: permission denied
```
It exits non-zero at that point, before opening any database. Create
the directory ahead of the first `docker run`:
```bash
mkdir -p /path/to/data
chown 1000:1000 /path/to/data
chmod 750 /path/to/data
```
The same `chown` is what a restore needs — see step 4 of
[Restore](#restore). A **named volume** does not have this problem:
Docker copies the image's ownership onto a volume it initializes, and
the image creates `/var/lib/webhooker` owned by `webhooker`.
**The file modes are not yours to set, and do not depend on the
directory.** `webhooker.db` holds target configuration in plaintext —
bearer tokens, API keys, Slack webhook URLs — along with the session
encryption key, so webhooker creates every SQLite file it owns `0600`:
each database and both of its `-wal` and `-shm` sidecars, across all
three tiers. Files an earlier build left `0644` are tightened when
they are opened. A `DATA_DIR` webhooker creates itself is `0750`, but
a bind mount supplies its own directory and Docker's default for one
it creates is `0755`; the `0600` files hold there regardless. The
`chmod 750` above is defence in depth — it stops other local users
listing the directory and learning your webhook UUIDs from the
`events-{uuid}.db` filenames — not the barrier protecting the
credentials.
## Deployment behind a reverse proxy
webhooker terminates no TLS of its own. It serves plaintext HTTP and
expects a reverse proxy in front of it, which is the deployment it is
built for: the proxy holds the certificate, and webhooker binds
loopback where only the proxy can reach it.
Five things have to be right. Each one is silent when it is wrong —
the service comes up, serves pages, and is broken in a way nothing
reports.
1. **Bind or firewall the app port.** The binary binds `127.0.0.1` by
default, so the cleartext listener is not published beside the
proxy. The image binds `0.0.0.0` inside its own network namespace
and relies on the publish address instead —
`-p 127.0.0.1:8080:8080`. Either way the port must reach the proxy
and nothing else; widen it only with a firewall or a publish
address in front of it. A cleartext port answering the internet
serves the admin login form and the unauthenticated receiver with
no TLS at all, and the proxy in front of it changes nothing about
that.
2. **Set `WEBHOOKER_ENVIRONMENT=prod`, and make sure the proxy sends
`X-Forwarded-Proto`.** These are two requirements, not one. The
environment setting decides CORS and nothing else: the default
`dev` answers every origin with `Access-Control-Allow-Origin: *`
(without credentials), which a server-rendered production
deployment has no use for. Cookie `Secure` and the strict
Origin/Referer mode are **not** tied to it — they are decided per
request from the transport, which behind a proxy means the
`X-Forwarded-Proto` header. The block below sets it; without it
every request is read as plaintext and cookies ship without
`Secure`. See [Configuration](#configuration).
3. **Set `TRUSTED_PROXIES` to the proxy's address.** Unset, every rate
limiter keys on the connecting peer, which behind a proxy is the
proxy on every request: all clients collapse into one global bucket
per limit and the receiver's per-IP limits become service-wide
ceilings. See [Trusted proxies](#trusted-proxies). List the proxy
and nothing else.
4. **Send `Host` as `$http_host`, not `$host`.** `$host` strips the
port. webhooker's Origin/Referer check compares against the host it
was given, so on any port other than 443 `$host` makes every form
POST — including login — fail with `403 origin invalid`, with
nothing in the error naming the cause.
5. **Keep the proxy's access log.** webhooker's own access log records
the peer address, which behind a proxy is always the proxy. The
proxy's log is the only record of which client sent what. nginx's
default `combined` format already logs `$remote_addr`; do not
replace it with one that drops the client address, and retain those
logs as long as you would want to answer a question about traffic.
### nginx
Complete server block. Replace the `server_name` and the two
certificate paths.
```nginx
server {
listen 443 ssl;
listen [::]:443 ssl;
http2 on; # nginx 1.25.1+; older: listen 443 ssl http2;
server_name webhooker.example.com;
ssl_certificate /etc/ssl/certs/webhooker.example.com.crt;
ssl_certificate_key /etc/ssl/private/webhooker.example.com.key;
ssl_protocols TLSv1.2 TLSv1.3;
# webhooker caps form POST bodies at 1 MB. nginx's default happens
# to match, so leaving this out breaks nothing today — but if you
# ever raise webhooker's cap, this is the limit you will still be
# hitting, and the rejection is nginx's HTML page rather than
# webhooker's message.
client_max_body_size 1m;
# $remote_addr is the client. webhooker's own log records this
# proxy and nothing else, so this file is the only place the
# client's address is written down.
access_log /var/log/nginx/webhooker.access.log combined;
location / {
# A literal address, not localhost: with BIND_ADDRESS at its
# 127.0.0.1 default, a localhost that resolves to ::1 first
# gets connection refused.
proxy_pass http://127.0.0.1:8080;
# $http_host, NOT $host. $host drops the port and every form
# POST fails with 403 origin invalid on any port but 443.
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Above webhooker's own 60s request timeout, so its 503
# reaches the client instead of nginx cutting the connection
# first and answering 504.
proxy_read_timeout 70s;
}
}
server {
listen 80;
listen [::]:80;
server_name webhooker.example.com;
return 308 https://$host$request_uri;
}
```
`X-Forwarded-For` must be **appended** to, which
`$proxy_add_x_forwarded_for` does. webhooker reads no other forwarded
client header: `X-Real-IP` and `True-Client-IP` are ignored from every
peer, so setting them has no effect. See
[Trusted proxies](#trusted-proxies) for how the chain is walked.
`X-Forwarded-Proto: https` is what tells webhooker the request arrived
over TLS, which decides the `Secure` flag on both the session and CSRF
cookies and the strict Origin/Referer mode. Without it, requests are
treated as plaintext and the cookies ship without `Secure`. Unlike
`X-Forwarded-For`, this header is read from any peer and is not gated
by `TRUSTED_PROXIES`, so the proxy must overwrite whatever a client
sent — `$scheme` above does.
With that block, webhooker's environment is:
```sh
WEBHOOKER_ENVIRONMENT=prod
BIND_ADDRESS=127.0.0.1 # the default; stated here to be explicit
TRUSTED_PROXIES=127.0.0.1
```
If nginx runs on another host, `BIND_ADDRESS` becomes the address it
connects to, `TRUSTED_PROXIES` becomes nginx's address, and the port
must be firewalled to that address — the traffic between them is
cleartext.
### HSTS is always sent, and is not configurable
Every response carries
`Strict-Transport-Security: max-age=63072000; includeSubDomains; preload`.
Two years, every subdomain, and a `preload` token. There is no setting
that changes or suppresses it.
This is worth knowing before the first request reaches a browser: a
client that sees it once will refuse plaintext HTTP to that hostname —
and to every subdomain of it — for two years, whatever else is served
there. Terminate TLS on a hostname you are prepared to keep on HTTPS.
## Backup, Restore, and Upgrades ## Backup, Restore, and Upgrades
### What to back up ### What to back up
@@ -899,18 +573,11 @@ so while the service is running each `{name}.db` has a `{name}.db-wal`
and a `{name}.db-shm` beside it. **`-wal` is part of the database, not a and a `{name}.db-shm` beside it. **`-wal` is part of the database, not a
scratch file**: it holds committed transactions that are not yet in the scratch file**: it holds committed transactions that are not yet in the
`.db`, so a copy of the `.db` without its `-wal` is missing data and may `.db`, so a copy of the `.db` without its `-wal` is missing data and may
have no readable schema at all. `-shm` is regenerable, but there is no have no readable schema at all. A clean shutdown checkpoints and removes
reason to separate the two — copy the directory and you have them. both sidecars, so a stopped deployment has none; a killed or crashed one
leaves them, and they must be carried with the `.db`. `-shm` is
A clean shutdown closes `webhooker.db` and every `events-*.db`, which regenerable, but there is no reason to separate the two — copy the
checkpoints and removes their sidecars; a killed or crashed instance directory.
leaves them, and they must be carried with the `.db`. **Archive
databases are different**: their handle is not closed at shutdown, so
`archive-*.db-wal` and `-shm` normally survive a clean stop and the
`-wal` can hold every row the archive has. Measured on a stopped
instance: `archive-….db` 4096 bytes with no table, its `-wal` 157 KB
holding all 8 archived events. Copying `DATA_DIR` in full is what makes
this a non-issue; copying `.db` files out of it by name is not.
Configuration is **not** in `DATA_DIR` — it comes from the environment Configuration is **not** in `DATA_DIR` — it comes from the environment
and from a `.env` file read out of the process working directory. Back and from a `.env` file read out of the process working directory. Back
@@ -966,30 +633,17 @@ run — it does not block ingestion — but back up with `.backup` or a
stopped copy. stopped copy.
Archive databases are the one exception the service is built for: the Archive databases are the one exception the service is built for: the
archive writer closes and reopens its handle around writes (debounced archive writer closes its handle after each write (debounced to at most
to at most one reopen per second), so an operator can move one reopen per second), so an operator can move `archive-{uuid}.db`
`archive-{uuid}.db` away for offline retention while the service runs, away for offline retention while the service runs, and it is recreated
and it is recreated on the next write. See on the next write. Closing the handle checkpoints and removes that
file's sidecars, so what is left to move is a single self-contained
`.db` — but if a `-wal` is there, a write is in flight, and it has to
move with it. See
[Database Architecture](#database-architecture). That is a [Database Architecture](#database-architecture). That is a
move-the-file-away workflow, not a substitute for the backup procedures move-the-file-away workflow, not a substitute for the backup procedures
above. above.
**Move the sidecars with it.** Under WAL that workflow is no longer a
single file, and the common case is the dangerous one. The reopen
happens on the *next* write after the debounce window elapses, so after
the last write of a burst nothing checkpoints: measured, 20 s after ten
events the `archive-….db` was 4096 bytes — a header, no table — with
all ten rows sitting in a 189 KB `-wal`. Copying the `.db` alone at that
moment yields a file that opens with `no such table: archived_events`.
The file becomes self-contained again when the handle closes, which
happens on the next write past the debounce window, when the connection
pool retires the idle connection (about a minute after the last write),
or at the idle archive sweep — measured, the same file was a complete
20 KB `.db` with no sidecars about a minute after its last write.
Shutdown is **not** on that list: the archive handle is not closed when
the service stops. So either move `archive-{uuid}.db` together with any
`-wal`/`-shm` beside it, or wait until there are none.
### Restore ### Restore
1. Stop the service. 1. Stop the service.
@@ -1005,14 +659,10 @@ the service stops. So either move `archive-{uuid}.db` together with any
3. Carry any `*.db-wal` and `*.db-shm` files that are in the backup. 3. Carry any `*.db-wal` and `*.db-shm` files that are in the backup.
They are part of the database, and dropping a `-wal` silently They are part of the database, and dropping a `-wal` silently
discards every transaction it still holds. An `.backup` set will not discards every transaction it still holds. Backups taken by either
contain any: it writes a single consolidated file per database. A procedure above will not contain them — `.backup` writes a single
stop-and-copy set has none for `webhooker.db` or the `events-*.db`, consolidated file, and a clean stop checkpoints the sidecars away —
because a clean stop closes those and checkpoints their sidecars but a copy salvaged from a crashed instance will, and it needs them.
away — but it will normally have them for `archive-*.db`, whose
handle stays open across shutdown, and those carry the archive's
rows. A copy salvaged from a crashed instance has them for
everything, and needs all of them.
4. **Fix ownership.** The container runs as the non-root `webhooker` 4. **Fix ownership.** The container runs as the non-root `webhooker`
user, UID 1000 / GID 1000. Restored files must be owned by (or user, UID 1000 / GID 1000. Restored files must be owned by (or
@@ -1052,21 +702,6 @@ Upgrade procedure:
3. Pull the new image and start it. 3. Pull the new image and start it.
4. Confirm `database migrations completed` in the logs before putting 4. Confirm `database migrations completed` in the logs before putting
traffic back on it. traffic back on it.
5. Confirm the new build is the one running:
`curl -s http://host:8080/.well-known/healthcheck` reports the
version it was stamped with (see [Version stamping](#version-stamping)).
**Upgrading past the introduction of `BIND_ADDRESS`:** earlier versions
always bound every interface. **Container deployments are unaffected**
— the image ships `ENV BIND_ADDRESS=0.0.0.0`, so a `docker run` or
Compose service that worked before still works with nothing changed.
A **bare binary** is the case that changes: the listener now binds
`127.0.0.1` unless `BIND_ADDRESS` says otherwise, so a deployment that
relied on reaching it from another host becomes unreachable until it
sets the address the proxy connects to. Check the `bindAddress` field
of the startup log to see what a running process bound. See
[Bind address](#bind-address).
**Downgrade is unsupported.** Once a newer binary has migrated the files **Downgrade is unsupported.** Once a newer binary has migrated the files
there is no way to move them back. `AutoMigrate` is additive — it adds there is no way to move them back. `AutoMigrate` is additive — it adds
@@ -1077,42 +712,6 @@ silent divergence, not a startup error. The only supported way back to
an older version is restoring the pre-upgrade backup, which discards an older version is restoring the pre-upgrade backup, which discards
everything received since that backup was taken. everything received since that backup was taken.
### Version stamping
The binary reports its version at `/.well-known/healthcheck` (the
`version` field), in the UI footer, and in the startup log line
(`msg=starting`, `version=...`). It is also the Sentry release name,
as `webhooker-{version}`. The value is stamped in at build time by the
linker; it is not read from a file at runtime, so it identifies the
build itself.
`script/version` produces the value and both build paths use it:
| Build | What it reports |
| --- | --- |
| Clean checkout at a tag | exactly that tag, e.g. `v1.0.0` |
| Commits past a tag | `v1.0.0-3-g1a2b3c4` — tag, commits since, short SHA |
| No tag reachable | the short SHA, e.g. `1a2b3c4` |
| Uncommitted changes | the above with a `-dirty` suffix |
| No git metadata | `unknown` |
`unknown` is what a source tarball or a `docker build .` with no
`--build-arg VERSION=...` reports. `.dockerignore` excludes `.git/`, so
the build context carries no git metadata and the image cannot derive
the version itself: `script/docker` (and so `make docker`) resolves it
on the host and passes it in as the `VERSION` build arg. A build that
reports `unknown` is a build nobody told what it was; it is not a
failure, but it cannot be traced back to a commit.
`make version` prints what the current checkout would stamp, and
`make build VERSION=v1.2.3` overrides it. An empty override — from
`make build VERSION=` or from `--build-arg VERSION=` — means unset
rather than `""`, and resolves the way an absent one does.
Nothing that varies between two builds of the same commit is stamped —
no timestamp, no hostname, no builder identity — so two builds of one
commit still produce a byte-identical binary.
### Backups contain secrets ### Backups contain secrets
Treat a backup with the same care as the credentials inside it. Encrypt Treat a backup with the same care as the credentials inside it. Encrypt
@@ -1145,29 +744,119 @@ backups at rest and restrict who can read them.
credential that was in a backup you cannot account for. credential that was in a backup you cannot account for.
- `webhooker.db` stores target config **unencrypted**, tracked at - `webhooker.db` stores target config **unencrypted**, tracked at
[issue #212](https://git.eeqj.de/sneak/webhooker/issues/212), next to [issue #212](https://git.eeqj.de/sneak/webhooker/issues/212), next to
the session encryption key and the Argon2id password hashes. the session encryption key and the Argon2id password hashes. It also
holds each entrypoint's inbound signature secret in the clear, for
the reason given under
[Inbound Signature Verification](#inbound-signature-verification):
HMAC verification needs the key itself, so it cannot be hashed.
## The entrypoint URL is the authentication secret ## Inbound Signature Verification
The receiver verifies nothing about an inbound request. The UUID in an A receiver URL is a bare v4 UUID in a path. That is unguessable, but it
entrypoint's URL is its credential: anyone who holds that URL can is not a credential: anyone who learns it — from a browser history, a
submit events to it, and the receiver checks nothing else about the proxy log, a screenshot, a copy-pasted support ticket — can post events
sender. Treat an entrypoint URL the way you would treat an API token. that webhooker stores and forwards, and the inbound headers are passed
on to your targets almost verbatim, so they also choose what the
downstream service sees. Verification is how an entrypoint stops
accepting anything that reaches its URL.
There is no way to rotate the UUID in place. To retire one, delete the It is optional and configured per entrypoint. An entrypoint with no
entrypoint (or deactivate it, which answers `410`) and create a new scheme selected is not verified, which is what every entrypoint was
one, then point the sender at the new URL. before this existed and what every entrypoint remains after an
upgrade — enabling verification is always a deliberate act, and no
existing deployment is locked out of its own receivers by installing a
new version.
When a scheme **is** selected, a request whose signature is missing,
malformed or wrong is answered `401` and **nothing is stored**: no
event row, no delivery row, no delivery attempt. Rejection happens
after the body is read (the signature covers it) and before the first
write.
### Supported schemes
| Scheme | Header | Check |
| -------- | --------------------- | ----- |
| `github` | `X-Hub-Signature-256` | HMAC-SHA256 of the raw request body under the shared secret, hex-encoded, prefixed `sha256=` |
| `gitlab` | `X-Gitlab-Token` | The header is the shared secret itself, compared as-is |
Both comparisons run in constant time (`hmac.Equal`). The HMAC is
computed over the request body exactly as received, before any parsing,
and under the same 1 MB body cap every other request obeys — an
unsigned sender cannot make webhooker buffer more than a signed one.
GitHub's older SHA-1 `X-Hub-Signature` is **not** accepted. Neither is
a GitHub digest sent without its `sha256=` prefix.
### Configuring a sender
The secret is a value you choose and enter in two places: at the sender
and in webhooker. webhooker never generates or displays one, so there
is no stored credential the UI can be made to reveal.
1. Generate a secret, e.g. `openssl rand -hex 32`.
2. In webhooker, open the webhook's page, find the entrypoint, and
click **Configure** (or **Rotate**, if it already has one). Select
the scheme and paste the secret. Surrounding whitespace is stripped,
so a value pasted with a trailing space still works; a secret whose
own first or last character is a space cannot be stored.
3. At the sender:
- **GitHub** — repository (or organization) → Settings → Webhooks →
the hook → **Secret**. GitHub then signs every delivery with
`X-Hub-Signature-256`.
- **GitLab** — project → Settings → Webhooks → the hook → **Secret
token**. GitLab sends it verbatim as `X-Gitlab-Token`.
**Rotation** is the same form: submit the new secret. Deliveries signed
with the old secret are rejected from that moment, so change it at the
sender in the same sitting. Selecting **None** removes verification and
deletes the stored secret with it.
The page shows which scheme an entrypoint uses and which header it
reads, never the secret. The value is stored in the clear — HMAC
verification needs the key itself, and a hash of it cannot recompute a
sender's digest — so it is handled like the other credentials
webhooker holds: excluded from JSON, kept out of templates by a
projection (`handlers.EntrypointView`), and absent from every log line,
including the ones written when verification fails.
### The credential is not stored or forwarded
Under the `gitlab` scheme the signature header **is** the secret. An
accepted request's headers are persisted on the event and forwarded to
every delivery target, so `X-Gitlab-Token` is removed from that copy
before the event is written — otherwise every target operator, every
backup and everyone with read access to `events-*.db` would hold the
value needed to forge signed requests to the entrypoint it protects.
The sender's other headers are untouched, and the request the receiver
itself verifies against is not modified.
The stripping is driven by the scheme's own description rather than by
a header name, and a scheme is stripped unless it declares that its
header carries a digest. `github` declares it: `X-Hub-Signature-256` is
an HMAC over the body, from which the key cannot be recovered, so it is
stored and forwarded intact. A scheme added later is stripped by
default.
### When configuration is broken
An entrypoint whose stored scheme this build does not recognise, or
which has one half of the scheme/secret pair and not the other, is
answered `500` and stores nothing. It is not treated as unverified. The
UI cannot create such a row — it rejects an unknown scheme with a `400`
— so this covers a hand-edited database or a downgrade to a build that
predates a scheme. Failing closed is the point: an entrypoint the
operator believes is protected must never quietly go back to accepting
anything.
## Entrypoints ## Entrypoints
This repository adheres to the This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: normalized scripts in `script/` are the entrypoints for the standard: normalized scripts in `script/` are the entrypoints for the
development workflow. Ten of the Makefile's seventeen targets are thin development workflow. Ten of the Makefile's sixteen targets are thin
shims that call them; `build`, `run`, `dev`, `deps`, `clean`, `css` and shims that call them; `build`, `run`, `dev`, `deps`, `clean` and `css`
`version` are inline commands with no script behind them, though are inline commands with no script behind them. We provide:
`build` and `version` both take their value from `script/version`. We
provide:
- `script/bootstrap` — install all dependencies (idempotent) - `script/bootstrap` — install all dependencies (idempotent)
- `script/setup` — make a fresh clone ready for development - `script/setup` — make a fresh clone ready for development
@@ -1180,11 +869,7 @@ provide:
- `script/fmt` — format all code (writes) - `script/fmt` — format all code (writes)
- `script/fmt-check` — check formatting (read-only) - `script/fmt-check` — check formatting (read-only)
- `script/check` — run test, lint, and fmt-check - `script/check` — run test, lint, and fmt-check
- `script/version` — output the version to stamp into the binary (see - `script/docker` — build the Docker image tagged via `script/projectname`
[Version stamping](#version-stamping))
- `script/docker` — build the Docker image tagged via
`script/projectname`, passing `script/version`'s output in as the
`VERSION` build arg
- `script/cibuild` — CI entrypoint: `docker build .` (the Dockerfile - `script/cibuild` — CI entrypoint: `docker build .` (the Dockerfile
runs the checks, so a green build implies a green repo) runs the checks, so a green build implies a green repo)
- `script/ci-mark-superseded` — CI helper: mark the commits whose run a - `script/ci-mark-superseded` — CI helper: mark the commits whose run a
@@ -1234,16 +919,6 @@ webhooker solves this by acting as a durable intermediary:
backoff. Every delivery attempt is logged with status codes, response backoff. Every delivery attempt is logged with status codes, response
bodies, and timing. bodies, and timing.
**That guarantee is at-least-once, not exactly-once.** When a send
reaches its target but the write recording that outcome fails, the
delivery is deliberately left in a recoverable state rather than
marked done — losing a delivery is the worse failure — so the
pending sweep picks it up about fifteen minutes later, or the next
restart does, and the target receives a payload it already got.
webhooker adds no delivery identifier of its own to an outbound
request, so **make your receiver idempotent** against whatever the
payload itself carries.
3. **Observability** — Full request/response logging for every webhook 3. **Observability** — Full request/response logging for every webhook
received and every delivery attempted. Prometheus metrics expose received and every delivery attempted. Prometheus metrics expose
volume, latency, and error rates. The web UI provides real-time volume, latency, and error rates. The web UI provides real-time
@@ -1452,12 +1127,18 @@ the full request and creates an Event.
| -------------- | ------- | ----------- | | -------------- | ------- | ----------- |
| `id` | UUID | Primary key | | `id` | UUID | Primary key |
| `webhook_id` | UUID | Foreign key → Webhook | | `webhook_id` | UUID | Foreign key → Webhook |
| `path` | string | Unique bare UUID, generated at creation. The `/webhook/` prefix is route only and is not stored: the receiver matches this column against the raw `{uuid}` path segment. It is also the entrypoint's credential; see [The entrypoint URL is the authentication secret](#the-entrypoint-url-is-the-authentication-secret) | | `path` | string | Unique bare UUID, generated at creation. The `/webhook/` prefix is route only and is not stored: the receiver matches this column against the raw `{uuid}` path segment |
| `description` | string | Optional description | | `description` | string | Optional description |
| `active` | boolean | Whether this entrypoint accepts events (default: true) | | `active` | boolean | Whether this entrypoint accepts events (default: true) |
| `signature_scheme` | string | How inbound requests are authenticated: `github`, `gitlab`, or empty for no verification (default: empty). See [Inbound Signature Verification](#inbound-signature-verification) |
| `signature_secret` | string | The secret shared with the sender, stored in the clear because HMAC verification needs the key itself. Never marshalled to JSON, never rendered, never logged. Empty when no scheme is set |
**Relations:** Belongs to Webhook. **Relations:** Belongs to Webhook.
Both signature columns arrive through `AutoMigrate` with an empty
default, so every entrypoint written before they existed migrates to
"not configured" and keeps accepting the traffic it already accepted.
A webhook can have multiple entrypoints. This allows separate URLs for A webhook can have multiple entrypoints. This allows separate URLs for
different event sources that all feed into the same processing pipeline different event sources that all feed into the same processing pipeline
(e.g., one entrypoint for GitHub, another for Stripe, both routing to (e.g., one entrypoint for GitHub, another for Stripe, both routing to
@@ -1651,6 +1332,10 @@ fired twenty times at a backend stays traceable. Resubmitting the same
event repeatedly is supported and is the point of the action — there is event repeatedly is supported and is the point of the action — there is
no in-flight refusal; the route's own rate limit is what bounds it. no in-flight refusal; the route's own rate limit is what bounds it.
Inbound signature verification is not re-run on a resubmit. There is no
inbound signature to check on a copy the operator submits, and the
route is authenticated and CSRF-protected as an operator action.
#### DeliveryResult #### DeliveryResult
The result of a single delivery attempt. Every attempt (including The result of a single delivery attempt. Every attempt (including
@@ -1690,11 +1375,6 @@ webhooker uses **separate SQLite database files**: a main application
database for configuration data and per-webhook databases for event database for configuration data and per-webhook databases for event
storage. All database files live in the `DATA_DIR` directory. storage. All database files live in the `DATA_DIR` directory.
Every one of them is created `0600`, and so is each `-wal` and `-shm`
sidecar. See
[Running with Docker](#running-with-docker) for what that does and
does not protect.
**Main Application Database** (`{DATA_DIR}/webhooker.db`) — stores **Main Application Database** (`{DATA_DIR}/webhooker.db`) — stores
configuration and application state: configuration and application state:
@@ -1818,15 +1498,17 @@ External Service
│ │ │ Stack │ │ Handler │ │ │ │ Stack │ │ Handler │
└─────────────┘ └──────────────┘ └──────┬───────┘ └─────────────┘ └──────────────┘ └──────┬───────┘
│ │
1. Look up Entrypoint by UUID — 404 if unknown, 1. Look up Entrypoint by UUID
410 if inactive
2. Read the body under the 1 MB cap 2. Read the body under the 1 MB cap
3. Capture full request as Event 3. Verify the signature, if the entrypoint has
4. Create Delivery records for each active Target one configured — 401 and no writes if it
5. Build self-contained delivery.Task structs fails (see Inbound Signature Verification)
4. Capture full request as Event
5. Create Delivery records for each active Target
6. Build self-contained delivery.Task structs
(target config + event data inline for (target config + event data inline for
bodies < 16 KiB) bodies < 16 KiB)
6. Notify Engine via channel (no DB read needed) 7. Notify Engine via channel (no DB read needed)
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
@@ -2022,40 +1704,6 @@ gauge. The outcome counters move only after the status change has been
written, so a transition the database rejected is never reported as an written, so a transition the database rejected is never reported as an
outcome that happened. outcome that happened.
#### Inbound HTTP metrics
The middleware records three more on the same registry:
| Metric | Type | Labels |
| ------ | ---- | ------ |
| `http_request_duration_seconds` | histogram | `service`, `handler`, `method`, `code` |
| `http_response_size_bytes` | histogram | `service`, `handler`, `method`, `code` |
| `http_requests_inflight` | gauge | `service`, `handler` |
Two of those labels are written once per request from bytes the client
chose, so both are bounded to something this service registers:
- `handler` is the chi route pattern — `/webhook/{uuid}`, never the
concrete path. A request matching no route carries `(unmatched)`,
and no entrypoint UUID ever reaches a label.
- `method` is the request method when the router can route it, and
`(unmatched)` otherwise. `net/http` accepts any RFC 9110 token as a
method, so the raw value bounds the label at nothing; the nine chi
matches routes for stay distinguishable, and a token that could only
ever have produced a 405 does not get a series of its own.
The other two are not request-controlled: `code` is the status one of
this service's own handlers wrote, and `service` is a fixed empty
string.
`http_requests_inflight` is deliberately aggregate — its `handler` is
always `(all)`, one series counting the requests in flight across the
whole service. The gauge is incremented before routing and decremented
after the handler returns, and the route pattern exists only between
those two moments, so labelling it by pattern would increment one
series and decrement another, leaving every pattern permanently off by
the number of requests it served.
### Rate Limiting ### Rate Limiting
Global blanket rate limiting middleware (e.g., a per-IP throttle shared Global blanket rate limiting middleware (e.g., a per-IP throttle shared
@@ -2198,16 +1846,13 @@ the rest. Nothing dropped is needed for the likeliest use, debugging a
CSRF rejection. Its three inputs are the TLS decision, `Origin` and CSRF rejection. Its three inputs are the TLS decision, `Origin` and
`Referer`; the latter two are kept, and the first is the scheme of the `Referer`; the latter two are kept, and the first is the scheme of the
retained URL, because the SDK derives that scheme from retained URL, because the SDK derives that scheme from
`r.TLS != nil || r.Header.Get("X-Forwarded-Proto") == "https"`. That `r.TLS != nil || r.Header.Get("X-Forwarded-Proto") == "https"` — byte
predicate is the SDK's own and is stricter than `internal/reqtls.IsTLS`, for byte the predicate `internal/middleware/csrf.go` uses to choose
which this service now uses everywhere it decides transport: the SDK between the `csrf.Secure(true)` and `csrf.Secure(false)` handlers.
reports `http` for the `HTTPS` and `https, http` spellings `reqtls` That is what the rewrite above preserves it for, and it is why
accepts. Only a reported scheme is affected, no decision is, so it is dropping `X-Forwarded-Proto` costs nothing. The dropped provider
left to the SDK rather than reimplemented. That is what the rewrite headers (`X-GitHub-Event`, `X-Gitlab-Event` and the like) are real
above preserves it for, and it is why dropping `X-Forwarded-Proto` signal but are recorded locally on the event, and
costs nothing. The dropped provider headers (`X-GitHub-Event`,
`X-Gitlab-Event` and the like) are real signal but are recorded
locally on the event, and
`Sentry-Trace`/`Baggage` are already reflected in the event's trace `Sentry-Trace`/`Baggage` are already reflected in the event's trace
context. context.
@@ -2626,15 +2271,13 @@ abuse limit later; they are tracked as future work.
| `POST` | `/source/{id}/edit` | Edit webhook submission | | `POST` | `/source/{id}/edit` | Edit webhook submission |
| `POST` | `/source/{id}/delete` | Delete webhook | | `POST` | `/source/{id}/delete` | Delete webhook |
| `GET` | `/source/{id}/logs` | Webhook event logs | | `GET` | `/source/{id}/logs` | Webhook event logs |
| `GET` | `/source/{id}/logs/{eventID}/body` | Download an event's full stored body. The log page renders each body only up to its cap, so this is the only route that serves a whole one; it is offered wherever a body is shown truncated |
| `POST` | `/source/{id}/deliveries/{deliveryID}/replay` | Replay a finished delivery: creates a new delivery for the same event against the target's current configuration (30 per minute per bucket, then `429`) | | `POST` | `/source/{id}/deliveries/{deliveryID}/replay` | Replay a finished delivery: creates a new delivery for the same event against the target's current configuration (30 per minute per bucket, then `429`) |
| `POST` | `/source/{id}/events/{eventID}/resubmit` | Resubmit a stored event: creates a new event copying it and fans that out to every currently active target (30 per minute per bucket, then `429`) | | `POST` | `/source/{id}/events/{eventID}/resubmit` | Resubmit a stored event: creates a new event copying it and fans that out to every currently active target (30 per minute per bucket, then `429`) |
| `POST` | `/source/{id}/entrypoints` | Add entrypoint to webhook | | `POST` | `/source/{id}/entrypoints` | Add entrypoint to webhook |
| `POST` | `/source/{id}/entrypoints/{entrypointID}/delete` | Delete an entrypoint | | `POST` | `/source/{id}/entrypoints/{entrypointID}/delete` | Delete an entrypoint |
| `POST` | `/source/{id}/entrypoints/{entrypointID}/toggle` | Enable or disable an entrypoint | | `POST` | `/source/{id}/entrypoints/{entrypointID}/toggle` | Enable or disable an entrypoint |
| `POST` | `/source/{id}/entrypoints/{entrypointID}/secret` | Set, rotate or remove the entrypoint's inbound signature scheme and secret (see [Inbound Signature Verification](#inbound-signature-verification)) |
| `POST` | `/source/{id}/targets` | Add target to webhook | | `POST` | `/source/{id}/targets` | Add target to webhook |
| `GET` | `/source/{id}/targets/{targetID}/edit` | Edit target form. The one page that renders a target's destination URL and header values in full, rather than masked |
| `POST` | `/source/{id}/targets/{targetID}/edit` | Edit target submission |
| `POST` | `/source/{id}/targets/{targetID}/delete` | Delete a target | | `POST` | `/source/{id}/targets/{targetID}/delete` | Delete a target |
| `POST` | `/source/{id}/targets/{targetID}/toggle` | Enable or disable a target | | `POST` | `/source/{id}/targets/{targetID}/toggle` | Enable or disable a target |
@@ -2673,8 +2316,6 @@ webhooker/
├── internal/ ├── internal/
│ ├── banner/ │ ├── banner/
│ │ └── banner.go # Ruled block for the one credential shown in the clear │ │ └── banner.go # Ruled block for the one credential shown in the clear
│ ├── ciscript/
│ │ └── doc.go # Tests for the CI shell scripts in script/; no runtime code
│ ├── resetpw/ │ ├── resetpw/
│ │ └── resetpw.go # `webhooker resetpw`: set an account's password, stopped deployments only │ │ └── resetpw.go # `webhooker resetpw`: set an account's password, stopped deployments only
│ ├── config/ │ ├── config/
@@ -2686,7 +2327,7 @@ webhooker/
│ │ ├── model_setting.go # Setting entity (key-value app config) │ │ ├── model_setting.go # Setting entity (key-value app config)
│ │ ├── model_user.go # User entity │ │ ├── model_user.go # User entity
│ │ ├── model_webhook.go # Webhook entity │ │ ├── model_webhook.go # Webhook entity
│ │ ├── model_entrypoint.go # Entrypoint entity │ │ ├── model_entrypoint.go # Entrypoint entity and SignatureScheme enum
│ │ ├── model_target.go # Target entity and TargetType enum │ │ ├── model_target.go # Target entity and TargetType enum
│ │ ├── model_event.go # Event entity (per-webhook DB) │ │ ├── model_event.go # Event entity (per-webhook DB)
│ │ ├── model_delivery.go # Delivery entity (per-webhook DB) │ │ ├── model_delivery.go # Delivery entity (per-webhook DB)
@@ -2744,8 +2385,6 @@ webhooker/
│ │ ├── ratelimit.go # Per-IP rate limiting middleware (go-chi/httprate) │ │ ├── ratelimit.go # Per-IP rate limiting middleware (go-chi/httprate)
│ │ ├── loginguard.go # Login failure counters and the Argon2id verification semaphore │ │ ├── loginguard.go # Login failure counters and the Argon2id verification semaphore
│ │ └── testing.go # NewForTest: Middleware without the fx lifecycle │ │ └── testing.go # NewForTest: Middleware without the fx lifecycle
│ ├── reqtls/
│ │ └── reqtls.go # IsTLS: the one TLS predicate, r.TLS or X-Forwarded-Proto
│ ├── server/ │ ├── server/
│ │ ├── server.go # Server struct, fx lifecycle, signal handling │ │ ├── server.go # Server struct, fx lifecycle, signal handling
│ │ ├── http.go # HTTP server setup with timeouts │ │ ├── http.go # HTTP server setup with timeouts
@@ -2753,8 +2392,8 @@ webhooker/
│ ├── session/ │ ├── session/
│ │ ├── session.go # Cookie-based session management │ │ ├── session.go # Cookie-based session management
│ │ └── testing.go # NewForTest: Session without the fx lifecycle │ │ └── testing.go # NewForTest: Session without the fx lifecycle
│ └── versionscript/ │ └── signature/
│ └── doc.go # Tests for script/version and the build files that use it │ └── signature.go # Inbound signature verification (GitHub, GitLab)
├── static/ ├── static/
│ ├── static.go # //go:embed directive │ ├── static.go # //go:embed directive
│ ├── css/input.css # Tailwind input, source for tailwind.css (make css) │ ├── css/input.css # Tailwind input, source for tailwind.css (make css)
@@ -2767,7 +2406,7 @@ webhooker/
├── script/ # Scripts to Rule Them All entrypoints ├── script/ # Scripts to Rule Them All entrypoints
├── Dockerfile # Three stages: lint, test+build, Alpine runtime ├── Dockerfile # Three stages: lint, test+build, Alpine runtime
├── Dockerfile.lint # Lint-only image built by script/lint ├── Dockerfile.lint # Lint-only image built by script/lint
├── Makefile # 10 of 17 targets shim script/; 7 are inline ├── Makefile # 10 of 16 targets shim script/; 6 are inline
├── go.mod / go.sum ├── go.mod / go.sum
└── .golangci.yml # Linter configuration └── .golangci.yml # Linter configuration
``` ```
@@ -2869,9 +2508,8 @@ check, see [The login endpoint](#the-login-endpoint).
- **Web UI:** Cookie-based sessions using gorilla/sessions with - **Web UI:** Cookie-based sessions using gorilla/sessions with
encrypted cookies. Sessions are configured with HttpOnly, SameSite encrypted cookies. Sessions are configured with HttpOnly, SameSite
Lax, and Secure whenever the request is on TLS — the flag follows the Lax, and Secure (in production). Absolute session lifetime is 7 days,
request's transport, not the environment. Absolute session lifetime with a sliding idle timeout on top of it (see
is 7 days, with a sliding idle timeout on top of it (see
[Sessions](#sessions)). [Sessions](#sessions)).
- **API (planned):** API key authentication via `Authorization: Bearer` - **API (planned):** API key authentication via `Authorization: Bearer`
header. API keys are stored per-user with usage tracking header. API keys are stored per-user with usage tracking
@@ -2884,10 +2522,7 @@ check, see [The login endpoint](#the-login-endpoint).
### Security ### Security
- Passwords hashed with Argon2id (64 MB memory cost) - Passwords hashed with Argon2id (64 MB memory cost)
- Session cookies are HttpOnly, SameSite Lax, and Secure on any request - Session cookies are HttpOnly, SameSite Lax, Secure (prod only)
that arrived over TLS (directly or through a reverse proxy reporting
it), decided per-request by `internal/reqtls.IsTLS` rather than by the
configured environment
- Session regeneration on login to prevent session fixation attacks - Session regeneration on login to prevent session fixation attacks
- Session key is a 32-byte value auto-generated on first startup and - Session key is a 32-byte value auto-generated on first startup and
stored in the database stored in the database
@@ -2900,14 +2535,18 @@ check, see [The login endpoint](#the-login-endpoint).
on all state-changing forms (cookie-based double-submit tokens with on all state-changing forms (cookie-based double-submit tokens with
HMAC authentication). Applied to `/pages`, `/sources`, `/source`, and HMAC authentication). Applied to `/pages`, `/sources`, `/source`, and
`/user` routes. Excluded from `/webhook` (inbound webhook POSTs) and `/user` routes. Excluded from `/webhook` (inbound webhook POSTs) and
`/api` (stateless API). The middleware detects TLS per-request through `/api` (stateless API). The middleware auto-detects TLS status
`internal/reqtls.IsTLS` — the same predicate the session cookie uses — per-request (via `r.TLS` and `X-Forwarded-Proto`) to set appropriate
to set appropriate cookie security flags and Origin/Referer validation cookie security flags and Origin/Referer validation mode
mode - **Optional inbound signature verification** per entrypoint (GitHub
- **The entrypoint URL is the receiver's only credential.** Nothing `X-Hub-Signature-256`, GitLab `X-Gitlab-Token`). Off by default and
about an inbound request is verified; possession of the UUID off after an upgrade, so behaviour is unchanged until an operator
authorises submission (see turns it on. Where it is on, an unsigned or wrongly signed request
[The entrypoint URL is the authentication secret](#the-entrypoint-url-is-the-authentication-secret)) is `401` and is not persisted, and a configuration the receiver
cannot apply fails closed rather than reverting to unverified. The
comparison is constant time and the secret never reaches a template,
a JSON response or a log line (see
[Inbound Signature Verification](#inbound-signature-verification))
- **SSRF prevention** for HTTP delivery targets: private/reserved IP - **SSRF prevention** for HTTP delivery targets: private/reserved IP
ranges (RFC 1918, loopback, link-local, cloud metadata) are blocked ranges (RFC 1918, loopback, link-local, cloud metadata) are blocked
both at target creation time (URL validation) and at delivery time both at target creation time (URL validation) and at delivery time
@@ -3067,11 +2706,7 @@ version is fixed independently of the compiler's:
stage passing (it copies a file from it), runs `script/fetch-assets` stage passing (it copies a file from it), runs `script/fetch-assets`
to download and verify the third-party browser assets, then runs to download and verify the third-party browser assets, then runs
`make test` and `make build`, and finally rebuilds the binary with `make test` and `make build`, and finally rebuilds the binary with
`CGO_ENABLED=1` and static linking so it runs on musl. Both builds `CGO_ENABLED=1` and static linking so it runs on musl.
go through `make build`, the relink adding its `-extldflags` via
`GO_LDFLAGS`, so neither can drop the `-X` that stamps the version.
The version arrives as the `VERSION` build arg, since the context
has no `.git` (see [Version stamping](#version-stamping)).
3. **Runtime stage** (`alpine:3.21`) — copies the static binary, 3. **Runtime stage** (`alpine:3.21`) — copies the static binary,
creates the `/var/lib/webhooker` directory for all SQLite databases, creates the `/var/lib/webhooker` directory for all SQLite databases,
runs as the non-root `webhooker` user (UID 1000), exposes port 8080, runs as the non-root `webhooker` user (UID 1000), exposes port 8080,

177
TODO.md
View File

@@ -18,27 +18,29 @@ Issue branches do NOT touch this file — the manager maintains it on
# Status # Status
The milestone (https://git.eeqj.de/sneak/webhooker/milestone/9) is the 1.0.0 is complete: 55 closed, 0 open. `next` (6874059) is 62 commits
authoritative list, and the only place to read a count or a state of ahead of `main` and a strict fast-forward. No git tags exist yet.
play from. This file records where the project is, not what is in
flight: a sentence whose truth depends on a branch being unmerged is
wrong the moment it merges, and this file has been wrong that way
before.
The durability defect that held the tag has landed The bar was not "the milestone is empty" but "sneak can deploy this and
(https://git.eeqj.de/sneak/webhooker/issues/256, commit `8d64259`). use it in low-volume production". Every gap the deployability audit
Every SQLite handle opens with WAL journaling and a busy timeout, a named against that bar is now closed:
bookkeeping write that fails leaves its delivery in a recoverable
state rather than a lying one, and recovery skips a delivery that
already has a successful result row. Final pre-tag verification
exercised it and confirmed it holds. Whatever the milestone still
shows open is what remains before `v1.0.0`.
Delivery is at-least-once by design, not by accident: a send whose - `DATA_DIR` locking, so two instances cannot both deliver
result row does not land is attempted again, so a receiver can see a (https://git.eeqj.de/sneak/webhooker/issues/201)
duplicate. That is deliberate — the alternative is a silent lost - shutdown on listener failure, rather than a live non-serving process
delivery — and the README says so under Rationale. It is not a defect (https://git.eeqj.de/sneak/webhooker/issues/200)
to re-file. - inbound signature verification
(https://git.eeqj.de/sneak/webhooker/issues/67)
- per-attempt delivery detail in the event log
(https://git.eeqj.de/sneak/webhooker/issues/202)
- replay of a terminally failed delivery
(https://git.eeqj.de/sneak/webhooker/issues/203)
- `ALLOWED_EGRESS_CIDRS`, an allowlist escape hatch for the SSRF guard
(https://git.eeqj.de/sneak/webhooker/issues/204)
- the three credential exposures
(https://git.eeqj.de/sneak/webhooker/issues/205,
https://git.eeqj.de/sneak/webhooker/issues/206,
https://git.eeqj.de/sneak/webhooker/issues/207)
One caveat on reading a green check: a docs-only commit deliberately One caveat on reading a green check: a docs-only commit deliberately
replays from the layer cache replays from the layer cache
@@ -48,129 +50,23 @@ commit invalidates the `COPY` layer and genuinely executes.
# Next Step # Next Step
Clear the rest of the open 1.0.0 milestone Merge the milestone PR (https://git.eeqj.de/sneak/webhooker/pulls/111)
(https://git.eeqj.de/sneak/webhooker/milestone/9) and tag `v1.0.0`. and tag `v1.0.0`. It is `merge-ready` and assigned to sneak; nothing
Merging `next` into `main` is a separate act from tagging and waits on else gates it.
neither of those: `next` is kept mergeable at all times, which is the
point of the branch. Post-1.0 follow-ups are open, none blocking the tag:
https://git.eeqj.de/sneak/webhooker/issues/245,
https://git.eeqj.de/sneak/webhooker/issues/246,
https://git.eeqj.de/sneak/webhooker/issues/247 and
https://git.eeqj.de/sneak/webhooker/issues/248. Also still open and
unmilestoned: https://git.eeqj.de/sneak/webhooker/issues/193 (a design
question, not a defect), https://git.eeqj.de/sneak/webhooker/issues/198
(`make test` is past the org 20s target) and
https://git.eeqj.de/sneak/webhooker/issues/212 (encrypting target config
at rest).
# Completed Steps # Completed Steps
- 2026-08-24 Bind the plaintext HTTP listener deliberately, via
`BIND_ADDRESS` defaulting to `127.0.0.1`, and document the
reverse-proxy deployment. A hostname, an empty value or a value
carrying a port is a startup error, and the `Dockerfile` sets
`0.0.0.0` because a loopback bind inside a container is unreachable
(https://git.eeqj.de/sneak/webhooker/issues/268). The same commit
removed the shutdown race: `httpServer` is built in the constructor
rather than assigned from the serving goroutine, which orders the
write before every fx hook and rules out the nil dereference a
SIGTERM arriving first would have caused, and `sentryEnabled` is an
`atomic.Bool` (https://git.eeqj.de/sneak/webhooker/issues/226)
- 2026-08-24 Remove inbound request signature verification. The
entrypoint UUID is the authentication secret, so the per-entrypoint
shared secret, the `internal/signature` package, the receiver check,
the model fields and the forms are all gone. This reverses the
feature that landed earlier in the same milestone
(https://git.eeqj.de/sneak/webhooker/issues/67,
https://git.eeqj.de/sneak/webhooker/issues/279)
- 2026-08-24 Stamp the build version into the binary and render it in
the UI footer. `script/version` is the single source — `$VERSION`,
else `git describe --tags --always --dirty`, else `unknown` — so a
`make build` binary and a `make docker` image from one checkout
report the same thing, and nothing in it varies between two builds
of the same commit, which the release gate's byte-identical
assertion would catch
(https://git.eeqj.de/sneak/webhooker/issues/253)
- 2026-08-24 Derive cookie `Secure` and CSRF strictness from the
request transport rather than from `WEBHOOKER_ENVIRONMENT`. Behind a
real TLS proxy with the environment left at its `dev` default, the
session cookie silently lost `Secure` while the CSRF cookie on the
same response kept it. `X-Forwarded-Proto` is now matched
case-insensitively on its first comma-separated element, so `HTTPS`
and `https, http` no longer fall to the relaxed CSRF path
(https://git.eeqj.de/sneak/webhooker/issues/269)
- 2026-08-24 Roll back a failed webhook deletion instead of committing
it. A failing delete committed whatever had already succeeded,
hard-deleted the per-webhook event database anyway, and redirected as
though it had worked — orphaned config plus permanently destroyed
history, reported as success. All three delete positions now roll
back with the event database intact
(https://git.eeqj.de/sneak/webhooker/issues/262)
- 2026-08-24 Name a deleted target on its historical deliveries, marked
`(deleted)`, rather than leaving the event log unable to say where a
delivery went. A deleted target's credentials stay masked exactly as
a live one's, and it cannot become deliverable again through the
receiver, resubmit, replay, the edit form or the toggle
(https://git.eeqj.de/sneak/webhooker/issues/211)
- 2026-08-24 Bound both request-controlled `/metrics` label dimensions,
so the unauthenticated receiver is no longer a memory-exhaustion
vector: `handler` carries the chi route pattern, and `method` folds
anything chi cannot route onto a single `(unmatched)` sentinel. Both
were reproduced before the fix — 300 random method tokens took the
series count from 106 to 7,631, and path flooding reached 62,532 —
and a label audit across a live scrape found no third unbounded
dimension (https://git.eeqj.de/sneak/webhooker/issues/254,
https://git.eeqj.de/sneak/webhooker/issues/261)
- 2026-08-24 Validate `max_retries` on both target forms. `abc`, `2.7`
and `-5` silently became 0 — fire-and-forget — including on the edit
path, where it destroyed a working value, and `999999999` stored
verbatim. The ceiling of 20 is the `max` both templates already
declared (https://git.eeqj.de/sneak/webhooker/issues/221)
- 2026-08-24 Resubmit a stored event as a new undelivered event, so a
backend under development can be tested against real captured
traffic. Per-delivery replay cannot serve that: it re-sends one
finished delivery to its own original target, and a target created
for a dev backend has no prior delivery to replay. Resubmit
re-injects the stored event at the top of the receiver path and fans
it out to whatever targets are active now
(https://git.eeqj.de/sneak/webhooker/issues/250)
- 2026-08-20 Take an exclusive lock on `DATA_DIR` at startup, so two
instances on one directory cannot both deliver
(https://git.eeqj.de/sneak/webhooker/issues/201)
- 2026-08-20 Shut down the app when the HTTP listener fails. The
`OnStart` hook returned as soon as the serving goroutine was
spawned, so a failed listen left fx reporting RUNNING and a live
process with nothing bound — invisible to systemd and Docker restart
policies (https://git.eeqj.de/sneak/webhooker/issues/200)
- 2026-08-20 Stop target credentials leaking into the per-webhook event
databases (https://git.eeqj.de/sneak/webhooker/issues/206), log SQL
with placeholders rather than bound values
(https://git.eeqj.de/sneak/webhooker/issues/207), and fail loudly on
half-set metrics auth credentials
(https://git.eeqj.de/sneak/webhooker/issues/205)
- 2026-08-20 Read queue depths with `Find`, not `Scan`. `Scan` swaps
GORM's own trace recorder in for the logging adapter, and that
recorder does not implement `gorm.ParamsFilter`, so those statements
logged their bound values interpolated and bypassed the suppression
above. The two units gated green against a `next` that lacked the
other, and `next` went red when both landed
(https://git.eeqj.de/sneak/webhooker/issues/234)
- 2026-08-20 Render per-attempt delivery detail in the event log
(https://git.eeqj.de/sneak/webhooker/issues/202) and add replay of a
terminally failed delivery
(https://git.eeqj.de/sneak/webhooker/issues/203)
- 2026-08-20 Expose delivery metrics on `/metrics`
(https://git.eeqj.de/sneak/webhooker/issues/209) and document the
backup, restore and upgrade procedures
(https://git.eeqj.de/sneak/webhooker/issues/210)
- 2026-08-20 Add a `webhooker resetpw` subcommand and a bootstrap
banner. The admin bootstrap password was printed once among roughly
45 fx lines, and under `docker run -d` went to container logs subject
to rotation; there was no reset path at all, so recovery meant
hand-deleting the users row, documented nowhere. The password is read
from stdin or generated, never from argv where `/proc` would publish
it (https://git.eeqj.de/sneak/webhooker/issues/208)
- 2026-08-20 Add `ALLOWED_EGRESS_CIDRS`, an allowlist-only escape hatch
for the SSRF guard, so a self-hosted proxy can forward into the
operator's own network. The guard's always-blocked set cannot be
reopened by configuration
(https://git.eeqj.de/sneak/webhooker/issues/204)
- 2026-08-20 Harden operator-set target headers, which were carried
unsafely across a redirect
(https://git.eeqj.de/sneak/webhooker/issues/233)
- 2026-08-20 Add a target edit form with headers and timeout fields
(https://git.eeqj.de/sneak/webhooker/issues/127)
- 2026-08-18 Raise `script/test`'s per-package timeout from 30s to 90s, - 2026-08-18 Raise `script/test`'s per-package timeout from 30s to 90s,
matching the org-wide backstop. `go test` applies `-timeout` per matching the org-wide backstop. `go test` applies `-timeout` per
package, and `internal/handlers` had grown past the old budget: a package, and `internal/handlers` had grown past the old budget: a
@@ -403,6 +299,9 @@ point of the branch.
- Per-webhook rate limiting in the receiver handler (per-webhook config - Per-webhook rate limiting in the receiver handler (per-webhook config
plus handler enforcement; global limits must not apply to receiver plus handler enforcement; global limits must not apply to receiver
endpoints) endpoints)
- Stripe HMAC signature verification. The GitHub and GitLab schemes
landed with inbound verification
(https://git.eeqj.de/sneak/webhooker/issues/67)
- API key authentication for programmatic access (APIKey model exists; - API key authentication for programmatic access (APIKey model exists;
Bearer token middleware does not) Bearer token middleware does not)
- REST API v1 - REST API v1

107
cmd/webhooker/dotenv_test.go
View File

@@ -1,107 +0,0 @@
package main
import (
"bytes"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/config"
)
// dotEnvKey is a throwaway variable name these tests write and read,
// so they cannot disturb real configuration.
const dotEnvKey = "WEBHOOKER_TEST_DISPATCH_VALUE"
// writeDotEnvInWorkingDir puts contents in a .env file in a fresh
// temporary directory and moves the process there.
//
// The callers are deliberately not parallel and must stay that way:
// t.Chdir moves the whole process. Go releases parallel tests only
// after every sequential test in the package has finished, so nothing
// else runs while these do.
func writeDotEnvInWorkingDir(t *testing.T, contents string) {
t.Helper()
dir := t.TempDir()
require.NoError(t, os.WriteFile(
filepath.Join(dir, config.DotEnvPath),
[]byte(contents), 0o600,
))
t.Chdir(dir)
}
// TestDispatch_MalformedDotEnvRefuses pins the second half of the
// defect. godotenv applies nothing at all when a file will not parse,
// so one mistyped line used to revert every variable in it to its
// default and start the server anyway, with no log line naming the
// file. The refusal has to arrive before any subcommand runs, which
// is why `help` — the one subcommand that touches nothing — is still
// refused here.
//
//nolint:paralleltest // t.Chdir moves the whole process.
func TestDispatch_MalformedDotEnvRefuses(t *testing.T) {
writeDotEnvInWorkingDir(t, "PORT 19615\n")
var stdout, stderr bytes.Buffer
code := dispatch(
[]string{helpCommand}, strings.NewReader(""), &stdout, &stderr,
)
require.Equal(t, 1, code, "a broken .env must exit non-zero")
assert.Contains(
t, stderr.String(), config.DotEnvPath,
"the refusal must name the file",
)
assert.Empty(
t, stdout.String(),
"the subcommand must not have run",
)
}
// TestDispatch_LoadsDotEnvBeforeSubcommands pins the ordering the
// godotenv/autoload import used to provide for free. It ran in an
// init(), so .env was in the environment before anything read it —
// including config.DataDir, which both the DATA_DIR lock and resetpw
// call outside the fx graph. Loading any later would let a .env that
// sets DATA_DIR lock one directory while the config opened databases
// in another.
func TestDispatch_LoadsDotEnvBeforeSubcommands(t *testing.T) {
t.Setenv(dotEnvKey, "placeholder")
require.NoError(t, os.Unsetenv(dotEnvKey))
writeDotEnvInWorkingDir(t, dotEnvKey+"=from-dot-env\n")
var stdout, stderr bytes.Buffer
code := dispatch(
[]string{helpCommand}, strings.NewReader(""), &stdout, &stderr,
)
require.Equal(t, 0, code)
assert.Equal(
t, "from-dot-env", os.Getenv(dotEnvKey),
"the file must be applied before the subcommand runs",
)
}
// TestDispatch_MissingDotEnvIsFine pins the case most deployments are
// in: no .env at all, which must stay a normal start.
//
//nolint:paralleltest // t.Chdir moves the whole process.
func TestDispatch_MissingDotEnvIsFine(t *testing.T) {
t.Chdir(t.TempDir())
var stdout, stderr bytes.Buffer
code := dispatch(
[]string{helpCommand}, strings.NewReader(""), &stdout, &stderr,
)
require.Equal(t, 0, code)
assert.Empty(t, stderr.String())
}

23
cmd/webhooker/main.go
View File

@@ -54,11 +54,6 @@ const stopTimeout = 5 * time.Second
// caller can tell "called wrong" from "declined". // caller can tell "called wrong" from "declined".
const exitUsage = 2 const exitUsage = 2
// helpCommand is the subcommand that prints usage. The flag spellings
// beside it in the switch are aliases; this is the name the usage text
// documents and the one tests invoke.
const helpCommand = "help"
// Build-time variables set via -ldflags. // Build-time variables set via -ldflags.
// //
//nolint:gochecknoglobals // Build-time variables injected by the linker. //nolint:gochecknoglobals // Build-time variables injected by the linker.
@@ -80,27 +75,11 @@ func main() {
// every existing deployment invoke; that path is unchanged, including // every existing deployment invoke; that path is unchanged, including
// where the DATA_DIR lock is taken relative to building the fx graph // where the DATA_DIR lock is taken relative to building the fx graph
// and how fx propagates a non-zero exit itself. // and how fx propagates a non-zero exit itself.
//
// The optional .env file is read here, before any subcommand and so
// before anything reads the environment — config.DataDir, which both
// the DATA_DIR lock and resetpw call outside the fx graph, above all.
// It used to be read from an init() in internal/config, which put it
// earlier still but threw the error away: a single malformed line
// applied none of the file and said nothing about it. A file that is
// not there stays fine, since .env is optional and most deployments
// do not have one.
func dispatch( func dispatch(
args []string, args []string,
stdin io.Reader, stdin io.Reader,
stdout, stderr io.Writer, stdout, stderr io.Writer,
) int { ) int {
err := config.LoadDotEnv()
if err != nil {
_, _ = fmt.Fprintf(stderr, "%s: %v\n", appname, err)
return 1
}
if len(args) == 0 { if len(args) == 0 {
return run(stderr) return run(stderr)
} }
@@ -108,7 +87,7 @@ func dispatch(
switch args[0] { switch args[0] {
case resetpw.Name: case resetpw.Name:
return resetpw.Run(args[1:], stdin, stdout, stderr) return resetpw.Run(args[1:], stdin, stdout, stderr)
case helpCommand, "-h", "-help", "--help": case "help", "-h", "-help", "--help":
usage(stdout) usage(stdout)
return 0 return 0

2
cmd/webhooker/main_test.go
View File

@@ -121,7 +121,7 @@ func TestDispatch_Help(t *testing.T) {
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
code := dispatch( code := dispatch(
[]string{helpCommand}, strings.NewReader(""), &stdout, &stderr, []string{"help"}, strings.NewReader(""), &stdout, &stderr,
) )
require.Equal(t, 0, code) require.Equal(t, 0, code)

220
internal/config/config.go
View File

@@ -4,7 +4,6 @@ package config
import ( import (
"errors" "errors"
"fmt" "fmt"
"io/fs"
"log/slog" "log/slog"
"net/netip" "net/netip"
"os" "os"
@@ -12,11 +11,13 @@ import (
"strings" "strings"
"time" "time"
"github.com/getsentry/sentry-go"
"github.com/joho/godotenv"
"go.uber.org/fx" "go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/globals" "sneak.berlin/go/webhooker/internal/globals"
"sneak.berlin/go/webhooker/internal/logger" "sneak.berlin/go/webhooker/internal/logger"
// Populates the environment from a ./.env file automatically for
// development configuration. Kept in one place only (here).
_ "github.com/joho/godotenv/autoload"
) )
const ( const (
@@ -32,34 +33,6 @@ const (
// defaultPort is the default HTTP listen port. // defaultPort is the default HTTP listen port.
defaultPort = 8080 defaultPort = 8080
// defaultBindAddress is the interface the plaintext HTTP
// listener claims when BIND_ADDRESS is unset.
//
// Loopback, because the listener speaks cleartext and serves
// both the admin UI and the unauthenticated receiver: a
// wildcard default publishes them on every interface of every
// host that never configured anything, which is the failure
// this default exists to prevent. Reaching webhooker from off
// the host is then a deliberate act — a reverse proxy in front
// of it, or an explicit BIND_ADDRESS.
//
// This is the binary's default only. The Dockerfile ships
// ENV BIND_ADDRESS=0.0.0.0, so a container deployment needs
// nothing set and is unaffected by this constant. The two
// differ because they answer different questions: a container's
// network namespace is already the boundary this default is
// reaching for, so binding every address inside it exposes
// nothing, and what decides exposure there is the publish flag
// (-p 127.0.0.1:8080:8080). A loopback bind inside a container
// buys no security and makes the process unreachable through
// its own published port.
//
// The split is expressed as two explicit defaults rather than
// container auto-detection, because a heuristic that guesses
// wrong opens the cleartext port exactly where nobody is
// looking.
defaultBindAddress = "127.0.0.1"
// defaultRetentionSweepInterval is how often the retention // defaultRetentionSweepInterval is how often the retention
// reaper deletes events older than each webhook's RetentionDays. // reaper deletes events older than each webhook's RetentionDays.
defaultRetentionSweepInterval = time.Hour defaultRetentionSweepInterval = time.Hour
@@ -83,12 +56,6 @@ const (
// IPv6 prefix spends on the ::ffff:0:0/96 wrapper, so a /104 // IPv6 prefix spends on the ::ffff:0:0/96 wrapper, so a /104
// covers the same addresses as an IPv4 /8. // covers the same addresses as an IPv4 /8.
mappedV4Offset = 96 mappedV4Offset = 96
// DotEnvPath is the optional file of KEY=value lines read into the
// environment at startup, relative to the process working
// directory. Exported so that documentation and tests name the
// same path the loader opens.
DotEnvPath = ".env"
) )
// ErrInvalidEnvironment is returned when WEBHOOKER_ENVIRONMENT // ErrInvalidEnvironment is returned when WEBHOOKER_ENVIRONMENT
@@ -108,19 +75,6 @@ var ErrInvalidPort = errors.New("invalid port")
// nor a bare IP address. // nor a bare IP address.
var ErrInvalidCIDR = errors.New("invalid CIDR") var ErrInvalidCIDR = errors.New("invalid CIDR")
// ErrInvalidBindAddress is returned when BIND_ADDRESS is set to
// something that is not an IP address literal.
var ErrInvalidBindAddress = errors.New("invalid bind address")
// ErrInvalidSentryDSN is returned when SENTRY_DSN is set to something
// the Sentry SDK cannot parse as a DSN.
var ErrInvalidSentryDSN = errors.New("invalid Sentry DSN")
// ErrDotEnvUnreadable is returned when the optional .env file exists
// but cannot be read or parsed. A file that is not there is not an
// error; a file that is there and broken is.
var ErrDotEnvUnreadable = errors.New("unreadable .env file")
// ErrIncompleteMetricsAuth is returned when exactly one of // ErrIncompleteMetricsAuth is returned when exactly one of
// METRICS_USERNAME and METRICS_PASSWORD carries a value. Neither // METRICS_USERNAME and METRICS_PASSWORD carries a value. Neither
// fallback is acceptable: serving /metrics on the username alone // fallback is acceptable: serving /metrics on the username alone
@@ -151,13 +105,6 @@ type Config struct {
Port int Port int
SentryDSN string SentryDSN string
// BindAddress is the IP address the plaintext HTTP listener
// binds, as an address literal. It defaults to
// defaultBindAddress and is never empty: an empty string would
// mean the wildcard to net.Listen, which is the opposite of the
// default this ships.
BindAddress string
// RetentionSweepInterval is how often the retention reaper runs. // RetentionSweepInterval is how often the retention reaper runs.
// Always positive: it becomes a time.NewTicker period. // Always positive: it becomes a time.NewTicker period.
RetentionSweepInterval time.Duration RetentionSweepInterval time.Duration
@@ -226,62 +173,12 @@ func (c *Config) MetricsAuthEnabled() bool {
return c.MetricsUsername != "" && c.MetricsPassword != "" return c.MetricsUsername != "" && c.MetricsPassword != ""
} }
// SentryEnabled reports whether error reporting is shipped to Sentry.
// It is the only answer to that question in the codebase: the SDK
// initialisation, the sentryhttp middleware registration and the
// startup log's sentryEnabled field all read this one method, so the
// log cannot report reporting as on while nothing is sending.
//
// A non-empty DSN is enough because loadFromEnv already parsed it with
// the SDK's own parser and refused to build a Config around one the
// SDK would reject, and because initialising the SDK with a DSN that
// parsed and failed anyway aborts the process rather than leaving this
// true and the client absent.
func (c *Config) SentryEnabled() bool {
return c.SentryDSN != ""
}
// envString returns the value of the named environment variable, // envString returns the value of the named environment variable,
// or an empty string if not set. // or an empty string if not set.
func envString(key string) string { func envString(key string) string {
return os.Getenv(key) return os.Getenv(key)
} }
// LoadDotEnv reads DotEnvPath into the environment when that file is
// present, and reports a file that is present but broken.
//
// It has to run before anything reads the environment, so that every
// reader agrees on what the environment holds — the DATA_DIR lock
// taken before the fx graph exists as much as loadFromEnv itself. A
// variable already set in the real environment wins: godotenv never
// overwrites one.
//
// A missing file is not an error. It is a development convenience and
// most deployments set the environment directly.
//
// Any other failure is. godotenv parses the whole file before setting
// anything, so a single malformed line applies none of it: every
// variable in the file silently reverts to its default, which defeats
// the fail-loud guarantee for all of them at once.
func LoadDotEnv() error {
return loadDotEnvFile(DotEnvPath)
}
// loadDotEnvFile is LoadDotEnv over a named file, so tests can point
// at a temporary one instead of the process working directory.
func loadDotEnvFile(path string) error {
err := godotenv.Load(path)
if err == nil || errors.Is(err, fs.ErrNotExist) {
return nil
}
return fmt.Errorf(
"%w: %s: %w; nothing in it was applied, so fix the file or "+
"remove it",
ErrDotEnvUnreadable, path, err,
)
}
// DataDir resolves DATA_DIR, applying DefaultDataDir when it is unset // DataDir resolves DATA_DIR, applying DefaultDataDir when it is unset
// or empty. It is exported so that entry points which must act on the // or empty. It is exported so that entry points which must act on the
// data directory before the fx graph exists — taking the exclusive // data directory before the fx graph exists — taking the exclusive
@@ -490,77 +387,6 @@ func envPrefixList(key string) ([]netip.Prefix, error) {
return prefixes, nil return prefixes, nil
} }
// envBindAddress returns the value of the named environment variable
// parsed as an IP address literal. An unset (or empty, or
// whitespace-only) value yields defaultValue.
//
// Only literals are accepted: no hostname is resolved, so `localhost`
// is an error rather than a DNS lookup at startup whose answer could
// be either loopback family, could change under the process, and
// could return several addresses of which only one would be bound. A
// value with a port in it (`127.0.0.1:8080`) is likewise an error —
// the port is PORT's business, and silently accepting it would bind
// something other than what was asked for.
//
// A set value that is not a literal is a hard error naming the key
// and the bad value, so startup fails loudly rather than falling back
// to a default the operator plainly did not want. A literal that is
// not an address of this host parses here and fails at listen time
// instead, which ends the process non-zero.
func envBindAddress(key, defaultValue string) (string, error) {
v := strings.TrimSpace(os.Getenv(key))
if v == "" {
return defaultValue, nil
}
addr, err := netip.ParseAddr(v)
if err != nil {
return "", fmt.Errorf(
"%w: %s: %q must be an IP address literal such as "+
"127.0.0.1, 0.0.0.0 or ::, not a hostname and not "+
"host:port: %w",
ErrInvalidBindAddress, key, v, err,
)
}
return addr.String(), nil
}
// envSentryDSN returns the value of the named environment variable
// checked as a Sentry DSN. An unset (or empty, or whitespace-only)
// value yields "", which means error reporting stays off — the common
// case, and a normal start.
//
// A set value is parsed with sentry.NewDsn, which is the call
// sentry.Init makes on the DSN it is handed, so what passes here is
// exactly what the SDK will accept later and the two cannot disagree.
// Reproducing the check by hand instead would cost this package its
// dependency on the SDK — already a module dependency, already linked
// into the binary — in exchange for a second definition of "valid DSN"
// free to drift from the one that decides.
//
// A set value that does not parse is a hard error naming the key, so
// startup fails loudly. Losing error reporting is the failure this
// variable exists to prevent, and a typo in a DSN is silent forever:
// nothing later in the process can notice that reports are going
// nowhere. The bad value is quoted because it is a URL to a public
// endpoint carrying a public key, not a secret.
func envSentryDSN(key string) (string, error) {
v := strings.TrimSpace(os.Getenv(key))
if v == "" {
return "", nil
}
_, err := sentry.NewDsn(v)
if err != nil {
return "", fmt.Errorf(
"%w: %s: %q: %w", ErrInvalidSentryDSN, key, v, err,
)
}
return v, nil
}
// resolveMetricsAuth reads the /metrics basic-auth credentials and // resolveMetricsAuth reads the /metrics basic-auth credentials and
// rejects a half-set pair, naming both variables either way. The // rejects a half-set pair, naming both variables either way. The
// error carries neither value: the password is a secret. // error carries neither value: the password is a secret.
@@ -605,27 +431,6 @@ func resolveEnvironment() (string, error) {
return environment, nil return environment, nil
} }
// resolveListener reads the two variables that describe the HTTP
// listener: which port it claims and which address it claims it on.
// They are read together because neither is meaningful alone, and
// because a validation failure in either has to abort startup before
// anything binds.
func resolveListener() (int, string, error) {
port, err := envPort("PORT", defaultPort)
if err != nil {
return 0, "", err
}
bindAddress, err := envBindAddress(
"BIND_ADDRESS", defaultBindAddress,
)
if err != nil {
return 0, "", err
}
return port, bindAddress, nil
}
// loadFromEnv builds a Config from the environment. Every value that // loadFromEnv builds a Config from the environment. Every value that
// needs parsing fails loudly when it is set but unparseable: the // needs parsing fails loudly when it is set but unparseable: the
// documented defaults apply only to variables that are unset (or // documented defaults apply only to variables that are unset (or
@@ -637,7 +442,7 @@ func loadFromEnv() (*Config, error) {
return nil, err return nil, err
} }
port, bindAddress, err := resolveListener() port, err := envPort("PORT", defaultPort)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -693,11 +498,6 @@ func loadFromEnv() (*Config, error) {
return nil, err return nil, err
} }
sentryDSN, err := envSentryDSN("SENTRY_DSN")
if err != nil {
return nil, err
}
return &Config{ return &Config{
DataDir: DataDir(), DataDir: DataDir(),
Debug: debug, Debug: debug,
@@ -706,8 +506,7 @@ func loadFromEnv() (*Config, error) {
MetricsUsername: metricsUsername, MetricsUsername: metricsUsername,
MetricsPassword: metricsPassword, MetricsPassword: metricsPassword,
Port: port, Port: port,
BindAddress: bindAddress, SentryDSN: envString("SENTRY_DSN"),
SentryDSN: sentryDSN,
RetentionSweepInterval: retentionSweepInterval, RetentionSweepInterval: retentionSweepInterval,
SessionIdleTimeout: sessionIdleTimeout, SessionIdleTimeout: sessionIdleTimeout,
ReceiverRateLimit: receiverRateLimit, ReceiverRateLimit: receiverRateLimit,
@@ -826,11 +625,6 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
log.Info("Configuration loaded", log.Info("Configuration loaded",
"environment", s.Environment, "environment", s.Environment,
"port", s.Port, "port", s.Port,
// Logged because which interfaces the cleartext listener
// answers on is not otherwise observable from inside a
// container, and it decides whether anything but the local
// host can reach the admin UI.
"bindAddress", s.BindAddress,
"debug", s.Debug, "debug", s.Debug,
"maintenanceMode", s.MaintenanceMode, "maintenanceMode", s.MaintenanceMode,
"dataDir", s.DataDir, "dataDir", s.DataDir,
@@ -842,7 +636,7 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
"receiverRateLimit", s.ReceiverRateLimit, "receiverRateLimit", s.ReceiverRateLimit,
"trustedProxies", len(s.TrustedProxies), "trustedProxies", len(s.TrustedProxies),
"allowedEgressCIDRs", len(s.AllowedEgressCIDRs), "allowedEgressCIDRs", len(s.AllowedEgressCIDRs),
"sentryEnabled", s.SentryEnabled(), "hasSentryDSN", s.SentryDSN != "",
"hasMetricsAuth", s.MetricsAuthEnabled(), "hasMetricsAuth", s.MetricsAuthEnabled(),
) )

158
internal/config/dotenv_test.go
View File

@@ -1,158 +0,0 @@
package config_test
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/config"
)
// dotEnvKey is a throwaway variable name the .env tests write and
// read, so they cannot disturb real configuration.
const dotEnvKey = "WEBHOOKER_TEST_DOTENV_VALUE"
// malformedDotEnv is a file godotenv cannot parse. The first line is
// the realistic typo — a space where the `=` belongs — and the rest
// make sure nothing downstream treats the file as salvageable line by
// line.
const malformedDotEnv = "PORT 19615\n" +
"this is not = valid ! syntax\n" +
"\"unclosed\n"
// unsetDotEnvKey makes dotEnvKey genuinely absent for the duration of
// the test and restores it afterwards. t.Setenv registers the restore;
// the Unsetenv that follows is what the test actually needs, because a
// variable set to the empty string is still present in os.Environ and
// godotenv would refuse to overwrite it.
func unsetDotEnvKey(t *testing.T) {
t.Helper()
t.Setenv(dotEnvKey, "placeholder")
require.NoError(t, os.Unsetenv(dotEnvKey))
}
// writeDotEnv writes contents to a .env file in a fresh temporary
// directory and returns its path.
func writeDotEnv(t *testing.T, contents string) string {
t.Helper()
path := filepath.Join(t.TempDir(), config.DotEnvPath)
require.NoError(t, os.WriteFile(path, []byte(contents), 0o600))
return path
}
// TestLoadDotEnv_MissingFileIsFine pins the case most deployments are
// in. The file is optional: it is a development convenience, and a
// deployment that configures the environment directly must start
// normally rather than be refused for a file it was never meant to
// have.
//
//nolint:paralleltest // unsetDotEnvKey uses t.Setenv.
func TestLoadDotEnv_MissingFileIsFine(t *testing.T) {
unsetDotEnvKey(t)
absent := filepath.Join(t.TempDir(), config.DotEnvPath)
require.NoError(t, config.LoadDotEnvFileForTest(absent))
_, present := os.LookupEnv(dotEnvKey)
assert.False(t, present, "nothing may be set from an absent file")
}
// TestLoadDotEnv_AppliesValues pins that a well-formed file still
// reaches the environment, which is the whole reason the file is read
// at all.
//
//nolint:paralleltest // unsetDotEnvKey uses t.Setenv.
func TestLoadDotEnv_AppliesValues(t *testing.T) {
unsetDotEnvKey(t)
path := writeDotEnv(t, "# a comment\n"+dotEnvKey+"=from-dot-env\n")
require.NoError(t, config.LoadDotEnvFileForTest(path))
assert.Equal(t, "from-dot-env", os.Getenv(dotEnvKey))
}
// TestLoadDotEnv_RealEnvironmentWins pins that the file cannot
// override a variable the process was actually started with. A
// deployment that sets DATA_DIR in its unit file must not have it
// silently replaced by a stale .env left in the working directory.
func TestLoadDotEnv_RealEnvironmentWins(t *testing.T) {
t.Setenv(dotEnvKey, "from-environment")
path := writeDotEnv(t, dotEnvKey+"=from-dot-env\n")
require.NoError(t, config.LoadDotEnvFileForTest(path))
assert.Equal(t, "from-environment", os.Getenv(dotEnvKey))
}
// TestLoadDotEnv_MalformedFileAborts is the defect this fixes. One bad
// line makes godotenv apply none of the file, so every variable in it
// reverts to its default; the process used to start that way with no
// log line naming the file at all.
//
//nolint:paralleltest // unsetDotEnvKey uses t.Setenv.
func TestLoadDotEnv_MalformedFileAborts(t *testing.T) {
unsetDotEnvKey(t)
path := writeDotEnv(
t, malformedDotEnv+dotEnvKey+"=from-dot-env\n",
)
err := config.LoadDotEnvFileForTest(path)
require.Error(t, err)
require.ErrorIs(t, err, config.ErrDotEnvUnreadable)
assert.Contains(
t, err.Error(), config.DotEnvPath,
"the failure must name the file it could not read",
)
_, present := os.LookupEnv(dotEnvKey)
assert.False(
t, present,
"a rejected file must apply nothing, not part of itself",
)
}
// TestLoadDotEnv_UnreadableFileAborts pins that only absence is
// tolerated. A .env that exists but cannot be read is a file the
// operator meant to be applied, so it fails like a malformed one
// rather than being treated as though it were not there.
func TestLoadDotEnv_UnreadableFileAborts(t *testing.T) {
t.Parallel()
// A directory in the file's place: open succeeds and the read
// fails, which no umask or root-ness can turn back into success
// the way a chmod could.
path := filepath.Join(t.TempDir(), config.DotEnvPath)
require.NoError(t, os.Mkdir(path, 0o750))
err := config.LoadDotEnvFileForTest(path)
require.Error(t, err)
require.ErrorIs(t, err, config.ErrDotEnvUnreadable)
}
// TestLoadDotEnv_ReadsTheWorkingDirectory pins the path LoadDotEnv
// itself opens, which the tests above bypass. It is relative to the
// process working directory, as it was under godotenv/autoload and as
// the README documents.
//
//nolint:paralleltest // t.Chdir moves the whole process.
func TestLoadDotEnv_ReadsTheWorkingDirectory(t *testing.T) {
unsetDotEnvKey(t)
dir := t.TempDir()
require.NoError(t, os.WriteFile(
filepath.Join(dir, config.DotEnvPath),
[]byte(dotEnvKey+"=from-working-directory\n"),
0o600,
))
t.Chdir(dir)
require.NoError(t, config.LoadDotEnv())
assert.Equal(t, "from-working-directory", os.Getenv(dotEnvKey))
}

383
internal/config/env_test.go
View File

@@ -21,22 +21,6 @@ const (
envKeyPort = "PORT" envKeyPort = "PORT"
envKeyDebug = "DEBUG" envKeyDebug = "DEBUG"
envKeyMaintenanceMode = "MAINTENANCE_MODE" envKeyMaintenanceMode = "MAINTENANCE_MODE"
envKeyBindAddress = "BIND_ADDRESS"
)
// Sample BIND_ADDRESS values used by the tables below.
const (
// bindAddressDefault is the shipped default. It is asserted
// against the package's own constant in
// TestNewUsesDefaultsWhenUnset, so the two cannot drift.
bindAddressDefault = "127.0.0.1"
// bindAddressWildcard is the value a container deployment sets.
bindAddressWildcard = "0.0.0.0"
// bindAddressSample is an arbitrary specific address, standing
// for "one interface of several".
bindAddressSample = "10.1.2.3"
) )
// envBoolCase is one row of the envBool table. // envBoolCase is one row of the envBool table.
@@ -307,160 +291,6 @@ func TestEnvPort(t *testing.T) {
} }
} }
// TestEnvBindAddress covers BIND_ADDRESS parsing.
//
// Only IP address literals are accepted. Every rejection below is a
// value an operator plausibly writes — a hostname, a host:port, a
// CIDR block — and each has to abort startup rather than fall back to
// the default, because falling back would bind an address other than
// the one asked for and, in the wildcard-default case this setting
// exists to end, publish cleartext on every interface.
func TestEnvBindAddress(t *testing.T) {
for _, tt := range envBindAddressCases() {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
if tt.set {
t.Setenv(testEnvKey, tt.value)
} else {
require.NoError(t, os.Unsetenv(testEnvKey))
}
got, err := config.EnvBindAddressForTest(
testEnvKey, bindAddressDefault,
)
if tt.expectError {
require.Error(t, err)
require.ErrorIs(t, err, config.ErrInvalidBindAddress)
assert.Contains(t, err.Error(), testEnvKey)
assert.Contains(t, err.Error(), tt.value)
return
}
require.NoError(t, err)
assert.Equal(t, tt.expected, got)
})
}
}
// envBindAddressCase is one row of the envBindAddress table.
type envBindAddressCase struct {
name string
set bool
value string
expectError bool
expected string
}
// envBindAddressCases is the envBindAddress table, kept out of the
// test body so the test itself stays readable.
func envBindAddressCases() []envBindAddressCase {
return append(
envBindAddressAcceptedCases(),
envBindAddressRejectedCases()...,
)
}
// envBindAddressAcceptedCases are the values that parse: the three
// spellings of "unset" that take the default, and the literals.
func envBindAddressAcceptedCases() []envBindAddressCase {
return []envBindAddressCase{
{
name: "unset returns the default",
expected: bindAddressDefault,
},
{
name: "empty returns the default",
set: true,
value: "",
expected: bindAddressDefault,
},
{
name: "whitespace returns the default",
set: true,
value: " ",
expected: bindAddressDefault,
},
{
name: "ipv4 wildcard is parsed",
set: true,
value: bindAddressWildcard,
expected: bindAddressWildcard,
},
{
name: "ipv4 literal is parsed",
set: true,
value: bindAddressSample,
expected: bindAddressSample,
},
{
name: "surrounding whitespace is trimmed",
set: true,
value: " " + bindAddressSample + " ",
expected: bindAddressSample,
},
{
name: "ipv6 wildcard is parsed",
set: true,
value: "::",
expected: "::",
},
{
name: "ipv6 literal is parsed",
set: true,
value: "2001:db8::5",
expected: "2001:db8::5",
},
}
}
// envBindAddressRejectedCases are the values that abort startup.
// Each is something an operator plausibly writes, and none may fall
// back to the default: the default is loopback, so a silent fallback
// would bind somewhere other than what was asked for.
func envBindAddressRejectedCases() []envBindAddressCase {
return []envBindAddressCase{
{
name: "garbage is rejected",
set: true,
value: "not-an-address",
expectError: true,
},
{
name: "hostname is rejected",
set: true,
value: "localhost",
expectError: true,
},
{
name: "unresolvable hostname is rejected",
set: true,
value: "no-such-host.invalid",
expectError: true,
},
{
name: "host and port is rejected",
set: true,
value: bindAddressDefault + ":8080",
expectError: true,
},
{
name: "bracketed ipv6 is rejected",
set: true,
value: "[::1]",
expectError: true,
},
{
name: "CIDR block is rejected",
set: true,
value: "10.0.0.0/8",
expectError: true,
},
}
}
// buildConfig constructs a Config through fx exactly as the // buildConfig constructs a Config through fx exactly as the
// application does, returning the config and any construction error. // application does, returning the config and any construction error.
func buildConfig(t *testing.T) (*config.Config, error) { func buildConfig(t *testing.T) (*config.Config, error) {
@@ -482,7 +312,58 @@ func buildConfig(t *testing.T) (*config.Config, error) {
} }
func TestNewRejectsBadEnvValues(t *testing.T) { func TestNewRejectsBadEnvValues(t *testing.T) {
for _, tt := range badEnvValueCases() { tests := []struct {
name string
key string
value string
expectError bool
check func(t *testing.T, cfg *config.Config)
}{
{
name: "valid PORT is used",
key: envKeyPort,
value: "9001",
check: func(t *testing.T, cfg *config.Config) {
t.Helper()
assert.Equal(t, 9001, cfg.Port)
},
},
{
name: "unparseable PORT aborts startup",
key: envKeyPort,
value: "eighty-eighty",
expectError: true,
},
{
name: "out-of-range PORT aborts startup",
key: envKeyPort,
value: "70000",
expectError: true,
},
{
name: "valid DEBUG is used",
key: envKeyDebug,
value: "true",
check: func(t *testing.T, cfg *config.Config) {
t.Helper()
assert.True(t, cfg.Debug)
},
},
{
name: "unparseable DEBUG aborts startup",
key: envKeyDebug,
value: "ture",
expectError: true,
},
{
name: "unparseable MAINTENANCE_MODE aborts startup",
key: envKeyMaintenanceMode,
value: "sometimes",
expectError: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv // Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests. // is incompatible with parallel subtests.
@@ -506,149 +387,6 @@ func TestNewRejectsBadEnvValues(t *testing.T) {
} }
} }
// badEnvValueCase is one row of the config.New table: a variable, the
// value it is set to, and either the assertion that startup fails
// naming both, or a check on the Config that resulted.
type badEnvValueCase struct {
name string
key string
value string
expectError bool
check func(t *testing.T, cfg *config.Config)
}
// badEnvValueCases is the config.New table, kept out of the test body
// so the test itself stays readable. It is assembled from per-variable
// groups because one literal covering every variable outgrew the
// function-length budget.
func badEnvValueCases() []badEnvValueCase {
cases := listenerEnvValueCases()
cases = append(cases, flagEnvValueCases()...)
cases = append(cases, sentryEnvValueCases()...)
return cases
}
// listenerEnvValueCases covers the two variables that describe the
// HTTP listener.
func listenerEnvValueCases() []badEnvValueCase {
return []badEnvValueCase{
{
name: "valid PORT is used",
key: envKeyPort,
value: "9001",
check: func(t *testing.T, cfg *config.Config) {
t.Helper()
assert.Equal(t, 9001, cfg.Port)
},
},
{
name: "unparseable PORT aborts startup",
key: envKeyPort,
value: "eighty-eighty",
expectError: true,
},
{
name: "out-of-range PORT aborts startup",
key: envKeyPort,
value: "70000",
expectError: true,
},
{
name: "valid BIND_ADDRESS is used",
key: envKeyBindAddress,
value: bindAddressWildcard,
check: func(t *testing.T, cfg *config.Config) {
t.Helper()
assert.Equal(
t, bindAddressWildcard, cfg.BindAddress,
)
},
},
{
name: "unparseable BIND_ADDRESS aborts startup",
key: envKeyBindAddress,
value: "not-an-address",
expectError: true,
},
{
name: "hostname BIND_ADDRESS aborts startup",
key: envKeyBindAddress,
value: "localhost",
expectError: true,
},
{
name: "BIND_ADDRESS with a port aborts startup",
key: envKeyBindAddress,
value: bindAddressDefault + ":8080",
expectError: true,
},
}
}
// flagEnvValueCases covers the boolean variables.
func flagEnvValueCases() []badEnvValueCase {
return []badEnvValueCase{
{
name: "valid DEBUG is used",
key: envKeyDebug,
value: "true",
check: func(t *testing.T, cfg *config.Config) {
t.Helper()
assert.True(t, cfg.Debug)
},
},
{
name: "unparseable DEBUG aborts startup",
key: envKeyDebug,
value: "ture",
expectError: true,
},
{
name: "unparseable MAINTENANCE_MODE aborts startup",
key: envKeyMaintenanceMode,
value: "sometimes",
expectError: true,
},
}
}
// sentryEnvValueCases covers SENTRY_DSN. The three rejected values are
// the ones measured on the defect: each initialised the SDK with an
// error and left the process serving with error reporting off.
func sentryEnvValueCases() []badEnvValueCase {
return []badEnvValueCase{
{
name: "valid SENTRY_DSN is used",
key: envKeySentryDSN,
value: validSentryDSN,
check: func(t *testing.T, cfg *config.Config) {
t.Helper()
assert.Equal(t, validSentryDSN, cfg.SentryDSN)
assert.True(t, cfg.SentryEnabled())
},
},
{
name: "unparseable SENTRY_DSN aborts startup",
key: envKeySentryDSN,
value: "not-a-dsn",
expectError: true,
},
{
name: "SENTRY_DSN that is not a URL aborts startup",
key: envKeySentryDSN,
value: "%%%",
expectError: true,
},
{
name: "keyless SENTRY_DSN aborts startup",
key: envKeySentryDSN,
value: "https://example.invalid/1",
expectError: true,
},
}
}
// TestNewUsesDefaultsWhenUnset proves the fail-loud behaviour did not // TestNewUsesDefaultsWhenUnset proves the fail-loud behaviour did not
// break the legitimate unset case: absent variables still get their // break the legitimate unset case: absent variables still get their
// documented defaults. // documented defaults.
@@ -657,7 +395,6 @@ func TestNewUsesDefaultsWhenUnset(t *testing.T) {
for _, key := range []string{ for _, key := range []string{
envKeyPort, envKeyDebug, envKeyMaintenanceMode, envKeyPort, envKeyDebug, envKeyMaintenanceMode,
envKeyBindAddress, envKeySentryDSN,
} { } {
require.NoError(t, os.Unsetenv(key)) require.NoError(t, os.Unsetenv(key))
} }
@@ -669,20 +406,4 @@ func TestNewUsesDefaultsWhenUnset(t *testing.T) {
assert.Equal(t, 8080, cfg.Port) assert.Equal(t, 8080, cfg.Port)
assert.False(t, cfg.Debug) assert.False(t, cfg.Debug)
assert.False(t, cfg.MaintenanceMode) assert.False(t, cfg.MaintenanceMode)
// Loopback, not the wildcard: the default must not publish the
// cleartext admin UI and the unauthenticated receiver on every
// interface of a host that configured nothing. The value is read
// from the package rather than repeated, so the README's
// documented default and the compiled-in one are pinned to the
// same constant.
assert.Equal(
t, config.DefaultBindAddressForTest, cfg.BindAddress,
)
assert.Equal(t, bindAddressDefault, cfg.BindAddress)
// An absent SENTRY_DSN is the common case and must stay a normal
// start with error reporting off, not a refusal.
assert.Empty(t, cfg.SentryDSN)
assert.False(t, cfg.SentryEnabled())
} }

22
internal/config/export_test.go
View File

@@ -50,25 +50,3 @@ func EnvPositiveIntForTest(key string, defaultValue int) (int, error) {
func EnvPortForTest(key string, defaultValue int) (int, error) { func EnvPortForTest(key string, defaultValue int) (int, error) {
return envPort(key, defaultValue) return envPort(key, defaultValue)
} }
// EnvSentryDSNForTest exposes envSentryDSN.
func EnvSentryDSNForTest(key string) (string, error) {
return envSentryDSN(key)
}
// LoadDotEnvFileForTest exposes the loader LoadDotEnv runs, over a
// caller-named file rather than the process working directory, so
// each .env state can be covered without moving the test process.
func LoadDotEnvFileForTest(path string) error {
return loadDotEnvFile(path)
}
// EnvBindAddressForTest exposes envBindAddress.
func EnvBindAddressForTest(key, defaultValue string) (string, error) {
return envBindAddress(key, defaultValue)
}
// DefaultBindAddressForTest exposes the compiled-in BIND_ADDRESS
// default, so a test pins the documented value rather than repeating
// a literal that could drift from it.
const DefaultBindAddressForTest = defaultBindAddress

141
internal/config/sentry_test.go
View File

@@ -1,141 +0,0 @@
package config_test
import (
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/config"
)
// envKeySentryDSN is the variable envSentryDSN reads in production.
const envKeySentryDSN = "SENTRY_DSN"
// validSentryDSN is a syntactically complete DSN. The host is under
// .invalid (RFC 2606), so nothing a test builds around it can reach a
// real Sentry installation.
const validSentryDSN = "https://abc123@sentry.invalid/42"
// envSentryDSNCase is one row of the envSentryDSN table.
type envSentryDSNCase struct {
name string
set bool
value string
expectError bool
expected string
}
// envSentryDSNCases is the envSentryDSN table. The three invalid
// values are the ones measured on the defect: each initialised the SDK
// with an error and left the process serving with reporting off.
func envSentryDSNCases() []envSentryDSNCase {
return []envSentryDSNCase{
{
name: "unset means reporting off",
expected: "",
},
{
name: "empty means reporting off",
set: true,
value: "",
expected: "",
},
{
name: "whitespace means reporting off",
set: true,
value: " ",
expected: "",
},
{
name: "a valid DSN is kept",
set: true,
value: validSentryDSN,
expected: validSentryDSN,
},
{
name: "surrounding whitespace is trimmed",
set: true,
value: " " + validSentryDSN + "\t",
expected: validSentryDSN,
},
{
name: "a value that is not a URL is rejected",
set: true,
value: "not-a-dsn",
expectError: true,
},
{
name: "an unparseable URL is rejected",
set: true,
value: "%%%",
expectError: true,
},
{
name: "a DSN without a public key is rejected",
set: true,
value: "https://example.invalid/1",
expectError: true,
},
{
name: "a DSN without a project id is rejected",
set: true,
value: "https://abc123@sentry.invalid/",
expectError: true,
},
{
name: "a non-HTTP scheme is rejected",
set: true,
value: "ftp://abc123@sentry.invalid/42",
expectError: true,
},
}
}
// TestEnvSentryDSN covers the helper directly. What it pins beyond the
// value is the failure shape: a set-but-unparseable DSN names the
// variable and the value, exactly as the other fail-loud helpers do,
// so an operator reads the fix off the message.
func TestEnvSentryDSN(t *testing.T) {
for _, tt := range envSentryDSNCases() {
t.Run(tt.name, func(t *testing.T) {
// Cannot use t.Parallel() here because t.Setenv
// is incompatible with parallel subtests.
if tt.set {
t.Setenv(envKeySentryDSN, tt.value)
} else {
require.NoError(t, os.Unsetenv(envKeySentryDSN))
}
got, err := config.EnvSentryDSNForTest(envKeySentryDSN)
if tt.expectError {
require.Error(t, err)
require.ErrorIs(t, err, config.ErrInvalidSentryDSN)
assert.Contains(t, err.Error(), envKeySentryDSN)
assert.Contains(t, err.Error(), tt.value)
assert.Empty(t, got)
return
}
require.NoError(t, err)
assert.Equal(t, tt.expected, got)
})
}
}
// TestSentryEnabled_TracksTheDSN pins that the one method answering
// "is anything being reported" agrees with the DSN in every state. The
// startup log, the SDK initialisation and the sentryhttp middleware
// all read it, so a log field cannot report reporting as on while
// nothing is sending.
func TestSentryEnabled_TracksTheDSN(t *testing.T) {
t.Parallel()
assert.False(t, (&config.Config{}).SentryEnabled())
assert.True(
t,
(&config.Config{SentryDSN: validSentryDSN}).SentryEnabled(),
)
}

85
internal/database/migration_entrypoint_test.go Normal file
View File

@@ -0,0 +1,85 @@
package database_test
import (
"net/http"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/signature"
)
// TestEntrypointSignatureColumnsMigrateToUnconfigured pins the
// upgrade path for a deployment that already has entrypoints.
//
// The signature columns arrive through GORM's AutoMigrate, so every
// row written before they existed acquires them with no value. That
// has to land on "not configured", because the alternative is an
// upgrade that rejects the traffic the operator was already
// receiving — a self-inflicted outage on a receiver whose senders
// cannot be told to start signing.
//
// The legacy schema is reproduced by dropping the columns from a
// migrated database and writing a row through the old shape, so the
// row really predates them rather than merely being blank.
func TestEntrypointSignatureColumnsMigrateToUnconfigured(t *testing.T) {
t.Parallel()
db, lc := setupTestDB(t)
lc.RequireStart()
t.Cleanup(lc.RequireStop)
for _, column := range []string{
"signature_scheme", "signature_secret",
} {
require.NoError(
t,
db.DB().Exec(
"ALTER TABLE entrypoints DROP COLUMN "+column,
).Error,
"dropping %s to reproduce the pre-upgrade schema",
column,
)
}
const legacyID = "legacy-entrypoint"
require.NoError(
t,
db.DB().Exec(
`INSERT INTO entrypoints
(id, created_at, updated_at, webhook_id, path,
description, active)
VALUES (?, ?, ?, ?, ?, ?, ?)`,
legacyID, "2026-01-01 00:00:00", "2026-01-01 00:00:00",
"legacy-webhook", "legacy-path", "predates signatures",
true,
).Error,
)
// The upgrade.
require.NoError(t, db.Migrate())
var ep database.Entrypoint
require.NoError(
t,
db.DB().Where("id = ?", legacyID).First(&ep).Error,
"the migrated row must still load; a NULL landing in a "+
"string column would fail here",
)
assert.Equal(t, database.SignatureSchemeNone, ep.SignatureScheme)
assert.Empty(t, ep.SignatureSecret)
assert.False(t, ep.SignatureConfigured())
assert.True(t, ep.Active, "the row's other columns survive")
// The behaviour that actually matters: an unsigned request to
// this entrypoint is still accepted.
assert.NoError(
t,
signature.Verify(&ep, http.Header{}, []byte(`{"a":1}`)),
)
}

59
internal/database/model_entrypoint.go
View File

@@ -1,20 +1,71 @@
package database package database
// SignatureScheme names the way an entrypoint authenticates inbound
// requests. A scheme fixes both the header the signature arrives in
// and the algorithm used to check it, so an operator cannot pair one
// sender's header with another sender's comparison.
type SignatureScheme string
// Signature scheme values. The empty scheme means the entrypoint
// performs no inbound verification: it is the default, and it is the
// state every entrypoint created before this column existed migrates
// to, so an existing deployment keeps accepting the requests it
// accepted before.
const (
SignatureSchemeNone SignatureScheme = ""
SignatureSchemeGitHub SignatureScheme = "github"
SignatureSchemeGitLab SignatureScheme = "gitlab"
)
// Entrypoint represents an inbound URL endpoint that feeds into a webhook // Entrypoint represents an inbound URL endpoint that feeds into a webhook
type Entrypoint struct { type Entrypoint struct {
BaseModel BaseModel
WebhookID string `gorm:"type:uuid;not null" json:"webhookId"` WebhookID string `gorm:"type:uuid;not null" json:"webhookId"`
// Path is the URL path for this entrypoint. It is the // Path is the URL path for this entrypoint.
// entrypoint's only credential: possession of the UUID
// authorises submission, so the receiver checks nothing else
// about the sender.
Path string `gorm:"uniqueIndex;not null" json:"path"` Path string `gorm:"uniqueIndex;not null" json:"path"`
Description string `json:"description"` Description string `json:"description"`
Active bool `gorm:"default:true" json:"active"` Active bool `gorm:"default:true" json:"active"`
// SignatureScheme selects how inbound requests to this
// entrypoint are authenticated. Empty means unauthenticated,
// which is what a UUID-only entrypoint has always been.
SignatureScheme SignatureScheme `gorm:"default:''" json:"signatureScheme"`
// SignatureSecret is the secret shared with the sender.
//
// It is stored in the clear because HMAC verification needs the
// key itself: a hash of it cannot recompute the sender's digest.
// It is therefore a live credential, and json:"-" keeps it out of
// any handler that marshals the model, the way APIKey.Key and
// Target.Config are kept out. handlers.EntrypointView is the
// matching barrier for the HTML path.
SignatureSecret string `gorm:"default:''" json:"-"`
// Relations // Relations
Webhook Webhook `json:"webhook,omitzero"` Webhook Webhook `json:"webhook,omitzero"`
} }
// SignatureConfigured reports whether this entrypoint verifies
// inbound requests. Both halves must be present: a scheme without a
// secret, or a secret without a scheme, is a broken configuration
// rather than a configured one, and signature.Verify fails those
// closed rather than treating them as "off".
func (e *Entrypoint) SignatureConfigured() bool {
return e.SignatureScheme != SignatureSchemeNone &&
e.SignatureSecret != ""
}
// SignatureHalfConfigured reports whether exactly one half of the
// scheme/secret pair is present. The receiver refuses such a row on
// every request, so the UI must not describe it as unverified. It
// reports the state without exposing the secret, which is why it
// lives here rather than in the display projection.
func (e *Entrypoint) SignatureHalfConfigured() bool {
hasScheme := e.SignatureScheme != SignatureSchemeNone
hasSecret := e.SignatureSecret != ""
return hasScheme != hasSecret
}

31
internal/database/model_secrets_test.go
View File

@@ -34,6 +34,8 @@ func marshalModel(t *testing.T, v any) string {
// - APIKey.Key is a bearer token outright. // - APIKey.Key is a bearer token outright.
// - Setting.Value holds the session encryption key. // - Setting.Value holds the session encryption key.
// - User.Password holds the Argon2 hash, and was already tagged. // - User.Password holds the Argon2 hash, and was already tagged.
// - Entrypoint.SignatureSecret is the secret its senders sign with,
// stored in the clear because HMAC verification needs the key.
func TestModelsDoNotMarshalTheirSecrets(t *testing.T) { func TestModelsDoNotMarshalTheirSecrets(t *testing.T) {
t.Parallel() t.Parallel()
@@ -72,6 +74,14 @@ func TestModelsDoNotMarshalTheirSecrets(t *testing.T) {
Password: marker, Password: marker,
}, },
}, },
{
name: "entrypoint signature secret",
model: database.Entrypoint{
Description: keptField,
SignatureScheme: database.SignatureSchemeGitHub,
SignatureSecret: marker,
},
},
} }
for _, tc := range cases { for _, tc := range cases {
@@ -105,3 +115,24 @@ func TestWebhookMarshalsNoTargetConfig(t *testing.T) {
assert.NotContains(t, encoded, marker) assert.NotContains(t, encoded, marker)
assert.Contains(t, encoded, keptField) assert.Contains(t, encoded, keptField)
} }
// TestWebhookMarshalsNoEntrypointSecret covers the same nested case
// for the entrypoint's inbound signature secret, which reaches a
// marshalled webhook through the Entrypoints association.
func TestWebhookMarshalsNoEntrypointSecret(t *testing.T) {
t.Parallel()
const marker = "QQENTRYPOINTMARKERQQ"
encoded := marshalModel(t, database.Webhook{
Name: keptField,
Entrypoints: []database.Entrypoint{{
Path: "some-uuid",
SignatureScheme: database.SignatureSchemeGitLab,
SignatureSecret: marker,
}},
})
assert.NotContains(t, encoded, marker)
assert.Contains(t, encoded, keptField)
}

240
internal/database/sqlite_mode_test.go
View File

@@ -1,240 +0,0 @@
package database_test
import (
"context"
"io/fs"
"net/http"
"os"
"path/filepath"
"testing"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/fx/fxtest"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/globals"
"sneak.berlin/go/webhooker/internal/logger"
)
// ownerOnly is the mode every SQLite file the service owns must have.
// Spelled out rather than referencing database.SQLiteFilePerm so the
// test fails if the constant itself is loosened.
const ownerOnly fs.FileMode = 0o600
// requireOwnerOnly asserts that path exists and is readable and
// writable by its owner and by nobody else.
func requireOwnerOnly(t *testing.T, path string) {
t.Helper()
info, err := os.Stat(path)
require.NoError(t, err, "%s must exist", path)
assert.Equal(
t,
ownerOnly,
info.Mode().Perm(),
"%s holds credentials and must not be readable by "+
"anyone but its owner",
path,
)
}
// requireDatabaseSetOwnerOnly asserts the mode of a database file and
// of both WAL sidecars. The sidecars carry the same rows as the
// database, so tightening only the main file fixes nothing.
func requireDatabaseSetOwnerOnly(t *testing.T, dbPath string) {
t.Helper()
requireOwnerOnly(t, dbPath)
requireOwnerOnly(t, dbPath+"-wal")
requireOwnerOnly(t, dbPath+"-shm")
}
// TestMainDatabaseFilesAreOwnerOnly covers the tier the defect was
// reported against: webhooker.db holds targets.config in plaintext —
// bearer tokens, API keys, Slack webhook URLs — and the session
// encryption key.
func TestMainDatabaseFilesAreOwnerOnly(t *testing.T) {
t.Parallel()
lc := fxtest.NewLifecycle(t)
l, err := logger.New(lc, logger.LoggerParams{
Globals: &globals.Globals{
Appname: testAppname,
Version: testVersion,
},
})
require.NoError(t, err)
// A directory the application creates itself, not one t.TempDir
// made at 0700, so the mode below is the application's.
dataDir := filepath.Join(t.TempDir(), "data")
db, err := database.New(lc, database.DatabaseParams{
Config: &config.Config{DataDir: dataDir},
Logger: l,
})
require.NoError(t, err)
ctx := context.Background()
require.NoError(t, lc.Start(ctx))
defer func() { require.NoError(t, lc.Stop(ctx)) }()
// Write through the real model so the WAL is populated and both
// sidecars are on disk while the handle is open.
require.NoError(t, db.DB().Create(&database.Webhook{
Name: testWebhookName,
}).Error)
requireDatabaseSetOwnerOnly(
t, filepath.Join(dataDir, database.MainDBFileName),
)
// The data directory grants nothing to `other`. Asserted as a
// property rather than as an exact 0750, because MkdirAll applies
// the ambient umask: the exact mode is the developer's umask as
// much as the application's request, and pinning it would make
// `make check` pass or fail on where it is run. The group bits are
// deliberately left unasserted — deployments may rely on them.
info, err := os.Stat(dataDir)
require.NoError(t, err)
assert.Zero(
t,
info.Mode().Perm()&0o007,
"the data directory must not be world-accessible",
)
}
// TestPerWebhookEventDatabaseFilesAreOwnerOnly covers the events-*.db
// tier. These carry no credential canaries since
// https://git.eeqj.de/sneak/webhooker/issues/206, but they hold every
// received request body and header.
func TestPerWebhookEventDatabaseFilesAreOwnerOnly(t *testing.T) {
t.Parallel()
mgr, lc := setupTestWebhookDBManager(t)
ctx := context.Background()
require.NoError(t, lc.Start(ctx))
defer func() { require.NoError(t, lc.Stop(ctx)) }()
webhookID := uuid.New().String()
db, err := mgr.GetDB(webhookID)
require.NoError(t, err)
require.NoError(t, db.Create(&database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Body: "{}",
}).Error)
requireDatabaseSetOwnerOnly(t, mgr.DBPath(webhookID))
}
// TestArchiveDatabaseFilesAreOwnerOnly covers the archive-*.db tier.
// internal/delivery builds that path and opens it through OpenSQLite,
// the same single open path exercised here, so the mode is settled for
// all three tiers in one place.
func TestArchiveDatabaseFilesAreOwnerOnly(t *testing.T) {
t.Parallel()
ctx := context.Background()
path := filepath.Join(
t.TempDir(), "archive-"+uuid.New().String()+".db",
)
sqlDB, err := database.OpenSQLite(path, database.SQLiteModeCreate)
require.NoError(t, err)
defer func() { require.NoError(t, sqlDB.Close()) }()
_, err = sqlDB.ExecContext(ctx, "create table t (id integer)")
require.NoError(t, err)
requireDatabaseSetOwnerOnly(t, path)
}
// TestOpenSQLiteTightensFilesLeftWorldReadable is the upgrade case: a
// data directory an earlier build left at 0644, including a
// developer's own scratch directory, is fixed when it is opened rather
// than staying exposed until it is recreated.
func TestOpenSQLiteTightensFilesLeftWorldReadable(t *testing.T) {
t.Parallel()
dir := t.TempDir()
path := filepath.Join(dir, database.MainDBFileName)
// A database and both sidecars as the pre-fix build left them.
for _, p := range []string{path, path + "-wal", path + "-shm"} {
require.NoError(t, os.WriteFile(p, nil, 0o644)) //nolint:gosec // the mode under test
}
sqlDB, err := database.OpenSQLite(path, database.SQLiteModeCreate)
require.NoError(t, err)
require.NoError(t, sqlDB.Close())
requireDatabaseSetOwnerOnly(t, path)
}
// TestOpenSQLiteExistingModeDoesNotCreateTheFile guards the mechanism
// the fix uses: OpenSQLite now creates the database file itself, and
// must not do so for a caller that asked for an existing database. An
// empty file materialized here would turn a missing-database error
// into a silently empty one.
func TestOpenSQLiteExistingModeDoesNotCreateTheFile(t *testing.T) {
t.Parallel()
ctx := context.Background()
path := filepath.Join(t.TempDir(), "absent.db")
sqlDB, err := database.OpenSQLite(path, database.SQLiteModeExisting)
if err == nil {
// sql.Open is lazy: force the connection that fails.
require.Error(t, sqlDB.PingContext(ctx))
require.NoError(t, sqlDB.Close())
}
_, statErr := os.Stat(path)
assert.ErrorIs(t, statErr, fs.ErrNotExist)
}
// TestReopenAfterRestartKeepsFilesOwnerOnly is the restart case: a
// process that closed its files must be able to open them again at
// 0600, including through a gorm handle, and the sidecars must come
// back at 0600 too rather than at SQLite's own default.
func TestReopenAfterRestartKeepsFilesOwnerOnly(t *testing.T) {
t.Parallel()
ctx := context.Background()
dir := t.TempDir()
path := filepath.Join(dir, database.MainDBFileName)
first, err := database.OpenSQLite(path, database.SQLiteModeCreate)
require.NoError(t, err)
_, err = first.ExecContext(ctx, "create table t (id integer)")
require.NoError(t, err)
require.NoError(t, first.Close())
second, err := database.OpenSQLite(path, database.SQLiteModeCreate)
require.NoError(t, err)
defer func() { require.NoError(t, second.Close()) }()
_, err = second.ExecContext(ctx, "insert into t (id) values (1)")
require.NoError(t, err)
requireDatabaseSetOwnerOnly(t, path)
var got int
require.NoError(t,
second.QueryRowContext(ctx, "select id from t").Scan(&got))
assert.Equal(t, 1, got)
}

107
internal/database/sqlite_open.go
View File

@@ -2,11 +2,8 @@ package database
import ( import (
"database/sql" "database/sql"
"errors"
"fmt" "fmt"
"io/fs"
"net/url" "net/url"
"os"
"time" "time"
_ "modernc.org/sqlite" // Pure Go SQLite driver _ "modernc.org/sqlite" // Pure Go SQLite driver
@@ -75,90 +72,6 @@ const (
sqliteConnMaxIdleTime = time.Minute sqliteConnMaxIdleTime = time.Minute
) )
// SQLiteFilePerm is the mode every SQLite file this service owns is
// created with and held at: owner read/write, nothing for group or
// other.
//
// These files hold credentials in plaintext. The main database stores
// `targets.config` — bearer tokens, API keys, Slack webhook URLs — and
// the session encryption key. SQLite left to itself creates them 0644
// (see reserveSQLiteFile), which made the 0750 data directory the only
// barrier; a bind-mounted directory supplied at 0755 removes it and
// every local user on the host can read every stored credential.
//
// This is a file-mode fix and not encryption at rest. An unattended
// process needs a key it can read without a human, so the key lands
// beside the data and an attacker who can read the database can read
// it too. See https://git.eeqj.de/sneak/webhooker/issues/212.
const SQLiteFilePerm fs.FileMode = 0o600
// reserveSQLiteFile puts path at SQLiteFilePerm before the driver ever
// touches it, and tightens any sidecar already on disk.
//
// The mode has to be settled here rather than by a chmod after opening,
// because SQLite picks it: robust_open substitutes
// SQLITE_DEFAULT_FILE_PERMISSIONS (0644) whenever it is handed mode 0,
// and findCreateFileMode yields 0 for a main database opened by URI
// with no `modeof` parameter. A chmod afterwards would leave a window
// in which the credentials are on disk world-readable.
//
// Creating the file ourselves also settles the sidecars, which is the
// half that could quietly not work. SQLite does not create those at a
// mode we choose — it derives both from the main database file:
// `-wal` through findCreateFileMode, which stats the path with the
// suffix stripped, and `-shm` in unixOpenSharedMemory from an fstat of
// the already-open database descriptor. A main file at 0600 therefore
// produces sidecars at 0600. A zero-length file is a valid empty
// database, so reserving it changes nothing else.
//
// create says whether the caller is opening in a mode that may create
// the database. When it is false a missing file is left missing, so
// SQLite still reports the absence rather than this function
// materializing an empty database the caller asked not to create.
//
// Chmod of a file that already exists is what tightens a data
// directory an earlier build left at 0644 — including a developer's
// own scratch directory — without any migration machinery.
func reserveSQLiteFile(path string, create bool) error {
if create {
// gosec G304: the path is the database file the caller asked
// to open, and the driver is about to open the same path
// anyway. Creating it here is what fixes its mode.
f, err := os.OpenFile( //nolint:gosec // see above
path, os.O_RDWR|os.O_CREATE, SQLiteFilePerm,
)
if err != nil {
return fmt.Errorf("creating %s: %w", path, err)
}
err = f.Close()
if err != nil {
return fmt.Errorf("closing %s: %w", path, err)
}
}
// O_CREATE leaves an existing file's mode alone, and umask can only
// have narrowed a new one. Chmod settles both cases at exactly
// SQLiteFilePerm.
for _, p := range append(
[]string{path}, sqliteSidecarPaths(path)...,
) {
err := os.Chmod(p, SQLiteFilePerm)
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return fmt.Errorf("securing %s: %w", p, err)
}
}
return nil
}
// sqliteSidecarPaths returns the files SQLite maintains beside a
// database under WAL. They carry the same rows as the database itself,
// so a fix that tightens only the main file has fixed nothing.
func sqliteSidecarPaths(path string) []string {
return []string{path + "-wal", path + "-shm"}
}
// SQLiteDSN builds the connection string for one database file. // SQLiteDSN builds the connection string for one database file.
// //
// mode is the SQLite URI open mode: "rwc" to create the file when it // mode is the SQLite URI open mode: "rwc" to create the file when it
@@ -195,20 +108,11 @@ func sqliteSidecarPaths(path string) []string {
// synchronous is deliberately left at SQLite's default of FULL: this // synchronous is deliberately left at SQLite's default of FULL: this
// is a webhook receiver whose one promise is that an event it answered // is a webhook receiver whose one promise is that an event it answered
// 200 for is durable. // 200 for is durable.
// The order of the _pragma parameters is load-bearing.
// modernc.org/sqlite executes them in the order they appear, on every
// new connection, before the connection is handed to the pool. Setting
// journal_mode first means that pragma itself runs with no busy
// handler installed: the pool opens connections lazily, so the moment
// a new one is created is a moment the database is under load, and
// PRAGMA journal_mode takes a lock. It would fail immediately with
// SQLITE_BUSY and fail the query that caused the connection to be
// opened. busy_timeout is therefore set first, so every pragma after
// it — and the whole life of the connection — is covered.
func SQLiteDSN(path, mode string) string { func SQLiteDSN(path, mode string) string {
q := url.Values{} q := url.Values{}
q.Set("mode", mode) q.Set("mode", mode)
q.Set("_txlock", "immediate") q.Set("_txlock", "immediate")
q.Add("_pragma", "journal_mode(WAL)")
q.Add( q.Add(
"_pragma", "_pragma",
fmt.Sprintf( fmt.Sprintf(
@@ -216,7 +120,6 @@ func SQLiteDSN(path, mode string) string {
SQLiteBusyTimeout.Milliseconds(), SQLiteBusyTimeout.Milliseconds(),
), ),
) )
q.Add("_pragma", "journal_mode(WAL)")
return "file:" + path + "?" + q.Encode() return "file:" + path + "?" + q.Encode()
} }
@@ -225,17 +128,9 @@ func SQLiteDSN(path, mode string) string {
// durability settings and pool bounds applied. mode is the SQLite URI // durability settings and pool bounds applied. mode is the SQLite URI
// open mode ("rwc" or "rw"). // open mode ("rwc" or "rw").
// //
// The file and its WAL sidecars are settled at SQLiteFilePerm before
// the driver sees the path; see reserveSQLiteFile.
//
// The handle is returned rather than a *gorm.DB because the callers // The handle is returned rather than a *gorm.DB because the callers
// wrap it in gorm themselves with their own logger. // wrap it in gorm themselves with their own logger.
func OpenSQLite(path, mode string) (*sql.DB, error) { func OpenSQLite(path, mode string) (*sql.DB, error) {
err := reserveSQLiteFile(path, mode == SQLiteModeCreate)
if err != nil {
return nil, err
}
sqlDB, err := sql.Open("sqlite", SQLiteDSN(path, mode)) sqlDB, err := sql.Open("sqlite", SQLiteDSN(path, mode))
if err != nil { if err != nil {
return nil, fmt.Errorf( return nil, fmt.Errorf(

11
internal/database/sqlite_open_test.go
View File

@@ -40,17 +40,6 @@ func TestSQLiteDSNCarriesTheDurabilitySettings(t *testing.T) {
assert.Contains(t, dsn, "_txlock=immediate") assert.Contains(t, dsn, "_txlock=immediate")
assert.Contains(t, dsn, "mode=rwc") assert.Contains(t, dsn, "mode=rwc")
// busy_timeout must come first. The driver runs these in order on
// every new connection, and PRAGMA journal_mode takes a lock — a
// connection opened while the database is busy would fail on that
// pragma, with no busy handler yet installed to wait it out.
assert.Less(
t,
strings.Index(dsn, "busy_timeout"),
strings.Index(dsn, "journal_mode"),
"busy_timeout must be applied before journal_mode",
)
// cache=shared turns an in-process conflict into SQLITE_LOCKED, // cache=shared turns an in-process conflict into SQLITE_LOCKED,
// which the busy handler does not retry. It must never come back. // which the busy handler does not retry. It must never come back.
// See https://git.eeqj.de/sneak/webhooker/issues/256. // See https://git.eeqj.de/sneak/webhooker/issues/256.

681
internal/delivery/engine.go
View File

@@ -4,7 +4,6 @@ package delivery
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"log/slog" "log/slog"
"net/http" "net/http"
@@ -42,23 +41,18 @@ const (
// sweep runs. // sweep runs.
retrySweepInterval = 60 * time.Second retrySweepInterval = 60 * time.Second
// pendingSweepMinAge is how long a delivery must have sat // pendingSweepMinAge is how long a delivery must have sat at
// untouched at pending before the sweep will look at it. // pending before the sweep treats it as stranded rather than as
// in flight.
// //
// It is not what keeps the sweep off live work — inflightSet is, // A delivery is pending from the moment it is created until its
// and it is exact. This bound sets the re-dispatch cadence for a // outcome is written, which includes the whole time a worker
// delivery that really is stranded: without it, a delivery the // spends on it, so the bound has to clear the longest a live
// database will not let the engine settle would be re-sent on // attempt can take: httpClientTimeout plus queueing behind the
// every 60-second tick. // other deliveries in front of it. Five minutes is far above
// // that, and still recovers a stranded delivery in minutes rather
// It is nonetheless set clear of the longest legitimate attempt, // than at the next restart.
// so that the two guards do not both have to be right. That pendingSweepMinAge = 5 * time.Minute
// length is MaxTargetTimeoutSeconds (300s), the per-target
// timeout the target form accepts — not httpClientTimeout, which
// is merely the default. Fifteen minutes leaves a margin of
// three times the ceiling rather than the zero margin the two
// equal values would have given.
pendingSweepMinAge = 15 * time.Minute
// pendingSweepBatch bounds how many stranded pending deliveries // pendingSweepBatch bounds how many stranded pending deliveries
// one sweep of one webhook re-dispatches. The sweep runs every // one sweep of one webhook re-dispatches. The sweep runs every
@@ -183,12 +177,6 @@ type Engine struct {
// dbTarget is retained so the engine can reach the archive // dbTarget is retained so the engine can reach the archive
// writer registry for webhook eviction and the idle sweep. // writer registry for webhook eviction and the idle sweep.
dbTarget *databaseTarget dbTarget *databaseTarget
// inflight is the set of deliveries this engine currently owns.
// Recovery and the sweeps re-dispatch only what it does not
// hold. Held by value: its zero value works, so no constructor
// can leave it out. See inflight.go.
inflight inflightSet
} }
// New creates and registers the delivery engine with the // New creates and registers the delivery engine with the
@@ -221,27 +209,12 @@ func New(
// are ready. // are ready.
func (e *Engine) Notify(tasks []Task) { func (e *Engine) Notify(tasks []Task) {
for i := range tasks { for i := range tasks {
// Owned before it is queued, and until the worker that runs
// it returns. A task can sit in a 10000-deep channel for a
// long time on a healthy system, and nothing may re-send it
// while it waits. See inflight.go.
if !e.inflight.retainIdle(tasks[i].DeliveryID) {
e.log.Warn(
"delivery already in flight, not queued again",
"delivery_id", tasks[i].DeliveryID,
"event_id", tasks[i].EventID,
)
continue
}
select { select {
case e.deliveryCh <- tasks[i]: case e.deliveryCh <- tasks[i]:
default: default:
e.inflight.release(tasks[i].DeliveryID)
e.log.Warn( e.log.Warn(
"delivery channel full, "+ "delivery channel full, "+
"task will be recovered by the sweep", "task will be recovered on restart",
"delivery_id", tasks[i].DeliveryID, "delivery_id", tasks[i].DeliveryID,
"event_id", tasks[i].EventID, "event_id", tasks[i].EventID,
) )
@@ -276,20 +249,10 @@ func (e *Engine) ScheduleRetry(
"next_attempt", task.AttemptNum, "next_attempt", task.AttemptNum,
) )
// The reference is taken here rather than when the timer fires,
// so the delivery stays owned across the whole backoff window.
// Its caller is a target inside Deliver, so the engine already
// owns it; this second reference is what keeps that ownership
// alive after the worker returns and the row sits at retrying
// with nothing running. Without it the sweep finds the row
// orphaned and sends it again.
e.inflight.retain(task.DeliveryID)
time.AfterFunc(delay, func() { time.AfterFunc(delay, func() {
select { select {
case e.retryCh <- task: case e.retryCh <- task:
default: default:
e.inflight.release(task.DeliveryID)
e.log.Warn( e.log.Warn(
"retry channel full, delivery "+ "retry channel full, delivery "+
"will be recovered by periodic sweep", "will be recovered by periodic sweep",
@@ -389,35 +352,13 @@ func (e *Engine) worker(ctx context.Context) {
case <-ctx.Done(): case <-ctx.Done():
return return
case task := <-e.deliveryCh: case task := <-e.deliveryCh:
e.runTask(ctx, &task, e.processNewTask) e.processNewTask(ctx, &task)
case task := <-e.retryCh: case task := <-e.retryCh:
e.runTask(ctx, &task, e.processRetryTask) e.processRetryTask(ctx, &task)
} }
} }
} }
// runTask runs one task and then drops the reference the queueing
// side took on its delivery.
//
// The release is deferred rather than written after the call because
// every early return inside the processing paths must drop it too: a
// delivery whose database could not be opened is one the engine has
// stopped working on, and leaving it owned would hide it from the
// sweep forever.
//
// Ownership does not necessarily end here. A target that scheduled a
// retry took its own reference before this one is dropped, so the
// delivery stays owned through the backoff window.
func (e *Engine) runTask(
ctx context.Context,
task *Task,
run func(context.Context, *Task),
) {
defer e.inflight.release(task.DeliveryID)
run(ctx, task)
}
func (e *Engine) recoverPending(ctx context.Context) { func (e *Engine) recoverPending(ctx context.Context) {
defer e.wg.Done() defer e.wg.Done()
@@ -440,7 +381,7 @@ func (e *Engine) processNewTask(
event := buildEventFromTask(task) event := buildEventFromTask(task)
event, err = e.hydrateEvent( event, err = e.resolveEventBody(
webhookDB, event, task, webhookDB, event, task,
) )
if err != nil { if err != nil {
@@ -506,13 +447,9 @@ func (e *Engine) processRetryTask(
return return
} }
if e.abandonRetryForMissingTarget(webhookDB, d, task) {
return
}
event := buildEventFromTask(task) event := buildEventFromTask(task)
event, err = e.hydrateEvent( event, err = e.resolveEventBody(
webhookDB, event, task, webhookDB, event, task,
) )
if err != nil { if err != nil {
@@ -534,64 +471,6 @@ func (e *Engine) processRetryTask(
e.processDelivery(ctx, webhookDB, d, task) e.processDelivery(ctx, webhookDB, d, task)
} }
// abandonRetryForMissingTarget stops a retry chain whose target has
// been deleted, and reports whether it did.
//
// A scheduled retry lives in memory as a time.AfterFunc holding the
// target's configuration as it was when the chain began, and nothing
// else on this path reads the target row. Without this check a
// deletion stops nothing: the timer keeps firing and keeps sending to
// the destination the operator removed, for the whole remaining
// backoff chain. Terminalising in the recovery and sweep paths alone
// is not enough, because those only see the delivery once nothing
// holds it in memory — which is to say after a restart.
//
// The worker already owns this delivery, so the terminal write happens
// here directly, exactly as a target's own Deliver fails one. Claiming
// it again through the recovery gate would only fail against the
// reference the worker itself is holding.
//
// A lookup that fails for any other reason is not a deletion — it is
// the main database being unreadable — and the delivery goes ahead as
// it did before. A guard that terminally failed deliveries on a
// transient fault would be worse than the bug it fixes.
func (e *Engine) abandonRetryForMissingTarget(
webhookDB *gorm.DB,
d *database.Delivery,
task *Task,
) bool {
_, err := e.loadTarget(task.TargetID)
if err == nil {
return false
}
if !errors.Is(err, gorm.ErrRecordNotFound) {
e.log.Warn(
"could not confirm the target of a retrying "+
"delivery still exists; attempting anyway",
"delivery_id", task.DeliveryID,
"target_id", task.TargetID,
"error", err,
)
return false
}
targetType, reason := e.missingTargetReason(task.TargetID)
e.log.Warn(
"abandoning scheduled retry: target is gone",
"webhook_id", task.WebhookID,
"delivery_id", task.DeliveryID,
"target_id", task.TargetID,
"target_type", targetType,
)
e.failDelivery(webhookDB, d, targetType, reason)
return true
}
func (e *Engine) recoverInFlight(ctx context.Context) { func (e *Engine) recoverInFlight(ctx context.Context) {
var webhookIDs []string var webhookIDs []string
@@ -667,16 +546,7 @@ func (e *Engine) recoverRetryingDeliveries(
return return
} }
settled := e.reconcileDelivered(
webhookDB, webhookID, retrying,
e.loadTargetMap(retrying),
)
for i := range retrying { for i := range retrying {
if _, ok := settled[retrying[i].ID]; ok {
continue
}
e.recoverSingleRetry( e.recoverSingleRetry(
webhookDB, webhookID, &retrying[i], webhookDB, webhookID, &retrying[i],
) )
@@ -696,20 +566,6 @@ func (e *Engine) recoverSingleRetry(
) { ) {
target, err := e.loadTarget(d.TargetID) target, err := e.loadTarget(d.TargetID)
if err != nil { if err != nil {
// A target that is merely gone is an operator action with a
// terminal answer. Any other failure is the main database
// refusing to read, which is transient and must leave the
// delivery alone: failing every retrying delivery of every
// webhook on one bad read would be a far larger fault than
// the strand it is meant to clear.
if errors.Is(err, gorm.ErrRecordNotFound) {
e.failMissingTargetRetry(
webhookDB, webhookID, d,
)
return
}
e.log.Error( e.log.Error(
"failed to load target for retrying "+ "failed to load target for retrying "+
"delivery recovery", "delivery recovery",
@@ -752,10 +608,6 @@ func (e *Engine) recoverSingleRetry(
d, webhookID, &event, &target, attemptNum+1, d, webhookID, &event, &target, attemptNum+1,
) )
if !e.rescheduleRecovered(webhookDB, task, remaining) {
return
}
e.log.Info( e.log.Info(
"recovering retrying delivery", "recovering retrying delivery",
"webhook_id", webhookID, "webhook_id", webhookID,
@@ -763,6 +615,8 @@ func (e *Engine) recoverSingleRetry(
"attempt", attemptNum, "attempt", attemptNum,
"remaining_backoff", remaining, "remaining_backoff", remaining,
) )
e.ScheduleRetry(task, remaining)
} }
func (e *Engine) recoverPendingDeliveries( func (e *Engine) recoverPendingDeliveries(
@@ -772,14 +626,12 @@ func (e *Engine) recoverPendingDeliveries(
) { ) {
var deliveries []database.Delivery var deliveries []database.Delivery
// No Preload: event bodies are read one at a time in
// sendRecoveredDeliveries, and only for the deliveries actually
// being sent.
result := webhookDB. result := webhookDB.
Where( Where(
"status = ?", "status = ?",
database.DeliveryStatusPending, database.DeliveryStatusPending,
). ).
Preload("Event").
Find(&deliveries) Find(&deliveries)
if result.Error != nil { if result.Error != nil {
@@ -829,26 +681,19 @@ func (e *Engine) recoverPendingBatch(
) )
} }
// reconcileDelivered finds the deliveries in a recovered batch that // reconcileDelivered finds the deliveries in a pending batch that
// already have a successful DeliveryResult, marks them delivered, and // already have a successful DeliveryResult, marks them delivered, and
// returns their ids so the caller does not send them a second time. // returns their ids so the caller does not send them a second time.
// //
// This is the state the engine previously had no way to represent. A // This is the state the engine previously had no way to represent. A
// delivery is left in a non-terminal state by a failed bookkeeping // delivery is left pending by a failed bookkeeping write, and that
// write, and that covers two different histories: nothing was ever // covers two different histories: nothing was ever sent, or the send
// sent, or the send reached the receiver and only the status write // reached the receiver and only the status write failed. Re-sending
// failed. Re-sending was the sole option, so every stranded row // was the sole option, so every stranded row produced a duplicate at
// produced a duplicate at the receiver and an event log that recorded // the receiver and an event log that recorded one attempt for two
// one attempt for two POSTs. A successful result row distinguishes // POSTs. A successful result row distinguishes them: it is written
// them: it is written before the status, so its presence means the // before the status, so its presence means the wire I/O happened and
// wire I/O happened and was recorded, and all that is missing is the // was recorded, and all that is missing is the status.
// status.
//
// Every recovery path runs this, not only the pending one. A delivery
// abandoned at retrying can hold a successful result just as a pending
// one can — a second attempt that reached the receiver and whose status
// write then failed sits at retrying with success recorded — and
// re-sending it is the same duplicate.
// //
// Deliveries whose result row itself never landed are not in the // Deliveries whose result row itself never landed are not in the
// returned set and are re-sent, recorded as the further attempt they // returned set and are re-sent, recorded as the further attempt they
@@ -904,7 +749,7 @@ func (e *Engine) reconcileDelivered(
} }
e.log.Info( e.log.Info(
"settling recovered deliveries that already succeeded", "settling pending deliveries that already succeeded",
"webhook_id", webhookID, "webhook_id", webhookID,
"count", len(settled), "count", len(settled),
) )
@@ -914,22 +759,12 @@ func (e *Engine) reconcileDelivered(
continue continue
} }
// A delivery the engine is working on right now settles
// itself; writing over it from here would race that worker.
if !e.inflight.retainIdle(deliveries[i].ID) {
delete(settled, deliveries[i].ID)
continue
}
e.settleStatus( e.settleStatus(
webhookDB, webhookDB,
&deliveries[i], &deliveries[i],
targetMap[deliveries[i].TargetID].Type, targetMap[deliveries[i].TargetID].Type,
database.DeliveryStatusDelivered, database.DeliveryStatusDelivered,
) )
e.inflight.release(deliveries[i].ID)
} }
return settled return settled
@@ -1016,11 +851,6 @@ func (e *Engine) sweepWebhookRetries(
return return
} }
settled := e.reconcileDelivered(
webhookDB, webhookID, retrying,
e.loadTargetMap(retrying),
)
for i := range retrying { for i := range retrying {
select { select {
case <-ctx.Done(): case <-ctx.Done():
@@ -1028,10 +858,6 @@ func (e *Engine) sweepWebhookRetries(
default: default:
} }
if _, ok := settled[retrying[i].ID]; ok {
continue
}
e.sweepSingleRetry( e.sweepSingleRetry(
webhookDB, webhookID, &retrying[i], webhookDB, webhookID, &retrying[i],
) )
@@ -1043,18 +869,20 @@ func (e *Engine) sweepWebhookRetries(
// sweepWebhookPending recovers deliveries stranded at pending. // sweepWebhookPending recovers deliveries stranded at pending.
// //
// A delivery is created pending and leaves that state only when its // A delivery is created pending and leaves that state only when its
// outcome is written, so a pending row the engine does not own is one // outcome is written, so a pending row older than the age bound is one
// whose bookkeeping write failed — the state that used to sit there // whose bookkeeping write failed — the state that used to sit there
// until a restart, and then produce a duplicate at the receiver. The // until a restart, and then produce a duplicate at the receiver. The
// sweep gives it the same reconcile-then-dispatch treatment restart // sweep gives it the same reconcile-then-dispatch treatment restart
// recovery gets, so it costs a minute rather than an operator // recovery gets, so it costs a minute rather than an operator
// noticing. // noticing.
// //
// What keeps the sweep off live work is ownership, checked per // The age bound is what keeps the sweep off deliveries the workers
// delivery in takeForRedispatch, not the age bound in this query. // still hold: a delivery in flight is pending too, and re-dispatching
// A delivery waiting in deliveryCh is pending and arbitrarily old — // one would race the worker that owns it. It is measured on updated_at
// the channel holds 10000 tasks and 10 workers drain it — so // rather than created_at because claimPending stamps that column when
// reasoning from the row's age alone re-sends it. See inflight.go. // a delivery is handed out, which is what stops the next sweep, a
// minute later, from sending the same delivery again while the first
// attempt is still running.
func (e *Engine) sweepWebhookPending( func (e *Engine) sweepWebhookPending(
ctx context.Context, ctx context.Context,
webhookDB *gorm.DB, webhookDB *gorm.DB,
@@ -1068,6 +896,7 @@ func (e *Engine) sweepWebhookPending(
database.DeliveryStatusPending, database.DeliveryStatusPending,
time.Now().Add(-pendingSweepMinAge), time.Now().Add(-pendingSweepMinAge),
). ).
Preload("Event").
Limit(pendingSweepBatch). Limit(pendingSweepBatch).
Find(&pending).Error Find(&pending).Error
if err != nil { if err != nil {
@@ -1105,16 +934,6 @@ func (e *Engine) sweepSingleRetry(
) { ) {
target, err := e.loadTarget(d.TargetID) target, err := e.loadTarget(d.TargetID)
if err != nil { if err != nil {
// Deleted is terminal, unreadable is not; see
// recoverSingleRetry.
if errors.Is(err, gorm.ErrRecordNotFound) {
e.failMissingTargetRetry(
webhookDB, webhookID, d,
)
return
}
e.log.Error( e.log.Error(
"retry sweep: failed to load target", "retry sweep: failed to load target",
"delivery_id", d.ID, "delivery_id", d.ID,
@@ -1158,20 +977,17 @@ func (e *Engine) sweepSingleRetry(
d, webhookID, &event, &target, attemptNum+1, d, webhookID, &event, &target, attemptNum+1,
) )
if !e.redispatch( select {
e.retryCh, webhookDB, task, case e.retryCh <- task:
database.DeliveryStatusRetrying, e.log.Info(
) { "retry sweep: "+
return "recovered orphaned retrying delivery",
"delivery_id", d.ID,
"webhook_id", webhookID,
"attempt", attemptNum+1,
)
default:
} }
e.log.Info(
"retry sweep: "+
"recovered orphaned retrying delivery",
"delivery_id", d.ID,
"webhook_id", webhookID,
"attempt", attemptNum+1,
)
} }
// failUnretryableRetry terminally fails an orphaned retrying // failUnretryableRetry terminally fails an orphaned retrying
@@ -1194,16 +1010,6 @@ func (e *Engine) failUnretryableRetry(
d *database.Delivery, d *database.Delivery,
target *database.Target, target *database.Target,
) { ) {
// Terminal, and reached from the recovery paths, so it takes
// ownership like every other write they make: a delivery the
// engine is still attempting must not be failed underneath the
// worker running it.
if !e.inflight.retainIdle(d.ID) {
return
}
defer e.inflight.release(d.ID)
e.log.Warn( e.log.Warn(
"failing orphaned retrying delivery: target "+ "failing orphaned retrying delivery: target "+
"type no longer supports retries", "type no longer supports retries",
@@ -1221,113 +1027,6 @@ func (e *Engine) failUnretryableRetry(
target.Type, target.Type,
) )
e.failDelivery(webhookDB, d, target.Type, reason)
}
// failMissingTargetRetry terminally fails an orphaned retrying
// delivery whose target row is gone. Both restart recovery and the
// periodic sweep call it, so the transition exists once.
//
// Until it existed both paths logged the failed lookup and returned,
// which left the delivery retrying for the life of the database and
// the sweep repeating the same error every minute forever. Failing it
// with a recorded reason is the treatment the other orphaned-retry
// cases already get, so all of them read alike in the event log.
//
// Logged at warn rather than error: a deleted target is an operator
// action, not a system fault.
func (e *Engine) failMissingTargetRetry(
webhookDB *gorm.DB,
webhookID string,
d *database.Delivery,
) {
// Terminal, and reached from the recovery paths, so it takes
// ownership like every other write they make.
if !e.inflight.retainIdle(d.ID) {
return
}
defer e.inflight.release(d.ID)
targetType, reason := e.missingTargetReason(d.TargetID)
e.log.Warn(
"failing orphaned retrying delivery: "+
"its target no longer exists",
"webhook_id", webhookID,
"delivery_id", d.ID,
"target_id", d.TargetID,
"target_type", targetType,
)
e.failDelivery(webhookDB, d, targetType, reason)
}
// missingTargetReason describes a target id that no longer resolves,
// and returns the type of the deleted row where there still is one.
//
// The lookup is Unscoped because deletes are soft: the row survives
// with deleted_at set, invisible to loadTarget's default scope.
// Reading it is what separates "you deleted this target" from "this id
// never named a row" — different things to whoever reads the event
// log, and only the first is something an operator did. The widened
// scope is deliberately confined to this terminal path: the engine's
// normal target loading must go on refusing a deleted target, or
// deleting one would stop nothing.
//
// The type comes back so the caller can label the delivery's status
// transition with it. Where the row is gone entirely there is no type
// to give, and updateDeliveryStatus leaves the counter alone rather
// than opening a series named by the empty string.
func (e *Engine) missingTargetReason(
targetID string,
) (database.TargetType, string) {
var target database.Target
err := e.database.DB().Unscoped().
First(&target, "id = ?", targetID).Error
if err != nil {
return "", fmt.Sprintf(
"target %s no longer exists; the delivery "+
"cannot be retried and has been failed "+
"terminally",
targetID,
)
}
return target.Type, fmt.Sprintf(
"target %q (type %s) was deleted; the delivery "+
"cannot be retried and has been failed terminally",
target.Name, target.Type,
)
}
// failDelivery records why a delivery is over and then marks it
// failed. The caller must already own the delivery: every call site is
// either a worker holding the reference runTask took, or a recovery
// path that took one through retainIdle.
//
// The result row is written first and a failure to write it stops the
// transition, which is what keeps a delivery from ending failed with
// an empty event log — the state that leaves an operator with nothing
// but a server log line to work out what happened. A delivery whose
// reason could not be recorded stays in the non-terminal state it
// already holds, where the sweep will find it again; see
// bookkeepingFailed.
//
// The target type is a parameter rather than read off d because the
// orphaned-retry callers deliberately hold a delivery loaded without
// its Target relation: populating d.Target would make GORM's
// SaveBeforeAssociations upsert the whole target row — plaintext
// config, which for a slack target is the credential — into the
// per-webhook event database. See
// https://git.eeqj.de/sneak/webhooker/issues/206.
func (e *Engine) failDelivery(
webhookDB *gorm.DB,
d *database.Delivery,
targetType database.TargetType,
reason string,
) {
err := e.recordResult( err := e.recordResult(
webhookDB, webhookDB,
d, d,
@@ -1344,8 +1043,14 @@ func (e *Engine) failDelivery(
return return
} }
// The type is passed rather than assigned onto d: the delivery
// is loaded here without its target relation, and populating
// d.Target would make GORM's SaveBeforeAssociations upsert the
// whole target row — plaintext config, which for a slack target
// is the credential — into the per-webhook event database. See
// https://git.eeqj.de/sneak/webhooker/issues/206.
e.settleStatus( e.settleStatus(
webhookDB, d, targetType, webhookDB, d, target.Type,
database.DeliveryStatusFailed, database.DeliveryStatusFailed,
) )
} }
@@ -1366,19 +1071,9 @@ func (e *Engine) processDelivery(
"type", d.Target.Type, "type", d.Target.Type,
) )
// The reason is recorded, not just logged. This branch used e.settleStatus(
// to fail the delivery with no DeliveryResult at all, which
// showed in the event log as "failed, no attempts recorded
// yet" and left one server log line as the only account of
// why anywhere.
e.failDelivery(
webhookDB, d, d.Target.Type, webhookDB, d, d.Target.Type,
fmt.Sprintf( database.DeliveryStatusFailed,
"unknown target type %q: this build has no "+
"delivery implementation for it, so no "+
"attempt was made",
d.Target.Type,
),
) )
return return
@@ -1508,13 +1203,7 @@ func (e *Engine) updateDeliveryStatus(
) )
} }
// An empty type means the target row is gone — a delivery being e.mtr.DeliveryStatusChanged(targetType, status)
// settled long after its target was deleted. The row still has to
// be settled, but the counter is left alone rather than given a
// series labelled with the empty string.
if targetType != "" {
e.mtr.DeliveryStatusChanged(targetType, status)
}
return nil return nil
} }
@@ -1547,11 +1236,6 @@ func truncate(s string, maxLen int) string {
// --- Helper functions --- // --- Helper functions ---
// buildEventFromTask reconstructs the event a Task describes, as far
// as the Task itself goes. The fields it cannot fill — the body when
// it was too large to inline, and the receipt time, which no Task
// carries — come from the stored row in hydrateEvent, which every
// caller of this function runs next.
func buildEventFromTask(task *Task) database.Event { func buildEventFromTask(task *Task) database.Event {
event := database.Event{ event := database.Event{
EntrypointID: task.EntrypointID, EntrypointID: task.EntrypointID,
@@ -1579,67 +1263,29 @@ func buildTargetFromTask(task *Task) database.Target {
return target return target
} }
// hydrateEvent fills in the event fields a Task does not carry, by func (e *Engine) resolveEventBody(
// reading the stored event row.
//
// CreatedAt is the event's receipt time and lives only in that row.
// The Slack target renders it into every message it sends, so an
// unhydrated event puts the zero time in front of a human on every
// notification the product delivers. See
// https://git.eeqj.de/sneak/webhooker/issues/257.
//
// The body comes from the same row when the Task did not inline it,
// which is the case for a body at or above MaxInlineBodySize.
//
// A read failure is fatal to the delivery only when the body depended
// on it. When the Task inlined the body, the delivery has everything
// it needs to be sent and goes ahead with the timestamp unset: the row
// can be gone under a retention reap while a queued delivery still
// holds its body, and dropping a deliverable event to protect one
// metadata field would be a worse failure than the one it prevents.
func (e *Engine) hydrateEvent(
webhookDB *gorm.DB, webhookDB *gorm.DB,
event database.Event, event database.Event,
task *Task, task *Task,
) (database.Event, error) { ) (database.Event, error) {
columns := []string{"created_at"} if task.Body != nil {
if task.Body == nil {
columns = append(columns, "body")
}
var dbEvent database.Event
err := webhookDB.Select(columns).
First(&dbEvent, "id = ?", task.EventID).Error
if err != nil {
if task.Body == nil {
return event, fmt.Errorf(
"fetching event body: %w", err,
)
}
e.log.Warn(
"could not read the stored event; delivering "+
"the inlined body without its receipt time",
"event_id", task.EventID,
"delivery_id", task.DeliveryID,
"error", err,
)
event.Body = *task.Body event.Body = *task.Body
return event, nil return event, nil
} }
event.CreatedAt = dbEvent.CreatedAt var dbEvent database.Event
if task.Body != nil { err := webhookDB.Select("body").
event.Body = *task.Body First(&dbEvent, "id = ?", task.EventID).Error
} else { if err != nil {
event.Body = dbEvent.Body return event, fmt.Errorf(
"fetching event body: %w", err,
)
} }
event.Body = dbEvent.Body
return event, nil return event, nil
} }
@@ -1671,133 +1317,44 @@ func (e *Engine) countAttempts(
return int(resultCount) return int(resultCount)
} }
// takeForRedispatch decides whether a recovered delivery may be sent // claimPending takes ownership of a pending delivery before it is
// again, and takes it if so. It is the single gate every re-dispatch // re-dispatched, and reports whether the claim succeeded.
// path goes through, and it asks two separate questions in order.
// //
// First, does the engine already own this delivery? Ownership is // The claim is a compare-and-set on the status: it takes effect only
// exact and mutually exclusive, so a delivery queued, being attempted, // while the delivery is still pending, so a worker that settled the
// or waiting out a retry backoff is refused here, and two dispatchers // delivery between the query and here wins and nothing is re-sent.
// racing for the same delivery cannot both win. See inflight.go. // Stamping updated_at is the claim itself — the sweep selects on that
// column, so a delivery handed out now is out of the sweep's reach for
// a further pendingSweepMinAge, rather than being sent again on every
// sweep for as long as the attempt takes.
// //
// Second, is the row still in the status that made it eligible? The // A claim that cannot be written means the database is refusing
// batch was read some time ago and a worker may have settled a row // writes, which is the condition that stranded this delivery in the
// since. The check is a conditional update rather than a read so the // first place. Not sending is then the right answer: the attempt
// answer cannot go stale between asking and acting. // could not be recorded either, and an unrecordable send is exactly
// // the duplicate this issue is about.
// Stamping updated_at is the same statement, and it is a cadence func (e *Engine) claimPending(
// control rather than a claim: the pending sweep selects on that webhookDB *gorm.DB, d *database.Delivery,
// column, so a delivery handed out now is not selected again on the
// next tick a minute later but after pendingSweepMinAge. A delivery
// the database refuses to settle is therefore retried on that
// interval instead of every tick.
//
// A failed write is a refusal. It means the database is not accepting
// writes, which is the condition that stranded the delivery in the
// first place; an attempt that cannot be recorded is exactly the
// unlogged duplicate this is all here to prevent.
//
// The caller must release ownership if it then fails to queue the
// task.
func (e *Engine) takeForRedispatch(
webhookDB *gorm.DB,
deliveryID string,
eligible database.DeliveryStatus,
) bool { ) bool {
if !e.inflight.retainIdle(deliveryID) {
return false
}
res := webhookDB. res := webhookDB.
Model(&database.Delivery{}). Model(&database.Delivery{}).
Where( Where(
"id = ? AND status = ?", deliveryID, eligible, "id = ? AND status = ?",
d.ID, database.DeliveryStatusPending,
). ).
UpdateColumn("updated_at", time.Now()) UpdateColumn("updated_at", time.Now())
if res.Error != nil { if res.Error != nil {
e.log.Error( e.log.Error(
"failed to mark delivery for re-dispatch; "+ "failed to claim pending delivery for recovery; "+
"leaving it for a later sweep", "leaving it for the next sweep",
"delivery_id", deliveryID, "delivery_id", d.ID,
"error", res.Error, "error", res.Error,
) )
e.inflight.release(deliveryID)
return false return false
} }
if res.RowsAffected != 1 { return res.RowsAffected == 1
// Settled underneath us between the query and here.
e.inflight.release(deliveryID)
return false
}
return true
}
// queueRecovered puts an owned delivery's task on a worker channel,
// dropping the ownership the gate took if it does not fit.
func (e *Engine) queueRecovered(
ch chan<- Task, task Task,
) bool {
select {
case ch <- task:
return true
default:
e.inflight.release(task.DeliveryID)
e.log.Warn(
"worker channel full during recovery; "+
"delivery will be recovered by a later sweep",
"delivery_id", task.DeliveryID,
"webhook_id", task.WebhookID,
)
return false
}
}
// redispatch hands a recovered delivery to a worker channel through
// takeForRedispatch, and reports whether the task was queued.
func (e *Engine) redispatch(
ch chan<- Task,
webhookDB *gorm.DB,
task Task,
eligible database.DeliveryStatus,
) bool {
if !e.takeForRedispatch(
webhookDB, task.DeliveryID, eligible,
) {
return false
}
return e.queueRecovered(ch, task)
}
// rescheduleRecovered hands an orphaned retrying delivery back to the
// retry timer, through the same gate. It reports whether the delivery
// was rescheduled.
//
// The reference taken by the gate is dropped as soon as ScheduleRetry
// has taken its own, which it does before returning: what keeps the
// delivery owned through the backoff window is ScheduleRetry's
// reference, not this one.
func (e *Engine) rescheduleRecovered(
webhookDB *gorm.DB, task Task, delay time.Duration,
) bool {
if !e.takeForRedispatch(
webhookDB, task.DeliveryID,
database.DeliveryStatusRetrying,
) {
return false
}
defer e.inflight.release(task.DeliveryID)
e.ScheduleRetry(task, delay)
return true
} }
// countAttemptsBatch counts the recorded attempts of every delivery // countAttemptsBatch counts the recorded attempts of every delivery
@@ -1916,10 +1473,6 @@ func buildRecoveryTask(
func (e *Engine) loadTargetMap( func (e *Engine) loadTargetMap(
deliveries []database.Delivery, deliveries []database.Delivery,
) map[string]database.Target { ) map[string]database.Target {
if len(deliveries) == 0 {
return nil
}
seen := make(map[string]bool) seen := make(map[string]bool)
targetIDs := make([]string, 0, len(deliveries)) targetIDs := make([]string, 0, len(deliveries))
@@ -1959,15 +1512,6 @@ func (e *Engine) loadTargetMap(
// sendRecoveredDeliveries re-dispatches pending deliveries, skipping // sendRecoveredDeliveries re-dispatches pending deliveries, skipping
// the ids in settled — those already reached their receiver and have // the ids in settled — those already reached their receiver and have
// been marked delivered by reconcileDelivered. // been marked delivered by reconcileDelivered.
//
// The skip and takeForRedispatch's status check answer different
// questions and neither replaces the other. This one is "did this
// delivery already succeed", which is what settles the row to
// delivered instead of sending it, and which is the only thing that
// keeps the retrying paths from terminally failing a delivery that
// reconcile just settled. The status check is "is the row still what
// the batch query said it was", which catches a worker settling it to
// anything at all in between.
func (e *Engine) sendRecoveredDeliveries( func (e *Engine) sendRecoveredDeliveries(
ctx context.Context, ctx context.Context,
webhookDB *gorm.DB, webhookDB *gorm.DB,
@@ -2005,40 +1549,27 @@ func (e *Engine) sendRecoveredDeliveries(
continue continue
} }
if !e.takeForRedispatch( if !e.claimPending(webhookDB, &deliveries[i]) {
webhookDB, deliveries[i].ID,
database.DeliveryStatusPending,
) {
continue
}
// The body is read here, one delivery at a time and only for
// deliveries that are actually being sent, rather than
// preloaded across the whole batch. A batch is up to
// pendingSweepBatch rows at up to the 1 MB ingest cap, and
// most of a sweep's batch is refused by the gate above — so
// preloading would hold hundreds of megabytes per webhook per
// tick to build tasks it then discards.
event, err := e.loadEvent(
webhookDB, deliveries[i].EventID,
)
if err != nil {
e.log.Error(
"failed to load event for recovered delivery",
"delivery_id", deliveries[i].ID,
"event_id", deliveries[i].EventID,
"error", err,
)
e.inflight.release(deliveries[i].ID)
continue continue
} }
task := buildRecoveryTask( task := buildRecoveryTask(
&deliveries[i], webhookID, &event, &target, &deliveries[i], webhookID,
&deliveries[i].Event, &target,
attempts[deliveries[i].ID]+1, attempts[deliveries[i].ID]+1,
) )
e.queueRecovered(e.deliveryCh, task) select {
case e.deliveryCh <- task:
default:
e.log.Warn(
"delivery channel full during "+
"recovery, remaining deliveries "+
"will be recovered on next restart",
"delivery_id", deliveries[i].ID,
)
return
}
} }
} }

23
internal/delivery/engine_integration_test.go
View File

@@ -377,17 +377,6 @@ func TestProcessRetryTask_SuccessfulRetry(t *testing.T) {
bodyStr := event.Body bodyStr := event.Body
cfg := iHTTPConfig(ts.URL) cfg := iHTTPConfig(ts.URL)
// The target row exists because the engine confirms a scheduled
// retry's target has not been deleted before it runs it. A retry
// task whose target id names no row at all is a state the service
// does not produce: the handler read that target to build the
// task. See https://git.eeqj.de/sneak/webhooker/issues/107.
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "retry-target",
database.TargetTypeHTTP, cfg, 5,
)
task := iTask( task := iTask(
d, event, s.WebhookID, targetID, d, event, s.WebhookID, targetID,
"retry-target", cfg, 5, 2, &bodyStr, "retry-target", cfg, 5, 2, &bodyStr,
@@ -467,12 +456,6 @@ func TestProcessRetryTask_LargeBody_FetchFromDB(
) )
cfg := iHTTPConfig(ts.URL) cfg := iHTTPConfig(ts.URL)
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "retry-large",
database.TargetTypeHTTP, cfg, 5,
)
task := iTask( task := iTask(
d, event, s.WebhookID, targetID, d, event, s.WebhookID, targetID,
"retry-large", cfg, 5, 2, nil, "retry-large", cfg, 5, 2, nil,
@@ -575,12 +558,6 @@ func TestWorkerLifecycle_ProcessesRetryChannel(
bodyStr := event.Body bodyStr := event.Body
cfg := iHTTPConfig(ts.URL) cfg := iHTTPConfig(ts.URL)
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "retry-chan-test",
database.TargetTypeHTTP, cfg, 5,
)
task := iTask( task := iTask(
d, event, s.WebhookID, targetID, d, event, s.WebhookID, targetID,
"retry-chan-test", cfg, 5, 2, &bodyStr, "retry-chan-test", cfg, 5, 2, &bodyStr,

10
internal/delivery/event_db_isolation_test.go
View File

@@ -96,15 +96,7 @@ func TestEventDBHoldsNoTargetRows(t *testing.T) {
) )
assertNoTargetRows(t, dbPath) assertNoTargetRows(t, dbPath)
// A retry. Its target exists in the main database, because the // A retry.
// engine confirms a scheduled retry's target has not been
// deleted before running it; see
// https://git.eeqj.de/sneak/webhooker/issues/107.
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "leaky-target",
database.TargetTypeHTTP, cfg, 5,
)
rd := iSeedDelivery( rd := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID, t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusRetrying, database.DeliveryStatusRetrying,

442
internal/delivery/event_timestamp_test.go
View File

@@ -1,442 +0,0 @@
package delivery_test
import (
"context"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
// tsEventCreatedAt is the receipt time seeded on the events these
// tests deliver. It is far enough from both the zero time and from
// now that neither can be mistaken for it.
func tsEventCreatedAt() time.Time {
return time.Date(
2026, time.March, 4, 5, 6, 7, 0, time.UTC,
)
}
// tsZeroStamp is what a Slack message renders when the event handed
// to FormatSlackMessage carries no CreatedAt.
const tsZeroStamp = "*Timestamp:* `0001-01-01T00:00:00Z`"
// tsEventBody is the body seeded on every event in this file. It is
// small enough that a Task can inline it.
const tsEventBody = `{"hello":"world"}`
// tsUndeliverableHook stands in for a Slack incoming webhook on the
// tests that never send: the config parser requires a URL, but no
// request is made.
const tsUndeliverableHook = "https://hooks.slack.com/services/T/B/x"
// tsSink is a stand-in Slack incoming webhook that records the raw
// body posted to it.
type tsSink struct {
*httptest.Server
bodies chan []byte
}
func newTSSink(t *testing.T) *tsSink {
t.Helper()
s := &tsSink{bodies: make(chan []byte, 8)}
s.Server = httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
select {
case s.bodies <- body:
default:
}
w.WriteHeader(http.StatusOK)
},
))
t.Cleanup(s.Close)
return s
}
// text returns the Slack message text from the single payload the
// sink received.
func (s *tsSink) text(t *testing.T) string {
t.Helper()
select {
case raw := <-s.bodies:
t.Logf("raw slack payload: %s", raw)
var payload struct {
Text string `json:"text"`
}
require.NoError(t, json.Unmarshal(raw, &payload))
return payload.Text
case <-time.After(5 * time.Second):
t.Fatal("slack sink received no payload")
return ""
}
}
func tsSlackConfig(t *testing.T, url string) string {
t.Helper()
data, err := json.Marshal(
delivery.SlackTargetConfig{WebhookURL: url},
)
require.NoError(t, err)
return string(data)
}
// tsSeedEvent writes an event whose CreatedAt is tsEventCreatedAt
// rather than the write time, so an assertion on the rendered
// timestamp cannot pass by accident against "roughly now".
func tsSeedEvent(
t *testing.T, db *gorm.DB, webhookID string,
) database.Event {
t.Helper()
event := database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
Headers: `{}`,
Body: tsEventBody,
ContentType: "application/json",
}
event.ID = uuid.New().String()
event.CreatedAt = tsEventCreatedAt()
event.UpdatedAt = tsEventCreatedAt()
require.NoError(t, db.Create(&event).Error)
var stored database.Event
require.NoError(t,
db.First(&stored, "id = ?", event.ID).Error,
)
require.Equal(t,
tsEventCreatedAt().UTC(), stored.CreatedAt.UTC(),
"seeded created_at did not round-trip",
)
return event
}
// tsSeedTarget writes the slack target row into the main database.
// The retry path confirms the target still exists before sending.
func tsSeedTarget(
t *testing.T, mainDB *gorm.DB, webhookID, config string,
) database.Target {
t.Helper()
target := database.Target{
WebhookID: webhookID,
Name: "slack-sink",
Type: database.TargetTypeSlack,
Config: config,
Active: true,
}
require.NoError(t, mainDB.Create(&target).Error)
return target
}
func tsTask(
d database.Delivery,
event database.Event,
webhookID string,
target database.Target,
attemptNum int,
body *string,
) delivery.Task {
return delivery.Task{
DeliveryID: d.ID,
EventID: event.ID,
WebhookID: webhookID,
EntrypointID: event.EntrypointID,
TargetID: target.ID,
TargetName: target.Name,
TargetType: database.TargetTypeSlack,
TargetConfig: target.Config,
MaxRetries: 0,
Method: event.Method,
Headers: event.Headers,
ContentType: event.ContentType,
Body: body,
AttemptNum: attemptNum,
}
}
func tsAssertRealTimestamp(t *testing.T, text string) {
t.Helper()
assert.NotContains(t, text, tsZeroStamp,
"slack message carries the zero timestamp",
)
assert.Contains(t, text,
"*Timestamp:* `"+
tsEventCreatedAt().UTC().Format(time.RFC3339)+"`",
"slack message does not carry the event's receipt time",
)
}
// tsCase is one end-to-end delivery of a seeded event to a slack
// sink, over whichever engine path `process` names.
type tsCase struct {
// status is the delivery row's status before the engine runs.
// The retry path refuses a delivery that is not retrying.
status database.DeliveryStatus
// inlineBody mirrors a Task built for a body under
// MaxInlineBodySize. When false the engine reads the body back
// from the stored row.
inlineBody bool
attemptNum int
process func(
ctx context.Context, e *delivery.Engine, task *delivery.Task,
)
}
// run delivers one event through the named path and returns the
// Slack message text the sink received.
func (c tsCase) run(t *testing.T) (iSetup, database.Delivery, string) {
t.Helper()
s := newISetup(t)
sink := newTSSink(t)
cfg := tsSlackConfig(t, sink.URL)
target := tsSeedTarget(t, s.MainDB, s.WebhookID, cfg)
event := tsSeedEvent(t, s.WebhookDB, s.WebhookID)
d := iSeedDelivery(
t, s.WebhookDB, event.ID, target.ID, c.status,
)
var body *string
if c.inlineBody {
bodyStr := event.Body
body = &bodyStr
}
task := tsTask(
d, event, s.WebhookID, target, c.attemptNum, body,
)
c.process(context.TODO(), s.Engine, &task)
return s, d, sink.text(t)
}
// TestSlackFirstAttemptCarriesEventTimestamp covers the path an
// event takes on its first delivery: the task comes from the
// receiver and the engine reconstructs the event from it.
func TestSlackFirstAttemptCarriesEventTimestamp(t *testing.T) {
t.Parallel()
s, d, text := tsCase{
status: database.DeliveryStatusPending,
inlineBody: true,
attemptNum: 1,
process: func(
ctx context.Context,
e *delivery.Engine,
task *delivery.Task,
) {
e.ExportProcessNewTask(ctx, task)
},
}.run(t)
tsAssertRealTimestamp(t, text)
iAssertStatus(t, s.WebhookDB, d.ID,
database.DeliveryStatusDelivered,
)
}
// TestSlackFirstAttemptLargeBodyCarriesEventTimestamp covers the
// first-attempt path for an event whose body exceeded
// MaxInlineBodySize, so the task carries no body and the engine
// reads it back from the stored row.
func TestSlackFirstAttemptLargeBodyCarriesEventTimestamp(
t *testing.T,
) {
t.Parallel()
_, _, text := tsCase{
status: database.DeliveryStatusPending,
inlineBody: false,
attemptNum: 1,
process: func(
ctx context.Context,
e *delivery.Engine,
task *delivery.Task,
) {
e.ExportProcessNewTask(ctx, task)
},
}.run(t)
tsAssertRealTimestamp(t, text)
}
// TestSlackRetryCarriesEventTimestamp covers the retry path, which
// reconstructs the event from the same task the first attempt used.
func TestSlackRetryCarriesEventTimestamp(t *testing.T) {
t.Parallel()
s, d, text := tsCase{
status: database.DeliveryStatusRetrying,
inlineBody: true,
attemptNum: 2,
process: func(
ctx context.Context,
e *delivery.Engine,
task *delivery.Task,
) {
e.ExportProcessRetryTask(ctx, task)
},
}.run(t)
tsAssertRealTimestamp(t, text)
iAssertStatus(t, s.WebhookDB, d.ID,
database.DeliveryStatusDelivered,
)
}
// TestFormatSlackMessageOverTaskReconstructedEvent asserts on the
// formatted message directly, over the event the delivery paths
// reconstruct from a Task. It is the unit-level guard under the
// end-to-end tests: revert the CreatedAt population in hydrateEvent
// and this fails on the zero timestamp.
func TestFormatSlackMessageOverTaskReconstructedEvent(
t *testing.T,
) {
t.Parallel()
s := newISetup(t)
cfg := tsSlackConfig(t, tsUndeliverableHook)
target := tsSeedTarget(t, s.MainDB, s.WebhookID, cfg)
event := tsSeedEvent(t, s.WebhookDB, s.WebhookID)
d := iSeedDelivery(
t, s.WebhookDB, event.ID, target.ID,
database.DeliveryStatusPending,
)
bodyStr := event.Body
task := tsTask(d, event, s.WebhookID, target, 1, &bodyStr)
rebuilt, err := s.Engine.ExportEventForTask(
s.WebhookDB, &task,
)
require.NoError(t, err)
assert.False(t, rebuilt.CreatedAt.IsZero(),
"reconstructed event carries the zero time",
)
assert.Equal(t,
tsEventCreatedAt().UTC(), rebuilt.CreatedAt.UTC(),
)
tsAssertRealTimestamp(
t, delivery.FormatSlackMessage(&rebuilt),
)
}
// TestFormatSlackMessageZeroTimestamp asserts the rendering choice
// directly, without going through the engine: a zero CreatedAt (the
// shape a reaped-row fallback produces) renders as "unknown" rather
// than the year-1 zero time, while a real CreatedAt still renders as
// RFC3339.
func TestFormatSlackMessageZeroTimestamp(t *testing.T) {
t.Parallel()
zeroEvent := database.Event{
Method: http.MethodPost,
ContentType: testContentType,
Body: tsEventBody,
}
zeroText := delivery.FormatSlackMessage(&zeroEvent)
assert.NotContains(t, zeroText, "0001-01-01",
"slack message carries the zero-time year",
)
assert.Contains(t, zeroText, "*Timestamp:* `unknown`",
"slack message does not mark an unset receipt time as unknown",
)
nonZeroEvent := zeroEvent
nonZeroEvent.CreatedAt = tsEventCreatedAt()
nonZeroText := delivery.FormatSlackMessage(&nonZeroEvent)
assert.Contains(t, nonZeroText,
"*Timestamp:* `"+
tsEventCreatedAt().UTC().Format(time.RFC3339)+"`",
"slack message does not render a real receipt time as RFC3339",
)
}
// TestEventReconstructionSurvivesAReapedRow pins the fallback: an
// event row reaped by retention while its delivery still holds the
// body inline is still delivered, with the receipt time unset,
// rather than dropped.
func TestEventReconstructionSurvivesAReapedRow(t *testing.T) {
t.Parallel()
s := newISetup(t)
cfg := tsSlackConfig(t, tsUndeliverableHook)
target := tsSeedTarget(t, s.MainDB, s.WebhookID, cfg)
event := tsSeedEvent(t, s.WebhookDB, s.WebhookID)
d := iSeedDelivery(
t, s.WebhookDB, event.ID, target.ID,
database.DeliveryStatusPending,
)
bodyStr := event.Body
task := tsTask(d, event, s.WebhookID, target, 1, &bodyStr)
require.NoError(t, s.WebhookDB.Unscoped().Delete(
&database.Event{}, "id = ?", event.ID,
).Error)
rebuilt, err := s.Engine.ExportEventForTask(
s.WebhookDB, &task,
)
require.NoError(t, err)
assert.Equal(t, bodyStr, rebuilt.Body)
assert.True(t, rebuilt.CreatedAt.IsZero())
// A task with no inlined body has nothing left to deliver, so
// the same reaped row is an error there.
noBody := task
noBody.Body = nil
_, err = s.Engine.ExportEventForTask(s.WebhookDB, &noBody)
require.Error(t, err)
}

30
internal/delivery/export_test.go
View File

@@ -151,16 +151,6 @@ func (e *Engine) ExportProcessRetryTask(
e.processRetryTask(ctx, task) e.processRetryTask(ctx, task)
} }
// ExportEventForTask exposes the event reconstruction the delivery
// paths run: buildEventFromTask followed by hydrateEvent.
func (e *Engine) ExportEventForTask(
webhookDB *gorm.DB, task *Task,
) (database.Event, error) {
return e.hydrateEvent(
webhookDB, buildEventFromTask(task), task,
)
}
// ExportProcessDelivery exposes processDelivery. // ExportProcessDelivery exposes processDelivery.
func (e *Engine) ExportProcessDelivery( func (e *Engine) ExportProcessDelivery(
ctx context.Context, ctx context.Context,
@@ -301,26 +291,6 @@ func (e *Engine) ExportWedgeWorker(release <-chan struct{}) {
}) })
} }
// ExportInflightHeld reports how many deliveries the engine currently
// owns, so a test can prove ownership is released rather than leaked.
func (e *Engine) ExportInflightHeld() int {
return e.inflight.held()
}
// ExportRetainDelivery takes the first reference on a delivery, as the
// queueing side does. It lets a test put a delivery into the state a
// worker or a full channel would, without running the pool.
func (e *Engine) ExportRetainDelivery(deliveryID string) bool {
return e.inflight.retainIdle(deliveryID)
}
// ExportRecoverRetryingDeliveries exposes recoverRetryingDeliveries.
func (e *Engine) ExportRecoverRetryingDeliveries(
webhookDB *gorm.DB, webhookID string,
) {
e.recoverRetryingDeliveries(webhookDB, webhookID)
}
// ExportDeliveryCh returns the delivery channel. // ExportDeliveryCh returns the delivery channel.
func (e *Engine) ExportDeliveryCh() chan Task { func (e *Engine) ExportDeliveryCh() chan Task {
return e.deliveryCh return e.deliveryCh

110
internal/delivery/inflight.go
View File

@@ -1,110 +0,0 @@
package delivery
import "sync"
// inflightSet records which deliveries the engine currently owns.
//
// A delivery is owned from the moment a task for it is handed to a
// channel or to a retry timer until the engine has no further plan for
// it in memory. Restart recovery and both arms of the periodic sweep
// re-dispatch only deliveries the set does not hold, which is what
// makes them exact rather than a guess about how long a row has sat at
// pending.
//
// This replaces reasoning from timestamps. A delivery's row says
// pending from creation until its outcome is written, which covers
// four different situations — never dispatched, waiting in a channel,
// being attempted right now, and genuinely stranded — and no column
// distinguishes them. Only the engine knows which, and it knows
// exactly. `deliveryChannelSize` is 10000 against 10 workers, so a
// perfectly healthy delivery can wait far longer than any age bound
// worth setting before its attempt even begins; an age bound alone
// re-sends it. See
// https://git.eeqj.de/sneak/webhooker/issues/256.
//
// In-memory state is sufficient because a data directory admits one
// process: internal/datadir takes an flock on it at startup and a
// second instance refuses to run. Deliveries owned by a process that
// died are not in any successor's set, and restart recovery is what
// picks those up.
//
// References are counted rather than held as a plain set because
// ownership outlives the worker that took it. A target that schedules
// a retry from inside Deliver adds a reference while the worker still
// holds one, so the delivery stays owned across the gap between the
// worker returning and the timer firing — the window in which a sweep
// would otherwise find the row at retrying and send it again.
//
// The zero value is ready to use, and the Engine holds one by value.
// That is deliberate: an engine built by a constructor that forgot to
// initialise this would not refuse to re-dispatch anything, and the
// symptom would be duplicate deliveries rather than a failure anybody
// notices.
type inflightSet struct {
mu sync.Mutex
ids map[string]int
}
// retain adds a reference to a delivery the caller already knows the
// engine owns, so that ownership survives the current holder letting
// go. It cannot fail.
func (s *inflightSet) retain(deliveryID string) {
s.mu.Lock()
defer s.mu.Unlock()
if s.ids == nil {
s.ids = make(map[string]int)
}
s.ids[deliveryID]++
}
// retainIdle takes the first reference on a delivery, and reports
// whether it got it. It fails when the engine already owns the
// delivery, which is what makes two claimants — restart recovery and
// the sweep run concurrently, or two sweep arms — mutually exclusive
// rather than merely atomic.
func (s *inflightSet) retainIdle(deliveryID string) bool {
s.mu.Lock()
defer s.mu.Unlock()
if s.ids[deliveryID] > 0 {
return false
}
if s.ids == nil {
s.ids = make(map[string]int)
}
s.ids[deliveryID] = 1
return true
}
// release drops one reference. The delivery becomes eligible for
// re-dispatch again once the last one goes.
func (s *inflightSet) release(deliveryID string) {
s.mu.Lock()
defer s.mu.Unlock()
n := s.ids[deliveryID] - 1
if n <= 0 {
delete(s.ids, deliveryID)
return
}
s.ids[deliveryID] = n
}
// held reports how many deliveries the engine currently owns. It
// exists so a test can assert that ownership is released rather than
// leaked: a reference that is never dropped hides its delivery from
// every sweep for the life of the process, which is the one way this
// mechanism can fail silently.
func (s *inflightSet) held() int {
s.mu.Lock()
defer s.mu.Unlock()
return len(s.ids)
}

428
internal/delivery/inflight_test.go
View File

@@ -1,428 +0,0 @@
package delivery_test
import (
"context"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
// These tests pin the rule that decides whether a delivery may be
// handed back to a worker: the engine re-dispatches only what it does
// not already own. Age alone is not that rule — a healthy delivery
// waiting in a 10000-deep channel is old and must not be re-sent. See
// https://git.eeqj.de/sneak/webhooker/issues/256.
// fSweepSetup seeds the main database with the webhook row the sweep
// enumerates, and returns the setup.
func fSweepSetup(
t *testing.T, targetID, name string,
) iSetup {
t.Helper()
s := newISetup(t)
iCreateTarget(t, s.MainDB, targetID,
s.WebhookID, name,
database.TargetTypeLog, "", 0,
)
require.NoError(t, s.MainDB.Create(&database.Webhook{
BaseModel: database.BaseModel{ID: s.WebhookID},
UserID: uuid.New().String(),
Name: name,
}).Error)
return s
}
// fDrain collects every task the engine has queued.
//
// Every caller drives the dispatch paths synchronously and has already
// waited for them to return, so anything they queued is in the channel
// by now. The short grace covers nothing but scheduler jitter, and is
// kept small because one of these tests runs the drain forty times.
func fDrain(e *delivery.Engine) []delivery.Task {
var out []delivery.Task
for {
select {
case task := <-e.ExportDeliveryCh():
out = append(out, task)
case task := <-e.ExportRetryCh():
out = append(out, task)
case <-time.After(25 * time.Millisecond):
return out
}
}
}
// TestArchiveHandleIsWAL closes the last gap in the durability
// evidence: the main and per-webhook tiers each assert their journal
// mode on a live handle, and the archive tier gets its settings from
// the same code path but nothing checked the running file.
func TestArchiveHandleIsWAL(t *testing.T) {
t.Parallel()
w := delivery.NewExportArchiveWriter(
filepath.Join(t.TempDir(), "archive-wal.db"),
archiveTestLogger(), 0,
)
require.NoError(t, w.Open(0))
var mode string
row := w.DB().Raw("pragma journal_mode").Row()
require.NoError(t, row.Scan(&mode))
assert.Equal(t, "wal", strings.ToLower(mode))
var busy string
row = w.DB().Raw("pragma busy_timeout").Row()
require.NoError(t, row.Scan(&busy))
assert.Equal(t, "10000", busy)
}
// TestSweepLeavesAQueuedDeliveryAlone is the case the age bound cannot
// see. The delivery is queued and untouched, so its row is arbitrarily
// old and still perfectly healthy; only ownership distinguishes it
// from a stranded one.
func TestSweepLeavesAQueuedDeliveryAlone(t *testing.T) {
t.Parallel()
targetID := uuid.New().String()
s := fSweepSetup(t, targetID, "queued")
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"queued":true}`,
)
d := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusPending,
)
rAgePending(t, s.WebhookDB, d.ID)
// Queued exactly as the receiver queues it, and never dequeued:
// no workers are running in this engine.
s.Engine.Notify([]delivery.Task{{
DeliveryID: d.ID,
EventID: event.ID,
WebhookID: s.WebhookID,
TargetID: targetID,
}})
require.Equal(t, 1, s.Engine.ExportInflightHeld())
s.Engine.ExportSweepWebhookRetries(
context.Background(), s.WebhookID,
)
tasks := fDrain(s.Engine)
assert.Len(
t, tasks, 1,
"the sweep must not queue a delivery that is "+
"already waiting for a worker",
)
}
// TestRecoveryAndSweepDoNotDoubleDispatch drives the two entry points
// the engine starts concurrently against one aged pending row. Before
// ownership they both dispatched it.
func TestRecoveryAndSweepDoNotDoubleDispatch(t *testing.T) {
t.Parallel()
targetID := uuid.New().String()
s := fSweepSetup(t, targetID, "racing")
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"racing":true}`,
)
ctx := context.Background()
for range 40 {
d := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusPending,
)
rAgePending(t, s.WebhookDB, d.ID)
var wg sync.WaitGroup
wg.Go(func() {
s.Engine.ExportRecoverPendingDeliveries(
ctx, s.WebhookDB, s.WebhookID,
)
})
wg.Go(func() {
s.Engine.ExportSweepWebhookRetries(
ctx, s.WebhookID,
)
})
wg.Wait()
tasks := fDrain(s.Engine)
require.Len(
t, tasks, 1,
"delivery %s dispatched %d times",
d.ID, len(tasks),
)
// No worker runs in this engine, so the reference the winner
// took is never released and earlier iterations' deliveries
// stay owned — which is itself the property under test, since
// both paths see them on every subsequent pass.
}
}
// TestConcurrentClaimsOfOneDeliveryYieldOneOwner exercises the
// exclusion directly, rather than arguing it from a SQL predicate.
func TestConcurrentClaimsOfOneDeliveryYieldOneOwner(
t *testing.T,
) {
t.Parallel()
eng := newISetup(t).Engine
deliveryID := uuid.New().String()
var (
wg sync.WaitGroup
mu sync.Mutex
won int
)
for range 64 {
wg.Go(func() {
if eng.ExportRetainDelivery(deliveryID) {
mu.Lock()
won++
mu.Unlock()
}
})
}
wg.Wait()
assert.Equal(t, 1, won)
assert.Equal(t, 1, eng.ExportInflightHeld())
}
// TestOwnershipIsReleasedAfterDelivery guards the other direction: a
// leaked reference hides a delivery from every sweep for the life of
// the process.
func TestOwnershipIsReleasedAfterDelivery(t *testing.T) {
t.Parallel()
s := newISetup(t)
targetID := uuid.New().String()
iCreateTarget(t, s.MainDB, targetID,
s.WebhookID, "released",
database.TargetTypeLog, "", 0,
)
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"released":true}`,
)
d := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusPending,
)
s.Engine.ExportStart()
defer func() {
require.NoError(
t, s.Engine.ExportStop(context.Background()),
)
}()
body := `{"released":true}`
s.Engine.Notify([]delivery.Task{{
DeliveryID: d.ID,
EventID: event.ID,
WebhookID: s.WebhookID,
TargetID: targetID,
TargetName: "released",
TargetType: database.TargetTypeLog,
Body: &body,
EntrypointID: event.EntrypointID,
}})
iWaitForDelivered(t, s.WebhookDB, d.ID)
assert.Eventually(
t,
func() bool {
return s.Engine.ExportInflightHeld() == 0
},
2*time.Second, 20*time.Millisecond,
"the delivery stayed owned after it was delivered",
)
}
// TestRetryingRecoverySkipsASuccessfulResult is the retrying-side twin
// of the pending reconcile. A second attempt that reached the receiver
// and whose status write then failed sits at retrying holding a
// successful result, and re-sending it is the same duplicate.
func TestRetryingRecoverySkipsASuccessfulResult(t *testing.T) {
t.Parallel()
targetID := uuid.New().String()
s := fSweepSetup(t, targetID, "retry-settled")
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"retry":true}`,
)
d := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusRetrying,
)
rSeedResult(t, s.WebhookDB, d.ID, 1, false)
rSeedResult(t, s.WebhookDB, d.ID, 2, true)
s.Engine.ExportRecoverRetryingDeliveries(
s.WebhookDB, s.WebhookID,
)
assert.Empty(
t, fDrain(s.Engine),
"a retrying delivery holding a successful result "+
"must not be sent again",
)
iAssertStatus(
t, s.WebhookDB, d.ID,
database.DeliveryStatusDelivered,
)
}
// TestRetryingSweepSkipsASuccessfulResult is the same rule on the
// periodic sweep's retrying arm.
func TestRetryingSweepSkipsASuccessfulResult(t *testing.T) {
t.Parallel()
targetID := uuid.New().String()
s := fSweepSetup(t, targetID, "retry-swept")
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"swept":true}`,
)
d := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusRetrying,
)
rSeedResult(t, s.WebhookDB, d.ID, 1, false)
rSeedResult(t, s.WebhookDB, d.ID, 2, true)
s.Engine.ExportSweepWebhookRetries(
context.Background(), s.WebhookID,
)
assert.Empty(t, fDrain(s.Engine))
iAssertStatus(
t, s.WebhookDB, d.ID,
database.DeliveryStatusDelivered,
)
var attempts int64
require.NoError(t, s.WebhookDB.
Model(&database.DeliveryResult{}).
Where("delivery_id = ?", d.ID).
Count(&attempts).Error)
assert.Equal(
t, int64(2), attempts,
"settling must not invent an attempt",
)
}
// TestScheduledRetryIsNotSweptDuringBackoff closes the window between
// a target scheduling a retry and the timer firing. The row says
// retrying and nothing is running, which is exactly what an orphaned
// retry looks like from the database.
func TestScheduledRetryIsNotSweptDuringBackoff(t *testing.T) {
t.Parallel()
targetID := uuid.New().String()
s := fSweepSetup(t, targetID, "backoff")
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"backoff":true}`,
)
d := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusRetrying,
)
s.Engine.ExportScheduleRetry(delivery.Task{
DeliveryID: d.ID,
EventID: event.ID,
WebhookID: s.WebhookID,
TargetID: targetID,
AttemptNum: 2,
}, time.Hour)
require.Equal(t, 1, s.Engine.ExportInflightHeld())
s.Engine.ExportSweepWebhookRetries(
context.Background(), s.WebhookID,
)
assert.Empty(
t, fDrain(s.Engine),
"the sweep must not duplicate a retry that is "+
"already scheduled",
)
}
// TestRedispatchStampsTheRow pins the cadence control: a stranded
// delivery that has just been handed out is not selected again by the
// next tick a minute later.
func TestRedispatchStampsTheRow(t *testing.T) {
t.Parallel()
targetID := uuid.New().String()
s := fSweepSetup(t, targetID, "stamped")
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"stamped":true}`,
)
d := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusPending,
)
rAgePending(t, s.WebhookDB, d.ID)
ctx := context.Background()
s.Engine.ExportSweepWebhookRetries(ctx, s.WebhookID)
require.Len(t, fDrain(s.Engine), 1)
var row database.Delivery
require.NoError(t, s.WebhookDB.
First(&row, "id = ?", d.ID).Error)
assert.WithinDuration(
t, time.Now(), row.UpdatedAt, time.Minute,
"a re-dispatched delivery must be stamped so the "+
"next tick does not select it again",
)
}

19
internal/delivery/metrics_test.go
View File

@@ -229,13 +229,6 @@ func mExhaustRetries(t *testing.T, s iSetup) {
body := event.Body body := event.Body
cfg := iHTTPConfig(ts.URL) cfg := iHTTPConfig(ts.URL)
// The retry below is only run if its target still exists; see
// https://git.eeqj.de/sneak/webhooker/issues/107.
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "metrics-fail",
database.TargetTypeHTTP, cfg, 2,
)
first := iTask( first := iTask(
d, event, s.WebhookID, targetID, d, event, s.WebhookID, targetID,
"metrics-fail", cfg, 2, 1, &body, "metrics-fail", cfg, 2, 1, &body,
@@ -296,13 +289,6 @@ func TestDeliveryMetrics_CircuitBreakerGauge(t *testing.T) {
// rather than the budget is what stops the delivery. // rather than the budget is what stops the delivery.
maxRetries := delivery.ExportDefaultFailureThreshold + 5 maxRetries := delivery.ExportDefaultFailureThreshold + 5
// The retries below are only run if their target still exists;
// see https://git.eeqj.de/sneak/webhooker/issues/107.
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "metrics-trip",
database.TargetTypeHTTP, cfg, maxRetries,
)
first := iTask( first := iTask(
d, event, s.WebhookID, targetID, d, event, s.WebhookID, targetID,
"metrics-trip", cfg, maxRetries, 1, &body, "metrics-trip", cfg, maxRetries, 1, &body,
@@ -367,11 +353,6 @@ func TestDeliveryMetrics_BreakerBlockedIsNotAnAttempt(
cfg := iHTTPConfig(ts.URL) cfg := iHTTPConfig(ts.URL)
maxRetries := delivery.ExportDefaultFailureThreshold + 5 maxRetries := delivery.ExportDefaultFailureThreshold + 5
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "metrics-blocked",
database.TargetTypeHTTP, cfg, maxRetries,
)
first := iTask( first := iTask(
d, event, s.WebhookID, targetID, d, event, s.WebhookID, targetID,
"metrics-blocked", cfg, maxRetries, 1, &body, "metrics-blocked", cfg, maxRetries, 1, &body,

39
internal/delivery/recovery_durability_test.go
View File

@@ -163,8 +163,19 @@ func TestRecoveryContinuesTheAttemptNumbering(t *testing.T) {
func TestSweepRecoversStrandedPending(t *testing.T) { func TestSweepRecoversStrandedPending(t *testing.T) {
t.Parallel() t.Parallel()
s := newISetup(t)
targetID := uuid.New().String() targetID := uuid.New().String()
s := fSweepSetup(t, targetID, "stranded")
iCreateTarget(t, s.MainDB, targetID,
s.WebhookID, "stranded",
database.TargetTypeLog, "", 0,
)
require.NoError(t, s.MainDB.Create(&database.Webhook{
BaseModel: database.BaseModel{ID: s.WebhookID},
UserID: uuid.New().String(),
Name: "stranded",
}).Error)
event := iSeedEvent( event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"stranded":true}`, t, s.WebhookDB, s.WebhookID, `{"stranded":true}`,
@@ -215,8 +226,19 @@ func TestSweepRecoversStrandedPending(t *testing.T) {
func TestSweepClaimsAStrandedDeliveryOnlyOnce(t *testing.T) { func TestSweepClaimsAStrandedDeliveryOnlyOnce(t *testing.T) {
t.Parallel() t.Parallel()
s := newISetup(t)
targetID := uuid.New().String() targetID := uuid.New().String()
s := fSweepSetup(t, targetID, "claimed")
iCreateTarget(t, s.MainDB, targetID,
s.WebhookID, "claimed",
database.TargetTypeLog, "", 0,
)
require.NoError(t, s.MainDB.Create(&database.Webhook{
BaseModel: database.BaseModel{ID: s.WebhookID},
UserID: uuid.New().String(),
Name: "claimed",
}).Error)
event := iSeedEvent( event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"claimed":true}`, t, s.WebhookDB, s.WebhookID, `{"claimed":true}`,
@@ -266,8 +288,19 @@ func TestSweepSettlesStrandedPendingWithoutResending(
) { ) {
t.Parallel() t.Parallel()
s := newISetup(t)
targetID := uuid.New().String() targetID := uuid.New().String()
s := fSweepSetup(t, targetID, "settled")
iCreateTarget(t, s.MainDB, targetID,
s.WebhookID, "settled",
database.TargetTypeLog, "", 0,
)
require.NoError(t, s.MainDB.Create(&database.Webhook{
BaseModel: database.BaseModel{ID: s.WebhookID},
UserID: uuid.New().String(),
Name: "settled",
}).Error)
event := iSeedEvent( event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"settled":true}`, t, s.WebhookDB, s.WebhookID, `{"settled":true}`,

45
internal/delivery/target_config_view.go
View File

@@ -23,51 +23,21 @@ type ConfigField struct {
Value string Value string
} }
// deletedNameSuffix marks the name of a target that no longer
// exists. Deletes are soft and delivery history outlives the
// target, so the event log shows names of targets that are gone;
// an operator reading one needs to know it cannot be delivered
// to, replayed to, or configured.
const deletedNameSuffix = " (deleted)"
// TargetView is the display-safe projection of a target for // TargetView is the display-safe projection of a target for
// the UI. It deliberately has no raw configuration field, so // the UI. It deliberately has no raw configuration field, so
// no template — present or future — can render the stored // no template — present or future — can render the stored
// blob. // blob.
type TargetView struct { type TargetView struct {
ID string ID string
Name string Name string
// Deleted reports that this target's row is soft deleted.
// Only views built for historical display carry it set:
// every other projection is of a live row.
Deleted bool
Type database.TargetType Type database.TargetType
Active bool Active bool
Config []ConfigField Config []ConfigField
} }
// DisplayName is the name to render, marked when the target has
// been deleted. Templates showing a name against historical data
// must use it rather than Name, which stays the stored name.
func (v TargetView) DisplayName() string {
if v.Deleted {
return v.Name + deletedNameSuffix
}
return v.Name
}
// NewTargetViews projects targets for rendering, replacing // NewTargetViews projects targets for rendering, replacing
// each stored configuration blob with named, display-safe // each stored configuration blob with named, display-safe
// fields. // fields.
//
// A soft-deleted row projects exactly as a live one does, minus
// the deleted marker on its name: masking is a property of the
// projection, not of the row's state, so a deleted target's
// credential is as unreachable from a template as a live
// target's.
func NewTargetViews( func NewTargetViews(
targets []database.Target, targets []database.Target,
) []TargetView { ) []TargetView {
@@ -77,12 +47,11 @@ func NewTargetViews(
t := &targets[i] t := &targets[i]
views = append(views, TargetView{ views = append(views, TargetView{
ID: t.ID, ID: t.ID,
Name: t.Name, Name: t.Name,
Deleted: t.DeletedAt.Valid, Type: t.Type,
Type: t.Type, Active: t.Active,
Active: t.Active, Config: targetConfigFields(t),
Config: targetConfigFields(t),
}) })
} }

68
internal/delivery/target_config_view_test.go
View File

@@ -2,11 +2,9 @@ package delivery_test
import ( import (
"testing" "testing"
"time"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database" "sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery" "sneak.berlin/go/webhooker/internal/delivery"
) )
@@ -19,14 +17,6 @@ const (
slackWebhookURL = "https://hooks.slack.com" + slackWebhookURL = "https://hooks.slack.com" +
slackSecretPath slackSecretPath
// slackMaskedURL is what a Slack webhook URL renders as
// once masked: scheme and host, path elided.
slackMaskedURL = "https://hooks.slack.com/..."
// slackTargetName is the target name the Slack projection
// tests use.
slackTargetName = "slack-target"
viewExampleOrigin = "https://example.com" viewExampleOrigin = "https://example.com"
viewExampleHook = viewExampleOrigin + "/hook" viewExampleHook = viewExampleOrigin + "/hook"
viewMaskedOrigin = viewExampleOrigin + "/..." viewMaskedOrigin = viewExampleOrigin + "/..."
@@ -43,7 +33,7 @@ func TestMaskedWebhookURL(t *testing.T) {
}{ }{
"slack webhook": { "slack webhook": {
url: slackWebhookURL, url: slackWebhookURL,
want: slackMaskedURL, want: "https://hooks.slack.com/...",
}, },
"query string dropped": { "query string dropped": {
url: viewExampleOrigin + "/a?token=secret", url: viewExampleOrigin + "/a?token=secret",
@@ -135,61 +125,23 @@ func viewFor(
return views[0] return views[0]
} }
// TestNewTargetViews_DeletedTarget proves the projection marks func TestNewTargetViews_Slack(t *testing.T) {
// a soft-deleted target's name and masks its configuration by
// the same rules a live target's is. Delivery history outlives
// the target it names, so this projection is what an operator
// reads about a target that no longer exists.
func TestNewTargetViews_DeletedTarget(t *testing.T) {
t.Parallel() t.Parallel()
target := slackTarget() view := viewFor(t, database.Target{
target.DeletedAt = gorm.DeletedAt{ Name: "slack-target",
Time: time.Now(),
Valid: true,
}
view := viewFor(t, target)
assert.True(t, view.Deleted)
assert.Equal(t, slackTargetName, view.Name)
assert.Equal(
t, slackTargetName+" (deleted)", view.DisplayName(),
)
assert.Equal(
t,
map[string]string{"Webhook URL": slackMaskedURL},
fieldMap(view.Config),
)
}
// slackTarget is the live Slack target the projection tests
// share.
func slackTarget() database.Target {
return database.Target{
Name: slackTargetName,
Type: database.TargetTypeSlack, Type: database.TargetTypeSlack,
Active: true, Active: true,
Config: `{"webhookUrl":"` + Config: `{"webhookUrl":"` +
slackWebhookURL + `"}`, slackWebhookURL + `"}`,
} })
}
func TestNewTargetViews_Slack(t *testing.T) {
t.Parallel()
view := viewFor(t, slackTarget())
assert.Equal(t, slackTargetName, view.Name)
// A live target is never marked, so the marker cannot
// reach a name that still exists.
assert.False(t, view.Deleted)
assert.Equal(t, slackTargetName, view.DisplayName())
assert.Equal(t, "slack-target", view.Name)
assert.Equal( assert.Equal(
t, t,
map[string]string{"Webhook URL": slackMaskedURL}, map[string]string{
"Webhook URL": "https://hooks.slack.com/...",
},
fieldMap(view.Config), fieldMap(view.Config),
) )
} }
@@ -260,7 +212,7 @@ func TestNewTargetViews_HTTPMasksDestinationURL(t *testing.T) {
assert.Equal( assert.Equal(
t, t,
slackMaskedURL, "https://hooks.slack.com/...",
fields["Destination URL"], fields["Destination URL"],
) )

142
internal/delivery/target_http_secret_test.go Normal file
View File

@@ -0,0 +1,142 @@
package delivery_test
import (
"context"
"encoding/json"
"net/http"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
"sneak.berlin/go/webhooker/internal/signature"
)
// gitlabDeliverySecret is the shared secret the entrypoint in these
// tests is configured with. No outbound request may contain it.
const gitlabDeliverySecret = "QQDELIVERYSECRETQQ"
// receivedEventHeaders builds the Event.Headers value the receiver
// stores for an inbound request, by running the request's headers
// through the same sanitizer the receive path uses. Going through
// signature.SanitizeHeaders rather than a literal is the point of
// the test: it joins the two egresses at the field they share, so a
// regression at either end shows up here.
func receivedEventHeaders(
t *testing.T,
scheme database.SignatureScheme,
inbound http.Header,
) string {
t.Helper()
ep := &database.Entrypoint{
SignatureScheme: scheme,
SignatureSecret: gitlabDeliverySecret,
}
encoded, err := json.Marshal(
signature.SanitizeHeaders(ep, inbound),
)
require.NoError(t, err)
return string(encoded)
}
// TestApplyRequestHeadersDropsInboundCredential proves a delivery to
// an HTTP target does not carry the GitLab shared secret.
//
// isForwardableHeader is a blocklist of hop-by-hop names, so it
// forwards X-Gitlab-Token like any other header; what keeps the
// secret out of the outbound request is that the receiver never
// stored it. Handing a target operator the token would hand them the
// ability to forge requests to the entrypoint it authenticates,
// which is the one control the receiver has.
func TestApplyRequestHeadersDropsInboundCredential(t *testing.T) {
t.Parallel()
inbound := http.Header{}
inbound.Set(signature.HeaderGitLab, gitlabDeliverySecret)
inbound.Set("X-Gitlab-Event", "Push Hook")
event := &database.Event{
Headers: receivedEventHeaders(
t, database.SignatureSchemeGitLab, inbound,
),
ContentType: "application/json",
}
req, err := http.NewRequestWithContext(
context.Background(),
http.MethodPost,
"https://target.example.com/hook",
http.NoBody,
)
require.NoError(t, err)
delivery.ExportApplyRequestHeaders(
req, event, &delivery.HTTPTargetConfig{},
)
assert.Empty(
t,
req.Header.Values(signature.HeaderGitLab),
"the shared secret header must not reach a target",
)
// Header.Values canonicalises, so a differently-cased spelling
// would be caught above; this catches the value arriving under
// some other name.
for name, values := range req.Header {
for _, v := range values {
assert.NotContains(
t, v, gitlabDeliverySecret,
"secret present in outbound header %s", name,
)
}
}
// The rest of the sender's headers still arrive. A fix that
// dropped everything would pass the assertions above while
// breaking delivery.
assert.Equal(
t,
"Push Hook",
req.Header.Get("X-Gitlab-Event"),
)
}
// TestApplyRequestHeadersKeepsGitHubDigest proves the stripping is
// scoped to headers that carry the secret itself. GitHub's
// X-Hub-Signature-256 is an HMAC over the body, so a target can be
// shown it without being handed the key.
func TestApplyRequestHeadersKeepsGitHubDigest(t *testing.T) {
t.Parallel()
const digest = "sha256=deadbeef"
inbound := http.Header{}
inbound.Set(signature.HeaderGitHub, digest)
event := &database.Event{
Headers: receivedEventHeaders(
t, database.SignatureSchemeGitHub, inbound,
),
}
req, err := http.NewRequestWithContext(
context.Background(),
http.MethodPost,
"https://target.example.com/hook",
http.NoBody,
)
require.NoError(t, err)
delivery.ExportApplyRequestHeaders(
req, event, &delivery.HTTPTargetConfig{},
)
assert.Equal(
t, digest, req.Header.Get(signature.HeaderGitHub),
)
}

7
internal/delivery/target_slack.go
View File

@@ -231,15 +231,10 @@ func FormatSlackMessage(
event.ContentType, event.ContentType,
) )
timestamp := "unknown"
if !event.CreatedAt.IsZero() {
timestamp = event.CreatedAt.UTC().Format(time.RFC3339)
}
fmt.Fprintf( fmt.Fprintf(
&b, &b,
"*Timestamp:* `%s`\n", "*Timestamp:* `%s`\n",
timestamp, event.CreatedAt.UTC().Format(time.RFC3339),
) )
fmt.Fprintf( fmt.Fprintf(

531
internal/delivery/terminal_state_test.go
View File

@@ -1,531 +0,0 @@
package delivery_test
import (
"context"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
// The two terminal-state gaps of
// https://git.eeqj.de/sneak/webhooker/issues/107: a delivery failed
// with nothing in its event log to say why, and a retrying delivery
// whose target was deleted, which used to keep sending and then never
// terminalise.
// tUnknownType is a target type no build implements. It stands in for
// a target whose type was written by a build that knew a type this one
// does not.
const tUnknownType = database.TargetType("pubsub")
// tSeedDeletedTarget creates a target, a retrying delivery against it
// with one recorded failed attempt, and then deletes the target the
// way the source page does.
//
// It asserts the delete is soft, because that is the whole reason the
// engine could not tell a deleted target from a target id that never
// named a row: the surviving row is invisible to a scoped read.
func tSeedDeletedTarget(
t *testing.T,
s iSetup,
name, url string,
) string {
t.Helper()
targetID := uuid.New().String()
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, name,
database.TargetTypeHTTP, iHTTPConfig(url), 5,
)
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"target":"deleted"}`,
)
d := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusRetrying,
)
iSeedFailedResult(t, s.WebhookDB, d.ID)
require.NoError(t, s.MainDB.Delete(
&database.Target{}, "id = ?", targetID,
).Error)
var scoped, unscoped int64
require.NoError(t, s.MainDB.
Model(&database.Target{}).
Where("id = ?", targetID).
Count(&scoped).Error)
require.NoError(t, s.MainDB.Unscoped().
Model(&database.Target{}).
Where("id = ?", targetID).
Count(&unscoped).Error)
require.Zero(t, scoped,
"the deleted target is still visible to a scoped read",
)
require.Equal(t, int64(1), unscoped,
"the delete was hard, so this test proves nothing about "+
"the soft-delete case it exists for",
)
return d.ID
}
// tLastResult returns a delivery's final recorded attempt, asserting
// the expected number of them.
func tLastResult(
t *testing.T,
s iSetup,
deliveryID string,
want int,
) database.DeliveryResult {
t.Helper()
results := iResults(t, s.WebhookDB, deliveryID)
require.Len(t, results, want)
return results[want-1]
}
// --- 1. A failure with nothing recorded ---
func TestProcessDelivery_UnknownTargetType_RecordsWhy(
t *testing.T,
) {
t.Parallel()
s := newISetup(t)
targetID := uuid.New().String()
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"unknown":"type"}`,
)
seeded := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusPending,
)
target := database.Target{
Name: "mystery",
Type: tUnknownType,
Config: iHTTPConfig("http://example.com/hook"),
}
target.ID = targetID
d := database.Delivery{
EventID: event.ID,
TargetID: targetID,
Status: database.DeliveryStatusPending,
Event: event,
Target: target,
}
d.ID = seeded.ID
body := event.Body
task := iTask(
seeded, event, s.WebhookID, targetID, "mystery",
target.Config, 0, 1, &body,
)
task.TargetType = tUnknownType
s.Engine.ExportProcessDelivery(
context.Background(), s.WebhookDB, &d, &task,
)
iAssertStatus(
t, s.WebhookDB, d.ID, database.DeliveryStatusFailed,
)
last := tLastResult(t, s, d.ID, 1)
assert.False(t, last.Success)
assert.Equal(t, 1, last.AttemptNum)
assert.Contains(t, last.Error, string(tUnknownType),
"the recorded reason does not name the offending type",
)
}
// --- 2. A retrying delivery whose target is gone ---
func TestRecoverSingleRetry_TargetDeleted(t *testing.T) {
t.Parallel()
s := newISetup(t)
iCreateWebhook(
t, s.MainDB, s.WebhookID, "deleted-target-recovery",
)
deliveryID := tSeedDeletedTarget(
t, s, "gone-on-recovery", "http://example.com/hook",
)
s.Engine.ExportRecoverWebhookDeliveries(
context.Background(), s.WebhookID,
)
iAssertStatus(
t, s.WebhookDB, deliveryID,
database.DeliveryStatusFailed,
)
last := tLastResult(t, s, deliveryID, 2)
assert.False(t, last.Success)
assert.Equal(t, 2, last.AttemptNum)
assert.Contains(t, last.Error, "gone-on-recovery")
assert.Contains(t, last.Error, "was deleted")
assert.Empty(t, s.Engine.ExportRetryCh(),
"a delivery whose target is gone was rescheduled",
)
assert.Zero(t, s.Engine.ExportInflightHeld(),
"the terminal path leaked its ownership reference",
)
}
func TestSweepSingleRetry_TargetDeleted(t *testing.T) {
t.Parallel()
s := newISetup(t)
iCreateWebhook(
t, s.MainDB, s.WebhookID, "deleted-target-sweep",
)
deliveryID := tSeedDeletedTarget(
t, s, "gone-on-sweep", "http://example.com/hook",
)
// Twice, because the bug was an error the sweep repeated every
// minute for the life of the database: the second sweep must
// find nothing left to do.
s.Engine.ExportSweepWebhookRetries(
context.Background(), s.WebhookID,
)
s.Engine.ExportSweepWebhookRetries(
context.Background(), s.WebhookID,
)
iAssertStatus(
t, s.WebhookDB, deliveryID,
database.DeliveryStatusFailed,
)
last := tLastResult(t, s, deliveryID, 2)
assert.Contains(t, last.Error, "gone-on-sweep")
assert.Contains(t, last.Error, "was deleted")
assert.Empty(t, s.Engine.ExportRetryCh())
assert.Zero(t, s.Engine.ExportInflightHeld())
}
// TestSweepSingleRetry_TargetNeverExisted covers the other half of the
// soft-delete distinction: an id with no row at all, deleted or
// otherwise, must not be reported as something the operator deleted.
func TestSweepSingleRetry_TargetNeverExisted(t *testing.T) {
t.Parallel()
s := newISetup(t)
iCreateWebhook(
t, s.MainDB, s.WebhookID, "target-never-existed",
)
targetID := uuid.New().String()
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"target":"absent"}`,
)
d := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusRetrying,
)
iSeedFailedResult(t, s.WebhookDB, d.ID)
s.Engine.ExportSweepWebhookRetries(
context.Background(), s.WebhookID,
)
iAssertStatus(
t, s.WebhookDB, d.ID, database.DeliveryStatusFailed,
)
last := tLastResult(t, s, d.ID, 2)
assert.Contains(t, last.Error, targetID)
assert.Contains(t, last.Error, "no longer exists")
assert.NotContains(t, last.Error, "was deleted",
"an id that never named a row was reported as a deletion",
)
}
// TestFailMissingTargetRetry_WritesNoTargetRow holds the new terminal
// path to the same rule as the existing one: no target row, and so no
// plaintext target config, may be written into the per-webhook event
// database. See https://git.eeqj.de/sneak/webhooker/issues/206.
func TestFailMissingTargetRetry_WritesNoTargetRow(
t *testing.T,
) {
t.Parallel()
s := newISetup(t)
iCreateWebhook(
t, s.MainDB, s.WebhookID, "no-target-row-deleted",
)
hookURL := "https://hooks.slack.com/services/T00/B00/x"
deliveryID := tSeedDeletedTarget(
t, s, "credential-bearing", hookURL,
)
s.Engine.ExportSweepWebhookRetries(
context.Background(), s.WebhookID,
)
iAssertStatus(
t, s.WebhookDB, deliveryID,
database.DeliveryStatusFailed,
)
var configs []string
require.NoError(t, s.WebhookDB.
Table("targets").
Pluck("config", &configs).Error)
assert.Empty(t, configs,
"the deleted-target terminal path wrote a target row "+
"into the per-webhook event database",
)
}
// --- 3. The scheduled retry chain ---
// tRetryChainSetup wires a counting sink and a retrying delivery
// against a live target pointing at it, and returns the task a
// scheduled retry would carry — config and all, snapshotted as
// ScheduleRetry snapshots it.
func tRetryChainSetup(
t *testing.T,
s iSetup,
name string,
hits *atomic.Int64,
) (delivery.Task, string) {
t.Helper()
ts := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
hits.Add(1)
w.WriteHeader(http.StatusOK)
},
))
t.Cleanup(ts.Close)
iCreateWebhook(t, s.MainDB, s.WebhookID, name)
targetID := uuid.New().String()
cfg := iHTTPConfig(ts.URL)
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, name,
database.TargetTypeHTTP, cfg, 5,
)
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"chain":"retry"}`,
)
d := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusRetrying,
)
iSeedFailedResult(t, s.WebhookDB, d.ID)
body := event.Body
return iTask(
d, event, s.WebhookID, targetID, name, cfg, 5, 2, &body,
), targetID
}
// TestProcessRetryTask_TargetDeleted_MakesNoAttempt is the half the
// deployability audit found worse than filed: terminalising on
// recovery and sweep alone leaves the already-scheduled timer chain
// running, and it holds the target's configuration from before the
// deletion, so it goes on sending to a destination that was removed.
func TestProcessRetryTask_TargetDeleted_MakesNoAttempt(
t *testing.T,
) {
t.Parallel()
s := newISetup(t)
var hits atomic.Int64
task, targetID := tRetryChainSetup(
t, s, "gone-mid-chain", &hits,
)
require.NoError(t, s.MainDB.Delete(
&database.Target{}, "id = ?", targetID,
).Error)
s.Engine.ExportProcessRetryTask(
context.Background(), &task,
)
assert.Zero(t, hits.Load(),
"a scheduled retry fired at a target the operator "+
"had already deleted",
)
iAssertStatus(
t, s.WebhookDB, task.DeliveryID,
database.DeliveryStatusFailed,
)
last := tLastResult(t, s, task.DeliveryID, 2)
assert.False(t, last.Success)
assert.Contains(t, last.Error, "was deleted")
assert.Zero(t, s.Engine.ExportInflightHeld())
}
// TestProcessRetryTask_TargetPresent_StillDelivers is the guard's
// mutation check: a liveness check that refused every retry would pass
// the test above and break every retry there is.
func TestProcessRetryTask_TargetPresent_StillDelivers(
t *testing.T,
) {
t.Parallel()
s := newISetup(t)
var hits atomic.Int64
task, _ := tRetryChainSetup(t, s, "still-there", &hits)
s.Engine.ExportProcessRetryTask(
context.Background(), &task,
)
assert.Equal(t, int64(1), hits.Load())
iAssertStatus(
t, s.WebhookDB, task.DeliveryID,
database.DeliveryStatusDelivered,
)
}
// TestProcessRetryTask_TargetUnreadable_StillDelivers pins the other
// half of the guard: only a target that is confirmed gone stops a
// retry. A main database that cannot be read is a transient fault, and
// a guard that abandoned deliveries on one would be a worse bug than
// the one it fixes.
func TestProcessRetryTask_TargetUnreadable_StillDelivers(
t *testing.T,
) {
t.Parallel()
s := newISetup(t)
var hits atomic.Int64
task, _ := tRetryChainSetup(t, s, "unreadable-main", &hits)
sqlDB, err := s.MainDB.DB()
require.NoError(t, err)
require.NoError(t, sqlDB.Close())
s.Engine.ExportProcessRetryTask(
context.Background(), &task,
)
assert.Equal(t, int64(1), hits.Load(),
"a retry was abandoned because the main database "+
"could not be read, not because its target was gone",
)
iAssertStatus(
t, s.WebhookDB, task.DeliveryID,
database.DeliveryStatusDelivered,
)
}
// TestRecoverSingleRetry_TargetUnreadable_LeavesDeliveryAlone is the
// same rule on the recovery path. A read failure that is not
// "record not found" must leave every retrying delivery of every
// webhook exactly as it was.
func TestRecoverSingleRetry_TargetUnreadable_LeavesDeliveryAlone(
t *testing.T,
) {
t.Parallel()
s := newISetup(t)
iCreateWebhook(
t, s.MainDB, s.WebhookID, "unreadable-on-recovery",
)
targetID := uuid.New().String()
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "healthy",
database.TargetTypeHTTP,
iHTTPConfig("http://example.com/hook"), 5,
)
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"still":"retrying"}`,
)
d := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusRetrying,
)
iSeedFailedResult(t, s.WebhookDB, d.ID)
sqlDB, err := s.MainDB.DB()
require.NoError(t, err)
require.NoError(t, sqlDB.Close())
s.Engine.ExportRecoverRetryingDeliveries(
s.WebhookDB, s.WebhookID,
)
iAssertStatus(
t, s.WebhookDB, d.ID,
database.DeliveryStatusRetrying,
)
assert.Len(t, iResults(t, s.WebhookDB, d.ID), 1,
"an unreadable main database produced a terminal "+
"failure row",
)
assert.Zero(t, s.Engine.ExportInflightHeld())
}

342
internal/handlers/entrypoint_secret_test.go Normal file
View File

@@ -0,0 +1,342 @@
package handlers_test
import (
"net/http"
"net/http/httptest"
"net/url"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
// submitEntrypointSecret posts the signature configuration form for
// an entrypoint and returns the recorder.
func submitEntrypointSecret(
t *testing.T,
h *handlers.Handlers,
cookies []*http.Cookie,
webhookID, entrypointID, scheme, secret string,
) *httptest.ResponseRecorder {
t.Helper()
form := url.Values{}
form.Set("signature_scheme", scheme)
form.Set("secret", secret)
req := formRequest(
"/source/"+webhookID+"/entrypoints/"+
entrypointID+"/secret",
cookies,
form,
map[string]string{
paramSourceID: webhookID,
entrypointIDParam: entrypointID,
},
)
w := httptest.NewRecorder()
h.HandleEntrypointSecret().ServeHTTP(w, req)
return w
}
// reloadEntrypoint reads an entrypoint back from the database,
// including the columns the model keeps out of JSON.
func reloadEntrypoint(
t *testing.T,
db *database.Database,
id string,
) database.Entrypoint {
t.Helper()
var ep database.Entrypoint
require.NoError(
t, db.DB().Where("id = ?", id).First(&ep).Error,
)
return ep
}
// TestEntrypointSecretSetRotateAndRemove walks the whole lifecycle
// the UI has to support: turning verification on, rotating the secret
// to a new value, and turning it back off.
func TestEntrypointSecretSetRotateAndRemove(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
)
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
wh := seedWebhook(t, db)
ep := seedSignedEntrypoint(
t, db, wh.ID, database.SignatureSchemeNone, "",
)
// Set.
w := submitEntrypointSecret(
t, h, cookies, wh.ID, ep.ID, "github", inboundSecret,
)
require.Equal(t, http.StatusSeeOther, w.Code)
stored := reloadEntrypoint(t, db, ep.ID)
assert.Equal(
t, database.SignatureSchemeGitHub, stored.SignatureScheme,
)
assert.Equal(t, inboundSecret, stored.SignatureSecret)
assert.True(t, stored.SignatureConfigured())
// Rotate: a new secret and a different scheme in one submission.
// The new value is submitted with surrounding whitespace, the way
// a secret pasted out of a password manager arrives; storing that
// verbatim would make every later request fail verification with
// nothing visible on either side to explain it.
const rotated = "QQROTATEDSECRETQQ"
w = submitEntrypointSecret(
t, h, cookies, wh.ID, ep.ID, "gitlab", " "+rotated+"\t",
)
require.Equal(t, http.StatusSeeOther, w.Code)
stored = reloadEntrypoint(t, db, ep.ID)
assert.Equal(
t, database.SignatureSchemeGitLab, stored.SignatureScheme,
)
assert.Equal(t, rotated, stored.SignatureSecret)
// Remove. The secret has to go with the scheme: a stored
// credential nothing reads is one more copy to leak.
w = submitEntrypointSecret(t, h, cookies, wh.ID, ep.ID, "", "")
require.Equal(t, http.StatusSeeOther, w.Code)
stored = reloadEntrypoint(t, db, ep.ID)
assert.Equal(
t, database.SignatureSchemeNone, stored.SignatureScheme,
)
assert.Empty(t, stored.SignatureSecret)
assert.False(t, stored.SignatureConfigured())
}
// TestEntrypointSecretRejectsBadInput proves the form cannot create a
// row the receiver would later have to refuse. Both rejections leave
// the stored configuration untouched rather than half-applied.
func TestEntrypointSecretRejectsBadInput(t *testing.T) {
t.Parallel()
cases := []struct {
name string
scheme string
secret string
}{
{
name: "unsupported scheme",
scheme: "stripe",
secret: inboundSecret,
},
{
name: "scheme with no secret",
scheme: "github",
secret: "",
},
{
// Whitespace is stripped, so a secret of spaces is an
// empty one.
name: "scheme with blank secret",
scheme: "github",
secret: " ",
},
}
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
)
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
for _, tc := range cases {
wh := seedWebhook(t, db)
ep := seedSignedEntrypoint(
t, db, wh.ID,
database.SignatureSchemeGitLab, inboundSecret,
)
w := submitEntrypointSecret(
t, h, cookies, wh.ID, ep.ID, tc.scheme, tc.secret,
)
assert.Equal(
t, http.StatusBadRequest, w.Code, "case %s", tc.name,
)
stored := reloadEntrypoint(t, db, ep.ID)
assert.Equal(
t,
database.SignatureSchemeGitLab,
stored.SignatureScheme,
"case %s", tc.name,
)
assert.Equal(
t, inboundSecret, stored.SignatureSecret,
"case %s", tc.name,
)
}
}
// TestEntrypointSecretRequiresOwnership proves the configuration
// endpoint is bound by the same ownership check as the rest of the
// webhook's pages: another user's entrypoint is a 404, and the secret
// is not touched.
func TestEntrypointSecretRequiresOwnership(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
)
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
ep := seedSignedEntrypoint(
t, db, wh.ID,
database.SignatureSchemeGitLab, inboundSecret,
)
stranger := authenticatedCookies(
t, sess, "someone-else", "someoneelse",
)
w := submitEntrypointSecret(
t, h, stranger, wh.ID, ep.ID, "github", "hijacked",
)
assert.Equal(t, http.StatusNotFound, w.Code)
assert.Equal(
t,
inboundSecret,
reloadEntrypoint(t, db, ep.ID).SignatureSecret,
)
}
// TestHandleSourceDetail_MasksEntrypointSecret is the regression test
// for the credential on the entrypoint: the page has to say that
// verification is configured and which header carries it, without the
// secret itself ever reaching the rendered HTML.
func TestHandleSourceDetail_MasksEntrypointSecret(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
)
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedSignedEntrypoint(
t, db, wh.ID,
database.SignatureSchemeGitHub, inboundSecret,
)
body := renderSourceDetailPage(t, h, sess, wh.ID)
assert.NotContains(t, body, inboundSecret)
assert.Contains(t, body, "GitHub")
assert.Contains(t, body, "X-Hub-Signature-256")
}
// TestEntrypointViewsDropTheSecret pins the projection itself, so the
// barrier survives a template rewrite that stops rendering the field
// the page test above looks at.
func TestEntrypointViewsDropTheSecret(t *testing.T) {
t.Parallel()
views := handlers.NewEntrypointViews([]database.Entrypoint{
{
Path: "p1",
Active: true,
SignatureScheme: database.SignatureSchemeGitHub,
SignatureSecret: inboundSecret,
},
{
Path: "p2",
},
{
// Half a configuration. The receiver 500s every request
// to this row, so the UI must not call it unverified.
Path: "p2a",
SignatureScheme: database.SignatureSchemeGitLab,
},
{
// The other half.
Path: "p2b",
SignatureSecret: inboundSecret,
},
{
// A scheme this build does not know: described as
// unavailable, never echoed back.
Path: "p3",
SignatureScheme: database.SignatureScheme("stripe"),
SignatureSecret: inboundSecret,
},
})
require.Len(t, views, 5)
assert.True(t, views[0].Configured)
assert.Equal(t, "GitHub", views[0].SchemeLabel)
assert.Equal(t, "X-Hub-Signature-256", views[0].SchemeHeader)
assert.False(t, views[1].Configured)
assert.Equal(t, "not verified", views[1].SchemeLabel)
assert.Empty(t, views[1].SchemeHeader)
for _, v := range []handlers.EntrypointView{views[2], views[3]} {
assert.False(t, v.Configured)
assert.Equal(t, "misconfigured", v.SchemeLabel)
assert.Empty(t, v.SchemeHeader)
}
assert.True(t, views[4].Configured)
assert.Equal(t, "(unavailable)", views[4].SchemeLabel)
// The struct has no field that could carry the secret, so this
// fails to compile rather than fails at runtime if one is added
// and populated. The assertion covers the labels it derives.
for _, v := range views {
assert.NotContains(t, v.SchemeLabel, inboundSecret)
assert.NotContains(t, v.SchemeHeader, inboundSecret)
assert.NotContains(t, string(v.Scheme), inboundSecret)
}
}

73
internal/handlers/entrypoint_view.go
View File

@@ -2,18 +2,56 @@ package handlers
import ( import (
"sneak.berlin/go/webhooker/internal/database" "sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/signature"
) )
// signatureUnavailable is what an entrypoint's scheme renders as when
// the stored value is not one this build supports. The stored string
// is never echoed as a fallback: it is operator-supplied and the row
// is already in a state the receiver refuses, so the UI says so
// rather than inventing a description for it.
const signatureUnavailable = "(unavailable)"
// signatureNotVerified is the label for an entrypoint that performs
// no inbound verification.
const signatureNotVerified = "not verified"
// signatureMisconfigured is the label for a row holding one half of
// the scheme/secret pair. The receiver answers every request to such
// an entrypoint 500, so calling it "not verified" would describe a
// receiver that is refusing everything as one that is accepting
// everything. The form cannot create the state; a hand-edited
// database or a downgrade past a scheme can.
const signatureMisconfigured = "misconfigured"
// EntrypointView is the display-safe projection of an entrypoint for // EntrypointView is the display-safe projection of an entrypoint for
// the UI, in the same way delivery.TargetView is one for a target. // the UI. It deliberately has no secret field, so no template —
// present or future — can render the shared secret, in the same way
// delivery.TargetView keeps a target's stored credential away from
// one.
type EntrypointView struct { type EntrypointView struct {
ID string ID string
Path string Path string
Description string Description string
Active bool Active bool
// Configured reports whether inbound requests to this entrypoint
// are verified.
Configured bool
// Scheme is the stored scheme, carried so the form can preselect
// it. It names an algorithm, not a secret.
Scheme database.SignatureScheme
// SchemeLabel and SchemeHeader describe the configured scheme for
// display: the sender's name, and the header its signature
// arrives in.
SchemeLabel string
SchemeHeader string
} }
// NewEntrypointViews projects entrypoints for rendering. // NewEntrypointViews projects entrypoints for rendering, dropping the
// shared secret on the way.
func NewEntrypointViews( func NewEntrypointViews(
entrypoints []database.Entrypoint, entrypoints []database.Entrypoint,
) []EntrypointView { ) []EntrypointView {
@@ -22,12 +60,31 @@ func NewEntrypointViews(
for i := range entrypoints { for i := range entrypoints {
e := &entrypoints[i] e := &entrypoints[i]
views = append(views, EntrypointView{ view := EntrypointView{
ID: e.ID, ID: e.ID,
Path: e.Path, Path: e.Path,
Description: e.Description, Description: e.Description,
Active: e.Active, Active: e.Active,
}) Configured: e.SignatureConfigured(),
Scheme: e.SignatureScheme,
SchemeLabel: signatureNotVerified,
SchemeHeader: "",
}
switch {
case view.Configured:
view.SchemeLabel = signatureUnavailable
info, ok := signature.Info(e.SignatureScheme)
if ok {
view.SchemeLabel = info.Label
view.SchemeHeader = info.Header
}
case e.SignatureHalfConfigured():
view.SchemeLabel = signatureMisconfigured
}
views = append(views, view)
} }
return views return views

4
internal/handlers/event_resubmit.go
View File

@@ -84,6 +84,10 @@ const resubmitColumns = "id, entrypoint_id, method, headers, " +
// is the stored EVENT. The response bodies and headers the original // is the stored EVENT. The response bodies and headers the original
// deliveries received stay where they are. // deliveries received stay where they are.
// //
// Inbound signature verification is deliberately not re-run. There is
// no inbound signature to check on a copy the operator submits; the
// route is authenticated and CSRF-protected as an operator action.
//
// Resubmitting the same event repeatedly is supported and is the point // Resubmitting the same event repeatedly is supported and is the point
// of the feature, so replay's in-flight refusal is deliberately not // of the feature, so replay's in-flight refusal is deliberately not
// applied here. The route's rate limit is what bounds a held-down // applied here. The route's rate limit is what bounds a held-down

38
internal/handlers/footer_version_test.go
View File

@@ -1,38 +0,0 @@
package handlers_test
import (
"testing"
"github.com/stretchr/testify/assert"
"sneak.berlin/go/webhooker/internal/globals"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
// The footer in base.html falls back to the literal "dev" when the
// template data carries no version, which is what every page rendered
// while nothing supplied one. The operator uses the footer to tell
// which build is live, so it has to carry the stamped value.
func TestFooterReportsStampedVersion(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
g *globals.Globals
)
app := newTestApp(t, &h, &sess, &g)
app.RequireStart()
t.Cleanup(app.RequireStop)
g.Version = "v9.9.9-test"
html := renderPage(t, h, sess, "login.html", map[string]any{
dataKeyError: "",
})
assert.Contains(t, html, "<span>v9.9.9-test</span>")
assert.NotContains(t, html, "<span>dev</span>")
}

9
internal/handlers/handlers.go
View File

@@ -184,7 +184,6 @@ type UserInfo struct {
type templateDataWrapper struct { type templateDataWrapper struct {
User *UserInfo User *UserInfo
CSRFToken string CSRFToken string
Version string
Data any Data any
} }
@@ -235,16 +234,9 @@ func (s *Handlers) renderTemplate(
userInfo := s.getUserInfo(r) userInfo := s.getUserInfo(r)
csrfToken := middleware.CSRFToken(r) csrfToken := middleware.CSRFToken(r)
// The footer in base.html renders .Version. Every page reaches it
// through here, so this is the one place that has to supply it;
// left unset, the footer falls back to its literal "dev" and the
// UI reports a build that is not the one running.
version := s.params.Globals.Version
if m, ok := data.(map[string]any); ok { if m, ok := data.(map[string]any); ok {
m["User"] = userInfo m["User"] = userInfo
m["CSRFToken"] = csrfToken m["CSRFToken"] = csrfToken
m["Version"] = version
s.executeTemplate(w, tmpl, m) s.executeTemplate(w, tmpl, m)
return return
@@ -253,7 +245,6 @@ func (s *Handlers) renderTemplate(
wrapper := templateDataWrapper{ wrapper := templateDataWrapper{
User: userInfo, User: userInfo,
CSRFToken: csrfToken, CSRFToken: csrfToken,
Version: version,
Data: data, Data: data,
} }

226
internal/handlers/source_delete_test.go
View File

@@ -2,7 +2,6 @@ package handlers_test
import ( import (
"context" "context"
"errors"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"os" "os"
@@ -12,7 +11,6 @@ import (
"github.com/go-chi/chi" "github.com/go-chi/chi"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gorm.io/gorm"
"gorm.io/gorm/clause" "gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database" "sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers" "sneak.berlin/go/webhooker/internal/handlers"
@@ -75,77 +73,6 @@ func seedTarget(
return tgt return tgt
} }
// errInjectedDelete is the failure failDeleteOnTable reports
// from a delete statement.
var errInjectedDelete = errors.New("injected delete failure")
// seedEntrypoint inserts an entrypoint for a webhook.
func seedEntrypoint(
t *testing.T,
db *database.Database,
webhookID string,
) {
t.Helper()
ep := &database.Entrypoint{
WebhookID: webhookID,
Path: "ep-" + webhookID,
Active: true,
}
require.NoError(
t,
db.DB().Omit(clause.Associations).Create(ep).Error,
)
}
// countRows counts the live (not soft-deleted) rows of a model
// matching column = value.
func countRows(
t *testing.T,
db *database.Database,
model any,
column, value string,
) int64 {
t.Helper()
var n int64
require.NoError(
t,
db.DB().Model(model).
Where(column+" = ?", value).
Count(&n).Error,
)
return n
}
// failDeleteOnTable makes every delete against the named table
// fail the way a database-level error does: the statement
// reports an error but leaves the surrounding transaction
// usable, so a caller that does not check it can go on to
// commit the statements that did succeed.
func failDeleteOnTable(
t *testing.T,
db *database.Database,
table string,
) {
t.Helper()
require.NoError(t, db.DB().Callback().Delete().
Before("gorm:delete").
Register(
"test:fail_delete_"+table,
func(tx *gorm.DB) {
if tx.Statement.Table == table {
_ = tx.AddError(errInjectedDelete)
}
},
),
)
}
// archivePathFor returns the archive database path the // archivePathFor returns the archive database path the
// delivery engine would use for a webhook: beside the webhook's // delivery engine would use for a webhook: beside the webhook's
// event database in the data directory. // event database in the data directory.
@@ -282,159 +209,6 @@ func TestHandleSourceDelete_KeepsArchiveFile(t *testing.T) {
) )
} }
// TestHandleSourceDelete_FailedDeleteKeepsEverything proves
// that a failing delete statement loses nothing: the
// configuration is rolled back whole, the event database
// survives, and the operator is told the deletion failed
// instead of being redirected as though it worked.
func TestHandleSourceDelete_FailedDeleteKeepsEverything(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
mgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &mgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedEntrypoint(t, db, wh.ID)
seedTarget(t, db, wh.ID, database.TargetTypeDatabase)
require.NoError(t, mgr.CreateDB(wh.ID))
eventDBPath := mgr.DBPath(wh.ID)
require.FileExists(t, eventDBPath)
// The entrypoint delete runs first and succeeds; the target
// delete then fails, which is what the whole transaction has
// to be rolled back over.
failDeleteOnTable(t, db, "targets")
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/delete",
cookies,
map[string]string{paramSourceID: wh.ID},
)
w := httptest.NewRecorder()
h.HandleSourceDelete().ServeHTTP(w, req)
assert.Equal(
t, http.StatusInternalServerError, w.Code,
"a failed deletion must be reported, not redirected",
)
assert.Empty(
t, w.Header().Get("Location"),
"a failed deletion must not redirect to /sources",
)
assert.Equal(
t, int64(1),
countRows(t, db, &database.Webhook{}, "id", wh.ID),
"the webhook must survive a failed deletion",
)
assert.Equal(
t, int64(1),
countRows(
t, db, &database.Entrypoint{}, "webhook_id", wh.ID,
),
"the entrypoint delete must be rolled back",
)
assert.Equal(
t, int64(1),
countRows(
t, db, &database.Target{}, "webhook_id", wh.ID,
),
"the target must survive a failed deletion",
)
assert.FileExists(
t, eventDBPath,
"event history must not be destroyed when the "+
"configuration delete did not commit",
)
}
// TestHandleSourceDelete_RemovesConfigAndEventDatabase is the
// positive control for the rollback above: an ordinary deletion
// still removes the webhook, its children and its event
// database.
func TestHandleSourceDelete_RemovesConfigAndEventDatabase(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
mgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &mgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedEntrypoint(t, db, wh.ID)
seedTarget(t, db, wh.ID, database.TargetTypeDatabase)
require.NoError(t, mgr.CreateDB(wh.ID))
eventDBPath := mgr.DBPath(wh.ID)
require.FileExists(t, eventDBPath)
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
req := postRequest(
"/source/"+wh.ID+"/delete",
cookies,
map[string]string{paramSourceID: wh.ID},
)
w := httptest.NewRecorder()
h.HandleSourceDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
assert.Equal(t, "/sources", w.Header().Get("Location"))
assert.Equal(
t, int64(0),
countRows(t, db, &database.Webhook{}, "id", wh.ID),
)
assert.Equal(
t, int64(0),
countRows(
t, db, &database.Entrypoint{}, "webhook_id", wh.ID,
),
)
assert.Equal(
t, int64(0),
countRows(
t, db, &database.Target{}, "webhook_id", wh.ID,
),
)
assert.NoFileExists(
t, eventDBPath,
"a successful deletion removes the event database",
)
}
// TestHandleTargetDelete_EvictsWhenLastDatabaseTargetGone // TestHandleTargetDelete_EvictsWhenLastDatabaseTargetGone
// proves that removing the last database target releases the // proves that removing the last database target releases the
// archive writer. // archive writer.

283
internal/handlers/source_detail_baseurl_test.go
View File

@@ -1,283 +0,0 @@
package handlers_test
import (
"context"
"crypto/tls"
"net/http"
"net/http/httptest"
"regexp"
"testing"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
// The only two schemes a rendered entrypoint URL may carry,
// whatever the request claimed.
const (
schemeHTTPS = "https"
schemeHTTP = "http"
)
// entrypointURLPattern captures the entrypoint URL the source
// detail page renders, which is the operator-visible product of
// BaseURL. Asserting on the extracted string rather than on a
// substring of the page proves the raw header value cannot reach
// the scheme by any route.
var entrypointURLPattern = regexp.MustCompile(
`<code id="entrypoint-url-[^"]*"[^>]*>([^<]*)</code>`,
)
// baseURLFixture is one started app plus the webhook whose
// entrypoint URL the BaseURL cases read.
type baseURLFixture struct {
handlers *handlers.Handlers
session *session.Session
webhook string
path string
}
// newBaseURLFixture starts the app and seeds a webhook with one
// entrypoint.
func newBaseURLFixture(t *testing.T) *baseURLFixture {
t.Helper()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
)
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedEntrypoint(t, db, wh.ID)
return &baseURLFixture{
handlers: h,
session: sess,
webhook: wh.ID,
path: "ep-" + wh.ID,
}
}
// entrypointURL renders the source detail page for the fixture's
// webhook over a request the caller shapes, and returns the
// entrypoint URL as an operator would copy it.
func (f *baseURLFixture) entrypointURL(
t *testing.T,
host string,
shape func(*http.Request),
) string {
t.Helper()
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet,
"/source/"+f.webhook,
nil,
)
req.Host = host
shape(req)
for _, c := range authenticatedCookies(
t, f.session, deleteTestUserID, deleteTestUsername,
) {
req.AddCookie(c)
}
rctx := chi.NewRouteContext()
rctx.URLParams.Add(paramSourceID, f.webhook)
req = req.WithContext(
context.WithValue(
req.Context(), chi.RouteCtxKey, rctx,
),
)
w := httptest.NewRecorder()
f.handlers.HandleSourceDetail().ServeHTTP(w, req)
require.Equal(t, http.StatusOK, w.Code)
match := entrypointURLPattern.FindStringSubmatch(w.Body.String())
require.Len(
t, match, 2,
"the page must render exactly one entrypoint URL",
)
return match[1]
}
// forwardedProto returns a request shaper setting
// X-Forwarded-Proto, or leaving the request alone for "".
func forwardedProto(value string) func(*http.Request) {
return func(r *http.Request) {
if value == "" {
return
}
r.Header.Set("X-Forwarded-Proto", value)
}
}
// baseURLCase is one X-Forwarded-Proto spelling and the scheme
// the rendered entrypoint URL owes it.
type baseURLCase struct {
name string
header string
scheme string
why string
}
// baseURLCases enumerate the spellings a proxy really emits. The
// scheme is only ever http or https: the header value itself is
// never a scheme, however it is spelled.
func baseURLCases() []baseURLCase {
return []baseURLCase{
{
name: "lowercase",
header: schemeHTTPS,
scheme: schemeHTTPS,
why: "the ordinary spelling",
},
{
name: "uppercase",
header: "HTTPS",
scheme: schemeHTTPS,
why: "the token is case-insensitive; the scheme " +
"in a copyable URL is not",
},
{
name: "chain with plaintext inner hop",
header: "https, http",
scheme: schemeHTTPS,
why: "a chained proxy appends its hop; the " +
"leftmost element faces the client",
},
{
name: "chain of two TLS hops",
header: "https,https",
scheme: schemeHTTPS,
why: "appended chain with no space after the comma",
},
{
name: "trailing space",
header: "https ",
scheme: schemeHTTPS,
why: "whitespace is not part of the token",
},
{
name: "plaintext",
header: schemeHTTP,
scheme: schemeHTTP,
why: "the negative control: the proxy reports plaintext",
},
{
name: "no header",
header: "",
scheme: schemeHTTP,
why: "a plaintext request asserting nothing is http",
},
{
name: "garbage token",
header: "javascript:alert(1)//",
scheme: schemeHTTP,
why: "anything that is not https is not TLS, and " +
"the token never becomes the scheme",
},
}
}
// TestSourceDetailBaseURL_ForwardedProtoSpellings is the
// regression test for the entrypoint URL an operator pastes into
// the sending system: a header spelling that used to land in the
// scheme verbatim produced a URL no sender could deliver to.
func TestSourceDetailBaseURL_ForwardedProtoSpellings(t *testing.T) {
t.Parallel()
const host = "hooks.example.com"
fixture := newBaseURLFixture(t)
for _, tc := range baseURLCases() {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
assert.Equal(
t,
tc.scheme+"://"+host+"/webhook/"+fixture.path,
fixture.entrypointURL(
t, host, forwardedProto(tc.header),
),
"X-Forwarded-Proto %q: %s", tc.header, tc.why,
)
})
}
}
// TestSourceDetailBaseURL_DirectTLSBeatsPlaintextHeader pins the
// precedence the old code had backwards: it let any present
// header overwrite what the connection itself proved, so a
// direct-TLS request behind a proxy reporting http rendered an
// http URL.
func TestSourceDetailBaseURL_DirectTLSBeatsPlaintextHeader(
t *testing.T,
) {
t.Parallel()
const host = "hooks.example.com"
fixture := newBaseURLFixture(t)
got := fixture.entrypointURL(t, host, func(r *http.Request) {
r.TLS = &tls.ConnectionState{}
r.Header.Set("X-Forwarded-Proto", "http")
})
assert.Equal(
t,
"https://"+host+"/webhook/"+fixture.path,
got,
"a connection this process terminated with TLS "+
"outranks a header claiming plaintext",
)
}
// TestSourceDetailBaseURL_KeepsHostAuthority pins the host half
// of the URL: it is taken from the request unchanged, so the
// deployments that do not sit on port 443 still get a URL that
// works. Constraining the host would break exactly these.
func TestSourceDetailBaseURL_KeepsHostAuthority(t *testing.T) {
t.Parallel()
fixture := newBaseURLFixture(t)
hosts := []string{
"hooks.example.com:8443",
"[2001:db8::1]:8443",
"internal-host",
}
for _, host := range hosts {
t.Run(host, func(t *testing.T) {
t.Parallel()
assert.Equal(
t,
"https://"+host+"/webhook/"+fixture.path,
fixture.entrypointURL(
t, host, forwardedProto("HTTPS"),
),
"the authority must survive verbatim, port and all",
)
})
}
}

144
internal/handlers/source_logs_deleted_target_test.go
View File

@@ -1,144 +0,0 @@
package handlers_test
import (
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
// deletedMarker is the suffix the event log appends to the name
// of a target that no longer exists.
const deletedMarker = " (deleted)"
// deleteTargetThroughHandler removes a target through the real
// deletion handler, so the test soft-deletes exactly the way the
// UI does rather than by writing the timestamp itself.
func deleteTargetThroughHandler(
t *testing.T,
h *handlers.Handlers,
sess *session.Session,
webhookID, targetID string,
) {
t.Helper()
req := postRequest(
"/source/"+webhookID+"/targets/"+targetID+"/delete",
authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
),
map[string]string{
paramSourceID: webhookID,
paramTargetID: targetID,
},
)
w := httptest.NewRecorder()
h.HandleTargetDelete().ServeHTTP(w, req)
require.Equal(t, http.StatusSeeOther, w.Code)
}
// TestHandleSourceLogs_NamesDeletedTarget proves a delivery
// produced by a since-deleted target still names it on the event
// log, marked as deleted.
//
// Deletes are soft and deliveries carry no foreign key to the
// target row, so the history outlives the target. Against a
// scoped lookup the delivery resolves to a zero view and the page
// renders ": delivered" with nothing saying what it was delivered
// to.
func TestHandleSourceLogs_NamesDeletedTarget(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
tgt := seedTarget(t, db, wh.ID, database.TargetTypeLog)
seedDeliveredEvent(t, dbMgr, wh.ID, tgt.ID)
// The control: the name is on the page while the target
// lives, and is not yet marked as deleted.
before := renderSourceLogsPage(t, h, sess, wh.ID)
assert.Contains(t, before, tgt.Name)
assert.NotContains(t, before, tgt.Name+deletedMarker)
deleteTargetThroughHandler(t, h, sess, wh.ID, tgt.ID)
after := renderSourceLogsPage(t, h, sess, wh.ID)
assert.Contains(
t, after, tgt.Name+deletedMarker,
"a delivery from a deleted target must keep its name, "+
"marked as no longer existing",
)
assert.Contains(
t, after, "delivered",
"the delivery history itself must survive the delete",
)
}
// TestHandleSourceLogs_MasksDeletedTargetConfig proves that
// naming a deleted target does not widen what the page shows of
// it: its stored configuration stays masked by exactly the rules
// a live target's is.
//
// The lookup behind the name reads soft-deleted rows, so it
// carries a full target row — credential blob included — into the
// place a zero value used to sit. The projection to TargetView is
// what keeps that blob away from the template, and it must hold
// for a deleted row too.
func TestHandleSourceLogs_MasksDeletedTargetConfig(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
tgt := seedConfiguredTarget(
t, db, wh.ID,
database.TargetTypeSlack,
`{"webhookUrl":"`+slackWebhookURL+`"}`,
)
seedDeliveredEvent(t, dbMgr, wh.ID, tgt.ID)
deleteTargetThroughHandler(t, h, sess, wh.ID, tgt.ID)
body := renderSourceLogsPage(t, h, sess, wh.ID)
assert.NotContains(t, body, slackSecretPath)
assert.NotContains(t, body, "T00000000")
assert.NotContains(t, body, "B00000000")
assert.NotContains(
t, body, "XXXXXXXXXXXXXXXXXXXXXXXX",
)
assert.NotContains(t, body, "webhookUrl")
// The name is there; only the credential is not.
assert.Contains(t, body, tgt.Name+deletedMarker)
}

309
internal/handlers/source_management.go
View File

@@ -13,7 +13,7 @@ import (
"gorm.io/gorm" "gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database" "sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery" "sneak.berlin/go/webhooker/internal/delivery"
"sneak.berlin/go/webhooker/internal/reqtls" "sneak.berlin/go/webhooker/internal/signature"
) )
// WebhookListItem holds data for the webhook list view. // WebhookListItem holds data for the webhook list view.
@@ -428,28 +428,31 @@ func (h *Handlers) renderSourceDetail(
} }
} }
scheme := "http" host := r.Host
if reqtls.IsTLS(r) { scheme := "https"
scheme = "https"
if r.TLS == nil {
scheme = "http"
} }
// The host is the client's Host header, unvalidated. It is if fwdProto := r.Header.Get("X-Forwarded-Proto"); fwdProto != "" {
// inert only because source_detail.html renders BaseURL as scheme = fwdProto
// text inside a <code> element; putting it in an href or any }
// other URL context needs it constrained first.
baseURL := scheme + "://" + r.Host
// The template calls Webhook methods, which take pointer // The template calls Webhook methods, which take pointer
// receivers; html/template cannot address a value stored in a map. // receivers; html/template cannot address a value stored in a map.
data := map[string]any{ data := map[string]any{
tmplKeyWebhook: &webhook, tmplKeyWebhook: &webhook,
// Targets are projected to a display-safe view: a // Entrypoints and targets are both projected to
// target's stored config blob holds a credential, and it // display-safe views: an entrypoint carries the shared
// must never reach a template. // secret its senders sign with and a target's stored
"Entrypoints": NewEntrypointViews(entrypoints), // config blob holds a credential, and neither must ever
"Targets": delivery.NewTargetViews(targets), // reach a template.
"Events": events, "Entrypoints": NewEntrypointViews(entrypoints),
"BaseURL": baseURL, "Targets": delivery.NewTargetViews(targets),
"SignatureSchemes": signature.Schemes(),
"Events": events,
"BaseURL": scheme + "://" + host,
} }
h.renderTemplate(w, r, "source_detail.html", data) h.renderTemplate(w, r, "source_detail.html", data)
@@ -622,26 +625,42 @@ func (h *Handlers) deleteWebhookResources(
webhook database.Webhook, webhook database.Webhook,
userID string, userID string,
) { ) {
// The configuration delete commits before the event database tx := h.db.DB().Begin()
// is touched. No transaction spans the main database and the if tx.Error != nil {
// filesystem, so one side has to go first: committing the h.log.Error(
// configuration first means a later failure leaves an unused "failed to begin transaction",
// event database file on disk, while removing the event "error", tx.Error,
// database first would mean a failed commit destroys the )
// history of a webhook that still exists. A leftover file can http.Error(
// be removed by hand; deleted history cannot be recovered. w, "Internal server error",
err := h.commitWebhookDeletion(&webhook) http.StatusInternalServerError,
if err != nil { )
h.serverError(w, "failed to delete webhook", err)
return return
} }
h.log.Info( tx.Where(
"webhook deleted", "webhook_id = ?", webhook.ID,
"webhook_id", webhook.ID, ).Delete(&database.Entrypoint{})
"user_id", userID,
) tx.Where(
"webhook_id = ?", webhook.ID,
).Delete(&database.Target{})
tx.Delete(&webhook)
err := tx.Commit().Error
if err != nil {
h.log.Error(
"failed to commit deletion", "error", err,
)
http.Error(
w, "Internal server error",
http.StatusInternalServerError,
)
return
}
// Release the delivery engine's per-webhook archiving state // Release the delivery engine's per-webhook archiving state
// so a deleted webhook's archive writer (and any handle open // so a deleted webhook's archive writer (and any handle open
@@ -652,63 +671,22 @@ func (h *Handlers) deleteWebhookResources(
err = h.dbMgr.DeleteDB(webhook.ID) err = h.dbMgr.DeleteDB(webhook.ID)
if err != nil { if err != nil {
// The configuration is committed, so the webhook is gone, h.log.Error(
// but its event database file is still on disk with "failed to delete webhook event database",
// nothing referencing it. Report the failure rather than "webhook_id", webhook.ID,
// redirecting as though everything succeeded: the file "error", err,
// needs removing by hand, and the logged error names it.
h.serverError(
w, "failed to delete webhook event database", err,
) )
return
} }
h.log.Info(
"webhook deleted",
"webhook_id", webhook.ID,
"user_id", userID,
)
http.Redirect(w, r, "/sources", http.StatusSeeOther) http.Redirect(w, r, "/sources", http.StatusSeeOther)
} }
// commitWebhookDeletion soft-deletes a webhook's entrypoints,
// targets and the webhook row in one transaction. Every
// statement is checked and any failure rolls the whole
// transaction back, so a caller that gets an error knows the
// configuration is untouched and the event database must be
// left alone.
func (h *Handlers) commitWebhookDeletion(
webhook *database.Webhook,
) error {
tx := h.db.DB().Begin()
if tx.Error != nil {
return tx.Error
}
err := tx.Where(
"webhook_id = ?", webhook.ID,
).Delete(&database.Entrypoint{}).Error
if err != nil {
tx.Rollback()
return err
}
err = tx.Where(
"webhook_id = ?", webhook.ID,
).Delete(&database.Target{}).Error
if err != nil {
tx.Rollback()
return err
}
err = tx.Delete(webhook).Error
if err != nil {
tx.Rollback()
return err
}
return tx.Commit().Error
}
// evictArchiveWriter asks the delivery engine to drop its // evictArchiveWriter asks the delivery engine to drop its
// cached archive writer for a webhook, closing the archive file // cached archive writer for a webhook, closing the archive file
// handle. // handle.
@@ -882,16 +860,11 @@ func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
// //
// The load is Unscoped because deleting a target only soft // The load is Unscoped because deleting a target only soft
// deletes the row while its deliveries survive in the // deletes the row while its deliveries survive in the
// per-webhook database. Both halves of the map need those rows: // per-webhook database: a scoped load leaves those deliveries
// a scoped load leaves an old delivery with a zero redactor, // with a zero redactor, which renders their response bodies
// which renders its response bodies unredacted, and with a zero // unredacted. Only the redactor half of the map is built from
// view, which renders its target as a blank name. // deleted rows. The view half, which is what the page lists,
// // stays scoped.
// This map is historical display only. It is built for the event
// log page and reaches nothing but DeliveryView.Target: the
// target list on the source detail page, the edit form and the
// replay path each resolve targets themselves, and a deleted row
// is refused there as before.
func (h *Handlers) loadTargetMap( func (h *Handlers) loadTargetMap(
webhookID string, webhookID string,
) (map[string]eventLogTarget, error) { ) (map[string]eventLogTarget, error) {
@@ -907,18 +880,21 @@ func (h *Handlers) loadTargetMap(
targetMap := make( targetMap := make(
map[string]eventLogTarget, len(targets), map[string]eventLogTarget, len(targets),
) )
live := make([]database.Target, 0, len(targets))
for i := range targets { for i := range targets {
targetMap[targets[i].ID] = eventLogTarget{ targetMap[targets[i].ID] = eventLogTarget{
Redactor: delivery.NewRedactor(&targets[i]), Redactor: delivery.NewRedactor(&targets[i]),
} }
if !targets[i].DeletedAt.Valid {
live = append(live, targets[i])
}
} }
// The views come from NewTargetViews rather than being // The views come from NewTargetViews rather than being
// rebuilt here, so the masking rules stay in one place and a // rebuilt here, so the masking rules stay in one place.
// deleted target's configuration is masked by the same code for _, v := range delivery.NewTargetViews(live) {
// that masks a live one's.
for _, v := range delivery.NewTargetViews(targets) {
entry := targetMap[v.ID] entry := targetMap[v.ID]
entry.View = v entry.View = v
targetMap[v.ID] = entry targetMap[v.ID] = entry
@@ -1262,6 +1238,145 @@ func (h *Handlers) HandleEntrypointCreate() http.HandlerFunc {
} }
} }
// HandleEntrypointSecret sets, rotates or removes the shared secret
// an entrypoint verifies inbound requests with.
//
// Setting and rotating are the same operation: the form always takes
// the secret afresh and the stored value is never sent to the browser
// to be edited, so there is no path by which the page can display a
// credential it holds. Rotation is therefore "submit the new secret",
// and the operator already has that value — both supported senders
// require them to enter the same string on the sender's side, so
// there is no generated value for webhooker to reveal once.
func (h *Handlers) HandleEntrypointSecret() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
webhook, ok := h.ownedWebhook(w, r)
if !ok {
return
}
// The body size cap is enforced by the MaxBodySize
// middleware, which runs before CSRF parses the form.
err := r.ParseForm()
if err != nil {
http.Error(
w, "Bad request", http.StatusBadRequest,
)
return
}
var entrypoint database.Entrypoint
err = h.db.DB().Where(
"id = ? AND webhook_id = ?",
chi.URLParam(r, "entrypointID"), webhook.ID,
).First(&entrypoint).Error
if err != nil {
http.NotFound(w, r)
return
}
h.applyEntrypointSecret(w, r, &entrypoint)
}
}
// applyEntrypointSecret validates the submitted scheme and secret and
// stores them.
//
// A scheme this build does not support is a 400, never a stored value
// the receiver would later have to interpret: the receiver fails such
// a row closed, so letting one be created would take the entrypoint
// offline through a form that reported success.
func (h *Handlers) applyEntrypointSecret(
w http.ResponseWriter,
r *http.Request,
entrypoint *database.Entrypoint,
) {
// PostFormValue, not FormValue: a credential must come from the
// body. FormValue falls back to the query string, and the request
// line — unlike the body — is what logs, proxies, Referer headers
// and error trackers record.
scheme := database.SignatureScheme(
r.PostFormValue("signature_scheme"),
)
// Surrounding whitespace is stripped, because a secret pasted from
// a password manager routinely carries some and the resulting
// mismatch is undiagnosable from the sender's side. A secret whose
// own first or last character is a space cannot be stored; the
// README says so.
secret := strings.TrimSpace(r.PostFormValue("secret"))
if !signature.Supported(scheme) {
http.Error(
w, "Invalid signature scheme",
http.StatusBadRequest,
)
return
}
if scheme == database.SignatureSchemeNone {
// Turning verification off drops the secret with it: a stored
// credential nothing reads is one more copy to leak, and
// Verify refuses that pairing in any case.
secret = ""
} else if secret == "" {
http.Error(
w,
"A shared secret is required for this signature scheme.",
http.StatusBadRequest,
)
return
}
h.storeEntrypointSecret(w, r, entrypoint, scheme, secret)
}
// storeEntrypointSecret writes a validated scheme and secret to an
// entrypoint and returns the operator to the webhook page.
func (h *Handlers) storeEntrypointSecret(
w http.ResponseWriter,
r *http.Request,
entrypoint *database.Entrypoint,
scheme database.SignatureScheme,
secret string,
) {
// Updates with a map rather than a struct: a struct update skips
// zero values, and the empty pair is exactly what has to be
// written when verification is being turned off.
err := h.db.DB().Model(entrypoint).Updates(map[string]any{
"signature_scheme": scheme,
"signature_secret": secret,
}).Error
if err != nil {
// The error is logged by serverError; GORM's error text
// carries the statement, not the bound values, so the secret
// does not travel with it.
h.serverError(
w, "failed to update entrypoint signature", err,
)
return
}
h.log.Info(
"entrypoint signature configuration updated",
"entrypoint_id", entrypoint.ID,
"webhook_id", entrypoint.WebhookID,
"scheme", string(scheme),
)
http.Redirect(
w, r,
"/source/"+entrypoint.WebhookID,
http.StatusSeeOther,
)
}
// HandleTargetCreate handles adding a new target to a webhook. // HandleTargetCreate handles adding a new target to a webhook.
func (h *Handlers) HandleTargetCreate() http.HandlerFunc { func (h *Handlers) HandleTargetCreate() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) {

14
internal/handlers/ui_copy_test.go
View File

@@ -13,6 +13,7 @@ import (
"sneak.berlin/go/webhooker/internal/delivery" "sneak.berlin/go/webhooker/internal/delivery"
"sneak.berlin/go/webhooker/internal/handlers" "sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session" "sneak.berlin/go/webhooker/internal/session"
"sneak.berlin/go/webhooker/internal/signature"
) )
// Template data keys the page templates read. The handlers package has // Template data keys the page templates read. The handlers package has
@@ -268,15 +269,16 @@ func TestEntrypointCopyButtonIsProgressiveEnhancement(t *testing.T) {
body := renderPage(t, h, sess, "source_detail.html", map[string]any{ body := renderPage(t, h, sess, "source_detail.html", map[string]any{
dataKeyWebhook: webhook, dataKeyWebhook: webhook,
// The handler passes projected views, never raw rows — a // The handler passes projected views, never raw rows — an
// target carries its stored credential — so the test data // entrypoint carries its shared secret and a target its
// has that same shape. // stored credential — so the test data has that same shape.
"Entrypoints": handlers.NewEntrypointViews( "Entrypoints": handlers.NewEntrypointViews(
[]database.Entrypoint{entrypoint}, []database.Entrypoint{entrypoint},
), ),
"Targets": delivery.NewTargetViews(nil), "Targets": delivery.NewTargetViews(nil),
"Events": []database.Event{}, "SignatureSchemes": signature.Schemes(),
"BaseURL": "https://hooks.example.com", "Events": []database.Event{},
"BaseURL": "https://hooks.example.com",
}) })
assert.Contains( assert.Contains(

88
internal/handlers/webhook.go
View File

@@ -2,6 +2,7 @@ package handlers
import ( import (
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
@@ -11,6 +12,7 @@ import (
"sneak.berlin/go/webhooker/internal/database" "sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery" "sneak.berlin/go/webhooker/internal/delivery"
"sneak.berlin/go/webhooker/internal/logfield" "sneak.berlin/go/webhooker/internal/logfield"
"sneak.berlin/go/webhooker/internal/signature"
) )
const ( const (
@@ -70,12 +72,8 @@ func (h *Handlers) HandleWebhook() http.HandlerFunc {
} }
} }
// processWebhookRequest reads the body, serializes headers, loads // processWebhookRequest reads the body, verifies the sender,
// targets, and delivers the event. // serializes headers, loads targets, and delivers the event.
//
// Nothing about the request itself is authenticated: the entrypoint
// UUID in the path is the credential, and reaching here means it
// matched an active entrypoint.
func (h *Handlers) processWebhookRequest( func (h *Handlers) processWebhookRequest(
w http.ResponseWriter, w http.ResponseWriter,
r *http.Request, r *http.Request,
@@ -86,7 +84,26 @@ func (h *Handlers) processWebhookRequest(
return return
} }
headersJSON, err := json.Marshal(r.Header) // Before anything is written. An unverified request must leave no
// event row, no delivery row and no delivery task behind, so this
// sits above every write rather than inside the transaction that
// performs them. It has to sit below the body read because the
// signature is computed over the body; readWebhookBody is what
// bounds that read, so an unauthenticated sender still cannot make
// the process hold more than the 1 MB cap.
if !h.verifyInboundSignature(w, entrypoint, r.Header, body) {
return
}
// These headers are about to be stored verbatim and handed to
// every delivery target, so the scheme's credential comes out
// first. Under GitLab's scheme the header is the shared secret
// itself, and leaving it in would hand the ability to forge
// signed requests to exactly the parties the signature is meant
// to exclude.
headersJSON, err := json.Marshal(
signature.SanitizeHeaders(&entrypoint, r.Header),
)
if err != nil { if err != nil {
h.serverError(w, "failed to serialize headers", err) h.serverError(w, "failed to serialize headers", err)
@@ -105,6 +122,63 @@ func (h *Handlers) processWebhookRequest(
) )
} }
// verifyInboundSignature authenticates the request against the
// entrypoint's configured secret, reporting false once it has written
// the response.
//
// An entrypoint with no secret configured is not checked and this
// returns true, which is the unchanged behaviour every existing
// entrypoint keeps.
//
// A configuration that cannot be applied — an unknown scheme, or one
// half of the pair missing — is a 500, not a 401: the request may well
// be authentic, and calling it unauthorized would tell a legitimate
// sender to go fix its own signing. Either way it is refused. Failing
// open here would mean an entrypoint the operator has protected
// quietly accepting anything.
func (h *Handlers) verifyInboundSignature(
w http.ResponseWriter,
entrypoint database.Entrypoint,
header http.Header,
body []byte,
) bool {
err := signature.Verify(&entrypoint, header, body)
if err == nil {
return true
}
if errors.Is(err, signature.ErrConfig) {
h.log.Error(
"entrypoint signature configuration cannot be applied",
"entrypoint_id", entrypoint.ID,
"webhook_id", entrypoint.WebhookID,
"error", err,
)
http.Error(
w, "Internal server error",
http.StatusInternalServerError,
)
return false
}
// Every field here is bounded and none is client-chosen: the ids
// are ours, the scheme is one of a fixed set, and the error is a
// static string carrying no part of the secret or of what the
// client presented. Reaching this line also requires a real
// entrypoint UUID, so it is not a line a stranger can drive.
h.log.Warn(
"inbound signature verification failed",
"entrypoint_id", entrypoint.ID,
"webhook_id", entrypoint.WebhookID,
"scheme", string(entrypoint.SignatureScheme),
"error", err,
)
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return false
}
// loadActiveTargets returns all active targets for a webhook. // loadActiveTargets returns all active targets for a webhook.
func (h *Handlers) loadActiveTargets( func (h *Handlers) loadActiveTargets(
webhookID string, webhookID string,

468
internal/handlers/webhook_signature_test.go Normal file
View File

@@ -0,0 +1,468 @@
package handlers_test
import (
"bytes"
"context"
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"log/slog"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/signature"
)
const (
// inboundSecret is the shared secret the signed-receiver tests
// configure on their entrypoint. It doubles as a marker: no log
// line and no rendered page may contain it.
inboundSecret = "QQINBOUNDSECRETQQ"
// inboundBody is the payload the sender signs.
inboundBody = `{"zen":"Non-blocking is better than blocking."}`
// entrypointIDParam is the chi URL parameter naming an entrypoint.
entrypointIDParam = "entrypointID"
)
// hubSignature returns the X-Hub-Signature-256 value a GitHub sender
// holding secret would send for inboundBody.
func hubSignature(secret string) string {
mac := hmac.New(sha256.New, []byte(secret))
_, _ = mac.Write([]byte(inboundBody))
return "sha256=" + hex.EncodeToString(mac.Sum(nil))
}
// seedSignedEntrypoint inserts an active entrypoint for a webhook
// with the given signature configuration and returns it.
func seedSignedEntrypoint(
t *testing.T,
db *database.Database,
webhookID string,
scheme database.SignatureScheme,
secret string,
) *database.Entrypoint {
t.Helper()
ep := &database.Entrypoint{
WebhookID: webhookID,
Path: "path-" + webhookID,
Description: "signed",
Active: true,
SignatureScheme: scheme,
SignatureSecret: secret,
}
require.NoError(
t,
db.DB().Omit(clause.Associations).Create(ep).Error,
)
return ep
}
// postToEntrypoint drives the real receiver handler at an
// entrypoint's path with one optional header set.
func postToEntrypoint(
t *testing.T,
h *handlers.Handlers,
path, body, headerName, headerValue string,
) *httptest.ResponseRecorder {
t.Helper()
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodPost,
"/webhook/"+path,
strings.NewReader(body),
)
req.Header.Set("Content-Type", "application/json")
if headerName != "" {
req.Header.Set(headerName, headerValue)
}
rctx := chi.NewRouteContext()
rctx.URLParams.Add("uuid", path)
req = req.WithContext(
context.WithValue(req.Context(), chi.RouteCtxKey, rctx),
)
w := httptest.NewRecorder()
h.HandleWebhook().ServeHTTP(w, req)
return w
}
// storedEvents counts the event rows a webhook's per-webhook database
// holds. A database that was never opened holds none, which is the
// state a rejected request has to leave behind.
func storedEvents(
t *testing.T,
mgr *database.WebhookDBManager,
webhookID string,
) int64 {
t.Helper()
if !mgr.DBExists(webhookID) {
return 0
}
db, err := mgr.GetDB(webhookID)
require.NoError(t, err)
var count int64
require.NoError(
t,
db.Model(&database.Event{}).
Where("webhook_id = ?", webhookID).
Count(&count).Error,
)
return count
}
// storedEventHeaders reads back the Headers column of the single
// event row a webhook's per-webhook database holds.
//
// It reads the database rather than an in-memory struct on purpose:
// what matters is what an operator, a backup or the reaper's archive
// would find on disk, not what the handler passed around.
func storedEventHeaders(
t *testing.T,
mgr *database.WebhookDBManager,
webhookID string,
) string {
t.Helper()
require.True(t, mgr.DBExists(webhookID))
db, err := mgr.GetDB(webhookID)
require.NoError(t, err)
var events []database.Event
require.NoError(
t,
db.Where("webhook_id = ?", webhookID).
Find(&events).Error,
)
require.Len(t, events, 1)
return events[0].Headers
}
// signedReceiverCase is one inbound request against an entrypoint
// with a given stored signature configuration.
type signedReceiverCase struct {
name string
scheme database.SignatureScheme
secret string
headerName string
headerValue string
body string
wantStatus int
}
// signedReceiverCases covers each supported scheme with a valid
// signature, an invalid one and none at all, plus the two states that
// are not "a client got it wrong": an entrypoint with nothing
// configured, and one whose stored configuration cannot be applied.
func signedReceiverCases() []signedReceiverCase {
return append(
schemeReceiverCases(), unverifiedReceiverCases()...,
)
}
// schemeReceiverCases covers the two supported schemes.
func schemeReceiverCases() []signedReceiverCase {
return []signedReceiverCase{
{
name: "github valid",
scheme: database.SignatureSchemeGitHub,
secret: inboundSecret,
headerName: signature.HeaderGitHub,
headerValue: hubSignature(inboundSecret),
body: inboundBody,
wantStatus: http.StatusOK,
},
{
name: "github wrong secret",
scheme: database.SignatureSchemeGitHub,
secret: inboundSecret,
headerName: signature.HeaderGitHub,
headerValue: hubSignature("wrong"),
body: inboundBody,
wantStatus: http.StatusUnauthorized,
},
{
// A digest that was valid for a different body: the
// check is over the bytes as received.
name: "github body tampered",
scheme: database.SignatureSchemeGitHub,
secret: inboundSecret,
headerName: signature.HeaderGitHub,
headerValue: hubSignature(inboundSecret),
body: inboundBody + " ",
wantStatus: http.StatusUnauthorized,
},
{
name: "github unsigned",
scheme: database.SignatureSchemeGitHub,
secret: inboundSecret,
body: inboundBody,
wantStatus: http.StatusUnauthorized,
},
{
name: "gitlab valid",
scheme: database.SignatureSchemeGitLab,
secret: inboundSecret,
headerName: signature.HeaderGitLab,
headerValue: inboundSecret,
body: inboundBody,
wantStatus: http.StatusOK,
},
{
name: "gitlab wrong token",
scheme: database.SignatureSchemeGitLab,
secret: inboundSecret,
headerName: signature.HeaderGitLab,
headerValue: "wrong",
body: inboundBody,
wantStatus: http.StatusUnauthorized,
},
{
name: "gitlab unsigned",
scheme: database.SignatureSchemeGitLab,
secret: inboundSecret,
body: inboundBody,
wantStatus: http.StatusUnauthorized,
},
}
}
// unverifiedReceiverCases covers the two entrypoint states that are
// not about a client getting its signature wrong: nothing configured
// at all, and a configuration the receiver cannot apply.
func unverifiedReceiverCases() []signedReceiverCase {
return []signedReceiverCase{
{
// The pass-through case. An entrypoint with nothing
// configured is what every deployment already has, and
// it must keep accepting unsigned requests so that an
// upgrade does not lock an operator out of their own
// receivers.
name: "unconfigured accepts unsigned",
scheme: database.SignatureSchemeNone,
body: inboundBody,
wantStatus: http.StatusOK,
},
{
// A stray signature header changes nothing when nothing
// is configured to check it.
name: "unconfigured ignores a stray header",
scheme: database.SignatureSchemeNone,
headerName: signature.HeaderGitHub,
headerValue: "sha256=deadbeef",
body: inboundBody,
wantStatus: http.StatusOK,
},
{
// A scheme this build cannot apply, reachable only by
// editing the database: refused, not waved through as
// unverified.
name: "unknown scheme fails closed",
scheme: database.SignatureScheme("stripe"),
secret: inboundSecret,
headerName: signature.HeaderGitHub,
headerValue: hubSignature(inboundSecret),
body: inboundBody,
wantStatus: http.StatusInternalServerError,
},
}
}
// TestReceiverVerifiesConfiguredEntrypoints is the load-bearing test
// for the feature: for each supported scheme a correctly signed
// request is accepted and stored, and an incorrectly signed or
// unsigned one is answered 401 having stored nothing.
//
// The event count is the half that matters most. A rejection that
// still wrote a row would leave the receiver a place for a stranger
// who knows a URL to deposit content, which is exactly what the
// signature is there to prevent.
//
// The cases share one application and take a webhook each, rather
// than each standing up its own: every newTestApp seeds an admin user
// and so pays an Argon2id hash at 64 MB, and this package's test
// budget is not large enough to spend one per table row.
func TestReceiverVerifiesConfiguredEntrypoints(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
db *database.Database
mgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &db, &mgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
for _, tc := range signedReceiverCases() {
wh := seedWebhook(t, db)
ep := seedSignedEntrypoint(
t, db, wh.ID, tc.scheme, tc.secret,
)
w := postToEntrypoint(
t, h, ep.Path, tc.body,
tc.headerName, tc.headerValue,
)
assert.Equal(t, tc.wantStatus, w.Code, "case %s", tc.name)
want := int64(0)
if tc.wantStatus == http.StatusOK {
want = 1
}
assert.Equal(
t, want, storedEvents(t, mgr, wh.ID),
"case %s: stored event rows after a %d response",
tc.name, w.Code,
)
}
}
// TestReceiverLogsNoSecret proves the rejection path does not write
// the shared secret, or what the client presented, into the log. A
// GitLab token arrives as the credential itself, so echoing the
// header value would put a live secret in the log of every deployment
// whose sender is briefly misconfigured.
func TestReceiverLogsNoSecret(t *testing.T) {
t.Parallel()
const presented = "QQPRESENTEDVALUEQQ"
var (
h *handlers.Handlers
db *database.Database
)
app := newTestApp(t, &h, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
var buf bytes.Buffer
h.SetLogForTest(slog.New(slog.NewJSONHandler(&buf, nil)))
wh := seedWebhook(t, db)
ep := seedSignedEntrypoint(
t, db, wh.ID,
database.SignatureSchemeGitLab, inboundSecret,
)
w := postToEntrypoint(
t, h, ep.Path, inboundBody,
signature.HeaderGitLab, presented,
)
require.Equal(t, http.StatusUnauthorized, w.Code)
// The rejection is recorded at all — a silent 401 leaves an
// operator no way to see a sender failing to authenticate.
assert.Contains(t, buf.String(), "verification failed")
assert.NotContains(t, buf.String(), inboundSecret)
assert.NotContains(t, buf.String(), presented)
}
// TestReceiverDoesNotStoreInboundCredential proves an accepted
// request leaves no copy of the shared secret in the event store.
//
// GitLab's X-Gitlab-Token is the credential itself, not a digest
// over the request. Stored headers are read back by the UI, copied
// into every backup and archive, and handed verbatim to every
// delivery target, so a stored token is the entrypoint's only
// authentication control disclosed to precisely the parties it
// exists to exclude.
//
// The two cases share one application: every newTestApp seeds an
// admin user and pays an Argon2id hash at 64 MB, and this package's
// test budget does not stretch to one per case.
func TestReceiverDoesNotStoreInboundCredential(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
db *database.Database
mgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &db, &mgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
gitlab := seedWebhook(t, db)
gitlabEP := seedSignedEntrypoint(
t, db, gitlab.ID,
database.SignatureSchemeGitLab, inboundSecret,
)
w := postToEntrypoint(
t, h, gitlabEP.Path, inboundBody,
signature.HeaderGitLab, inboundSecret,
)
require.Equal(t, http.StatusOK, w.Code)
stored := storedEventHeaders(t, mgr, gitlab.ID)
assert.NotContains(
t, stored, inboundSecret,
"the shared secret must not be persisted",
)
assert.NotContains(
t, stored, signature.HeaderGitLab,
"the credential header must not be persisted at all",
)
// Everything else the sender set is still there. A fix that
// stored no headers would satisfy the assertions above while
// discarding the record the receiver exists to keep.
assert.Contains(t, stored, "Content-Type")
// A GitHub digest is an HMAC over the body, so the key cannot be
// recovered from it and it stays: the stripping is scoped to
// what actually carries the secret.
github := seedWebhook(t, db)
githubEP := seedSignedEntrypoint(
t, db, github.ID,
database.SignatureSchemeGitHub, inboundSecret,
)
w = postToEntrypoint(
t, h, githubEP.Path, inboundBody,
signature.HeaderGitHub, hubSignature(inboundSecret),
)
require.Equal(t, http.StatusOK, w.Code)
stored = storedEventHeaders(t, mgr, github.ID)
assert.Contains(t, stored, signature.HeaderGitHub)
assert.NotContains(t, stored, inboundSecret)
}

17
internal/middleware/csrf.go
View File

@@ -5,7 +5,6 @@ import (
"github.com/gorilla/csrf" "github.com/gorilla/csrf"
"sneak.berlin/go/webhooker/internal/logfield" "sneak.berlin/go/webhooker/internal/logfield"
"sneak.berlin/go/webhooker/internal/reqtls"
) )
// CSRFToken retrieves the CSRF token from the request context. // CSRFToken retrieves the CSRF token from the request context.
@@ -14,6 +13,13 @@ func CSRFToken(r *http.Request) string {
return csrf.Token(r) return csrf.Token(r)
} }
// isClientTLS reports whether the client-facing connection uses TLS.
// It checks for a direct TLS connection (r.TLS) or a TLS-terminating
// reverse proxy that sets the standard X-Forwarded-Proto header.
func isClientTLS(r *http.Request) bool {
return r.TLS != nil || r.Header.Get("X-Forwarded-Proto") == "https"
}
// CSRF returns middleware that provides CSRF protection using the // CSRF returns middleware that provides CSRF protection using the
// gorilla/csrf library. The middleware uses the session authentication // gorilla/csrf library. The middleware uses the session authentication
// key to sign a CSRF cookie and validates a masked token submitted via // key to sign a CSRF cookie and validates a masked token submitted via
@@ -21,10 +27,9 @@ func CSRFToken(r *http.Request) string {
// POST/PUT/PATCH/DELETE requests. Requests with an invalid or missing // POST/PUT/PATCH/DELETE requests. Requests with an invalid or missing
// token receive a 403 Forbidden response. // token receive a 403 Forbidden response.
// //
// The middleware detects the client-facing transport protocol // The middleware detects the client-facing transport protocol per-request
// per-request via reqtls.IsTLS, the single TLS predicate the session // using r.TLS and the X-Forwarded-Proto header. This allows correct
// cookie also uses. This allows correct behavior in all deployment // behavior in all deployment scenarios:
// scenarios:
// //
// - Direct HTTPS: strict Referer/Origin checks, Secure cookies. // - Direct HTTPS: strict Referer/Origin checks, Secure cookies.
// - Behind a TLS-terminating reverse proxy: strict checks (the // - Behind a TLS-terminating reverse proxy: strict checks (the
@@ -78,7 +83,7 @@ func (m *Middleware) CSRF() func(http.Handler) http.Handler {
httpCSRF := httpProtect(next) httpCSRF := httpProtect(next)
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if reqtls.IsTLS(r) { if isClientTLS(r) {
// Client is on TLS (directly or via reverse proxy). // Client is on TLS (directly or via reverse proxy).
// Use Secure cookies and strict Origin/Referer checks. // Use Secure cookies and strict Origin/Referer checks.
tlsCSRF.ServeHTTP(w, r) tlsCSRF.ServeHTTP(w, r)

197
internal/middleware/csrf_test.go
View File

@@ -297,176 +297,55 @@ func TestCSRFToken_NoMiddleware(t *testing.T) {
} }
// --- TLS Detection Tests --- // --- TLS Detection Tests ---
//
// The predicate itself is tested in internal/reqtls. What is tested
// here is the consequence that actually matters: which of the two
// gorilla/csrf instances a request is routed to.
//
// The two are told apart behaviourally rather than by inspection. On
// the STRICT (TLS) instance, a state-changing request carrying no
// Origin header must supply a Referer -- gorilla/csrf rejects it with
// ErrNoReferer before it ever looks at the token, to defend a
// TLS site against an HTTP machine-in-the-middle injecting a form. On
// the RELAXED (plaintext) instance that check is skipped and a valid
// token is enough. So: valid token, no Origin, no Referer, and the
// outcome names the instance.
//
// Landing on the relaxed instance for a genuinely-HTTPS deployment is
// the defect: an exact == "https" comparison did exactly that for the
// uppercase and comma-appended spellings below.
// csrfTookStrictPath reports whether the CSRF middleware routed a func TestIsClientTLS_DirectTLS(t *testing.T) {
// request with the given transport to the strict instance. It also
// asserts the CSRF cookie's Secure attribute agrees, since the two are
// set by the same choice and must never disagree.
func csrfTookStrictPath(
t *testing.T,
env string,
directTLS bool,
fwdProto string,
) bool {
t.Helper()
m, _ := testMiddleware(t, env)
csrfMW := m.CSRF()
newReq := func(method string) *http.Request {
r := httptest.NewRequestWithContext(
context.Background(), method,
"http://example.com/form", nil,
)
if directTLS {
r.TLS = &tls.ConnectionState{}
}
if fwdProto != "" {
r.Header.Set("X-Forwarded-Proto", fwdProto)
}
return r
}
token, cookies := csrfGetToken(t, csrfMW, newReq(http.MethodGet))
// Deliberately no Origin and no Referer: that is what makes the
// two instances distinguishable.
called, code := csrfPostWithToken(
t, csrfMW, newReq(http.MethodPost), token, cookies,
)
strict := !called
if strict {
assert.Equal(
t, http.StatusForbidden, code,
"the strict instance rejects a Referer-less POST",
)
}
for _, c := range cookies {
if c.Name == csrfCookieName {
assert.Equal(
t, strict, c.Secure,
"the CSRF cookie's Secure attribute and the "+
"chosen instance come from one decision "+
"and must agree",
)
}
}
return strict
}
// TestCSRF_ForwardedProtoSpellingsTakeStrictPath runs the header
// spellings a real proxy emits through the middleware. The environment
// is dev -- the DEFAULT when WEBHOOKER_ENVIRONMENT is unset -- to pin
// that the routing is a per-request transport decision and owes
// nothing to configuration.
func TestCSRF_ForwardedProtoSpellingsTakeStrictPath(t *testing.T) {
t.Parallel() t.Parallel()
cases := []struct { r := httptest.NewRequestWithContext(
name string context.Background(), http.MethodGet, "/", nil)
header string r.TLS = &tls.ConnectionState{}
strict bool
why string
}{
{
name: "lowercase",
header: "https",
strict: true,
why: "the ordinary spelling",
},
{
name: "uppercase",
header: "HTTPS",
strict: true,
why: "the header value is a case-insensitive token",
},
{
name: "chain with plaintext inner hop",
header: "https, http",
strict: true,
why: "a chained proxy appends its hop; the leftmost " +
"element is the browser's connection",
},
{
name: "chain of two TLS hops",
header: "https,https",
strict: true,
why: "appended chain with no space after the comma",
},
{
name: "trailing space",
header: "https ",
strict: true,
why: "whitespace is not part of the token",
},
{
name: "plaintext",
header: "http",
strict: false,
why: "the negative control: the proxy reports a " +
"plaintext client connection",
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
assert.Equal(
t, tc.strict,
csrfTookStrictPath(
t, config.EnvironmentDev, false, tc.header,
),
"X-Forwarded-Proto %q: %s", tc.header, tc.why,
)
})
}
}
// TestCSRF_DirectTLSTakesStrictPath covers the no-proxy TLS
// deployment, and TestCSRF_PlaintextTakesRelaxedPath the no-proxy
// plaintext one -- the local development case that must keep working.
func TestCSRF_DirectTLSTakesStrictPath(t *testing.T) {
t.Parallel()
assert.True( assert.True(
t, t, middleware.IsClientTLS(r),
csrfTookStrictPath(t, config.EnvironmentDev, true, ""), "should detect direct TLS connection",
"a request that arrived over TLS takes the strict path",
) )
} }
func TestCSRF_PlaintextTakesRelaxedPath(t *testing.T) { func TestIsClientTLS_XForwardedProto(t *testing.T) {
t.Parallel() t.Parallel()
r := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/", nil)
r.Header.Set("X-Forwarded-Proto", "https")
assert.True(
t, middleware.IsClientTLS(r),
"should detect TLS via X-Forwarded-Proto",
)
}
func TestIsClientTLS_PlaintextHTTP(t *testing.T) {
t.Parallel()
r := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/", nil)
assert.False( assert.False(
t, t, middleware.IsClientTLS(r),
csrfTookStrictPath(t, config.EnvironmentProd, false, ""), "should detect plaintext HTTP",
"no TLS and no proxy header is plaintext, in any environment", )
}
func TestIsClientTLS_XForwardedProtoHTTP(t *testing.T) {
t.Parallel()
r := httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/", nil)
r.Header.Set("X-Forwarded-Proto", "http")
assert.False(
t, middleware.IsClientTLS(r),
"should detect plaintext when X-Forwarded-Proto is http",
) )
} }

9
internal/middleware/export_test.go
View File

@@ -26,10 +26,6 @@ const UnmatchedRouteConst = unmatchedRoute
// inflight gauge. // inflight gauge.
const InflightHandlerConst = inflightHandler const InflightHandlerConst = inflightHandler
// UnmatchedMethodConst exposes the sentinel that stands in for a
// method the router can never route.
const UnmatchedMethodConst = unmatchedMethod
// NewLoggingResponseWriterForTest wraps newLoggingResponseWriter // NewLoggingResponseWriterForTest wraps newLoggingResponseWriter
// for use in external test packages. // for use in external test packages.
func NewLoggingResponseWriterForTest( func NewLoggingResponseWriterForTest(
@@ -56,6 +52,11 @@ func ClientKeyForTest(m *Middleware, r *http.Request) string {
return m.clientKey(r) return m.clientKey(r)
} }
// IsClientTLS exposes isClientTLS for testing.
func IsClientTLS(r *http.Request) bool {
return isClientTLS(r)
}
// LoginRateLimitConst exposes the loginRateLimit constant: the // LoginRateLimitConst exposes the loginRateLimit constant: the
// number of FAILED login attempts one client may make against one // number of FAILED login attempts one client may make against one
// submitted username per interval. // submitted username per interval.

67
internal/middleware/metrics.go
View File

@@ -26,15 +26,6 @@ import (
// counting the requests in flight across the whole service. // counting the requests in flight across the whole service.
const inflightHandler = "(all)" const inflightHandler = "(all)"
// unmatchedMethod is the `method` label for a request whose method
// the router can never route.
//
// It is deliberately the same sentinel as unmatchedRoute rather than
// a spelling of its own: both stand for a client-chosen token that
// matched nothing this service registers, and giving one idea two
// spellings would read in a scrape as two different unmatched states.
const unmatchedMethod = unmatchedRoute
// routePatternID is the `handler` label for a request: the chi route // routePatternID is the `handler` label for a request: the chi route
// pattern, never the concrete path. // pattern, never the concrete path.
// //
@@ -59,45 +50,9 @@ func routePatternID(ctx context.Context) string {
return unmatchedRoute return unmatchedRoute
} }
// methodID is the `method` label for a request: the request method // routePatternRecorder wraps a go-http-metrics recorder and replaces
// when the router can route it, and the unmatched sentinel otherwise. // the handler id on every observation with the request's route
// // pattern.
// net/http accepts any RFC 9110 token as a method and hands it
// through verbatim, so the raw method is client-chosen bytes and
// bounds the label at nothing — the same unauthenticated
// series-minting the handler label carried, reached through a second
// dimension. What bounds it is the set chi's router will match a
// route for: its methodMap, which is unexported, so it is restated
// here against the net/http constants it is built from. A token
// outside that set can only ever produce chi's 405, so folding every
// one of them onto a single series loses no information a scrape
// could have used, while the nine methods that can reach a handler
// stay distinguishable.
//
// chi.RegisterMethod would extend the router's set at runtime; this
// service never calls it, and a caller that started to would have to
// extend this switch with it.
func methodID(method string) string {
switch method {
case http.MethodConnect,
http.MethodDelete,
http.MethodGet,
http.MethodHead,
http.MethodOptions,
http.MethodPatch,
http.MethodPost,
http.MethodPut,
http.MethodTrace:
return method
default:
return unmatchedMethod
}
}
// boundedLabelRecorder wraps a go-http-metrics recorder and replaces
// the request-controlled labels on every observation with bounded
// ones: the handler id becomes the request's route pattern, and the
// method becomes one the router can route.
// //
// This is the seam that makes the pattern usable at all. The metrics // This is the seam that makes the pattern usable at all. The metrics
// middleware is global (see Server.setupGlobalMiddleware), so it is // middleware is global (see Server.setupGlobalMiddleware), so it is
@@ -116,31 +71,29 @@ func methodID(method string) string {
// Those never reach a handler, but chi has already matched the route // Those never reach a handler, but chi has already matched the route
// by the time the limiter runs, so their 429s land on the pattern // by the time the limiter runs, so their 429s land on the pattern
// like any other response. // like any other response.
type boundedLabelRecorder struct { type routePatternRecorder struct {
inner httpmetrics.Recorder inner httpmetrics.Recorder
} }
func (r boundedLabelRecorder) ObserveHTTPRequestDuration( func (r routePatternRecorder) ObserveHTTPRequestDuration(
ctx context.Context, ctx context.Context,
props httpmetrics.HTTPReqProperties, props httpmetrics.HTTPReqProperties,
duration time.Duration, duration time.Duration,
) { ) {
props.ID = routePatternID(ctx) props.ID = routePatternID(ctx)
props.Method = methodID(props.Method)
r.inner.ObserveHTTPRequestDuration(ctx, props, duration) r.inner.ObserveHTTPRequestDuration(ctx, props, duration)
} }
func (r boundedLabelRecorder) ObserveHTTPResponseSize( func (r routePatternRecorder) ObserveHTTPResponseSize(
ctx context.Context, ctx context.Context,
props httpmetrics.HTTPReqProperties, props httpmetrics.HTTPReqProperties,
sizeBytes int64, sizeBytes int64,
) { ) {
props.ID = routePatternID(ctx) props.ID = routePatternID(ctx)
props.Method = methodID(props.Method)
r.inner.ObserveHTTPResponseSize(ctx, props, sizeBytes) r.inner.ObserveHTTPResponseSize(ctx, props, sizeBytes)
} }
func (r boundedLabelRecorder) AddInflightRequests( func (r routePatternRecorder) AddInflightRequests(
ctx context.Context, ctx context.Context,
props httpmetrics.HTTPProperties, props httpmetrics.HTTPProperties,
quantity int, quantity int,
@@ -149,7 +102,7 @@ func (r boundedLabelRecorder) AddInflightRequests(
r.inner.AddInflightRequests(ctx, props, quantity) r.inner.AddInflightRequests(ctx, props, quantity)
} }
var _ httpmetrics.Recorder = boundedLabelRecorder{} var _ httpmetrics.Recorder = routePatternRecorder{}
// Metrics returns middleware that records Prometheus HTTP metrics on // Metrics returns middleware that records Prometheus HTTP metrics on
// the default registry, which is the one the /metrics route gathers. // the default registry, which is the one the /metrics route gathers.
@@ -166,14 +119,14 @@ func metricsMiddleware(
rec httpmetrics.Recorder, rec httpmetrics.Recorder,
) func(http.Handler) http.Handler { ) func(http.Handler) http.Handler {
mdlw := ghmm.New(ghmm.Config{ mdlw := ghmm.New(ghmm.Config{
Recorder: boundedLabelRecorder{inner: rec}, Recorder: routePatternRecorder{inner: rec},
}) })
return func(next http.Handler) http.Handler { return func(next http.Handler) http.Handler {
// The handler id is unmatchedRoute rather than "" so that // The handler id is unmatchedRoute rather than "" so that
// the client-chosen URL path never enters the metrics // the client-chosen URL path never enters the metrics
// pipeline at all: an empty id is the library's signal to // pipeline at all: an empty id is the library's signal to
// substitute it. boundedLabelRecorder overwrites this value // substitute it. routePatternRecorder overwrites this value
// on every observation, so it is reachable only if that // on every observation, so it is reachable only if that
// decorator is removed — in which case the metrics collapse // decorator is removed — in which case the metrics collapse
// to one series instead of leaking again. // to one series instead of leaking again.

309
internal/middleware/metrics_method_test.go
View File

@@ -1,309 +0,0 @@
package middleware_test
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/google/uuid"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
dto "github.com/prometheus/client_model/go"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/middleware"
)
const (
// metricsProbeMethods is how many distinct invented method tokens
// each cardinality assertion drives. The measurement on the issue
// took 300 tokens from 106 exposition lines to 7,631 — about 25
// permanent lines per token, never reclaimed — so a probe of this
// size puts a regression thousands of lines over the bound rather
// than leaving it to a rounding argument.
metricsProbeMethods = 300
// probeMethodLen is how many characters each invented method
// token carries, matching the 12 the issue measured with.
probeMethodLen = 12
// methodLabel is the label these tests are about.
methodLabel = "method"
)
// realMethods is the positive control's domain: the methods chi's
// router can match a route for, every one of which a client
// legitimately sends and every one of which must keep a series of its
// own. Bounding the label by collapsing these into one bucket would
// destroy the metric it is meant to protect.
func realMethods() []string {
return []string{
http.MethodConnect, http.MethodDelete, http.MethodGet,
http.MethodHead, http.MethodOptions, http.MethodPatch,
http.MethodPost, http.MethodPut, http.MethodTrace,
}
}
// methodProbePath returns the one receiver path a method probe
// targets. Holding the path fixed leaves the method as the only
// dimension varying, so any series growth a probe produces is the
// method label's and nothing else's.
func methodProbePath() string {
return "/webhook/" + uuid.NewString()
}
// inventedMethods returns n distinct RFC 9110 method tokens that no
// router will ever match: uppercase hex from a fresh UUID, which is
// both the shape and the length an unauthenticated flood would send.
// net/http accepts any token as a method, so every one of these
// reaches the metrics pipeline exactly as a real method does.
func inventedMethods(n int) []string {
methods := make([]string, 0, n)
for range n {
token := strings.ToUpper(
strings.ReplaceAll(uuid.NewString(), "-", ""),
)
methods = append(methods, token[:probeMethodLen])
}
return methods
}
// driveMethods sends one request per supplied method to a single
// fixed path.
func driveMethods(
t *testing.T,
h http.Handler,
path string,
methods []string,
) map[int]int {
t.Helper()
probes := make([]probe, 0, len(methods))
for _, m := range methods {
probes = append(probes, probe{method: m, path: path})
}
return drive(t, h, probes)
}
// methodLabels returns the set of distinct `method` values across
// every gathered series that carries the label at all. The inflight
// gauge does not carry it, and so contributes nothing rather than an
// empty-string member.
func methodLabels(families []*dto.MetricFamily) map[string]struct{} {
seen := make(map[string]struct{})
for _, fam := range families {
for _, m := range fam.GetMetric() {
for _, pair := range m.GetLabel() {
if pair.GetName() == methodLabel {
seen[pair.GetValue()] = struct{}{}
}
}
}
}
return seen
}
// scrapeLines renders the registry through the same promhttp handler
// /metrics is mounted on and counts the sample lines it produced.
//
// This is the quantity the issue measured and the one a Prometheus
// server pays for on every scrape: one histogram label set is a
// single gathered series but around 25 lines of exposition, which is
// why 300 method tokens cost thousands of lines rather than hundreds.
func scrapeLines(t *testing.T, reg *prometheus.Registry) int {
t.Helper()
h := promhttp.HandlerFor(reg, promhttp.HandlerOpts{})
req := httptest.NewRequestWithContext(
t.Context(), http.MethodGet, "/metrics", nil,
)
w := httptest.NewRecorder()
h.ServeHTTP(w, req)
require.Equal(t, http.StatusOK, w.Code)
lines := 0
for line := range strings.SplitSeq(w.Body.String(), "\n") {
if line == "" || strings.HasPrefix(line, "#") {
continue
}
lines++
}
return lines
}
// TestMetrics_MethodSentinelIsTheRouteSentinel pins the convention
// rather than the mechanism. An unroutable method and an unmatched
// path are the same fact — a client-chosen token matching nothing
// this service registers — so they carry one spelling. Two spellings
// would read in a scrape as two different unmatched states.
func TestMetrics_MethodSentinelIsTheRouteSentinel(t *testing.T) {
t.Parallel()
assert.Equal(
t,
middleware.UnmatchedRouteConst,
middleware.UnmatchedMethodConst,
"the unmatched sentinel must have exactly one spelling",
)
}
// TestMetrics_InventedMethodsMintOneLabelSet is the direct assertion
// the issue asks for: N requests carrying N distinct invented method
// tokens must produce exactly ONE method label. Before the fix this
// produced N of them, on an unauthenticated route with no rate
// limiter.
func TestMetrics_InventedMethodsMintOneLabelSet(t *testing.T) {
t.Parallel()
h, reg := metricsTestRouter(t, generousReceiverLimit)
methods := inventedMethods(metricsProbeMethods)
codes := driveMethods(t, h, methodProbePath(), methods)
require.Equal(
t, metricsProbeMethods, codes[http.StatusMethodNotAllowed],
"every invented token should have been unroutable",
)
labels := methodLabels(gatherMetrics(t, reg))
// Asserted on the count rather than on the set, so that a
// regression reports one number instead of dumping every token it
// minted.
distinct := len(labels)
assert.Equal(
t, 1, distinct,
"invented methods must collapse onto one label",
)
assert.Contains(
t, keys(labels), middleware.UnmatchedMethodConst,
"that one label must be the unmatched sentinel",
)
// The scrape must not republish the tokens it was driven with
// either: a label that merely looks bounded while still echoing
// client bytes is the same defect wearing a different name.
echoed := 0
for _, m := range methods {
for label := range labels {
if strings.Contains(label, m) {
echoed++
}
}
}
assert.Equal(
t, 0, echoed,
"invented method tokens reached the metrics labels",
)
}
// TestMetrics_MethodSeriesCountIsFlatUnderAFlood reproduces the
// measurement on the issue in miniature: scrape, drive several
// hundred distinct method tokens, scrape again, and require the
// second scrape to be no larger than the first. The first batch
// establishes every label set the route can produce; a flood five
// times its size must land on exactly those.
func TestMetrics_MethodSeriesCountIsFlatUnderAFlood(t *testing.T) {
t.Parallel()
h, reg := metricsTestRouter(t, generousReceiverLimit)
path := methodProbePath()
driveMethods(t, h, path, inventedMethods(metricsProbeMethods))
seededSeries := seriesCount(gatherMetrics(t, reg))
seededLines := scrapeLines(t, reg)
driveMethods(t, h, path, inventedMethods(metricsProbeMethods*4))
floodedSeries := seriesCount(gatherMetrics(t, reg))
floodedLines := scrapeLines(t, reg)
t.Logf(
"after %d invented methods: %d series, %d lines; "+
"after %d more: %d series, %d lines",
metricsProbeMethods, seededSeries, seededLines,
metricsProbeMethods*4, floodedSeries, floodedLines,
)
assert.Equal(
t, seededSeries, floodedSeries,
"a flood of invented methods must not mint series",
)
assert.Equal(
t, seededLines, floodedLines,
"a flood of invented methods must not grow the scrape",
)
}
// TestMetrics_RealMethodsStayDistinct is the positive control. The
// bound is worth nothing if it is bought by flattening the metric:
// every method the router can route must still carry a series of its
// own, one sample each, under the route pattern it was sent to.
func TestMetrics_RealMethodsStayDistinct(t *testing.T) {
t.Parallel()
h, reg := metricsTestRouter(t, generousReceiverLimit)
methods := realMethods()
codes := driveMethods(t, h, methodProbePath(), methods)
require.Equal(
t, len(methods), codes[http.StatusNotFound],
"every real method should have reached the receiver",
)
families := gatherMetrics(t, reg)
want := make(map[string]struct{}, len(methods))
for _, m := range methods {
want[m] = struct{}{}
}
assert.Equal(
t, want, methodLabels(families),
"real methods must remain distinguishable",
)
// Appearing somewhere in the scrape is not enough: each method
// must own its duration series, holding the one sample it sent.
observed := 0
for _, fam := range families {
if !strings.HasSuffix(fam.GetName(), "request_duration_seconds") {
continue
}
for _, m := range fam.GetMetric() {
observed++
assert.Equal(
t, receiverRoutePattern,
labelValue(m, "handler"),
)
assert.Equal(
t, uint64(1),
m.GetHistogram().GetSampleCount(),
"method %q shares a series",
labelValue(m, methodLabel),
)
}
}
assert.Equal(
t, len(methods), observed,
"one duration series per routable method",
)
}

41
internal/middleware/metrics_test.go
View File

@@ -97,34 +97,8 @@ func metricsTestRouter(
return r, reg return r, reg
} }
// probe is one request a cardinality assertion sends. Both label // drivePaths sends one POST per supplied path and returns how many
// dimensions that have leaked are request-controlled — the path and // responses carried each status code.
// the method — so both vary here and one driver sends them.
type probe struct {
method string
path string
}
// drive sends every probe and returns how many responses carried each
// status code.
func drive(t *testing.T, h http.Handler, probes []probe) map[int]int {
t.Helper()
codes := make(map[int]int)
for _, p := range probes {
req := httptest.NewRequestWithContext(
t.Context(), p.method, p.path, nil,
)
w := httptest.NewRecorder()
h.ServeHTTP(w, req)
codes[w.Code]++
}
return codes
}
// drivePaths sends one POST per supplied path.
func drivePaths( func drivePaths(
t *testing.T, t *testing.T,
h http.Handler, h http.Handler,
@@ -132,15 +106,18 @@ func drivePaths(
) map[int]int { ) map[int]int {
t.Helper() t.Helper()
probes := make([]probe, 0, len(paths)) codes := make(map[int]int)
for _, p := range paths { for _, p := range paths {
probes = append( req := httptest.NewRequestWithContext(
probes, probe{method: http.MethodPost, path: p}, t.Context(), http.MethodPost, p, nil,
) )
w := httptest.NewRecorder()
h.ServeHTTP(w, req)
codes[w.Code]++
} }
return drive(t, h, probes) return codes
} }
// receiverPaths returns n distinct /webhook/ paths, each naming a // receiverPaths returns n distinct /webhook/ paths, each naming a

59
internal/reqtls/reqtls.go
View File

@@ -1,59 +0,0 @@
// Package reqtls answers one question, in one place, for the whole
// application: did this request reach the service over TLS?
//
// It exists because that question used to be answered independently in
// several packages, by hand, and the answers disagreed. The session
// cookie's Secure attribute was decided at startup from the configured
// environment while the CSRF cookie's was decided per-request, so a
// deployment behind a TLS proxy in the default environment emitted one
// Secure cookie and one non-Secure cookie on the same response.
// Everything kept working, which is exactly why nobody noticed.
//
// Any code that needs a scheme or a Secure flag must call IsTLS rather
// than reading the request itself.
package reqtls
import (
"net/http"
"strings"
)
// forwardedProtoHeader is the de-facto standard header by which a
// TLS-terminating reverse proxy reports the protocol the CLIENT used.
const forwardedProtoHeader = "X-Forwarded-Proto"
// IsTLS reports whether the client-facing connection uses TLS: either
// the request arrived over TLS directly, or a reverse proxy terminated
// TLS and said so in X-Forwarded-Proto.
//
// The header is only as trustworthy as whatever sits in front of the
// listener. A proxy that overwrites it -- which is what the deployment
// documentation requires -- makes it authoritative; a listener exposed
// directly to clients lets any client assert it. That is the same
// exposure every X-Forwarded-* consumer carries.
func IsTLS(r *http.Request) bool {
return r.TLS != nil || forwardedProto(r) == "https"
}
// forwardedProto reduces X-Forwarded-Proto to a bare, comparable
// protocol token, or "" when the header is absent or blank.
//
// Two shapes that real infrastructure emits do not survive an exact
// comparison against "https", and both name a TLS client connection:
//
// - "HTTPS", because the header value is a case-insensitive token and
// nothing obliges a proxy to emit it lowercased.
// - "https, http", because a proxy chained behind another proxy
// APPENDS its own hop instead of replacing the value. As with
// X-Forwarded-For, the leftmost element is the one nearest the
// client, so it is the element that describes the browser's
// connection -- the only hop a cookie's Secure attribute is about.
//
// Landing on the plaintext path for either of those spellings is not a
// cosmetic error: it stops gorilla/csrf enforcing the strict Referer
// check on a site that genuinely is HTTPS.
func forwardedProto(r *http.Request) string {
first, _, _ := strings.Cut(r.Header.Get(forwardedProtoHeader), ",")
return strings.ToLower(strings.TrimSpace(first))
}

209
internal/reqtls/reqtls_test.go
View File

@@ -1,209 +0,0 @@
package reqtls_test
import (
"context"
"crypto/tls"
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
"sneak.berlin/go/webhooker/internal/reqtls"
)
// newReq builds a plaintext request with no forwarding headers.
func newReq(t *testing.T) *http.Request {
t.Helper()
return httptest.NewRequestWithContext(
context.Background(), http.MethodGet, "/", nil,
)
}
func TestIsTLS_DirectTLS(t *testing.T) {
t.Parallel()
r := newReq(t)
r.TLS = &tls.ConnectionState{}
assert.True(
t, reqtls.IsTLS(r),
"a request that arrived over TLS is TLS",
)
}
func TestIsTLS_PlaintextNoHeader(t *testing.T) {
t.Parallel()
assert.False(
t, reqtls.IsTLS(newReq(t)),
"no TLS connection and no header means plaintext",
)
}
// protoCase is one X-Forwarded-Proto spelling and the answer IsTLS
// owes it.
type protoCase struct {
name string
header string
want bool
why string
}
// protoCases enumerates the header values real infrastructure emits.
func protoCases() []protoCase {
return append(protoTLSCases(), protoPlaintextCases()...)
}
// protoTLSCases are the spellings that name a TLS client connection.
// Every one but the first is a spelling an exact == "https"
// comparison used to miss, silently downgrading a genuinely-HTTPS
// deployment to the plaintext path.
func protoTLSCases() []protoCase {
return []protoCase{
{
name: "lowercase",
header: "https",
want: true,
why: "the ordinary spelling",
},
{
name: "uppercase",
header: "HTTPS",
want: true,
why: "the value is a case-insensitive token; " +
"nothing obliges a proxy to lowercase it",
},
{
name: "mixed case",
header: "HttpS",
want: true,
why: "case folding must be total, not just the two extremes",
},
{
name: "chain with plaintext inner hop",
header: "https, http",
want: true,
why: "a chained proxy appends its hop; the leftmost " +
"element is the client-facing one",
},
{
name: "chain of two TLS hops",
header: "https,https",
want: true,
why: "appended chain with no space after the comma",
},
{
name: "trailing space",
header: "https ",
want: true,
why: "surrounding whitespace is not part of the token",
},
{
name: "leading space",
header: " https",
want: true,
why: "surrounding whitespace is not part of the token",
},
{
name: "uppercase chain",
header: "HTTPS, HTTP",
want: true,
why: "case folding and chain splitting must compose",
},
}
}
// protoPlaintextCases are the values that must NOT be read as TLS.
func protoPlaintextCases() []protoCase {
return []protoCase{
{
name: "plaintext",
header: "http",
want: false,
why: "the negative control: the proxy reports plaintext",
},
{
name: "plaintext chain with TLS inner hop",
header: "http, https",
want: false,
why: "the client-facing hop is plaintext even though " +
"an inner hop used TLS",
},
{
name: "empty",
header: "",
want: false,
why: "an empty header asserts nothing",
},
{
name: "whitespace only",
header: " ",
want: false,
why: "a blank header asserts nothing",
},
{
name: "unrelated token",
header: "ftp",
want: false,
why: "only https means TLS",
},
{
name: "https as a substring",
header: "nothttps",
want: false,
why: "matching must be on the whole token, not a substring",
},
}
}
func TestIsTLS_ForwardedProtoSpellings(t *testing.T) {
t.Parallel()
for _, tc := range protoCases() {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
r := newReq(t)
r.Header.Set("X-Forwarded-Proto", tc.header)
assert.Equal(
t, tc.want, reqtls.IsTLS(r),
"X-Forwarded-Proto %q: %s", tc.header, tc.why,
)
})
}
}
// TestIsTLS_DirectTLSBeatsPlaintextHeader pins the precedence: a
// connection this process itself terminated with TLS is a fact, and a
// header claiming otherwise does not override it.
func TestIsTLS_DirectTLSBeatsPlaintextHeader(t *testing.T) {
t.Parallel()
r := newReq(t)
r.TLS = &tls.ConnectionState{}
r.Header.Set("X-Forwarded-Proto", "http")
assert.True(
t, reqtls.IsTLS(r),
"an actual TLS connection outranks a header claiming plaintext",
)
}
// TestIsTLS_FirstHeaderValueWins covers a proxy that adds a second
// header line rather than appending to the existing one. net/http
// keeps them as separate values; the first is the client-facing hop,
// matching how the comma-separated form is read.
func TestIsTLS_FirstHeaderValueWins(t *testing.T) {
t.Parallel()
r := newReq(t)
r.Header.Add("X-Forwarded-Proto", "https")
r.Header.Add("X-Forwarded-Proto", "http")
assert.True(
t, reqtls.IsTLS(r),
"the first header line is the client-facing hop",
)
}

336
internal/server/bind_address_test.go
View File

@@ -1,336 +0,0 @@
package server_test
import (
"context"
"net"
"net/http"
"strconv"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/globals"
"sneak.berlin/go/webhooker/internal/server"
)
const (
// loopbackV4 is the shipped BIND_ADDRESS default.
loopbackV4 = "127.0.0.1"
// wildcardV4 is the value a container deployment must set,
// where a loopback-bound process is unreachable from outside
// its network namespace even with a published port.
wildcardV4 = "0.0.0.0"
// unavailableAddr is a TEST-NET-1 address (RFC 5737). It is a
// well-formed literal that no host is assigned, so binding it
// fails with EADDRNOTAVAIL rather than succeeding somewhere
// unexpected.
unavailableAddr = "192.0.2.1"
// listenReadyTimeout bounds the wait for the listener to accept
// connections. The bind itself is immediate; this only covers
// goroutine scheduling.
listenReadyTimeout = 3 * time.Second
// listenPollInterval is how often the readiness wait retries.
listenPollInterval = 10 * time.Millisecond
// dialTimeout bounds a single connection attempt in these
// tests. Everything dialled here is on this host, so a dial
// that is not answered immediately is a failure, not slowness.
dialTimeout = time.Second
)
// freePort returns a TCP port that is free on every local address at
// the moment it returns, by taking one on the wildcard and releasing
// it. The window between release and re-bind is the standard one
// every "pick a free port" helper carries.
func freePort(t *testing.T) int {
t.Helper()
var listenCfg net.ListenConfig
l, err := listenCfg.Listen(t.Context(), "tcp", "0.0.0.0:0")
require.NoError(t, err)
addr, ok := l.Addr().(*net.TCPAddr)
require.True(t, ok, "listener is not TCP")
require.NoError(t, l.Close())
return addr.Port
}
// otherLocalAddr returns a local IPv4 address that is not
// loopbackV4, or skips the test when the host has none.
//
// The bind-address tests need a second address of this host to stand
// in for "another interface": what a wildcard bind claims and a
// loopback bind does not. 127.0.0.2 is that address on Linux, where
// the whole 127.0.0.0/8 is local; elsewhere an interface address is
// used instead. Each candidate is proven bindable before it is
// returned, so a host that offers neither skips rather than fails on
// something that was never about the code under test.
func otherLocalAddr(t *testing.T) string {
t.Helper()
candidates := []string{"127.0.0.2"}
ifaceAddrs, err := net.InterfaceAddrs()
require.NoError(t, err)
for _, a := range ifaceAddrs {
ipNet, ok := a.(*net.IPNet)
if !ok {
continue
}
ip4 := ipNet.IP.To4()
if ip4 == nil || ip4.String() == loopbackV4 {
continue
}
candidates = append(candidates, ip4.String())
}
var listenCfg net.ListenConfig
for _, candidate := range candidates {
l, listenErr := listenCfg.Listen(
t.Context(), "tcp", net.JoinHostPort(candidate, "0"),
)
if listenErr != nil {
continue
}
require.NoError(t, l.Close())
return candidate
}
t.Skip("host has no second local IPv4 address to bind")
return ""
}
// startBoundServer starts the wired app with the given bind address
// on a free port and returns that port. The app is stopped on
// cleanup.
func startBoundServer(t *testing.T, bindAddress string) int {
t.Helper()
port := freePort(t)
env := newTestEnv(t)
env.cfg.BindAddress = bindAddress
env.cfg.Port = port
app := fx.New(
fx.NopLogger,
fx.Supply(env.log, env.cfg, env.mw, env.hnd),
fx.Provide(globals.New, server.New),
fx.Invoke(func(*server.Server) {}),
)
startCtx, cancelStart := context.WithTimeout(
context.Background(), lifecycleTimeout,
)
defer cancelStart()
require.NoError(t, app.Start(startCtx))
t.Cleanup(func() {
stopCtx, cancelStop := context.WithTimeout(
context.Background(), lifecycleTimeout,
)
defer cancelStop()
require.NoError(t, app.Stop(stopCtx))
})
return port
}
// dialable reports whether a TCP connection to addr succeeds.
func dialable(ctx context.Context, addr string) bool {
dialer := net.Dialer{Timeout: dialTimeout}
conn, err := dialer.DialContext(ctx, "tcp", addr)
if err != nil {
return false
}
_ = conn.Close()
return true
}
// requireDialable waits for addr to accept connections, failing the
// test if it never does.
func requireDialable(t *testing.T, addr string) {
t.Helper()
deadline := time.Now().Add(listenReadyTimeout)
for time.Now().Before(deadline) {
if dialable(t.Context(), addr) {
return
}
time.Sleep(listenPollInterval)
}
t.Fatalf("nothing accepted connections on %s", addr)
}
// TestListenAddr pins how BindAddress and Port are rendered into the
// listen address.
//
// The defect this covers was a bare fmt.Sprintf(":%d", port), which
// binds every interface with no way to say otherwise. The IPv6 rows
// are here because an unbracketed IPv6 host would produce an address
// net.Listen rejects, turning a valid configuration into a startup
// failure.
func TestListenAddr(t *testing.T) {
t.Parallel()
tests := []struct {
name string
bindAddress string
port int
expected string
}{
{
name: "loopback default",
bindAddress: loopbackV4,
port: 8080,
expected: "127.0.0.1:8080",
},
{
name: "ipv4 wildcard",
bindAddress: wildcardV4,
port: 8080,
expected: "0.0.0.0:8080",
},
{
name: "ipv6 wildcard is bracketed",
bindAddress: "::",
port: 8080,
expected: "[::]:8080",
},
{
name: "ipv6 literal is bracketed",
bindAddress: "2001:db8::5",
port: 9001,
expected: "[2001:db8::5]:9001",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
assert.Equal(t, tt.expected, server.ListenAddrForTest(
&config.Config{
BindAddress: tt.bindAddress,
Port: tt.port,
},
))
})
}
}
// TestBindAddress_LoopbackIsNotOnOtherAddresses proves the fix end to
// end: with BIND_ADDRESS at its loopback default, the cleartext
// listener answers on loopback and has not claimed any other address
// of this host.
//
// The second address is proven free by binding it on the same port
// while the server runs. That is the assertion that fails against the
// old wildcard bind — a wildcard listener owns the port on every
// address, so this bind would return EADDRINUSE. Dialling from
// another machine is what the operator cares about, and this is the
// in-process form of it: the socket the remote host would connect to
// does not exist.
func TestBindAddress_LoopbackIsNotOnOtherAddresses(t *testing.T) {
t.Parallel()
other := otherLocalAddr(t)
port := startBoundServer(t, loopbackV4)
// Positive control: the service really is up and serving.
requireDialable(t, net.JoinHostPort(loopbackV4, strconv.Itoa(port)))
var listenCfg net.ListenConfig
l, err := listenCfg.Listen(
t.Context(), "tcp",
net.JoinHostPort(other, strconv.Itoa(port)),
)
require.NoError(
t, err,
"port %d on %s is taken while bound to %s: the listener "+
"claimed more than its configured address",
port, other, loopbackV4,
)
require.NoError(t, l.Close())
}
// TestBindAddress_WildcardReachesOtherAddresses is the counterpart:
// the value a container deployment sets does reach the addresses the
// default withholds. Without this, a loopback-only bind would pass
// the test above by never listening at all.
func TestBindAddress_WildcardReachesOtherAddresses(t *testing.T) {
t.Parallel()
other := otherLocalAddr(t)
port := startBoundServer(t, wildcardV4)
requireDialable(t, net.JoinHostPort(other, strconv.Itoa(port)))
}
// TestBindAddress_ServesRequestsOnConfiguredAddress proves the bound
// listener serves the application rather than merely accepting TCP,
// so a bind address that is honoured cannot be mistaken for one that
// is honoured and broken.
func TestBindAddress_ServesRequestsOnConfiguredAddress(t *testing.T) {
t.Parallel()
port := startBoundServer(t, loopbackV4)
addr := net.JoinHostPort(loopbackV4, strconv.Itoa(port))
requireDialable(t, addr)
req, err := http.NewRequestWithContext(
t.Context(), http.MethodGet,
"http://"+addr+"/.well-known/healthcheck", nil,
)
require.NoError(t, err)
client := &http.Client{Timeout: dialTimeout}
resp, err := client.Do(req)
require.NoError(t, err)
defer func() { _ = resp.Body.Close() }()
assert.Equal(t, http.StatusOK, resp.StatusCode)
}
// TestBindAddress_UnavailableAddressShutsDownTheApp covers the half
// of the fail-loud rule that configuration parsing cannot reach. A
// syntactically valid address that is not assigned to this host
// parses fine and fails at bind time, after fx has already reported
// RUNNING. It must end the process non-zero rather than leave it
// alive with nothing listening.
func TestBindAddress_UnavailableAddressShutsDownTheApp(t *testing.T) {
t.Parallel()
env := newTestEnv(t)
env.cfg.BindAddress = unavailableAddr
env.cfg.Port = freePort(t)
requireListenFailureExit(t, env)
}

91
internal/server/early_shutdown_test.go
View File

@@ -1,91 +0,0 @@
package server_test
import (
"context"
"testing"
"github.com/stretchr/testify/require"
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/globals"
"sneak.berlin/go/webhooker/internal/server"
)
// earlyStopIterations is how many start/stop cycles the race test
// runs. The window it aims at is the gap between the OnStart hook
// returning and the serving goroutine reaching its first field
// access, which is microseconds wide. The race detector reports an
// unsynchronised pair whenever it observes one, but it has to observe
// one, so a single cycle can miss purely on scheduling. Repetition
// makes the observation reliable; the collaborators are built once,
// so the cycles themselves are cheap.
const earlyStopIterations = 25
// TestEarlyShutdown_NoPanicAndNoRace stops the application
// immediately after starting it, before the serving goroutine has
// necessarily run at all.
//
// Two defects live in that window. The OnStart hook returns as soon
// as it has spawned the serving goroutine, so fx runs the stop
// sequence against a Server whose serving goroutine may not have
// executed a single line. cleanShutdown called Shutdown on an
// httpServer that goroutine was supposed to assign, which was a nil
// dereference on an early SIGTERM; and it read httpServer and
// sentryEnabled with nothing ordering those reads against the
// goroutine's writes, which is a data race that only surfaces once
// something both starts and stops the server. Nothing did before this
// test: the listen-failure test never binds, and the router tests
// bypass the lifecycle entirely.
//
// httpServer is now built in New, on the constructing goroutine, so
// it is written before any hook exists and can never be nil.
// sentryEnabled is atomic. This test is what catches either one
// coming back — under -race, which is how the suite runs.
func TestEarlyShutdown_NoPanicAndNoRace(t *testing.T) {
t.Parallel()
// Built once: the collaborators are not what is under test, and
// standing up a database per iteration would make repetition too
// expensive to be worth having.
env := newTestEnv(t)
env.cfg.BindAddress = loopbackV4
for range earlyStopIterations {
requireStartStopIsClean(t, env)
}
}
// requireStartStopIsClean runs one start/stop cycle with no wait in
// between, failing the test if either half errors.
//
// Each cycle gets a fresh fx app, so the Server under test is
// constructed anew every time — that construction is where the
// httpServer write now happens, and reusing one Server would test it
// only once.
func requireStartStopIsClean(t *testing.T, env *testEnv) {
t.Helper()
env.cfg.Port = freePort(t)
app := fx.New(
fx.NopLogger,
fx.Supply(env.log, env.cfg, env.mw, env.hnd),
fx.Provide(globals.New, server.New),
fx.Invoke(func(*server.Server) {}),
)
startCtx, cancelStart := context.WithTimeout(
context.Background(), lifecycleTimeout,
)
defer cancelStart()
require.NoError(t, app.Start(startCtx))
// No sleep and no readiness wait: stopping while the serving
// goroutine is still in flight is the whole point.
stopCtx, cancelStop := context.WithTimeout(
context.Background(), lifecycleTimeout,
)
defer cancelStop()
require.NoError(t, app.Stop(stopCtx))
}

20
internal/server/export_test.go
View File

@@ -55,16 +55,6 @@ func NewRouterForTest(
return s.router return s.router
} }
// ListenAddrForTest exposes the address the HTTP listener binds for
// a given Config, so the rendering of host and port — IPv6
// bracketing above all — can be pinned without standing up a
// listener.
func ListenAddrForTest(cfg *config.Config) string {
s := &Server{params: ServerParams{Config: cfg}}
return s.listenAddr()
}
// ProbePattern is the route NewRouterWithProbeForTest adds to the // ProbePattern is the route NewRouterWithProbeForTest adds to the
// production route tree. // production route tree.
const ProbePattern = "/probe" const ProbePattern = "/probe"
@@ -90,12 +80,12 @@ func NewRouterWithProbeForTest(
probe http.HandlerFunc, probe http.HandlerFunc,
) http.Handler { ) http.Handler {
s := &Server{ s := &Server{
log: log, log: log,
mw: mw, mw: mw,
h: h, h: h,
params: ServerParams{Config: cfg}, params: ServerParams{Config: cfg},
sentryEnabled: sentryEnabled,
} }
s.sentryEnabled.Store(sentryEnabled)
s.SetupRoutes() s.SetupRoutes()
s.router.Handle(ProbePattern, probe) s.router.Handle(ProbePattern, probe)

41
internal/server/http.go
View File

@@ -2,9 +2,8 @@ package server
import ( import (
"errors" "errors"
"net" "fmt"
"net/http" "net/http"
"strconv"
"time" "time"
) )
@@ -25,52 +24,26 @@ const (
httpMaxHeaderBytes = 1 << 20 httpMaxHeaderBytes = 1 << 20
) )
// listenAddr renders the address the HTTP listener binds. func (s *Server) serveUntilShutdown() {
// listenAddr := fmt.Sprintf(":%d", s.params.Config.Port)
// The host half is always present: an empty host would be the s.httpServer = &http.Server{
// wildcard, and the whole point of BIND_ADDRESS is that binding every Addr: listenAddr,
// interface is a choice the operator makes rather than one the
// process makes for them. Config guarantees a literal, so
// JoinHostPort's bracketing is enough to make IPv6 well formed.
func (s *Server) listenAddr() string {
return net.JoinHostPort(
s.params.Config.BindAddress,
strconv.Itoa(s.params.Config.Port),
)
}
// newHTTPServer builds the HTTP server for this Server's
// configuration.
//
// It is called from New, on the constructing goroutine, rather than
// from the serving goroutine that used to assign s.httpServer
// directly. Two goroutines reach that field — the serving goroutine
// and the fx stop hook, which calls Shutdown on it — with nothing
// ordering them. Constructing it during New puts the write before
// every hook fx will later run, which both removes the race and rules
// out the nil dereference a stop that arrived before the serving
// goroutine had run would have caused.
func (s *Server) newHTTPServer() *http.Server {
return &http.Server{
Addr: s.listenAddr(),
ReadTimeout: httpReadTimeout, ReadTimeout: httpReadTimeout,
WriteTimeout: httpWriteTimeout, WriteTimeout: httpWriteTimeout,
MaxHeaderBytes: httpMaxHeaderBytes, MaxHeaderBytes: httpMaxHeaderBytes,
Handler: s, Handler: s,
} }
}
func (s *Server) serveUntilShutdown() {
// add routes // add routes
// this does any necessary setup in each handler // this does any necessary setup in each handler
s.SetupRoutes() s.SetupRoutes()
s.log.Info("http begin listen", "listenaddr", s.httpServer.Addr) s.log.Info("http begin listen", "listenaddr", listenAddr)
err := s.httpServer.ListenAndServe() err := s.httpServer.ListenAndServe()
if err != nil && !errors.Is(err, http.ErrServerClosed) { if err != nil && !errors.Is(err, http.ErrServerClosed) {
s.log.Error("listen error", "error", err) s.log.Error("listen error", "error", err)
s.shutdownWithFailure() s.shutdownOnListenFailure()
} }
} }

28
internal/server/listen_failure_test.go
View File

@@ -36,9 +36,9 @@ const lifecycleTimeout = 15 * time.Second
// //
// The port is occupied by a listener this test holds open, on a // The port is occupied by a listener this test holds open, on a
// kernel-chosen port, so the failure is the real EADDRINUSE the // kernel-chosen port, so the failure is the real EADDRINUSE the
// operator hits when a second instance starts. The server is pointed // operator hits when a second instance starts. Loopback is enough to
// at the same loopback address, so the collision is a direct one on // collide with the server's wildcard bind: a listening socket on a
// the exact address it asks the kernel for. // specific address blocks the wildcard from claiming the same port.
func TestListenFailure_ShutsDownTheApp(t *testing.T) { func TestListenFailure_ShutsDownTheApp(t *testing.T) {
t.Parallel() t.Parallel()
@@ -55,27 +55,11 @@ func TestListenFailure_ShutsDownTheApp(t *testing.T) {
require.True(t, ok, "listener is not TCP") require.True(t, ok, "listener is not TCP")
// The collaborators come from the wired graph rather than stubs, // The collaborators come from the wired graph rather than stubs,
// so the Server under test is the one that ships. Only the // so the Server under test is the one that ships. Only the port
// listen address is test-specific. // is test-specific.
env := newTestEnv(t) env := newTestEnv(t)
env.cfg.BindAddress = loopbackV4
env.cfg.Port = addr.Port env.cfg.Port = addr.Port
requireListenFailureExit(t, env)
}
// requireListenFailureExit starts the wired app over env and asserts
// that it gives up on its own with the listen-failure status, then
// completes its stop sequence.
//
// Two different listen failures share it — a port already in use and
// an address that is not on this host — because what has to hold for
// both is the same: the failure is discovered after fx has already
// reported RUNNING, so the only thing that can turn it into a visible
// exit is the shutdown path under test.
func requireListenFailureExit(t *testing.T, env *testEnv) {
t.Helper()
app := fx.New( app := fx.New(
fx.NopLogger, fx.NopLogger,
fx.Supply(env.log, env.cfg, env.mw, env.hnd), fx.Supply(env.log, env.cfg, env.mw, env.hnd),
@@ -93,7 +77,7 @@ func requireListenFailureExit(t *testing.T, env *testEnv) {
select { select {
case sig := <-app.Wait(): case sig := <-app.Wait():
require.Equal( require.Equal(
t, server.StartupFailureExitCode, sig.ExitCode, t, server.ListenFailureExitCode, sig.ExitCode,
"listen failure must exit non-zero", "listen failure must exit non-zero",
) )
case <-time.After(listenFailureDeadline): case <-time.After(listenFailureDeadline):

6
internal/server/routes.go
View File

@@ -81,7 +81,7 @@ func (s *Server) setupGlobalMiddleware() {
// Sentry error reporting (if SENTRY_DSN is set). Repanic is // Sentry error reporting (if SENTRY_DSN is set). Repanic is
// true so panics still bubble up to the Recoverer middleware // true so panics still bubble up to the Recoverer middleware
// registered immediately above. // registered immediately above.
if s.sentryEnabled.Load() { if s.sentryEnabled {
sentryHandler := sentryhttp.New(sentryhttp.Options{ sentryHandler := sentryhttp.New(sentryhttp.Options{
Repanic: true, Repanic: true,
}) })
@@ -237,6 +237,10 @@ func (s *Server) setupSourceRoutes() {
"/entrypoints/{entrypointID}/toggle", "/entrypoints/{entrypointID}/toggle",
s.h.HandleEntrypointToggle(), s.h.HandleEntrypointToggle(),
) )
r.Post(
"/entrypoints/{entrypointID}/secret",
s.h.HandleEntrypointSecret(),
)
r.Post("/targets", s.h.HandleTargetCreate()) r.Post("/targets", s.h.HandleTargetCreate())
// The edit form is the one page that renders a target's // The edit form is the one page that renders a target's
// destination URL and header values in full; see // destination URL and header values in full; see

11
internal/server/sentry.go
View File

@@ -147,13 +147,10 @@ func sentryRoutePattern(hint *sentry.EventHint) string {
// //
// The scheme is load-bearing and is kept: the SDK derives it from // The scheme is load-bearing and is kept: the SDK derives it from
// r.TLS != nil || r.Header.Get("X-Forwarded-Proto") == "https" // r.TLS != nil || r.Header.Get("X-Forwarded-Proto") == "https"
// (interfaces.go:180), which is the reason dropping X-Forwarded-Proto // (interfaces.go:180), byte for byte the predicate
// from the header allowlist costs nothing. That predicate is the SDK's // internal/middleware/csrf.go uses, so it is the CSRF TLS decision and
// own and is stricter than reqtls.IsTLS, which this service now uses // the reason dropping X-Forwarded-Proto from the header allowlist
// everywhere it decides transport: the SDK reports "http" for the // costs nothing. The host is parsed.Host of the SDK's
// "HTTPS" and "https, http" spellings reqtls accepts. Only a reported
// scheme is affected, no decision is, so it is left to the SDK rather
// than reimplemented. The host is parsed.Host of the SDK's
// scheme://r.Host/path, so it is whatever the client's Host header // scheme://r.Host/path, so it is whatever the client's Host header
// carried: this service validates no hostname. It is kept because that // carried: this service validates no hostname. It is kept because that
// same header is on the allowlist, so scrubbing it here would withhold // same header is on the allowlist, so scrubbing it here would withhold

99
internal/server/sentry_failure_test.go
View File

@@ -1,99 +0,0 @@
package server_test
import (
"context"
"net"
"strconv"
"testing"
"time"
"github.com/stretchr/testify/require"
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/globals"
"sneak.berlin/go/webhooker/internal/server"
)
// TestSentryInitFailure_ShutsDownTheApp pins that error reporting
// which is configured and cannot be started ends the application
// instead of serving without it.
//
// The measured defect logged `sentry init failure` and kept running,
// so the deployment served traffic with reporting off while every
// other signal — SENTRY_DSN still set, the startup summary's own
// field — said it was on. Nothing later in the process can notice
// that reports are going nowhere, which is why this exits rather than
// degrades.
//
// The DSN is placed on a hand-built Config, which is the only way to
// reach this branch at all: loadFromEnv now parses SENTRY_DSN with
// sentry.NewDsn, the same call sentry.Init makes, so a DSN that
// survives configuration cannot fail initialisation in the SDK
// version this pins. The branch stays because that is a property of
// the SDK's current implementation rather than of its contract.
func TestSentryInitFailure_ShutsDownTheApp(t *testing.T) {
t.Parallel()
port := freePort(t)
env := newTestEnvWithConfig(t, &config.Config{
DataDir: t.TempDir(),
Environment: config.EnvironmentDev,
BindAddress: loopbackV4,
Port: port,
SentryDSN: "not-a-dsn",
})
app := fx.New(
fx.NopLogger,
fx.Supply(env.log, env.cfg, env.mw, env.hnd),
fx.Provide(globals.New, server.New),
fx.Invoke(func(*server.Server) {}),
)
startCtx, cancelStart := context.WithTimeout(
context.Background(), lifecycleTimeout,
)
defer cancelStart()
require.NoError(t, app.Start(startCtx))
select {
case sig := <-app.Wait():
require.Equal(
t, server.StartupFailureExitCode, sig.ExitCode,
"a sentry failure must exit non-zero",
)
case <-time.After(listenFailureDeadline):
t.Fatal("a sentry failure left the app running")
}
// The stop sequence still has to complete: the failure must reach
// shutdown through fx rather than around it.
stopCtx, cancelStop := context.WithTimeout(
context.Background(), lifecycleTimeout,
)
defer cancelStop()
require.NoError(t, app.Stop(stopCtx))
// And it must give up before it listens. A process that bound the
// port and then exited would have accepted requests it could not
// report on, which is the state under test in miniature.
requireBindable(t, port)
}
// requireBindable asserts that the port is free, which it is only if
// the server under test never claimed it.
func requireBindable(t *testing.T, port int) {
t.Helper()
var listenCfg net.ListenConfig
listener, err := listenCfg.Listen(
t.Context(), "tcp",
net.JoinHostPort(loopbackV4, strconv.Itoa(port)),
)
require.NoError(t, err, "the server bound a port it then gave up")
require.NoError(t, listener.Close())
}

102
internal/server/server.go
View File

@@ -9,7 +9,6 @@ import (
"net/http" "net/http"
"os" "os"
"os/signal" "os/signal"
"sync/atomic"
"syscall" "syscall"
"time" "time"
@@ -51,13 +50,12 @@ const (
minSentryFlush = 250 * time.Millisecond minSentryFlush = 250 * time.Millisecond
) )
// StartupFailureExitCode is the status the process exits with when // ListenFailureExitCode is the status the process exits with when the
// the serving goroutine gives up: the HTTP listener cannot be // HTTP listener cannot be established, or dies for a reason other
// established or dies for a reason other than a requested shutdown, or // than a requested shutdown. It must stay non-zero: systemd
// error reporting is configured and cannot be started. It must stay // `Restart=on-failure` and Docker's restart policies key off it, and a
// non-zero: systemd `Restart=on-failure` and Docker's restart policies // zero exit would read as a deliberate stop.
// key off it, and a zero exit would read as a deliberate stop. const ListenFailureExitCode = 1
const StartupFailureExitCode = 1
// SentryFlushBudget reports how long the Sentry flush may run when // SentryFlushBudget reports how long the Sentry flush may run when
// remaining is the time left on the fx stop context after the HTTP // remaining is the time left on the fx stop context after the HTTP
@@ -90,15 +88,8 @@ type ServerParams struct {
// Server is the main HTTP server that wires up routes and manages // Server is the main HTTP server that wires up routes and manages
// graceful shutdown. // graceful shutdown.
type Server struct { type Server struct {
startupTime time.Time startupTime time.Time
sentryEnabled bool
// sentryEnabled is written by the serving goroutine, in
// enableSentry, and read by the fx stop hook in cleanShutdown.
// Nothing orders those two: the OnStart hook returns as soon as
// the goroutine is spawned, so a stop can be running while
// enableSentry is still deciding. It is atomic to supply the
// edge the goroutines do not.
sentryEnabled atomic.Bool
log *slog.Logger log *slog.Logger
cancelFunc context.CancelFunc cancelFunc context.CancelFunc
httpServer *http.Server httpServer *http.Server
@@ -116,7 +107,6 @@ func New(lc fx.Lifecycle, params ServerParams) (*Server, error) {
s.mw = params.Middleware s.mw = params.Middleware
s.h = params.Handlers s.h = params.Handlers
s.log = params.Logger.Get() s.log = params.Logger.Get()
s.httpServer = s.newHTTPServer()
lc.Append(fx.Hook{ lc.Append(fx.Hook{
OnStart: func(_ context.Context) error { OnStart: func(_ context.Context) error {
@@ -136,25 +126,11 @@ func New(lc fx.Lifecycle, params ServerParams) (*Server, error) {
} }
// Run configures Sentry and starts serving HTTP requests. // Run configures Sentry and starts serving HTTP requests.
//
// A Sentry failure ends the application instead of listening. It runs
// before the listener rather than after it so that the process never
// binds a port it is about to give up.
func (s *Server) Run() { func (s *Server) Run() {
s.configure() s.configure()
// logging before sentry, because sentry logs // logging before sentry, because sentry logs
err := s.enableSentry() s.enableSentry()
if err != nil {
s.log.Error(
"SENTRY_DSN is set but error reporting could not be "+
"started; refusing to serve with it off",
"error", err,
)
s.shutdownWithFailure()
return
}
s.serve() s.serve()
} }
@@ -165,23 +141,11 @@ func (s *Server) MaintenanceMode() bool {
return s.params.Config.MaintenanceMode return s.params.Config.MaintenanceMode
} }
// enableSentry initialises the Sentry SDK when error reporting is func (s *Server) enableSentry() {
// configured, and reports the failure when it is configured and cannot s.sentryEnabled = false
// be initialised. A DSN that is not set is not a failure: reporting
// stays off and the server starts normally.
//
// There is no fallback to running with reporting off. An operator who
// set SENTRY_DSN asked for failures to be visible, and serving traffic
// with reporting quietly off is the one state nothing can ever tell
// them about — the DSN is still set, so every later signal says it is
// on. Config already refused a DSN the SDK cannot parse, which is what
// a typo produces, so reaching this branch means the SDK refused
// something that parsed: not a condition to guess at either.
func (s *Server) enableSentry() error {
s.sentryEnabled.Store(false)
if !s.params.Config.SentryEnabled() { if s.params.Config.SentryDSN == "" {
return nil return
} }
err := sentry.Init(sentryClientOptions( err := sentry.Init(sentryClientOptions(
@@ -193,19 +157,19 @@ func (s *Server) enableSentry() error {
), ),
)) ))
if err != nil { if err != nil {
return fmt.Errorf("initialising sentry: %w", err) s.log.Error("sentry init failure", "error", err)
// Don't use fatal since we still want the service to run
return
} }
s.log.Info("sentry error reporting activated") s.log.Info("sentry error reporting activated")
s.sentryEnabled.Store(true) s.sentryEnabled = true
return nil
} }
// serve installs the signal watcher, starts the listener and blocks // serve installs the signal watcher, starts the listener and blocks
// until the server's context is cancelled. The process exit status is // until the server's context is cancelled. The process exit status is
// fx's to decide — from a signal, or from the code // fx's to decide — from a signal, or from the code
// shutdownWithFailure hands the Shutdowner — so this reports // shutdownOnListenFailure hands the Shutdowner — so this reports
// nothing back to its caller. // nothing back to its caller.
func (s *Server) serve() { func (s *Server) serve() {
ctx, cancelFunc := context.WithCancel(context.Background()) ctx, cancelFunc := context.WithCancel(context.Background())
@@ -235,24 +199,20 @@ func (s *Server) serve() {
// Do not call cleanShutdown() here to avoid double invocation. // Do not call cleanShutdown() here to avoid double invocation.
} }
// shutdownWithFailure ends the application non-zero from the serving // shutdownOnListenFailure ends the application after the HTTP
// goroutine. It is how anything on that goroutine fails fatally: the // listener failed. The fx OnStart hook returns as soon as the serving
// fx OnStart hook returns as soon as the goroutine is spawned, so // goroutine is spawned, so nothing downstream of it ever learns that
// nothing downstream of it ever learns that the goroutine gave up. fx // the listen failed: fx reports RUNNING and the process sits alive
// reports RUNNING and the process sits alive having done neither what // with nothing bound, which is invisible to systemd and Docker
// it was asked nor anything visible instead, which systemd and // restart policies. Asking the Shutdowner to stop the app with a
// Docker restart policies cannot see. Asking the Shutdowner to stop // non-zero code is what turns that into a visible failure.
// the app with a non-zero code is what turns that into a visible
// failure, and it is the whole of "fatal" here — no panic, no
// os.Exit, and every stop hook still runs.
// //
// The context cancel that follows only unwinds serve()'s own wait, // The context cancel that follows only unwinds serve()'s own wait.
// and is skipped before serve has installed one. The shutdown itself // The shutdown itself runs through fx's normal stop sequence, so the
// runs through fx's normal stop sequence, so the clean-shutdown drain // clean-shutdown drain in cleanShutdown is reached unchanged.
// in cleanShutdown is reached unchanged. func (s *Server) shutdownOnListenFailure() {
func (s *Server) shutdownWithFailure() {
err := s.params.Shutdowner.Shutdown( err := s.params.Shutdowner.Shutdown(
fx.ExitCode(StartupFailureExitCode), fx.ExitCode(ListenFailureExitCode),
) )
if err != nil { if err != nil {
s.log.Error("shutdown request failed", "error", err) s.log.Error("shutdown request failed", "error", err)
@@ -282,7 +242,7 @@ func (s *Server) cleanShutdown(ctx context.Context) {
s.cleanupForExit() s.cleanupForExit()
if s.sentryEnabled.Load() { if s.sentryEnabled {
s.flushSentry(ctx) s.flushSentry(ctx)
} }
} }

4
internal/session/export_test.go
View File

@@ -5,6 +5,6 @@ import "github.com/gorilla/sessions"
// NewStore exposes the production cookie-store constructor so tests // NewStore exposes the production cookie-store constructor so tests
// exercise the store the application actually runs with, rather than a // exercise the store the application actually runs with, rather than a
// lookalike assembled in the test. // lookalike assembled in the test.
func NewStore(key []byte) *sessions.CookieStore { func NewStore(key []byte, secure bool) *sessions.CookieStore {
return newStore(key) return newStore(key, secure)
} }

66
internal/session/session.go
View File

@@ -17,7 +17,6 @@ import (
"sneak.berlin/go/webhooker/internal/config" "sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database" "sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/logger" "sneak.berlin/go/webhooker/internal/logger"
"sneak.berlin/go/webhooker/internal/reqtls"
) )
const ( const (
@@ -85,9 +84,10 @@ type Params struct {
// Session manages encrypted session storage. // Session manages encrypted session storage.
type Session struct { type Session struct {
store *sessions.CookieStore store *sessions.CookieStore
key []byte // raw 32-byte auth key, also used for CSRF cookie signing key []byte // raw 32-byte auth key, also used for CSRF cookie signing
log *slog.Logger log *slog.Logger
config *config.Config
// idleTimeout is the sliding inactivity window. A session that // idleTimeout is the sliding inactivity window. A session that
// sees no authenticated request within this window expires, // sees no authenticated request within this window expires,
@@ -104,10 +104,6 @@ type Session struct {
// cookie. MaxAge is deliberately left at its zero value: for a store // cookie. MaxAge is deliberately left at its zero value: for a store
// it is set through CookieStore.MaxAge (see newStore), and for a // it is set through CookieStore.MaxAge (see newStore), and for a
// single session it is copied from the store's options. // single session it is copied from the store's options.
//
// Secure is a parameter rather than a constant because it is the one
// attribute here that is not a policy -- it is a fact about the
// connection carrying this particular response. See applyTransport.
func cookieOptions(secure bool) *sessions.Options { func cookieOptions(secure bool) *sessions.Options {
return &sessions.Options{ return &sessions.Options{
Path: "/", Path: "/",
@@ -125,52 +121,14 @@ func cookieOptions(secure bool) *sessions.Options {
// Options never touches Codecs -- so a store configured that way still // Options never touches Codecs -- so a store configured that way still
// decodes a 30-day-old cookie, leaving the cookie attribute and the // decodes a 30-day-old cookie, leaving the cookie attribute and the
// codec disagreeing about the same policy. store.MaxAge sets both. // codec disagreeing about the same policy. store.MaxAge sets both.
// func newStore(key []byte, secure bool) *sessions.CookieStore {
// The store's Secure is fixed at true, and is only a template: every
// write path overwrites it for the request in hand (applyTransport).
// It is true rather than false so that a write path added later which
// forgets to call applyTransport fails loudly -- the browser drops the
// cookie over plaintext HTTP and the developer sees it immediately --
// instead of silently shipping the authentication credential without
// Secure, which is the exact failure this store already had once.
func newStore(key []byte) *sessions.CookieStore {
store := sessions.NewCookieStore(key) store := sessions.NewCookieStore(key)
store.Options = cookieOptions(true) store.Options = cookieOptions(secure)
store.MaxAge(secondsPerDay * sessionMaxAgeDays) store.MaxAge(secondsPerDay * sessionMaxAgeDays)
return store return store
} }
// applyTransport sets the session cookie's Secure attribute from the
// transport of the request being answered.
//
// This is decided per-request, not once at startup. Deciding it at
// startup from the configured environment is what this replaces, and
// it got the DEFAULT posture wrong: "dev" is the environment when
// WEBHOOKER_ENVIRONMENT is unset, so a deployment terminating TLS at a
// proxy without also setting the environment emitted the
// authentication cookie with no Secure attribute -- silently, and on
// the same response as a CSRF cookie that did have one.
//
// gorilla/sessions makes this cheap and local: CookieStore.New gives
// every session its own copy of the store's Options, and
// CookieStore.Save renders the cookie from that copy rather than from
// the store. So the flag is set on the one session being saved,
// without a second store and without reaching across concurrent
// requests.
//
// The flag tracks the transport in BOTH directions rather than being
// latched on once seen. Secure on a plaintext response is worse than
// useless: the browser discards such a cookie without any error, so a
// latched flag would make a plain-HTTP local run impossible to log
// into. It is also why every write path must call this, including the
// deletion cookies in Destroy and Regenerate -- a Secure deletion
// cookie sent over plaintext is dropped too, leaving the session the
// caller believed it had just revoked.
func applyTransport(r *http.Request, sess *sessions.Session) {
sess.Options.Secure = reqtls.IsTLS(r)
}
// New creates a new session manager. The cookie store is // New creates a new session manager. The cookie store is
// initialized during the fx OnStart phase after the database is // initialized during the fx OnStart phase after the database is
// connected, using a session key that is auto-generated and stored // connected, using a session key that is auto-generated and stored
@@ -181,6 +139,7 @@ func New(
) (*Session, error) { ) (*Session, error) {
s := &Session{ s := &Session{
log: params.Logger.Get(), log: params.Logger.Get(),
config: params.Config,
idleTimeout: params.Config.SessionIdleTimeout, idleTimeout: params.Config.SessionIdleTimeout,
now: time.Now, now: time.Now,
} }
@@ -213,7 +172,7 @@ func New(
} }
s.key = keyBytes s.key = keyBytes
s.store = newStore(keyBytes) s.store = newStore(keyBytes, !params.Config.IsDev())
s.log.Info("session manager initialized") s.log.Info("session manager initialized")
return nil return nil
@@ -237,16 +196,12 @@ func (s *Session) GetKey() []byte {
return s.key return s.key
} }
// Save saves the session. Every session-cookie write in the // Save saves the session.
// application goes through here or through Regenerate, which is what
// makes applyTransport a complete answer rather than a best effort.
func (s *Session) Save( func (s *Session) Save(
r *http.Request, r *http.Request,
w http.ResponseWriter, w http.ResponseWriter,
sess *sessions.Session, sess *sessions.Session,
) error { ) error {
applyTransport(r, sess)
return sess.Save(r, w) return sess.Save(r, w)
} }
@@ -385,7 +340,6 @@ func (s *Session) Regenerate(
// Destroy the old session // Destroy the old session
oldSess.Options.MaxAge = -1 oldSess.Options.MaxAge = -1
s.ClearUser(oldSess) s.ClearUser(oldSess)
applyTransport(r, oldSess)
err := oldSess.Save(r, w) err := oldSess.Save(r, w)
if err != nil { if err != nil {
@@ -414,7 +368,7 @@ func (s *Session) Regenerate(
// Apply the standard session options (the destroyed old // Apply the standard session options (the destroyed old
// session had MaxAge = -1, which store.New might inherit // session had MaxAge = -1, which store.New might inherit
// from the cookie). // from the cookie).
newSess.Options = cookieOptions(reqtls.IsTLS(r)) newSess.Options = cookieOptions(!s.config.IsDev())
newSess.Options.MaxAge = secondsPerDay * sessionMaxAgeDays newSess.Options.MaxAge = secondsPerDay * sessionMaxAgeDays
return newSess, nil return newSess, nil

264
internal/session/session_test.go
View File

@@ -2,7 +2,6 @@ package session_test
import ( import (
"context" "context"
"crypto/tls"
"log/slog" "log/slog"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
@@ -74,7 +73,7 @@ func testSessionWithClock(
t.Helper() t.Helper()
key := testKey() key := testKey()
store := session.NewStore(key) store := session.NewStore(key, false)
cfg := &config.Config{ cfg := &config.Config{
Environment: config.EnvironmentDev, Environment: config.EnvironmentDev,
@@ -881,264 +880,3 @@ func TestDestroy_ThenSave_DeletesCookie(t *testing.T) {
"destroyed session cookie should have negative MaxAge", "destroyed session cookie should have negative MaxAge",
) )
} }
// --- Secure Attribute / Transport Tests ---
// transportCase describes one client-facing transport and the Secure
// attribute the session cookie must carry for it.
type transportCase struct {
name string
tls bool
header string
want bool
why string
}
// transportCases enumerates the transports the session cookie has to
// get right. Every https spelling here is one a real proxy emits.
func transportCases() []transportCase {
return []transportCase{
{
name: "direct TLS",
tls: true,
want: true,
why: "this process terminated TLS itself",
},
{
name: "proxy reports https",
header: "https",
want: true,
why: "the ordinary reverse-proxy deployment",
},
{
name: "proxy reports HTTPS",
header: "HTTPS",
want: true,
why: "the header value is a case-insensitive token",
},
{
name: "appended chain https, http",
header: "https, http",
want: true,
why: "the leftmost hop is the browser's connection",
},
{
name: "appended chain https,https",
header: "https,https",
want: true,
why: "two TLS hops, no space after the comma",
},
{
name: "trailing space",
header: "https ",
want: true,
why: "whitespace is not part of the token",
},
{
name: "proxy reports http",
header: "http",
want: false,
why: "the negative control: Secure over plaintext is " +
"dropped by the browser without a word",
},
{
name: "plaintext, no proxy",
want: false,
why: "a plain local run must stay loggable-in",
},
}
}
// transportRequest builds a request carrying the case's transport.
func (tc transportCase) request(t *testing.T) *http.Request {
t.Helper()
r := httptest.NewRequestWithContext(
context.Background(), http.MethodGet,
"http://example.com/", nil,
)
if tc.tls {
r.TLS = &tls.ConnectionState{}
}
if tc.header != "" {
r.Header.Set("X-Forwarded-Proto", tc.header)
}
return r
}
// sessionCookieFrom returns the session cookie from a response, or
// fails the test if there is none.
func sessionCookieFrom(
t *testing.T,
w *httptest.ResponseRecorder,
) *http.Cookie {
t.Helper()
for _, c := range w.Result().Cookies() {
if c.Name == session.SessionName {
return c
}
}
require.FailNow(t, "no session cookie in response")
return nil
}
// TestSave_SecureFollowsRequestTransport is the regression test for
// the defect this replaces: Secure was fixed at startup from the
// configured environment, and "dev" is the environment when
// WEBHOOKER_ENVIRONMENT is unset. A deployment behind a TLS proxy in
// that DEFAULT posture shipped the authentication cookie with no
// Secure attribute and said nothing about it.
//
// testSession builds its config with EnvironmentDev precisely so that
// the https cases below fail against the old startup-fixed behaviour.
func TestSave_SecureFollowsRequestTransport(t *testing.T) {
t.Parallel()
for _, tc := range transportCases() {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
s := testSession(t)
r := tc.request(t)
w := httptest.NewRecorder()
sess, err := s.Get(r)
require.NoError(t, err)
s.SetUser(sess, "user-1", "alice")
require.NoError(t, s.Save(r, w, sess))
assert.Equal(
t, tc.want, sessionCookieFrom(t, w).Secure,
"session cookie Secure for %q: %s",
tc.name, tc.why,
)
})
}
}
// TestSave_SecureTracksTransportBothWays pins that the flag is not
// latched. One store serves every request, so a Secure cookie set for
// a proxied request must not leak into a later plaintext response --
// the browser would silently discard that one, and a local run would
// become impossible to log into.
func TestSave_SecureTracksTransportBothWays(t *testing.T) {
t.Parallel()
s := testSession(t)
secureReq := httptest.NewRequestWithContext(
context.Background(), http.MethodGet,
"http://example.com/", nil,
)
secureReq.Header.Set("X-Forwarded-Proto", "https")
secureW := httptest.NewRecorder()
secureSess, err := s.Get(secureReq)
require.NoError(t, err)
require.NoError(t, s.Save(secureReq, secureW, secureSess))
require.True(
t, sessionCookieFrom(t, secureW).Secure,
"proxied request should produce a Secure cookie",
)
plainReq := httptest.NewRequestWithContext(
context.Background(), http.MethodGet,
"http://example.com/", nil,
)
plainW := httptest.NewRecorder()
plainSess, err := s.Get(plainReq)
require.NoError(t, err)
require.NoError(t, s.Save(plainReq, plainW, plainSess))
assert.False(
t, sessionCookieFrom(t, plainW).Secure,
"a later plaintext request must not inherit Secure from "+
"the earlier proxied one",
)
}
// TestDestroy_DeletionCookieFollowsTransport covers the trap in the
// deletion path. The store's template Secure is true, so a logout over
// plaintext that failed to track the transport would emit a Secure
// deletion cookie -- which the browser drops, leaving the session the
// user just tried to end still sitting in the jar.
func TestDestroy_DeletionCookieFollowsTransport(t *testing.T) {
t.Parallel()
s := testSession(t)
r := httptest.NewRequestWithContext(
context.Background(), http.MethodGet,
"http://example.com/", nil,
)
w := httptest.NewRecorder()
sess, err := s.Get(r)
require.NoError(t, err)
s.Destroy(sess)
require.NoError(t, s.Save(r, w, sess))
cookie := sessionCookieFrom(t, w)
require.Negative(
t, cookie.MaxAge,
"Destroy then Save should emit a deletion cookie",
)
assert.False(
t, cookie.Secure,
"a deletion cookie sent over plaintext must not be Secure, "+
"or the browser discards it and the session survives",
)
}
// TestRegenerate_BothCookiesFollowTransport covers the login path.
// Regenerate writes two cookies -- a deletion for the pre-login
// session and the new authenticated one -- and both have to match the
// transport or one of them is silently dropped.
func TestRegenerate_BothCookiesFollowTransport(t *testing.T) {
t.Parallel()
for _, tc := range transportCases() {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
s := testSession(t)
r := tc.request(t)
w := httptest.NewRecorder()
oldSess, err := s.Get(r)
require.NoError(t, err)
newSess, err := s.Regenerate(r, w, oldSess)
require.NoError(t, err)
s.SetUser(newSess, "user-1", "alice")
require.NoError(t, s.Save(r, w, newSess))
cookies := w.Result().Cookies()
require.Len(
t, cookies, 2,
"Regenerate then Save writes a deletion cookie "+
"and a replacement",
)
for _, c := range cookies {
assert.Equal(
t, tc.want, c.Secure,
"cookie %d Secure for %q: %s",
c.MaxAge, tc.name, tc.why,
)
}
})
}
}

1
internal/session/testing.go
View File

@@ -32,6 +32,7 @@ func NewForTest(
return &Session{ return &Session{
store: store, store: store,
key: key, key: key,
config: cfg,
log: log, log: log,
idleTimeout: cfg.SessionIdleTimeout, idleTimeout: cfg.SessionIdleTimeout,
now: now, now: now,

283
internal/signature/signature.go Normal file
View File

@@ -0,0 +1,283 @@
// Package signature verifies that an inbound webhook request really
// came from the sender an entrypoint was configured for.
//
// Verification is optional and per entrypoint. An entrypoint with no
// scheme configured is not verified at all, which is what every
// entrypoint was before this package existed. An entrypoint whose
// configuration is present but incoherent is failed closed, never
// treated as unverified: the whole point of the feature is that
// turning it on cannot silently turn itself back off.
package signature
import (
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"net/http"
"strings"
"sneak.berlin/go/webhooker/internal/database"
)
// Header names each supported scheme reads its signature from.
const (
// HeaderGitHub is GitHub's HMAC-SHA256 signature header. GitHub
// also sends the older SHA-1 X-Hub-Signature; it is not accepted.
HeaderGitHub = "X-Hub-Signature-256"
// HeaderGitLab is GitLab's plain shared-token header.
HeaderGitLab = "X-Gitlab-Token"
)
// githubPrefix is the algorithm label GitHub puts in front of the hex
// digest. It is required, not optional: accepting a bare digest too
// would mean accepting a spelling no supported sender produces.
const githubPrefix = "sha256="
// ErrConfig marks a failure caused by the entrypoint's stored
// configuration rather than by the request. A caller must fail these
// closed — refuse the request — because the alternative is an
// entrypoint the operator believes is verified silently accepting
// anything.
var ErrConfig = errors.New("entrypoint signature configuration invalid")
// ErrUnauthorized marks a request that failed verification. A caller
// answers these 401.
var ErrUnauthorized = errors.New("inbound signature verification failed")
// Configuration failures. None of these carry any part of the secret.
var (
errSchemeUnknown = fmt.Errorf(
"%w: unsupported scheme", ErrConfig,
)
errSecretMissing = fmt.Errorf(
"%w: scheme set with no secret", ErrConfig,
)
errSchemeMissing = fmt.Errorf(
"%w: secret set with no scheme", ErrConfig,
)
)
// Request failures. These are logged, so none of them carries the
// value the client sent: under the GitLab scheme that value is a
// guess at the token, and under either scheme a misconfigured sender
// could be presenting the real one.
var (
errHeaderMissing = fmt.Errorf(
"%w: signature header absent", ErrUnauthorized,
)
errHeaderMalformed = fmt.Errorf(
"%w: signature header malformed", ErrUnauthorized,
)
errSignatureMismatch = fmt.Errorf(
"%w: signature does not match", ErrUnauthorized,
)
)
// SchemeInfo describes one supported scheme for the UI.
type SchemeInfo struct {
Scheme database.SignatureScheme
Label string
Header string
// HeaderIsDigest reports that Header carries a value derived from
// the request rather than the shared secret itself, and so may be
// kept when the request is stored and forwarded.
//
// The polarity is deliberate: false — the zero value — means the
// header is the credential and must be stripped. A scheme added
// later is therefore stripped unless whoever adds it positively
// declares the header safe to keep.
HeaderIsDigest bool
}
// Schemes returns the supported schemes in the order the UI offers
// them. It returns a fresh slice per call so no caller can edit the
// set out from under another.
func Schemes() []SchemeInfo {
return []SchemeInfo{
{
Scheme: database.SignatureSchemeGitHub,
Label: "GitHub",
Header: HeaderGitHub,
// An HMAC over the body, not the key. Keeping it lets an
// operator see what the sender sent.
HeaderIsDigest: true,
},
{
Scheme: database.SignatureSchemeGitLab,
Label: "GitLab",
Header: HeaderGitLab,
// X-Gitlab-Token is the shared secret in plaintext.
HeaderIsDigest: false,
},
}
}
// Info returns the description of a supported scheme. It reports
// false for the empty scheme and for anything unrecognised, which is
// what a row hand-edited in the database could hold.
func Info(scheme database.SignatureScheme) (SchemeInfo, bool) {
for _, s := range Schemes() {
if s.Scheme == scheme {
return s, true
}
}
return SchemeInfo{}, false
}
// Supported reports whether a scheme may be stored on an entrypoint.
// The empty scheme is supported: it means no verification.
func Supported(scheme database.SignatureScheme) bool {
if scheme == database.SignatureSchemeNone {
return true
}
_, ok := Info(scheme)
return ok
}
// SanitizeHeaders returns a copy of an accepted request's headers
// with the entrypoint's credential removed.
//
// Under a scheme whose header is the shared secret itself — GitLab's
// X-Gitlab-Token — every downstream use of the inbound headers is a
// disclosure of the credential: they are persisted verbatim in the
// per-webhook event store and forwarded to every delivery target, so
// a target operator or anyone who reads the event database could
// forge signed requests to the very entrypoint the secret protects.
// Stripping happens here, once, above the first write, rather than
// at each egress, so a new consumer of Event.Headers cannot reopen
// the leak by forgetting to filter.
//
// header is never modified; the caller's request keeps its headers
// intact for anything that still needs the original.
//
// An entrypoint with no scheme, or one whose stored scheme this
// build does not know, is returned unchanged: there is no configured
// credential to remove, and the unknown case is refused by Verify
// before a request reaches storage.
func SanitizeHeaders(
entrypoint *database.Entrypoint,
header http.Header,
) http.Header {
clone := header.Clone()
if clone == nil {
return header
}
info, ok := Info(entrypoint.SignatureScheme)
if !ok || info.HeaderIsDigest {
return clone
}
clone.Del(info.Header)
return clone
}
// Verify checks an inbound request against an entrypoint's
// configuration and returns nil when the request may be accepted.
//
// body must be the raw bytes exactly as received, before any parsing
// or normalisation: the sender computed its digest over those bytes,
// so anything that re-encodes them produces a different digest and a
// spurious rejection. The caller is also responsible for bounding
// that read; this package hashes what it is handed.
//
// Every non-nil error is either ErrConfig or ErrUnauthorized, so a
// caller can tell "the server is misconfigured" from "the client did
// not authenticate" with errors.Is.
func Verify(
entrypoint *database.Entrypoint,
header http.Header,
body []byte,
) error {
scheme := entrypoint.SignatureScheme
secret := entrypoint.SignatureSecret
if scheme == database.SignatureSchemeNone {
// A secret with no scheme names no header and no algorithm,
// so there is nothing to check it with. Accepting the request
// would make a half-applied configuration indistinguishable
// from no configuration at all.
if secret != "" {
return errSchemeMissing
}
return nil
}
if secret == "" {
return errSecretMissing
}
switch scheme {
case database.SignatureSchemeGitHub:
return verifyGitHub(secret, header.Get(HeaderGitHub), body)
case database.SignatureSchemeGitLab:
return verifyGitLab(secret, header.Get(HeaderGitLab))
case database.SignatureSchemeNone:
// Handled above; restated so the switch stays exhaustive and
// adding a scheme has to be decided here.
return nil
default:
return errSchemeUnknown
}
}
// verifyGitHub checks a GitHub-style X-Hub-Signature-256: the string
// "sha256=" followed by the hex HMAC-SHA256 of the raw body under the
// shared secret.
func verifyGitHub(secret, provided string, body []byte) error {
if provided == "" {
return errHeaderMissing
}
encoded, ok := strings.CutPrefix(provided, githubPrefix)
if !ok {
return errHeaderMalformed
}
got, err := hex.DecodeString(encoded)
if err != nil {
return errHeaderMalformed
}
mac := hmac.New(sha256.New, []byte(secret))
// hash.Hash.Write is documented never to return an error.
_, _ = mac.Write(body)
// hmac.Equal, never ==: string comparison stops at the first
// differing byte, which tells a client how much of a forged
// digest it got right and turns forgery into a per-byte search.
if !hmac.Equal(mac.Sum(nil), got) {
return errSignatureMismatch
}
return nil
}
// verifyGitLab checks a GitLab-style X-Gitlab-Token, which is the
// shared secret itself rather than a digest over the body.
//
// The comparison is constant time in the same way as the HMAC one.
// hmac.Equal returns early for unequal lengths, so the length of the
// token is not hidden; its contents are, and length alone does not
// let a client search for the value.
func verifyGitLab(secret, provided string) error {
if provided == "" {
return errHeaderMissing
}
if !hmac.Equal([]byte(provided), []byte(secret)) {
return errSignatureMismatch
}
return nil
}

340
internal/signature/signature_test.go Normal file
View File

@@ -0,0 +1,340 @@
package signature_test
import (
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"net/http"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/signature"
)
const (
// testSharedKey is the shared secret under test. It is not named
// "secret": gosec reads a credential-shaped name bound to a
// high-entropy literal as a leaked credential, which is the right
// rule and the wrong finding here.
testSharedKey = "s3kr1t-shared-value"
testBody = `{"action":"opened","number":1}`
)
// githubSignature returns the X-Hub-Signature-256 value GitHub would
// send for testBody signed with secret.
func githubSignature(secret string) string {
mac := hmac.New(sha256.New, []byte(secret))
_, _ = mac.Write([]byte(testBody))
return "sha256=" + hex.EncodeToString(mac.Sum(nil))
}
// headerWith builds a request header carrying one value.
func headerWith(name, value string) http.Header {
h := http.Header{}
if name != "" {
h.Set(name, value)
}
return h
}
// entrypoint builds an entrypoint with a signature configuration.
func entrypoint(
scheme database.SignatureScheme, secret string,
) *database.Entrypoint {
return &database.Entrypoint{
SignatureScheme: scheme,
SignatureSecret: secret,
}
}
// TestVerifyUnconfiguredAcceptsAnything pins the pass-through case:
// an entrypoint with no scheme is the entrypoint every deployment
// already has, and it must keep accepting requests that carry no
// signature at all.
func TestVerifyUnconfiguredAcceptsAnything(t *testing.T) {
t.Parallel()
ep := entrypoint(database.SignatureSchemeNone, "")
require.NoError(
t, signature.Verify(ep, http.Header{}, []byte(testBody)),
)
require.NoError(
t,
signature.Verify(
ep,
headerWith(signature.HeaderGitHub, "sha256=deadbeef"),
[]byte(testBody),
),
)
}
// githubCase is one inbound request against a GitHub-scheme
// entrypoint.
type githubCase struct {
name string
header string
value string
body string
want error
}
// githubCases enumerates the shapes a GitHub signature can arrive in.
func githubCases() []githubCase {
valid := githubSignature(testSharedKey)
return []githubCase{
{
name: "valid",
header: signature.HeaderGitHub,
value: valid,
body: testBody,
want: nil,
},
{
name: "absent header",
header: "",
body: testBody,
want: signature.ErrUnauthorized,
},
{
name: "wrong secret",
header: signature.HeaderGitHub,
value: githubSignature("not-the-shared-value"),
body: testBody,
want: signature.ErrUnauthorized,
},
{
// The digest is valid for a different body: the check
// has to be over the bytes actually received.
name: "body altered in flight",
header: signature.HeaderGitHub,
value: valid,
body: testBody + " ",
want: signature.ErrUnauthorized,
},
{
name: "missing algorithm prefix",
header: signature.HeaderGitHub,
value: valid[len("sha256="):],
body: testBody,
want: signature.ErrUnauthorized,
},
{
name: "not hex",
header: signature.HeaderGitHub,
value: "sha256=zzzz",
body: testBody,
want: signature.ErrUnauthorized,
},
{
name: "empty digest",
header: signature.HeaderGitHub,
value: "sha256=",
body: testBody,
want: signature.ErrUnauthorized,
},
{
// GitLab's header does not authenticate a GitHub
// entrypoint, even holding the right secret.
name: "wrong header for the scheme",
header: signature.HeaderGitLab,
value: testSharedKey,
body: testBody,
want: signature.ErrUnauthorized,
},
}
}
func TestVerifyGitHub(t *testing.T) {
t.Parallel()
for _, tc := range githubCases() {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
err := signature.Verify(
entrypoint(
database.SignatureSchemeGitHub, testSharedKey,
),
headerWith(tc.header, tc.value),
[]byte(tc.body),
)
if tc.want == nil {
require.NoError(t, err)
return
}
require.ErrorIs(t, err, tc.want)
})
}
}
func TestVerifyGitLab(t *testing.T) {
t.Parallel()
cases := []struct {
name string
header string
value string
want error
}{
{
name: "valid",
header: signature.HeaderGitLab,
value: testSharedKey,
want: nil,
},
{
name: "absent header",
header: "",
want: signature.ErrUnauthorized,
},
{
name: "wrong token",
header: signature.HeaderGitLab,
value: "not-the-shared-value",
want: signature.ErrUnauthorized,
},
{
name: "token prefix only",
header: signature.HeaderGitLab,
value: testSharedKey[:5],
want: signature.ErrUnauthorized,
},
{
name: "wrong header for the scheme",
header: signature.HeaderGitHub,
value: githubSignature(testSharedKey),
want: signature.ErrUnauthorized,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
err := signature.Verify(
entrypoint(
database.SignatureSchemeGitLab, testSharedKey,
),
headerWith(tc.header, tc.value),
[]byte(testBody),
)
if tc.want == nil {
require.NoError(t, err)
return
}
require.ErrorIs(t, err, tc.want)
})
}
}
// TestVerifyBrokenConfigurationFailsClosed covers the rows a caller
// must refuse rather than wave through. Each is a state an operator
// could only reach outside the UI, and each one would otherwise be
// indistinguishable from "verification is off".
func TestVerifyBrokenConfigurationFailsClosed(t *testing.T) {
t.Parallel()
cases := []struct {
name string
scheme database.SignatureScheme
secret string
}{
{
name: "unknown scheme",
scheme: database.SignatureScheme("stripe"),
secret: testSharedKey,
},
{
name: "scheme without secret",
scheme: database.SignatureSchemeGitHub,
secret: "",
},
{
name: "secret without scheme",
scheme: database.SignatureSchemeNone,
secret: testSharedKey,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
err := signature.Verify(
entrypoint(tc.scheme, tc.secret),
headerWith(
signature.HeaderGitHub,
githubSignature(testSharedKey),
),
[]byte(testBody),
)
require.ErrorIs(t, err, signature.ErrConfig)
assert.NotErrorIs(t, err, signature.ErrUnauthorized)
})
}
}
// TestErrorsCarryNoSecret proves the strings that reach the log hold
// no part of the shared secret or of what the client presented.
func TestErrorsCarryNoSecret(t *testing.T) {
t.Parallel()
const presented = "QQPRESENTEDTOKENQQ"
for _, scheme := range []database.SignatureScheme{
database.SignatureSchemeGitHub,
database.SignatureSchemeGitLab,
} {
for _, header := range []string{
signature.HeaderGitHub, signature.HeaderGitLab,
} {
err := signature.Verify(
entrypoint(scheme, testSharedKey),
headerWith(header, presented),
[]byte(testBody),
)
require.Error(t, err)
assert.NotContains(t, err.Error(), testSharedKey)
assert.NotContains(t, err.Error(), presented)
}
}
}
func TestSchemeMetadata(t *testing.T) {
t.Parallel()
assert.True(t, signature.Supported(database.SignatureSchemeNone))
assert.True(t, signature.Supported(database.SignatureSchemeGitHub))
assert.True(t, signature.Supported(database.SignatureSchemeGitLab))
assert.False(
t, signature.Supported(database.SignatureScheme("stripe")),
)
// The empty scheme describes no sender, so it has no info even
// though it is a storable value.
_, ok := signature.Info(database.SignatureSchemeNone)
assert.False(t, ok)
info, ok := signature.Info(database.SignatureSchemeGitHub)
require.True(t, ok)
assert.Equal(t, "GitHub", info.Label)
assert.Equal(t, signature.HeaderGitHub, info.Header)
info, ok = signature.Info(database.SignatureSchemeGitLab)
require.True(t, ok)
assert.Equal(t, "GitLab", info.Label)
assert.Equal(t, signature.HeaderGitLab, info.Header)
}

12
internal/versionscript/doc.go
View File

@@ -1,12 +0,0 @@
// Package versionscript holds the tests for script/version and for the
// build files that consume it. It carries no runtime code: the version
// string is produced by a shell script at build time and reaches the
// binary through a linker flag, so nothing in the Go build graph can
// assert it, but the behaviour still has to be verified by the test
// suite.
//
// The files under test are outside the Go build graph, so `go test`'s
// result cache serves a stale PASS when only script/version, the
// Makefile or the Dockerfile changed: run the container build, or
// GOFLAGS=-count=1, to trust a result here after editing them.
package versionscript

345
internal/versionscript/version_script_test.go
View File

@@ -1,345 +0,0 @@
package versionscript_test
import (
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/require"
)
const (
repoRoot = "../.."
scriptPath = "../../script/version"
makefilePath = "../../Makefile"
dockerfilePath = "../../Dockerfile"
// unknown is what a tree with no git metadata and no $VERSION must
// report: a source tarball has no way to know its version, and the
// one thing it must not do is name a tag it may not be at.
unknown = "unknown"
// scriptMode keeps the copied script runnable; dirMode and fileMode
// are the ordinary permissions for the throwaway tree around it.
scriptMode = 0o755
dirMode = 0o750
fileMode = 0o600
)
// checkout is a throwaway working tree carrying a copy of the script
// under test at the same path it lives at in this repository, since the
// script resolves the checkout root from its own location.
type checkout struct {
dir string
head string
}
func TestVersion_CleanTagReportsExactlyTheTag(t *testing.T) {
t.Parallel()
c := newCheckout(t)
c.git(t, "tag", "v1.2.3")
require.Equal(t, "v1.2.3", c.version(t))
}
func TestVersion_CommitAfterTagCarriesDistanceAndSHA(t *testing.T) {
t.Parallel()
c := newCheckout(t)
c.git(t, "tag", "v1.2.3")
head := c.commit(t, "after the tag")
got := c.version(t)
require.NotEqual(t, "v1.2.3", got,
"a commit past the tag must not claim to be the tag")
require.True(t, strings.HasPrefix(got, "v1.2.3-1-g"),
"want describe form v1.2.3-1-g<sha>, got %q", got)
require.True(t, strings.HasPrefix(head, strings.TrimPrefix(
got, "v1.2.3-1-g")),
"%q must carry the abbreviated head SHA of %q", got, head)
}
func TestVersion_UntaggedHistoryReportsShortSHA(t *testing.T) {
t.Parallel()
c := newCheckout(t)
got := c.version(t)
require.NotEmpty(t, got)
require.NotEqual(t, unknown, got)
require.True(t, strings.HasPrefix(c.head, got),
"%q must be an abbreviation of head %q", got, c.head)
}
func TestVersion_UncommittedChangesAreMarkedDirty(t *testing.T) {
t.Parallel()
c := newCheckout(t)
c.git(t, "tag", "v1.2.3")
require.NoError(t, os.WriteFile(
filepath.Join(c.dir, "tracked.txt"), []byte("edited\n"), fileMode,
))
require.Equal(t, "v1.2.3-dirty", c.version(t))
}
// A source tarball, or any build context without .git, still has to
// build. It reports "unknown" rather than failing or naming a tag.
func TestVersion_NoGitMetadataReportsUnknown(t *testing.T) {
t.Parallel()
c := newCheckout(t)
require.NoError(t, os.RemoveAll(filepath.Join(c.dir, ".git")))
require.Equal(t, unknown, c.version(t))
}
// An unpacked tarball can sit inside an unrelated working copy. The
// enclosing repository's version is not this tree's version.
func TestVersion_EnclosingRepositoryIsNotUsed(t *testing.T) {
t.Parallel()
outer := newCheckout(t)
outer.git(t, "tag", "v9.9.9")
inner := filepath.Join(outer.dir, "unpacked")
require.NoError(t, os.MkdirAll(filepath.Join(inner, "script"), dirMode))
copyScript(t, inner)
require.Equal(t, unknown, runScript(t, inner, nil))
}
// The Docker build has no git metadata, so the version arrives as an
// environment override. It wins over anything derivable.
func TestVersion_EnvironmentOverrideWins(t *testing.T) {
t.Parallel()
c := newCheckout(t)
c.git(t, "tag", "v1.2.3")
require.Equal(t, "v4.5.6",
runScript(t, c.dir, []string{"VERSION=v4.5.6"}))
}
// An empty VERSION is treated as unset rather than stamping an empty
// string: the Dockerfile's build arg has a non-empty default, but a
// caller exporting VERSION= must not produce a binary reporting "".
func TestVersion_EmptyOverrideFallsBackToGit(t *testing.T) {
t.Parallel()
c := newCheckout(t)
c.git(t, "tag", "v1.2.3")
require.Equal(t, "v1.2.3", runScript(t, c.dir, []string{"VERSION="}))
}
// Two builds of the same commit must produce a byte-identical binary,
// which they cannot if the stamped value moves between invocations.
func TestVersion_IsStableAcrossInvocations(t *testing.T) {
t.Parallel()
c := newCheckout(t)
c.git(t, "tag", "v1.2.3")
first := c.version(t)
second := c.version(t)
require.Equal(t, first, second)
}
// The build target is the only place composing linker flags. If a
// future edit drops either half, the binary silently reports "dev"
// again (the defect this package exists for) or fails to link on
// Alpine.
func TestMakefile_BuildComposesVersionAndExtraFlags(t *testing.T) {
t.Parallel()
makefile := read(t, makefilePath)
require.Contains(t, makefile, "-X main.version=$(VERSION)")
require.Contains(t, makefile, "$(GO_LDFLAGS)")
require.Contains(t, makefile, "VERSION ?= $(shell script/version)")
}
// A caller can define VERSION as the empty string -- `make build
// VERSION=`, or a `--build-arg VERSION=` reaching the Dockerfile's `make
// build VERSION="$VERSION"`. script/version's own guard does not cover
// that: the value never passes through the script. Stamping "" would
// leave the binary reporting no version and the footer on "dev", which
// is the defect this package exists for.
func TestMakefile_EmptyOverrideResolvesLikeAnUnsetOne(t *testing.T) {
t.Parallel()
// A plain assignment would be ignored here: a command-line
// definition outranks it, and that is the case being corrected.
require.Contains(t, read(t, makefilePath), "override VERSION :=")
requireMake(t)
derived := makeVersion(t)
require.NotEmpty(t, derived)
require.Equal(t, derived, makeVersion(t, "VERSION="),
"an empty VERSION must resolve the way an unset one does")
require.Equal(t, "v9.9.9", makeVersion(t, "VERSION=v9.9.9"),
"the empty guard must not clobber a real override")
}
// makeVersion runs this repository's `version` target, which prints the
// value `make build` would stamp, with the given command-line
// definitions.
func makeVersion(t *testing.T, defs ...string) string {
t.Helper()
//nolint:gosec // fixed argv, arguments are test constants
cmd := exec.CommandContext(t.Context(), "make",
append([]string{"--no-print-directory", "version"}, defs...)...)
cmd.Dir = repoRoot
// Only the command-line definitions may decide the outcome: an
// inherited VERSION would change what an unset one resolves to, and
// an inherited MAKEFLAGS carries the parent's jobserver.
cmd.Env = append(os.Environ(), "VERSION=", "MAKEFLAGS=", "MAKELEVEL=")
out, err := cmd.CombinedOutput()
require.NoError(t, err, string(out))
return strings.TrimSpace(string(out))
}
func requireMake(t *testing.T) {
t.Helper()
_, err := exec.LookPath("make")
if err != nil {
t.Skipf("make is not installed: %v", err)
}
}
// Every compile in the image goes through the build target, so the
// static relink cannot replace the flags that carry the stamp.
func TestDockerfile_BuildsThroughTheMakeTarget(t *testing.T) {
t.Parallel()
dockerfile := read(t, dockerfilePath)
require.NotContains(t, dockerfile, "go build",
"a raw go build bypasses the Makefile's -X flag")
require.Contains(t, dockerfile, "ARG VERSION=")
require.Contains(t, dockerfile,
`make build VERSION="$VERSION" GO_LDFLAGS='-extldflags "-static"'`)
}
func read(t *testing.T, path string) string {
t.Helper()
//nolint:gosec // repo-local build file under test, fixed path
b, err := os.ReadFile(path)
require.NoError(t, err)
return string(b)
}
// version runs the script in this checkout with no overrides.
func (c checkout) version(t *testing.T) string {
t.Helper()
return runScript(t, c.dir, nil)
}
func runScript(t *testing.T, dir string, env []string) string {
t.Helper()
//nolint:gosec // fixed argv, repo-local script under test
cmd := exec.CommandContext(t.Context(), "sh",
filepath.Join(dir, "script", "version"))
cmd.Dir = dir
cmd.Env = append(os.Environ(), env...)
out, err := cmd.CombinedOutput()
require.NoError(t, err, string(out))
return strings.TrimSpace(string(out))
}
func (c checkout) git(t *testing.T, args ...string) string {
t.Helper()
//nolint:gosec // fixed argv, arguments are test constants
cmd := exec.CommandContext(t.Context(), "git", args...)
cmd.Dir = c.dir
out, err := cmd.CombinedOutput()
require.NoError(t, err, string(out))
return strings.TrimSpace(string(out))
}
func (c checkout) commit(t *testing.T, message string) string {
t.Helper()
c.git(t,
"-c", "user.email=ci@example.invalid",
"-c", "user.name=ci",
"-c", "commit.gpgsign=false",
"commit", "-q", "--allow-empty", "-m", message,
)
return c.git(t, "rev-parse", "HEAD")
}
// newCheckout builds a one-commit repository with a tracked file, so a
// later edit to that file makes the tree dirty, and with a copy of the
// script at the path it occupies in this repository.
func newCheckout(t *testing.T) checkout {
t.Helper()
requireGit(t)
dir := t.TempDir()
c := checkout{dir: dir}
require.NoError(t, os.MkdirAll(filepath.Join(dir, "script"), dirMode))
copyScript(t, dir)
require.NoError(t, os.WriteFile(
filepath.Join(dir, "tracked.txt"), []byte("original\n"), fileMode,
))
c.git(t, "init", "-q", "-b", "main")
c.git(t, "add", "tracked.txt")
c.head = c.commit(t, "initial")
return c
}
func copyScript(t *testing.T, dir string) {
t.Helper()
body, err := os.ReadFile(scriptPath)
require.NoError(t, err)
//nolint:gosec // the copy has to stay executable to be run
err = os.WriteFile(
filepath.Join(dir, "script", "version"), body, scriptMode,
)
require.NoError(t, err)
}
func requireGit(t *testing.T) {
t.Helper()
for _, tool := range []string{"sh", "git"} {
_, err := exec.LookPath(tool)
if err != nil {
t.Skipf("%s is not installed: %v", tool, err)
}
}
}

11
script/docker
View File

@@ -1,10 +1,7 @@
#!/bin/sh #!/bin/sh
# script/docker: build the Docker image tagged with the project name. # script/docker: build the Docker image tagged with the project name.
# The tag comes from script/projectname. # Identical in all repos; the tag comes from script/projectname.
# # Generic: needs no adaptation.
# .dockerignore excludes .git/, so the builder stage cannot derive the
# version itself. It is resolved here, where the checkout is, and passed
# in as a build arg; without it the image would stamp itself "unknown".
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -12,9 +9,7 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
docker build \ docker build -t "$("$SCRIPT_DIR/projectname")" .
--build-arg VERSION="$("$SCRIPT_DIR/version")" \
-t "$("$SCRIPT_DIR/projectname")" .
} }
main "$@" main "$@"

65
script/version
View File

@@ -1,65 +0,0 @@
#!/bin/sh
# script/version: output the version string the binary is stamped with.
# Our own extension to scripts-to-rule-them-all. The Makefile's build
# target and script/docker both take the value from here, so a `make
# build` binary and a `make docker` image built from the same checkout
# report the same thing.
#
# Order of precedence:
#
# 1. $VERSION, if set and non-empty. This is how the value reaches a
# build that cannot derive it: .dockerignore excludes .git/, so the
# builder stage has no git metadata and the Dockerfile takes the
# value as a build arg instead.
# 2. `git describe --tags --always --dirty` against this checkout. At
# a clean tagged commit that is exactly the tag; otherwise it
# carries the short SHA, the commit distance when a tag is
# reachable, and a -dirty suffix for uncommitted changes.
# 3. "unknown", for a tree with no git metadata and no $VERSION -- a
# source tarball, or `docker build .` with no --build-arg. That
# case must not fail the build and must not name a tag the tree may
# not be at, so it names nothing.
#
# The git step insists the enclosing repository is this checkout, not
# merely some repository above it: an unpacked tarball sitting inside an
# unrelated working copy would otherwise be stamped with that copy's
# version.
#
# Nothing here may vary between two builds of the same commit: the
# release gate asserts the binary is byte-identical across builds. That
# rules out a build timestamp, a hostname, and a builder identity.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# in_this_checkout succeeds when git can read metadata for a repository
# whose work tree root is $ROOT.
in_this_checkout() {
command -v git >/dev/null 2>&1 || return 1
top="$(git rev-parse --show-toplevel 2>/dev/null)" || return 1
[ -n "$top" ] || return 1
top="$(cd "$top" 2>/dev/null && pwd -P)" || return 1
[ "$top" = "$ROOT" ]
}
main() {
if [ -n "${VERSION:-}" ]; then
echo "$VERSION"
return 0
fi
cd "$ROOT"
if in_this_checkout; then
# --always keeps an untagged history from failing the build: it
# falls back to the bare short SHA.
git describe --tags --always --dirty 2>/dev/null && return 0
fi
echo "unknown"
}
main "$@"

36
templates/source_detail.html
View File

@@ -48,7 +48,7 @@
<div class="divide-y divide-gray-100"> <div class="divide-y divide-gray-100">
{{range .Entrypoints}} {{range .Entrypoints}}
<div class="p-4"> <div class="p-4" x-data="{ showSecret: false }">
<div class="flex items-center justify-between mb-1"> <div class="flex items-center justify-between mb-1">
<span class="text-sm font-medium text-gray-900">{{if .Description}}{{.Description}}{{else}}Entrypoint{{end}}</span> <span class="text-sm font-medium text-gray-900">{{if .Description}}{{.Description}}{{else}}Entrypoint{{end}}</span>
<div class="flex items-center gap-2"> <div class="flex items-center gap-2">
@@ -75,8 +75,38 @@
script the URL above stays selectable. --> script the URL above stays selectable. -->
<button type="button" hidden data-copy-target="entrypoint-url-{{.ID}}" class="text-xs text-gray-500 hover:text-primary-600">Copy</button> <button type="button" hidden data-copy-target="entrypoint-url-{{.ID}}" class="text-xs text-gray-500 hover:text-primary-600">Copy</button>
</div> </div>
<!-- The URL above is the entrypoint's credential: <div class="flex items-center gap-2 mt-2">
anyone holding it can submit events. --> <span class="text-xs text-gray-500">
Signature: {{.SchemeLabel}}{{if .SchemeHeader}} ({{.SchemeHeader}}){{end}}
</span>
<button type="button" @click="showSecret = !showSecret" class="text-xs text-gray-500 hover:text-primary-600">
{{if .Configured}}Rotate{{else}}Configure{{end}}
</button>
</div>
<!-- The stored secret is never sent to the browser: the
form takes a new one every time, so setting and
rotating are the same submission. -->
<div x-show="showSecret" x-cloak class="mt-2">
<form method="POST" action="/source/{{$.Webhook.ID}}/entrypoints/{{.ID}}/secret" class="flex gap-2">
<input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">
<select name="signature_scheme" class="input text-sm w-28">
<!-- Selection follows the stored scheme, not
whether the pair is complete: a row with a
scheme and no secret would otherwise mark
both this option and its own selected. -->
<option value="" {{if not .Scheme}}selected{{end}}>None</option>
{{$current := .Scheme}}
{{range $.SignatureSchemes}}
<option value="{{.Scheme}}" {{if eq .Scheme $current}}selected{{end}}>{{.Label}}</option>
{{end}}
</select>
<input type="password" name="secret" autocomplete="new-password" placeholder="Shared secret" class="input text-sm flex-1">
<button type="submit" class="btn-primary text-sm">Save</button>
</form>
<p class="text-xs text-gray-500 mt-1">
Enter the same secret you configured at the sender. Selecting None removes verification.
</p>
</div>
</div> </div>
{{else}} {{else}}
<div class="p-4 text-sm text-gray-500">No entrypoints configured.</div> <div class="p-4 text-sm text-gray-500">No entrypoints configured.</div>

4
templates/source_logs.html
View File

@@ -39,7 +39,7 @@
<div class="flex items-center gap-4"> <div class="flex items-center gap-4">
{{range .Deliveries}} {{range .Deliveries}}
<span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}"> <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">
{{.Target.DisplayName}}: {{.Status}} {{.Target.Name}}: {{.Status}}
</span> </span>
{{end}} {{end}}
<span class="text-xs text-gray-400">{{.CreatedAt.Format "2006-01-02 15:04:05"}}</span> <span class="text-xs text-gray-400">{{.CreatedAt.Format "2006-01-02 15:04:05"}}</span>
@@ -74,7 +74,7 @@
<div class="py-2" x-data="{ attempts: false }"> <div class="py-2" x-data="{ attempts: false }">
<div class="flex items-center justify-between cursor-pointer" @click="attempts = !attempts"> <div class="flex items-center justify-between cursor-pointer" @click="attempts = !attempts">
<div class="flex items-center gap-3"> <div class="flex items-center gap-3">
<span class="text-sm text-gray-700">{{.Target.DisplayName}}</span> <span class="text-sm text-gray-700">{{.Target.Name}}</span>
<span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{.Status}}</span> <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{.Status}}</span>
</div> </div>
<div class="flex items-center gap-3"> <div class="flex items-center gap-3">