Author SHA1 Message Date
clawbot ff0018cf43 Add a statistics pane to the webhook page (closes #368)
check / check (push) Waiting to run
Each webhook's event database keeps running totals: one row for its
events, and one row per target for that target's deliveries, delivered
and failed, each with what retention removed. Every write to them
shares the transaction of the rows it counts. Deliveries get a
finished_at column; it and target_id end the status index, so each
target's deliveries finished in a window come from one index-range
query grouped by target. Retention deletes 1000 expired events per
transaction. The pane is its own template, its figures in tables.

The schema changes in place with nothing back-filled, so an existing
database must be recreated.

Model: opus-5-5
2026-10-01 21:05:05 +00:00
clawbot 9d29baaa2d Commit the Alpine.js tarball in 3p/ and extract it at build time (closes #345)
check / check (push) Waiting to run
The build no longer downloads Alpine.js. Its npm package tarball is committed as 3p/alpinejs-3.14.9.tgz, byte for byte the file script/fetch-assets downloaded, with the sha256 that script pinned. script/assets (make assets) extracts package/dist/cdn.min.js to the ignored static/js/alpine.min.js; script/test runs it, so make test, make check, the pre-commit hook and the Dockerfile get the file with no network access, and make build, run and dev run it too.

Removed: script/fetch-assets, its Dockerfile step, static/vendor.sha256 and static/vendor_test.go. The README describes the new flow.

Model: opus-5-5
2026-10-01 22:44:41 +02:00
clawbot 507980a347 Bound username length at creation (closes #184)
check / check (push) Waiting to run
Usernames are limited to 1024 bytes, so no account can exist that is unable to log in. The username rides in the session cookie, which browsers and securecookie refuse past about 4 KB, leaving room for roughly 2000 bytes of username; the limit is about half that.

User.BeforeSave returns ErrUsernameTooLong when a whole User is created or saved. A byte-counting check constraint on users.username catches every other write, including a column update. The limit appears in the constant and in the struct tag; a test fails if they disagree.

Model: opus-5-5
2026-10-01 21:38:48 +02:00
clawbot b79e4649a1 Container sets its data directory's owner and mode itself (#353)
check / check (push) Canceled after 0s
Closes #340.

The image no longer sets `USER`. Its new `ENTRYPOINT`, `deploy/docker-entrypoint.sh`, starts as root, creates `DATA_DIR` if missing, gives the directory and anything in it owned by another user to `webhooker` (UID 1000), sets the directory to `0750`, and runs the command as `webhooker` through `su-exec`. An empty root-owned bind mount, or data left by another UID, now works as mounted; the app never runs as root and is still PID 1. `CMD` is still `/app/webhooker`, so the `resetpw` commands are unchanged. Started with `--user`, the script only runs the command. It is in `/usr/local/bin`, not `/app`, which belongs to `webhooker`.

README: the UID 1000 ownership block, the upaas pre-deploy commands and the restore ownership step are gone; the upaas volume bullet names only the path.

- Judgement call: `su-exec` over `setpriv`: Alpine's small tool for this, needing only musl; busybox's `setpriv` cannot change user, and util-linux's adds `libcap-ng`.
- Deviation: `su-exec` is pinned by version (`0.2-r3`), not by hash; `ca-certificates` beside it is unpinned.
- Judgement call: each start reads every entry's owner but changes only entries owned by someone else.
- `docker exec` and the health check now run as root, since the image sets no `USER`.
- No automated test covers the script: the suite runs inside `docker build`, which cannot start a container.
- A missing host directory under upaas is sneak/upaas#235.

Model: opus-5-5
Reviewed-on: #353
Co-authored-by: clawbot <35+clawbot@noreply.example.org>
2026-09-29 13:05:16 +02:00
clawbot 1428154bbd Let a browser with cookies from an earlier database log in (closes #359)
check / check (push) Canceled after 0s
A new database brings a new session key. A browser still holding the
old session cookie got a 500 on a correct login: Session.Get returned
the cookie's decode error and the login handler answered it with a
500. Get now treats a cookie that does not decode as absent, and
logging in replaces it. gorilla/csrf already did the same for the
CSRF cookie.

A start that creates webhooker.db now logs "created a new, empty
database" at WARN with its path, shortly before the first-boot banner,
so an unexpectedly empty DATA_DIR is noticed.

The codec tests now decode through the store, since Get no longer
reports the codec's reason.

Model: opus-5-5
2026-09-29 12:58:44 +02:00
sneak 8ad2a86e4b Sneak/testdeploy (#356)
check / check (push) Successful in 11s
Reviewed-on: #356
2026-09-29 12:01:33 +02:00
sneak a891b726e5 Milestone: next into main (#342)
check / check (push) Successful in 9s
Reviewed-on: #342
2026-09-29 11:53:19 +02:00
clawbot ab63b5f777 Restrict /s/* to GET and HEAD (closes #169)
check / check (push) Successful in 3m37s
The static file server was attached with Mount, which registers every
method, so POST, PUT and DELETE on an asset were answered 200 with the
file. It is now registered for GET and HEAD only, inside a /s group
whose method-not-allowed handler answers 405 with Allow: GET, HEAD.
A method chi does not route at all, such as PROPFIND, still gets 405
from the top-level router, without Allow. The inverted test and the
README route table say the same.

Model: opus-5-5
2026-09-29 11:10:27 +02:00
46 changed files with 2136 additions and 514 deletions
+2 -4
View File
@@ -3,10 +3,8 @@
# stage of the Dockerfile. # stage of the Dockerfile.
.git/ .git/
bin/ bin/
# Third-party browser assets are fetched and hash-verified inside the build by # Extracted from 3p/ by `make assets` inside the build; a host copy is not
# script/fetch-assets. Excluding any host copy keeps a developer's working tree # needed. The tarball in 3p/ must stay in the context.
# from supplying the bytes that get shipped. The script and its
# static/vendor.sha256 manifest stay in the context.
static/js/alpine.min.js static/js/alpine.min.js
*.md *.md
LICENSE LICENSE
+3 -4
View File
@@ -46,7 +46,6 @@ temp/
# CI cache barrier, written into the build context by the check workflow # CI cache barrier, written into the build context by the check workflow
.ci-fingerprint .ci-fingerprint
# Third-party browser assets, fetched and hash-verified by # Alpine.js, extracted by `make assets` from its tarball in 3p/, which is
# script/fetch-assets against static/vendor.sha256. Not committed: # what is committed.
# REPO_POLICIES.md forbids minified bundles in version control. /static/js/alpine.min.js
/static/js/alpine.min.js
Binary file not shown.
+13 -13
View File
@@ -51,15 +51,8 @@ RUN go mod download
# the lint stage above. # the lint stage above.
COPY . . COPY . .
# Fetch the third-party browser assets the UI serves. They are not committed # Run tests and build. Both first run script/assets, which extracts Alpine.js
# (REPO_POLICIES.md forbids minified bundles in version control) and # from its tarball in 3p/.
# .dockerignore keeps any host copy out of the build context, so this step is
# the only way they enter the image. Each download is checked against a
# hardcoded sha256 and the build fails on mismatch; make test re-checks the
# hashes against the bytes go:embed actually put in the binary.
RUN script/fetch-assets
# Run tests and build
RUN make test RUN make test
# Version stamped into the binary. .dockerignore excludes .git/, so # Version stamped into the binary. .dockerignore excludes .git/, so
@@ -67,8 +60,8 @@ RUN make test
# host and passes it in. The default is what a bare `docker build .` # host and passes it in. The default is what a bare `docker build .`
# with no --build-arg gets, and it names no tag the tree may not be at. # with no --build-arg gets, and it names no tag the tree may not be at.
# #
# Declared here, below the test and asset steps, so a changed version # Declared here, below the test step, so a changed version does not
# does not invalidate their cached layers. # invalidate its cached layer.
ARG VERSION=unknown ARG VERSION=unknown
RUN make build VERSION="$VERSION" RUN make build VERSION="$VERSION"
@@ -88,7 +81,9 @@ RUN CGO_ENABLED=1 make build VERSION="$VERSION" GO_LDFLAGS='-extldflags "-static
# alpine:3.21, 2026-03-17 # alpine:3.21, 2026-03-17
FROM alpine:3.21@sha256:c3f8e73fdb79deaebaa2037150150191b9dcbfba68b4a46d70103204c53f4709 FROM alpine:3.21@sha256:c3f8e73fdb79deaebaa2037150150191b9dcbfba68b4a46d70103204c53f4709
RUN apk --no-cache add ca-certificates # su-exec 0.2-r3 (Alpine 3.21), 2026-09-29: the entrypoint runs the app
# as webhooker with it.
RUN apk --no-cache add ca-certificates su-exec=0.2-r3
# Create non-root user # Create non-root user
RUN addgroup -g 1000 -S webhooker && \ RUN addgroup -g 1000 -S webhooker && \
@@ -99,13 +94,17 @@ WORKDIR /app
# Copy binary from builder # Copy binary from builder
COPY --from=builder /build/bin/webhooker /app/webhooker COPY --from=builder /build/bin/webhooker /app/webhooker
# Not under /app, which belongs to webhooker: this script runs as root.
COPY deploy/docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
# Create data directory for all SQLite databases (main app DB + # Create data directory for all SQLite databases (main app DB +
# per-webhook event DBs). DATA_DIR defaults to /var/lib/webhooker. # per-webhook event DBs). DATA_DIR defaults to /var/lib/webhooker.
RUN mkdir -p /var/lib/webhooker RUN mkdir -p /var/lib/webhooker
RUN chown -R webhooker:webhooker /app /var/lib/webhooker RUN chown -R webhooker:webhooker /app /var/lib/webhooker
USER webhooker # No USER: the entrypoint starts as root to make the data directory
# webhooker's, then runs the app as webhooker.
EXPOSE 8080 EXPOSE 8080
@@ -124,4 +123,5 @@ ENV BIND_ADDRESS=0.0.0.0
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \ HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:8080/.well-known/healthcheck || exit 1 CMD wget --no-verbose --tries=1 --spider http://localhost:8080/.well-known/healthcheck || exit 1
ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]
CMD ["/app/webhooker"] CMD ["/app/webhooker"]
+3 -3
View File
@@ -28,7 +28,7 @@ setup:
@script/setup @script/setup
assets: assets:
@script/fetch-assets @script/assets
test: test:
@script/test @script/test
@@ -45,13 +45,13 @@ fmt-check:
check: check:
@script/check @script/check
build: build: assets
go build -ldflags '$(strip -X main.version=$(VERSION) $(GO_LDFLAGS))' -o bin/webhooker ./cmd/webhooker go build -ldflags '$(strip -X main.version=$(VERSION) $(GO_LDFLAGS))' -o bin/webhooker ./cmd/webhooker
run: build run: build
./bin/webhooker ./bin/webhooker
dev: dev: assets
go run ./cmd/webhooker go run ./cmd/webhooker
deps: deps:
+164 -147
View File
@@ -21,9 +21,6 @@ before deploying one.
- Go 1.26.1+ (the version in `go.mod`) - Go 1.26.1+ (the version in `go.mod`)
- Docker (for linting, for the test stage of the CI gate, and for - Docker (for linting, for the test stage of the CI gate, and for
containerized deployment) containerized deployment)
- `curl`, used by `script/fetch-assets` to download the third-party
browser assets, which are not committed (`make bootstrap` installs
it if missing)
golangci-lint is not a prerequisite and must not be installed on the golangci-lint is not a prerequisite and must not be installed on the
host: `script/bootstrap` does not install it, and `make lint` runs the host: `script/bootstrap` does not install it, and `make lint` runs the
@@ -36,9 +33,7 @@ digest-pinned linter image via `Dockerfile.lint`.
git clone https://git.eeqj.de/sneak/webhooker.git git clone https://git.eeqj.de/sneak/webhooker.git
cd webhooker cd webhooker
# Install Go dependencies and the third-party browser assets. # Install the Go toolchain if missing, and the Go dependencies
# `make deps` alone is not enough: it only runs go mod download/tidy,
# and the checks below need the fetched assets.
make bootstrap make bootstrap
# Run all checks (test, lint, format check) # Run all checks (test, lint, format check)
@@ -58,7 +53,7 @@ make docker
```bash ```bash
make bootstrap # Install all dependencies (idempotent) make bootstrap # Install all dependencies (idempotent)
make setup # Bootstrap + install git pre-commit hook make setup # Bootstrap + install git pre-commit hook
make assets # Fetch + verify third-party browser assets make assets # Extract Alpine.js from 3p/ (test, check, build, dev run it)
make fmt # Format code (gofmt + goimports) make fmt # Format code (gofmt + goimports)
make fmt-check # Fail if gofmt would change anything (writes nothing) make fmt-check # Fail if gofmt would change anything (writes nothing)
make lint # Run golangci-lint in Docker (Dockerfile.lint) make lint # Run golangci-lint in Docker (Dockerfile.lint)
@@ -538,6 +533,12 @@ its Argon2id hash. There is no second account and no forgot-password
flow, so the banner and the reset command below are the only two ways flow, so the banner and the reset command below are the only two ways
in. in.
A start that finds no `webhooker.db` in `DATA_DIR` also logs
`created a new, empty database` at `WARN`, with the file's path,
shortly before the banner. On a deployment that has run before, that
line means `DATA_DIR` was empty, most often because its volume is not
mounted.
#### Recovering a lost admin password #### Recovering a lost admin password
`webhooker resetpw` sets an existing account's password from the `webhooker resetpw` sets an existing account's password from the
@@ -552,8 +553,9 @@ printf '%s' "$NEW_PASSWORD" | \
DATA_DIR=/var/lib/webhooker webhooker resetpw admin DATA_DIR=/var/lib/webhooker webhooker resetpw admin
``` ```
In a container it is the same binary, which the image sets as `CMD` In a container it is the same binary. The image's `CMD` is
rather than `ENTRYPOINT`, so the whole command has to be given: `/app/webhooker`, and a command given to `docker run` replaces all of
it, so the whole command has to be given:
```bash ```bash
docker run --rm -v webhooker-data:/var/lib/webhooker \ docker run --rm -v webhooker-data:/var/lib/webhooker \
@@ -690,38 +692,22 @@ those three values rather than trusting the figure. Measured at 65s on
Docker 29.7.2.) A container `unhealthy` with `connection refused` in Docker 29.7.2.) A container `unhealthy` with `connection refused` in
its health log, or a published port that resets connections, is this. its health log, or a published port that resets connections, is this.
The container runs as a non-root user (`webhooker`, UID 1000), exposes The app runs as a non-root user (`webhooker`, UID 1000), exposes port
port 8080, and includes a health check against 8080, and includes a health check against `/.well-known/healthcheck`.
`/.well-known/healthcheck`. The `/var/lib/webhooker` volume holds all The `/var/lib/webhooker` volume holds all SQLite databases: the main
SQLite databases: the main application database (`webhooker.db`), the application database (`webhooker.db`), the per-webhook event databases
per-webhook event databases (`events-{uuid}.db`), and any archive (`events-{uuid}.db`), and any archive databases written by `database`
databases written by `database` targets (`archive-{uuid}.db`). Mount targets (`archive-{uuid}.db`). Mount this as a persistent volume to
this as a persistent volume to preserve data across container preserve data across container restarts.
restarts.
**The bind-mounted directory must be owned by UID 1000, or the **The container sets its data directory's owner and mode itself
container does not start.** Docker creates a `-v` source path that before the app starts**, so a host directory can be mounted as it is,
does not exist yet as `root:root`, and the process runs as UID 1000, whoever owns it. The image's `ENTRYPOINT`,
so it cannot take its `DATA_DIR` lock: `deploy/docker-entrypoint.sh`, starts as root, creates `DATA_DIR` if
it is missing, gives the directory and anything in it that belongs to
``` another user to `webhooker`, sets the directory to `0750`, and only
webhooker: locking data directory /var/lib/webhooker: open then runs the app as `webhooker`. Started with `--user`, it changes
/var/lib/webhooker/webhooker.lock: permission denied nothing and runs the app as that user.
```
It exits non-zero at that point, before opening any database. Create
the directory ahead of the first `docker run`:
```bash
mkdir -p /path/to/data
chown 1000:1000 /path/to/data
chmod 750 /path/to/data
```
The same `chown` is what a restore needs — see step 4 of
[Restore](#restore). A **named volume** does not have this problem:
Docker copies the image's ownership onto a volume it initializes, and
the image creates `/var/lib/webhooker` owned by `webhooker`.
**The file modes are not yours to set, and do not depend on the **The file modes are not yours to set, and do not depend on the
directory.** `webhooker.db` holds target configuration in plaintext — directory.** `webhooker.db` holds target configuration in plaintext —
@@ -729,13 +715,10 @@ bearer tokens, API keys, Slack webhook URLs — along with the session
encryption key, so webhooker creates every SQLite file it owns `0600`: encryption key, so webhooker creates every SQLite file it owns `0600`:
each database and both of its `-wal` and `-shm` sidecars, across all each database and both of its `-wal` and `-shm` sidecars, across all
three tiers. Files an earlier build left `0644` are tightened when three tiers. Files an earlier build left `0644` are tightened when
they are opened. A `DATA_DIR` webhooker creates itself is `0750`, but they are opened. The directory's `0750` is defence in depth — it stops
a bind mount supplies its own directory and Docker's default for one other local users listing the directory and learning your webhook
it creates is `0755`; the `0600` files hold there regardless. The UUIDs from the `events-{uuid}.db` filenames — not the barrier
`chmod 750` above is defence in depth — it stops other local users protecting the credentials.
listing the directory and learning your webhook UUIDs from the
`events-{uuid}.db` filenames — not the barrier protecting the
credentials.
### Running under upaas ### Running under upaas
@@ -751,17 +734,6 @@ repository's `Dockerfile` and runs it. The app needs:
app name, port `8080`. Leave `PORT` unset: the image's health check app name, port `8080`. Leave `PORT` unset: the image's health check
probes `8080`. probes `8080`.
- **Volume:** one host directory mounted at `/var/lib/webhooker`. - **Volume:** one host directory mounted at `/var/lib/webhooker`.
upaas bind-mounts the host path it is given and does not create it,
and the container does not start unless UID 1000 owns it (see
[Running with Docker](#running-with-docker)). Create it before the
first deploy:
```bash
mkdir -p /path/to/data
chown 1000:1000 /path/to/data
chmod 750 /path/to/data
```
- **Environment variables:** - **Environment variables:**
- `WEBHOOKER_ENVIRONMENT=prod` - `WEBHOOKER_ENVIRONMENT=prod`
- `TRUSTED_PROXIES`: your reverse proxy's address on that Docker - `TRUSTED_PROXIES`: your reverse proxy's address on that Docker
@@ -1020,12 +992,12 @@ done
`.backup` reads through the WAL and writes a single consistent file with `.backup` reads through the WAL and writes a single consistent file with
no sidecars of its own, so the destination is complete as it stands. no sidecars of its own, so the destination is complete as it stands.
Two caveats. First, the runtime image is `alpine:3.21` with only Two caveats. First, the runtime image is `alpine:3.21` with only
`ca-certificates` added — the `sqlite3` CLI is **not** in it, so run `ca-certificates` and `su-exec` added — the `sqlite3` CLI is **not** in
this on the host against the volume path, or from a throwaway container it, so run this on the host against the volume path, or from a
that mounts the volume. Second, each file is captured at its own throwaway container that mounts the volume. Second, each file is
instant, so a webhook created or an event delivered between two files captured at its own instant, so a webhook created or an event delivered
being copied lands in one and not the other. If you need the whole set between two files being copied lands in one and not the other. If you
coherent as of a single moment, stop the service. need the whole set coherent as of a single moment, stop the service.
Note that `sqlite3 <db> .dump` is **not** one of these procedures: it is Note that `sqlite3 <db> .dump` is **not** one of these procedures: it is
an export, it holds a read transaction open for as long as it runs, and an export, it holds a read transaction open for as long as it runs, and
@@ -1079,21 +1051,11 @@ with any `-wal`/`-shm` beside it, or wait until there are none.
archive not opened since a crash. A copy salvaged from a crashed archive not opened since a crash. A copy salvaged from a crashed
instance has them for everything, and needs all of them. instance has them for everything, and needs all of them.
4. **Fix ownership.** The container runs as the non-root `webhooker` 4. Start the service. The container gives the directory and the
user, UID 1000 / GID 1000. Restored files must be owned by (or restored files to the `webhooker` user before the app starts,
writable by) that UID, and so must the directory itself — SQLite whoever restored them (see
creates the `-wal` and `-shm` sidecars beside the database, so a [Running with Docker](#running-with-docker)). `AutoMigrate` runs
writable file inside a directory it cannot write is not enough: against each restored database as it is opened.
```bash
chown -R 1000:1000 /path/to/data
```
Restoring as `root` on the host and forgetting this step is the
usual way a restore fails.
5. Start the service. `AutoMigrate` runs against each restored database
as it is opened.
### Upgrades ### Upgrades
@@ -1103,7 +1065,7 @@ unconditionally against whatever files it finds:
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`, - the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
`Entrypoint`, `Target` `Entrypoint`, `Target`
- each event database when it is lazily opened — `Event`, `Delivery`, - each event database when it is lazily opened — `Event`, `Delivery`,
`DeliveryResult` `DeliveryResult`, `EventTotals`, `TargetTotals`
- each archive database on every open and reopen - each archive database on every open and reopen
There is no schema version table, no migration ledger, and no down There is no schema version table, no migration ledger, and no down
@@ -1254,14 +1216,15 @@ This repository adheres to the
standard: normalized scripts in `script/` are the entrypoints for the standard: normalized scripts in `script/` are the entrypoints for the
development workflow. Ten of the Makefile's seventeen targets are thin development workflow. Ten of the Makefile's seventeen targets are thin
shims that call them; `build`, `run`, `dev`, `deps`, `clean`, `css` and shims that call them; `build`, `run`, `dev`, `deps`, `clean`, `css` and
`version` are inline commands with no script behind them, though `version` are inline commands with no script behind them, though `build`,
`build` and `version` both take their value from `script/version`. `run` and `dev` first run `script/assets`, and `build` and `version` both
take their value from `script/version`.
`make check` needs the third-party browser assets in `static/`, which `script/test`, `make build` and `make dev` each run `script/assets`
are not committed, so run `make bootstrap` (or just `make assets`) once first, which writes the ignored `static/js/alpine.min.js` (see
after cloning. Without them the tests fail with a message naming that [Third-party browser assets](#third-party-browser-assets)), so
remedy. `make check` does not fetch them itself because it must not `make test`, `make check` and the pre-commit hook work on a fresh clone
change any files in the repo. without a separate step.
We provide: We provide:
@@ -1269,8 +1232,8 @@ We provide:
- `script/setup` — make a fresh clone ready for development - `script/setup` — make a fresh clone ready for development
(bootstrap, then install-precommit) (bootstrap, then install-precommit)
- `script/projectname` — output the project name ("webhooker") - `script/projectname` — output the project name ("webhooker")
- `script/fetch-assets` — download the third-party browser assets into - `script/assets` — extract Alpine.js from its tarball in `3p/` (see
`static/`, verifying each against its pinned sha256 [Third-party browser assets](#third-party-browser-assets))
- `script/test` — run the test suite - `script/test` — run the test suite
- `script/lint` — run golangci-lint in Docker (see Linting below) - `script/lint` — run golangci-lint in Docker (see Linting below)
- `script/fmt` — format all code (writes) - `script/fmt` — format all code (writes)
@@ -1292,24 +1255,25 @@ We provide:
## Third-party browser assets ## Third-party browser assets
The web UI serves one third-party script, Alpine.js. It is **not** committed: The web UI serves one third-party script, Alpine.js. Its npm package tarball
a minified bundle in the tree is unreviewable, and `REPO_POLICIES.md` bars is committed as `3p/alpinejs-3.14.9.tgz`, byte for byte as the npm registry
both committed build artifacts and unpinned external references. publishes it. It is a dependency, not this repo's build output, so
`REPO_POLICIES.md`'s rule against committed build artifacts does not apply.
The directory is `3p/` rather than `vendor/` because Go treats a root
`vendor/` directory as its module vendor directory.
Instead `script/fetch-assets` downloads it from a pinned URL, checks the `script/assets` (`make assets`) extracts the browser build,
download against a hardcoded sha256, and installs it under `static/`. The `package/dist/cdn.min.js`, from the tarball to `static/js/alpine.min.js`,
sha256 of every installed asset is recorded in `static/vendor.sha256`, and where `go:embed` picks it up. `script/test`, `make build` and `make dev` run
`static/vendor_test.go` re-hashes the bytes `go:embed` put in the binary it first, and the Dockerfile builds through `make test` and `make build`, so
against that manifest — so the pin is enforced on what actually ships, not nothing downloads Alpine.js. The extracted file is not committed, and
merely written down. Any mismatch fails the build. `.dockerignore` keeps any host copy out of the build context.
`make bootstrap` runs the fetch for local development, and the Dockerfile To move to a new version: download
runs it in the build stage; `.gitignore` and `.dockerignore` keep the `https://registry.npmjs.org/alpinejs/-/alpinejs-<version>.tgz`, check it
artifact out of both the repo and the build context. against the `dist.integrity` hash listed at
`https://registry.npmjs.org/alpinejs/<version>`, replace the tarball in `3p/`
To move to a new version: update the version, URL, and tarball sha256 in with it, update its file name in `script/assets`, and run `make check`.
`script/fetch-assets` and the asset sha256 in `static/vendor.sha256`, then
run `make assets && make check`.
## Rationale ## Rationale
@@ -1417,7 +1381,7 @@ The codebase uses consistent naming throughout (rename completed in
### Data Model ### Data Model
webhooker's data model has nine entities organized into two tiers: the webhooker's data model has eleven entities organized into two tiers: the
**application tier** (user and webhook configuration) and the **event **application tier** (user and webhook configuration) and the **event
tier** (event ingestion, delivery, and logging). tier** (event ingestion, delivery, and logging).
@@ -1446,6 +1410,13 @@ tier** (event ingestion, delivery, and logging).
│ ┌──────────┐ ┌──────────┐ ┌─────────────────┐ │ │ ┌──────────┐ ┌──────────┐ ┌─────────────────┐ │
│ │ Event │──1:N──│ Delivery │──1:N──│ DeliveryResult │ │ │ │ Event │──1:N──│ Delivery │──1:N──│ DeliveryResult │ │
│ └──────────┘ └──────────┘ └─────────────────┘ │ │ └──────────┘ └──────────┘ └─────────────────┘ │
│ │
│ ┌──────────────┐ (one row: running counts of events) │
│ │ EventTotals │ │
│ └──────────────┘ │
│ ┌──────────────┐ (one row per target: running counts │
│ │ TargetTotals │ of its deliveries) │
│ └──────────────┘ │
└─────────────────────────────────────────────────────────────┘ └─────────────────────────────────────────────────────────────┘
``` ```
@@ -1472,7 +1443,7 @@ A registered user of the webhooker service.
| Field | Type | Description | | Field | Type | Description |
| ---------- | -------- | ----------- | | ---------- | -------- | ----------- |
| `id` | UUID | Primary key | | `id` | UUID | Primary key |
| `username` | string | Unique login name | | `username` | string | Unique login name, at most 1024 bytes so that it fits in the session cookie |
| `password` | string | Argon2id hash (never exposed via API) | | `password` | string | Argon2id hash (never exposed via API) |
**Relations:** Has many Webhooks. Has many APIKeys. **Relations:** Has many Webhooks. Has many APIKeys.
@@ -1698,6 +1669,7 @@ status across potentially multiple attempts.
| `event_id` | UUID | Foreign key → Event | | `event_id` | UUID | Foreign key → Event |
| `target_id`| UUID | Foreign key → Target | | `target_id`| UUID | Foreign key → Target |
| `status` | DeliveryStatus | One of: `pending`, `delivered`, `failed`, `retrying` | | `status` | DeliveryStatus | One of: `pending`, `delivered`, `failed`, `retrying` |
| `finished_at` | timestamp | When the delivery became `delivered` or `failed` (nullable; empty while `pending` or `retrying`) |
**Relations:** Belongs to Event. Belongs to Target. Has many **Relations:** Belongs to Event. Belongs to Target. Has many
DeliveryResults. DeliveryResults.
@@ -1765,33 +1737,65 @@ retries) is individually logged for full observability.
**Relations:** Belongs to Delivery. **Relations:** Belongs to Delivery.
#### EventTotals and TargetTotals
Running counts in each event database, read by the statistics pane at the
top of the webhook page. `EventTotals` is one row:
| Field | Type | Description |
| ---------------- | ------- | ----------- |
| `events` | integer | Events ever stored, resubmitted copies included |
| `events_removed` | integer | Events retention has deleted |
`TargetTotals` is one row per target, created by the first delivery to it:
| Field | Type | Description |
| -------------------- | ------- | ----------- |
| `target_id` | UUID | The target (primary key) |
| `deliveries` | integer | Deliveries to it ever created, replays included |
| `delivered` | integer | Of those, how many became `delivered` |
| `failed` | integer | Of those, how many became `failed` |
| `deliveries_removed` | integer | Its deliveries retention has deleted |
| `failed_removed` | integer | Its failed deliveries retention has deleted |
Each count changes in the transaction that writes or deletes the rows it
counts. The pane's lifetime events are `events`, and its lifetime
deliveries and failures are `deliveries` and `failed` summed over the
targets; each figure within retention is the same less what retention
removed, so neither needs the rows themselves. Its last-10-minutes and
last-24-hours figures are counted from the `events` and `deliveries`
indexes over just that window, the deliveries in one query grouped by
target. Its failure percentage for a window is the deliveries that became
`failed` in it out of all that became `delivered` or `failed` in it, and
a dash when none did.
#### Event-tier indexes #### Event-tier indexes
These indexes on the per-webhook event databases are declared in the model These indexes on the per-webhook event databases are declared in the model
tags, so `AutoMigrate` creates them on a fresh and on an existing database: tags, so `AutoMigrate` creates them on a fresh database:
| Table | Columns | Serves | | Table | Columns | Serves |
| ------------------ | --------------------------- | ------ | | ------------------ | --------------------------- | ------ |
| `deliveries` | `status`, `deleted_at` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status | | `deliveries` | `status`, `deleted_at`, `finished_at`, `target_id` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, and the webhook page's statistics, which count each target's deliveries by status and when they finished |
| `deliveries` | `event_id`, `deleted_at` | The event log, which loads each event's deliveries, and retention, which selects and deletes the deliveries of expired events | | `deliveries` | `event_id`, `deleted_at` | The event log, which loads each event's deliveries, and retention, which counts and deletes the deliveries of expired events |
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events | | `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
| `events` | `deleted_at`, `created_at` | Retention, which selects expired events by age | | `events` | `deleted_at`, `created_at` | The webhook page's statistics, which count recent events and find the newest |
| `events` | `created_at` | Retention's delete of the expired events themselves | | `events` | `created_at` | Retention, which selects expired events by age |
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention's GORM's soft delete adds `deleted_at IS NULL` to these queries; retention
deletes leave it out, but their lookups of expired rows keep it. SQLite keeps leaves it out. SQLite keeps no statistics on these tables, and without them it
no statistics on these tables, and without them it rates the `deleted_at` rates the `deleted_at` index, which every live row matches, above an index on
index, which every live row matches, above an index on a column matched a column matched against several values or compared with `<`. So every index
against several values or compared with `<`. So every index but the last also but the last also covers `deleted_at`. It comes second, so that retention can
covers `deleted_at`. It comes second, so that retention's deletes can use the use the index without it, except in `events`, where `created_at` is compared
index without it, except in `events`, where `created_at` is compared with `<` with `<` and SQLite narrows by a `<` only on the last column it uses.
and SQLite narrows by a `<` only on the last column it uses.
#### Common Fields #### Common Fields
Every entity except `Setting` includes these fields from `BaseModel`. Every entity except `Setting`, `EventTotals` and `TargetTotals` includes
`Setting` is a bare key-value row with no `id`, no timestamps and no these fields from `BaseModel`. `Setting` is a bare key-value row with no
soft delete: `id`, no timestamps and no soft delete, and the two totals tables hold
only counts, keyed by a numeric `id` and by `target_id`:
| Field | Type | Description | | Field | Type | Description |
| ------------ | --------- | ----------- | | ------------ | --------- | ----------- |
@@ -1833,6 +1837,8 @@ encryption key is generated and stored, and an `admin` user is created.
- **Events** — captured incoming webhook payloads - **Events** — captured incoming webhook payloads
- **Deliveries** — event-to-target pairings and their status - **Deliveries** — event-to-target pairings and their status
- **DeliveryResults** — individual delivery attempt logs - **DeliveryResults** — individual delivery attempt logs
- **EventTotals** and **TargetTotals** — running counts of the above,
the deliveries per target, kept through retention
Per-webhook databases are created automatically when a webhook is Per-webhook databases are created automatically when a webhook is
created (and lazily on first access for webhooks that predate this created (and lazily on first access for webhooks that predate this
@@ -2412,14 +2418,14 @@ Removing either cap fails 14 subtests.
`internal/middleware/logbound_test.go` and `internal/middleware/logbound_test.go` and
`internal/handlers/logbound_test.go` drive 8 KB of client-chosen text `internal/handlers/logbound_test.go` drive 8 KB of client-chosen text
at each of these — 1 KB at `invalid password`, whose accounts are at each of these — just under 1 KB at `invalid password`, whose
shared with the successful-login line, where a username past 4 KB accounts are shared with the successful-login line and so must stay
overflows the session cookie and answers 500 before that line is within the 1024-byte username limit — through both handlers, and
written — through both handlers, and through seven fills: plain text through seven fills: plain text as the baseline, and then the
as the baseline, and then the quotation mark, backslash, tab, newline, quotation mark, backslash, tab, newline, C0 control and astral
C0 control and astral non-printable, six characters the wider of the non-printable, six characters the wider of the two handlers spends
two handlers spends more on than the client spent sending them. Every more on than the client spent sending them. Every case holds each
case holds each line to the 2,560-byte ceiling. That per-line ceiling line to the 2,560-byte ceiling. That per-line ceiling
is what the figure above states, and every row establishes it. is what the figure above states, and every row establishes it.
Three of the sites go further and bound the whole flood's output — the Three of the sites go further and bound the whole flood's output — the
@@ -2721,7 +2727,7 @@ abuse limit later; they are tracked as future work.
| ------ | --------------------------- | ----------- | | ------ | --------------------------- | ----------- |
| `GET` | `/` | Root redirect, 303 (authenticated → `/sources`, unauthenticated → `/pages/login`) | | `GET` | `/` | Root redirect, 303 (authenticated → `/sources`, unauthenticated → `/pages/login`) |
| `GET` | `/.well-known/healthcheck` | Health check (JSON: `status`, `now`, `uptimeSeconds`, `uptimeHuman`, `version`, `appname`, `maintenanceMode`) | | `GET` | `/.well-known/healthcheck` | Health check (JSON: `status`, `now`, `uptimeSeconds`, `uptimeHuman`, `version`, `appname`, `maintenanceMode`) |
| any | `/s/*` | Static file serving (embedded CSS, JS). Mounted for every method, not just `GET`/`HEAD`: chi's `Mount` registers all methods and `http.FileServer` special-cases only `HEAD` (by omitting the body), so a `POST` or `DELETE` to an asset is answered `200` with the file. Pinned by `TestStaticServesEveryMethod` | | `GET`, `HEAD` | `/s/*` | Static file serving (embedded CSS, JS). `GET` and `HEAD` only — `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` and `CONNECT` are answered `405 Method Not Allowed` with `Allow: GET, HEAD`. Any other method (such as `PROPFIND`) is refused by chi before it reaches this route, and gets `405` without an `Allow` header. Pinned by `TestStaticServesOnlyGetAndHead` |
| `POST` | `/webhook/{uuid}` | Webhook receiver endpoint. `POST` only — every other method is answered `405 Method Not Allowed` with `Allow: POST`. Rate limited (see [Rate Limiting](#rate-limiting)) | | `POST` | `/webhook/{uuid}` | Webhook receiver endpoint. `POST` only — every other method is answered `405 Method Not Allowed` with `Allow: POST`. Rate limited (see [Rate Limiting](#rate-limiting)) |
#### Authentication Endpoints #### Authentication Endpoints
@@ -2788,6 +2794,8 @@ imports. The entry point is `cmd/webhooker/main.go`.
``` ```
webhooker/ webhooker/
├── 3p/
│ └── alpinejs-3.14.9.tgz # Alpine.js npm package, extracted by make assets
├── cmd/webhooker/ ├── cmd/webhooker/
│ └── main.go # Entry point: subcommand dispatch; no args locks DATA_DIR and wires fx │ └── main.go # Entry point: subcommand dispatch; no args locks DATA_DIR and wires fx
├── internal/ ├── internal/
@@ -2811,6 +2819,7 @@ webhooker/
│ │ ├── model_event.go # Event entity (per-webhook DB) │ │ ├── model_event.go # Event entity (per-webhook DB)
│ │ ├── model_delivery.go # Delivery entity (per-webhook DB) │ │ ├── model_delivery.go # Delivery entity (per-webhook DB)
│ │ ├── model_delivery_result.go # DeliveryResult entity (per-webhook DB) │ │ ├── model_delivery_result.go # DeliveryResult entity (per-webhook DB)
│ │ ├── model_totals.go # EventTotals and TargetTotals (per-webhook DB)
│ │ ├── model_apikey.go # APIKey entity │ │ ├── model_apikey.go # APIKey entity
│ │ ├── password.go # Argon2id hashing and verification │ │ ├── password.go # Argon2id hashing and verification
│ │ ├── retention.go # Retention reaper (per-webhook event expiry) │ │ ├── retention.go # Retention reaper (per-webhook event expiry)
@@ -2881,8 +2890,7 @@ webhooker/
│ ├── css/tailwind.css # Generated stylesheet the pages load │ ├── css/tailwind.css # Generated stylesheet the pages load
│ ├── css/style.css # Older hand-written stylesheet, no longer loaded │ ├── css/style.css # Older hand-written stylesheet, no longer loaded
│ ├── js/app.js # Progressive-enhancement copy-to-clipboard │ ├── js/app.js # Progressive-enhancement copy-to-clipboard
│ ├── js/alpine.min.js # Alpine.js, fetched by script/fetch-assets, not committed │ └── js/alpine.min.js # Alpine.js, extracted from 3p/ by make assets, not committed
│ └── vendor.sha256 # Pinned hashes the fetched assets are verified against
├── templates/ # Go HTML templates (base, login, sources, etc.) ├── templates/ # Go HTML templates (base, login, sources, etc.)
├── script/ # Scripts to Rule Them All entrypoints ├── script/ # Scripts to Rule Them All entrypoints
├── Dockerfile # Three stages: lint, test+build, Alpine runtime ├── Dockerfile # Three stages: lint, test+build, Alpine runtime
@@ -3071,7 +3079,11 @@ check, see [The login endpoint](#the-login-endpoint).
- Prometheus metrics behind basic auth - Prometheus metrics behind basic auth
- Static assets embedded in binary (no filesystem access needed at - Static assets embedded in binary (no filesystem access needed at
runtime) runtime)
- Container runs as non-root user (UID 1000) - The app runs as the non-root `webhooker` user (UID 1000) in the
container. The image sets no `USER`, so these run as root: the
`ENTRYPOINT` script, which sets the data directory's owner and mode
before the app starts; the image's health check; and `docker exec`,
unless given `--user`
- GORM soft deletes on every entity that carries `BaseModel`, which is - GORM soft deletes on every entity that carries `BaseModel`, which is
all of them but `Setting` (data preserved for audit) all of them but `Setting` (data preserved for audit)
@@ -3190,18 +3202,21 @@ version is fixed independently of the compiler's:
`make fmt-check`, then `golangci-lint config verify` and `make fmt-check`, then `golangci-lint config verify` and
`golangci-lint run`, both with `--network=none`. `golangci-lint run`, both with `--network=none`.
2. **Builder stage** (`golang:1.26.1-bookworm`) — depends on the lint 2. **Builder stage** (`golang:1.26.1-bookworm`) — depends on the lint
stage passing (it copies a file from it), runs `script/fetch-assets` stage passing (it copies a file from it), runs `make test` and
to download and verify the third-party browser assets, then runs `make build` (both extract Alpine.js from `3p/` first), and finally
`make test` and `make build`, and finally rebuilds the binary with rebuilds the binary with `CGO_ENABLED=1` and static linking so it
`CGO_ENABLED=1` and static linking so it runs on musl. Both builds runs on musl. Both builds go through `make build`, the relink adding
go through `make build`, the relink adding its `-extldflags` via its `-extldflags` via `GO_LDFLAGS`, so neither can drop the `-X` that
`GO_LDFLAGS`, so neither can drop the `-X` that stamps the version. stamps the version. The version arrives as the `VERSION` build arg,
The version arrives as the `VERSION` build arg, since the context since the context has no `.git` (see
has no `.git` (see [Version stamping](#version-stamping)). [Version stamping](#version-stamping)).
3. **Runtime stage** (`alpine:3.21`) — copies the static binary, 3. **Runtime stage** (`alpine:3.21`) — copies the static binary and
creates the `/var/lib/webhooker` directory for all SQLite databases, `deploy/docker-entrypoint.sh`, creates the `/var/lib/webhooker`
runs as the non-root `webhooker` user (UID 1000), exposes port 8080, directory for all SQLite databases, exposes port 8080, and includes
and includes a health check against `/.well-known/healthcheck`. a health check against `/.well-known/healthcheck`. It sets no
`USER`: the `ENTRYPOINT` script starts as root, sets the data
directory's owner and mode, and runs the app as the non-root
`webhooker` user (UID 1000) through `su-exec`.
The lint stage invokes `golangci-lint` directly rather than `make lint`: The lint stage invokes `golangci-lint` directly rather than `make lint`:
it is already the pinned linter image, and `make lint` builds it is already the pinned linter image, and `make lint` builds
@@ -3280,3 +3295,5 @@ MIT
## Author ## Author
[@sneak](https://sneak.berlin) [@sneak](https://sneak.berlin)
+22
View File
@@ -0,0 +1,22 @@
#!/bin/sh
# deploy/docker-entrypoint.sh: the image's ENTRYPOINT. A bind-mounted
# data directory keeps its owner from the host, often root, and the app
# could not write to it. Started as root, this creates DATA_DIR if
# needed, gives it and everything in it to webhooker, sets its mode, and
# runs the command as webhooker, so the app never runs as root. Started
# as another user, it only runs the command.
set -eu
main() {
if [ "$(id -u)" != 0 ]; then
exec "$@"
fi
dir="${DATA_DIR:-/var/lib/webhooker}"
mkdir -p "$dir"
find "$dir" ! -user webhooker -exec chown -h webhooker:webhooker {} +
chmod 750 "$dir"
exec su-exec webhooker "$@"
}
main "$@"
@@ -3,6 +3,8 @@ package database_test
import ( import (
"bytes" "bytes"
"context" "context"
"log/slog"
"path/filepath"
"strings" "strings"
"testing" "testing"
@@ -83,3 +85,37 @@ func TestFirstBoot_PrintsTheAdminPasswordAsABanner(t *testing.T) {
t, ok, "the printed password must open the seeded account", t, ok, "the printed password must open the seeded account",
) )
} }
// TestNewDatabase_IsLoggedWithItsPath is the log half of
// https://git.eeqj.de/sneak/webhooker/issues/359. A DATA_DIR that is
// unexpectedly empty boots exactly like a first start, so the start
// that creates the database must say so, and where. Opening that
// database again must not.
func TestNewDatabase_IsLoggedWithItsPath(t *testing.T) {
t.Parallel()
dir := t.TempDir()
open := func() string {
var out bytes.Buffer
db, err := database.Open(dir, slog.New(slog.NewTextHandler(&out, nil)))
require.NoError(t, err)
require.NoError(t, db.Close())
return out.String()
}
const created = `level=WARN msg="created a new, empty database"`
first := open()
second := open()
assert.Contains(
t, first,
created+" path="+filepath.Join(dir, database.MainDBFileName),
)
assert.NotContains(
t, second, created, "an existing database is not new",
)
}
+13 -1
View File
@@ -8,6 +8,7 @@ import (
"errors" "errors"
"fmt" "fmt"
"io" "io"
"io/fs"
"log/slog" "log/slog"
"os" "os"
"path/filepath" "path/filepath"
@@ -199,6 +200,12 @@ func (d *Database) connectTo(dataDir string) error {
// Construct the main application database path inside DATA_DIR. // Construct the main application database path inside DATA_DIR.
dbPath := filepath.Join(dataDir, MainDBFileName) dbPath := filepath.Join(dataDir, MainDBFileName)
// Checked before opening, which creates the file. A DATA_DIR that
// is unexpectedly empty -- its volume not mounted, say -- looks
// exactly like a first start, so a new database is a warning.
_, statErr := os.Stat(dbPath)
created := errors.Is(statErr, fs.ErrNotExist)
// Opened through OpenSQLite so this handle carries the same WAL // Opened through OpenSQLite so this handle carries the same WAL
// journaling, busy timeout, immediate-transaction locking, and pool // journaling, busy timeout, immediate-transaction locking, and pool
// bounds as every other database file. See sqlite_open.go. // bounds as every other database file. See sqlite_open.go.
@@ -229,7 +236,12 @@ func (d *Database) connectTo(dataDir string) error {
} }
d.db = db d.db = db
d.log.Info("connected to database", "path", dbPath)
if created {
d.log.Warn("created a new, empty database", "path", dbPath)
} else {
d.log.Info("connected to database", "path", dbPath)
}
// Run migrations // Run migrations
return d.migrate() return d.migrate()
+94 -21
View File
@@ -93,11 +93,11 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
deliveries []database.Delivery deliveries []database.Delivery
results []database.DeliveryResult results []database.DeliveryResult
depths []struct{ Depth int } depths []struct{ Depth int }
removed []database.TargetTotals
) )
byStatus := "idx_deliveries_status (status=? AND deleted_at=?)" byStatus := "idx_deliveries_status (status=? AND deleted_at=?)"
byEvent := "idx_deliveries_event_id (event_id=? AND deleted_at=?)" byEvent := "idx_deliveries_event_id (event_id=? AND deleted_at=?)"
byAge := "idx_events_deleted_at_created_at (deleted_at=? AND created_at<?)"
// The delivery engine: recovery and the retry sweep, the sweep for // The delivery engine: recovery and the retry sweep, the sweep for
// stranded pending deliveries, and the queue depth count. // stranded pending deliveries, and the queue depth count.
@@ -123,25 +123,89 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
Order("attempt_num ASC").Find(&results), Order("attempt_num ASC").Find(&results),
"idx_delivery_results_delivery_id (delivery_id=? AND deleted_at=?)") "idx_delivery_results_delivery_id (delivery_id=? AND deleted_at=?)")
// Retention's three deletes (reapExpired), whose subqueries are built // Retention (reapExpired, deleteEvents): one batch of expired
// afresh for each statement as it builds them. // events, then their attempts, deliveries and the events.
expiredEventIDs := func() *gorm.DB { var expired []string
return dry.Model(&database.Event{}).Select("id").
Where("created_at < ?", cutoff)
}
assertPlanUses(t, db, dry.Unscoped().Model(&database.Event{}).
Where("created_at < ?", cutoff).
Limit(database.ExportReapBatchSize).Pluck("id", &expired),
"idx_events_created_at (created_at<?)")
assertPlanUses(t, db, dry.Unscoped().Where( assertPlanUses(t, db, dry.Unscoped().Where(
"delivery_id IN (?)", dry.Model(&database.Delivery{}). "delivery_id IN (?)", dry.Unscoped().Model(&database.Delivery{}).
Select("id").Where("event_id IN (?)", expiredEventIDs()), Select("id").Where("event_id IN ?", ids),
).Delete(&database.DeliveryResult{}), ).Delete(&database.DeliveryResult{}),
"idx_delivery_results_delivery_id (delivery_id=?)", byEvent, byAge) "idx_delivery_results_delivery_id (delivery_id=?)",
assertPlanUses(t, db, dry.Unscoped().Where( "idx_deliveries_event_id (event_id=?)")
"event_id IN (?)", expiredEventIDs(), assertPlanUses(t, db, dry.Unscoped().Model(&database.Delivery{}).
).Delete(&database.Delivery{}), Select("target_id, count(*) AS deliveries_removed, "+
"idx_deliveries_event_id (event_id=?)", byAge) "count(CASE WHEN status = ? THEN 1 END) AS failed_removed",
assertPlanUses(t, db, dry.Unscoped().Where( database.DeliveryStatusFailed).
"created_at < ?", cutoff, Where("event_id IN ?", ids).Group("target_id").Find(&removed),
).Delete(&database.Event{}), "idx_events_created_at (created_at<?)") "idx_deliveries_event_id (event_id=?)")
assertPlanUses(t, db, dry.Unscoped().Where("event_id IN ?", ids).
Delete(&database.Delivery{}), "idx_deliveries_event_id (event_id=?)")
assertPlanUses(t, db, dry.Unscoped().Where("id IN ?", ids).
Delete(&database.Event{}), "sqlite_autoindex_events_1 (id=?)")
}
// TestStatisticsQueriesUseTheirIndexes does the same for the webhook
// page's statistics (readEventStats in the handlers): deliveries in
// progress, each target's deliveries finished since a time, which must
// come from the index alone, events received since a time, and the
// newest event, which must come straight off an index rather than from
// sorting every event.
func TestStatisticsQueriesUseTheirIndexes(t *testing.T) {
t.Parallel()
mgr, lc := setupTestWebhookDBManager(t)
ctx := context.Background()
require.NoError(t, lc.Start(ctx))
defer func() { require.NoError(t, lc.Stop(ctx)) }()
db, err := mgr.GetDB(uuid.New().String())
require.NoError(t, err)
dry := db.Session(&gorm.Session{DryRun: true})
since := time.Now()
var (
count int64
newest []time.Time
byTarget []struct{ TargetID string }
)
assertPlanUses(t, db, dry.Model(&database.Delivery{}).
Where("status IN ?", []database.DeliveryStatus{
database.DeliveryStatusPending,
database.DeliveryStatusRetrying,
}).Count(&count),
"idx_deliveries_status (status=? AND deleted_at=?)")
assertPlanUses(t, db, dry.Model(&database.Delivery{}).
Select("target_id, "+
"count(CASE WHEN status = ? THEN 1 END) AS delivered, "+
"count(CASE WHEN status = ? THEN 1 END) AS failed",
database.DeliveryStatusDelivered,
database.DeliveryStatusFailed).
Where("status IN ? AND finished_at >= ?",
[]database.DeliveryStatus{
database.DeliveryStatusDelivered,
database.DeliveryStatusFailed,
}, since).
Group("target_id").Find(&byTarget),
"COVERING INDEX idx_deliveries_status "+
"(status=? AND deleted_at=? AND finished_at>?)")
assertPlanUses(t, db, dry.Model(&database.Event{}).
Where("created_at >= ?", since).Count(&count),
"idx_events_deleted_at_created_at "+
"(deleted_at=? AND created_at>?)")
newestEvent := dry.Model(&database.Event{}).
Order("created_at DESC").Limit(1).Pluck("created_at", &newest)
assertPlanUses(t, db, newestEvent,
"idx_events_deleted_at_created_at (deleted_at=?)")
assert.NotContains(t, queryPlan(t, db, newestEvent), "TEMP B-TREE")
} }
// assertPlanUses asserts that SQLite's plan for a statement GORM built // assertPlanUses asserts that SQLite's plan for a statement GORM built
@@ -152,6 +216,18 @@ func assertPlanUses(
) { ) {
t.Helper() t.Helper()
plan := queryPlan(t, db, built)
for _, index := range indexes {
assert.Contains(t, plan, index, built.Statement.SQL.String())
}
}
// queryPlan returns SQLite's plan for a statement GORM built in a dry
// run, run with the same SQL and arguments GORM would send.
func queryPlan(t *testing.T, db, built *gorm.DB) string {
t.Helper()
var plan []struct{ Detail string } var plan []struct{ Detail string }
require.NoError(t, db.Raw( require.NoError(t, db.Raw(
@@ -159,8 +235,5 @@ func assertPlanUses(
built.Statement.Vars..., built.Statement.Vars...,
).Scan(&plan).Error) ).Scan(&plan).Error)
for _, index := range indexes { return fmt.Sprint(plan)
assert.Contains(t, fmt.Sprint(plan), index,
built.Statement.SQL.String())
}
} }
+4
View File
@@ -28,6 +28,10 @@ func NewTestRetentionReaper(
} }
} }
// ExportReapBatchSize exposes how many expired events one retention
// transaction deletes.
const ExportReapBatchSize = reapBatchSize
// ExportSweep runs a single retention sweep synchronously for tests. // ExportSweep runs a single retention sweep synchronously for tests.
func (r *RetentionReaper) ExportSweep(ctx context.Context) { func (r *RetentionReaper) ExportSweep(ctx context.Context) {
r.sweep(ctx) r.sweep(ctx)
+13 -2
View File
@@ -1,6 +1,10 @@
package database package database
import "gorm.io/gorm" import (
"time"
"gorm.io/gorm"
)
// DeliveryStatus represents the status of a delivery // DeliveryStatus represents the status of a delivery
type DeliveryStatus string type DeliveryStatus string
@@ -37,7 +41,7 @@ type Delivery struct {
BaseModel BaseModel
EventID string `gorm:"type:uuid;not null;index:idx_deliveries_event_id,priority:1" json:"eventId"` EventID string `gorm:"type:uuid;not null;index:idx_deliveries_event_id,priority:1" json:"eventId"`
TargetID string `gorm:"type:uuid;not null" json:"targetId"` TargetID string `gorm:"type:uuid;not null;index:idx_deliveries_status,priority:4" json:"targetId"`
Status DeliveryStatus `gorm:"not null;default:'pending';index:idx_deliveries_status,priority:1" json:"status"` Status DeliveryStatus `gorm:"not null;default:'pending';index:idx_deliveries_status,priority:1" json:"status"`
// DeletedAt repeats the BaseModel field only to be the second column // DeletedAt repeats the BaseModel field only to be the second column
@@ -45,6 +49,13 @@ type Delivery struct {
// gives. // gives.
DeletedAt gorm.DeletedAt `gorm:"index:idx_deliveries_event_id,priority:2;index:idx_deliveries_status,priority:2" json:"deletedAt,omitzero"` DeletedAt gorm.DeletedAt `gorm:"index:idx_deliveries_event_id,priority:2;index:idx_deliveries_status,priority:2" json:"deletedAt,omitzero"`
// FinishedAt is when the delivery became delivered or failed, and
// nil while it is pending or retrying. It and then TargetID end the
// status index, so the webhook page counts each target's deliveries
// that finished in a recent window by reading just that window from
// the index.
FinishedAt *time.Time `gorm:"index:idx_deliveries_status,priority:3" json:"finishedAt,omitempty"`
// Relations // Relations
Event Event `json:"event,omitzero"` Event Event `json:"event,omitzero"`
Target Target `json:"target,omitzero"` Target Target `json:"target,omitzero"`
+91
View File
@@ -0,0 +1,91 @@
package database
import (
"fmt"
"gorm.io/gorm"
)
// The running totals in a webhook's event database keep the webhook
// page's lifetime figures right after retention has removed the rows
// they count, and let the page show them without counting every row.
// Each total changes in the transaction that writes or deletes the
// rows it counts.
// EventTotals is the single row counting a webhook's events: every
// event ever stored, and how many of them retention has deleted.
type EventTotals struct {
ID int64 `gorm:"primaryKey"`
Events int64 `gorm:"not null"`
EventsRemoved int64 `gorm:"not null"`
}
// TableName names the table AddEventTotals updates.
func (EventTotals) TableName() string {
return "event_totals"
}
// TargetTotals is one row per target counting its deliveries: every
// delivery ever created, how many became delivered and how many
// failed, and how many deliveries and failed deliveries retention has
// deleted. The webhook's delivery figures are these rows summed.
type TargetTotals struct {
TargetID string `gorm:"type:uuid;primaryKey"`
Deliveries int64 `gorm:"not null"`
Delivered int64 `gorm:"not null"`
Failed int64 `gorm:"not null"`
DeliveriesRemoved int64 `gorm:"not null"`
FailedRemoved int64 `gorm:"not null"`
}
// TableName names the table AddTargetTotals updates.
func (TargetTotals) TableName() string {
return "target_totals"
}
// AddEventTotals adds each count in add to the webhook's event totals.
// Call it on the transaction that writes or deletes the events it
// counts.
func AddEventTotals(tx *gorm.DB, add EventTotals) error {
err := tx.Exec(
`UPDATE event_totals SET
events = events + ?,
events_removed = events_removed + ?`,
add.Events, add.EventsRemoved,
).Error
if err != nil {
return fmt.Errorf("adding to event totals: %w", err)
}
return nil
}
// AddTargetTotals adds each count in add to the totals of the target
// add.TargetID names, creating its row the first time. Call it on the
// transaction that writes or deletes the deliveries it counts.
func AddTargetTotals(tx *gorm.DB, add TargetTotals) error {
err := tx.Exec(
`INSERT INTO target_totals (target_id, deliveries, delivered,
failed, deliveries_removed, failed_removed)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT (target_id) DO UPDATE SET
deliveries = deliveries + excluded.deliveries,
delivered = delivered + excluded.delivered,
failed = failed + excluded.failed,
deliveries_removed =
deliveries_removed + excluded.deliveries_removed,
failed_removed = failed_removed + excluded.failed_removed`,
add.TargetID, add.Deliveries, add.Delivered,
add.Failed, add.DeliveriesRemoved, add.FailedRemoved,
).Error
if err != nil {
return fmt.Errorf(
"adding to totals of target %s: %w", add.TargetID, err,
)
}
return nil
}
+47 -2
View File
@@ -1,13 +1,58 @@
package database package database
import (
"errors"
"fmt"
"gorm.io/gorm"
)
// MaxUsernameBytes is the longest username, in bytes, that a user may
// have. The same number appears in the check constraint on
// User.Username, because a struct tag cannot reference a constant.
//
// A login stores the username in the session cookie, and both
// securecookie and browsers refuse a cookie value past about 4096
// bytes. That value is the session base64-encoded twice, so it holds
// 4096 × 3/4 × 3/4 = 2304 bytes of session, and the signature,
// timestamp and the session's other values take about 270 of those: a
// username longer than about 2030 bytes can never log in. The limit is
// about half that, so the session can carry more values later without
// locking out an account whose username is already at the limit.
const MaxUsernameBytes = 1024
// ErrUsernameTooLong is returned when a user is saved with a username
// longer than MaxUsernameBytes.
var ErrUsernameTooLong = errors.New("username is too long")
// User represents a user of the webhooker service // User represents a user of the webhooker service
//
//nolint:lll // a struct tag cannot wrap
type User struct { type User struct {
BaseModel BaseModel
Username string `gorm:"uniqueIndex;not null" json:"username"` Username string `gorm:"uniqueIndex;not null;check:length(CAST(username AS BLOB)) <= 1024" json:"username"`
Password string `gorm:"not null" json:"-"` // Argon2 hashed Password string `gorm:"not null" json:"-"` // Argon2 hashed
// Relations // Relations
Webhooks []Webhook `json:"webhooks,omitempty"` Webhooks []Webhook `json:"webhooks,omitempty"`
APIKeys []APIKey `json:"apiKeys,omitempty"` APIKeys []APIKey `json:"apiKeys,omitempty"`
} }
// BeforeSave rejects a username longer than MaxUsernameBytes when a whole
// User is created or saved, so those calls get ErrUsernameTooLong rather
// than the database's constraint error. A column update such as
// Update("username", ...) is caught only by the check constraint, as is
// any path that writes the table without this model.
func (u *User) BeforeSave(_ *gorm.DB) error {
if len(u.Username) > MaxUsernameBytes {
return fmt.Errorf(
"%w: %d bytes, limit is %d",
ErrUsernameTooLong,
len(u.Username),
MaxUsernameBytes,
)
}
return nil
}
+65
View File
@@ -0,0 +1,65 @@
package database_test
import (
"strings"
"testing"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
)
// usernameAtLimit is exactly MaxUsernameBytes long, built from a
// two-byte character. A check that counted characters rather than bytes
// would see half the length and let the one-byte-longer name through.
func usernameAtLimit() string {
return strings.Repeat("é", database.MaxUsernameBytes/2)
}
func TestUserCreate_RejectsOverlongUsername(t *testing.T) {
t.Parallel()
db := startedTestDB(t)
err := db.Create(&database.User{
Username: usernameAtLimit() + "x",
Password: "hash",
}).Error
require.ErrorIs(t, err, database.ErrUsernameTooLong)
}
func TestUserCreate_AcceptsUsernameAtLimit(t *testing.T) {
t.Parallel()
db := startedTestDB(t)
require.NoError(t, db.Create(&database.User{
Username: usernameAtLimit(),
Password: "hash",
}).Error)
}
// TestUsersTable_EnforcesUsernameLimitWithoutTheModel inserts with raw
// SQL, as a path that bypassed User.BeforeSave would, so only the
// table's check constraint stands between it and an over-long
// username. Accepting the name at the limit and refusing the next byte
// also pins the constraint's number to MaxUsernameBytes.
func TestUsersTable_EnforcesUsernameLimitWithoutTheModel(t *testing.T) {
t.Parallel()
db := startedTestDB(t)
insert := "INSERT INTO users (id, username, password) VALUES (?, ?, ?)"
require.NoError(t, db.Exec(
insert, uuid.New().String(), usernameAtLimit(), "hash",
).Error)
err := db.Exec(
insert, uuid.New().String(), usernameAtLimit()+"x", "hash",
).Error
require.Error(t, err)
assert.Contains(t, err.Error(), "CHECK constraint failed")
}
+2 -1
View File
@@ -2,7 +2,8 @@ package database
// Migrate runs database migrations for the main application database. // Migrate runs database migrations for the main application database.
// Only configuration-tier models are stored in the main database. // Only configuration-tier models are stored in the main database.
// Event-tier models (Event, Delivery, DeliveryResult) live in // Event-tier models (Event, Delivery, DeliveryResult, EventTotals,
// TargetTotals) live in
// per-webhook dedicated databases managed by WebhookDBManager. // per-webhook dedicated databases managed by WebhookDBManager.
func (d *Database) Migrate() error { func (d *Database) Migrate() error {
return d.db.AutoMigrate( return d.db.AutoMigrate(
+94 -47
View File
@@ -18,6 +18,13 @@ import (
// computation. // computation.
const hoursPerDay = 24 const hoursPerDay = 24
// reapBatchSize is how many expired events one retention transaction
// deletes. A transaction holds the event database's write lock, which
// the receiver and the delivery workers wait for, so a large prune is
// split into transactions each short enough to finish well inside the
// busy timeout.
const reapBatchSize = 1000
// RetentionReaperParams holds the fx dependencies for the // RetentionReaperParams holds the fx dependencies for the
// RetentionReaper. // RetentionReaper.
type RetentionReaperParams struct { type RetentionReaperParams struct {
@@ -265,57 +272,97 @@ func retentionCutoff(
), true ), true
} }
// reapExpired hard-deletes, in foreign-key-safe order, the delivery // reapExpired hard-deletes the events older than cutoff, with their
// results, deliveries, and events associated with events older than // deliveries and delivery results, reapBatchSize events per
// cutoff. Deletes are unscoped so rows are physically removed rather // transaction until none is left. It returns the number of events
// than soft-deleted, reclaiming disk. It returns the number of events
// deleted. // deleted.
func reapExpired(db *gorm.DB, cutoff time.Time) (int64, error) { func reapExpired(db *gorm.DB, cutoff time.Time) (int64, error) {
// Fresh subqueries are built per statement to avoid reusing a var total int64
// mutated builder across executions.
expiredEventIDs := func() *gorm.DB {
return db.Model(&Event{}).
Select("id").
Where("created_at < ?", cutoff)
}
expiredDeliveryIDs := func() *gorm.DB {
return db.Model(&Delivery{}).
Select("id").
Where("event_id IN (?)", expiredEventIDs())
}
// 1. Delivery results whose delivery belongs to an expired event. for {
res := db.Unscoped(). var eventIDs []string
Where("delivery_id IN (?)", expiredDeliveryIDs()).
Delete(&DeliveryResult{})
if res.Error != nil {
return 0, fmt.Errorf(
"deleting expired delivery results: %w",
res.Error,
)
}
// 2. Deliveries belonging to an expired event. err := db.Transaction(func(tx *gorm.DB) error {
del := db.Unscoped(). err := tx.Unscoped().Model(&Event{}).
Where("event_id IN (?)", expiredEventIDs()). Where("created_at < ?", cutoff).
Delete(&Delivery{}) Limit(reapBatchSize).
if del.Error != nil { Pluck("id", &eventIDs).Error
return 0, fmt.Errorf( if err != nil {
"deleting expired deliveries: %w", return fmt.Errorf("selecting expired events: %w", err)
del.Error, }
)
}
// 3. The expired events themselves. if len(eventIDs) == 0 {
ev := db.Unscoped(). return nil
Where("created_at < ?", cutoff). }
Delete(&Event{})
if ev.Error != nil {
return 0, fmt.Errorf(
"deleting expired events: %w",
ev.Error,
)
}
return ev.RowsAffected, nil return deleteEvents(tx, eventIDs)
})
if err != nil {
return total, err
}
total += int64(len(eventIDs))
if len(eventIDs) < reapBatchSize {
return total, nil
}
}
}
// deleteEvents hard-deletes the given events and, in foreign-key-safe
// order before them, their delivery results and deliveries, then adds
// what it deleted to the running totals. It runs on reapExpired's
// transaction, so the totals change exactly when the rows do. Deletes
// are unscoped so rows are physically removed rather than
// soft-deleted, reclaiming disk.
func deleteEvents(tx *gorm.DB, eventIDs []string) error {
// 1. The delivery results of the events' deliveries.
err := tx.Unscoped().
Where("delivery_id IN (?)", tx.Unscoped().Model(&Delivery{}).
Select("id").
Where("event_id IN ?", eventIDs)).
Delete(&DeliveryResult{}).Error
if err != nil {
return fmt.Errorf("deleting expired delivery results: %w", err)
}
// 2. The events' deliveries, after counting them, and the failed
// ones among them, per target. The status is tested in the select
// list rather than the WHERE clause: there, SQLite would read every
// failed delivery the webhook has through the status index,
// instead of only these through the event_id index.
var removed []TargetTotals
err = tx.Unscoped().Model(&Delivery{}).
Select("target_id, count(*) AS deliveries_removed, "+
"count(CASE WHEN status = ? THEN 1 END) AS failed_removed",
DeliveryStatusFailed).
Where("event_id IN ?", eventIDs).
Group("target_id").
Find(&removed).Error
if err != nil {
return fmt.Errorf("counting expired deliveries: %w", err)
}
err = tx.Unscoped().
Where("event_id IN ?", eventIDs).
Delete(&Delivery{}).Error
if err != nil {
return fmt.Errorf("deleting expired deliveries: %w", err)
}
// 3. The events themselves.
ev := tx.Unscoped().Where("id IN ?", eventIDs).Delete(&Event{})
if ev.Error != nil {
return fmt.Errorf("deleting expired events: %w", ev.Error)
}
for i := range removed {
err = AddTargetTotals(tx, removed[i])
if err != nil {
return err
}
}
return AddEventTotals(tx, EventTotals{EventsRemoved: ev.RowsAffected})
} }
+215
View File
@@ -0,0 +1,215 @@
package database_test
import (
"context"
"net/http"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// readEventTotals reads a webhook database's row of event totals,
// asserting that it has exactly one.
func readEventTotals(t *testing.T, db *gorm.DB) database.EventTotals {
t.Helper()
var rows []database.EventTotals
require.NoError(t, db.Find(&rows).Error)
require.Len(t, rows, 1)
return rows[0]
}
// readTargetTotals reads a webhook database's target totals, keyed by
// target.
func readTargetTotals(
t *testing.T, db *gorm.DB,
) map[string]database.TargetTotals {
t.Helper()
var rows []database.TargetTotals
require.NoError(t, db.Find(&rows).Error)
byTarget := make(map[string]database.TargetTotals, len(rows))
for _, row := range rows {
byTarget[row.TargetID] = row
}
return byTarget
}
// TestWebhookDBManager_TotalsSurviveReopen verifies that a new event
// database starts with one row of zero event totals and no target
// totals, that adding to a target twice adds to the one row, and that
// opening the database again keeps everything added.
func TestWebhookDBManager_TotalsSurviveReopen(t *testing.T) {
t.Parallel()
mgr, lc := setupTestWebhookDBManager(t)
ctx := context.Background()
require.NoError(t, lc.Start(ctx))
defer func() { require.NoError(t, lc.Stop(ctx)) }()
webhookID := uuid.New().String()
db, err := mgr.GetDB(webhookID)
require.NoError(t, err)
fresh := readEventTotals(t, db)
assert.Equal(t, database.EventTotals{ID: fresh.ID}, fresh)
assert.Empty(t, readTargetTotals(t, db))
first, second := uuid.New().String(), uuid.New().String()
require.NoError(t, database.AddEventTotals(db, database.EventTotals{
Events: 2,
}))
require.NoError(t, database.AddTargetTotals(db, database.TargetTotals{
TargetID: first, Deliveries: 2, Delivered: 1,
}))
require.NoError(t, database.AddTargetTotals(db, database.TargetTotals{
TargetID: first, Failed: 1,
}))
require.NoError(t, database.AddTargetTotals(db, database.TargetTotals{
TargetID: second, Deliveries: 1,
}))
// Drop the cached connection so the next open reopens the file,
// as a restart would.
require.NoError(t, mgr.CloseAll())
db, err = mgr.GetDB(webhookID)
require.NoError(t, err)
assert.Equal(t, database.EventTotals{ID: fresh.ID, Events: 2},
readEventTotals(t, db))
assert.Equal(t, map[string]database.TargetTotals{
first: {
TargetID: first, Deliveries: 2, Delivered: 1, Failed: 1,
},
second: {TargetID: second, Deliveries: 1},
}, readTargetTotals(t, db))
}
// TestRetentionReaper_PrunesMoreThanOneBatch verifies that a prune
// larger than one transaction's batch removes every expired event with
// its deliveries and delivery results, keeps the recent event, and
// adds what it removed to the event and target totals, so the totals
// within retention match the rows still stored.
func TestRetentionReaper_PrunesMoreThanOneBatch(t *testing.T) {
t.Parallel()
env := setupRetentionTest(t)
webhookID := createWebhook(t, env.mainDB.DB(), 30)
db, err := env.mgr.GetDB(webhookID)
require.NoError(t, err)
// Every expired event has a delivered delivery to one target and a
// failed one to the other, each with one attempt.
expired := database.ExportReapBatchSize + 1
delivered, failed := uuid.New().String(), uuid.New().String()
old := time.Now().Add(-40 * 24 * time.Hour)
events := make([]database.Event, expired)
deliveries := make([]database.Delivery, 0, 2*expired)
for i := range events {
events[i] = database.Event{
WebhookID: webhookID,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
}
events[i].ID = uuid.New().String()
events[i].CreatedAt = old
deliveries = append(deliveries,
database.Delivery{
EventID: events[i].ID,
TargetID: delivered,
Status: database.DeliveryStatusDelivered,
},
database.Delivery{
EventID: events[i].ID,
TargetID: failed,
Status: database.DeliveryStatusFailed,
},
)
}
require.NoError(t, db.CreateInBatches(events, 500).Error)
require.NoError(t, db.CreateInBatches(deliveries, 500).Error)
results := make([]database.DeliveryResult, len(deliveries))
for i := range deliveries {
results[i] = database.DeliveryResult{
DeliveryID: deliveries[i].ID, AttemptNum: 1,
}
}
require.NoError(t, db.CreateInBatches(results, 500).Error)
// One recent event, delivered to the first target.
recent := seedEventChain(t, db, webhookID, time.Now())
require.NoError(t, db.Model(&database.Delivery{}).
Where("id = ?", recent.deliveryID).
Update("target_id", delivered).Error)
// The totals storing those rows would have left.
n := int64(expired)
require.NoError(t, database.AddEventTotals(db, database.EventTotals{
Events: n + 1,
}))
require.NoError(t, database.AddTargetTotals(db, database.TargetTotals{
TargetID: delivered, Deliveries: n + 1, Delivered: n + 1,
}))
require.NoError(t, database.AddTargetTotals(db, database.TargetTotals{
TargetID: failed, Deliveries: n, Failed: n,
}))
env.reaper.ExportSweep(context.Background())
// Only the recent event's rows are left.
for _, model := range []any{
&database.Event{}, &database.Delivery{}, &database.DeliveryResult{},
} {
var count int64
require.NoError(t, db.Model(model).Count(&count).Error)
assert.Equal(t, int64(1), count, "%T rows left", model)
}
assertChainPresent(t, db, recent)
eventTotals := readEventTotals(t, db)
assert.Equal(t, database.EventTotals{
ID: eventTotals.ID, Events: n + 1, EventsRemoved: n,
}, eventTotals)
targetTotals := readTargetTotals(t, db)
assert.Equal(t, map[string]database.TargetTotals{
delivered: {
TargetID: delivered, Deliveries: n + 1, Delivered: n + 1,
DeliveriesRemoved: n,
},
failed: {
TargetID: failed, Deliveries: n, Failed: n,
DeliveriesRemoved: n, FailedRemoved: n,
},
}, targetTotals)
// A sweep with nothing left to remove changes nothing.
env.reaper.ExportSweep(context.Background())
assert.Equal(t, eventTotals, readEventTotals(t, db))
assert.Equal(t, targetTotals, readTargetTotals(t, db))
}
+15 -1
View File
@@ -35,7 +35,8 @@ var errInvalidCachedDBType = errors.New(
// WebhookDBManager manages per-webhook SQLite database files // WebhookDBManager manages per-webhook SQLite database files
// for event storage. Each webhook gets its own dedicated // for event storage. Each webhook gets its own dedicated
// database containing Events, Deliveries, and DeliveryResults. // database containing Events, Deliveries, DeliveryResults and the
// running totals of them (EventTotals, TargetTotals).
// Database connections are opened lazily and cached. // Database connections are opened lazily and cached.
type WebhookDBManager struct { type WebhookDBManager struct {
dataDir string dataDir string
@@ -295,6 +296,7 @@ func (m *WebhookDBManager) openDB(
// Run migrations for event-tier models only // Run migrations for event-tier models only
err = db.AutoMigrate( err = db.AutoMigrate(
&Event{}, &Delivery{}, &DeliveryResult{}, &Event{}, &Delivery{}, &DeliveryResult{},
&EventTotals{}, &TargetTotals{},
) )
if err != nil { if err != nil {
_ = sqlDB.Close() _ = sqlDB.Close()
@@ -305,6 +307,18 @@ func (m *WebhookDBManager) openDB(
) )
} }
// A new database gets its row of event totals, all zero. Target
// totals rows are created by the first delivery to each target.
err = db.FirstOrCreate(&EventTotals{}).Error
if err != nil {
_ = sqlDB.Close()
return nil, fmt.Errorf(
"creating event totals for webhook database %s: %w",
webhookID, err,
)
}
m.log.Info( m.log.Info(
"opened per-webhook database", "opened per-webhook database",
"webhook_id", webhookID, "webhook_id", webhookID,
+116
View File
@@ -0,0 +1,116 @@
package delivery_test
import (
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// targetTotals reads one target's totals from a webhook database, all
// zero when it has no row.
func targetTotals(
t *testing.T, db *gorm.DB, targetID string,
) database.TargetTotals {
t.Helper()
var rows []database.TargetTotals
require.NoError(t, db.Where("target_id = ?", targetID).
Find(&rows).Error)
if len(rows) == 0 {
return database.TargetTotals{TargetID: targetID}
}
return rows[0]
}
// TestUpdateDeliveryStatus_FinishTimeAndTargetTotals pins what a status
// write records for the webhook page's statistics: the time a delivery
// finished, set only when it becomes delivered or failed, and one more
// on its target's delivered or failed total.
func TestUpdateDeliveryStatus_FinishTimeAndTargetTotals(t *testing.T) {
t.Parallel()
tests := []struct {
status database.DeliveryStatus
finished bool
delivered int64
failed int64
}{
{database.DeliveryStatusRetrying, false, 0, 0},
{database.DeliveryStatusDelivered, true, 1, 0},
{database.DeliveryStatusFailed, true, 0, 1},
}
for _, tt := range tests {
t.Run(string(tt.status), func(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
e := testEngine(t, 1)
event := seedEvent(t, db, `{}`)
targetID := uuid.New().String()
d := seedDelivery(
t, db, event.ID, targetID,
database.DeliveryStatusPending,
)
before := time.Now()
require.NoError(t, e.ExportUpdateDeliveryStatus(
db, &d, tt.status,
))
var stored database.Delivery
require.NoError(t, db.First(&stored, "id = ?", d.ID).Error)
assert.Equal(t, tt.status, stored.Status)
if tt.finished {
require.NotNil(t, stored.FinishedAt)
assert.False(t, stored.FinishedAt.Before(before))
} else {
assert.Nil(t, stored.FinishedAt)
}
assert.Equal(t, database.TargetTotals{
TargetID: targetID,
Delivered: tt.delivered,
Failed: tt.failed,
}, targetTotals(t, db, targetID))
})
}
}
// TestUpdateDeliveryStatus_DeletedDeliveryIsNotCounted covers a
// delivery retention deleted while the engine still held it. Failing
// it afterwards writes no row, so it adds no failure either: retention
// has already counted what it removed.
func TestUpdateDeliveryStatus_DeletedDeliveryIsNotCounted(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
e := testEngine(t, 1)
event := seedEvent(t, db, `{}`)
targetID := uuid.New().String()
d := seedDelivery(
t, db, event.ID, targetID,
database.DeliveryStatusRetrying,
)
require.NoError(t, db.Unscoped().
Delete(&database.Delivery{}, "id = ?", d.ID).Error)
require.NoError(t, e.ExportUpdateDeliveryStatus(
db, &d, database.DeliveryStatusFailed,
))
assert.Equal(t, database.TargetTotals{TargetID: targetID},
targetTotals(t, db, targetID))
}
+33 -2
View File
@@ -1554,8 +1554,9 @@ func (e *Engine) updateDeliveryStatus(
targetType database.TargetType, targetType database.TargetType,
status database.DeliveryStatus, status database.DeliveryStatus,
) error { ) error {
err := webhookDB.Model(d). err := webhookDB.Transaction(func(tx *gorm.DB) error {
Update("status", status).Error return writeDeliveryStatus(tx, d, status)
})
if err != nil { if err != nil {
return fmt.Errorf( return fmt.Errorf(
"updating delivery %s to status %s: %w", "updating delivery %s to status %s: %w",
@@ -1574,6 +1575,36 @@ func (e *Engine) updateDeliveryStatus(
return nil return nil
} }
// writeDeliveryStatus writes a delivery's new status. A delivery that
// becomes delivered or failed also gets the time it finished, and is
// added to its target's delivered or failed total. It is counted only
// if the row was still there to update: retention may have deleted it
// while the engine was working on it.
func writeDeliveryStatus(
tx *gorm.DB,
d *database.Delivery,
status database.DeliveryStatus,
) error {
if !status.Terminal() {
return tx.Model(d).Update("status", status).Error
}
res := tx.Model(d).Updates(map[string]any{
"status": status,
"finished_at": time.Now(),
})
if res.Error != nil || res.RowsAffected == 0 {
return res.Error
}
add := database.TargetTotals{TargetID: d.TargetID, Delivered: 1}
if status == database.DeliveryStatusFailed {
add = database.TargetTotals{TargetID: d.TargetID, Failed: 1}
}
return database.AddTargetTotals(tx, add)
}
// settleStatus moves a delivery to its outcome status and reports a // settleStatus moves a delivery to its outcome status and reports a
// failed write through bookkeepingFailed, which leaves the row // failed write through bookkeepingFailed, which leaves the row
// recoverable. It exists so the target call sites read as one // recoverable. It exists so the target call sites read as one
+3
View File
@@ -57,7 +57,10 @@ func testWebhookDB(t *testing.T) *gorm.DB {
&database.Event{}, &database.Event{},
&database.Delivery{}, &database.Delivery{},
&database.DeliveryResult{}, &database.DeliveryResult{},
&database.EventTotals{},
&database.TargetTotals{},
)) ))
require.NoError(t, db.Create(&database.EventTotals{}).Error)
return db return db
} }
+10
View File
@@ -150,6 +150,16 @@ func (e *Engine) ExportDeliverSlack(
) )
} }
// ExportUpdateDeliveryStatus exposes updateDeliveryStatus. It passes no
// target type, so no metric moves.
func (e *Engine) ExportUpdateDeliveryStatus(
webhookDB *gorm.DB,
d *database.Delivery,
status database.DeliveryStatus,
) error {
return e.updateDeliveryStatus(webhookDB, d, "", status)
}
// ExportProcessNewTask exposes processNewTask. // ExportProcessNewTask exposes processNewTask.
func (e *Engine) ExportProcessNewTask( func (e *Engine) ExportProcessNewTask(
ctx context.Context, task *Task, ctx context.Context, task *Task,
+30
View File
@@ -453,3 +453,33 @@ func TestLogin_SuccessCreatesSession(t *testing.T) {
"the issued cookie must carry an authenticated session", "the issued cookie must carry an authenticated session",
) )
} }
// TestLogin_UsernameAtLimitCanLogIn shows that a username of exactly
// database.MaxUsernameBytes still fits in the session cookie. Past
// what the cookie can carry, a correct login answers 500.
func TestLogin_UsernameAtLimitCanLogIn(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
db *database.Database
)
app := newTestApp(t, &h, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
username := strings.Repeat("a", database.MaxUsernameBytes)
hash, err := database.HashPassword(operatorPassword)
require.NoError(t, err)
require.NoError(t, db.DB().Create(&database.User{
Username: username,
Password: hash,
}).Error)
w := submitLogin(h, sharedProxyPeer, username, operatorPassword)
assert.Equal(t, http.StatusSeeOther, w.Code)
}
+13 -3
View File
@@ -299,8 +299,9 @@ func countInFlightDeliveries(
return count, err return count, err
} }
// createReplayDelivery writes the new pending delivery row and returns // createReplayDelivery writes the new pending delivery row, adds it to
// the task that carries it to the delivery engine. // its target's totals in the same transaction, and returns the task
// that carries it to the delivery engine.
// //
// The row is written with associations omitted, and neither Event nor // The row is written with associations omitted, and neither Event nor
// Target is populated on it: GORM's SaveBeforeAssociations would // Target is populated on it: GORM's SaveBeforeAssociations would
@@ -319,7 +320,16 @@ func createReplayDelivery(
Status: database.DeliveryStatusPending, Status: database.DeliveryStatusPending,
} }
err := webhookDB.Omit(clause.Associations).Create(dlv).Error err := webhookDB.Transaction(func(tx *gorm.DB) error {
err := tx.Omit(clause.Associations).Create(dlv).Error
if err != nil {
return err
}
return database.AddTargetTotals(tx, database.TargetTotals{
TargetID: dlv.TargetID, Deliveries: 1,
})
})
if err != nil { if err != nil {
return delivery.Task{}, err return delivery.Task{}, err
} }
+25
View File
@@ -4,7 +4,9 @@ import (
"html/template" "html/template"
"log/slog" "log/slog"
"net/http" "net/http"
"time"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database" "sneak.berlin/go/webhooker/internal/database"
) )
@@ -69,6 +71,29 @@ func (s *Handlers) LoadEventLogViewsForTest(
return views return views
} }
// WebhookStatsForTest returns the figures the statistics pane on a
// webhook's page shows, from the webhook's entrypoints and targets
// loaded as that page loads them.
func (s *Handlers) WebhookStatsForTest(webhookID string) *WebhookStats {
var entrypoints []database.Entrypoint
s.db.DB().Where("webhook_id = ?", webhookID).Find(&entrypoints)
var targets []database.Target
s.db.DB().Where("webhook_id = ?", webhookID).Find(&targets)
return s.loadWebhookStats(webhookID, entrypoints, targets)
}
// FinishedByTargetForTest exposes finishedByTarget for use in the
// handlers_test package.
func FinishedByTargetForTest(
webhookDB *gorm.DB, since time.Time,
) ([]TargetFinished, error) {
return finishedByTarget(webhookDB, since)
}
// AddTemplateForTest registers a template under a page name so that // AddTemplateForTest registers a template under a page name so that
// the handlers_test package can drive the render path with a // the handlers_test package can drive the render path with a
// template of its own. // template of its own.
+16 -12
View File
@@ -91,18 +91,22 @@ type Handlers struct {
// parsePageTemplate parses a page-specific template set from the // parsePageTemplate parses a page-specific template set from the
// embedded FS. Each page template is combined with the shared // embedded FS. Each page template is combined with the shared
// base, htmlheader, and navbar templates. The page file must be // base, htmlheader, and navbar templates, and with any further files
// listed first so that its root action ({{template "base" .}}) // the page includes. The page file must be listed first so that its
// becomes the template set's entry point. // root action ({{template "base" .}}) becomes the template set's entry
func parsePageTemplate(pageFile string) *template.Template { // point.
func parsePageTemplate(
pageFile string, included ...string,
) *template.Template {
files := append([]string{
pageFile,
"base.html",
"htmlheader.html",
"navbar.html",
}, included...)
return template.Must( return template.Must(
template.ParseFS( template.ParseFS(templates.Templates, files...),
templates.Templates,
pageFile,
"base.html",
"htmlheader.html",
"navbar.html",
),
) )
} }
@@ -131,7 +135,7 @@ func New(
"profile.html": parsePageTemplate("profile.html"), "profile.html": parsePageTemplate("profile.html"),
"sources_list.html": parsePageTemplate("sources_list.html"), "sources_list.html": parsePageTemplate("sources_list.html"),
"sources_new.html": parsePageTemplate("sources_new.html"), "sources_new.html": parsePageTemplate("sources_new.html"),
"source_detail.html": parsePageTemplate("source_detail.html"), "source_detail.html": parsePageTemplate("source_detail.html", "webhook_stats.html"),
"source_edit.html": parsePageTemplate("source_edit.html"), "source_edit.html": parsePageTemplate("source_edit.html"),
"source_logs.html": parsePageTemplate("source_logs.html"), "source_logs.html": parsePageTemplate("source_logs.html"),
"target_edit.html": parsePageTemplate("target_edit.html"), "target_edit.html": parsePageTemplate("target_edit.html"),
+3 -5
View File
@@ -339,11 +339,9 @@ const storedUserPassword = "correct-horse-battery-staple"
// storedFillBytes is the raw length of the client-chosen value in // storedFillBytes is the raw length of the client-chosen value in
// those accounts' usernames. It is well past the 512-byte field // those accounts' usernames. It is well past the 512-byte field
// budget, so the line is still truncated, but short enough that the // budget, so the line is still truncated, but short enough that the
// session cookie a successful login writes stays inside // whole username, markers and fill name included, stays within
// securecookie's 4 KB limit: the cookie is written BEFORE the // database.MaxUsernameBytes.
// "user logged in" line, so an 8 KB username answers 500 and never const storedFillBytes = 960
// reaches it.
const storedFillBytes = 1024
// storedFill builds a username fill of storedFillBytes raw bytes out // storedFill builds a username fill of storedFillBytes raw bytes out
// of repetitions of ch, with both markers at its far end. // of repetitions of ch, with both markers at its far end.
+1
View File
@@ -450,6 +450,7 @@ func (h *Handlers) renderSourceDetail(
"Targets": delivery.NewTargetViews(targets), "Targets": delivery.NewTargetViews(targets),
"Events": events, "Events": events,
"BaseURL": baseURL, "BaseURL": baseURL,
"Stats": h.loadWebhookStats(webhook.ID, entrypoints, targets),
} }
h.renderTemplate(w, r, "source_detail.html", data) h.renderTemplate(w, r, "source_detail.html", data)
+23 -7
View File
@@ -252,11 +252,12 @@ func requestEventSource(
} }
} }
// createAndFanOut writes the event and one pending delivery per target // createAndFanOut writes the event and one pending delivery per target,
// in a single transaction, then hands the tasks to the delivery // and adds them to the webhook's running totals, in a single
// engine. It is the only path by which an event and its deliveries are // transaction, then hands the tasks to the delivery engine. It is the
// created, so a resubmitted event is retried, SSRF-guarded and // only path by which an event and its deliveries are created, so a
// circuit-broken exactly as a received one is. // resubmitted event is retried, SSRF-guarded and circuit-broken
// exactly as a received one is.
// //
// The tasks are returned as well as queued, so a caller can report how // The tasks are returned as well as queued, so a caller can report how
// many targets the event went to. // many targets the event went to.
@@ -296,6 +297,13 @@ func (h *Handlers) createAndFanOut(
return nil, nil, err return nil, nil, err
} }
err = database.AddEventTotals(tx, database.EventTotals{Events: 1})
if err != nil {
tx.Rollback()
return nil, nil, err
}
err = tx.Commit().Error err = tx.Commit().Error
if err != nil { if err != nil {
return nil, nil, fmt.Errorf( return nil, nil, fmt.Errorf(
@@ -354,8 +362,9 @@ func (h *Handlers) finishWebhookResponse(
} }
// buildDeliveryTasks creates one pending delivery per target in the // buildDeliveryTasks creates one pending delivery per target in the
// transaction and returns the tasks for the delivery engine. The // transaction, adds each to its target's totals, and returns the tasks
// caller owns the transaction and rolls it back on error. // for the delivery engine. The caller owns the transaction and rolls
// it back on error.
func buildDeliveryTasks( func buildDeliveryTasks(
tx *gorm.DB, tx *gorm.DB,
event *database.Event, event *database.Event,
@@ -379,6 +388,13 @@ func buildDeliveryTasks(
) )
} }
err = database.AddTargetTotals(tx, database.TargetTotals{
TargetID: targets[i].ID, Deliveries: 1,
})
if err != nil {
return nil, err
}
tasks = append(tasks, delivery.Task{ tasks = append(tasks, delivery.Task{
DeliveryID: dlv.ID, DeliveryID: dlv.ID,
EventID: event.ID, EventID: event.ID,
+271
View File
@@ -0,0 +1,271 @@
package handlers
import (
"fmt"
"time"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// The spans of the two recent windows the statistics pane reports on:
// the last 10 minutes and the last 24 hours.
const (
shortWindow = 10 * time.Minute
longWindow = 24 * time.Hour
)
// percent turns a fraction into a percentage.
const percent = 100
// WebhookStats holds the figures in the statistics pane at the top of
// the webhook page.
type WebhookStats struct {
Entrypoints int
ActiveEntrypoints int
Targets int
ActiveTargets int
// Lifetime counts every event, delivery and failure the webhook
// has had, and WithinRetention those still stored.
Lifetime Counts
WithinRetention Counts
// InProgress counts the deliveries still pending or retrying.
InProgress int64
// LastEventAt is when the newest stored event arrived, or nil when
// none is stored.
LastEventAt *time.Time
Last10Minutes RecentWindow
Last24Hours RecentWindow
}
// Counts holds a number of events, of deliveries and of failed
// deliveries.
type Counts struct {
Events int64
Deliveries int64
Failures int64
}
// RecentWindow holds what happened in one recent window: the events
// received in it, and the deliveries that became delivered or failed in
// it.
type RecentWindow struct {
Events int64
Delivered int64
Failed int64
}
// TargetFinished is how many of one target's deliveries became
// delivered, and how many failed, in a recent window.
type TargetFinished struct {
TargetID string
Delivered int64
Failed int64
}
// FailurePercent is the share of the deliveries finished in the window
// that failed, or a dash when none finished. Deliveries still pending
// or retrying are not counted either way.
func (w RecentWindow) FailurePercent() string {
finished := w.Delivered + w.Failed
if finished == 0 {
return "—"
}
return fmt.Sprintf(
"%.1f%%", percent*float64(w.Failed)/float64(finished),
)
}
// loadWebhookStats gathers the figures for the statistics pane from the
// webhook's entrypoints and targets, as the page has already loaded
// them, and from its event database. It returns nil, and logs why, when
// the event database cannot be read.
func (h *Handlers) loadWebhookStats(
webhookID string,
entrypoints []database.Entrypoint,
targets []database.Target,
) *WebhookStats {
stats := &WebhookStats{
Entrypoints: len(entrypoints),
Targets: len(targets),
}
for i := range entrypoints {
if entrypoints[i].Active {
stats.ActiveEntrypoints++
}
}
for i := range targets {
if targets[i].Active {
stats.ActiveTargets++
}
}
// Opening an event database that does not exist would create it,
// and it would hold nothing to count.
if !h.dbMgr.DBExists(webhookID) {
return stats
}
webhookDB, err := h.dbMgr.GetDB(webhookID)
if err == nil {
err = readEventStats(webhookDB, time.Now(), stats)
}
if err != nil {
h.log.Error(
"failed to read webhook statistics",
"webhook_id", webhookID,
"error", err,
)
return nil
}
return stats
}
// readEventStats fills in the figures that come from the webhook's
// event database. None of them reads every stored row: the totals are
// one row for the events and one per target for the deliveries, and
// every other figure is read from an index, over only the rows it
// counts.
func readEventStats(
db *gorm.DB, now time.Time, stats *WebhookStats,
) error {
err := readTotals(db, stats)
if err != nil {
return err
}
err = db.Model(&database.Delivery{}).
Where("status IN ?", []database.DeliveryStatus{
database.DeliveryStatusPending,
database.DeliveryStatusRetrying,
}).
Count(&stats.InProgress).Error
if err != nil {
return fmt.Errorf("counting deliveries in progress: %w", err)
}
var newest []time.Time
err = db.Model(&database.Event{}).
Order("created_at DESC").
Limit(1).
Pluck("created_at", &newest).Error
if err != nil {
return fmt.Errorf("reading newest event time: %w", err)
}
if len(newest) > 0 {
stats.LastEventAt = &newest[0]
}
stats.Last10Minutes, err = readRecentWindow(
db, now.Add(-shortWindow),
)
if err != nil {
return err
}
stats.Last24Hours, err = readRecentWindow(
db, now.Add(-longWindow),
)
return err
}
// readTotals fills in the lifetime and within-retention figures from
// the running totals: the events' row, and the targets' rows summed.
func readTotals(db *gorm.DB, stats *WebhookStats) error {
var events database.EventTotals
err := db.Take(&events).Error
if err != nil {
return fmt.Errorf("reading event totals: %w", err)
}
var targets []database.TargetTotals
err = db.Find(&targets).Error
if err != nil {
return fmt.Errorf("reading target totals: %w", err)
}
stats.Lifetime.Events = events.Events
stats.WithinRetention.Events = events.Events - events.EventsRemoved
for _, t := range targets {
stats.Lifetime.Deliveries += t.Deliveries
stats.Lifetime.Failures += t.Failed
stats.WithinRetention.Deliveries += t.Deliveries - t.DeliveriesRemoved
stats.WithinRetention.Failures += t.Failed - t.FailedRemoved
}
return nil
}
// readRecentWindow counts the events received, and the deliveries that
// became delivered or failed, since the given time.
func readRecentWindow(
db *gorm.DB, since time.Time,
) (RecentWindow, error) {
var w RecentWindow
err := db.Model(&database.Event{}).
Where("created_at >= ?", since).
Count(&w.Events).Error
if err != nil {
return w, fmt.Errorf("counting recent events: %w", err)
}
byTarget, err := finishedByTarget(db, since)
if err != nil {
return w, err
}
for _, f := range byTarget {
w.Delivered += f.Delivered
w.Failed += f.Failed
}
return w, nil
}
// finishedByTarget counts, for each target, the deliveries that became
// delivered and those that failed since the given time, in one query
// over just that window of the deliveries' status index. A target with
// neither is left out.
func finishedByTarget(
db *gorm.DB, since time.Time,
) ([]TargetFinished, error) {
var byTarget []TargetFinished
err := db.Model(&database.Delivery{}).
Select("target_id, "+
"count(CASE WHEN status = ? THEN 1 END) AS delivered, "+
"count(CASE WHEN status = ? THEN 1 END) AS failed",
database.DeliveryStatusDelivered,
database.DeliveryStatusFailed).
Where("status IN ? AND finished_at >= ?",
[]database.DeliveryStatus{
database.DeliveryStatusDelivered,
database.DeliveryStatusFailed,
}, since).
Group("target_id").
Find(&byTarget).Error
if err != nil {
return nil, fmt.Errorf(
"counting deliveries finished by target: %w", err,
)
}
return byTarget, nil
}
+432
View File
@@ -0,0 +1,432 @@
package handlers_test
import (
"net/http"
"strings"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/fx/fxtest"
"gorm.io/gorm"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/logger"
"sneak.berlin/go/webhooker/internal/session"
)
// statsEntrypoint adds an entrypoint to a webhook and returns its path.
func statsEntrypoint(
t *testing.T, db *database.Database, webhookID string, active bool,
) string {
t.Helper()
ep := &database.Entrypoint{
WebhookID: webhookID,
Path: uuid.New().String(),
}
require.NoError(t, db.DB().Omit(clause.Associations).Create(ep).Error)
require.NoError(t, db.DB().Model(ep).Update("active", active).Error)
return ep.Path
}
// statsDelivery returns an event's delivery to a target.
func statsDelivery(
t *testing.T, webhookDB *gorm.DB, eventID, targetID string,
) database.Delivery {
t.Helper()
var d database.Delivery
require.NoError(t, webhookDB.Where(
"event_id = ? AND target_id = ?", eventID, targetID,
).First(&d).Error)
return d
}
// statsFinish settles a delivery as the delivery engine does: its
// final status and the time it finished, and one more on its target's
// delivered or failed total, in one transaction.
func statsFinish(
t *testing.T,
webhookDB *gorm.DB,
d database.Delivery,
status database.DeliveryStatus,
at time.Time,
) {
t.Helper()
add := database.TargetTotals{TargetID: d.TargetID, Delivered: 1}
if status == database.DeliveryStatusFailed {
add = database.TargetTotals{TargetID: d.TargetID, Failed: 1}
}
require.NoError(t, webhookDB.Transaction(func(tx *gorm.DB) error {
err := tx.Model(&database.Delivery{}).
Where("id = ?", d.ID).
Updates(map[string]any{"status": status, "finished_at": at}).
Error
if err != nil {
return err
}
return database.AddTargetTotals(tx, add)
}))
}
// statsAge moves an event's arrival back to the given time.
func statsAge(
t *testing.T, webhookDB *gorm.DB, eventID string, at time.Time,
) {
t.Helper()
require.NoError(t, webhookDB.Model(&database.Event{}).
Where("id = ?", eventID).
Update("created_at", at).Error)
}
// statsTargetTotals reads a webhook database's target totals, keyed by
// target.
func statsTargetTotals(
t *testing.T, webhookDB *gorm.DB,
) map[string]database.TargetTotals {
t.Helper()
var rows []database.TargetTotals
require.NoError(t, webhookDB.Find(&rows).Error)
byTarget := make(map[string]database.TargetTotals, len(rows))
for _, row := range rows {
byTarget[row.TargetID] = row
}
return byTarget
}
// statsHistory is the webhook seedStatsHistory builds: its event
// database, its newest event, and its two active targets.
type statsHistory struct {
webhook *database.Webhook
webhookDB *gorm.DB
newest database.Event
first, second string
}
// seedStatsHistory builds the webhook the statistics test checks: one
// day of retention, two entrypoints (one inactive) and three targets
// (one inactive). Three events arrive through the receiver, and so
// each has a delivery to the two active targets. The oldest event is
// past retention, the middle one six hours old, the newest just in.
// Their deliveries are settled as the delivery engine would, and a
// replay adds a pending delivery to the oldest event.
func seedStatsHistory(
t *testing.T,
h *handlers.Handlers,
sess *session.Session,
db *database.Database,
dbMgr *database.WebhookDBManager,
) statsHistory {
t.Helper()
wh := &database.Webhook{
UserID: deleteTestUserID, Name: "stats", RetentionDays: 1,
}
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
path := statsEntrypoint(t, db, wh.ID, true)
statsEntrypoint(t, db, wh.ID, false)
first := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
`{"url":"`+replayTargetURL+`"}`,
)
second := seedTarget(t, db, wh.ID, database.TargetTypeLog)
inactive := seedTarget(t, db, wh.ID, database.TargetTypeLog)
require.NoError(t, db.DB().Model(inactive).
Update("active", false).Error)
router := receiverRouter(h)
for range 3 {
require.Equal(t, http.StatusOK, postReceiver(t, router, path))
}
webhookDB, err := dbMgr.GetDB(wh.ID)
require.NoError(t, err)
events := listEvents(t, webhookDB)
require.Len(t, events, 3)
oldest, middle, newest := events[0], events[1], events[2]
now := time.Now()
statsAge(t, webhookDB, oldest.ID, now.Add(-50*time.Hour))
statsAge(t, webhookDB, middle.ID, now.Add(-6*time.Hour))
oldestFailure := statsDelivery(t, webhookDB, oldest.ID, first.ID)
statsFinish(t, webhookDB, oldestFailure,
database.DeliveryStatusFailed, now.Add(-49*time.Hour))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, oldest.ID, second.ID),
database.DeliveryStatusDelivered, now.Add(-49*time.Hour))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, middle.ID, first.ID),
database.DeliveryStatusFailed, now.Add(-5*time.Hour))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, middle.ID, second.ID),
database.DeliveryStatusFailed, now.Add(-time.Minute))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, newest.ID, first.ID),
database.DeliveryStatusDelivered, now.Add(-2*time.Minute))
require.Equal(t, http.StatusSeeOther,
postReplay(t, h, sess, wh.ID, oldestFailure.ID).Code)
return statsHistory{
webhook: wh,
webhookDB: webhookDB,
newest: newest,
first: first.ID,
second: second.ID,
}
}
// statsPrune runs the real retention reaper until it has removed one
// event from the webhook's database, then stops it.
func statsPrune(
t *testing.T,
db *database.Database,
dbMgr *database.WebhookDBManager,
log *logger.Logger,
webhookDB *gorm.DB,
) {
t.Helper()
lc := fxtest.NewLifecycle(t)
database.NewRetentionReaper(lc, database.RetentionReaperParams{
Config: &config.Config{
RetentionSweepInterval: 10 * time.Millisecond,
},
Database: db,
DBManager: dbMgr,
Logger: log,
})
lc.RequireStart()
require.Eventually(t, func() bool {
var totals database.EventTotals
err := webhookDB.Take(&totals).Error
return err == nil && totals.EventsRemoved == 1
}, 10*time.Second, 10*time.Millisecond)
lc.RequireStop()
}
// statsPane returns the statistics pane from a rendered webhook page:
// everything from its heading to the next heading on the page.
func statsPane(t *testing.T, page string) string {
t.Helper()
_, pane, found := strings.Cut(page, ">Statistics</h2>")
require.True(t, found, "the page has no statistics pane")
pane, _, _ = strings.Cut(pane, "<h2")
return pane
}
// TestWebhookStats_EveryFigureAcrossRetentionPrune checks every figure
// the statistics pane shows for the history seedStatsHistory builds,
// and each target's totals and recent figures, before and after the
// real retention reaper removes the oldest event.
func TestWebhookStats_EveryFigureAcrossRetentionPrune(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
log *logger.Logger
)
app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
app.RequireStart()
t.Cleanup(app.RequireStop)
hist := seedStatsHistory(t, h, sess, db, dbMgr)
first, second := hist.first, hist.second
stats := h.WebhookStatsForTest(hist.webhook.ID)
require.NotNil(t, stats)
assert.Equal(t, 2, stats.Entrypoints)
assert.Equal(t, 1, stats.ActiveEntrypoints)
assert.Equal(t, 3, stats.Targets)
assert.Equal(t, 2, stats.ActiveTargets)
assert.Equal(t, handlers.Counts{Events: 3, Deliveries: 7, Failures: 3},
stats.Lifetime)
assert.Equal(t, stats.Lifetime, stats.WithinRetention)
assert.Equal(t, int64(2), stats.InProgress)
require.NotNil(t, stats.LastEventAt)
assert.True(t, hist.newest.CreatedAt.Equal(*stats.LastEventAt))
assert.Equal(t, handlers.RecentWindow{
Events: 1, Delivered: 1, Failed: 1,
}, stats.Last10Minutes)
assert.Equal(t, handlers.RecentWindow{
Events: 2, Delivered: 1, Failed: 2,
}, stats.Last24Hours)
assert.Equal(t, "50.0%", stats.Last10Minutes.FailurePercent())
assert.Equal(t, "66.7%", stats.Last24Hours.FailurePercent())
// The first target has three deliveries and the replay, the second
// three; the inactive target has none and so no row.
assert.Equal(t, map[string]database.TargetTotals{
first: {TargetID: first, Deliveries: 4, Delivered: 1, Failed: 2},
second: {
TargetID: second, Deliveries: 3, Delivered: 1, Failed: 1,
},
}, statsTargetTotals(t, hist.webhookDB))
lastDay, err := handlers.FinishedByTargetForTest(
hist.webhookDB, time.Now().Add(-24*time.Hour),
)
require.NoError(t, err)
assert.ElementsMatch(t, []handlers.TargetFinished{
{TargetID: first, Delivered: 1, Failed: 1},
{TargetID: second, Failed: 1},
}, lastDay)
// Retention removes the oldest event with its three deliveries:
// the first target's failed one and the pending replay, and the
// second target's delivered one.
statsPrune(t, db, dbMgr, log, hist.webhookDB)
after := h.WebhookStatsForTest(hist.webhook.ID)
require.NotNil(t, after)
assert.Equal(t, stats.Lifetime, after.Lifetime)
assert.Equal(t, handlers.Counts{Events: 2, Deliveries: 4, Failures: 2},
after.WithinRetention)
assert.Equal(t, int64(1), after.InProgress)
assert.Equal(t, stats.LastEventAt, after.LastEventAt)
assert.Equal(t, stats.Last10Minutes, after.Last10Minutes)
assert.Equal(t, stats.Last24Hours, after.Last24Hours)
assert.Equal(t, map[string]database.TargetTotals{
first: {
TargetID: first, Deliveries: 4, Delivered: 1, Failed: 2,
DeliveriesRemoved: 2, FailedRemoved: 1,
},
second: {
TargetID: second, Deliveries: 3, Delivered: 1, Failed: 1,
DeliveriesRemoved: 1,
},
}, statsTargetTotals(t, hist.webhookDB))
pane := statsPane(t, renderSourceDetailPage(t, h, sess, hist.webhook.ID))
assert.Contains(t, pane, "Within retention")
assert.Contains(t, pane, "50.0%")
assert.Contains(t, pane, "66.7%")
}
// TestWebhookStats_PaneShowsRetentionPeriod checks that the statistics
// pane itself, not only the line at the foot of the page, shows the
// webhook's retention period, for a finite one and for forever.
func TestWebhookStats_PaneShowsRetentionPeriod(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
)
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
tests := []struct {
retentionDays int
want string
}{
{30, "30 days"},
{database.RetentionForeverDays, "forever"},
}
for _, tt := range tests {
wh := &database.Webhook{
UserID: deleteTestUserID,
Name: "retention",
RetentionDays: tt.retentionDays,
}
require.NoError(t,
db.DB().Omit(clause.Associations).Create(wh).Error)
pane := statsPane(t, renderSourceDetailPage(t, h, sess, wh.ID))
assert.Contains(t, pane, "Retention", tt.want)
assert.Contains(t, pane, tt.want)
}
}
// TestWebhookStats_WebhookWithNoEvents covers a webhook whose event
// database has never been opened: every count is zero, the
// percentages are a dash, and showing the page does not create the
// database.
func TestWebhookStats_WebhookWithNoEvents(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
assert.Equal(t, &handlers.WebhookStats{}, h.WebhookStatsForTest(wh.ID))
assert.Equal(t, "—", handlers.RecentWindow{}.FailurePercent())
statsPane(t, renderSourceDetailPage(t, h, sess, wh.ID))
assert.False(t, dbMgr.DBExists(wh.ID))
}
// TestRecentWindow_FailurePercent pins the percentage: failed
// deliveries out of all that finished in the window.
func TestRecentWindow_FailurePercent(t *testing.T) {
t.Parallel()
tests := []struct {
window handlers.RecentWindow
want string
}{
{handlers.RecentWindow{}, "—"},
{handlers.RecentWindow{Events: 4}, "—"},
{handlers.RecentWindow{Delivered: 3, Failed: 1}, "25.0%"},
{handlers.RecentWindow{Failed: 2}, "100.0%"},
{handlers.RecentWindow{Delivered: 2}, "0.0%"},
}
for _, tt := range tests {
assert.Equal(t, tt.want, tt.window.FailurePercent(), tt.window)
}
}
+17 -3
View File
@@ -92,11 +92,25 @@ func (s *Server) setupGlobalMiddleware() {
func (s *Server) setupRoutes() { func (s *Server) setupRoutes() {
s.router.Get("/", s.h.HandleIndex()) s.router.Get("/", s.h.HandleIndex())
s.router.Mount( // Static assets answer GET and HEAD only. chi's default 405
"/s", // carries no Allow header, so this group supplies its own.
http.StripPrefix("/s", http.FileServer(http.FS(static.Static))), staticFiles := http.StripPrefix(
"/s", http.FileServer(http.FS(static.Static)),
) )
s.router.Route("/s", func(r chi.Router) {
r.MethodNotAllowed(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Allow", "GET, HEAD")
http.Error(
w,
"Method Not Allowed",
http.StatusMethodNotAllowed,
)
})
r.Method(http.MethodGet, "/*", staticFiles)
r.Method(http.MethodHead, "/*", staticFiles)
})
s.router.Route("/api/v1", func(_ chi.Router) { s.router.Route("/api/v1", func(_ chi.Router) {
// API routes will be added here. // API routes will be added here.
}) })
+108 -19
View File
@@ -7,6 +7,7 @@ import (
"net/http/httptest" "net/http/httptest"
"net/url" "net/url"
"regexp" "regexp"
"slices"
"strconv" "strconv"
"strings" "strings"
"testing" "testing"
@@ -220,9 +221,21 @@ func (e *testEnv) csrfFrom(
// out of the markup has to be unescaped before it is submitted. // out of the markup has to be unescaped before it is submitted.
token := html.UnescapeString(match[1]) token := html.UnescapeString(match[1])
combined := make([]*http.Cookie, 0, len(cookies)) // A cookie the page sets replaces the one of the same name, as in
combined = append(combined, cookies...) // a browser. Sent both, the server would read the first, older one.
combined = append(combined, w.Result().Cookies()...) set := w.Result().Cookies()
combined := make([]*http.Cookie, 0, len(cookies)+len(set))
for _, c := range cookies {
replaced := slices.ContainsFunc(set, func(n *http.Cookie) bool {
return n.Name == c.Name
})
if !replaced {
combined = append(combined, c)
}
}
combined = append(combined, set...)
return token, combined return token, combined
} }
@@ -396,13 +409,15 @@ func (e *testEnv) storedHash(t *testing.T, username string) string {
// --- /s static group --- // --- /s static group ---
// TestStaticServesEveryMethod pins what the static mount actually // TestStaticServesOnlyGetAndHead pins the methods the static group
// answers. chi's Mount registers the handler for all methods and // answers: GET and HEAD are served the asset, and the other methods
// http.FileServer only special-cases HEAD (by suppressing the body), // chi routes (POST, PUT, DELETE and the rest) are refused with 405
// so a POST or a DELETE to an asset is served the file rather than // and an Allow header naming those two. A method chi does not route,
// refused. The README documents this; the test is what keeps the two // such as PROPFIND, is refused with 405 by the top-level router
// from drifting. // before it reaches the static group, so it gets no Allow header.
func TestStaticServesEveryMethod(t *testing.T) { // The README documents this; the test is what keeps the two from
// drifting.
func TestStaticServesOnlyGetAndHead(t *testing.T) {
t.Parallel() t.Parallel()
env := newTestEnv(t) env := newTestEnv(t)
@@ -417,6 +432,7 @@ func TestStaticServesEveryMethod(t *testing.T) {
http.MethodPost, http.MethodPost,
http.MethodPut, http.MethodPut,
http.MethodDelete, http.MethodDelete,
"PROPFIND",
} { } {
t.Run(method, func(t *testing.T) { t.Run(method, func(t *testing.T) {
t.Parallel() t.Parallel()
@@ -428,18 +444,38 @@ func TestStaticServesEveryMethod(t *testing.T) {
w := httptest.NewRecorder() w := httptest.NewRecorder()
env.router.ServeHTTP(w, req) env.router.ServeHTTP(w, req)
assert.Equal(t, http.StatusOK, w.Code, switch method {
"static mount answers every method") case http.MethodGet:
assert.Equal(t, http.StatusOK, w.Code)
if method == http.MethodHead { assert.Equal(t, body, w.Body.Bytes(),
"the asset itself is returned")
case http.MethodHead:
assert.Equal(t, http.StatusOK, w.Code)
assert.Empty(t, w.Body.Bytes(), assert.Empty(t, w.Body.Bytes(),
"HEAD must not carry a body") "HEAD must not carry a body")
case "PROPFIND":
return assert.Equal(
t, http.StatusMethodNotAllowed, w.Code,
)
assert.Empty(t, w.Header().Get("Allow"),
"chi refuses a method it does not route "+
"before the static group runs")
assert.NotContains(
t, w.Body.String(), string(body),
"a refused method must not get the asset",
)
default:
assert.Equal(
t, http.StatusMethodNotAllowed, w.Code,
)
assert.Equal(
t, "GET, HEAD", w.Header().Get("Allow"),
)
assert.NotContains(
t, w.Body.String(), string(body),
"a refused method must not get the asset",
)
} }
assert.Equal(t, body, w.Body.Bytes(),
"the asset itself is returned")
}) })
} }
} }
@@ -591,6 +627,59 @@ func TestPagesLogin_CorrectPasswordSurvivesASpentBudget(
) )
} }
// TestPagesLogin_CookiesFromAnEarlierDatabase is
// https://git.eeqj.de/sneak/webhooker/issues/359. A new database
// brings a new session key, and the operator's browser still holds
// the session and CSRF cookies signed with the old one. Logging in
// must work as from a fresh browser and leave cookies the new key
// accepts.
func TestPagesLogin_CookiesFromAnEarlierDatabase(t *testing.T) {
t.Parallel()
const (
username = "operator"
password = "correct-horse-battery-staple"
)
earlier := newTestEnv(t)
earlierID, _ := earlier.seedUser(t, username, password)
_, stale := earlier.csrfFrom(t, "/pages/login", nil)
stale = append(stale, earlier.authCookies(t, earlierID, username)...)
env := newTestEnv(t)
env.seedUser(t, username, password)
token, cookies := env.csrfFrom(t, "/pages/login", stale)
form := url.Values{}
form.Set("csrf_token", token)
form.Set("username", username)
form.Set("password", password)
w := env.post("/pages/login", form, cookies)
require.Equal(
t, http.StatusSeeOther, w.Code,
"a session cookie from another key must not fail the login",
)
// The response deletes the old session cookie and then sets the
// new one; a browser keeps the last.
var fresh *http.Cookie
for _, c := range w.Result().Cookies() {
if c.Name == session.SessionName {
fresh = c
}
}
require.NotNil(t, fresh, "login must set a session cookie")
assert.Equal(
t, "/sources",
env.get("/", []*http.Cookie{fresh}).Header().Get("Location"),
"the new session cookie must authenticate",
)
}
// --- /user/{username} group --- // --- /user/{username} group ---
// TestPasswordChange_OversizeBody_RejectedAndPasswordUnchanged // TestPasswordChange_OversizeBody_RejectedAndPasswordUnchanged
+3 -3
View File
@@ -13,9 +13,9 @@ import (
// TestBaseTemplateScriptsAreServed walks every /s/ script the base // TestBaseTemplateScriptsAreServed walks every /s/ script the base
// template loads on each page and fetches it through the real router. // template loads on each page and fetches it through the real router.
// Alpine.js is fetched at build time rather than committed, so nothing // Alpine.js is extracted from its tarball in 3p/ at build time, so the
// in the repo guarantees it is present: this is the check that the page // file is not in the tree: this is the check that the page still gets
// still gets the JavaScript it asks for. // the JavaScript it asks for.
func TestBaseTemplateScriptsAreServed(t *testing.T) { func TestBaseTemplateScriptsAreServed(t *testing.T) {
t.Parallel() t.Parallel()
+8 -7
View File
@@ -19,8 +19,8 @@ import (
) )
// The tests below exercise the securecookie codecs underneath the // The tests below exercise the securecookie codecs underneath the
// store and nothing else: Session.Get only decodes, so no server-side // store and nothing else: they decode through the store itself, so no
// expiry check takes part in the result. They exist because // server-side expiry check takes part in the result. They exist because
// NewCookieStore gives its codecs a 30-day max age that assigning // NewCookieStore gives its codecs a 30-day max age that assigning
// store.Options does not override, which would let the codec accept a // store.Options does not override, which would let the codec accept a
// cookie weeks past the cap the cookie attribute advertises. // cookie weeks past the cap the cookie attribute advertises.
@@ -75,10 +75,11 @@ func restamp(
return base64.URLEncoding.EncodeToString(payload) return base64.URLEncoding.EncodeToString(payload)
} }
// decodeCookie feeds value back through the store's decode path. // decodeCookie feeds value back through the store's decode path. It
// asks the store rather than Session.Get, which treats a cookie that
// does not decode as absent and so hides the codec's reason.
func decodeCookie( func decodeCookie(
t *testing.T, t *testing.T,
s *session.Session,
value string, value string,
) (*sessions.Session, error) { ) (*sessions.Session, error) {
t.Helper() t.Helper()
@@ -94,7 +95,7 @@ func decodeCookie(
SameSite: http.SameSiteLaxMode, SameSite: http.SameSiteLaxMode,
}) })
sess, err := s.Get(req) sess, err := session.NewStore(testKey()).Get(req, session.SessionName)
require.NotNil(t, sess) require.NotNil(t, sess)
return sess, err return sess, err
@@ -105,7 +106,7 @@ func TestCodec_AcceptsCookieInsideAbsoluteCap(t *testing.T) {
s := testSession(t) s := testSession(t)
sess, err := decodeCookie(t, s, restamp( sess, err := decodeCookie(t, restamp(
t, t,
issuedCookie(t, s), issuedCookie(t, s),
time.Now().Add(-(testAbsoluteMaxAge-time.Hour)), time.Now().Add(-(testAbsoluteMaxAge-time.Hour)),
@@ -126,7 +127,7 @@ func TestCodec_RejectsCookiePastAbsoluteCap(t *testing.T) {
s := testSession(t) s := testSession(t)
sess, err := decodeCookie(t, s, restamp( sess, err := decodeCookie(t, restamp(
t, t,
issuedCookie(t, s), issuedCookie(t, s),
time.Now().Add(-(testAbsoluteMaxAge+time.Hour)), time.Now().Add(-(testAbsoluteMaxAge+time.Hour)),
+13 -1
View File
@@ -224,10 +224,22 @@ func New(
} }
// Get retrieves a session for the request. // Get retrieves a session for the request.
//
// A session cookie that does not decode -- one signed with an earlier
// session key, say, because the database was made anew -- is treated
// as absent: the caller gets a new, empty session and no error, and
// the next save replaces the cookie.
func (s *Session) Get( func (s *Session) Get(
r *http.Request, r *http.Request,
) (*sessions.Session, error) { ) (*sessions.Session, error) {
return s.store.Get(r, SessionName) sess, err := s.store.Get(r, SessionName)
if sess == nil {
return nil, err
}
// For a cookie that does not decode, gorilla/sessions returns a
// new, empty session alongside the error that is dropped here.
return sess, nil
} }
// GetKey returns the raw 32-byte authentication key used for // GetKey returns the raw 32-byte authentication key used for
Executable
+16
View File
@@ -0,0 +1,16 @@
#!/bin/sh
# script/assets: extract Alpine.js from its npm package tarball, committed
# in 3p/, to static/js/alpine.min.js, where go:embed reads it. The
# extracted file is not committed. script/test, make build and make dev run
# this first.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
tar -xzOf 3p/alpinejs-3.14.9.tgz package/dist/cdn.min.js \
>static/js/alpine.min.js
}
main "$@"
+1 -8
View File
@@ -4,9 +4,7 @@
# installed tools are skipped. Base tooling comes from nix, apt, brew, # installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes NOTHING is present (not git, # or apk (detected in that order); assumes NOTHING is present (not git,
# make, or go). golangci-lint is deliberately not installed: linting runs # make, or go). golangci-lint is deliberately not installed: linting runs
# only in docker, via script/lint and Dockerfile.lint. Finishes by running # only in docker, via script/lint and Dockerfile.lint.
# script/fetch-assets, which installs the hash-pinned third-party browser
# assets the repo does not commit.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
@@ -69,11 +67,6 @@ main() {
go mod download go mod download
# Third-party browser assets are not committed; fetch and verify them
# so a fresh clone can build and test.
if missing curl; then pkg_install curl curl curl curl; fi
"$ROOT/script/fetch-assets"
echo "bootstrap complete" echo "bootstrap complete"
} }
+2 -1
View File
@@ -1,6 +1,7 @@
#!/bin/sh #!/bin/sh
# script/check: run all checks (test, lint, fmt-check). Our own # script/check: run all checks (test, lint, fmt-check). Our own
# extension to scripts-to-rule-them-all. Must not modify any files. # extension to scripts-to-rule-them-all.
# Writes only the ignored static/js/alpine.min.js, through script/test.
# Generic: usually needs no adaptation. # Generic: usually needs no adaptation.
set -eu set -eu
-104
View File
@@ -1,104 +0,0 @@
#!/bin/sh
# script/fetch-assets: download the third-party browser assets the web UI
# ships and install them under static/. Minified bundles are not committed
# (REPO_POLICIES.md: no build artifacts in version control), so the build
# fetches them here. Every download is verified against a hardcoded sha256
# before it is installed, and any mismatch aborts. Idempotent: an asset
# already present with its pinned hash is left alone.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# The sha256 of each installed asset lives in static/vendor.sha256, in
# sha256sum(1) format, with paths relative to static/. That file is the
# single source of truth: this script verifies against it, and
# static/vendor_test.go asserts the bytes embedded into the binary match
# it, so the hash cannot rot into a value nothing checks.
MANIFEST="static/vendor.sha256"
# Alpine.js 3.14.9, 2026-08-17. Fetched from registry.npmjs.org, the
# publisher of record; the jsDelivr and unpkg copies are mirrors of this
# same tarball. dist/cdn.min.js is the browser build Alpine publishes for
# a <script> tag.
ALPINE_VERSION="3.14.9"
ALPINE_URL="https://registry.npmjs.org/alpinejs/-/alpinejs-${ALPINE_VERSION}.tgz"
# sha256 of alpinejs-3.14.9.tgz
ALPINE_TARBALL_SHA256="97dad7c0c81e659cfc8e7700055da9770f8186187cb9a8a76efb57e00d5ce52a"
ALPINE_MEMBER="package/dist/cdn.min.js"
ALPINE_DEST="js/alpine.min.js"
sha256_of() {
if command -v sha256sum >/dev/null 2>&1; then
sha256sum "$1" | cut -d' ' -f1
else
shasum -a 256 "$1" | cut -d' ' -f1
fi
}
# expected_sha256 <path-relative-to-static>
expected_sha256() {
awk -v want="$1" '$2 == want { print $1; found = 1 }
END { if (!found) exit 1 }' "$ROOT/$MANIFEST"
}
# verify <file> <expected-sha256> <what>
verify() {
actual="$(sha256_of "$1")"
if [ "$actual" != "$2" ]; then
echo "fetch-assets: sha256 mismatch for $3" >&2
echo " expected: $2" >&2
echo " actual: $actual" >&2
exit 1
fi
}
# up_to_date <path-relative-to-static> <expected-sha256>
up_to_date() {
[ -f "$ROOT/static/$1" ] || return 1
[ "$(sha256_of "$ROOT/static/$1")" = "$2" ]
}
fetch_alpine() {
want="$(expected_sha256 "$ALPINE_DEST")"
if up_to_date "$ALPINE_DEST" "$want"; then
echo "fetch-assets: static/$ALPINE_DEST already at $want"
return 0
fi
echo "fetch-assets: fetching Alpine.js $ALPINE_VERSION from $ALPINE_URL"
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT INT TERM
curl -fsSL -o "$tmp/alpine.tgz" "$ALPINE_URL"
verify "$tmp/alpine.tgz" "$ALPINE_TARBALL_SHA256" "alpinejs-${ALPINE_VERSION}.tgz"
tar -xzOf "$tmp/alpine.tgz" "$ALPINE_MEMBER" >"$tmp/alpine.min.js"
verify "$tmp/alpine.min.js" "$want" "$ALPINE_MEMBER from alpinejs-${ALPINE_VERSION}.tgz"
mkdir -p "$(dirname "$ROOT/static/$ALPINE_DEST")"
cp "$tmp/alpine.min.js" "$ROOT/static/$ALPINE_DEST"
rm -rf "$tmp"
trap - EXIT INT TERM
echo "fetch-assets: installed static/$ALPINE_DEST ($want)"
}
# Re-check every manifest entry against what is now on disk, so an entry
# no script installs fails loudly instead of passing silently.
verify_manifest() {
while read -r want path; do
case "$want" in '' | '#'*) continue ;; esac
if [ ! -f "$ROOT/static/$path" ]; then
echo "fetch-assets: $MANIFEST lists static/$path, which is missing" >&2
exit 1
fi
verify "$ROOT/static/$path" "$want" "static/$path"
done <"$ROOT/$MANIFEST"
}
main() {
cd "$ROOT"
fetch_alpine
verify_manifest
echo "fetch-assets: all assets in $MANIFEST verified"
}
main "$@"
+1
View File
@@ -28,6 +28,7 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
"$ROOT/script/assets"
go test -v -race -timeout 90s ./... go test -v -race -timeout 90s ./...
} }
-1
View File
@@ -1 +0,0 @@
3ed1eed252488921df65e363d6715deb04d7f92aaedb9e52199fdf73cb1e0ad3 js/alpine.min.js
-92
View File
@@ -1,92 +0,0 @@
package static_test
import (
"bufio"
"crypto/sha256"
"encoding/hex"
"os"
"strings"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/static"
)
const manifestPath = "vendor.sha256"
// fetchHint is appended to every failure here: the assets the manifest
// covers are fetched by the build, not committed, so a fresh clone that
// has not run script/fetch-assets fails this test and should be told why.
const fetchHint = "run `script/fetch-assets` (or `make assets`) to install " +
"the pinned third-party assets"
// TestVendoredAssetsMatchManifest asserts that every asset listed in
// static/vendor.sha256 is embedded in the binary with exactly the pinned
// bytes. script/fetch-assets verifies the same hashes at download time;
// this test verifies them again on what actually ships, so a build that
// skipped, cached, or subverted the fetch cannot produce a binary serving
// unpinned third-party JavaScript.
func TestVendoredAssetsMatchManifest(t *testing.T) {
t.Parallel()
entries := readManifest(t)
require.NotEmpty(t, entries, "%s lists no assets", manifestPath)
for path, want := range entries {
t.Run(path, func(t *testing.T) {
t.Parallel()
data, err := static.Static.ReadFile(path)
require.NoErrorf(
t, err,
"%s is listed in %s but is not embedded; %s",
path, manifestPath, fetchHint,
)
sum := sha256.Sum256(data)
got := hex.EncodeToString(sum[:])
require.Equalf(
t, want, got,
"embedded %s does not match its pinned sha256 in %s; %s",
path, manifestPath, fetchHint,
)
})
}
}
// readManifest parses static/vendor.sha256, which is in sha256sum(1)
// format with paths relative to static/.
func readManifest(t *testing.T) map[string]string {
t.Helper()
f, err := os.Open(manifestPath)
require.NoError(t, err, "opening %s", manifestPath)
defer func() { require.NoError(t, f.Close()) }()
entries := make(map[string]string)
scanner := bufio.NewScanner(f)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line == "" || strings.HasPrefix(line, "#") {
continue
}
fields := strings.Fields(line)
require.Lenf(
t, fields, 2,
"%s: malformed entry %q, want \"<sha256> <path>\"",
manifestPath, line,
)
sum, path := fields[0], fields[1]
require.Lenf(t, sum, 64, "%s: %q is not a sha256", manifestPath, sum)
entries[path] = sum
}
require.NoError(t, scanner.Err(), "reading %s", manifestPath)
return entries
}
+2
View File
@@ -24,6 +24,8 @@
</div> </div>
</div> </div>
{{template "webhook_stats" .}}
<div class="grid grid-cols-1 lg:grid-cols-2 gap-6"> <div class="grid grid-cols-1 lg:grid-cols-2 gap-6">
<!-- Entrypoints --> <!-- Entrypoints -->
<div class="card"> <div class="card">
+93
View File
@@ -0,0 +1,93 @@
{{define "webhook_stats"}}
<!-- Statistics pane at the top of the webhook page. -->
<div class="card mb-6">
<div class="p-4 border-b border-gray-200">
<h2 class="text-lg font-medium text-gray-900">Statistics</h2>
</div>
{{with .Stats}}
<div class="p-4 flex flex-wrap gap-6 text-sm border-b border-gray-200">
<div>
<span class="text-gray-500">Entrypoints</span>
<span class="font-medium text-gray-900">{{.Entrypoints}}</span>
<span class="text-gray-500">({{.ActiveEntrypoints}} active)</span>
</div>
<div>
<span class="text-gray-500">Targets</span>
<span class="font-medium text-gray-900">{{.Targets}}</span>
<span class="text-gray-500">({{.ActiveTargets}} active)</span>
</div>
<div>
<span class="text-gray-500">Deliveries in progress</span>
<span class="font-medium text-gray-900">{{.InProgress}}</span>
</div>
<div>
<span class="text-gray-500">Last event</span>
<span class="font-medium text-gray-900">{{with .LastEventAt}}{{.Format "2006-01-02 15:04:05 UTC"}}{{else}}none{{end}}</span>
</div>
<div>
<span class="text-gray-500">Retention</span>
<span class="font-medium text-gray-900">{{$.Webhook.RetentionLabel}}</span>
</div>
</div>
<div class="p-4 grid grid-cols-1 lg:grid-cols-2 gap-6 text-sm">
<table class="w-full text-center text-gray-900">
<thead>
<tr class="border-b border-gray-200 text-xs text-gray-500 uppercase tracking-wide">
<th></th>
<th class="py-2 font-medium">Lifetime</th>
<th class="py-2 font-medium">Within retention</th>
</tr>
</thead>
<tbody>
<tr>
<td class="py-2 text-left text-gray-600">Events</td>
<td class="py-2">{{.Lifetime.Events}}</td>
<td class="py-2">{{.WithinRetention.Events}}</td>
</tr>
<tr>
<td class="py-2 text-left text-gray-600">Deliveries</td>
<td class="py-2">{{.Lifetime.Deliveries}}</td>
<td class="py-2">{{.WithinRetention.Deliveries}}</td>
</tr>
<tr>
<td class="py-2 text-left text-gray-600">Failures</td>
<td class="py-2">{{.Lifetime.Failures}}</td>
<td class="py-2">{{.WithinRetention.Failures}}</td>
</tr>
</tbody>
</table>
<div>
<table class="w-full text-center text-gray-900">
<thead>
<tr class="border-b border-gray-200 text-xs text-gray-500 uppercase tracking-wide">
<th></th>
<th class="py-2 font-medium">Last 10 minutes</th>
<th class="py-2 font-medium">Last 24 hours</th>
</tr>
</thead>
<tbody>
<tr>
<td class="py-2 text-left text-gray-600">Events</td>
<td class="py-2">{{.Last10Minutes.Events}}</td>
<td class="py-2">{{.Last24Hours.Events}}</td>
</tr>
<tr>
<td class="py-2 text-left text-gray-600">Failures</td>
<td class="py-2">{{.Last10Minutes.Failed}}</td>
<td class="py-2">{{.Last24Hours.Failed}}</td>
</tr>
<tr>
<td class="py-2 text-left text-gray-600">Failure percentage</td>
<td class="py-2">{{.Last10Minutes.FailurePercent}}</td>
<td class="py-2">{{.Last24Hours.FailurePercent}}</td>
</tr>
</tbody>
</table>
<p class="mt-2 text-xs text-gray-500">Failure percentage is the failed deliveries out of all deliveries that finished in the window. Deliveries still pending or retrying are not counted.</p>
</div>
</div>
{{else}}
<div class="p-4 text-sm text-gray-500">The statistics could not be read.</div>
{{end}}
</div>
{{end}}