Author SHA1 Message Date
clawbot 7287e163e7 Add a statistics pane to the webhook page (closes #368)
check / check (push) Waiting to run
Each webhook's event database keeps one row of running totals: events,
deliveries and failures, and how many of each retention removed.
Storing an event, creating a delivery, a delivery becoming failed and
the retention sweep each update it in the transaction that writes or
deletes the rows it counts. Deliveries get a finished_at column, the
last column of the status index, so the last-10-minutes and
last-24-hours figures are index-range counts. The pane is its own
template, included at the top of the page.

The schema changes in place with nothing back-filled, so an existing
database must be recreated.

Model: opus-5-5
2026-10-01 19:49:18 +00:00
clawbot 507980a347 Bound username length at creation (closes #184)
check / check (push) Waiting to run
Usernames are limited to 1024 bytes, so no account can exist that is unable to log in. The username rides in the session cookie, which browsers and securecookie refuse past about 4 KB, leaving room for roughly 2000 bytes of username; the limit is about half that.

User.BeforeSave returns ErrUsernameTooLong when a whole User is created or saved. A byte-counting check constraint on users.username catches every other write, including a column update. The limit appears in the constant and in the struct tag; a test fails if they disagree.

Model: opus-5-5
2026-10-01 21:38:48 +02:00
clawbot b79e4649a1 Container sets its data directory's owner and mode itself (#353)
check / check (push) Canceled after 0s
Closes #340.

The image no longer sets `USER`. Its new `ENTRYPOINT`, `deploy/docker-entrypoint.sh`, starts as root, creates `DATA_DIR` if missing, gives the directory and anything in it owned by another user to `webhooker` (UID 1000), sets the directory to `0750`, and runs the command as `webhooker` through `su-exec`. An empty root-owned bind mount, or data left by another UID, now works as mounted; the app never runs as root and is still PID 1. `CMD` is still `/app/webhooker`, so the `resetpw` commands are unchanged. Started with `--user`, the script only runs the command. It is in `/usr/local/bin`, not `/app`, which belongs to `webhooker`.

README: the UID 1000 ownership block, the upaas pre-deploy commands and the restore ownership step are gone; the upaas volume bullet names only the path.

- Judgement call: `su-exec` over `setpriv`: Alpine's small tool for this, needing only musl; busybox's `setpriv` cannot change user, and util-linux's adds `libcap-ng`.
- Deviation: `su-exec` is pinned by version (`0.2-r3`), not by hash; `ca-certificates` beside it is unpinned.
- Judgement call: each start reads every entry's owner but changes only entries owned by someone else.
- `docker exec` and the health check now run as root, since the image sets no `USER`.
- No automated test covers the script: the suite runs inside `docker build`, which cannot start a container.
- A missing host directory under upaas is sneak/upaas#235.

Model: opus-5-5
Reviewed-on: #353
Co-authored-by: clawbot <35+clawbot@noreply.example.org>
2026-09-29 13:05:16 +02:00
clawbot 1428154bbd Let a browser with cookies from an earlier database log in (closes #359)
check / check (push) Canceled after 0s
A new database brings a new session key. A browser still holding the
old session cookie got a 500 on a correct login: Session.Get returned
the cookie's decode error and the login handler answered it with a
500. Get now treats a cookie that does not decode as absent, and
logging in replaces it. gorilla/csrf already did the same for the
CSRF cookie.

A start that creates webhooker.db now logs "created a new, empty
database" at WARN with its path, shortly before the first-boot banner,
so an unexpectedly empty DATA_DIR is noticed.

The codec tests now decode through the store, since Get no longer
reports the codec's reason.

Model: opus-5-5
2026-09-29 12:58:44 +02:00
sneak 8ad2a86e4b Sneak/testdeploy (#356)
check / check (push) Successful in 11s
Reviewed-on: #356
2026-09-29 12:01:33 +02:00
sneak a891b726e5 Milestone: next into main (#342)
check / check (push) Successful in 9s
Reviewed-on: #342
2026-09-29 11:53:19 +02:00
clawbot ab63b5f777 Restrict /s/* to GET and HEAD (closes #169)
check / check (push) Successful in 3m37s
The static file server was attached with Mount, which registers every
method, so POST, PUT and DELETE on an asset were answered 200 with the
file. It is now registered for GET and HEAD only, inside a /s group
whose method-not-allowed handler answers 405 with Allow: GET, HEAD.
A method chi does not route at all, such as PROPFIND, still gets 405
from the top-level router, without Allow. The inverted test and the
README route table say the same.

Model: opus-5-5
2026-09-29 11:10:27 +02:00
33 changed files with 1653 additions and 177 deletions
+9 -2
View File
@@ -88,7 +88,9 @@ RUN CGO_ENABLED=1 make build VERSION="$VERSION" GO_LDFLAGS='-extldflags "-static
# alpine:3.21, 2026-03-17
FROM alpine:3.21@sha256:c3f8e73fdb79deaebaa2037150150191b9dcbfba68b4a46d70103204c53f4709
RUN apk --no-cache add ca-certificates
# su-exec 0.2-r3 (Alpine 3.21), 2026-09-29: the entrypoint runs the app
# as webhooker with it.
RUN apk --no-cache add ca-certificates su-exec=0.2-r3
# Create non-root user
RUN addgroup -g 1000 -S webhooker && \
@@ -99,13 +101,17 @@ WORKDIR /app
# Copy binary from builder
COPY --from=builder /build/bin/webhooker /app/webhooker
# Not under /app, which belongs to webhooker: this script runs as root.
COPY deploy/docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
# Create data directory for all SQLite databases (main app DB +
# per-webhook event DBs). DATA_DIR defaults to /var/lib/webhooker.
RUN mkdir -p /var/lib/webhooker
RUN chown -R webhooker:webhooker /app /var/lib/webhooker
USER webhooker
# No USER: the entrypoint starts as root to make the data directory
# webhooker's, then runs the app as webhooker.
EXPOSE 8080
@@ -124,4 +130,5 @@ ENV BIND_ADDRESS=0.0.0.0
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:8080/.well-known/healthcheck || exit 1
ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]
CMD ["/app/webhooker"]
+100 -94
View File
@@ -538,6 +538,12 @@ its Argon2id hash. There is no second account and no forgot-password
flow, so the banner and the reset command below are the only two ways
in.
A start that finds no `webhooker.db` in `DATA_DIR` also logs
`created a new, empty database` at `WARN`, with the file's path,
shortly before the banner. On a deployment that has run before, that
line means `DATA_DIR` was empty, most often because its volume is not
mounted.
#### Recovering a lost admin password
`webhooker resetpw` sets an existing account's password from the
@@ -552,8 +558,9 @@ printf '%s' "$NEW_PASSWORD" | \
DATA_DIR=/var/lib/webhooker webhooker resetpw admin
```
In a container it is the same binary, which the image sets as `CMD`
rather than `ENTRYPOINT`, so the whole command has to be given:
In a container it is the same binary. The image's `CMD` is
`/app/webhooker`, and a command given to `docker run` replaces all of
it, so the whole command has to be given:
```bash
docker run --rm -v webhooker-data:/var/lib/webhooker \
@@ -690,38 +697,22 @@ those three values rather than trusting the figure. Measured at 65s on
Docker 29.7.2.) A container `unhealthy` with `connection refused` in
its health log, or a published port that resets connections, is this.
The container runs as a non-root user (`webhooker`, UID 1000), exposes
port 8080, and includes a health check against
`/.well-known/healthcheck`. The `/var/lib/webhooker` volume holds all
SQLite databases: the main application database (`webhooker.db`), the
per-webhook event databases (`events-{uuid}.db`), and any archive
databases written by `database` targets (`archive-{uuid}.db`). Mount
this as a persistent volume to preserve data across container
restarts.
The app runs as a non-root user (`webhooker`, UID 1000), exposes port
8080, and includes a health check against `/.well-known/healthcheck`.
The `/var/lib/webhooker` volume holds all SQLite databases: the main
application database (`webhooker.db`), the per-webhook event databases
(`events-{uuid}.db`), and any archive databases written by `database`
targets (`archive-{uuid}.db`). Mount this as a persistent volume to
preserve data across container restarts.
**The bind-mounted directory must be owned by UID 1000, or the
container does not start.** Docker creates a `-v` source path that
does not exist yet as `root:root`, and the process runs as UID 1000,
so it cannot take its `DATA_DIR` lock:
```
webhooker: locking data directory /var/lib/webhooker: open
/var/lib/webhooker/webhooker.lock: permission denied
```
It exits non-zero at that point, before opening any database. Create
the directory ahead of the first `docker run`:
```bash
mkdir -p /path/to/data
chown 1000:1000 /path/to/data
chmod 750 /path/to/data
```
The same `chown` is what a restore needs — see step 4 of
[Restore](#restore). A **named volume** does not have this problem:
Docker copies the image's ownership onto a volume it initializes, and
the image creates `/var/lib/webhooker` owned by `webhooker`.
**The container sets its data directory's owner and mode itself
before the app starts**, so a host directory can be mounted as it is,
whoever owns it. The image's `ENTRYPOINT`,
`deploy/docker-entrypoint.sh`, starts as root, creates `DATA_DIR` if
it is missing, gives the directory and anything in it that belongs to
another user to `webhooker`, sets the directory to `0750`, and only
then runs the app as `webhooker`. Started with `--user`, it changes
nothing and runs the app as that user.
**The file modes are not yours to set, and do not depend on the
directory.** `webhooker.db` holds target configuration in plaintext —
@@ -729,13 +720,10 @@ bearer tokens, API keys, Slack webhook URLs — along with the session
encryption key, so webhooker creates every SQLite file it owns `0600`:
each database and both of its `-wal` and `-shm` sidecars, across all
three tiers. Files an earlier build left `0644` are tightened when
they are opened. A `DATA_DIR` webhooker creates itself is `0750`, but
a bind mount supplies its own directory and Docker's default for one
it creates is `0755`; the `0600` files hold there regardless. The
`chmod 750` above is defence in depth — it stops other local users
listing the directory and learning your webhook UUIDs from the
`events-{uuid}.db` filenames — not the barrier protecting the
credentials.
they are opened. The directory's `0750` is defence in depth — it stops
other local users listing the directory and learning your webhook
UUIDs from the `events-{uuid}.db` filenames — not the barrier
protecting the credentials.
### Running under upaas
@@ -751,17 +739,6 @@ repository's `Dockerfile` and runs it. The app needs:
app name, port `8080`. Leave `PORT` unset: the image's health check
probes `8080`.
- **Volume:** one host directory mounted at `/var/lib/webhooker`.
upaas bind-mounts the host path it is given and does not create it,
and the container does not start unless UID 1000 owns it (see
[Running with Docker](#running-with-docker)). Create it before the
first deploy:
```bash
mkdir -p /path/to/data
chown 1000:1000 /path/to/data
chmod 750 /path/to/data
```
- **Environment variables:**
- `WEBHOOKER_ENVIRONMENT=prod`
- `TRUSTED_PROXIES`: your reverse proxy's address on that Docker
@@ -1020,12 +997,12 @@ done
`.backup` reads through the WAL and writes a single consistent file with
no sidecars of its own, so the destination is complete as it stands.
Two caveats. First, the runtime image is `alpine:3.21` with only
`ca-certificates` added — the `sqlite3` CLI is **not** in it, so run
this on the host against the volume path, or from a throwaway container
that mounts the volume. Second, each file is captured at its own
instant, so a webhook created or an event delivered between two files
being copied lands in one and not the other. If you need the whole set
coherent as of a single moment, stop the service.
`ca-certificates` and `su-exec` added — the `sqlite3` CLI is **not** in
it, so run this on the host against the volume path, or from a
throwaway container that mounts the volume. Second, each file is
captured at its own instant, so a webhook created or an event delivered
between two files being copied lands in one and not the other. If you
need the whole set coherent as of a single moment, stop the service.
Note that `sqlite3 <db> .dump` is **not** one of these procedures: it is
an export, it holds a read transaction open for as long as it runs, and
@@ -1079,21 +1056,11 @@ with any `-wal`/`-shm` beside it, or wait until there are none.
archive not opened since a crash. A copy salvaged from a crashed
instance has them for everything, and needs all of them.
4. **Fix ownership.** The container runs as the non-root `webhooker`
user, UID 1000 / GID 1000. Restored files must be owned by (or
writable by) that UID, and so must the directory itself — SQLite
creates the `-wal` and `-shm` sidecars beside the database, so a
writable file inside a directory it cannot write is not enough:
```bash
chown -R 1000:1000 /path/to/data
```
Restoring as `root` on the host and forgetting this step is the
usual way a restore fails.
5. Start the service. `AutoMigrate` runs against each restored database
as it is opened.
4. Start the service. The container gives the directory and the
restored files to the `webhooker` user before the app starts,
whoever restored them (see
[Running with Docker](#running-with-docker)). `AutoMigrate` runs
against each restored database as it is opened.
### Upgrades
@@ -1103,7 +1070,7 @@ unconditionally against whatever files it finds:
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
`Entrypoint`, `Target`
- each event database when it is lazily opened — `Event`, `Delivery`,
`DeliveryResult`
`DeliveryResult`, `Totals`
- each archive database on every open and reopen
There is no schema version table, no migration ledger, and no down
@@ -1417,7 +1384,7 @@ The codebase uses consistent naming throughout (rename completed in
### Data Model
webhooker's data model has nine entities organized into two tiers: the
webhooker's data model has ten entities organized into two tiers: the
**application tier** (user and webhook configuration) and the **event
tier** (event ingestion, delivery, and logging).
@@ -1446,6 +1413,10 @@ tier** (event ingestion, delivery, and logging).
│ ┌──────────┐ ┌──────────┐ ┌─────────────────┐ │
│ │ Event │──1:N──│ Delivery │──1:N──│ DeliveryResult │ │
│ └──────────┘ └──────────┘ └─────────────────┘ │
│ │
│ ┌──────────┐ │
│ │ Totals │ (one row of running counts) │
│ └──────────┘ │
└─────────────────────────────────────────────────────────────┘
```
@@ -1472,7 +1443,7 @@ A registered user of the webhooker service.
| Field | Type | Description |
| ---------- | -------- | ----------- |
| `id` | UUID | Primary key |
| `username` | string | Unique login name |
| `username` | string | Unique login name, at most 1024 bytes so that it fits in the session cookie |
| `password` | string | Argon2id hash (never exposed via API) |
**Relations:** Has many Webhooks. Has many APIKeys.
@@ -1698,6 +1669,7 @@ status across potentially multiple attempts.
| `event_id` | UUID | Foreign key → Event |
| `target_id`| UUID | Foreign key → Target |
| `status` | DeliveryStatus | One of: `pending`, `delivered`, `failed`, `retrying` |
| `finished_at` | timestamp | When the delivery became `delivered` or `failed` (nullable; empty while `pending` or `retrying`) |
**Relations:** Belongs to Event. Belongs to Target. Has many
DeliveryResults.
@@ -1765,6 +1737,29 @@ retries) is individually logged for full observability.
**Relations:** Belongs to Delivery.
#### Totals
The one row of running counts in each event database, read by the
statistics pane at the top of the webhook page.
| Field | Type | Description |
| -------------------- | ------- | ----------- |
| `events` | integer | Events ever stored, resubmitted copies included |
| `deliveries` | integer | Deliveries ever created, replays included |
| `failures` | integer | Deliveries that ever became `failed` |
| `events_removed` | integer | Events retention has deleted |
| `deliveries_removed` | integer | Deliveries retention has deleted |
| `failures_removed` | integer | Failed deliveries retention has deleted |
Each count changes in the transaction that writes or deletes the rows it
counts. The pane shows each of the first three as a lifetime figure, and
less what retention removed as the figure within retention, so neither
needs the rows themselves. Its last-10-minutes and last-24-hours figures
are counted from the `events` and `deliveries` indexes over just that
window. Its failure percentage for a window is the deliveries that became
`failed` in it out of all that became `delivered` or `failed` in it, and
a dash when none did.
#### Event-tier indexes
These indexes on the per-webhook event databases are declared in the model
@@ -1772,10 +1767,10 @@ tags, so `AutoMigrate` creates them on a fresh and on an existing database:
| Table | Columns | Serves |
| ------------------ | --------------------------- | ------ |
| `deliveries` | `status`, `deleted_at` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status |
| `deliveries` | `status`, `deleted_at`, `finished_at` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, and the webhook page's statistics, which count deliveries by status and when they finished |
| `deliveries` | `event_id`, `deleted_at` | The event log, which loads each event's deliveries, and retention, which selects and deletes the deliveries of expired events |
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
| `events` | `deleted_at`, `created_at` | Retention, which selects expired events by age |
| `events` | `deleted_at`, `created_at` | Retention, which selects expired events by age, and the webhook page's statistics, which count recent events and find the newest |
| `events` | `created_at` | Retention's delete of the expired events themselves |
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention's
@@ -1789,9 +1784,10 @@ and SQLite narrows by a `<` only on the last column it uses.
#### Common Fields
Every entity except `Setting` includes these fields from `BaseModel`.
`Setting` is a bare key-value row with no `id`, no timestamps and no
soft delete:
Every entity except `Setting` and `Totals` includes these fields from
`BaseModel`. `Setting` is a bare key-value row with no `id`, no
timestamps and no soft delete, and `Totals` is a single row of counts
with only a numeric `id`:
| Field | Type | Description |
| ------------ | --------- | ----------- |
@@ -1833,6 +1829,7 @@ encryption key is generated and stored, and an `admin` user is created.
- **Events** — captured incoming webhook payloads
- **Deliveries** — event-to-target pairings and their status
- **DeliveryResults** — individual delivery attempt logs
- **Totals** — running counts of the above, kept through retention
Per-webhook databases are created automatically when a webhook is
created (and lazily on first access for webhooks that predate this
@@ -2412,14 +2409,14 @@ Removing either cap fails 14 subtests.
`internal/middleware/logbound_test.go` and
`internal/handlers/logbound_test.go` drive 8 KB of client-chosen text
at each of these — 1 KB at `invalid password`, whose accounts are
shared with the successful-login line, where a username past 4 KB
overflows the session cookie and answers 500 before that line is
written — through both handlers, and through seven fills: plain text
as the baseline, and then the quotation mark, backslash, tab, newline,
C0 control and astral non-printable, six characters the wider of the
two handlers spends more on than the client spent sending them. Every
case holds each line to the 2,560-byte ceiling. That per-line ceiling
at each of these — just under 1 KB at `invalid password`, whose
accounts are shared with the successful-login line and so must stay
within the 1024-byte username limit — through both handlers, and
through seven fills: plain text as the baseline, and then the
quotation mark, backslash, tab, newline, C0 control and astral
non-printable, six characters the wider of the two handlers spends
more on than the client spent sending them. Every case holds each
line to the 2,560-byte ceiling. That per-line ceiling
is what the figure above states, and every row establishes it.
Three of the sites go further and bound the whole flood's output — the
@@ -2721,7 +2718,7 @@ abuse limit later; they are tracked as future work.
| ------ | --------------------------- | ----------- |
| `GET` | `/` | Root redirect, 303 (authenticated → `/sources`, unauthenticated → `/pages/login`) |
| `GET` | `/.well-known/healthcheck` | Health check (JSON: `status`, `now`, `uptimeSeconds`, `uptimeHuman`, `version`, `appname`, `maintenanceMode`) |
| any | `/s/*` | Static file serving (embedded CSS, JS). Mounted for every method, not just `GET`/`HEAD`: chi's `Mount` registers all methods and `http.FileServer` special-cases only `HEAD` (by omitting the body), so a `POST` or `DELETE` to an asset is answered `200` with the file. Pinned by `TestStaticServesEveryMethod` |
| `GET`, `HEAD` | `/s/*` | Static file serving (embedded CSS, JS). `GET` and `HEAD` only — `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` and `CONNECT` are answered `405 Method Not Allowed` with `Allow: GET, HEAD`. Any other method (such as `PROPFIND`) is refused by chi before it reaches this route, and gets `405` without an `Allow` header. Pinned by `TestStaticServesOnlyGetAndHead` |
| `POST` | `/webhook/{uuid}` | Webhook receiver endpoint. `POST` only — every other method is answered `405 Method Not Allowed` with `Allow: POST`. Rate limited (see [Rate Limiting](#rate-limiting)) |
#### Authentication Endpoints
@@ -3071,7 +3068,11 @@ check, see [The login endpoint](#the-login-endpoint).
- Prometheus metrics behind basic auth
- Static assets embedded in binary (no filesystem access needed at
runtime)
- Container runs as non-root user (UID 1000)
- The app runs as the non-root `webhooker` user (UID 1000) in the
container. The image sets no `USER`, so these run as root: the
`ENTRYPOINT` script, which sets the data directory's owner and mode
before the app starts; the image's health check; and `docker exec`,
unless given `--user`
- GORM soft deletes on every entity that carries `BaseModel`, which is
all of them but `Setting` (data preserved for audit)
@@ -3198,10 +3199,13 @@ version is fixed independently of the compiler's:
`GO_LDFLAGS`, so neither can drop the `-X` that stamps the version.
The version arrives as the `VERSION` build arg, since the context
has no `.git` (see [Version stamping](#version-stamping)).
3. **Runtime stage** (`alpine:3.21`) — copies the static binary,
creates the `/var/lib/webhooker` directory for all SQLite databases,
runs as the non-root `webhooker` user (UID 1000), exposes port 8080,
and includes a health check against `/.well-known/healthcheck`.
3. **Runtime stage** (`alpine:3.21`) — copies the static binary and
`deploy/docker-entrypoint.sh`, creates the `/var/lib/webhooker`
directory for all SQLite databases, exposes port 8080, and includes
a health check against `/.well-known/healthcheck`. It sets no
`USER`: the `ENTRYPOINT` script starts as root, sets the data
directory's owner and mode, and runs the app as the non-root
`webhooker` user (UID 1000) through `su-exec`.
The lint stage invokes `golangci-lint` directly rather than `make lint`:
it is already the pinned linter image, and `make lint` builds
@@ -3280,3 +3284,5 @@ MIT
## Author
[@sneak](https://sneak.berlin)
+22
View File
@@ -0,0 +1,22 @@
#!/bin/sh
# deploy/docker-entrypoint.sh: the image's ENTRYPOINT. A bind-mounted
# data directory keeps its owner from the host, often root, and the app
# could not write to it. Started as root, this creates DATA_DIR if
# needed, gives it and everything in it to webhooker, sets its mode, and
# runs the command as webhooker, so the app never runs as root. Started
# as another user, it only runs the command.
set -eu
main() {
if [ "$(id -u)" != 0 ]; then
exec "$@"
fi
dir="${DATA_DIR:-/var/lib/webhooker}"
mkdir -p "$dir"
find "$dir" ! -user webhooker -exec chown -h webhooker:webhooker {} +
chmod 750 "$dir"
exec su-exec webhooker "$@"
}
main "$@"
@@ -3,6 +3,8 @@ package database_test
import (
"bytes"
"context"
"log/slog"
"path/filepath"
"strings"
"testing"
@@ -83,3 +85,37 @@ func TestFirstBoot_PrintsTheAdminPasswordAsABanner(t *testing.T) {
t, ok, "the printed password must open the seeded account",
)
}
// TestNewDatabase_IsLoggedWithItsPath is the log half of
// https://git.eeqj.de/sneak/webhooker/issues/359. A DATA_DIR that is
// unexpectedly empty boots exactly like a first start, so the start
// that creates the database must say so, and where. Opening that
// database again must not.
func TestNewDatabase_IsLoggedWithItsPath(t *testing.T) {
t.Parallel()
dir := t.TempDir()
open := func() string {
var out bytes.Buffer
db, err := database.Open(dir, slog.New(slog.NewTextHandler(&out, nil)))
require.NoError(t, err)
require.NoError(t, db.Close())
return out.String()
}
const created = `level=WARN msg="created a new, empty database"`
first := open()
second := open()
assert.Contains(
t, first,
created+" path="+filepath.Join(dir, database.MainDBFileName),
)
assert.NotContains(
t, second, created, "an existing database is not new",
)
}
+13 -1
View File
@@ -8,6 +8,7 @@ import (
"errors"
"fmt"
"io"
"io/fs"
"log/slog"
"os"
"path/filepath"
@@ -199,6 +200,12 @@ func (d *Database) connectTo(dataDir string) error {
// Construct the main application database path inside DATA_DIR.
dbPath := filepath.Join(dataDir, MainDBFileName)
// Checked before opening, which creates the file. A DATA_DIR that
// is unexpectedly empty -- its volume not mounted, say -- looks
// exactly like a first start, so a new database is a warning.
_, statErr := os.Stat(dbPath)
created := errors.Is(statErr, fs.ErrNotExist)
// Opened through OpenSQLite so this handle carries the same WAL
// journaling, busy timeout, immediate-transaction locking, and pool
// bounds as every other database file. See sqlite_open.go.
@@ -229,7 +236,12 @@ func (d *Database) connectTo(dataDir string) error {
}
d.db = db
d.log.Info("connected to database", "path", dbPath)
if created {
d.log.Warn("created a new, empty database", "path", dbPath)
} else {
d.log.Info("connected to database", "path", dbPath)
}
// Run migrations
return d.migrate()
+69 -5
View File
@@ -93,6 +93,7 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
deliveries []database.Delivery
results []database.DeliveryResult
depths []struct{ Depth int }
failed struct{ Count int64 }
)
byStatus := "idx_deliveries_status (status=? AND deleted_at=?)"
@@ -123,7 +124,7 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
Order("attempt_num ASC").Find(&results),
"idx_delivery_results_delivery_id (delivery_id=? AND deleted_at=?)")
// Retention's three deletes (reapExpired), whose subqueries are built
// Retention's deletes (deleteExpired), whose subqueries are built
// afresh for each statement as it builds them.
expiredEventIDs := func() *gorm.DB {
return dry.Model(&database.Event{}).Select("id").
@@ -135,6 +136,12 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
Select("id").Where("event_id IN (?)", expiredEventIDs()),
).Delete(&database.DeliveryResult{}),
"idx_delivery_results_delivery_id (delivery_id=?)", byEvent, byAge)
assertPlanUses(t, db, dry.Unscoped().Model(&database.Delivery{}).
Select("count(CASE WHEN status = ? THEN 1 END) AS count",
database.DeliveryStatusFailed).
Where("event_id IN (?)", expiredEventIDs()).
Take(&failed),
"idx_deliveries_event_id (event_id=?)", byAge)
assertPlanUses(t, db, dry.Unscoped().Where(
"event_id IN (?)", expiredEventIDs(),
).Delete(&database.Delivery{}),
@@ -144,6 +151,54 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
).Delete(&database.Event{}), "idx_events_created_at (created_at<?)")
}
// TestStatisticsQueriesUseTheirIndexes does the same for the webhook
// page's statistics (readEventStats in the handlers): deliveries in
// progress, deliveries finished and events received since a time, and
// the newest event, which must come straight off an index rather than
// from sorting every event.
func TestStatisticsQueriesUseTheirIndexes(t *testing.T) {
t.Parallel()
mgr, lc := setupTestWebhookDBManager(t)
ctx := context.Background()
require.NoError(t, lc.Start(ctx))
defer func() { require.NoError(t, lc.Stop(ctx)) }()
db, err := mgr.GetDB(uuid.New().String())
require.NoError(t, err)
dry := db.Session(&gorm.Session{DryRun: true})
since := time.Now()
var (
count int64
newest []time.Time
)
assertPlanUses(t, db, dry.Model(&database.Delivery{}).
Where("status IN ?", []database.DeliveryStatus{
database.DeliveryStatusPending,
database.DeliveryStatusRetrying,
}).Count(&count),
"idx_deliveries_status (status=? AND deleted_at=?)")
assertPlanUses(t, db, dry.Model(&database.Delivery{}).
Where("status = ? AND finished_at >= ?",
database.DeliveryStatusFailed, since).Count(&count),
"idx_deliveries_status "+
"(status=? AND deleted_at=? AND finished_at>?)")
assertPlanUses(t, db, dry.Model(&database.Event{}).
Where("created_at >= ?", since).Count(&count),
"idx_events_deleted_at_created_at "+
"(deleted_at=? AND created_at>?)")
newestEvent := dry.Model(&database.Event{}).
Order("created_at DESC").Limit(1).Pluck("created_at", &newest)
assertPlanUses(t, db, newestEvent,
"idx_events_deleted_at_created_at (deleted_at=?)")
assert.NotContains(t, queryPlan(t, db, newestEvent), "TEMP B-TREE")
}
// assertPlanUses asserts that SQLite's plan for a statement GORM built
// in a dry run, run with the same SQL and arguments GORM would send,
// names each of the given indexes.
@@ -152,6 +207,18 @@ func assertPlanUses(
) {
t.Helper()
plan := queryPlan(t, db, built)
for _, index := range indexes {
assert.Contains(t, plan, index, built.Statement.SQL.String())
}
}
// queryPlan returns SQLite's plan for a statement GORM built in a dry
// run, run with the same SQL and arguments GORM would send.
func queryPlan(t *testing.T, db, built *gorm.DB) string {
t.Helper()
var plan []struct{ Detail string }
require.NoError(t, db.Raw(
@@ -159,8 +226,5 @@ func assertPlanUses(
built.Statement.Vars...,
).Scan(&plan).Error)
for _, index := range indexes {
assert.Contains(t, fmt.Sprint(plan), index,
built.Statement.SQL.String())
}
return fmt.Sprint(plan)
}
+11 -1
View File
@@ -1,6 +1,10 @@
package database
import "gorm.io/gorm"
import (
"time"
"gorm.io/gorm"
)
// DeliveryStatus represents the status of a delivery
type DeliveryStatus string
@@ -45,6 +49,12 @@ type Delivery struct {
// gives.
DeletedAt gorm.DeletedAt `gorm:"index:idx_deliveries_event_id,priority:2;index:idx_deliveries_status,priority:2" json:"deletedAt,omitzero"`
// FinishedAt is when the delivery became delivered or failed, and
// nil while it is pending or retrying. It ends the status index,
// so the webhook page counts the deliveries that finished in a
// recent window by reading that window from the index.
FinishedAt *time.Time `gorm:"index:idx_deliveries_status,priority:3" json:"finishedAt,omitempty"`
// Relations
Event Event `json:"event,omitzero"`
Target Target `json:"target,omitzero"`
+73
View File
@@ -0,0 +1,73 @@
package database
import (
"fmt"
"gorm.io/gorm"
)
// Totals is the single row of running totals in a webhook's event
// database. It is what keeps the webhook page's lifetime figures right
// after retention has removed the rows they count, and what lets the
// page show them without counting every row.
//
// Storing an event, creating a delivery and failing a delivery each
// add one, and retention adds what it deletes to the Removed columns.
// Every addition goes through AddTotals, in the transaction that
// writes or deletes the rows it counts.
type Totals struct {
ID int64 `gorm:"primaryKey"`
Events int64 `gorm:"not null"`
Deliveries int64 `gorm:"not null"`
Failures int64 `gorm:"not null"`
EventsRemoved int64 `gorm:"not null"`
DeliveriesRemoved int64 `gorm:"not null"`
FailuresRemoved int64 `gorm:"not null"`
}
// TableName names the table AddTotals updates.
func (Totals) TableName() string {
return "totals"
}
// EventsWithinRetention is how many of the webhook's events are still
// stored.
func (t Totals) EventsWithinRetention() int64 {
return t.Events - t.EventsRemoved
}
// DeliveriesWithinRetention is how many of the webhook's deliveries
// are still stored.
func (t Totals) DeliveriesWithinRetention() int64 {
return t.Deliveries - t.DeliveriesRemoved
}
// FailuresWithinRetention is how many of the webhook's failed
// deliveries are still stored.
func (t Totals) FailuresWithinRetention() int64 {
return t.Failures - t.FailuresRemoved
}
// AddTotals adds each count in add to the webhook's running totals.
// Call it on the transaction that writes or deletes the rows it
// counts, so the totals change exactly when those rows do.
func AddTotals(tx *gorm.DB, add Totals) error {
err := tx.Exec(
`UPDATE totals SET
events = events + ?,
deliveries = deliveries + ?,
failures = failures + ?,
events_removed = events_removed + ?,
deliveries_removed = deliveries_removed + ?,
failures_removed = failures_removed + ?`,
add.Events, add.Deliveries, add.Failures,
add.EventsRemoved, add.DeliveriesRemoved, add.FailuresRemoved,
).Error
if err != nil {
return fmt.Errorf("adding to running totals: %w", err)
}
return nil
}
+47 -2
View File
@@ -1,13 +1,58 @@
package database
import (
"errors"
"fmt"
"gorm.io/gorm"
)
// MaxUsernameBytes is the longest username, in bytes, that a user may
// have. The same number appears in the check constraint on
// User.Username, because a struct tag cannot reference a constant.
//
// A login stores the username in the session cookie, and both
// securecookie and browsers refuse a cookie value past about 4096
// bytes. That value is the session base64-encoded twice, so it holds
// 4096 × 3/4 × 3/4 = 2304 bytes of session, and the signature,
// timestamp and the session's other values take about 270 of those: a
// username longer than about 2030 bytes can never log in. The limit is
// about half that, so the session can carry more values later without
// locking out an account whose username is already at the limit.
const MaxUsernameBytes = 1024
// ErrUsernameTooLong is returned when a user is saved with a username
// longer than MaxUsernameBytes.
var ErrUsernameTooLong = errors.New("username is too long")
// User represents a user of the webhooker service
//
//nolint:lll // a struct tag cannot wrap
type User struct {
BaseModel
Username string `gorm:"uniqueIndex;not null" json:"username"`
Password string `gorm:"not null" json:"-"` // Argon2 hashed
Username string `gorm:"uniqueIndex;not null;check:length(CAST(username AS BLOB)) <= 1024" json:"username"`
Password string `gorm:"not null" json:"-"` // Argon2 hashed
// Relations
Webhooks []Webhook `json:"webhooks,omitempty"`
APIKeys []APIKey `json:"apiKeys,omitempty"`
}
// BeforeSave rejects a username longer than MaxUsernameBytes when a whole
// User is created or saved, so those calls get ErrUsernameTooLong rather
// than the database's constraint error. A column update such as
// Update("username", ...) is caught only by the check constraint, as is
// any path that writes the table without this model.
func (u *User) BeforeSave(_ *gorm.DB) error {
if len(u.Username) > MaxUsernameBytes {
return fmt.Errorf(
"%w: %d bytes, limit is %d",
ErrUsernameTooLong,
len(u.Username),
MaxUsernameBytes,
)
}
return nil
}
+65
View File
@@ -0,0 +1,65 @@
package database_test
import (
"strings"
"testing"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
)
// usernameAtLimit is exactly MaxUsernameBytes long, built from a
// two-byte character. A check that counted characters rather than bytes
// would see half the length and let the one-byte-longer name through.
func usernameAtLimit() string {
return strings.Repeat("é", database.MaxUsernameBytes/2)
}
func TestUserCreate_RejectsOverlongUsername(t *testing.T) {
t.Parallel()
db := startedTestDB(t)
err := db.Create(&database.User{
Username: usernameAtLimit() + "x",
Password: "hash",
}).Error
require.ErrorIs(t, err, database.ErrUsernameTooLong)
}
func TestUserCreate_AcceptsUsernameAtLimit(t *testing.T) {
t.Parallel()
db := startedTestDB(t)
require.NoError(t, db.Create(&database.User{
Username: usernameAtLimit(),
Password: "hash",
}).Error)
}
// TestUsersTable_EnforcesUsernameLimitWithoutTheModel inserts with raw
// SQL, as a path that bypassed User.BeforeSave would, so only the
// table's check constraint stands between it and an over-long
// username. Accepting the name at the limit and refusing the next byte
// also pins the constraint's number to MaxUsernameBytes.
func TestUsersTable_EnforcesUsernameLimitWithoutTheModel(t *testing.T) {
t.Parallel()
db := startedTestDB(t)
insert := "INSERT INTO users (id, username, password) VALUES (?, ?, ?)"
require.NoError(t, db.Exec(
insert, uuid.New().String(), usernameAtLimit(), "hash",
).Error)
err := db.Exec(
insert, uuid.New().String(), usernameAtLimit()+"x", "hash",
).Error
require.Error(t, err)
assert.Contains(t, err.Error(), "CHECK constraint failed")
}
+1 -1
View File
@@ -2,7 +2,7 @@ package database
// Migrate runs database migrations for the main application database.
// Only configuration-tier models are stored in the main database.
// Event-tier models (Event, Delivery, DeliveryResult) live in
// Event-tier models (Event, Delivery, DeliveryResult, Totals) live in
// per-webhook dedicated databases managed by WebhookDBManager.
func (d *Database) Migrate() error {
return d.db.AutoMigrate(
+59 -13
View File
@@ -267,55 +267,101 @@ func retentionCutoff(
// reapExpired hard-deletes, in foreign-key-safe order, the delivery
// results, deliveries, and events associated with events older than
// cutoff. Deletes are unscoped so rows are physically removed rather
// than soft-deleted, reclaiming disk. It returns the number of events
// deleted.
// cutoff, and adds what it deleted to the running totals, all in one
// transaction. Deletes are unscoped so rows are physically removed
// rather than soft-deleted, reclaiming disk. It returns the number of
// events deleted.
func reapExpired(db *gorm.DB, cutoff time.Time) (int64, error) {
var removed Totals
err := db.Transaction(func(tx *gorm.DB) error {
var err error
removed, err = deleteExpired(tx, cutoff)
if err != nil {
return err
}
return AddTotals(tx, removed)
})
if err != nil {
return 0, err
}
return removed.EventsRemoved, nil
}
// deleteExpired runs reapExpired's deletes and returns how many
// events, deliveries and failed deliveries they removed.
func deleteExpired(tx *gorm.DB, cutoff time.Time) (Totals, error) {
var removed Totals
// Fresh subqueries are built per statement to avoid reusing a
// mutated builder across executions.
expiredEventIDs := func() *gorm.DB {
return db.Model(&Event{}).
return tx.Model(&Event{}).
Select("id").
Where("created_at < ?", cutoff)
}
expiredDeliveryIDs := func() *gorm.DB {
return db.Model(&Delivery{}).
return tx.Model(&Delivery{}).
Select("id").
Where("event_id IN (?)", expiredEventIDs())
}
// 1. Delivery results whose delivery belongs to an expired event.
res := db.Unscoped().
res := tx.Unscoped().
Where("delivery_id IN (?)", expiredDeliveryIDs()).
Delete(&DeliveryResult{})
if res.Error != nil {
return 0, fmt.Errorf(
return removed, fmt.Errorf(
"deleting expired delivery results: %w",
res.Error,
)
}
// 2. Deliveries belonging to an expired event.
del := db.Unscoped().
// 2. Deliveries belonging to an expired event, after counting the
// failed ones among them. The status is tested in the select list
// rather than the WHERE clause: there, SQLite would read every
// failed delivery the webhook has through the status index,
// instead of only the expired ones through the event_id index.
var failed struct{ Count int64 }
err := tx.Unscoped().Model(&Delivery{}).
Select("count(CASE WHEN status = ? THEN 1 END) AS count",
DeliveryStatusFailed).
Where("event_id IN (?)", expiredEventIDs()).
Take(&failed).Error
if err != nil {
return removed, fmt.Errorf(
"counting expired failed deliveries: %w", err,
)
}
del := tx.Unscoped().
Where("event_id IN (?)", expiredEventIDs()).
Delete(&Delivery{})
if del.Error != nil {
return 0, fmt.Errorf(
return removed, fmt.Errorf(
"deleting expired deliveries: %w",
del.Error,
)
}
// 3. The expired events themselves.
ev := db.Unscoped().
ev := tx.Unscoped().
Where("created_at < ?", cutoff).
Delete(&Event{})
if ev.Error != nil {
return 0, fmt.Errorf(
return removed, fmt.Errorf(
"deleting expired events: %w",
ev.Error,
)
}
return ev.RowsAffected, nil
removed.EventsRemoved = ev.RowsAffected
removed.DeliveriesRemoved = del.RowsAffected
removed.FailuresRemoved = failed.Count
return removed, nil
}
+126
View File
@@ -0,0 +1,126 @@
package database_test
import (
"context"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// readTotals reads a webhook database's row of running totals,
// asserting that it has exactly one.
func readTotals(t *testing.T, db *gorm.DB) database.Totals {
t.Helper()
var rows []database.Totals
require.NoError(t, db.Find(&rows).Error)
require.Len(t, rows, 1)
return rows[0]
}
// TestWebhookDBManager_TotalsRowSurvivesReopen verifies that a new
// event database starts with one row of zero totals, and that opening
// it again keeps that row and what was added to it.
func TestWebhookDBManager_TotalsRowSurvivesReopen(t *testing.T) {
t.Parallel()
mgr, lc := setupTestWebhookDBManager(t)
ctx := context.Background()
require.NoError(t, lc.Start(ctx))
defer func() { require.NoError(t, lc.Stop(ctx)) }()
webhookID := uuid.New().String()
db, err := mgr.GetDB(webhookID)
require.NoError(t, err)
fresh := readTotals(t, db)
assert.Equal(t, database.Totals{ID: fresh.ID}, fresh)
require.NoError(t, database.AddTotals(db, database.Totals{
Events: 2, Deliveries: 3, Failures: 1,
}))
// Drop the cached connection so the next open reopens the file,
// as a restart would.
require.NoError(t, mgr.CloseAll())
db, err = mgr.GetDB(webhookID)
require.NoError(t, err)
assert.Equal(t, database.Totals{
ID: fresh.ID, Events: 2, Deliveries: 3, Failures: 1,
}, readTotals(t, db))
}
// TestRetentionReaper_AddsWhatItRemovesToTotals verifies that a sweep
// leaves the lifetime totals alone and adds the events, deliveries and
// failed deliveries it deletes to the removed totals, so the totals
// within retention match the rows still stored.
func TestRetentionReaper_AddsWhatItRemovesToTotals(t *testing.T) {
t.Parallel()
env := setupRetentionTest(t)
webhookID := createWebhook(t, env.mainDB.DB(), 30)
db, err := env.mgr.GetDB(webhookID)
require.NoError(t, err)
now := time.Now()
expired := now.Add(-40 * 24 * time.Hour)
seedEventChain(t, db, webhookID, expired)
expiredFailure := seedEventChain(t, db, webhookID, expired)
recentFailure := seedEventChain(
t, db, webhookID, now.Add(-24*time.Hour),
)
for _, id := range []string{
expiredFailure.deliveryID, recentFailure.deliveryID,
} {
require.NoError(t, db.Model(&database.Delivery{}).
Where("id = ?", id).
Update("status", database.DeliveryStatusFailed).Error)
}
// The totals storing those rows would have left.
require.NoError(t, database.AddTotals(db, database.Totals{
Events: 3, Deliveries: 3, Failures: 2,
}))
env.reaper.ExportSweep(context.Background())
totals := readTotals(t, db)
assert.Equal(t, database.Totals{
ID: totals.ID,
Events: 3, Deliveries: 3, Failures: 2,
EventsRemoved: 2, DeliveriesRemoved: 2, FailuresRemoved: 1,
}, totals)
var events, deliveries, failures int64
require.NoError(t, db.Model(&database.Event{}).Count(&events).Error)
require.NoError(t, db.Model(&database.Delivery{}).
Count(&deliveries).Error)
require.NoError(t, db.Model(&database.Delivery{}).
Where("status = ?", database.DeliveryStatusFailed).
Count(&failures).Error)
assert.Equal(t, events, totals.EventsWithinRetention())
assert.Equal(t, deliveries, totals.DeliveriesWithinRetention())
assert.Equal(t, failures, totals.FailuresWithinRetention())
// A sweep with nothing left to remove changes nothing.
env.reaper.ExportSweep(context.Background())
assert.Equal(t, totals, readTotals(t, db))
}
+14 -2
View File
@@ -35,7 +35,8 @@ var errInvalidCachedDBType = errors.New(
// WebhookDBManager manages per-webhook SQLite database files
// for event storage. Each webhook gets its own dedicated
// database containing Events, Deliveries, and DeliveryResults.
// database containing Events, Deliveries, DeliveryResults and the
// running Totals of them.
// Database connections are opened lazily and cached.
type WebhookDBManager struct {
dataDir string
@@ -294,7 +295,7 @@ func (m *WebhookDBManager) openDB(
// Run migrations for event-tier models only
err = db.AutoMigrate(
&Event{}, &Delivery{}, &DeliveryResult{},
&Event{}, &Delivery{}, &DeliveryResult{}, &Totals{},
)
if err != nil {
_ = sqlDB.Close()
@@ -305,6 +306,17 @@ func (m *WebhookDBManager) openDB(
)
}
// A new database gets its row of running totals, all zero.
err = db.FirstOrCreate(&Totals{}).Error
if err != nil {
_ = sqlDB.Close()
return nil, fmt.Errorf(
"creating running totals for webhook database %s: %w",
webhookID, err,
)
}
m.log.Info(
"opened per-webhook database",
"webhook_id", webhookID,
+100
View File
@@ -0,0 +1,100 @@
package delivery_test
import (
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// failureTotal reads the running failure total of a webhook database.
func failureTotal(t *testing.T, db *gorm.DB) int64 {
t.Helper()
var totals database.Totals
require.NoError(t, db.Take(&totals).Error)
return totals.Failures
}
// TestUpdateDeliveryStatus_FinishTimeAndFailureTotal pins what a status
// write records for the webhook page's statistics: the time a delivery
// finished, set only when it becomes delivered or failed, and one more
// on the failure total when it fails.
func TestUpdateDeliveryStatus_FinishTimeAndFailureTotal(t *testing.T) {
t.Parallel()
tests := []struct {
status database.DeliveryStatus
finished bool
failures int64
}{
{database.DeliveryStatusRetrying, false, 0},
{database.DeliveryStatusDelivered, true, 0},
{database.DeliveryStatusFailed, true, 1},
}
for _, tt := range tests {
t.Run(string(tt.status), func(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
e := testEngine(t, 1)
event := seedEvent(t, db, `{}`)
d := seedDelivery(
t, db, event.ID, uuid.New().String(),
database.DeliveryStatusPending,
)
before := time.Now()
require.NoError(t, e.ExportUpdateDeliveryStatus(
db, &d, tt.status,
))
var stored database.Delivery
require.NoError(t, db.First(&stored, "id = ?", d.ID).Error)
assert.Equal(t, tt.status, stored.Status)
if tt.finished {
require.NotNil(t, stored.FinishedAt)
assert.False(t, stored.FinishedAt.Before(before))
} else {
assert.Nil(t, stored.FinishedAt)
}
assert.Equal(t, tt.failures, failureTotal(t, db))
})
}
}
// TestUpdateDeliveryStatus_DeletedDeliveryIsNotCounted covers a
// delivery retention deleted while the engine still held it. Failing
// it afterwards writes no row, so it adds no failure either: retention
// has already counted what it removed.
func TestUpdateDeliveryStatus_DeletedDeliveryIsNotCounted(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
e := testEngine(t, 1)
event := seedEvent(t, db, `{}`)
d := seedDelivery(
t, db, event.ID, uuid.New().String(),
database.DeliveryStatusRetrying,
)
require.NoError(t, db.Unscoped().
Delete(&database.Delivery{}, "id = ?", d.ID).Error)
require.NoError(t, e.ExportUpdateDeliveryStatus(
db, &d, database.DeliveryStatusFailed,
))
assert.Zero(t, failureTotal(t, db))
}
+30 -2
View File
@@ -1554,8 +1554,9 @@ func (e *Engine) updateDeliveryStatus(
targetType database.TargetType,
status database.DeliveryStatus,
) error {
err := webhookDB.Model(d).
Update("status", status).Error
err := webhookDB.Transaction(func(tx *gorm.DB) error {
return writeDeliveryStatus(tx, d, status)
})
if err != nil {
return fmt.Errorf(
"updating delivery %s to status %s: %w",
@@ -1574,6 +1575,33 @@ func (e *Engine) updateDeliveryStatus(
return nil
}
// writeDeliveryStatus writes a delivery's new status. A delivery that
// becomes delivered or failed also gets the time it finished, and a
// failed one is added to the webhook's running failure total. The
// failure is counted only if the row was still there to update:
// retention may have deleted it while the engine was working on it.
func writeDeliveryStatus(
tx *gorm.DB,
d *database.Delivery,
status database.DeliveryStatus,
) error {
columns := map[string]any{"status": status}
if status.Terminal() {
columns["finished_at"] = time.Now()
}
res := tx.Model(d).Updates(columns)
if res.Error != nil {
return res.Error
}
if status == database.DeliveryStatusFailed && res.RowsAffected > 0 {
return database.AddTotals(tx, database.Totals{Failures: 1})
}
return nil
}
// settleStatus moves a delivery to its outcome status and reports a
// failed write through bookkeepingFailed, which leaves the row
// recoverable. It exists so the target call sites read as one
+2
View File
@@ -57,7 +57,9 @@ func testWebhookDB(t *testing.T) *gorm.DB {
&database.Event{},
&database.Delivery{},
&database.DeliveryResult{},
&database.Totals{},
))
require.NoError(t, db.Create(&database.Totals{}).Error)
return db
}
+10
View File
@@ -150,6 +150,16 @@ func (e *Engine) ExportDeliverSlack(
)
}
// ExportUpdateDeliveryStatus exposes updateDeliveryStatus. It passes no
// target type, so no metric moves.
func (e *Engine) ExportUpdateDeliveryStatus(
webhookDB *gorm.DB,
d *database.Delivery,
status database.DeliveryStatus,
) error {
return e.updateDeliveryStatus(webhookDB, d, "", status)
}
// ExportProcessNewTask exposes processNewTask.
func (e *Engine) ExportProcessNewTask(
ctx context.Context, task *Task,
+30
View File
@@ -453,3 +453,33 @@ func TestLogin_SuccessCreatesSession(t *testing.T) {
"the issued cookie must carry an authenticated session",
)
}
// TestLogin_UsernameAtLimitCanLogIn shows that a username of exactly
// database.MaxUsernameBytes still fits in the session cookie. Past
// what the cookie can carry, a correct login answers 500.
func TestLogin_UsernameAtLimitCanLogIn(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
db *database.Database
)
app := newTestApp(t, &h, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
username := strings.Repeat("a", database.MaxUsernameBytes)
hash, err := database.HashPassword(operatorPassword)
require.NoError(t, err)
require.NoError(t, db.DB().Create(&database.User{
Username: username,
Password: hash,
}).Error)
w := submitLogin(h, sharedProxyPeer, username, operatorPassword)
assert.Equal(t, http.StatusSeeOther, w.Code)
}
+10 -2
View File
@@ -299,7 +299,8 @@ func countInFlightDeliveries(
return count, err
}
// createReplayDelivery writes the new pending delivery row and returns
// createReplayDelivery writes the new pending delivery row, adds it to
// the webhook's running totals in the same transaction, and returns
// the task that carries it to the delivery engine.
//
// The row is written with associations omitted, and neither Event nor
@@ -319,7 +320,14 @@ func createReplayDelivery(
Status: database.DeliveryStatusPending,
}
err := webhookDB.Omit(clause.Associations).Create(dlv).Error
err := webhookDB.Transaction(func(tx *gorm.DB) error {
err := tx.Omit(clause.Associations).Create(dlv).Error
if err != nil {
return err
}
return database.AddTotals(tx, database.Totals{Deliveries: 1})
})
if err != nil {
return delivery.Task{}, err
}
+15
View File
@@ -69,6 +69,21 @@ func (s *Handlers) LoadEventLogViewsForTest(
return views
}
// WebhookStatsForTest returns the figures the statistics pane on a
// webhook's page shows, from the webhook's entrypoints and targets
// loaded as that page loads them.
func (s *Handlers) WebhookStatsForTest(webhookID string) *WebhookStats {
var entrypoints []database.Entrypoint
s.db.DB().Where("webhook_id = ?", webhookID).Find(&entrypoints)
var targets []database.Target
s.db.DB().Where("webhook_id = ?", webhookID).Find(&targets)
return s.loadWebhookStats(webhookID, entrypoints, targets)
}
// AddTemplateForTest registers a template under a page name so that
// the handlers_test package can drive the render path with a
// template of its own.
+16 -12
View File
@@ -91,18 +91,22 @@ type Handlers struct {
// parsePageTemplate parses a page-specific template set from the
// embedded FS. Each page template is combined with the shared
// base, htmlheader, and navbar templates. The page file must be
// listed first so that its root action ({{template "base" .}})
// becomes the template set's entry point.
func parsePageTemplate(pageFile string) *template.Template {
// base, htmlheader, and navbar templates, and with any further files
// the page includes. The page file must be listed first so that its
// root action ({{template "base" .}}) becomes the template set's entry
// point.
func parsePageTemplate(
pageFile string, included ...string,
) *template.Template {
files := append([]string{
pageFile,
"base.html",
"htmlheader.html",
"navbar.html",
}, included...)
return template.Must(
template.ParseFS(
templates.Templates,
pageFile,
"base.html",
"htmlheader.html",
"navbar.html",
),
template.ParseFS(templates.Templates, files...),
)
}
@@ -131,7 +135,7 @@ func New(
"profile.html": parsePageTemplate("profile.html"),
"sources_list.html": parsePageTemplate("sources_list.html"),
"sources_new.html": parsePageTemplate("sources_new.html"),
"source_detail.html": parsePageTemplate("source_detail.html"),
"source_detail.html": parsePageTemplate("source_detail.html", "webhook_stats.html"),
"source_edit.html": parsePageTemplate("source_edit.html"),
"source_logs.html": parsePageTemplate("source_logs.html"),
"target_edit.html": parsePageTemplate("target_edit.html"),
+3 -5
View File
@@ -339,11 +339,9 @@ const storedUserPassword = "correct-horse-battery-staple"
// storedFillBytes is the raw length of the client-chosen value in
// those accounts' usernames. It is well past the 512-byte field
// budget, so the line is still truncated, but short enough that the
// session cookie a successful login writes stays inside
// securecookie's 4 KB limit: the cookie is written BEFORE the
// "user logged in" line, so an 8 KB username answers 500 and never
// reaches it.
const storedFillBytes = 1024
// whole username, markers and fill name included, stays within
// database.MaxUsernameBytes.
const storedFillBytes = 960
// storedFill builds a username fill of storedFillBytes raw bytes out
// of repetitions of ch, with both markers at its far end.
+1
View File
@@ -450,6 +450,7 @@ func (h *Handlers) renderSourceDetail(
"Targets": delivery.NewTargetViews(targets),
"Events": events,
"BaseURL": baseURL,
"Stats": h.loadWebhookStats(webhook.ID, entrypoints, targets),
}
h.renderTemplate(w, r, "source_detail.html", data)
+16 -5
View File
@@ -252,11 +252,12 @@ func requestEventSource(
}
}
// createAndFanOut writes the event and one pending delivery per target
// in a single transaction, then hands the tasks to the delivery
// engine. It is the only path by which an event and its deliveries are
// created, so a resubmitted event is retried, SSRF-guarded and
// circuit-broken exactly as a received one is.
// createAndFanOut writes the event and one pending delivery per target,
// and adds them to the webhook's running totals, in a single
// transaction, then hands the tasks to the delivery engine. It is the
// only path by which an event and its deliveries are created, so a
// resubmitted event is retried, SSRF-guarded and circuit-broken
// exactly as a received one is.
//
// The tasks are returned as well as queued, so a caller can report how
// many targets the event went to.
@@ -296,6 +297,16 @@ func (h *Handlers) createAndFanOut(
return nil, nil, err
}
err = database.AddTotals(tx, database.Totals{
Events: 1,
Deliveries: int64(len(tasks)),
})
if err != nil {
tx.Rollback()
return nil, nil, err
}
err = tx.Commit().Error
if err != nil {
return nil, nil, fmt.Errorf(
+212
View File
@@ -0,0 +1,212 @@
package handlers
import (
"fmt"
"time"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// The spans of the two recent windows the statistics pane reports on:
// the last 10 minutes and the last 24 hours.
const (
shortWindow = 10 * time.Minute
longWindow = 24 * time.Hour
)
// percent turns a fraction into a percentage.
const percent = 100
// WebhookStats holds the figures in the statistics pane at the top of
// the webhook page.
type WebhookStats struct {
Entrypoints int
ActiveEntrypoints int
Targets int
ActiveTargets int
// Totals holds the lifetime counts of events, deliveries and
// failures, and how many of each retention has removed.
Totals database.Totals
// InProgress counts the deliveries still pending or retrying.
InProgress int64
// LastEventAt is when the newest stored event arrived, or nil when
// none is stored.
LastEventAt *time.Time
Last10Minutes RecentWindow
Last24Hours RecentWindow
}
// RecentWindow holds what happened in one recent window: the events
// received in it, and the deliveries that became delivered or failed in
// it.
type RecentWindow struct {
Events int64
Delivered int64
Failed int64
}
// FailurePercent is the share of the deliveries finished in the window
// that failed, or a dash when none finished. Deliveries still pending
// or retrying are not counted either way.
func (w RecentWindow) FailurePercent() string {
finished := w.Delivered + w.Failed
if finished == 0 {
return "—"
}
return fmt.Sprintf(
"%.1f%%", percent*float64(w.Failed)/float64(finished),
)
}
// loadWebhookStats gathers the figures for the statistics pane from the
// webhook's entrypoints and targets, as the page has already loaded
// them, and from its event database. It returns nil, and logs why, when
// the event database cannot be read.
func (h *Handlers) loadWebhookStats(
webhookID string,
entrypoints []database.Entrypoint,
targets []database.Target,
) *WebhookStats {
stats := &WebhookStats{
Entrypoints: len(entrypoints),
Targets: len(targets),
}
for i := range entrypoints {
if entrypoints[i].Active {
stats.ActiveEntrypoints++
}
}
for i := range targets {
if targets[i].Active {
stats.ActiveTargets++
}
}
// Opening an event database that does not exist would create it,
// and it would hold nothing to count.
if !h.dbMgr.DBExists(webhookID) {
return stats
}
webhookDB, err := h.dbMgr.GetDB(webhookID)
if err == nil {
err = readEventStats(webhookDB, time.Now(), stats)
}
if err != nil {
h.log.Error(
"failed to read webhook statistics",
"webhook_id", webhookID,
"error", err,
)
return nil
}
return stats
}
// readEventStats fills in the figures that come from the webhook's
// event database. None of them reads every stored row: the totals are
// one row, and every other figure is read from an index, over only the
// rows it counts.
func readEventStats(
db *gorm.DB, now time.Time, stats *WebhookStats,
) error {
err := db.Take(&stats.Totals).Error
if err != nil {
return fmt.Errorf("reading running totals: %w", err)
}
err = db.Model(&database.Delivery{}).
Where("status IN ?", []database.DeliveryStatus{
database.DeliveryStatusPending,
database.DeliveryStatusRetrying,
}).
Count(&stats.InProgress).Error
if err != nil {
return fmt.Errorf("counting deliveries in progress: %w", err)
}
var newest []time.Time
err = db.Model(&database.Event{}).
Order("created_at DESC").
Limit(1).
Pluck("created_at", &newest).Error
if err != nil {
return fmt.Errorf("reading newest event time: %w", err)
}
if len(newest) > 0 {
stats.LastEventAt = &newest[0]
}
stats.Last10Minutes, err = readRecentWindow(
db, now.Add(-shortWindow),
)
if err != nil {
return err
}
stats.Last24Hours, err = readRecentWindow(
db, now.Add(-longWindow),
)
return err
}
// readRecentWindow counts the events received, and the deliveries that
// became delivered or failed, since the given time.
func readRecentWindow(
db *gorm.DB, since time.Time,
) (RecentWindow, error) {
var w RecentWindow
err := db.Model(&database.Event{}).
Where("created_at >= ?", since).
Count(&w.Events).Error
if err != nil {
return w, fmt.Errorf("counting recent events: %w", err)
}
w.Delivered, err = countFinishedSince(
db, database.DeliveryStatusDelivered, since,
)
if err != nil {
return w, err
}
w.Failed, err = countFinishedSince(
db, database.DeliveryStatusFailed, since,
)
return w, err
}
// countFinishedSince counts the deliveries that reached the given
// final status since the given time.
func countFinishedSince(
db *gorm.DB, status database.DeliveryStatus, since time.Time,
) (int64, error) {
var n int64
err := db.Model(&database.Delivery{}).
Where("status = ? AND finished_at >= ?", status, since).
Count(&n).Error
if err != nil {
return 0, fmt.Errorf(
"counting deliveries %s recently: %w", status, err,
)
}
return n, nil
}
+328
View File
@@ -0,0 +1,328 @@
package handlers_test
import (
"net/http"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/fx/fxtest"
"gorm.io/gorm"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/logger"
"sneak.berlin/go/webhooker/internal/session"
)
// statsEntrypoint adds an entrypoint to a webhook and returns its path.
func statsEntrypoint(
t *testing.T, db *database.Database, webhookID string, active bool,
) string {
t.Helper()
ep := &database.Entrypoint{
WebhookID: webhookID,
Path: uuid.New().String(),
}
require.NoError(t, db.DB().Omit(clause.Associations).Create(ep).Error)
require.NoError(t, db.DB().Model(ep).Update("active", active).Error)
return ep.Path
}
// statsDelivery returns the id of an event's delivery to a target.
func statsDelivery(
t *testing.T, webhookDB *gorm.DB, eventID, targetID string,
) string {
t.Helper()
var d database.Delivery
require.NoError(t, webhookDB.Where(
"event_id = ? AND target_id = ?", eventID, targetID,
).First(&d).Error)
return d.ID
}
// statsFinish settles a delivery as the delivery engine does: its
// final status and the time it finished, and for a failure one more on
// the webhook's failure total, in one transaction.
func statsFinish(
t *testing.T,
webhookDB *gorm.DB,
deliveryID string,
status database.DeliveryStatus,
at time.Time,
) {
t.Helper()
require.NoError(t, webhookDB.Transaction(func(tx *gorm.DB) error {
err := tx.Model(&database.Delivery{}).
Where("id = ?", deliveryID).
Updates(map[string]any{"status": status, "finished_at": at}).
Error
if err != nil || status != database.DeliveryStatusFailed {
return err
}
return database.AddTotals(tx, database.Totals{Failures: 1})
}))
}
// statsAge moves an event's arrival back to the given time.
func statsAge(
t *testing.T, webhookDB *gorm.DB, eventID string, at time.Time,
) {
t.Helper()
require.NoError(t, webhookDB.Model(&database.Event{}).
Where("id = ?", eventID).
Update("created_at", at).Error)
}
// seedStatsHistory builds the webhook the statistics test checks: one
// day of retention, two entrypoints (one inactive) and three targets
// (one inactive). Three events arrive through the receiver, and so
// each has a delivery to the two active targets. The oldest event is
// past retention, the middle one six hours old, the newest just in.
// Their deliveries are settled as the delivery engine would, and a
// replay adds a pending delivery to the oldest event. It returns the
// webhook, its event database and the newest event.
func seedStatsHistory(
t *testing.T,
h *handlers.Handlers,
sess *session.Session,
db *database.Database,
dbMgr *database.WebhookDBManager,
) (*database.Webhook, *gorm.DB, database.Event) {
t.Helper()
wh := &database.Webhook{
UserID: deleteTestUserID, Name: "stats", RetentionDays: 1,
}
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
path := statsEntrypoint(t, db, wh.ID, true)
statsEntrypoint(t, db, wh.ID, false)
first := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
`{"url":"`+replayTargetURL+`"}`,
)
second := seedTarget(t, db, wh.ID, database.TargetTypeLog)
inactive := seedTarget(t, db, wh.ID, database.TargetTypeLog)
require.NoError(t, db.DB().Model(inactive).
Update("active", false).Error)
router := receiverRouter(h)
for range 3 {
require.Equal(t, http.StatusOK, postReceiver(t, router, path))
}
webhookDB, err := dbMgr.GetDB(wh.ID)
require.NoError(t, err)
events := listEvents(t, webhookDB)
require.Len(t, events, 3)
oldest, middle, newest := events[0], events[1], events[2]
now := time.Now()
statsAge(t, webhookDB, oldest.ID, now.Add(-50*time.Hour))
statsAge(t, webhookDB, middle.ID, now.Add(-6*time.Hour))
oldestFailure := statsDelivery(t, webhookDB, oldest.ID, first.ID)
statsFinish(t, webhookDB, oldestFailure,
database.DeliveryStatusFailed, now.Add(-49*time.Hour))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, oldest.ID, second.ID),
database.DeliveryStatusDelivered, now.Add(-49*time.Hour))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, middle.ID, first.ID),
database.DeliveryStatusFailed, now.Add(-5*time.Hour))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, middle.ID, second.ID),
database.DeliveryStatusFailed, now.Add(-time.Minute))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, newest.ID, first.ID),
database.DeliveryStatusDelivered, now.Add(-2*time.Minute))
require.Equal(t, http.StatusSeeOther,
postReplay(t, h, sess, wh.ID, oldestFailure).Code)
return wh, webhookDB, newest
}
// statsPrune runs the real retention reaper until it has removed one
// event from the webhook's database, then stops it.
func statsPrune(
t *testing.T,
db *database.Database,
dbMgr *database.WebhookDBManager,
log *logger.Logger,
webhookDB *gorm.DB,
) {
t.Helper()
lc := fxtest.NewLifecycle(t)
database.NewRetentionReaper(lc, database.RetentionReaperParams{
Config: &config.Config{
RetentionSweepInterval: 10 * time.Millisecond,
},
Database: db,
DBManager: dbMgr,
Logger: log,
})
lc.RequireStart()
require.Eventually(t, func() bool {
var totals database.Totals
err := webhookDB.Take(&totals).Error
return err == nil && totals.EventsRemoved == 1
}, 10*time.Second, 10*time.Millisecond)
lc.RequireStop()
}
// assertStatsTotals checks the lifetime events, deliveries and
// failures, and those within retention.
func assertStatsTotals(
t *testing.T, totals database.Totals, lifetime, within [3]int64,
) {
t.Helper()
gotLifetime := [3]int64{
totals.Events, totals.Deliveries, totals.Failures,
}
gotWithin := [3]int64{
totals.EventsWithinRetention(),
totals.DeliveriesWithinRetention(),
totals.FailuresWithinRetention(),
}
assert.Equal(t, lifetime, gotLifetime,
"lifetime events, deliveries, failures")
assert.Equal(t, within, gotWithin,
"events, deliveries, failures within retention")
}
// TestWebhookStats_EveryFigureAcrossRetentionPrune checks every figure
// the statistics pane shows for the history seedStatsHistory builds,
// before and after the real retention reaper removes the oldest event.
func TestWebhookStats_EveryFigureAcrossRetentionPrune(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
log *logger.Logger
)
app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh, webhookDB, newest := seedStatsHistory(t, h, sess, db, dbMgr)
stats := h.WebhookStatsForTest(wh.ID)
require.NotNil(t, stats)
assert.Equal(t, 2, stats.Entrypoints)
assert.Equal(t, 1, stats.ActiveEntrypoints)
assert.Equal(t, 3, stats.Targets)
assert.Equal(t, 2, stats.ActiveTargets)
assertStatsTotals(t, stats.Totals, [3]int64{3, 7, 3}, [3]int64{3, 7, 3})
assert.Equal(t, int64(2), stats.InProgress)
require.NotNil(t, stats.LastEventAt)
assert.True(t, newest.CreatedAt.Equal(*stats.LastEventAt))
assert.Equal(t, handlers.RecentWindow{
Events: 1, Delivered: 1, Failed: 1,
}, stats.Last10Minutes)
assert.Equal(t, handlers.RecentWindow{
Events: 2, Delivered: 1, Failed: 2,
}, stats.Last24Hours)
assert.Equal(t, "50.0%", stats.Last10Minutes.FailurePercent())
assert.Equal(t, "66.7%", stats.Last24Hours.FailurePercent())
// Retention removes the oldest event with its three deliveries,
// one of them failed and one the pending replay.
statsPrune(t, db, dbMgr, log, webhookDB)
after := h.WebhookStatsForTest(wh.ID)
require.NotNil(t, after)
assertStatsTotals(t, after.Totals, [3]int64{3, 7, 3}, [3]int64{2, 4, 2})
assert.Equal(t, int64(1), after.InProgress)
assert.Equal(t, stats.LastEventAt, after.LastEventAt)
assert.Equal(t, stats.Last10Minutes, after.Last10Minutes)
assert.Equal(t, stats.Last24Hours, after.Last24Hours)
body := renderSourceDetailPage(t, h, sess, wh.ID)
assert.Contains(t, body, "Statistics")
assert.Contains(t, body, "Within retention")
assert.Contains(t, body, "50.0%")
assert.Contains(t, body, "66.7%")
}
// TestWebhookStats_WebhookWithNoEvents covers a webhook whose event
// database has never been opened: every count is zero, the
// percentages are a dash, and showing the page does not create the
// database.
func TestWebhookStats_WebhookWithNoEvents(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
assert.Equal(t, &handlers.WebhookStats{}, h.WebhookStatsForTest(wh.ID))
assert.Equal(t, "—", handlers.RecentWindow{}.FailurePercent())
body := renderSourceDetailPage(t, h, sess, wh.ID)
assert.Contains(t, body, "Statistics")
assert.False(t, dbMgr.DBExists(wh.ID))
}
// TestRecentWindow_FailurePercent pins the percentage: failed
// deliveries out of all that finished in the window.
func TestRecentWindow_FailurePercent(t *testing.T) {
t.Parallel()
tests := []struct {
window handlers.RecentWindow
want string
}{
{handlers.RecentWindow{}, "—"},
{handlers.RecentWindow{Events: 4}, "—"},
{handlers.RecentWindow{Delivered: 3, Failed: 1}, "25.0%"},
{handlers.RecentWindow{Failed: 2}, "100.0%"},
{handlers.RecentWindow{Delivered: 2}, "0.0%"},
}
for _, tt := range tests {
assert.Equal(t, tt.want, tt.window.FailurePercent(), tt.window)
}
}
+17 -3
View File
@@ -92,11 +92,25 @@ func (s *Server) setupGlobalMiddleware() {
func (s *Server) setupRoutes() {
s.router.Get("/", s.h.HandleIndex())
s.router.Mount(
"/s",
http.StripPrefix("/s", http.FileServer(http.FS(static.Static))),
// Static assets answer GET and HEAD only. chi's default 405
// carries no Allow header, so this group supplies its own.
staticFiles := http.StripPrefix(
"/s", http.FileServer(http.FS(static.Static)),
)
s.router.Route("/s", func(r chi.Router) {
r.MethodNotAllowed(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Allow", "GET, HEAD")
http.Error(
w,
"Method Not Allowed",
http.StatusMethodNotAllowed,
)
})
r.Method(http.MethodGet, "/*", staticFiles)
r.Method(http.MethodHead, "/*", staticFiles)
})
s.router.Route("/api/v1", func(_ chi.Router) {
// API routes will be added here.
})
+108 -19
View File
@@ -7,6 +7,7 @@ import (
"net/http/httptest"
"net/url"
"regexp"
"slices"
"strconv"
"strings"
"testing"
@@ -220,9 +221,21 @@ func (e *testEnv) csrfFrom(
// out of the markup has to be unescaped before it is submitted.
token := html.UnescapeString(match[1])
combined := make([]*http.Cookie, 0, len(cookies))
combined = append(combined, cookies...)
combined = append(combined, w.Result().Cookies()...)
// A cookie the page sets replaces the one of the same name, as in
// a browser. Sent both, the server would read the first, older one.
set := w.Result().Cookies()
combined := make([]*http.Cookie, 0, len(cookies)+len(set))
for _, c := range cookies {
replaced := slices.ContainsFunc(set, func(n *http.Cookie) bool {
return n.Name == c.Name
})
if !replaced {
combined = append(combined, c)
}
}
combined = append(combined, set...)
return token, combined
}
@@ -396,13 +409,15 @@ func (e *testEnv) storedHash(t *testing.T, username string) string {
// --- /s static group ---
// TestStaticServesEveryMethod pins what the static mount actually
// answers. chi's Mount registers the handler for all methods and
// http.FileServer only special-cases HEAD (by suppressing the body),
// so a POST or a DELETE to an asset is served the file rather than
// refused. The README documents this; the test is what keeps the two
// from drifting.
func TestStaticServesEveryMethod(t *testing.T) {
// TestStaticServesOnlyGetAndHead pins the methods the static group
// answers: GET and HEAD are served the asset, and the other methods
// chi routes (POST, PUT, DELETE and the rest) are refused with 405
// and an Allow header naming those two. A method chi does not route,
// such as PROPFIND, is refused with 405 by the top-level router
// before it reaches the static group, so it gets no Allow header.
// The README documents this; the test is what keeps the two from
// drifting.
func TestStaticServesOnlyGetAndHead(t *testing.T) {
t.Parallel()
env := newTestEnv(t)
@@ -417,6 +432,7 @@ func TestStaticServesEveryMethod(t *testing.T) {
http.MethodPost,
http.MethodPut,
http.MethodDelete,
"PROPFIND",
} {
t.Run(method, func(t *testing.T) {
t.Parallel()
@@ -428,18 +444,38 @@ func TestStaticServesEveryMethod(t *testing.T) {
w := httptest.NewRecorder()
env.router.ServeHTTP(w, req)
assert.Equal(t, http.StatusOK, w.Code,
"static mount answers every method")
if method == http.MethodHead {
switch method {
case http.MethodGet:
assert.Equal(t, http.StatusOK, w.Code)
assert.Equal(t, body, w.Body.Bytes(),
"the asset itself is returned")
case http.MethodHead:
assert.Equal(t, http.StatusOK, w.Code)
assert.Empty(t, w.Body.Bytes(),
"HEAD must not carry a body")
return
case "PROPFIND":
assert.Equal(
t, http.StatusMethodNotAllowed, w.Code,
)
assert.Empty(t, w.Header().Get("Allow"),
"chi refuses a method it does not route "+
"before the static group runs")
assert.NotContains(
t, w.Body.String(), string(body),
"a refused method must not get the asset",
)
default:
assert.Equal(
t, http.StatusMethodNotAllowed, w.Code,
)
assert.Equal(
t, "GET, HEAD", w.Header().Get("Allow"),
)
assert.NotContains(
t, w.Body.String(), string(body),
"a refused method must not get the asset",
)
}
assert.Equal(t, body, w.Body.Bytes(),
"the asset itself is returned")
})
}
}
@@ -591,6 +627,59 @@ func TestPagesLogin_CorrectPasswordSurvivesASpentBudget(
)
}
// TestPagesLogin_CookiesFromAnEarlierDatabase is
// https://git.eeqj.de/sneak/webhooker/issues/359. A new database
// brings a new session key, and the operator's browser still holds
// the session and CSRF cookies signed with the old one. Logging in
// must work as from a fresh browser and leave cookies the new key
// accepts.
func TestPagesLogin_CookiesFromAnEarlierDatabase(t *testing.T) {
t.Parallel()
const (
username = "operator"
password = "correct-horse-battery-staple"
)
earlier := newTestEnv(t)
earlierID, _ := earlier.seedUser(t, username, password)
_, stale := earlier.csrfFrom(t, "/pages/login", nil)
stale = append(stale, earlier.authCookies(t, earlierID, username)...)
env := newTestEnv(t)
env.seedUser(t, username, password)
token, cookies := env.csrfFrom(t, "/pages/login", stale)
form := url.Values{}
form.Set("csrf_token", token)
form.Set("username", username)
form.Set("password", password)
w := env.post("/pages/login", form, cookies)
require.Equal(
t, http.StatusSeeOther, w.Code,
"a session cookie from another key must not fail the login",
)
// The response deletes the old session cookie and then sets the
// new one; a browser keeps the last.
var fresh *http.Cookie
for _, c := range w.Result().Cookies() {
if c.Name == session.SessionName {
fresh = c
}
}
require.NotNil(t, fresh, "login must set a session cookie")
assert.Equal(
t, "/sources",
env.get("/", []*http.Cookie{fresh}).Header().Get("Location"),
"the new session cookie must authenticate",
)
}
// --- /user/{username} group ---
// TestPasswordChange_OversizeBody_RejectedAndPasswordUnchanged
+8 -7
View File
@@ -19,8 +19,8 @@ import (
)
// The tests below exercise the securecookie codecs underneath the
// store and nothing else: Session.Get only decodes, so no server-side
// expiry check takes part in the result. They exist because
// store and nothing else: they decode through the store itself, so no
// server-side expiry check takes part in the result. They exist because
// NewCookieStore gives its codecs a 30-day max age that assigning
// store.Options does not override, which would let the codec accept a
// cookie weeks past the cap the cookie attribute advertises.
@@ -75,10 +75,11 @@ func restamp(
return base64.URLEncoding.EncodeToString(payload)
}
// decodeCookie feeds value back through the store's decode path.
// decodeCookie feeds value back through the store's decode path. It
// asks the store rather than Session.Get, which treats a cookie that
// does not decode as absent and so hides the codec's reason.
func decodeCookie(
t *testing.T,
s *session.Session,
value string,
) (*sessions.Session, error) {
t.Helper()
@@ -94,7 +95,7 @@ func decodeCookie(
SameSite: http.SameSiteLaxMode,
})
sess, err := s.Get(req)
sess, err := session.NewStore(testKey()).Get(req, session.SessionName)
require.NotNil(t, sess)
return sess, err
@@ -105,7 +106,7 @@ func TestCodec_AcceptsCookieInsideAbsoluteCap(t *testing.T) {
s := testSession(t)
sess, err := decodeCookie(t, s, restamp(
sess, err := decodeCookie(t, restamp(
t,
issuedCookie(t, s),
time.Now().Add(-(testAbsoluteMaxAge-time.Hour)),
@@ -126,7 +127,7 @@ func TestCodec_RejectsCookiePastAbsoluteCap(t *testing.T) {
s := testSession(t)
sess, err := decodeCookie(t, s, restamp(
sess, err := decodeCookie(t, restamp(
t,
issuedCookie(t, s),
time.Now().Add(-(testAbsoluteMaxAge+time.Hour)),
+13 -1
View File
@@ -224,10 +224,22 @@ func New(
}
// Get retrieves a session for the request.
//
// A session cookie that does not decode -- one signed with an earlier
// session key, say, because the database was made anew -- is treated
// as absent: the caller gets a new, empty session and no error, and
// the next save replaces the cookie.
func (s *Session) Get(
r *http.Request,
) (*sessions.Session, error) {
return s.store.Get(r, SessionName)
sess, err := s.store.Get(r, SessionName)
if sess == nil {
return nil, err
}
// For a cookie that does not decode, gorilla/sessions returns a
// new, empty session alongside the error that is dropped here.
return sess, nil
}
// GetKey returns the raw 32-byte authentication key used for
+2
View File
@@ -24,6 +24,8 @@
</div>
</div>
{{template "webhook_stats" .}}
<div class="grid grid-cols-1 lg:grid-cols-2 gap-6">
<!-- Entrypoints -->
<div class="card">
+87
View File
@@ -0,0 +1,87 @@
{{define "webhook_stats"}}
<!-- Statistics pane at the top of the webhook page. -->
<div class="card mb-6">
<div class="p-4 border-b border-gray-200">
<h2 class="text-lg font-medium text-gray-900">Statistics</h2>
</div>
{{with .Stats}}
<div class="p-4 flex flex-wrap gap-6 text-sm border-b border-gray-200">
<div>
<span class="text-gray-500">Entrypoints</span>
<span class="font-medium text-gray-900">{{.Entrypoints}}</span>
<span class="text-gray-500">({{.ActiveEntrypoints}} active)</span>
</div>
<div>
<span class="text-gray-500">Targets</span>
<span class="font-medium text-gray-900">{{.Targets}}</span>
<span class="text-gray-500">({{.ActiveTargets}} active)</span>
</div>
<div>
<span class="text-gray-500">Deliveries in progress</span>
<span class="font-medium text-gray-900">{{.InProgress}}</span>
</div>
<div>
<span class="text-gray-500">Last event</span>
<span class="font-medium text-gray-900">{{with .LastEventAt}}{{.Format "2006-01-02 15:04:05 UTC"}}{{else}}none{{end}}</span>
</div>
<div>
<span class="text-gray-500">Retention</span>
<span class="font-medium text-gray-900">{{$.Webhook.RetentionLabel}}</span>
</div>
</div>
<div class="p-4 grid grid-cols-1 lg:grid-cols-2 gap-6 text-sm">
<div>
<div class="flex py-2 border-b border-gray-200 text-xs text-gray-500 uppercase tracking-wide">
<span class="flex-1"></span>
<span class="w-32 text-center">Lifetime</span>
<span class="w-32 text-center">Within retention</span>
</div>
<div class="divide-y divide-gray-100">
<div class="flex py-2">
<span class="flex-1 text-gray-600">Events</span>
<span class="w-32 text-center text-gray-900">{{.Totals.Events}}</span>
<span class="w-32 text-center text-gray-900">{{.Totals.EventsWithinRetention}}</span>
</div>
<div class="flex py-2">
<span class="flex-1 text-gray-600">Deliveries</span>
<span class="w-32 text-center text-gray-900">{{.Totals.Deliveries}}</span>
<span class="w-32 text-center text-gray-900">{{.Totals.DeliveriesWithinRetention}}</span>
</div>
<div class="flex py-2">
<span class="flex-1 text-gray-600">Failures</span>
<span class="w-32 text-center text-gray-900">{{.Totals.Failures}}</span>
<span class="w-32 text-center text-gray-900">{{.Totals.FailuresWithinRetention}}</span>
</div>
</div>
</div>
<div>
<div class="flex py-2 border-b border-gray-200 text-xs text-gray-500 uppercase tracking-wide">
<span class="flex-1"></span>
<span class="w-32 text-center">Last 10 minutes</span>
<span class="w-32 text-center">Last 24 hours</span>
</div>
<div class="divide-y divide-gray-100">
<div class="flex py-2">
<span class="flex-1 text-gray-600">Events</span>
<span class="w-32 text-center text-gray-900">{{.Last10Minutes.Events}}</span>
<span class="w-32 text-center text-gray-900">{{.Last24Hours.Events}}</span>
</div>
<div class="flex py-2">
<span class="flex-1 text-gray-600">Failures</span>
<span class="w-32 text-center text-gray-900">{{.Last10Minutes.Failed}}</span>
<span class="w-32 text-center text-gray-900">{{.Last24Hours.Failed}}</span>
</div>
<div class="flex py-2">
<span class="flex-1 text-gray-600">Failure percentage</span>
<span class="w-32 text-center text-gray-900">{{.Last10Minutes.FailurePercent}}</span>
<span class="w-32 text-center text-gray-900">{{.Last24Hours.FailurePercent}}</span>
</div>
</div>
<p class="mt-2 text-xs text-gray-500">Failure percentage is the failed deliveries out of all deliveries that finished in the window. Deliveries still pending or retrying are not counted.</p>
</div>
</div>
{{else}}
<div class="p-4 text-sm text-gray-500">The statistics could not be read.</div>
{{end}}
</div>
{{end}}