Compare commits
7
Commits
prod
...
7287e163e7
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7287e163e7 | ||
|
|
507980a347 | ||
|
|
b79e4649a1 | ||
|
|
1428154bbd | ||
|
|
8ad2a86e4b | ||
|
|
a891b726e5 | ||
|
|
ab63b5f777 |
+9
-2
@@ -88,7 +88,9 @@ RUN CGO_ENABLED=1 make build VERSION="$VERSION" GO_LDFLAGS='-extldflags "-static
|
|||||||
# alpine:3.21, 2026-03-17
|
# alpine:3.21, 2026-03-17
|
||||||
FROM alpine:3.21@sha256:c3f8e73fdb79deaebaa2037150150191b9dcbfba68b4a46d70103204c53f4709
|
FROM alpine:3.21@sha256:c3f8e73fdb79deaebaa2037150150191b9dcbfba68b4a46d70103204c53f4709
|
||||||
|
|
||||||
RUN apk --no-cache add ca-certificates
|
# su-exec 0.2-r3 (Alpine 3.21), 2026-09-29: the entrypoint runs the app
|
||||||
|
# as webhooker with it.
|
||||||
|
RUN apk --no-cache add ca-certificates su-exec=0.2-r3
|
||||||
|
|
||||||
# Create non-root user
|
# Create non-root user
|
||||||
RUN addgroup -g 1000 -S webhooker && \
|
RUN addgroup -g 1000 -S webhooker && \
|
||||||
@@ -99,13 +101,17 @@ WORKDIR /app
|
|||||||
# Copy binary from builder
|
# Copy binary from builder
|
||||||
COPY --from=builder /build/bin/webhooker /app/webhooker
|
COPY --from=builder /build/bin/webhooker /app/webhooker
|
||||||
|
|
||||||
|
# Not under /app, which belongs to webhooker: this script runs as root.
|
||||||
|
COPY deploy/docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
|
||||||
|
|
||||||
# Create data directory for all SQLite databases (main app DB +
|
# Create data directory for all SQLite databases (main app DB +
|
||||||
# per-webhook event DBs). DATA_DIR defaults to /var/lib/webhooker.
|
# per-webhook event DBs). DATA_DIR defaults to /var/lib/webhooker.
|
||||||
RUN mkdir -p /var/lib/webhooker
|
RUN mkdir -p /var/lib/webhooker
|
||||||
|
|
||||||
RUN chown -R webhooker:webhooker /app /var/lib/webhooker
|
RUN chown -R webhooker:webhooker /app /var/lib/webhooker
|
||||||
|
|
||||||
USER webhooker
|
# No USER: the entrypoint starts as root to make the data directory
|
||||||
|
# webhooker's, then runs the app as webhooker.
|
||||||
|
|
||||||
EXPOSE 8080
|
EXPOSE 8080
|
||||||
|
|
||||||
@@ -124,4 +130,5 @@ ENV BIND_ADDRESS=0.0.0.0
|
|||||||
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
|
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
|
||||||
CMD wget --no-verbose --tries=1 --spider http://localhost:8080/.well-known/healthcheck || exit 1
|
CMD wget --no-verbose --tries=1 --spider http://localhost:8080/.well-known/healthcheck || exit 1
|
||||||
|
|
||||||
|
ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]
|
||||||
CMD ["/app/webhooker"]
|
CMD ["/app/webhooker"]
|
||||||
|
|||||||
@@ -538,6 +538,12 @@ its Argon2id hash. There is no second account and no forgot-password
|
|||||||
flow, so the banner and the reset command below are the only two ways
|
flow, so the banner and the reset command below are the only two ways
|
||||||
in.
|
in.
|
||||||
|
|
||||||
|
A start that finds no `webhooker.db` in `DATA_DIR` also logs
|
||||||
|
`created a new, empty database` at `WARN`, with the file's path,
|
||||||
|
shortly before the banner. On a deployment that has run before, that
|
||||||
|
line means `DATA_DIR` was empty, most often because its volume is not
|
||||||
|
mounted.
|
||||||
|
|
||||||
#### Recovering a lost admin password
|
#### Recovering a lost admin password
|
||||||
|
|
||||||
`webhooker resetpw` sets an existing account's password from the
|
`webhooker resetpw` sets an existing account's password from the
|
||||||
@@ -552,8 +558,9 @@ printf '%s' "$NEW_PASSWORD" | \
|
|||||||
DATA_DIR=/var/lib/webhooker webhooker resetpw admin
|
DATA_DIR=/var/lib/webhooker webhooker resetpw admin
|
||||||
```
|
```
|
||||||
|
|
||||||
In a container it is the same binary, which the image sets as `CMD`
|
In a container it is the same binary. The image's `CMD` is
|
||||||
rather than `ENTRYPOINT`, so the whole command has to be given:
|
`/app/webhooker`, and a command given to `docker run` replaces all of
|
||||||
|
it, so the whole command has to be given:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker run --rm -v webhooker-data:/var/lib/webhooker \
|
docker run --rm -v webhooker-data:/var/lib/webhooker \
|
||||||
@@ -690,38 +697,22 @@ those three values rather than trusting the figure. Measured at 65s on
|
|||||||
Docker 29.7.2.) A container `unhealthy` with `connection refused` in
|
Docker 29.7.2.) A container `unhealthy` with `connection refused` in
|
||||||
its health log, or a published port that resets connections, is this.
|
its health log, or a published port that resets connections, is this.
|
||||||
|
|
||||||
The container runs as a non-root user (`webhooker`, UID 1000), exposes
|
The app runs as a non-root user (`webhooker`, UID 1000), exposes port
|
||||||
port 8080, and includes a health check against
|
8080, and includes a health check against `/.well-known/healthcheck`.
|
||||||
`/.well-known/healthcheck`. The `/var/lib/webhooker` volume holds all
|
The `/var/lib/webhooker` volume holds all SQLite databases: the main
|
||||||
SQLite databases: the main application database (`webhooker.db`), the
|
application database (`webhooker.db`), the per-webhook event databases
|
||||||
per-webhook event databases (`events-{uuid}.db`), and any archive
|
(`events-{uuid}.db`), and any archive databases written by `database`
|
||||||
databases written by `database` targets (`archive-{uuid}.db`). Mount
|
targets (`archive-{uuid}.db`). Mount this as a persistent volume to
|
||||||
this as a persistent volume to preserve data across container
|
preserve data across container restarts.
|
||||||
restarts.
|
|
||||||
|
|
||||||
**The bind-mounted directory must be owned by UID 1000, or the
|
**The container sets its data directory's owner and mode itself
|
||||||
container does not start.** Docker creates a `-v` source path that
|
before the app starts**, so a host directory can be mounted as it is,
|
||||||
does not exist yet as `root:root`, and the process runs as UID 1000,
|
whoever owns it. The image's `ENTRYPOINT`,
|
||||||
so it cannot take its `DATA_DIR` lock:
|
`deploy/docker-entrypoint.sh`, starts as root, creates `DATA_DIR` if
|
||||||
|
it is missing, gives the directory and anything in it that belongs to
|
||||||
```
|
another user to `webhooker`, sets the directory to `0750`, and only
|
||||||
webhooker: locking data directory /var/lib/webhooker: open
|
then runs the app as `webhooker`. Started with `--user`, it changes
|
||||||
/var/lib/webhooker/webhooker.lock: permission denied
|
nothing and runs the app as that user.
|
||||||
```
|
|
||||||
|
|
||||||
It exits non-zero at that point, before opening any database. Create
|
|
||||||
the directory ahead of the first `docker run`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mkdir -p /path/to/data
|
|
||||||
chown 1000:1000 /path/to/data
|
|
||||||
chmod 750 /path/to/data
|
|
||||||
```
|
|
||||||
|
|
||||||
The same `chown` is what a restore needs — see step 4 of
|
|
||||||
[Restore](#restore). A **named volume** does not have this problem:
|
|
||||||
Docker copies the image's ownership onto a volume it initializes, and
|
|
||||||
the image creates `/var/lib/webhooker` owned by `webhooker`.
|
|
||||||
|
|
||||||
**The file modes are not yours to set, and do not depend on the
|
**The file modes are not yours to set, and do not depend on the
|
||||||
directory.** `webhooker.db` holds target configuration in plaintext —
|
directory.** `webhooker.db` holds target configuration in plaintext —
|
||||||
@@ -729,13 +720,10 @@ bearer tokens, API keys, Slack webhook URLs — along with the session
|
|||||||
encryption key, so webhooker creates every SQLite file it owns `0600`:
|
encryption key, so webhooker creates every SQLite file it owns `0600`:
|
||||||
each database and both of its `-wal` and `-shm` sidecars, across all
|
each database and both of its `-wal` and `-shm` sidecars, across all
|
||||||
three tiers. Files an earlier build left `0644` are tightened when
|
three tiers. Files an earlier build left `0644` are tightened when
|
||||||
they are opened. A `DATA_DIR` webhooker creates itself is `0750`, but
|
they are opened. The directory's `0750` is defence in depth — it stops
|
||||||
a bind mount supplies its own directory and Docker's default for one
|
other local users listing the directory and learning your webhook
|
||||||
it creates is `0755`; the `0600` files hold there regardless. The
|
UUIDs from the `events-{uuid}.db` filenames — not the barrier
|
||||||
`chmod 750` above is defence in depth — it stops other local users
|
protecting the credentials.
|
||||||
listing the directory and learning your webhook UUIDs from the
|
|
||||||
`events-{uuid}.db` filenames — not the barrier protecting the
|
|
||||||
credentials.
|
|
||||||
|
|
||||||
### Running under upaas
|
### Running under upaas
|
||||||
|
|
||||||
@@ -751,17 +739,6 @@ repository's `Dockerfile` and runs it. The app needs:
|
|||||||
app name, port `8080`. Leave `PORT` unset: the image's health check
|
app name, port `8080`. Leave `PORT` unset: the image's health check
|
||||||
probes `8080`.
|
probes `8080`.
|
||||||
- **Volume:** one host directory mounted at `/var/lib/webhooker`.
|
- **Volume:** one host directory mounted at `/var/lib/webhooker`.
|
||||||
upaas bind-mounts the host path it is given and does not create it,
|
|
||||||
and the container does not start unless UID 1000 owns it (see
|
|
||||||
[Running with Docker](#running-with-docker)). Create it before the
|
|
||||||
first deploy:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mkdir -p /path/to/data
|
|
||||||
chown 1000:1000 /path/to/data
|
|
||||||
chmod 750 /path/to/data
|
|
||||||
```
|
|
||||||
|
|
||||||
- **Environment variables:**
|
- **Environment variables:**
|
||||||
- `WEBHOOKER_ENVIRONMENT=prod`
|
- `WEBHOOKER_ENVIRONMENT=prod`
|
||||||
- `TRUSTED_PROXIES`: your reverse proxy's address on that Docker
|
- `TRUSTED_PROXIES`: your reverse proxy's address on that Docker
|
||||||
@@ -1020,12 +997,12 @@ done
|
|||||||
`.backup` reads through the WAL and writes a single consistent file with
|
`.backup` reads through the WAL and writes a single consistent file with
|
||||||
no sidecars of its own, so the destination is complete as it stands.
|
no sidecars of its own, so the destination is complete as it stands.
|
||||||
Two caveats. First, the runtime image is `alpine:3.21` with only
|
Two caveats. First, the runtime image is `alpine:3.21` with only
|
||||||
`ca-certificates` added — the `sqlite3` CLI is **not** in it, so run
|
`ca-certificates` and `su-exec` added — the `sqlite3` CLI is **not** in
|
||||||
this on the host against the volume path, or from a throwaway container
|
it, so run this on the host against the volume path, or from a
|
||||||
that mounts the volume. Second, each file is captured at its own
|
throwaway container that mounts the volume. Second, each file is
|
||||||
instant, so a webhook created or an event delivered between two files
|
captured at its own instant, so a webhook created or an event delivered
|
||||||
being copied lands in one and not the other. If you need the whole set
|
between two files being copied lands in one and not the other. If you
|
||||||
coherent as of a single moment, stop the service.
|
need the whole set coherent as of a single moment, stop the service.
|
||||||
|
|
||||||
Note that `sqlite3 <db> .dump` is **not** one of these procedures: it is
|
Note that `sqlite3 <db> .dump` is **not** one of these procedures: it is
|
||||||
an export, it holds a read transaction open for as long as it runs, and
|
an export, it holds a read transaction open for as long as it runs, and
|
||||||
@@ -1079,21 +1056,11 @@ with any `-wal`/`-shm` beside it, or wait until there are none.
|
|||||||
archive not opened since a crash. A copy salvaged from a crashed
|
archive not opened since a crash. A copy salvaged from a crashed
|
||||||
instance has them for everything, and needs all of them.
|
instance has them for everything, and needs all of them.
|
||||||
|
|
||||||
4. **Fix ownership.** The container runs as the non-root `webhooker`
|
4. Start the service. The container gives the directory and the
|
||||||
user, UID 1000 / GID 1000. Restored files must be owned by (or
|
restored files to the `webhooker` user before the app starts,
|
||||||
writable by) that UID, and so must the directory itself — SQLite
|
whoever restored them (see
|
||||||
creates the `-wal` and `-shm` sidecars beside the database, so a
|
[Running with Docker](#running-with-docker)). `AutoMigrate` runs
|
||||||
writable file inside a directory it cannot write is not enough:
|
against each restored database as it is opened.
|
||||||
|
|
||||||
```bash
|
|
||||||
chown -R 1000:1000 /path/to/data
|
|
||||||
```
|
|
||||||
|
|
||||||
Restoring as `root` on the host and forgetting this step is the
|
|
||||||
usual way a restore fails.
|
|
||||||
|
|
||||||
5. Start the service. `AutoMigrate` runs against each restored database
|
|
||||||
as it is opened.
|
|
||||||
|
|
||||||
### Upgrades
|
### Upgrades
|
||||||
|
|
||||||
@@ -1103,7 +1070,7 @@ unconditionally against whatever files it finds:
|
|||||||
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
|
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
|
||||||
`Entrypoint`, `Target`
|
`Entrypoint`, `Target`
|
||||||
- each event database when it is lazily opened — `Event`, `Delivery`,
|
- each event database when it is lazily opened — `Event`, `Delivery`,
|
||||||
`DeliveryResult`
|
`DeliveryResult`, `Totals`
|
||||||
- each archive database on every open and reopen
|
- each archive database on every open and reopen
|
||||||
|
|
||||||
There is no schema version table, no migration ledger, and no down
|
There is no schema version table, no migration ledger, and no down
|
||||||
@@ -1417,7 +1384,7 @@ The codebase uses consistent naming throughout (rename completed in
|
|||||||
|
|
||||||
### Data Model
|
### Data Model
|
||||||
|
|
||||||
webhooker's data model has nine entities organized into two tiers: the
|
webhooker's data model has ten entities organized into two tiers: the
|
||||||
**application tier** (user and webhook configuration) and the **event
|
**application tier** (user and webhook configuration) and the **event
|
||||||
tier** (event ingestion, delivery, and logging).
|
tier** (event ingestion, delivery, and logging).
|
||||||
|
|
||||||
@@ -1446,6 +1413,10 @@ tier** (event ingestion, delivery, and logging).
|
|||||||
│ ┌──────────┐ ┌──────────┐ ┌─────────────────┐ │
|
│ ┌──────────┐ ┌──────────┐ ┌─────────────────┐ │
|
||||||
│ │ Event │──1:N──│ Delivery │──1:N──│ DeliveryResult │ │
|
│ │ Event │──1:N──│ Delivery │──1:N──│ DeliveryResult │ │
|
||||||
│ └──────────┘ └──────────┘ └─────────────────┘ │
|
│ └──────────┘ └──────────┘ └─────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ ┌──────────┐ │
|
||||||
|
│ │ Totals │ (one row of running counts) │
|
||||||
|
│ └──────────┘ │
|
||||||
└─────────────────────────────────────────────────────────────┘
|
└─────────────────────────────────────────────────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -1472,7 +1443,7 @@ A registered user of the webhooker service.
|
|||||||
| Field | Type | Description |
|
| Field | Type | Description |
|
||||||
| ---------- | -------- | ----------- |
|
| ---------- | -------- | ----------- |
|
||||||
| `id` | UUID | Primary key |
|
| `id` | UUID | Primary key |
|
||||||
| `username` | string | Unique login name |
|
| `username` | string | Unique login name, at most 1024 bytes so that it fits in the session cookie |
|
||||||
| `password` | string | Argon2id hash (never exposed via API) |
|
| `password` | string | Argon2id hash (never exposed via API) |
|
||||||
|
|
||||||
**Relations:** Has many Webhooks. Has many APIKeys.
|
**Relations:** Has many Webhooks. Has many APIKeys.
|
||||||
@@ -1698,6 +1669,7 @@ status across potentially multiple attempts.
|
|||||||
| `event_id` | UUID | Foreign key → Event |
|
| `event_id` | UUID | Foreign key → Event |
|
||||||
| `target_id`| UUID | Foreign key → Target |
|
| `target_id`| UUID | Foreign key → Target |
|
||||||
| `status` | DeliveryStatus | One of: `pending`, `delivered`, `failed`, `retrying` |
|
| `status` | DeliveryStatus | One of: `pending`, `delivered`, `failed`, `retrying` |
|
||||||
|
| `finished_at` | timestamp | When the delivery became `delivered` or `failed` (nullable; empty while `pending` or `retrying`) |
|
||||||
|
|
||||||
**Relations:** Belongs to Event. Belongs to Target. Has many
|
**Relations:** Belongs to Event. Belongs to Target. Has many
|
||||||
DeliveryResults.
|
DeliveryResults.
|
||||||
@@ -1765,6 +1737,29 @@ retries) is individually logged for full observability.
|
|||||||
|
|
||||||
**Relations:** Belongs to Delivery.
|
**Relations:** Belongs to Delivery.
|
||||||
|
|
||||||
|
#### Totals
|
||||||
|
|
||||||
|
The one row of running counts in each event database, read by the
|
||||||
|
statistics pane at the top of the webhook page.
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
| -------------------- | ------- | ----------- |
|
||||||
|
| `events` | integer | Events ever stored, resubmitted copies included |
|
||||||
|
| `deliveries` | integer | Deliveries ever created, replays included |
|
||||||
|
| `failures` | integer | Deliveries that ever became `failed` |
|
||||||
|
| `events_removed` | integer | Events retention has deleted |
|
||||||
|
| `deliveries_removed` | integer | Deliveries retention has deleted |
|
||||||
|
| `failures_removed` | integer | Failed deliveries retention has deleted |
|
||||||
|
|
||||||
|
Each count changes in the transaction that writes or deletes the rows it
|
||||||
|
counts. The pane shows each of the first three as a lifetime figure, and
|
||||||
|
less what retention removed as the figure within retention, so neither
|
||||||
|
needs the rows themselves. Its last-10-minutes and last-24-hours figures
|
||||||
|
are counted from the `events` and `deliveries` indexes over just that
|
||||||
|
window. Its failure percentage for a window is the deliveries that became
|
||||||
|
`failed` in it out of all that became `delivered` or `failed` in it, and
|
||||||
|
a dash when none did.
|
||||||
|
|
||||||
#### Event-tier indexes
|
#### Event-tier indexes
|
||||||
|
|
||||||
These indexes on the per-webhook event databases are declared in the model
|
These indexes on the per-webhook event databases are declared in the model
|
||||||
@@ -1772,10 +1767,10 @@ tags, so `AutoMigrate` creates them on a fresh and on an existing database:
|
|||||||
|
|
||||||
| Table | Columns | Serves |
|
| Table | Columns | Serves |
|
||||||
| ------------------ | --------------------------- | ------ |
|
| ------------------ | --------------------------- | ------ |
|
||||||
| `deliveries` | `status`, `deleted_at` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status |
|
| `deliveries` | `status`, `deleted_at`, `finished_at` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, and the webhook page's statistics, which count deliveries by status and when they finished |
|
||||||
| `deliveries` | `event_id`, `deleted_at` | The event log, which loads each event's deliveries, and retention, which selects and deletes the deliveries of expired events |
|
| `deliveries` | `event_id`, `deleted_at` | The event log, which loads each event's deliveries, and retention, which selects and deletes the deliveries of expired events |
|
||||||
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
|
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
|
||||||
| `events` | `deleted_at`, `created_at` | Retention, which selects expired events by age |
|
| `events` | `deleted_at`, `created_at` | Retention, which selects expired events by age, and the webhook page's statistics, which count recent events and find the newest |
|
||||||
| `events` | `created_at` | Retention's delete of the expired events themselves |
|
| `events` | `created_at` | Retention's delete of the expired events themselves |
|
||||||
|
|
||||||
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention's
|
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention's
|
||||||
@@ -1789,9 +1784,10 @@ and SQLite narrows by a `<` only on the last column it uses.
|
|||||||
|
|
||||||
#### Common Fields
|
#### Common Fields
|
||||||
|
|
||||||
Every entity except `Setting` includes these fields from `BaseModel`.
|
Every entity except `Setting` and `Totals` includes these fields from
|
||||||
`Setting` is a bare key-value row with no `id`, no timestamps and no
|
`BaseModel`. `Setting` is a bare key-value row with no `id`, no
|
||||||
soft delete:
|
timestamps and no soft delete, and `Totals` is a single row of counts
|
||||||
|
with only a numeric `id`:
|
||||||
|
|
||||||
| Field | Type | Description |
|
| Field | Type | Description |
|
||||||
| ------------ | --------- | ----------- |
|
| ------------ | --------- | ----------- |
|
||||||
@@ -1833,6 +1829,7 @@ encryption key is generated and stored, and an `admin` user is created.
|
|||||||
- **Events** — captured incoming webhook payloads
|
- **Events** — captured incoming webhook payloads
|
||||||
- **Deliveries** — event-to-target pairings and their status
|
- **Deliveries** — event-to-target pairings and their status
|
||||||
- **DeliveryResults** — individual delivery attempt logs
|
- **DeliveryResults** — individual delivery attempt logs
|
||||||
|
- **Totals** — running counts of the above, kept through retention
|
||||||
|
|
||||||
Per-webhook databases are created automatically when a webhook is
|
Per-webhook databases are created automatically when a webhook is
|
||||||
created (and lazily on first access for webhooks that predate this
|
created (and lazily on first access for webhooks that predate this
|
||||||
@@ -2412,14 +2409,14 @@ Removing either cap fails 14 subtests.
|
|||||||
|
|
||||||
`internal/middleware/logbound_test.go` and
|
`internal/middleware/logbound_test.go` and
|
||||||
`internal/handlers/logbound_test.go` drive 8 KB of client-chosen text
|
`internal/handlers/logbound_test.go` drive 8 KB of client-chosen text
|
||||||
at each of these — 1 KB at `invalid password`, whose accounts are
|
at each of these — just under 1 KB at `invalid password`, whose
|
||||||
shared with the successful-login line, where a username past 4 KB
|
accounts are shared with the successful-login line and so must stay
|
||||||
overflows the session cookie and answers 500 before that line is
|
within the 1024-byte username limit — through both handlers, and
|
||||||
written — through both handlers, and through seven fills: plain text
|
through seven fills: plain text as the baseline, and then the
|
||||||
as the baseline, and then the quotation mark, backslash, tab, newline,
|
quotation mark, backslash, tab, newline, C0 control and astral
|
||||||
C0 control and astral non-printable, six characters the wider of the
|
non-printable, six characters the wider of the two handlers spends
|
||||||
two handlers spends more on than the client spent sending them. Every
|
more on than the client spent sending them. Every case holds each
|
||||||
case holds each line to the 2,560-byte ceiling. That per-line ceiling
|
line to the 2,560-byte ceiling. That per-line ceiling
|
||||||
is what the figure above states, and every row establishes it.
|
is what the figure above states, and every row establishes it.
|
||||||
|
|
||||||
Three of the sites go further and bound the whole flood's output — the
|
Three of the sites go further and bound the whole flood's output — the
|
||||||
@@ -2721,7 +2718,7 @@ abuse limit later; they are tracked as future work.
|
|||||||
| ------ | --------------------------- | ----------- |
|
| ------ | --------------------------- | ----------- |
|
||||||
| `GET` | `/` | Root redirect, 303 (authenticated → `/sources`, unauthenticated → `/pages/login`) |
|
| `GET` | `/` | Root redirect, 303 (authenticated → `/sources`, unauthenticated → `/pages/login`) |
|
||||||
| `GET` | `/.well-known/healthcheck` | Health check (JSON: `status`, `now`, `uptimeSeconds`, `uptimeHuman`, `version`, `appname`, `maintenanceMode`) |
|
| `GET` | `/.well-known/healthcheck` | Health check (JSON: `status`, `now`, `uptimeSeconds`, `uptimeHuman`, `version`, `appname`, `maintenanceMode`) |
|
||||||
| any | `/s/*` | Static file serving (embedded CSS, JS). Mounted for every method, not just `GET`/`HEAD`: chi's `Mount` registers all methods and `http.FileServer` special-cases only `HEAD` (by omitting the body), so a `POST` or `DELETE` to an asset is answered `200` with the file. Pinned by `TestStaticServesEveryMethod` |
|
| `GET`, `HEAD` | `/s/*` | Static file serving (embedded CSS, JS). `GET` and `HEAD` only — `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` and `CONNECT` are answered `405 Method Not Allowed` with `Allow: GET, HEAD`. Any other method (such as `PROPFIND`) is refused by chi before it reaches this route, and gets `405` without an `Allow` header. Pinned by `TestStaticServesOnlyGetAndHead` |
|
||||||
| `POST` | `/webhook/{uuid}` | Webhook receiver endpoint. `POST` only — every other method is answered `405 Method Not Allowed` with `Allow: POST`. Rate limited (see [Rate Limiting](#rate-limiting)) |
|
| `POST` | `/webhook/{uuid}` | Webhook receiver endpoint. `POST` only — every other method is answered `405 Method Not Allowed` with `Allow: POST`. Rate limited (see [Rate Limiting](#rate-limiting)) |
|
||||||
|
|
||||||
#### Authentication Endpoints
|
#### Authentication Endpoints
|
||||||
@@ -3071,7 +3068,11 @@ check, see [The login endpoint](#the-login-endpoint).
|
|||||||
- Prometheus metrics behind basic auth
|
- Prometheus metrics behind basic auth
|
||||||
- Static assets embedded in binary (no filesystem access needed at
|
- Static assets embedded in binary (no filesystem access needed at
|
||||||
runtime)
|
runtime)
|
||||||
- Container runs as non-root user (UID 1000)
|
- The app runs as the non-root `webhooker` user (UID 1000) in the
|
||||||
|
container. The image sets no `USER`, so these run as root: the
|
||||||
|
`ENTRYPOINT` script, which sets the data directory's owner and mode
|
||||||
|
before the app starts; the image's health check; and `docker exec`,
|
||||||
|
unless given `--user`
|
||||||
- GORM soft deletes on every entity that carries `BaseModel`, which is
|
- GORM soft deletes on every entity that carries `BaseModel`, which is
|
||||||
all of them but `Setting` (data preserved for audit)
|
all of them but `Setting` (data preserved for audit)
|
||||||
|
|
||||||
@@ -3198,10 +3199,13 @@ version is fixed independently of the compiler's:
|
|||||||
`GO_LDFLAGS`, so neither can drop the `-X` that stamps the version.
|
`GO_LDFLAGS`, so neither can drop the `-X` that stamps the version.
|
||||||
The version arrives as the `VERSION` build arg, since the context
|
The version arrives as the `VERSION` build arg, since the context
|
||||||
has no `.git` (see [Version stamping](#version-stamping)).
|
has no `.git` (see [Version stamping](#version-stamping)).
|
||||||
3. **Runtime stage** (`alpine:3.21`) — copies the static binary,
|
3. **Runtime stage** (`alpine:3.21`) — copies the static binary and
|
||||||
creates the `/var/lib/webhooker` directory for all SQLite databases,
|
`deploy/docker-entrypoint.sh`, creates the `/var/lib/webhooker`
|
||||||
runs as the non-root `webhooker` user (UID 1000), exposes port 8080,
|
directory for all SQLite databases, exposes port 8080, and includes
|
||||||
and includes a health check against `/.well-known/healthcheck`.
|
a health check against `/.well-known/healthcheck`. It sets no
|
||||||
|
`USER`: the `ENTRYPOINT` script starts as root, sets the data
|
||||||
|
directory's owner and mode, and runs the app as the non-root
|
||||||
|
`webhooker` user (UID 1000) through `su-exec`.
|
||||||
|
|
||||||
The lint stage invokes `golangci-lint` directly rather than `make lint`:
|
The lint stage invokes `golangci-lint` directly rather than `make lint`:
|
||||||
it is already the pinned linter image, and `make lint` builds
|
it is already the pinned linter image, and `make lint` builds
|
||||||
@@ -3280,3 +3284,5 @@ MIT
|
|||||||
## Author
|
## Author
|
||||||
|
|
||||||
[@sneak](https://sneak.berlin)
|
[@sneak](https://sneak.berlin)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
Executable
+22
@@ -0,0 +1,22 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# deploy/docker-entrypoint.sh: the image's ENTRYPOINT. A bind-mounted
|
||||||
|
# data directory keeps its owner from the host, often root, and the app
|
||||||
|
# could not write to it. Started as root, this creates DATA_DIR if
|
||||||
|
# needed, gives it and everything in it to webhooker, sets its mode, and
|
||||||
|
# runs the command as webhooker, so the app never runs as root. Started
|
||||||
|
# as another user, it only runs the command.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
main() {
|
||||||
|
if [ "$(id -u)" != 0 ]; then
|
||||||
|
exec "$@"
|
||||||
|
fi
|
||||||
|
|
||||||
|
dir="${DATA_DIR:-/var/lib/webhooker}"
|
||||||
|
mkdir -p "$dir"
|
||||||
|
find "$dir" ! -user webhooker -exec chown -h webhooker:webhooker {} +
|
||||||
|
chmod 750 "$dir"
|
||||||
|
exec su-exec webhooker "$@"
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
@@ -3,6 +3,8 @@ package database_test
|
|||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
|
"log/slog"
|
||||||
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
@@ -83,3 +85,37 @@ func TestFirstBoot_PrintsTheAdminPasswordAsABanner(t *testing.T) {
|
|||||||
t, ok, "the printed password must open the seeded account",
|
t, ok, "the printed password must open the seeded account",
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestNewDatabase_IsLoggedWithItsPath is the log half of
|
||||||
|
// https://git.eeqj.de/sneak/webhooker/issues/359. A DATA_DIR that is
|
||||||
|
// unexpectedly empty boots exactly like a first start, so the start
|
||||||
|
// that creates the database must say so, and where. Opening that
|
||||||
|
// database again must not.
|
||||||
|
func TestNewDatabase_IsLoggedWithItsPath(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
|
||||||
|
open := func() string {
|
||||||
|
var out bytes.Buffer
|
||||||
|
|
||||||
|
db, err := database.Open(dir, slog.New(slog.NewTextHandler(&out, nil)))
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, db.Close())
|
||||||
|
|
||||||
|
return out.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
const created = `level=WARN msg="created a new, empty database"`
|
||||||
|
|
||||||
|
first := open()
|
||||||
|
second := open()
|
||||||
|
|
||||||
|
assert.Contains(
|
||||||
|
t, first,
|
||||||
|
created+" path="+filepath.Join(dir, database.MainDBFileName),
|
||||||
|
)
|
||||||
|
assert.NotContains(
|
||||||
|
t, second, created, "an existing database is not new",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
|
"io/fs"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
@@ -199,6 +200,12 @@ func (d *Database) connectTo(dataDir string) error {
|
|||||||
// Construct the main application database path inside DATA_DIR.
|
// Construct the main application database path inside DATA_DIR.
|
||||||
dbPath := filepath.Join(dataDir, MainDBFileName)
|
dbPath := filepath.Join(dataDir, MainDBFileName)
|
||||||
|
|
||||||
|
// Checked before opening, which creates the file. A DATA_DIR that
|
||||||
|
// is unexpectedly empty -- its volume not mounted, say -- looks
|
||||||
|
// exactly like a first start, so a new database is a warning.
|
||||||
|
_, statErr := os.Stat(dbPath)
|
||||||
|
created := errors.Is(statErr, fs.ErrNotExist)
|
||||||
|
|
||||||
// Opened through OpenSQLite so this handle carries the same WAL
|
// Opened through OpenSQLite so this handle carries the same WAL
|
||||||
// journaling, busy timeout, immediate-transaction locking, and pool
|
// journaling, busy timeout, immediate-transaction locking, and pool
|
||||||
// bounds as every other database file. See sqlite_open.go.
|
// bounds as every other database file. See sqlite_open.go.
|
||||||
@@ -229,7 +236,12 @@ func (d *Database) connectTo(dataDir string) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
d.db = db
|
d.db = db
|
||||||
|
|
||||||
|
if created {
|
||||||
|
d.log.Warn("created a new, empty database", "path", dbPath)
|
||||||
|
} else {
|
||||||
d.log.Info("connected to database", "path", dbPath)
|
d.log.Info("connected to database", "path", dbPath)
|
||||||
|
}
|
||||||
|
|
||||||
// Run migrations
|
// Run migrations
|
||||||
return d.migrate()
|
return d.migrate()
|
||||||
|
|||||||
@@ -93,6 +93,7 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
|
|||||||
deliveries []database.Delivery
|
deliveries []database.Delivery
|
||||||
results []database.DeliveryResult
|
results []database.DeliveryResult
|
||||||
depths []struct{ Depth int }
|
depths []struct{ Depth int }
|
||||||
|
failed struct{ Count int64 }
|
||||||
)
|
)
|
||||||
|
|
||||||
byStatus := "idx_deliveries_status (status=? AND deleted_at=?)"
|
byStatus := "idx_deliveries_status (status=? AND deleted_at=?)"
|
||||||
@@ -123,7 +124,7 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
|
|||||||
Order("attempt_num ASC").Find(&results),
|
Order("attempt_num ASC").Find(&results),
|
||||||
"idx_delivery_results_delivery_id (delivery_id=? AND deleted_at=?)")
|
"idx_delivery_results_delivery_id (delivery_id=? AND deleted_at=?)")
|
||||||
|
|
||||||
// Retention's three deletes (reapExpired), whose subqueries are built
|
// Retention's deletes (deleteExpired), whose subqueries are built
|
||||||
// afresh for each statement as it builds them.
|
// afresh for each statement as it builds them.
|
||||||
expiredEventIDs := func() *gorm.DB {
|
expiredEventIDs := func() *gorm.DB {
|
||||||
return dry.Model(&database.Event{}).Select("id").
|
return dry.Model(&database.Event{}).Select("id").
|
||||||
@@ -135,6 +136,12 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
|
|||||||
Select("id").Where("event_id IN (?)", expiredEventIDs()),
|
Select("id").Where("event_id IN (?)", expiredEventIDs()),
|
||||||
).Delete(&database.DeliveryResult{}),
|
).Delete(&database.DeliveryResult{}),
|
||||||
"idx_delivery_results_delivery_id (delivery_id=?)", byEvent, byAge)
|
"idx_delivery_results_delivery_id (delivery_id=?)", byEvent, byAge)
|
||||||
|
assertPlanUses(t, db, dry.Unscoped().Model(&database.Delivery{}).
|
||||||
|
Select("count(CASE WHEN status = ? THEN 1 END) AS count",
|
||||||
|
database.DeliveryStatusFailed).
|
||||||
|
Where("event_id IN (?)", expiredEventIDs()).
|
||||||
|
Take(&failed),
|
||||||
|
"idx_deliveries_event_id (event_id=?)", byAge)
|
||||||
assertPlanUses(t, db, dry.Unscoped().Where(
|
assertPlanUses(t, db, dry.Unscoped().Where(
|
||||||
"event_id IN (?)", expiredEventIDs(),
|
"event_id IN (?)", expiredEventIDs(),
|
||||||
).Delete(&database.Delivery{}),
|
).Delete(&database.Delivery{}),
|
||||||
@@ -144,6 +151,54 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
|
|||||||
).Delete(&database.Event{}), "idx_events_created_at (created_at<?)")
|
).Delete(&database.Event{}), "idx_events_created_at (created_at<?)")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestStatisticsQueriesUseTheirIndexes does the same for the webhook
|
||||||
|
// page's statistics (readEventStats in the handlers): deliveries in
|
||||||
|
// progress, deliveries finished and events received since a time, and
|
||||||
|
// the newest event, which must come straight off an index rather than
|
||||||
|
// from sorting every event.
|
||||||
|
func TestStatisticsQueriesUseTheirIndexes(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
mgr, lc := setupTestWebhookDBManager(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
require.NoError(t, lc.Start(ctx))
|
||||||
|
|
||||||
|
defer func() { require.NoError(t, lc.Stop(ctx)) }()
|
||||||
|
|
||||||
|
db, err := mgr.GetDB(uuid.New().String())
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
dry := db.Session(&gorm.Session{DryRun: true})
|
||||||
|
since := time.Now()
|
||||||
|
|
||||||
|
var (
|
||||||
|
count int64
|
||||||
|
newest []time.Time
|
||||||
|
)
|
||||||
|
|
||||||
|
assertPlanUses(t, db, dry.Model(&database.Delivery{}).
|
||||||
|
Where("status IN ?", []database.DeliveryStatus{
|
||||||
|
database.DeliveryStatusPending,
|
||||||
|
database.DeliveryStatusRetrying,
|
||||||
|
}).Count(&count),
|
||||||
|
"idx_deliveries_status (status=? AND deleted_at=?)")
|
||||||
|
assertPlanUses(t, db, dry.Model(&database.Delivery{}).
|
||||||
|
Where("status = ? AND finished_at >= ?",
|
||||||
|
database.DeliveryStatusFailed, since).Count(&count),
|
||||||
|
"idx_deliveries_status "+
|
||||||
|
"(status=? AND deleted_at=? AND finished_at>?)")
|
||||||
|
assertPlanUses(t, db, dry.Model(&database.Event{}).
|
||||||
|
Where("created_at >= ?", since).Count(&count),
|
||||||
|
"idx_events_deleted_at_created_at "+
|
||||||
|
"(deleted_at=? AND created_at>?)")
|
||||||
|
|
||||||
|
newestEvent := dry.Model(&database.Event{}).
|
||||||
|
Order("created_at DESC").Limit(1).Pluck("created_at", &newest)
|
||||||
|
assertPlanUses(t, db, newestEvent,
|
||||||
|
"idx_events_deleted_at_created_at (deleted_at=?)")
|
||||||
|
assert.NotContains(t, queryPlan(t, db, newestEvent), "TEMP B-TREE")
|
||||||
|
}
|
||||||
|
|
||||||
// assertPlanUses asserts that SQLite's plan for a statement GORM built
|
// assertPlanUses asserts that SQLite's plan for a statement GORM built
|
||||||
// in a dry run, run with the same SQL and arguments GORM would send,
|
// in a dry run, run with the same SQL and arguments GORM would send,
|
||||||
// names each of the given indexes.
|
// names each of the given indexes.
|
||||||
@@ -152,6 +207,18 @@ func assertPlanUses(
|
|||||||
) {
|
) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
|
plan := queryPlan(t, db, built)
|
||||||
|
|
||||||
|
for _, index := range indexes {
|
||||||
|
assert.Contains(t, plan, index, built.Statement.SQL.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// queryPlan returns SQLite's plan for a statement GORM built in a dry
|
||||||
|
// run, run with the same SQL and arguments GORM would send.
|
||||||
|
func queryPlan(t *testing.T, db, built *gorm.DB) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
var plan []struct{ Detail string }
|
var plan []struct{ Detail string }
|
||||||
|
|
||||||
require.NoError(t, db.Raw(
|
require.NoError(t, db.Raw(
|
||||||
@@ -159,8 +226,5 @@ func assertPlanUses(
|
|||||||
built.Statement.Vars...,
|
built.Statement.Vars...,
|
||||||
).Scan(&plan).Error)
|
).Scan(&plan).Error)
|
||||||
|
|
||||||
for _, index := range indexes {
|
return fmt.Sprint(plan)
|
||||||
assert.Contains(t, fmt.Sprint(plan), index,
|
|
||||||
built.Statement.SQL.String())
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,10 @@
|
|||||||
package database
|
package database
|
||||||
|
|
||||||
import "gorm.io/gorm"
|
import (
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gorm.io/gorm"
|
||||||
|
)
|
||||||
|
|
||||||
// DeliveryStatus represents the status of a delivery
|
// DeliveryStatus represents the status of a delivery
|
||||||
type DeliveryStatus string
|
type DeliveryStatus string
|
||||||
@@ -45,6 +49,12 @@ type Delivery struct {
|
|||||||
// gives.
|
// gives.
|
||||||
DeletedAt gorm.DeletedAt `gorm:"index:idx_deliveries_event_id,priority:2;index:idx_deliveries_status,priority:2" json:"deletedAt,omitzero"`
|
DeletedAt gorm.DeletedAt `gorm:"index:idx_deliveries_event_id,priority:2;index:idx_deliveries_status,priority:2" json:"deletedAt,omitzero"`
|
||||||
|
|
||||||
|
// FinishedAt is when the delivery became delivered or failed, and
|
||||||
|
// nil while it is pending or retrying. It ends the status index,
|
||||||
|
// so the webhook page counts the deliveries that finished in a
|
||||||
|
// recent window by reading that window from the index.
|
||||||
|
FinishedAt *time.Time `gorm:"index:idx_deliveries_status,priority:3" json:"finishedAt,omitempty"`
|
||||||
|
|
||||||
// Relations
|
// Relations
|
||||||
Event Event `json:"event,omitzero"`
|
Event Event `json:"event,omitzero"`
|
||||||
Target Target `json:"target,omitzero"`
|
Target Target `json:"target,omitzero"`
|
||||||
|
|||||||
@@ -0,0 +1,73 @@
|
|||||||
|
package database
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"gorm.io/gorm"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Totals is the single row of running totals in a webhook's event
|
||||||
|
// database. It is what keeps the webhook page's lifetime figures right
|
||||||
|
// after retention has removed the rows they count, and what lets the
|
||||||
|
// page show them without counting every row.
|
||||||
|
//
|
||||||
|
// Storing an event, creating a delivery and failing a delivery each
|
||||||
|
// add one, and retention adds what it deletes to the Removed columns.
|
||||||
|
// Every addition goes through AddTotals, in the transaction that
|
||||||
|
// writes or deletes the rows it counts.
|
||||||
|
type Totals struct {
|
||||||
|
ID int64 `gorm:"primaryKey"`
|
||||||
|
|
||||||
|
Events int64 `gorm:"not null"`
|
||||||
|
Deliveries int64 `gorm:"not null"`
|
||||||
|
Failures int64 `gorm:"not null"`
|
||||||
|
|
||||||
|
EventsRemoved int64 `gorm:"not null"`
|
||||||
|
DeliveriesRemoved int64 `gorm:"not null"`
|
||||||
|
FailuresRemoved int64 `gorm:"not null"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// TableName names the table AddTotals updates.
|
||||||
|
func (Totals) TableName() string {
|
||||||
|
return "totals"
|
||||||
|
}
|
||||||
|
|
||||||
|
// EventsWithinRetention is how many of the webhook's events are still
|
||||||
|
// stored.
|
||||||
|
func (t Totals) EventsWithinRetention() int64 {
|
||||||
|
return t.Events - t.EventsRemoved
|
||||||
|
}
|
||||||
|
|
||||||
|
// DeliveriesWithinRetention is how many of the webhook's deliveries
|
||||||
|
// are still stored.
|
||||||
|
func (t Totals) DeliveriesWithinRetention() int64 {
|
||||||
|
return t.Deliveries - t.DeliveriesRemoved
|
||||||
|
}
|
||||||
|
|
||||||
|
// FailuresWithinRetention is how many of the webhook's failed
|
||||||
|
// deliveries are still stored.
|
||||||
|
func (t Totals) FailuresWithinRetention() int64 {
|
||||||
|
return t.Failures - t.FailuresRemoved
|
||||||
|
}
|
||||||
|
|
||||||
|
// AddTotals adds each count in add to the webhook's running totals.
|
||||||
|
// Call it on the transaction that writes or deletes the rows it
|
||||||
|
// counts, so the totals change exactly when those rows do.
|
||||||
|
func AddTotals(tx *gorm.DB, add Totals) error {
|
||||||
|
err := tx.Exec(
|
||||||
|
`UPDATE totals SET
|
||||||
|
events = events + ?,
|
||||||
|
deliveries = deliveries + ?,
|
||||||
|
failures = failures + ?,
|
||||||
|
events_removed = events_removed + ?,
|
||||||
|
deliveries_removed = deliveries_removed + ?,
|
||||||
|
failures_removed = failures_removed + ?`,
|
||||||
|
add.Events, add.Deliveries, add.Failures,
|
||||||
|
add.EventsRemoved, add.DeliveriesRemoved, add.FailuresRemoved,
|
||||||
|
).Error
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("adding to running totals: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -1,13 +1,58 @@
|
|||||||
package database
|
package database
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"gorm.io/gorm"
|
||||||
|
)
|
||||||
|
|
||||||
|
// MaxUsernameBytes is the longest username, in bytes, that a user may
|
||||||
|
// have. The same number appears in the check constraint on
|
||||||
|
// User.Username, because a struct tag cannot reference a constant.
|
||||||
|
//
|
||||||
|
// A login stores the username in the session cookie, and both
|
||||||
|
// securecookie and browsers refuse a cookie value past about 4096
|
||||||
|
// bytes. That value is the session base64-encoded twice, so it holds
|
||||||
|
// 4096 × 3/4 × 3/4 = 2304 bytes of session, and the signature,
|
||||||
|
// timestamp and the session's other values take about 270 of those: a
|
||||||
|
// username longer than about 2030 bytes can never log in. The limit is
|
||||||
|
// about half that, so the session can carry more values later without
|
||||||
|
// locking out an account whose username is already at the limit.
|
||||||
|
const MaxUsernameBytes = 1024
|
||||||
|
|
||||||
|
// ErrUsernameTooLong is returned when a user is saved with a username
|
||||||
|
// longer than MaxUsernameBytes.
|
||||||
|
var ErrUsernameTooLong = errors.New("username is too long")
|
||||||
|
|
||||||
// User represents a user of the webhooker service
|
// User represents a user of the webhooker service
|
||||||
|
//
|
||||||
|
//nolint:lll // a struct tag cannot wrap
|
||||||
type User struct {
|
type User struct {
|
||||||
BaseModel
|
BaseModel
|
||||||
|
|
||||||
Username string `gorm:"uniqueIndex;not null" json:"username"`
|
Username string `gorm:"uniqueIndex;not null;check:length(CAST(username AS BLOB)) <= 1024" json:"username"`
|
||||||
Password string `gorm:"not null" json:"-"` // Argon2 hashed
|
Password string `gorm:"not null" json:"-"` // Argon2 hashed
|
||||||
|
|
||||||
// Relations
|
// Relations
|
||||||
Webhooks []Webhook `json:"webhooks,omitempty"`
|
Webhooks []Webhook `json:"webhooks,omitempty"`
|
||||||
APIKeys []APIKey `json:"apiKeys,omitempty"`
|
APIKeys []APIKey `json:"apiKeys,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// BeforeSave rejects a username longer than MaxUsernameBytes when a whole
|
||||||
|
// User is created or saved, so those calls get ErrUsernameTooLong rather
|
||||||
|
// than the database's constraint error. A column update such as
|
||||||
|
// Update("username", ...) is caught only by the check constraint, as is
|
||||||
|
// any path that writes the table without this model.
|
||||||
|
func (u *User) BeforeSave(_ *gorm.DB) error {
|
||||||
|
if len(u.Username) > MaxUsernameBytes {
|
||||||
|
return fmt.Errorf(
|
||||||
|
"%w: %d bytes, limit is %d",
|
||||||
|
ErrUsernameTooLong,
|
||||||
|
len(u.Username),
|
||||||
|
MaxUsernameBytes,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,65 @@
|
|||||||
|
package database_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
)
|
||||||
|
|
||||||
|
// usernameAtLimit is exactly MaxUsernameBytes long, built from a
|
||||||
|
// two-byte character. A check that counted characters rather than bytes
|
||||||
|
// would see half the length and let the one-byte-longer name through.
|
||||||
|
func usernameAtLimit() string {
|
||||||
|
return strings.Repeat("é", database.MaxUsernameBytes/2)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUserCreate_RejectsOverlongUsername(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
db := startedTestDB(t)
|
||||||
|
|
||||||
|
err := db.Create(&database.User{
|
||||||
|
Username: usernameAtLimit() + "x",
|
||||||
|
Password: "hash",
|
||||||
|
}).Error
|
||||||
|
|
||||||
|
require.ErrorIs(t, err, database.ErrUsernameTooLong)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUserCreate_AcceptsUsernameAtLimit(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
db := startedTestDB(t)
|
||||||
|
|
||||||
|
require.NoError(t, db.Create(&database.User{
|
||||||
|
Username: usernameAtLimit(),
|
||||||
|
Password: "hash",
|
||||||
|
}).Error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUsersTable_EnforcesUsernameLimitWithoutTheModel inserts with raw
|
||||||
|
// SQL, as a path that bypassed User.BeforeSave would, so only the
|
||||||
|
// table's check constraint stands between it and an over-long
|
||||||
|
// username. Accepting the name at the limit and refusing the next byte
|
||||||
|
// also pins the constraint's number to MaxUsernameBytes.
|
||||||
|
func TestUsersTable_EnforcesUsernameLimitWithoutTheModel(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
db := startedTestDB(t)
|
||||||
|
|
||||||
|
insert := "INSERT INTO users (id, username, password) VALUES (?, ?, ?)"
|
||||||
|
|
||||||
|
require.NoError(t, db.Exec(
|
||||||
|
insert, uuid.New().String(), usernameAtLimit(), "hash",
|
||||||
|
).Error)
|
||||||
|
|
||||||
|
err := db.Exec(
|
||||||
|
insert, uuid.New().String(), usernameAtLimit()+"x", "hash",
|
||||||
|
).Error
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.Contains(t, err.Error(), "CHECK constraint failed")
|
||||||
|
}
|
||||||
@@ -2,7 +2,7 @@ package database
|
|||||||
|
|
||||||
// Migrate runs database migrations for the main application database.
|
// Migrate runs database migrations for the main application database.
|
||||||
// Only configuration-tier models are stored in the main database.
|
// Only configuration-tier models are stored in the main database.
|
||||||
// Event-tier models (Event, Delivery, DeliveryResult) live in
|
// Event-tier models (Event, Delivery, DeliveryResult, Totals) live in
|
||||||
// per-webhook dedicated databases managed by WebhookDBManager.
|
// per-webhook dedicated databases managed by WebhookDBManager.
|
||||||
func (d *Database) Migrate() error {
|
func (d *Database) Migrate() error {
|
||||||
return d.db.AutoMigrate(
|
return d.db.AutoMigrate(
|
||||||
|
|||||||
@@ -267,55 +267,101 @@ func retentionCutoff(
|
|||||||
|
|
||||||
// reapExpired hard-deletes, in foreign-key-safe order, the delivery
|
// reapExpired hard-deletes, in foreign-key-safe order, the delivery
|
||||||
// results, deliveries, and events associated with events older than
|
// results, deliveries, and events associated with events older than
|
||||||
// cutoff. Deletes are unscoped so rows are physically removed rather
|
// cutoff, and adds what it deleted to the running totals, all in one
|
||||||
// than soft-deleted, reclaiming disk. It returns the number of events
|
// transaction. Deletes are unscoped so rows are physically removed
|
||||||
// deleted.
|
// rather than soft-deleted, reclaiming disk. It returns the number of
|
||||||
|
// events deleted.
|
||||||
func reapExpired(db *gorm.DB, cutoff time.Time) (int64, error) {
|
func reapExpired(db *gorm.DB, cutoff time.Time) (int64, error) {
|
||||||
|
var removed Totals
|
||||||
|
|
||||||
|
err := db.Transaction(func(tx *gorm.DB) error {
|
||||||
|
var err error
|
||||||
|
|
||||||
|
removed, err = deleteExpired(tx, cutoff)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
return AddTotals(tx, removed)
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return removed.EventsRemoved, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// deleteExpired runs reapExpired's deletes and returns how many
|
||||||
|
// events, deliveries and failed deliveries they removed.
|
||||||
|
func deleteExpired(tx *gorm.DB, cutoff time.Time) (Totals, error) {
|
||||||
|
var removed Totals
|
||||||
|
|
||||||
// Fresh subqueries are built per statement to avoid reusing a
|
// Fresh subqueries are built per statement to avoid reusing a
|
||||||
// mutated builder across executions.
|
// mutated builder across executions.
|
||||||
expiredEventIDs := func() *gorm.DB {
|
expiredEventIDs := func() *gorm.DB {
|
||||||
return db.Model(&Event{}).
|
return tx.Model(&Event{}).
|
||||||
Select("id").
|
Select("id").
|
||||||
Where("created_at < ?", cutoff)
|
Where("created_at < ?", cutoff)
|
||||||
}
|
}
|
||||||
expiredDeliveryIDs := func() *gorm.DB {
|
expiredDeliveryIDs := func() *gorm.DB {
|
||||||
return db.Model(&Delivery{}).
|
return tx.Model(&Delivery{}).
|
||||||
Select("id").
|
Select("id").
|
||||||
Where("event_id IN (?)", expiredEventIDs())
|
Where("event_id IN (?)", expiredEventIDs())
|
||||||
}
|
}
|
||||||
|
|
||||||
// 1. Delivery results whose delivery belongs to an expired event.
|
// 1. Delivery results whose delivery belongs to an expired event.
|
||||||
res := db.Unscoped().
|
res := tx.Unscoped().
|
||||||
Where("delivery_id IN (?)", expiredDeliveryIDs()).
|
Where("delivery_id IN (?)", expiredDeliveryIDs()).
|
||||||
Delete(&DeliveryResult{})
|
Delete(&DeliveryResult{})
|
||||||
if res.Error != nil {
|
if res.Error != nil {
|
||||||
return 0, fmt.Errorf(
|
return removed, fmt.Errorf(
|
||||||
"deleting expired delivery results: %w",
|
"deleting expired delivery results: %w",
|
||||||
res.Error,
|
res.Error,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// 2. Deliveries belonging to an expired event.
|
// 2. Deliveries belonging to an expired event, after counting the
|
||||||
del := db.Unscoped().
|
// failed ones among them. The status is tested in the select list
|
||||||
|
// rather than the WHERE clause: there, SQLite would read every
|
||||||
|
// failed delivery the webhook has through the status index,
|
||||||
|
// instead of only the expired ones through the event_id index.
|
||||||
|
var failed struct{ Count int64 }
|
||||||
|
|
||||||
|
err := tx.Unscoped().Model(&Delivery{}).
|
||||||
|
Select("count(CASE WHEN status = ? THEN 1 END) AS count",
|
||||||
|
DeliveryStatusFailed).
|
||||||
|
Where("event_id IN (?)", expiredEventIDs()).
|
||||||
|
Take(&failed).Error
|
||||||
|
if err != nil {
|
||||||
|
return removed, fmt.Errorf(
|
||||||
|
"counting expired failed deliveries: %w", err,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
del := tx.Unscoped().
|
||||||
Where("event_id IN (?)", expiredEventIDs()).
|
Where("event_id IN (?)", expiredEventIDs()).
|
||||||
Delete(&Delivery{})
|
Delete(&Delivery{})
|
||||||
if del.Error != nil {
|
if del.Error != nil {
|
||||||
return 0, fmt.Errorf(
|
return removed, fmt.Errorf(
|
||||||
"deleting expired deliveries: %w",
|
"deleting expired deliveries: %w",
|
||||||
del.Error,
|
del.Error,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// 3. The expired events themselves.
|
// 3. The expired events themselves.
|
||||||
ev := db.Unscoped().
|
ev := tx.Unscoped().
|
||||||
Where("created_at < ?", cutoff).
|
Where("created_at < ?", cutoff).
|
||||||
Delete(&Event{})
|
Delete(&Event{})
|
||||||
if ev.Error != nil {
|
if ev.Error != nil {
|
||||||
return 0, fmt.Errorf(
|
return removed, fmt.Errorf(
|
||||||
"deleting expired events: %w",
|
"deleting expired events: %w",
|
||||||
ev.Error,
|
ev.Error,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
return ev.RowsAffected, nil
|
removed.EventsRemoved = ev.RowsAffected
|
||||||
|
removed.DeliveriesRemoved = del.RowsAffected
|
||||||
|
removed.FailuresRemoved = failed.Count
|
||||||
|
|
||||||
|
return removed, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,126 @@
|
|||||||
|
package database_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"gorm.io/gorm"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
)
|
||||||
|
|
||||||
|
// readTotals reads a webhook database's row of running totals,
|
||||||
|
// asserting that it has exactly one.
|
||||||
|
func readTotals(t *testing.T, db *gorm.DB) database.Totals {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var rows []database.Totals
|
||||||
|
|
||||||
|
require.NoError(t, db.Find(&rows).Error)
|
||||||
|
require.Len(t, rows, 1)
|
||||||
|
|
||||||
|
return rows[0]
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestWebhookDBManager_TotalsRowSurvivesReopen verifies that a new
|
||||||
|
// event database starts with one row of zero totals, and that opening
|
||||||
|
// it again keeps that row and what was added to it.
|
||||||
|
func TestWebhookDBManager_TotalsRowSurvivesReopen(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
mgr, lc := setupTestWebhookDBManager(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
require.NoError(t, lc.Start(ctx))
|
||||||
|
|
||||||
|
defer func() { require.NoError(t, lc.Stop(ctx)) }()
|
||||||
|
|
||||||
|
webhookID := uuid.New().String()
|
||||||
|
|
||||||
|
db, err := mgr.GetDB(webhookID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
fresh := readTotals(t, db)
|
||||||
|
assert.Equal(t, database.Totals{ID: fresh.ID}, fresh)
|
||||||
|
|
||||||
|
require.NoError(t, database.AddTotals(db, database.Totals{
|
||||||
|
Events: 2, Deliveries: 3, Failures: 1,
|
||||||
|
}))
|
||||||
|
|
||||||
|
// Drop the cached connection so the next open reopens the file,
|
||||||
|
// as a restart would.
|
||||||
|
require.NoError(t, mgr.CloseAll())
|
||||||
|
|
||||||
|
db, err = mgr.GetDB(webhookID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
assert.Equal(t, database.Totals{
|
||||||
|
ID: fresh.ID, Events: 2, Deliveries: 3, Failures: 1,
|
||||||
|
}, readTotals(t, db))
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRetentionReaper_AddsWhatItRemovesToTotals verifies that a sweep
|
||||||
|
// leaves the lifetime totals alone and adds the events, deliveries and
|
||||||
|
// failed deliveries it deletes to the removed totals, so the totals
|
||||||
|
// within retention match the rows still stored.
|
||||||
|
func TestRetentionReaper_AddsWhatItRemovesToTotals(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
env := setupRetentionTest(t)
|
||||||
|
|
||||||
|
webhookID := createWebhook(t, env.mainDB.DB(), 30)
|
||||||
|
|
||||||
|
db, err := env.mgr.GetDB(webhookID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
now := time.Now()
|
||||||
|
expired := now.Add(-40 * 24 * time.Hour)
|
||||||
|
|
||||||
|
seedEventChain(t, db, webhookID, expired)
|
||||||
|
expiredFailure := seedEventChain(t, db, webhookID, expired)
|
||||||
|
recentFailure := seedEventChain(
|
||||||
|
t, db, webhookID, now.Add(-24*time.Hour),
|
||||||
|
)
|
||||||
|
|
||||||
|
for _, id := range []string{
|
||||||
|
expiredFailure.deliveryID, recentFailure.deliveryID,
|
||||||
|
} {
|
||||||
|
require.NoError(t, db.Model(&database.Delivery{}).
|
||||||
|
Where("id = ?", id).
|
||||||
|
Update("status", database.DeliveryStatusFailed).Error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The totals storing those rows would have left.
|
||||||
|
require.NoError(t, database.AddTotals(db, database.Totals{
|
||||||
|
Events: 3, Deliveries: 3, Failures: 2,
|
||||||
|
}))
|
||||||
|
|
||||||
|
env.reaper.ExportSweep(context.Background())
|
||||||
|
|
||||||
|
totals := readTotals(t, db)
|
||||||
|
assert.Equal(t, database.Totals{
|
||||||
|
ID: totals.ID,
|
||||||
|
Events: 3, Deliveries: 3, Failures: 2,
|
||||||
|
EventsRemoved: 2, DeliveriesRemoved: 2, FailuresRemoved: 1,
|
||||||
|
}, totals)
|
||||||
|
|
||||||
|
var events, deliveries, failures int64
|
||||||
|
|
||||||
|
require.NoError(t, db.Model(&database.Event{}).Count(&events).Error)
|
||||||
|
require.NoError(t, db.Model(&database.Delivery{}).
|
||||||
|
Count(&deliveries).Error)
|
||||||
|
require.NoError(t, db.Model(&database.Delivery{}).
|
||||||
|
Where("status = ?", database.DeliveryStatusFailed).
|
||||||
|
Count(&failures).Error)
|
||||||
|
|
||||||
|
assert.Equal(t, events, totals.EventsWithinRetention())
|
||||||
|
assert.Equal(t, deliveries, totals.DeliveriesWithinRetention())
|
||||||
|
assert.Equal(t, failures, totals.FailuresWithinRetention())
|
||||||
|
|
||||||
|
// A sweep with nothing left to remove changes nothing.
|
||||||
|
env.reaper.ExportSweep(context.Background())
|
||||||
|
|
||||||
|
assert.Equal(t, totals, readTotals(t, db))
|
||||||
|
}
|
||||||
@@ -35,7 +35,8 @@ var errInvalidCachedDBType = errors.New(
|
|||||||
|
|
||||||
// WebhookDBManager manages per-webhook SQLite database files
|
// WebhookDBManager manages per-webhook SQLite database files
|
||||||
// for event storage. Each webhook gets its own dedicated
|
// for event storage. Each webhook gets its own dedicated
|
||||||
// database containing Events, Deliveries, and DeliveryResults.
|
// database containing Events, Deliveries, DeliveryResults and the
|
||||||
|
// running Totals of them.
|
||||||
// Database connections are opened lazily and cached.
|
// Database connections are opened lazily and cached.
|
||||||
type WebhookDBManager struct {
|
type WebhookDBManager struct {
|
||||||
dataDir string
|
dataDir string
|
||||||
@@ -294,7 +295,7 @@ func (m *WebhookDBManager) openDB(
|
|||||||
|
|
||||||
// Run migrations for event-tier models only
|
// Run migrations for event-tier models only
|
||||||
err = db.AutoMigrate(
|
err = db.AutoMigrate(
|
||||||
&Event{}, &Delivery{}, &DeliveryResult{},
|
&Event{}, &Delivery{}, &DeliveryResult{}, &Totals{},
|
||||||
)
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
_ = sqlDB.Close()
|
_ = sqlDB.Close()
|
||||||
@@ -305,6 +306,17 @@ func (m *WebhookDBManager) openDB(
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A new database gets its row of running totals, all zero.
|
||||||
|
err = db.FirstOrCreate(&Totals{}).Error
|
||||||
|
if err != nil {
|
||||||
|
_ = sqlDB.Close()
|
||||||
|
|
||||||
|
return nil, fmt.Errorf(
|
||||||
|
"creating running totals for webhook database %s: %w",
|
||||||
|
webhookID, err,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
m.log.Info(
|
m.log.Info(
|
||||||
"opened per-webhook database",
|
"opened per-webhook database",
|
||||||
"webhook_id", webhookID,
|
"webhook_id", webhookID,
|
||||||
|
|||||||
@@ -0,0 +1,100 @@
|
|||||||
|
package delivery_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"gorm.io/gorm"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
)
|
||||||
|
|
||||||
|
// failureTotal reads the running failure total of a webhook database.
|
||||||
|
func failureTotal(t *testing.T, db *gorm.DB) int64 {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var totals database.Totals
|
||||||
|
|
||||||
|
require.NoError(t, db.Take(&totals).Error)
|
||||||
|
|
||||||
|
return totals.Failures
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUpdateDeliveryStatus_FinishTimeAndFailureTotal pins what a status
|
||||||
|
// write records for the webhook page's statistics: the time a delivery
|
||||||
|
// finished, set only when it becomes delivered or failed, and one more
|
||||||
|
// on the failure total when it fails.
|
||||||
|
func TestUpdateDeliveryStatus_FinishTimeAndFailureTotal(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
status database.DeliveryStatus
|
||||||
|
finished bool
|
||||||
|
failures int64
|
||||||
|
}{
|
||||||
|
{database.DeliveryStatusRetrying, false, 0},
|
||||||
|
{database.DeliveryStatusDelivered, true, 0},
|
||||||
|
{database.DeliveryStatusFailed, true, 1},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(string(tt.status), func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
db := testWebhookDB(t)
|
||||||
|
e := testEngine(t, 1)
|
||||||
|
event := seedEvent(t, db, `{}`)
|
||||||
|
d := seedDelivery(
|
||||||
|
t, db, event.ID, uuid.New().String(),
|
||||||
|
database.DeliveryStatusPending,
|
||||||
|
)
|
||||||
|
|
||||||
|
before := time.Now()
|
||||||
|
|
||||||
|
require.NoError(t, e.ExportUpdateDeliveryStatus(
|
||||||
|
db, &d, tt.status,
|
||||||
|
))
|
||||||
|
|
||||||
|
var stored database.Delivery
|
||||||
|
|
||||||
|
require.NoError(t, db.First(&stored, "id = ?", d.ID).Error)
|
||||||
|
assert.Equal(t, tt.status, stored.Status)
|
||||||
|
|
||||||
|
if tt.finished {
|
||||||
|
require.NotNil(t, stored.FinishedAt)
|
||||||
|
assert.False(t, stored.FinishedAt.Before(before))
|
||||||
|
} else {
|
||||||
|
assert.Nil(t, stored.FinishedAt)
|
||||||
|
}
|
||||||
|
|
||||||
|
assert.Equal(t, tt.failures, failureTotal(t, db))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUpdateDeliveryStatus_DeletedDeliveryIsNotCounted covers a
|
||||||
|
// delivery retention deleted while the engine still held it. Failing
|
||||||
|
// it afterwards writes no row, so it adds no failure either: retention
|
||||||
|
// has already counted what it removed.
|
||||||
|
func TestUpdateDeliveryStatus_DeletedDeliveryIsNotCounted(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
db := testWebhookDB(t)
|
||||||
|
e := testEngine(t, 1)
|
||||||
|
event := seedEvent(t, db, `{}`)
|
||||||
|
d := seedDelivery(
|
||||||
|
t, db, event.ID, uuid.New().String(),
|
||||||
|
database.DeliveryStatusRetrying,
|
||||||
|
)
|
||||||
|
|
||||||
|
require.NoError(t, db.Unscoped().
|
||||||
|
Delete(&database.Delivery{}, "id = ?", d.ID).Error)
|
||||||
|
|
||||||
|
require.NoError(t, e.ExportUpdateDeliveryStatus(
|
||||||
|
db, &d, database.DeliveryStatusFailed,
|
||||||
|
))
|
||||||
|
|
||||||
|
assert.Zero(t, failureTotal(t, db))
|
||||||
|
}
|
||||||
@@ -1554,8 +1554,9 @@ func (e *Engine) updateDeliveryStatus(
|
|||||||
targetType database.TargetType,
|
targetType database.TargetType,
|
||||||
status database.DeliveryStatus,
|
status database.DeliveryStatus,
|
||||||
) error {
|
) error {
|
||||||
err := webhookDB.Model(d).
|
err := webhookDB.Transaction(func(tx *gorm.DB) error {
|
||||||
Update("status", status).Error
|
return writeDeliveryStatus(tx, d, status)
|
||||||
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf(
|
return fmt.Errorf(
|
||||||
"updating delivery %s to status %s: %w",
|
"updating delivery %s to status %s: %w",
|
||||||
@@ -1574,6 +1575,33 @@ func (e *Engine) updateDeliveryStatus(
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// writeDeliveryStatus writes a delivery's new status. A delivery that
|
||||||
|
// becomes delivered or failed also gets the time it finished, and a
|
||||||
|
// failed one is added to the webhook's running failure total. The
|
||||||
|
// failure is counted only if the row was still there to update:
|
||||||
|
// retention may have deleted it while the engine was working on it.
|
||||||
|
func writeDeliveryStatus(
|
||||||
|
tx *gorm.DB,
|
||||||
|
d *database.Delivery,
|
||||||
|
status database.DeliveryStatus,
|
||||||
|
) error {
|
||||||
|
columns := map[string]any{"status": status}
|
||||||
|
if status.Terminal() {
|
||||||
|
columns["finished_at"] = time.Now()
|
||||||
|
}
|
||||||
|
|
||||||
|
res := tx.Model(d).Updates(columns)
|
||||||
|
if res.Error != nil {
|
||||||
|
return res.Error
|
||||||
|
}
|
||||||
|
|
||||||
|
if status == database.DeliveryStatusFailed && res.RowsAffected > 0 {
|
||||||
|
return database.AddTotals(tx, database.Totals{Failures: 1})
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// settleStatus moves a delivery to its outcome status and reports a
|
// settleStatus moves a delivery to its outcome status and reports a
|
||||||
// failed write through bookkeepingFailed, which leaves the row
|
// failed write through bookkeepingFailed, which leaves the row
|
||||||
// recoverable. It exists so the target call sites read as one
|
// recoverable. It exists so the target call sites read as one
|
||||||
|
|||||||
@@ -57,7 +57,9 @@ func testWebhookDB(t *testing.T) *gorm.DB {
|
|||||||
&database.Event{},
|
&database.Event{},
|
||||||
&database.Delivery{},
|
&database.Delivery{},
|
||||||
&database.DeliveryResult{},
|
&database.DeliveryResult{},
|
||||||
|
&database.Totals{},
|
||||||
))
|
))
|
||||||
|
require.NoError(t, db.Create(&database.Totals{}).Error)
|
||||||
|
|
||||||
return db
|
return db
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -150,6 +150,16 @@ func (e *Engine) ExportDeliverSlack(
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ExportUpdateDeliveryStatus exposes updateDeliveryStatus. It passes no
|
||||||
|
// target type, so no metric moves.
|
||||||
|
func (e *Engine) ExportUpdateDeliveryStatus(
|
||||||
|
webhookDB *gorm.DB,
|
||||||
|
d *database.Delivery,
|
||||||
|
status database.DeliveryStatus,
|
||||||
|
) error {
|
||||||
|
return e.updateDeliveryStatus(webhookDB, d, "", status)
|
||||||
|
}
|
||||||
|
|
||||||
// ExportProcessNewTask exposes processNewTask.
|
// ExportProcessNewTask exposes processNewTask.
|
||||||
func (e *Engine) ExportProcessNewTask(
|
func (e *Engine) ExportProcessNewTask(
|
||||||
ctx context.Context, task *Task,
|
ctx context.Context, task *Task,
|
||||||
|
|||||||
@@ -453,3 +453,33 @@ func TestLogin_SuccessCreatesSession(t *testing.T) {
|
|||||||
"the issued cookie must carry an authenticated session",
|
"the issued cookie must carry an authenticated session",
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestLogin_UsernameAtLimitCanLogIn shows that a username of exactly
|
||||||
|
// database.MaxUsernameBytes still fits in the session cookie. Past
|
||||||
|
// what the cookie can carry, a correct login answers 500.
|
||||||
|
func TestLogin_UsernameAtLimitCanLogIn(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
db *database.Database
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &db)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
username := strings.Repeat("a", database.MaxUsernameBytes)
|
||||||
|
|
||||||
|
hash, err := database.HashPassword(operatorPassword)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, db.DB().Create(&database.User{
|
||||||
|
Username: username,
|
||||||
|
Password: hash,
|
||||||
|
}).Error)
|
||||||
|
|
||||||
|
w := submitLogin(h, sharedProxyPeer, username, operatorPassword)
|
||||||
|
|
||||||
|
assert.Equal(t, http.StatusSeeOther, w.Code)
|
||||||
|
}
|
||||||
|
|||||||
@@ -299,7 +299,8 @@ func countInFlightDeliveries(
|
|||||||
return count, err
|
return count, err
|
||||||
}
|
}
|
||||||
|
|
||||||
// createReplayDelivery writes the new pending delivery row and returns
|
// createReplayDelivery writes the new pending delivery row, adds it to
|
||||||
|
// the webhook's running totals in the same transaction, and returns
|
||||||
// the task that carries it to the delivery engine.
|
// the task that carries it to the delivery engine.
|
||||||
//
|
//
|
||||||
// The row is written with associations omitted, and neither Event nor
|
// The row is written with associations omitted, and neither Event nor
|
||||||
@@ -319,7 +320,14 @@ func createReplayDelivery(
|
|||||||
Status: database.DeliveryStatusPending,
|
Status: database.DeliveryStatusPending,
|
||||||
}
|
}
|
||||||
|
|
||||||
err := webhookDB.Omit(clause.Associations).Create(dlv).Error
|
err := webhookDB.Transaction(func(tx *gorm.DB) error {
|
||||||
|
err := tx.Omit(clause.Associations).Create(dlv).Error
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
return database.AddTotals(tx, database.Totals{Deliveries: 1})
|
||||||
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return delivery.Task{}, err
|
return delivery.Task{}, err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -69,6 +69,21 @@ func (s *Handlers) LoadEventLogViewsForTest(
|
|||||||
return views
|
return views
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// WebhookStatsForTest returns the figures the statistics pane on a
|
||||||
|
// webhook's page shows, from the webhook's entrypoints and targets
|
||||||
|
// loaded as that page loads them.
|
||||||
|
func (s *Handlers) WebhookStatsForTest(webhookID string) *WebhookStats {
|
||||||
|
var entrypoints []database.Entrypoint
|
||||||
|
|
||||||
|
s.db.DB().Where("webhook_id = ?", webhookID).Find(&entrypoints)
|
||||||
|
|
||||||
|
var targets []database.Target
|
||||||
|
|
||||||
|
s.db.DB().Where("webhook_id = ?", webhookID).Find(&targets)
|
||||||
|
|
||||||
|
return s.loadWebhookStats(webhookID, entrypoints, targets)
|
||||||
|
}
|
||||||
|
|
||||||
// AddTemplateForTest registers a template under a page name so that
|
// AddTemplateForTest registers a template under a page name so that
|
||||||
// the handlers_test package can drive the render path with a
|
// the handlers_test package can drive the render path with a
|
||||||
// template of its own.
|
// template of its own.
|
||||||
|
|||||||
@@ -91,18 +91,22 @@ type Handlers struct {
|
|||||||
|
|
||||||
// parsePageTemplate parses a page-specific template set from the
|
// parsePageTemplate parses a page-specific template set from the
|
||||||
// embedded FS. Each page template is combined with the shared
|
// embedded FS. Each page template is combined with the shared
|
||||||
// base, htmlheader, and navbar templates. The page file must be
|
// base, htmlheader, and navbar templates, and with any further files
|
||||||
// listed first so that its root action ({{template "base" .}})
|
// the page includes. The page file must be listed first so that its
|
||||||
// becomes the template set's entry point.
|
// root action ({{template "base" .}}) becomes the template set's entry
|
||||||
func parsePageTemplate(pageFile string) *template.Template {
|
// point.
|
||||||
return template.Must(
|
func parsePageTemplate(
|
||||||
template.ParseFS(
|
pageFile string, included ...string,
|
||||||
templates.Templates,
|
) *template.Template {
|
||||||
|
files := append([]string{
|
||||||
pageFile,
|
pageFile,
|
||||||
"base.html",
|
"base.html",
|
||||||
"htmlheader.html",
|
"htmlheader.html",
|
||||||
"navbar.html",
|
"navbar.html",
|
||||||
),
|
}, included...)
|
||||||
|
|
||||||
|
return template.Must(
|
||||||
|
template.ParseFS(templates.Templates, files...),
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -131,7 +135,7 @@ func New(
|
|||||||
"profile.html": parsePageTemplate("profile.html"),
|
"profile.html": parsePageTemplate("profile.html"),
|
||||||
"sources_list.html": parsePageTemplate("sources_list.html"),
|
"sources_list.html": parsePageTemplate("sources_list.html"),
|
||||||
"sources_new.html": parsePageTemplate("sources_new.html"),
|
"sources_new.html": parsePageTemplate("sources_new.html"),
|
||||||
"source_detail.html": parsePageTemplate("source_detail.html"),
|
"source_detail.html": parsePageTemplate("source_detail.html", "webhook_stats.html"),
|
||||||
"source_edit.html": parsePageTemplate("source_edit.html"),
|
"source_edit.html": parsePageTemplate("source_edit.html"),
|
||||||
"source_logs.html": parsePageTemplate("source_logs.html"),
|
"source_logs.html": parsePageTemplate("source_logs.html"),
|
||||||
"target_edit.html": parsePageTemplate("target_edit.html"),
|
"target_edit.html": parsePageTemplate("target_edit.html"),
|
||||||
|
|||||||
@@ -339,11 +339,9 @@ const storedUserPassword = "correct-horse-battery-staple"
|
|||||||
// storedFillBytes is the raw length of the client-chosen value in
|
// storedFillBytes is the raw length of the client-chosen value in
|
||||||
// those accounts' usernames. It is well past the 512-byte field
|
// those accounts' usernames. It is well past the 512-byte field
|
||||||
// budget, so the line is still truncated, but short enough that the
|
// budget, so the line is still truncated, but short enough that the
|
||||||
// session cookie a successful login writes stays inside
|
// whole username, markers and fill name included, stays within
|
||||||
// securecookie's 4 KB limit: the cookie is written BEFORE the
|
// database.MaxUsernameBytes.
|
||||||
// "user logged in" line, so an 8 KB username answers 500 and never
|
const storedFillBytes = 960
|
||||||
// reaches it.
|
|
||||||
const storedFillBytes = 1024
|
|
||||||
|
|
||||||
// storedFill builds a username fill of storedFillBytes raw bytes out
|
// storedFill builds a username fill of storedFillBytes raw bytes out
|
||||||
// of repetitions of ch, with both markers at its far end.
|
// of repetitions of ch, with both markers at its far end.
|
||||||
|
|||||||
@@ -450,6 +450,7 @@ func (h *Handlers) renderSourceDetail(
|
|||||||
"Targets": delivery.NewTargetViews(targets),
|
"Targets": delivery.NewTargetViews(targets),
|
||||||
"Events": events,
|
"Events": events,
|
||||||
"BaseURL": baseURL,
|
"BaseURL": baseURL,
|
||||||
|
"Stats": h.loadWebhookStats(webhook.ID, entrypoints, targets),
|
||||||
}
|
}
|
||||||
|
|
||||||
h.renderTemplate(w, r, "source_detail.html", data)
|
h.renderTemplate(w, r, "source_detail.html", data)
|
||||||
|
|||||||
@@ -252,11 +252,12 @@ func requestEventSource(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// createAndFanOut writes the event and one pending delivery per target
|
// createAndFanOut writes the event and one pending delivery per target,
|
||||||
// in a single transaction, then hands the tasks to the delivery
|
// and adds them to the webhook's running totals, in a single
|
||||||
// engine. It is the only path by which an event and its deliveries are
|
// transaction, then hands the tasks to the delivery engine. It is the
|
||||||
// created, so a resubmitted event is retried, SSRF-guarded and
|
// only path by which an event and its deliveries are created, so a
|
||||||
// circuit-broken exactly as a received one is.
|
// resubmitted event is retried, SSRF-guarded and circuit-broken
|
||||||
|
// exactly as a received one is.
|
||||||
//
|
//
|
||||||
// The tasks are returned as well as queued, so a caller can report how
|
// The tasks are returned as well as queued, so a caller can report how
|
||||||
// many targets the event went to.
|
// many targets the event went to.
|
||||||
@@ -296,6 +297,16 @@ func (h *Handlers) createAndFanOut(
|
|||||||
return nil, nil, err
|
return nil, nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
err = database.AddTotals(tx, database.Totals{
|
||||||
|
Events: 1,
|
||||||
|
Deliveries: int64(len(tasks)),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
tx.Rollback()
|
||||||
|
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
|
||||||
err = tx.Commit().Error
|
err = tx.Commit().Error
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, nil, fmt.Errorf(
|
return nil, nil, fmt.Errorf(
|
||||||
|
|||||||
@@ -0,0 +1,212 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gorm.io/gorm"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The spans of the two recent windows the statistics pane reports on:
|
||||||
|
// the last 10 minutes and the last 24 hours.
|
||||||
|
const (
|
||||||
|
shortWindow = 10 * time.Minute
|
||||||
|
longWindow = 24 * time.Hour
|
||||||
|
)
|
||||||
|
|
||||||
|
// percent turns a fraction into a percentage.
|
||||||
|
const percent = 100
|
||||||
|
|
||||||
|
// WebhookStats holds the figures in the statistics pane at the top of
|
||||||
|
// the webhook page.
|
||||||
|
type WebhookStats struct {
|
||||||
|
Entrypoints int
|
||||||
|
ActiveEntrypoints int
|
||||||
|
Targets int
|
||||||
|
ActiveTargets int
|
||||||
|
|
||||||
|
// Totals holds the lifetime counts of events, deliveries and
|
||||||
|
// failures, and how many of each retention has removed.
|
||||||
|
Totals database.Totals
|
||||||
|
|
||||||
|
// InProgress counts the deliveries still pending or retrying.
|
||||||
|
InProgress int64
|
||||||
|
|
||||||
|
// LastEventAt is when the newest stored event arrived, or nil when
|
||||||
|
// none is stored.
|
||||||
|
LastEventAt *time.Time
|
||||||
|
|
||||||
|
Last10Minutes RecentWindow
|
||||||
|
Last24Hours RecentWindow
|
||||||
|
}
|
||||||
|
|
||||||
|
// RecentWindow holds what happened in one recent window: the events
|
||||||
|
// received in it, and the deliveries that became delivered or failed in
|
||||||
|
// it.
|
||||||
|
type RecentWindow struct {
|
||||||
|
Events int64
|
||||||
|
Delivered int64
|
||||||
|
Failed int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// FailurePercent is the share of the deliveries finished in the window
|
||||||
|
// that failed, or a dash when none finished. Deliveries still pending
|
||||||
|
// or retrying are not counted either way.
|
||||||
|
func (w RecentWindow) FailurePercent() string {
|
||||||
|
finished := w.Delivered + w.Failed
|
||||||
|
if finished == 0 {
|
||||||
|
return "—"
|
||||||
|
}
|
||||||
|
|
||||||
|
return fmt.Sprintf(
|
||||||
|
"%.1f%%", percent*float64(w.Failed)/float64(finished),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// loadWebhookStats gathers the figures for the statistics pane from the
|
||||||
|
// webhook's entrypoints and targets, as the page has already loaded
|
||||||
|
// them, and from its event database. It returns nil, and logs why, when
|
||||||
|
// the event database cannot be read.
|
||||||
|
func (h *Handlers) loadWebhookStats(
|
||||||
|
webhookID string,
|
||||||
|
entrypoints []database.Entrypoint,
|
||||||
|
targets []database.Target,
|
||||||
|
) *WebhookStats {
|
||||||
|
stats := &WebhookStats{
|
||||||
|
Entrypoints: len(entrypoints),
|
||||||
|
Targets: len(targets),
|
||||||
|
}
|
||||||
|
|
||||||
|
for i := range entrypoints {
|
||||||
|
if entrypoints[i].Active {
|
||||||
|
stats.ActiveEntrypoints++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for i := range targets {
|
||||||
|
if targets[i].Active {
|
||||||
|
stats.ActiveTargets++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Opening an event database that does not exist would create it,
|
||||||
|
// and it would hold nothing to count.
|
||||||
|
if !h.dbMgr.DBExists(webhookID) {
|
||||||
|
return stats
|
||||||
|
}
|
||||||
|
|
||||||
|
webhookDB, err := h.dbMgr.GetDB(webhookID)
|
||||||
|
if err == nil {
|
||||||
|
err = readEventStats(webhookDB, time.Now(), stats)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
h.log.Error(
|
||||||
|
"failed to read webhook statistics",
|
||||||
|
"webhook_id", webhookID,
|
||||||
|
"error", err,
|
||||||
|
)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return stats
|
||||||
|
}
|
||||||
|
|
||||||
|
// readEventStats fills in the figures that come from the webhook's
|
||||||
|
// event database. None of them reads every stored row: the totals are
|
||||||
|
// one row, and every other figure is read from an index, over only the
|
||||||
|
// rows it counts.
|
||||||
|
func readEventStats(
|
||||||
|
db *gorm.DB, now time.Time, stats *WebhookStats,
|
||||||
|
) error {
|
||||||
|
err := db.Take(&stats.Totals).Error
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("reading running totals: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = db.Model(&database.Delivery{}).
|
||||||
|
Where("status IN ?", []database.DeliveryStatus{
|
||||||
|
database.DeliveryStatusPending,
|
||||||
|
database.DeliveryStatusRetrying,
|
||||||
|
}).
|
||||||
|
Count(&stats.InProgress).Error
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("counting deliveries in progress: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var newest []time.Time
|
||||||
|
|
||||||
|
err = db.Model(&database.Event{}).
|
||||||
|
Order("created_at DESC").
|
||||||
|
Limit(1).
|
||||||
|
Pluck("created_at", &newest).Error
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("reading newest event time: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(newest) > 0 {
|
||||||
|
stats.LastEventAt = &newest[0]
|
||||||
|
}
|
||||||
|
|
||||||
|
stats.Last10Minutes, err = readRecentWindow(
|
||||||
|
db, now.Add(-shortWindow),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
stats.Last24Hours, err = readRecentWindow(
|
||||||
|
db, now.Add(-longWindow),
|
||||||
|
)
|
||||||
|
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// readRecentWindow counts the events received, and the deliveries that
|
||||||
|
// became delivered or failed, since the given time.
|
||||||
|
func readRecentWindow(
|
||||||
|
db *gorm.DB, since time.Time,
|
||||||
|
) (RecentWindow, error) {
|
||||||
|
var w RecentWindow
|
||||||
|
|
||||||
|
err := db.Model(&database.Event{}).
|
||||||
|
Where("created_at >= ?", since).
|
||||||
|
Count(&w.Events).Error
|
||||||
|
if err != nil {
|
||||||
|
return w, fmt.Errorf("counting recent events: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
w.Delivered, err = countFinishedSince(
|
||||||
|
db, database.DeliveryStatusDelivered, since,
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
return w, err
|
||||||
|
}
|
||||||
|
|
||||||
|
w.Failed, err = countFinishedSince(
|
||||||
|
db, database.DeliveryStatusFailed, since,
|
||||||
|
)
|
||||||
|
|
||||||
|
return w, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// countFinishedSince counts the deliveries that reached the given
|
||||||
|
// final status since the given time.
|
||||||
|
func countFinishedSince(
|
||||||
|
db *gorm.DB, status database.DeliveryStatus, since time.Time,
|
||||||
|
) (int64, error) {
|
||||||
|
var n int64
|
||||||
|
|
||||||
|
err := db.Model(&database.Delivery{}).
|
||||||
|
Where("status = ? AND finished_at >= ?", status, since).
|
||||||
|
Count(&n).Error
|
||||||
|
if err != nil {
|
||||||
|
return 0, fmt.Errorf(
|
||||||
|
"counting deliveries %s recently: %w", status, err,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
return n, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,328 @@
|
|||||||
|
package handlers_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"go.uber.org/fx/fxtest"
|
||||||
|
"gorm.io/gorm"
|
||||||
|
"gorm.io/gorm/clause"
|
||||||
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/handlers"
|
||||||
|
"sneak.berlin/go/webhooker/internal/logger"
|
||||||
|
"sneak.berlin/go/webhooker/internal/session"
|
||||||
|
)
|
||||||
|
|
||||||
|
// statsEntrypoint adds an entrypoint to a webhook and returns its path.
|
||||||
|
func statsEntrypoint(
|
||||||
|
t *testing.T, db *database.Database, webhookID string, active bool,
|
||||||
|
) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
ep := &database.Entrypoint{
|
||||||
|
WebhookID: webhookID,
|
||||||
|
Path: uuid.New().String(),
|
||||||
|
}
|
||||||
|
|
||||||
|
require.NoError(t, db.DB().Omit(clause.Associations).Create(ep).Error)
|
||||||
|
require.NoError(t, db.DB().Model(ep).Update("active", active).Error)
|
||||||
|
|
||||||
|
return ep.Path
|
||||||
|
}
|
||||||
|
|
||||||
|
// statsDelivery returns the id of an event's delivery to a target.
|
||||||
|
func statsDelivery(
|
||||||
|
t *testing.T, webhookDB *gorm.DB, eventID, targetID string,
|
||||||
|
) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var d database.Delivery
|
||||||
|
|
||||||
|
require.NoError(t, webhookDB.Where(
|
||||||
|
"event_id = ? AND target_id = ?", eventID, targetID,
|
||||||
|
).First(&d).Error)
|
||||||
|
|
||||||
|
return d.ID
|
||||||
|
}
|
||||||
|
|
||||||
|
// statsFinish settles a delivery as the delivery engine does: its
|
||||||
|
// final status and the time it finished, and for a failure one more on
|
||||||
|
// the webhook's failure total, in one transaction.
|
||||||
|
func statsFinish(
|
||||||
|
t *testing.T,
|
||||||
|
webhookDB *gorm.DB,
|
||||||
|
deliveryID string,
|
||||||
|
status database.DeliveryStatus,
|
||||||
|
at time.Time,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
require.NoError(t, webhookDB.Transaction(func(tx *gorm.DB) error {
|
||||||
|
err := tx.Model(&database.Delivery{}).
|
||||||
|
Where("id = ?", deliveryID).
|
||||||
|
Updates(map[string]any{"status": status, "finished_at": at}).
|
||||||
|
Error
|
||||||
|
if err != nil || status != database.DeliveryStatusFailed {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
return database.AddTotals(tx, database.Totals{Failures: 1})
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
|
||||||
|
// statsAge moves an event's arrival back to the given time.
|
||||||
|
func statsAge(
|
||||||
|
t *testing.T, webhookDB *gorm.DB, eventID string, at time.Time,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
require.NoError(t, webhookDB.Model(&database.Event{}).
|
||||||
|
Where("id = ?", eventID).
|
||||||
|
Update("created_at", at).Error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// seedStatsHistory builds the webhook the statistics test checks: one
|
||||||
|
// day of retention, two entrypoints (one inactive) and three targets
|
||||||
|
// (one inactive). Three events arrive through the receiver, and so
|
||||||
|
// each has a delivery to the two active targets. The oldest event is
|
||||||
|
// past retention, the middle one six hours old, the newest just in.
|
||||||
|
// Their deliveries are settled as the delivery engine would, and a
|
||||||
|
// replay adds a pending delivery to the oldest event. It returns the
|
||||||
|
// webhook, its event database and the newest event.
|
||||||
|
func seedStatsHistory(
|
||||||
|
t *testing.T,
|
||||||
|
h *handlers.Handlers,
|
||||||
|
sess *session.Session,
|
||||||
|
db *database.Database,
|
||||||
|
dbMgr *database.WebhookDBManager,
|
||||||
|
) (*database.Webhook, *gorm.DB, database.Event) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
wh := &database.Webhook{
|
||||||
|
UserID: deleteTestUserID, Name: "stats", RetentionDays: 1,
|
||||||
|
}
|
||||||
|
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||||
|
|
||||||
|
path := statsEntrypoint(t, db, wh.ID, true)
|
||||||
|
statsEntrypoint(t, db, wh.ID, false)
|
||||||
|
|
||||||
|
first := seedConfiguredTarget(
|
||||||
|
t, db, wh.ID, database.TargetTypeHTTP,
|
||||||
|
`{"url":"`+replayTargetURL+`"}`,
|
||||||
|
)
|
||||||
|
second := seedTarget(t, db, wh.ID, database.TargetTypeLog)
|
||||||
|
inactive := seedTarget(t, db, wh.ID, database.TargetTypeLog)
|
||||||
|
require.NoError(t, db.DB().Model(inactive).
|
||||||
|
Update("active", false).Error)
|
||||||
|
|
||||||
|
router := receiverRouter(h)
|
||||||
|
|
||||||
|
for range 3 {
|
||||||
|
require.Equal(t, http.StatusOK, postReceiver(t, router, path))
|
||||||
|
}
|
||||||
|
|
||||||
|
webhookDB, err := dbMgr.GetDB(wh.ID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
events := listEvents(t, webhookDB)
|
||||||
|
require.Len(t, events, 3)
|
||||||
|
|
||||||
|
oldest, middle, newest := events[0], events[1], events[2]
|
||||||
|
now := time.Now()
|
||||||
|
|
||||||
|
statsAge(t, webhookDB, oldest.ID, now.Add(-50*time.Hour))
|
||||||
|
statsAge(t, webhookDB, middle.ID, now.Add(-6*time.Hour))
|
||||||
|
|
||||||
|
oldestFailure := statsDelivery(t, webhookDB, oldest.ID, first.ID)
|
||||||
|
statsFinish(t, webhookDB, oldestFailure,
|
||||||
|
database.DeliveryStatusFailed, now.Add(-49*time.Hour))
|
||||||
|
statsFinish(t, webhookDB,
|
||||||
|
statsDelivery(t, webhookDB, oldest.ID, second.ID),
|
||||||
|
database.DeliveryStatusDelivered, now.Add(-49*time.Hour))
|
||||||
|
statsFinish(t, webhookDB,
|
||||||
|
statsDelivery(t, webhookDB, middle.ID, first.ID),
|
||||||
|
database.DeliveryStatusFailed, now.Add(-5*time.Hour))
|
||||||
|
statsFinish(t, webhookDB,
|
||||||
|
statsDelivery(t, webhookDB, middle.ID, second.ID),
|
||||||
|
database.DeliveryStatusFailed, now.Add(-time.Minute))
|
||||||
|
statsFinish(t, webhookDB,
|
||||||
|
statsDelivery(t, webhookDB, newest.ID, first.ID),
|
||||||
|
database.DeliveryStatusDelivered, now.Add(-2*time.Minute))
|
||||||
|
|
||||||
|
require.Equal(t, http.StatusSeeOther,
|
||||||
|
postReplay(t, h, sess, wh.ID, oldestFailure).Code)
|
||||||
|
|
||||||
|
return wh, webhookDB, newest
|
||||||
|
}
|
||||||
|
|
||||||
|
// statsPrune runs the real retention reaper until it has removed one
|
||||||
|
// event from the webhook's database, then stops it.
|
||||||
|
func statsPrune(
|
||||||
|
t *testing.T,
|
||||||
|
db *database.Database,
|
||||||
|
dbMgr *database.WebhookDBManager,
|
||||||
|
log *logger.Logger,
|
||||||
|
webhookDB *gorm.DB,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
lc := fxtest.NewLifecycle(t)
|
||||||
|
database.NewRetentionReaper(lc, database.RetentionReaperParams{
|
||||||
|
Config: &config.Config{
|
||||||
|
RetentionSweepInterval: 10 * time.Millisecond,
|
||||||
|
},
|
||||||
|
Database: db,
|
||||||
|
DBManager: dbMgr,
|
||||||
|
Logger: log,
|
||||||
|
})
|
||||||
|
|
||||||
|
lc.RequireStart()
|
||||||
|
|
||||||
|
require.Eventually(t, func() bool {
|
||||||
|
var totals database.Totals
|
||||||
|
|
||||||
|
err := webhookDB.Take(&totals).Error
|
||||||
|
|
||||||
|
return err == nil && totals.EventsRemoved == 1
|
||||||
|
}, 10*time.Second, 10*time.Millisecond)
|
||||||
|
|
||||||
|
lc.RequireStop()
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertStatsTotals checks the lifetime events, deliveries and
|
||||||
|
// failures, and those within retention.
|
||||||
|
func assertStatsTotals(
|
||||||
|
t *testing.T, totals database.Totals, lifetime, within [3]int64,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
gotLifetime := [3]int64{
|
||||||
|
totals.Events, totals.Deliveries, totals.Failures,
|
||||||
|
}
|
||||||
|
gotWithin := [3]int64{
|
||||||
|
totals.EventsWithinRetention(),
|
||||||
|
totals.DeliveriesWithinRetention(),
|
||||||
|
totals.FailuresWithinRetention(),
|
||||||
|
}
|
||||||
|
|
||||||
|
assert.Equal(t, lifetime, gotLifetime,
|
||||||
|
"lifetime events, deliveries, failures")
|
||||||
|
assert.Equal(t, within, gotWithin,
|
||||||
|
"events, deliveries, failures within retention")
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestWebhookStats_EveryFigureAcrossRetentionPrune checks every figure
|
||||||
|
// the statistics pane shows for the history seedStatsHistory builds,
|
||||||
|
// before and after the real retention reaper removes the oldest event.
|
||||||
|
func TestWebhookStats_EveryFigureAcrossRetentionPrune(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
log *logger.Logger
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
wh, webhookDB, newest := seedStatsHistory(t, h, sess, db, dbMgr)
|
||||||
|
|
||||||
|
stats := h.WebhookStatsForTest(wh.ID)
|
||||||
|
require.NotNil(t, stats)
|
||||||
|
|
||||||
|
assert.Equal(t, 2, stats.Entrypoints)
|
||||||
|
assert.Equal(t, 1, stats.ActiveEntrypoints)
|
||||||
|
assert.Equal(t, 3, stats.Targets)
|
||||||
|
assert.Equal(t, 2, stats.ActiveTargets)
|
||||||
|
assertStatsTotals(t, stats.Totals, [3]int64{3, 7, 3}, [3]int64{3, 7, 3})
|
||||||
|
assert.Equal(t, int64(2), stats.InProgress)
|
||||||
|
require.NotNil(t, stats.LastEventAt)
|
||||||
|
assert.True(t, newest.CreatedAt.Equal(*stats.LastEventAt))
|
||||||
|
assert.Equal(t, handlers.RecentWindow{
|
||||||
|
Events: 1, Delivered: 1, Failed: 1,
|
||||||
|
}, stats.Last10Minutes)
|
||||||
|
assert.Equal(t, handlers.RecentWindow{
|
||||||
|
Events: 2, Delivered: 1, Failed: 2,
|
||||||
|
}, stats.Last24Hours)
|
||||||
|
assert.Equal(t, "50.0%", stats.Last10Minutes.FailurePercent())
|
||||||
|
assert.Equal(t, "66.7%", stats.Last24Hours.FailurePercent())
|
||||||
|
|
||||||
|
// Retention removes the oldest event with its three deliveries,
|
||||||
|
// one of them failed and one the pending replay.
|
||||||
|
statsPrune(t, db, dbMgr, log, webhookDB)
|
||||||
|
|
||||||
|
after := h.WebhookStatsForTest(wh.ID)
|
||||||
|
require.NotNil(t, after)
|
||||||
|
|
||||||
|
assertStatsTotals(t, after.Totals, [3]int64{3, 7, 3}, [3]int64{2, 4, 2})
|
||||||
|
assert.Equal(t, int64(1), after.InProgress)
|
||||||
|
assert.Equal(t, stats.LastEventAt, after.LastEventAt)
|
||||||
|
assert.Equal(t, stats.Last10Minutes, after.Last10Minutes)
|
||||||
|
assert.Equal(t, stats.Last24Hours, after.Last24Hours)
|
||||||
|
|
||||||
|
body := renderSourceDetailPage(t, h, sess, wh.ID)
|
||||||
|
assert.Contains(t, body, "Statistics")
|
||||||
|
assert.Contains(t, body, "Within retention")
|
||||||
|
assert.Contains(t, body, "50.0%")
|
||||||
|
assert.Contains(t, body, "66.7%")
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestWebhookStats_WebhookWithNoEvents covers a webhook whose event
|
||||||
|
// database has never been opened: every count is zero, the
|
||||||
|
// percentages are a dash, and showing the page does not create the
|
||||||
|
// database.
|
||||||
|
func TestWebhookStats_WebhookWithNoEvents(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
wh := seedWebhook(t, db)
|
||||||
|
|
||||||
|
assert.Equal(t, &handlers.WebhookStats{}, h.WebhookStatsForTest(wh.ID))
|
||||||
|
assert.Equal(t, "—", handlers.RecentWindow{}.FailurePercent())
|
||||||
|
|
||||||
|
body := renderSourceDetailPage(t, h, sess, wh.ID)
|
||||||
|
assert.Contains(t, body, "Statistics")
|
||||||
|
assert.False(t, dbMgr.DBExists(wh.ID))
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRecentWindow_FailurePercent pins the percentage: failed
|
||||||
|
// deliveries out of all that finished in the window.
|
||||||
|
func TestRecentWindow_FailurePercent(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
window handlers.RecentWindow
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{handlers.RecentWindow{}, "—"},
|
||||||
|
{handlers.RecentWindow{Events: 4}, "—"},
|
||||||
|
{handlers.RecentWindow{Delivered: 3, Failed: 1}, "25.0%"},
|
||||||
|
{handlers.RecentWindow{Failed: 2}, "100.0%"},
|
||||||
|
{handlers.RecentWindow{Delivered: 2}, "0.0%"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
assert.Equal(t, tt.want, tt.window.FailurePercent(), tt.window)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -92,11 +92,25 @@ func (s *Server) setupGlobalMiddleware() {
|
|||||||
func (s *Server) setupRoutes() {
|
func (s *Server) setupRoutes() {
|
||||||
s.router.Get("/", s.h.HandleIndex())
|
s.router.Get("/", s.h.HandleIndex())
|
||||||
|
|
||||||
s.router.Mount(
|
// Static assets answer GET and HEAD only. chi's default 405
|
||||||
"/s",
|
// carries no Allow header, so this group supplies its own.
|
||||||
http.StripPrefix("/s", http.FileServer(http.FS(static.Static))),
|
staticFiles := http.StripPrefix(
|
||||||
|
"/s", http.FileServer(http.FS(static.Static)),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
s.router.Route("/s", func(r chi.Router) {
|
||||||
|
r.MethodNotAllowed(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.Header().Set("Allow", "GET, HEAD")
|
||||||
|
http.Error(
|
||||||
|
w,
|
||||||
|
"Method Not Allowed",
|
||||||
|
http.StatusMethodNotAllowed,
|
||||||
|
)
|
||||||
|
})
|
||||||
|
r.Method(http.MethodGet, "/*", staticFiles)
|
||||||
|
r.Method(http.MethodHead, "/*", staticFiles)
|
||||||
|
})
|
||||||
|
|
||||||
s.router.Route("/api/v1", func(_ chi.Router) {
|
s.router.Route("/api/v1", func(_ chi.Router) {
|
||||||
// API routes will be added here.
|
// API routes will be added here.
|
||||||
})
|
})
|
||||||
|
|||||||
+109
-20
@@ -7,6 +7,7 @@ import (
|
|||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"net/url"
|
"net/url"
|
||||||
"regexp"
|
"regexp"
|
||||||
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -220,9 +221,21 @@ func (e *testEnv) csrfFrom(
|
|||||||
// out of the markup has to be unescaped before it is submitted.
|
// out of the markup has to be unescaped before it is submitted.
|
||||||
token := html.UnescapeString(match[1])
|
token := html.UnescapeString(match[1])
|
||||||
|
|
||||||
combined := make([]*http.Cookie, 0, len(cookies))
|
// A cookie the page sets replaces the one of the same name, as in
|
||||||
combined = append(combined, cookies...)
|
// a browser. Sent both, the server would read the first, older one.
|
||||||
combined = append(combined, w.Result().Cookies()...)
|
set := w.Result().Cookies()
|
||||||
|
combined := make([]*http.Cookie, 0, len(cookies)+len(set))
|
||||||
|
|
||||||
|
for _, c := range cookies {
|
||||||
|
replaced := slices.ContainsFunc(set, func(n *http.Cookie) bool {
|
||||||
|
return n.Name == c.Name
|
||||||
|
})
|
||||||
|
if !replaced {
|
||||||
|
combined = append(combined, c)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
combined = append(combined, set...)
|
||||||
|
|
||||||
return token, combined
|
return token, combined
|
||||||
}
|
}
|
||||||
@@ -396,13 +409,15 @@ func (e *testEnv) storedHash(t *testing.T, username string) string {
|
|||||||
|
|
||||||
// --- /s static group ---
|
// --- /s static group ---
|
||||||
|
|
||||||
// TestStaticServesEveryMethod pins what the static mount actually
|
// TestStaticServesOnlyGetAndHead pins the methods the static group
|
||||||
// answers. chi's Mount registers the handler for all methods and
|
// answers: GET and HEAD are served the asset, and the other methods
|
||||||
// http.FileServer only special-cases HEAD (by suppressing the body),
|
// chi routes (POST, PUT, DELETE and the rest) are refused with 405
|
||||||
// so a POST or a DELETE to an asset is served the file rather than
|
// and an Allow header naming those two. A method chi does not route,
|
||||||
// refused. The README documents this; the test is what keeps the two
|
// such as PROPFIND, is refused with 405 by the top-level router
|
||||||
// from drifting.
|
// before it reaches the static group, so it gets no Allow header.
|
||||||
func TestStaticServesEveryMethod(t *testing.T) {
|
// The README documents this; the test is what keeps the two from
|
||||||
|
// drifting.
|
||||||
|
func TestStaticServesOnlyGetAndHead(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
env := newTestEnv(t)
|
env := newTestEnv(t)
|
||||||
@@ -417,6 +432,7 @@ func TestStaticServesEveryMethod(t *testing.T) {
|
|||||||
http.MethodPost,
|
http.MethodPost,
|
||||||
http.MethodPut,
|
http.MethodPut,
|
||||||
http.MethodDelete,
|
http.MethodDelete,
|
||||||
|
"PROPFIND",
|
||||||
} {
|
} {
|
||||||
t.Run(method, func(t *testing.T) {
|
t.Run(method, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
@@ -428,18 +444,38 @@ func TestStaticServesEveryMethod(t *testing.T) {
|
|||||||
w := httptest.NewRecorder()
|
w := httptest.NewRecorder()
|
||||||
env.router.ServeHTTP(w, req)
|
env.router.ServeHTTP(w, req)
|
||||||
|
|
||||||
assert.Equal(t, http.StatusOK, w.Code,
|
switch method {
|
||||||
"static mount answers every method")
|
case http.MethodGet:
|
||||||
|
assert.Equal(t, http.StatusOK, w.Code)
|
||||||
if method == http.MethodHead {
|
|
||||||
assert.Empty(t, w.Body.Bytes(),
|
|
||||||
"HEAD must not carry a body")
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
assert.Equal(t, body, w.Body.Bytes(),
|
assert.Equal(t, body, w.Body.Bytes(),
|
||||||
"the asset itself is returned")
|
"the asset itself is returned")
|
||||||
|
case http.MethodHead:
|
||||||
|
assert.Equal(t, http.StatusOK, w.Code)
|
||||||
|
assert.Empty(t, w.Body.Bytes(),
|
||||||
|
"HEAD must not carry a body")
|
||||||
|
case "PROPFIND":
|
||||||
|
assert.Equal(
|
||||||
|
t, http.StatusMethodNotAllowed, w.Code,
|
||||||
|
)
|
||||||
|
assert.Empty(t, w.Header().Get("Allow"),
|
||||||
|
"chi refuses a method it does not route "+
|
||||||
|
"before the static group runs")
|
||||||
|
assert.NotContains(
|
||||||
|
t, w.Body.String(), string(body),
|
||||||
|
"a refused method must not get the asset",
|
||||||
|
)
|
||||||
|
default:
|
||||||
|
assert.Equal(
|
||||||
|
t, http.StatusMethodNotAllowed, w.Code,
|
||||||
|
)
|
||||||
|
assert.Equal(
|
||||||
|
t, "GET, HEAD", w.Header().Get("Allow"),
|
||||||
|
)
|
||||||
|
assert.NotContains(
|
||||||
|
t, w.Body.String(), string(body),
|
||||||
|
"a refused method must not get the asset",
|
||||||
|
)
|
||||||
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -591,6 +627,59 @@ func TestPagesLogin_CorrectPasswordSurvivesASpentBudget(
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestPagesLogin_CookiesFromAnEarlierDatabase is
|
||||||
|
// https://git.eeqj.de/sneak/webhooker/issues/359. A new database
|
||||||
|
// brings a new session key, and the operator's browser still holds
|
||||||
|
// the session and CSRF cookies signed with the old one. Logging in
|
||||||
|
// must work as from a fresh browser and leave cookies the new key
|
||||||
|
// accepts.
|
||||||
|
func TestPagesLogin_CookiesFromAnEarlierDatabase(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const (
|
||||||
|
username = "operator"
|
||||||
|
password = "correct-horse-battery-staple"
|
||||||
|
)
|
||||||
|
|
||||||
|
earlier := newTestEnv(t)
|
||||||
|
earlierID, _ := earlier.seedUser(t, username, password)
|
||||||
|
_, stale := earlier.csrfFrom(t, "/pages/login", nil)
|
||||||
|
stale = append(stale, earlier.authCookies(t, earlierID, username)...)
|
||||||
|
|
||||||
|
env := newTestEnv(t)
|
||||||
|
env.seedUser(t, username, password)
|
||||||
|
|
||||||
|
token, cookies := env.csrfFrom(t, "/pages/login", stale)
|
||||||
|
|
||||||
|
form := url.Values{}
|
||||||
|
form.Set("csrf_token", token)
|
||||||
|
form.Set("username", username)
|
||||||
|
form.Set("password", password)
|
||||||
|
|
||||||
|
w := env.post("/pages/login", form, cookies)
|
||||||
|
require.Equal(
|
||||||
|
t, http.StatusSeeOther, w.Code,
|
||||||
|
"a session cookie from another key must not fail the login",
|
||||||
|
)
|
||||||
|
|
||||||
|
// The response deletes the old session cookie and then sets the
|
||||||
|
// new one; a browser keeps the last.
|
||||||
|
var fresh *http.Cookie
|
||||||
|
|
||||||
|
for _, c := range w.Result().Cookies() {
|
||||||
|
if c.Name == session.SessionName {
|
||||||
|
fresh = c
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
require.NotNil(t, fresh, "login must set a session cookie")
|
||||||
|
assert.Equal(
|
||||||
|
t, "/sources",
|
||||||
|
env.get("/", []*http.Cookie{fresh}).Header().Get("Location"),
|
||||||
|
"the new session cookie must authenticate",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
// --- /user/{username} group ---
|
// --- /user/{username} group ---
|
||||||
|
|
||||||
// TestPasswordChange_OversizeBody_RejectedAndPasswordUnchanged
|
// TestPasswordChange_OversizeBody_RejectedAndPasswordUnchanged
|
||||||
|
|||||||
@@ -19,8 +19,8 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// The tests below exercise the securecookie codecs underneath the
|
// The tests below exercise the securecookie codecs underneath the
|
||||||
// store and nothing else: Session.Get only decodes, so no server-side
|
// store and nothing else: they decode through the store itself, so no
|
||||||
// expiry check takes part in the result. They exist because
|
// server-side expiry check takes part in the result. They exist because
|
||||||
// NewCookieStore gives its codecs a 30-day max age that assigning
|
// NewCookieStore gives its codecs a 30-day max age that assigning
|
||||||
// store.Options does not override, which would let the codec accept a
|
// store.Options does not override, which would let the codec accept a
|
||||||
// cookie weeks past the cap the cookie attribute advertises.
|
// cookie weeks past the cap the cookie attribute advertises.
|
||||||
@@ -75,10 +75,11 @@ func restamp(
|
|||||||
return base64.URLEncoding.EncodeToString(payload)
|
return base64.URLEncoding.EncodeToString(payload)
|
||||||
}
|
}
|
||||||
|
|
||||||
// decodeCookie feeds value back through the store's decode path.
|
// decodeCookie feeds value back through the store's decode path. It
|
||||||
|
// asks the store rather than Session.Get, which treats a cookie that
|
||||||
|
// does not decode as absent and so hides the codec's reason.
|
||||||
func decodeCookie(
|
func decodeCookie(
|
||||||
t *testing.T,
|
t *testing.T,
|
||||||
s *session.Session,
|
|
||||||
value string,
|
value string,
|
||||||
) (*sessions.Session, error) {
|
) (*sessions.Session, error) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
@@ -94,7 +95,7 @@ func decodeCookie(
|
|||||||
SameSite: http.SameSiteLaxMode,
|
SameSite: http.SameSiteLaxMode,
|
||||||
})
|
})
|
||||||
|
|
||||||
sess, err := s.Get(req)
|
sess, err := session.NewStore(testKey()).Get(req, session.SessionName)
|
||||||
require.NotNil(t, sess)
|
require.NotNil(t, sess)
|
||||||
|
|
||||||
return sess, err
|
return sess, err
|
||||||
@@ -105,7 +106,7 @@ func TestCodec_AcceptsCookieInsideAbsoluteCap(t *testing.T) {
|
|||||||
|
|
||||||
s := testSession(t)
|
s := testSession(t)
|
||||||
|
|
||||||
sess, err := decodeCookie(t, s, restamp(
|
sess, err := decodeCookie(t, restamp(
|
||||||
t,
|
t,
|
||||||
issuedCookie(t, s),
|
issuedCookie(t, s),
|
||||||
time.Now().Add(-(testAbsoluteMaxAge-time.Hour)),
|
time.Now().Add(-(testAbsoluteMaxAge-time.Hour)),
|
||||||
@@ -126,7 +127,7 @@ func TestCodec_RejectsCookiePastAbsoluteCap(t *testing.T) {
|
|||||||
|
|
||||||
s := testSession(t)
|
s := testSession(t)
|
||||||
|
|
||||||
sess, err := decodeCookie(t, s, restamp(
|
sess, err := decodeCookie(t, restamp(
|
||||||
t,
|
t,
|
||||||
issuedCookie(t, s),
|
issuedCookie(t, s),
|
||||||
time.Now().Add(-(testAbsoluteMaxAge+time.Hour)),
|
time.Now().Add(-(testAbsoluteMaxAge+time.Hour)),
|
||||||
|
|||||||
@@ -224,10 +224,22 @@ func New(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Get retrieves a session for the request.
|
// Get retrieves a session for the request.
|
||||||
|
//
|
||||||
|
// A session cookie that does not decode -- one signed with an earlier
|
||||||
|
// session key, say, because the database was made anew -- is treated
|
||||||
|
// as absent: the caller gets a new, empty session and no error, and
|
||||||
|
// the next save replaces the cookie.
|
||||||
func (s *Session) Get(
|
func (s *Session) Get(
|
||||||
r *http.Request,
|
r *http.Request,
|
||||||
) (*sessions.Session, error) {
|
) (*sessions.Session, error) {
|
||||||
return s.store.Get(r, SessionName)
|
sess, err := s.store.Get(r, SessionName)
|
||||||
|
if sess == nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// For a cookie that does not decode, gorilla/sessions returns a
|
||||||
|
// new, empty session alongside the error that is dropped here.
|
||||||
|
return sess, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetKey returns the raw 32-byte authentication key used for
|
// GetKey returns the raw 32-byte authentication key used for
|
||||||
|
|||||||
@@ -24,6 +24,8 @@
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
{{template "webhook_stats" .}}
|
||||||
|
|
||||||
<div class="grid grid-cols-1 lg:grid-cols-2 gap-6">
|
<div class="grid grid-cols-1 lg:grid-cols-2 gap-6">
|
||||||
<!-- Entrypoints -->
|
<!-- Entrypoints -->
|
||||||
<div class="card">
|
<div class="card">
|
||||||
|
|||||||
@@ -0,0 +1,87 @@
|
|||||||
|
{{define "webhook_stats"}}
|
||||||
|
<!-- Statistics pane at the top of the webhook page. -->
|
||||||
|
<div class="card mb-6">
|
||||||
|
<div class="p-4 border-b border-gray-200">
|
||||||
|
<h2 class="text-lg font-medium text-gray-900">Statistics</h2>
|
||||||
|
</div>
|
||||||
|
{{with .Stats}}
|
||||||
|
<div class="p-4 flex flex-wrap gap-6 text-sm border-b border-gray-200">
|
||||||
|
<div>
|
||||||
|
<span class="text-gray-500">Entrypoints</span>
|
||||||
|
<span class="font-medium text-gray-900">{{.Entrypoints}}</span>
|
||||||
|
<span class="text-gray-500">({{.ActiveEntrypoints}} active)</span>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<span class="text-gray-500">Targets</span>
|
||||||
|
<span class="font-medium text-gray-900">{{.Targets}}</span>
|
||||||
|
<span class="text-gray-500">({{.ActiveTargets}} active)</span>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<span class="text-gray-500">Deliveries in progress</span>
|
||||||
|
<span class="font-medium text-gray-900">{{.InProgress}}</span>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<span class="text-gray-500">Last event</span>
|
||||||
|
<span class="font-medium text-gray-900">{{with .LastEventAt}}{{.Format "2006-01-02 15:04:05 UTC"}}{{else}}none{{end}}</span>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<span class="text-gray-500">Retention</span>
|
||||||
|
<span class="font-medium text-gray-900">{{$.Webhook.RetentionLabel}}</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="p-4 grid grid-cols-1 lg:grid-cols-2 gap-6 text-sm">
|
||||||
|
<div>
|
||||||
|
<div class="flex py-2 border-b border-gray-200 text-xs text-gray-500 uppercase tracking-wide">
|
||||||
|
<span class="flex-1"></span>
|
||||||
|
<span class="w-32 text-center">Lifetime</span>
|
||||||
|
<span class="w-32 text-center">Within retention</span>
|
||||||
|
</div>
|
||||||
|
<div class="divide-y divide-gray-100">
|
||||||
|
<div class="flex py-2">
|
||||||
|
<span class="flex-1 text-gray-600">Events</span>
|
||||||
|
<span class="w-32 text-center text-gray-900">{{.Totals.Events}}</span>
|
||||||
|
<span class="w-32 text-center text-gray-900">{{.Totals.EventsWithinRetention}}</span>
|
||||||
|
</div>
|
||||||
|
<div class="flex py-2">
|
||||||
|
<span class="flex-1 text-gray-600">Deliveries</span>
|
||||||
|
<span class="w-32 text-center text-gray-900">{{.Totals.Deliveries}}</span>
|
||||||
|
<span class="w-32 text-center text-gray-900">{{.Totals.DeliveriesWithinRetention}}</span>
|
||||||
|
</div>
|
||||||
|
<div class="flex py-2">
|
||||||
|
<span class="flex-1 text-gray-600">Failures</span>
|
||||||
|
<span class="w-32 text-center text-gray-900">{{.Totals.Failures}}</span>
|
||||||
|
<span class="w-32 text-center text-gray-900">{{.Totals.FailuresWithinRetention}}</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<div class="flex py-2 border-b border-gray-200 text-xs text-gray-500 uppercase tracking-wide">
|
||||||
|
<span class="flex-1"></span>
|
||||||
|
<span class="w-32 text-center">Last 10 minutes</span>
|
||||||
|
<span class="w-32 text-center">Last 24 hours</span>
|
||||||
|
</div>
|
||||||
|
<div class="divide-y divide-gray-100">
|
||||||
|
<div class="flex py-2">
|
||||||
|
<span class="flex-1 text-gray-600">Events</span>
|
||||||
|
<span class="w-32 text-center text-gray-900">{{.Last10Minutes.Events}}</span>
|
||||||
|
<span class="w-32 text-center text-gray-900">{{.Last24Hours.Events}}</span>
|
||||||
|
</div>
|
||||||
|
<div class="flex py-2">
|
||||||
|
<span class="flex-1 text-gray-600">Failures</span>
|
||||||
|
<span class="w-32 text-center text-gray-900">{{.Last10Minutes.Failed}}</span>
|
||||||
|
<span class="w-32 text-center text-gray-900">{{.Last24Hours.Failed}}</span>
|
||||||
|
</div>
|
||||||
|
<div class="flex py-2">
|
||||||
|
<span class="flex-1 text-gray-600">Failure percentage</span>
|
||||||
|
<span class="w-32 text-center text-gray-900">{{.Last10Minutes.FailurePercent}}</span>
|
||||||
|
<span class="w-32 text-center text-gray-900">{{.Last24Hours.FailurePercent}}</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<p class="mt-2 text-xs text-gray-500">Failure percentage is the failed deliveries out of all deliveries that finished in the window. Deliveries still pending or retrying are not counted.</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
{{else}}
|
||||||
|
<div class="p-4 text-sm text-gray-500">The statistics could not be read.</div>
|
||||||
|
{{end}}
|
||||||
|
</div>
|
||||||
|
{{end}}
|
||||||
Reference in New Issue
Block a user