Compare commits
14
Commits
3bba2df314
...
v1.0.0-rc1
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
573f7f986a | ||
|
|
373a2f828d | ||
|
|
267d5b12f2 | ||
|
|
e73beb20df | ||
|
|
c8efbd41d4 | ||
|
|
15377f792b | ||
|
|
f86ab6321d | ||
|
|
957de7f99f | ||
|
|
9199544999 | ||
|
|
128eb1b644 | ||
|
|
3de345fe6f | ||
|
|
a9d77e20d7 | ||
|
|
faf3da9a75 | ||
|
|
fa6a9ed4dc |
+83
-29
@@ -1,30 +1,84 @@
|
|||||||
# .git is sent so the build can derive the version it stamps into the binary
|
# .dockerignore does NOT use .gitignore semantics. Docker matches with
|
||||||
# (script/version). Its config, which can hold a remote URL carrying a
|
# moby/patternmatcher: filepath.Match plus `**`, so `*` does not cross
|
||||||
# credential and which `git describe` does not need, is left out of a
|
# `/` and an unprefixed pattern is anchored at the context root. Every
|
||||||
# directory context. A context sent as a tar is not filtered by this file, so
|
# depth-independent pattern therefore needs `**/`, or `config/.env` and
|
||||||
# it carries .git/config unless its sender leaves it out.
|
# `certs/server.key` still ship while this file reads as solved. Only
|
||||||
.git/config
|
# genuinely root-anchored entries go unprefixed. Never transplant these
|
||||||
|
# into .gitignore, where `**/` is wrong.
|
||||||
# No tracked file may be listed here: git in the build would see it as
|
|
||||||
# deleted and mark the version -dirty.
|
|
||||||
#
|
#
|
||||||
# .ci-fingerprint is deliberately NOT excluded: it is the CI cache barrier
|
# Matching is case-sensitive, so secrets use character ranges rather
|
||||||
# that keeps the check stages from replaying a cached pass. See the lint
|
# than an ALL-CAPS twin, which would still miss `Server.Key`.
|
||||||
# stage of the Dockerfile.
|
#
|
||||||
bin/
|
# Extend with this repo's own host-built artifacts, written anchored:
|
||||||
# Extracted from 3p/ by `make assets` inside the build; a host copy is not
|
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
|
||||||
# needed. The tarball in 3p/ must stay in the context.
|
# deletes the package directory from the context.
|
||||||
static/js/alpine.min.js
|
|
||||||
# The js-deps stage installs ESLint and prettier; a host copy would overwrite
|
# .git is sent without its config. Without a VERSION build argument the
|
||||||
# them at the `COPY . .` of the stages built on it.
|
# stage that compiles runs `git describe --tags --always` on .git, which
|
||||||
node_modules/
|
# does not need .git/config; that file can hold a credential, such as a
|
||||||
.env
|
# password in a remote URL or the token the CI checkout step stores there.
|
||||||
.env.*
|
# Each submodule keeps a config with the same exposure in its git directory
|
||||||
*.db
|
# under .git/modules/, nested again for a submodule's own submodules, or in
|
||||||
*.sqlite
|
# its own .git directory when it keeps one.
|
||||||
*.sqlite3
|
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
|
||||||
.DS_Store
|
# `deploy/config`, `config/lib`) loses its whole git directory, because
|
||||||
.idea/
|
# `**/.git/modules/**/config` also matches that segment's directory
|
||||||
.vscode/
|
# under .git/modules/. Go's version stamping then fails the build;
|
||||||
tmp/
|
# nothing leaks. Name such a submodule without that segment:
|
||||||
temp/
|
# `git submodule add --name`.
|
||||||
|
**/.git/config
|
||||||
|
**/.git/modules/**/config
|
||||||
|
|
||||||
|
# Agent scratch: one full checkout of the repo per in-flight agent.
|
||||||
|
# Anchored because it occurs once where agents run at the repo root.
|
||||||
|
# KNOWN GAP: a repo running agents in subdirectories still ships
|
||||||
|
# `services/api/.claude/` and must add its own anchored entry.
|
||||||
|
.claude
|
||||||
|
|
||||||
|
# Environment files. `*.env` covers bare `.env` and the `prod.env`
|
||||||
|
# convention. Re-include a committed template with a negation if the
|
||||||
|
# build needs one: `!docs/example.env`.
|
||||||
|
**/*.[eE][nN][vV]
|
||||||
|
**/.[eE][nN][vV].*
|
||||||
|
**/.[eE][nN][vV][rR][cC]
|
||||||
|
|
||||||
|
# Private keys and the bundles carrying them. Public certificates
|
||||||
|
# (*.crt, *.cer) are deliberately absent: they are legitimate inputs.
|
||||||
|
**/*.[pP][eE][mM]
|
||||||
|
**/*.[kK][eE][yY]
|
||||||
|
**/*.[pP]12
|
||||||
|
**/*.[pP][fF][xX]
|
||||||
|
**/[iI][dD]_[rR][sS][aA]
|
||||||
|
**/[iI][dD]_[dD][sS][aA]
|
||||||
|
**/[iI][dD]_[eE][cC][dD][sS][aA]
|
||||||
|
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
|
||||||
|
**/[iI][dD]_[eE][dD]25519
|
||||||
|
**/[iI][dD]_[eE][dD]25519_[sS][kK]
|
||||||
|
|
||||||
|
# Dependencies: restored inside the image, never copied in.
|
||||||
|
**/node_modules
|
||||||
|
|
||||||
|
# OS metadata.
|
||||||
|
**/.DS_Store
|
||||||
|
**/Thumbs.db
|
||||||
|
|
||||||
|
# Editor state: never a build input, and it churns COPY.
|
||||||
|
**/*.swp
|
||||||
|
**/*.swo
|
||||||
|
**/*~
|
||||||
|
**/*.bak
|
||||||
|
**/.idea
|
||||||
|
**/.vscode
|
||||||
|
**/*.sublime-*
|
||||||
|
|
||||||
|
# This repository's own host-built artifacts: the binary `make build`
|
||||||
|
# writes, and the Alpine.js file `make assets` extracts from 3p/ (the
|
||||||
|
# build extracts its own).
|
||||||
|
/bin
|
||||||
|
/static/js/alpine.min.js
|
||||||
|
|
||||||
|
# SQLite databases, which hold the session key and webhook payloads, at
|
||||||
|
# any depth.
|
||||||
|
**/*.db
|
||||||
|
**/*.sqlite
|
||||||
|
**/*.sqlite3
|
||||||
|
|||||||
@@ -10,3 +10,6 @@ insert_final_newline = true
|
|||||||
|
|
||||||
[Makefile]
|
[Makefile]
|
||||||
indent_style = tab
|
indent_style = tab
|
||||||
|
|
||||||
|
[*.go]
|
||||||
|
indent_style = tab
|
||||||
|
|||||||
@@ -1,37 +1,9 @@
|
|||||||
name: check
|
name: check
|
||||||
|
on: [push]
|
||||||
on:
|
|
||||||
push:
|
|
||||||
branches:
|
|
||||||
- '**'
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
check:
|
check:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout
|
# actions/checkout v4.2.2, 2026-02-22
|
||||||
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 2024-10-23
|
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
|
||||||
with:
|
- run: script/cibuild
|
||||||
# The superseded-status step needs history to walk ancestors (it
|
|
||||||
# aborts on a shallow clone).
|
|
||||||
fetch-depth: 0
|
|
||||||
|
|
||||||
- name: Mark superseded run statuses
|
|
||||||
# Gitea cancels the in-flight run when another commit is pushed to the
|
|
||||||
# same branch and records the cancellation as `failure`, so a commit
|
|
||||||
# that was never tested reads as a test result. The script rewrites
|
|
||||||
# those statuses to say what happened. See its header for why the
|
|
||||||
# state stays `failure` and not `skipped`.
|
|
||||||
env:
|
|
||||||
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
|
|
||||||
run: script/ci-mark-superseded
|
|
||||||
|
|
||||||
- name: Fingerprint the build context
|
|
||||||
# Writes the hash of the commit being checked into the context, which
|
|
||||||
# invalidates the `COPY . .` layer of every check stage: a commit
|
|
||||||
# that was never linted, format-checked, stylesheet-checked, tested
|
|
||||||
# and built cannot report success from cache.
|
|
||||||
run: git rev-parse HEAD > .ci-fingerprint
|
|
||||||
|
|
||||||
- name: Build Docker image (runs the gofmt check, golangci-lint, the stylesheet check, ESLint, the Markdown check, make test, make build)
|
|
||||||
run: script/cibuild
|
|
||||||
|
|||||||
+50
-21
@@ -1,3 +1,53 @@
|
|||||||
|
# OS
|
||||||
|
.DS_Store
|
||||||
|
Thumbs.db
|
||||||
|
|
||||||
|
# Editors
|
||||||
|
*.swp
|
||||||
|
*.swo
|
||||||
|
*~
|
||||||
|
*.bak
|
||||||
|
.idea/
|
||||||
|
.vscode/
|
||||||
|
*.sublime-*
|
||||||
|
|
||||||
|
# Agent scratch (worktrees of this repo, created and destroyed by
|
||||||
|
# in-flight tooling). Unanchored: .gitignore patterns already match at
|
||||||
|
# every depth, so no prefix is wanted here. This is not a .dockerignore
|
||||||
|
# entry and must not be given a `**/` prefix on the way into one.
|
||||||
|
.claude/
|
||||||
|
|
||||||
|
# Node
|
||||||
|
node_modules/
|
||||||
|
|
||||||
|
# Secrets. Unanchored like every entry above, so each matches at every
|
||||||
|
# depth. Matching is case-sensitive on Linux, so names use character
|
||||||
|
# ranges rather than a lowercase form that misses `Server.Key`.
|
||||||
|
|
||||||
|
# Environment files. `*.env` covers bare `.env` and the `prod.env`
|
||||||
|
# convention. Only the templates `example.env` and `sample.env` are
|
||||||
|
# re-included below. A repository that commits any other template adds
|
||||||
|
# its own negation after these lines, for example `!.env.example`.
|
||||||
|
*.[eE][nN][vV]
|
||||||
|
.[eE][nN][vV].*
|
||||||
|
.[eE][nN][vV][rR][cC]
|
||||||
|
!example.env
|
||||||
|
!sample.env
|
||||||
|
|
||||||
|
# Private keys and the bundles carrying them.
|
||||||
|
*.[pP][eE][mM]
|
||||||
|
*.[kK][eE][yY]
|
||||||
|
*.[pP]12
|
||||||
|
*.[pP][fF][xX]
|
||||||
|
[iI][dD]_[rR][sS][aA]
|
||||||
|
[iI][dD]_[dD][sS][aA]
|
||||||
|
[iI][dD]_[eE][cC][dD][sS][aA]
|
||||||
|
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
|
||||||
|
[iI][dD]_[eE][dD]25519
|
||||||
|
[iI][dD]_[eE][dD]25519_[sS][kK]
|
||||||
|
|
||||||
|
# This repository's own entries, after the shared content above.
|
||||||
|
|
||||||
# Binaries
|
# Binaries
|
||||||
*.exe
|
*.exe
|
||||||
*.dll
|
*.dll
|
||||||
@@ -15,24 +65,6 @@ bin/
|
|||||||
# Go vendor directory
|
# Go vendor directory
|
||||||
vendor/
|
vendor/
|
||||||
|
|
||||||
# ESLint, prettier and their dependencies, installed from yarn.lock
|
|
||||||
node_modules/
|
|
||||||
|
|
||||||
# IDE specific files
|
|
||||||
.idea/
|
|
||||||
*.swp
|
|
||||||
*.swo
|
|
||||||
*~
|
|
||||||
.vscode/
|
|
||||||
|
|
||||||
# OS specific files
|
|
||||||
.DS_Store
|
|
||||||
Thumbs.db
|
|
||||||
|
|
||||||
# Environment and config files
|
|
||||||
.env
|
|
||||||
.env.local
|
|
||||||
|
|
||||||
# Data directory (SQLite databases)
|
# Data directory (SQLite databases)
|
||||||
data/
|
data/
|
||||||
*.db
|
*.db
|
||||||
@@ -46,9 +78,6 @@ data/
|
|||||||
tmp/
|
tmp/
|
||||||
temp/
|
temp/
|
||||||
|
|
||||||
# CI cache barrier, written into the build context by the check workflow
|
|
||||||
.ci-fingerprint
|
|
||||||
|
|
||||||
# Alpine.js, extracted by `make assets` from its tarball in 3p/, which is
|
# Alpine.js, extracted by `make assets` from its tarball in 3p/, which is
|
||||||
# what is committed.
|
# what is committed.
|
||||||
/static/js/alpine.min.js
|
/static/js/alpine.min.js
|
||||||
|
|||||||
+73
-2
@@ -10,14 +10,21 @@ run:
|
|||||||
|
|
||||||
linters:
|
linters:
|
||||||
default: all
|
default: all
|
||||||
|
enable:
|
||||||
|
# Successor to the deprecated gomodguard. Named explicitly, rather than
|
||||||
|
# left to `default: all`, because it carries the module policy below.
|
||||||
|
- gomodguard_v2
|
||||||
disable:
|
disable:
|
||||||
# Genuinely incompatible with project patterns
|
# Genuinely incompatible with project patterns
|
||||||
- exhaustruct # Requires all struct fields
|
- exhaustruct # Requires all struct fields
|
||||||
- depguard # Dependency allow/block lists
|
- exhaustruct_v5 # Requires all struct fields (successor to exhaustruct)
|
||||||
- godot # Requires comments to end with periods
|
- godot # Requires comments to end with periods
|
||||||
- wsl # Deprecated, replaced by wsl_v5
|
|
||||||
- wrapcheck # Too verbose for internal packages
|
- wrapcheck # Too verbose for internal packages
|
||||||
- varnamelen # Short names like db, id are idiomatic Go
|
- varnamelen # Short names like db, id are idiomatic Go
|
||||||
|
# Deprecated: the warning is attached to the old name, so it is
|
||||||
|
# silenced by disabling that name, not by enabling the successor.
|
||||||
|
- wsl # Deprecated, replaced by wsl_v5
|
||||||
|
- gomodguard # Deprecated, replaced by gomodguard_v2
|
||||||
settings:
|
settings:
|
||||||
lll:
|
lll:
|
||||||
line-length: 88
|
line-length: 88
|
||||||
@@ -28,6 +35,70 @@ linters:
|
|||||||
max-complexity: 15
|
max-complexity: 15
|
||||||
dupl:
|
dupl:
|
||||||
threshold: 100
|
threshold: 100
|
||||||
|
depguard:
|
||||||
|
# Test-support code must not be compiled into the shipped binary. A
|
||||||
|
# test-support package exists to hand a test privileges the program
|
||||||
|
# itself must never have, so a file that is not a test must not import
|
||||||
|
# one. Test files, and the files inside a package whose directory name
|
||||||
|
# ends in `test`, are where that code belongs, and are exempt.
|
||||||
|
#
|
||||||
|
# The deny list below is the one part of this file a repository is
|
||||||
|
# expected to extend, and the only part it may. depguard matches an
|
||||||
|
# import path against a list of prefixes, so it cannot be told "any path
|
||||||
|
# whose last segment ends in test"; a repository's own test-support
|
||||||
|
# packages have to be named here one at a time, by full import path,
|
||||||
|
# under a module path that differs from repository to repository. Add
|
||||||
|
# them; change nothing else.
|
||||||
|
rules:
|
||||||
|
test-support:
|
||||||
|
list-mode: lax
|
||||||
|
files:
|
||||||
|
- "$all"
|
||||||
|
- "!$test"
|
||||||
|
- "!**/*test/**"
|
||||||
|
deny:
|
||||||
|
- pkg: net/http/httptest
|
||||||
|
desc: >-
|
||||||
|
Test-support code belongs in test files and in packages whose
|
||||||
|
directory name ends in test, not in the shipped binary.
|
||||||
|
- pkg: sneak.berlin/go/webhooker/internal/config/configtest
|
||||||
|
desc: test support; a file that is not a test must not import it
|
||||||
|
- pkg: sneak.berlin/go/webhooker/internal/database/databasetest
|
||||||
|
desc: test support; a file that is not a test must not import it
|
||||||
|
- pkg: sneak.berlin/go/webhooker/internal/middleware/middlewaretest
|
||||||
|
desc: test support; a file that is not a test must not import it
|
||||||
|
# Only decisions already recorded in the Go package defaults are
|
||||||
|
# listed here. Every entry matches the module path exactly.
|
||||||
|
gomodguard_v2:
|
||||||
|
blocked:
|
||||||
|
- module: github.com/rs/zerolog
|
||||||
|
recommendations:
|
||||||
|
- log/slog
|
||||||
|
reason: "Structured logging is stdlib log/slog."
|
||||||
|
# One entry per pre-fork module path, because the later releases
|
||||||
|
# are separate paths. A prefix match would be shorter but would
|
||||||
|
# also reach github.com/go-redis/redismock, the test double for
|
||||||
|
# the successor these entries recommend.
|
||||||
|
- module: github.com/go-redis/redis
|
||||||
|
recommendations:
|
||||||
|
- github.com/redis/go-redis/v9
|
||||||
|
reason: "Pre-fork module; use the maintained go-redis v9."
|
||||||
|
- module: github.com/go-redis/redis/v7
|
||||||
|
recommendations:
|
||||||
|
- github.com/redis/go-redis/v9
|
||||||
|
reason: "Pre-fork module; use the maintained go-redis v9."
|
||||||
|
- module: github.com/go-redis/redis/v8
|
||||||
|
recommendations:
|
||||||
|
- github.com/redis/go-redis/v9
|
||||||
|
reason: "Pre-fork module; use the maintained go-redis v9."
|
||||||
|
- module: github.com/sergi/go-diff
|
||||||
|
recommendations:
|
||||||
|
- github.com/aymanbagabas/go-udiff
|
||||||
|
reason: "No unified diff output; use go-udiff."
|
||||||
|
- module: github.com/hexops/gotextdiff
|
||||||
|
recommendations:
|
||||||
|
- github.com/aymanbagabas/go-udiff
|
||||||
|
reason: "Unmaintained fork; use go-udiff."
|
||||||
|
|
||||||
issues:
|
issues:
|
||||||
max-issues-per-linter: 0
|
max-issues-per-linter: 0
|
||||||
|
|||||||
@@ -0,0 +1,2 @@
|
|||||||
|
node_modules/
|
||||||
|
yarn.lock
|
||||||
+128
-67
@@ -1,34 +1,3 @@
|
|||||||
# Lint stage
|
|
||||||
# golangci/golangci-lint:v2.12.2 (Debian-based), 2026-08-07
|
|
||||||
# Using Debian-based image because mattn/go-sqlite3 (CGO) does not
|
|
||||||
# compile on Alpine musl (off64_t is a glibc type).
|
|
||||||
FROM golangci/golangci-lint:v2.12.2@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
|
|
||||||
|
|
||||||
WORKDIR /src
|
|
||||||
|
|
||||||
# Copy go mod files first for better layer caching
|
|
||||||
COPY go.mod go.sum ./
|
|
||||||
RUN go mod download
|
|
||||||
|
|
||||||
# Copy source code. In CI the context also carries .ci-fingerprint, which
|
|
||||||
# holds the hash of the commit being checked (see
|
|
||||||
# .gitea/workflows/check.yml). That invalidates this layer, so the checks
|
|
||||||
# below cannot report success by replaying a cached pass. Do not add it to
|
|
||||||
# .dockerignore.
|
|
||||||
COPY . .
|
|
||||||
|
|
||||||
# Run the Go formatting check and the linter. gofmt and golangci-lint are
|
|
||||||
# invoked directly rather than through `make fmt-check` and `make lint`: this
|
|
||||||
# stage is already the pinned linter image, and both scripts build docker
|
|
||||||
# stages, so calling them here would need a docker daemon inside the build.
|
|
||||||
# The Markdown half of `make fmt-check` is the markdown-check stage below.
|
|
||||||
# Keep the golangci-lint steps in step with Dockerfile.lint, including
|
|
||||||
# --network=none (see its header for why).
|
|
||||||
RUN if [ -n "$(gofmt -s -l .)" ]; then echo "gofmt needed on:"; gofmt -s -l .; exit 1; fi
|
|
||||||
RUN script/assets
|
|
||||||
RUN --network=none golangci-lint config verify --config .golangci.yml
|
|
||||||
RUN --network=none golangci-lint run --config .golangci.yml --build-tags browser ./...
|
|
||||||
|
|
||||||
# Stylesheet stages. static/css/tailwind.css is generated, by this pinned
|
# Stylesheet stages. static/css/tailwind.css is generated, by this pinned
|
||||||
# tailwindcss, from static/css/input.css and the files its @source lines
|
# tailwindcss, from static/css/input.css and the files its @source lines
|
||||||
# name. `make css` (script/css) writes it out from the css-output stage.
|
# name. `make css` (script/css) writes it out from the css-output stage.
|
||||||
@@ -67,10 +36,7 @@ RUN sed 's/}/}\n/g' static/css/tailwind.css > /tmp/committed.css \
|
|||||||
|
|
||||||
# JavaScript lint stages: ESLint, at the version package.json and yarn.lock
|
# JavaScript lint stages: ESLint, at the version package.json and yarn.lock
|
||||||
# pin, checks static/js/ against eslint.config.mjs. js-deps installs it, and
|
# pin, checks static/js/ against eslint.config.mjs. js-deps installs it, and
|
||||||
# prettier for the Markdown stages below, and stays cached until package.json,
|
# prettier for the Markdown stages below. The lint phase below runs js-lint.
|
||||||
# yarn.lock or .yarnrc.yml changes. script/lint forces only js-lint to re-run,
|
|
||||||
# and the build stage below runs it too. COPY . . brings in the CI cache
|
|
||||||
# barrier described in the lint stage above.
|
|
||||||
#
|
#
|
||||||
# The image's own corepack runs the yarn that package.json's packageManager
|
# The image's own corepack runs the yarn that package.json's packageManager
|
||||||
# field names, yarn 4.18.1 (released 2026-09-24), and checks it against the
|
# field names, yarn 4.18.1 (released 2026-09-24), and checks it against the
|
||||||
@@ -104,23 +70,116 @@ FROM js-deps AS markdown-check
|
|||||||
COPY . .
|
COPY . .
|
||||||
RUN --network=none node_modules/.bin/prettier --check '**/*.md'
|
RUN --network=none node_modules/.bin/prettier --check '**/*.md'
|
||||||
|
|
||||||
|
# Lint phase: the Go formatting check and golangci-lint over the Go code,
|
||||||
|
# and ESLint over static/js/ through the copy from js-lint at the end.
|
||||||
|
# `make lint` (script/lint) builds this stage alone; the build stage below
|
||||||
|
# depends on it.
|
||||||
|
#
|
||||||
|
# golangci/golangci-lint:v2.14.0 (Debian-based), 2026-09-24
|
||||||
|
# Using Debian-based image because mattn/go-sqlite3 (CGO) does not
|
||||||
|
# compile on Alpine musl (off64_t is a glibc type).
|
||||||
|
FROM golangci/golangci-lint:v2.14.0@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
|
||||||
|
|
||||||
|
WORKDIR /src
|
||||||
|
|
||||||
|
# Copy go mod files first for better layer caching
|
||||||
|
COPY go.mod go.sum ./
|
||||||
|
RUN go mod download
|
||||||
|
|
||||||
|
COPY . .
|
||||||
|
|
||||||
|
# gofmt and golangci-lint are invoked directly rather than through `make
|
||||||
|
# fmt-check` and `make lint`, which are themselves docker builds and would
|
||||||
|
# need a docker daemon inside this one. The Markdown half of `make
|
||||||
|
# fmt-check` is the markdown-check stage above.
|
||||||
|
RUN if [ -n "$(gofmt -s -l .)" ]; then echo "gofmt needed on:"; gofmt -s -l .; exit 1; fi
|
||||||
|
|
||||||
|
# static/static.go embeds the Alpine.js file this extracts from 3p/; without
|
||||||
|
# it the static package does not compile and cannot be linted.
|
||||||
|
RUN script/assets
|
||||||
|
|
||||||
|
# The golangci-lint steps run with --network=none. `golangci-lint config
|
||||||
|
# verify` is documented as fetching its JSON schema over HTTPS; this pinned
|
||||||
|
# image resolves the schema without any network, and --network=none enforces
|
||||||
|
# that. It also proves no linter reaches out at analysis time.
|
||||||
|
#
|
||||||
|
# `run` silently ignores config keys it does not recognize, so a typo would
|
||||||
|
# disable a setting without a word. `config verify` is what catches that.
|
||||||
|
RUN --network=none golangci-lint config verify --config .golangci.yml
|
||||||
|
# --build-tags browser also lints the browser test, which is built only with
|
||||||
|
# that tag (make test-browser).
|
||||||
|
RUN --network=none golangci-lint run --config .golangci.yml --build-tags browser ./...
|
||||||
|
|
||||||
|
# Nothing is wanted from js-lint; the copy is what makes this phase run it.
|
||||||
|
COPY --from=js-lint /src/yarn.lock /dev/null
|
||||||
|
|
||||||
|
# Test phase. -race needs cgo and so a C compiler, which the Debian Go image
|
||||||
|
# ships and the alpine one does not. `make test` (script/test) builds this
|
||||||
|
# stage alone; the build stage below depends on it.
|
||||||
|
#
|
||||||
|
# golang:1.26.1-bookworm (Debian-based), 2026-03-17
|
||||||
|
FROM golang:1.26.1-bookworm@sha256:4465644228bc2857a954b092167e12aa59c006a3492282a6c820bf4755fd64a4 AS test
|
||||||
|
|
||||||
|
WORKDIR /src
|
||||||
|
|
||||||
|
COPY go.mod go.sum ./
|
||||||
|
RUN go mod download
|
||||||
|
|
||||||
|
COPY . .
|
||||||
|
|
||||||
|
# static/static.go embeds the Alpine.js file this extracts from 3p/.
|
||||||
|
RUN script/assets
|
||||||
|
|
||||||
|
# -timeout applies to each package on its own, so 90s has only to clear the
|
||||||
|
# slowest one. -p 4 -parallel 8 keep the run under 2 GB of memory: at most
|
||||||
|
# four test binaries build or run at once, each with at most eight parallel
|
||||||
|
# tests. Under -race every test binary and every link costs a few hundred MB,
|
||||||
|
# so the defaults (one per core) add up to several GB on a many-core host.
|
||||||
|
#
|
||||||
|
# The first run has no -v: go test then prints one result line per package,
|
||||||
|
# with its coverage, and for a package that fails, everything its tests
|
||||||
|
# wrote. Verbose output from the whole suite passes the 2 MiB at which the
|
||||||
|
# Docker build cuts off a step's log, so on a failure only the tests that
|
||||||
|
# failed run again, with -v. go test reports a failed test as a line starting
|
||||||
|
# "--- FAIL: TestName" (a failed subtest's line is indented, and reruns with
|
||||||
|
# its parent) and a failed package as "FAIL<tab>package/path<tab>...". A
|
||||||
|
# failure that names no test, such as a build error or a timeout, is already
|
||||||
|
# shown in full, so there is nothing to rerun. The step fails after the rerun
|
||||||
|
# whatever its result: the first run already showed the suite is broken.
|
||||||
|
#
|
||||||
|
# TMPDIR, where the tests keep their SQLite databases, is a tmpfs: SQLite
|
||||||
|
# waits for the disk at every commit, and on a busy host that waiting was
|
||||||
|
# about 40% of the slowest package's run time. GOTMPDIR keeps go's own
|
||||||
|
# build files, the test binaries among them, on disk.
|
||||||
|
#
|
||||||
|
# bash with pipefail, so that the first run's status is go test's, not tee's.
|
||||||
|
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
|
||||||
|
RUN --mount=type=tmpfs,target=/tmp/tests,size=512m \
|
||||||
|
export TMPDIR=/tmp/tests GOTMPDIR=/tmp; \
|
||||||
|
go test -race -cover -p 4 -parallel 8 -timeout 90s ./... 2>&1 | tee /tmp/go-test.log && exit 0; \
|
||||||
|
tests="$(awk '/^--- FAIL: / { print $3 }' /tmp/go-test.log | paste -s -d '|' -)"; \
|
||||||
|
packages="$(awk '/^FAIL\t/ { print $2 }' /tmp/go-test.log)"; \
|
||||||
|
if [ -n "$tests" ]; then \
|
||||||
|
echo "--- Rerunning the failed tests with -v for details ---"; \
|
||||||
|
go test -race -v -p 4 -parallel 8 -timeout 90s -run "^($tests)\$" $packages; \
|
||||||
|
fi; \
|
||||||
|
exit 1
|
||||||
|
|
||||||
# Build stage
|
# Build stage
|
||||||
# golang:1.26.1-bookworm (Debian-based), 2026-03-17
|
# golang:1.26.1-bookworm (Debian-based), 2026-03-17
|
||||||
# Using Debian-based image because gorm.io/driver/sqlite pulls in
|
# Using Debian-based image because gorm.io/driver/sqlite pulls in
|
||||||
# mattn/go-sqlite3 (CGO), which does not compile on Alpine musl.
|
# mattn/go-sqlite3 (CGO), which does not compile on Alpine musl. The image
|
||||||
|
# ships git and make, which the version step below uses.
|
||||||
FROM golang:1.26.1-bookworm@sha256:4465644228bc2857a954b092167e12aa59c006a3492282a6c820bf4755fd64a4 AS builder
|
FROM golang:1.26.1-bookworm@sha256:4465644228bc2857a954b092167e12aa59c006a3492282a6c820bf4755fd64a4 AS builder
|
||||||
|
|
||||||
# Depend on the lint, stylesheet check, JavaScript lint and Markdown check
|
# Nothing is wanted from the lint and test phases or from the stylesheet and
|
||||||
# stages passing
|
# Markdown checks; the copies are what make BuildKit build them first, so
|
||||||
|
# this stage cannot run unless they all passed.
|
||||||
COPY --from=lint /src/go.sum /dev/null
|
COPY --from=lint /src/go.sum /dev/null
|
||||||
|
COPY --from=test /src/go.sum /dev/null
|
||||||
COPY --from=css-check /out/tailwind.css /dev/null
|
COPY --from=css-check /out/tailwind.css /dev/null
|
||||||
COPY --from=js-lint /src/yarn.lock /dev/null
|
|
||||||
COPY --from=markdown-check /src/yarn.lock /dev/null
|
COPY --from=markdown-check /src/yarn.lock /dev/null
|
||||||
|
|
||||||
# jq is a runtime dependency of script/ci-mark-superseded, which the test
|
|
||||||
# suite executes. git is what script/version derives the version with.
|
|
||||||
RUN apt-get update && apt-get install -y --no-install-recommends make curl ca-certificates jq git && rm -rf /var/lib/apt/lists/*
|
|
||||||
|
|
||||||
# A build context sent as a tar archive keeps its files' owners, and git
|
# A build context sent as a tar archive keeps its files' owners, and git
|
||||||
# refuses to read a checkout owned by another user. Trust this one
|
# refuses to read a checkout owned by another user. Trust this one
|
||||||
# whoever owns it.
|
# whoever owns it.
|
||||||
@@ -132,43 +191,45 @@ WORKDIR /build
|
|||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
|
|
||||||
# Copy source code, including the .ci-fingerprint cache barrier described in
|
|
||||||
# the lint stage above.
|
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# Run tests and build. Both first run script/assets, which extracts Alpine.js
|
|
||||||
# from its tarball in 3p/.
|
|
||||||
RUN make test
|
|
||||||
|
|
||||||
# Version stamped into the binary: the VERSION build arg when one is
|
# Version stamped into the binary: the VERSION build arg when one is
|
||||||
# given, otherwise what script/version derives from the .git the build
|
# given, otherwise what script/version derives from the .git the build
|
||||||
# context carries, so any `docker build .` of a clone stamps its commit.
|
# context carries, so any `docker build .` of a clone stamps its commit.
|
||||||
# With neither, as from a source tarball, it is "unknown".
|
# With neither, as from a source tarball, it is "unknown".
|
||||||
#
|
|
||||||
# Declared here, below the test step, so a changed version does not
|
|
||||||
# invalidate its cached layer.
|
|
||||||
ARG VERSION
|
ARG VERSION
|
||||||
|
|
||||||
# A context that carries .git must not stamp "unknown": that means git is
|
# A context that carries .git must not stamp an empty version, "dev" or
|
||||||
# missing here or could not read the checkout, and the image could not be
|
# "unknown": that means git is missing here or could not read the
|
||||||
# traced back to its commit.
|
# checkout, and the image could not be traced back to its commit.
|
||||||
RUN if [ -d .git ] && [ "$(make version VERSION="$VERSION")" = unknown ]; then \
|
RUN version="$(make version VERSION="$VERSION")"; \
|
||||||
echo "version is unknown although the build context carries .git" >&2; \
|
if [ -e .git ]; then \
|
||||||
exit 1; \
|
case "$version" in ""|dev|unknown) \
|
||||||
|
echo "version is '$version' although .git is present" >&2; \
|
||||||
|
exit 1 ;; \
|
||||||
|
esac; \
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Builds through the Makefile's build target, which runs script/assets
|
||||||
|
# (Alpine.js, extracted from its tarball in 3p/) first.
|
||||||
RUN make build VERSION="$VERSION"
|
RUN make build VERSION="$VERSION"
|
||||||
|
|
||||||
# Rebuild with static linking for Alpine runtime.
|
# Rebuild with static linking for the Alpine runtime. The cgo binary the
|
||||||
# make build already verified compilation.
|
# step above builds is dynamically linked against glibc, which Alpine
|
||||||
# The CGO binary from `make build` is dynamically linked against glibc,
|
# (musl) does not have.
|
||||||
# which doesn't exist on Alpine (musl). Rebuild with static linking so
|
|
||||||
# the binary runs on Alpine without glibc.
|
|
||||||
#
|
#
|
||||||
# The static flags go in through GO_LDFLAGS rather than a -ldflags of
|
# A statically linked glibc still loads shared libraries at run time for
|
||||||
# their own: the build target composes them with the -X that stamps the
|
# getaddrinfo and dlopen, and Alpine has none to load, so the build tags
|
||||||
# version, so this relink cannot silently drop the stamp.
|
# keep both out of the binary: netgo and osusergo select Go's own DNS
|
||||||
RUN CGO_ENABLED=1 make build VERSION="$VERSION" GO_LDFLAGS='-extldflags "-static"'
|
# resolver and user lookup instead of glibc's, and
|
||||||
|
# sqlite_omit_load_extension leaves out SQLite extension loading, which
|
||||||
|
# webhooker never uses.
|
||||||
|
#
|
||||||
|
# The static flags and tags go in through GO_LDFLAGS and GO_TAGS rather
|
||||||
|
# than a -ldflags of their own: the build target composes them with the
|
||||||
|
# -X that stamps the version, so this relink cannot silently drop the
|
||||||
|
# stamp.
|
||||||
|
RUN CGO_ENABLED=1 make build VERSION="$VERSION" GO_TAGS=netgo,osusergo,sqlite_omit_load_extension GO_LDFLAGS='-extldflags "-static"'
|
||||||
|
|
||||||
# Runtime stage
|
# Runtime stage
|
||||||
# alpine:3.21, 2026-03-17
|
# alpine:3.21, 2026-03-17
|
||||||
|
|||||||
+1
-1
@@ -16,7 +16,7 @@ COPY . .
|
|||||||
|
|
||||||
# The test binary embeds the templates and static files, so the browser
|
# The test binary embeds the templates and static files, so the browser
|
||||||
# stage needs nothing else. -p 4 keeps the compile's memory down, as in
|
# stage needs nothing else. -p 4 keeps the compile's memory down, as in
|
||||||
# script/test.
|
# the test phase of Dockerfile.
|
||||||
RUN make assets && go test -c -p 4 -tags browser -o /browser.test ./internal/server
|
RUN make assets && go test -c -p 4 -tags browser -o /browser.test ./internal/server
|
||||||
|
|
||||||
# chromedp/headless-shell:151.0.7922.109 (Debian trixie), 2026-08-11. The
|
# chromedp/headless-shell:151.0.7922.109 (Debian trixie), 2026-08-11. The
|
||||||
|
|||||||
@@ -1,43 +0,0 @@
|
|||||||
# Lint-only image, built by script/lint. golangci-lint is never installed on
|
|
||||||
# the host: the repo is COPYed into the pinned image and linted as a build
|
|
||||||
# step, so a successful build IS a clean lint. This works even when the docker
|
|
||||||
# daemon is remote and bind mounts are impossible.
|
|
||||||
#
|
|
||||||
# script/lint passes --no-cache-filter=lint. Without it an unchanged tree
|
|
||||||
# replays the lint stage from cache and the build succeeds in under a second
|
|
||||||
# having run no linter at all. Do not drop that flag.
|
|
||||||
#
|
|
||||||
# The lint steps run with --network=none. `golangci-lint config verify` is
|
|
||||||
# documented as fetching its JSON schema over HTTPS, which would make linting
|
|
||||||
# depend on an unpinned remote artifact; this pinned image resolves the schema
|
|
||||||
# without any network, and --network=none enforces that rather than trusting
|
|
||||||
# it. It also proves no linter reaches out at analysis time. If a future image
|
|
||||||
# bump makes either step need the network, this build fails loudly instead of
|
|
||||||
# quietly acquiring an unpinned dependency.
|
|
||||||
|
|
||||||
# golangci/golangci-lint:v2.12.2 (Debian-based), 2026-08-07
|
|
||||||
# Using Debian-based image because mattn/go-sqlite3 (CGO) does not
|
|
||||||
# compile on Alpine musl (off64_t is a glibc type).
|
|
||||||
FROM golangci/golangci-lint:v2.12.2@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS deps
|
|
||||||
|
|
||||||
WORKDIR /src
|
|
||||||
|
|
||||||
# Copy go mod files first for better layer caching. This stage is cacheable;
|
|
||||||
# only the lint stage below is forced to re-execute.
|
|
||||||
COPY go.mod go.sum ./
|
|
||||||
RUN go mod download
|
|
||||||
|
|
||||||
FROM deps AS lint
|
|
||||||
|
|
||||||
COPY . .
|
|
||||||
|
|
||||||
# static/static.go embeds the Alpine.js file this extracts from 3p/; without
|
|
||||||
# it the static package does not compile and cannot be linted.
|
|
||||||
RUN script/assets
|
|
||||||
|
|
||||||
# `run` silently ignores config keys it does not recognize, so a typo would
|
|
||||||
# disable a setting without a word. `config verify` is what catches that.
|
|
||||||
RUN --network=none golangci-lint config verify --config .golangci.yml
|
|
||||||
# --build-tags browser also lints the browser test, which is built only with
|
|
||||||
# that tag (make test-browser).
|
|
||||||
RUN --network=none golangci-lint run --config .golangci.yml --build-tags browser ./...
|
|
||||||
@@ -16,10 +16,16 @@ VERSION ?= $(shell script/version)
|
|||||||
# exists to correct.
|
# exists to correct.
|
||||||
override VERSION := $(or $(strip $(VERSION)),$(shell script/version))
|
override VERSION := $(or $(strip $(VERSION)),$(shell script/version))
|
||||||
|
|
||||||
# Extra linker flags for the build target. The static relink in the
|
# Extra linker flags and build tags for the build target. The static
|
||||||
# Dockerfile adds -extldflags here rather than passing its own -ldflags,
|
# relink in the Dockerfile passes -extldflags and its build tags through
|
||||||
# so composing flags cannot drop the version stamp.
|
# these rather than a -ldflags of its own, so composing flags cannot drop
|
||||||
|
# the version stamp.
|
||||||
|
#
|
||||||
|
# The build target itself always passes -trimpath and -s -w, as the Go
|
||||||
|
# Dockerfile in REPO_POLICIES.md does: no build paths, symbol table or
|
||||||
|
# debug information in the binary.
|
||||||
GO_LDFLAGS ?=
|
GO_LDFLAGS ?=
|
||||||
|
GO_TAGS ?=
|
||||||
|
|
||||||
bootstrap:
|
bootstrap:
|
||||||
@script/bootstrap
|
@script/bootstrap
|
||||||
@@ -49,7 +55,7 @@ check:
|
|||||||
@script/check
|
@script/check
|
||||||
|
|
||||||
build: assets
|
build: assets
|
||||||
go build -ldflags '$(strip -X main.version=$(VERSION) $(GO_LDFLAGS))' -o bin/webhooker ./cmd/webhooker
|
go build -trimpath -tags '$(GO_TAGS)' -ldflags '$(strip -s -w -X main.version=$(VERSION) $(GO_LDFLAGS))' -o bin/webhooker ./cmd/webhooker
|
||||||
|
|
||||||
run: build
|
run: build
|
||||||
./bin/webhooker
|
./bin/webhooker
|
||||||
|
|||||||
@@ -17,14 +17,14 @@ before deploying one.
|
|||||||
### Prerequisites
|
### Prerequisites
|
||||||
|
|
||||||
- Go 1.26.1+ (the version in `go.mod`)
|
- Go 1.26.1+ (the version in `go.mod`)
|
||||||
- Docker (for `make lint`, `make fmt` and `make css`, and so for `make check`,
|
- Docker (for `make test`, `make lint`, `make fmt` and `make css`, and so for
|
||||||
for the browser test in `make test-browser`, for the CI gate, and for
|
`make check`, for the browser test in `make test-browser`, for the CI gate,
|
||||||
containerized deployment)
|
and for containerized deployment)
|
||||||
|
|
||||||
golangci-lint is not a prerequisite and must not be installed on the host:
|
golangci-lint is not a prerequisite and must not be installed on the host:
|
||||||
`script/bootstrap` does not install it, and `make lint` runs the digest-pinned
|
`script/bootstrap` does not install it, and `make lint` runs the digest-pinned
|
||||||
linter image via `Dockerfile.lint`. The same holds for tailwindcss (see
|
linter image in the Dockerfile's `lint` phase. The same holds for tailwindcss
|
||||||
[Stylesheet](#stylesheet)). ESLint, prettier, node and yarn are not
|
(see [Stylesheet](#stylesheet)). ESLint, prettier, node and yarn are not
|
||||||
prerequisites either, and `make lint` and `make fmt` never use a host copy of
|
prerequisites either, and `make lint` and `make fmt` never use a host copy of
|
||||||
them (see [Linting](#linting)).
|
them (see [Linting](#linting)).
|
||||||
|
|
||||||
@@ -43,8 +43,9 @@ make check
|
|||||||
|
|
||||||
# Run the server from the clone. DATA_DIR defaults to
|
# Run the server from the clone. DATA_DIR defaults to
|
||||||
# /var/lib/webhooker in every environment, so set it (in .env or the
|
# /var/lib/webhooker in every environment, so set it (in .env or the
|
||||||
# shell) to a writable directory.
|
# shell) to a writable directory outside the clone: the databases hold
|
||||||
DATA_DIR=./data make dev
|
# the session key.
|
||||||
|
DATA_DIR=../webhooker-data make dev
|
||||||
|
|
||||||
# Build Docker image
|
# Build Docker image
|
||||||
make docker
|
make docker
|
||||||
@@ -55,11 +56,11 @@ make docker
|
|||||||
```bash
|
```bash
|
||||||
make bootstrap # Install all dependencies (idempotent)
|
make bootstrap # Install all dependencies (idempotent)
|
||||||
make setup # Bootstrap + install git pre-commit hook
|
make setup # Bootstrap + install git pre-commit hook
|
||||||
make assets # Extract Alpine.js from 3p/ (test, check, build, dev run it)
|
make assets # Extract Alpine.js from 3p/ (build and dev run it)
|
||||||
make fmt # Format Go (gofmt + goimports) and Markdown (prettier, in Docker)
|
make fmt # Format Go (gofmt + goimports) and Markdown (prettier, in Docker)
|
||||||
make fmt-check # Fail if gofmt or prettier would change anything (writes nothing)
|
make fmt-check # Fail if gofmt or prettier would change anything (writes nothing)
|
||||||
make lint # Run golangci-lint and ESLint in Docker
|
make lint # Run golangci-lint and ESLint in Docker
|
||||||
make test # Run tests with race detection
|
make test # Run tests with race detection, in Docker
|
||||||
make test-browser # Run the browser test in Docker (Dockerfile.browser)
|
make test-browser # Run the browser test in Docker (Dockerfile.browser)
|
||||||
make check # test + lint + fmt-check + css-check (CI gate)
|
make check # test + lint + fmt-check + css-check (CI gate)
|
||||||
make build # Build binary to bin/webhooker (version-stamped)
|
make build # Build binary to bin/webhooker (version-stamped)
|
||||||
@@ -140,6 +141,7 @@ security headers are always applied.
|
|||||||
| `SENTRY_DSN` | Sentry error reporting DSN. Unset leaves error reporting off; a value the Sentry SDK cannot parse fails startup rather than serving with reporting silently off | `""` |
|
| `SENTRY_DSN` | Sentry error reporting DSN. Unset leaves error reporting off; a value the Sentry SDK cannot parse fails startup rather than serving with reporting silently off | `""` |
|
||||||
| `RETENTION_SWEEP_INTERVAL` | How often the retention reaper and archive sweeper run (Go duration, must be positive). A value that does not parse, or is zero or negative, fails startup | `1h` |
|
| `RETENTION_SWEEP_INTERVAL` | How often the retention reaper and archive sweeper run (Go duration, must be positive). A value that does not parse, or is zero or negative, fails startup | `1h` |
|
||||||
| `SESSION_IDLE_TIMEOUT` | Idle session timeout (Go duration) | `24h` |
|
| `SESSION_IDLE_TIMEOUT` | Idle session timeout (Go duration) | `24h` |
|
||||||
|
| `HSTS_MAX_AGE` | `max-age` of the `Strict-Transport-Security` header, which also carries `includeSubDomains` and `preload` (Go duration, whole seconds). `0` sends no header. Read [HSTS](#hsts) before changing it | `87600h` (ten years) |
|
||||||
| `RECEIVER_RATE_LIMIT` | Receiver requests/minute per IP per entrypoint (10x that per IP across the route) | `120` |
|
| `RECEIVER_RATE_LIMIT` | Receiver requests/minute per IP per entrypoint (10x that per IP across the route) | `120` |
|
||||||
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted. A set value replaces the default. If any client can reach webhooker, or the proxy in front of it, from an RFC 1918 source address, set it to the proxy's address alone. See [Trusted proxies](#trusted-proxies) | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` (RFC 1918) |
|
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted. A set value replaces the default. If any client can reach webhooker, or the proxy in front of it, from an RFC 1918 source address, set it to the proxy's address alone. See [Trusted proxies](#trusted-proxies) | `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16` (RFC 1918) |
|
||||||
| `ALLOWED_EGRESS_CIDRS` | CIDRs that delivery targets may reach despite the SSRF blocklist. Read [Allowing egress to your own network](#allowing-egress-to-your-own-network) before setting it | `""` (none) |
|
| `ALLOWED_EGRESS_CIDRS` | CIDRs that delivery targets may reach despite the SSRF blocklist. Read [Allowing egress to your own network](#allowing-egress-to-your-own-network) before setting it | `""` (none) |
|
||||||
@@ -463,10 +465,7 @@ idle timestamp is rewritten at most once per tenth of the idle window rather
|
|||||||
than on every request, which means a session may expire up to 10% early relative
|
than on every request, which means a session may expire up to 10% early relative
|
||||||
to the user's true last request, but never late.
|
to the user's true last request, but never late.
|
||||||
|
|
||||||
Both clocks are anchored by timestamps stored in the session cookie. Sessions
|
Both clocks are anchored by timestamps stored in the session cookie.
|
||||||
issued before this feature existed carry neither, so they are treated as
|
|
||||||
expired: upgrading to a build that has it logs every existing session out once,
|
|
||||||
and those users sign in again.
|
|
||||||
|
|
||||||
#### Invalid values abort startup
|
#### Invalid values abort startup
|
||||||
|
|
||||||
@@ -482,9 +481,10 @@ crash the reaper after startup), every entry in `TRUSTED_PROXIES` and
|
|||||||
`ALLOWED_EGRESS_CIDRS` must be a CIDR block or a bare IP address, and
|
`ALLOWED_EGRESS_CIDRS` must be a CIDR block or a bare IP address, and
|
||||||
`BIND_ADDRESS` must be an IP address literal — `localhost`, `127.0.0.1:8080` and
|
`BIND_ADDRESS` must be an IP address literal — `localhost`, `127.0.0.1:8080` and
|
||||||
`10.0.0.0/8` are each rejected rather than resolved, split, or narrowed to
|
`10.0.0.0/8` are each rejected rather than resolved, split, or narrowed to
|
||||||
something they do not say — and `SENTRY_DSN` must parse as a Sentry DSN.
|
something they do not say — `SENTRY_DSN` must parse as a Sentry DSN, and
|
||||||
`SESSION_IDLE_TIMEOUT` is the exception: a non-positive value there means idle
|
`HSTS_MAX_AGE` must be a whole number of seconds, zero or more (`1.5s` and `-1h`
|
||||||
expiry is disabled, not invalid.
|
are rejected). `SESSION_IDLE_TIMEOUT` is the exception: a non-positive value
|
||||||
|
there means idle expiry is disabled, not invalid.
|
||||||
|
|
||||||
`SENTRY_DSN` is checked with the Sentry SDK's own parser, the same call the SDK
|
`SENTRY_DSN` is checked with the Sentry SDK's own parser, the same call the SDK
|
||||||
makes on the DSN it is later handed, so what configuration accepts is exactly
|
makes on the DSN it is later handed, so what configuration accepts is exactly
|
||||||
@@ -699,11 +699,11 @@ app as that user.
|
|||||||
`webhooker.db` holds target configuration in plaintext — bearer tokens, API
|
`webhooker.db` holds target configuration in plaintext — bearer tokens, API
|
||||||
keys, Slack webhook URLs — along with the session encryption key, so webhooker
|
keys, Slack webhook URLs — along with the session encryption key, so webhooker
|
||||||
creates every SQLite file it owns `0600`: each database and both of its `-wal`
|
creates every SQLite file it owns `0600`: each database and both of its `-wal`
|
||||||
and `-shm` sidecars, across all three tiers. Files an earlier build left `0644`
|
and `-shm` sidecars, across all three tiers. Any of them found at another mode
|
||||||
are tightened when they are opened. The directory's `0750` is defence in depth —
|
is set to `0600` when the database is opened. The directory's `0750` is defence
|
||||||
it stops other local users listing the directory and learning your webhook UUIDs
|
in depth — it stops other local users listing the directory and learning your
|
||||||
from the `events-{uuid}.db` filenames — not the barrier protecting the
|
webhook UUIDs from the `events-{uuid}.db` filenames — not the barrier protecting
|
||||||
credentials.
|
the credentials.
|
||||||
|
|
||||||
### Running under upaas
|
### Running under upaas
|
||||||
|
|
||||||
@@ -881,19 +881,41 @@ If nginx runs on another host, `BIND_ADDRESS` becomes the address it connects
|
|||||||
to, `TRUSTED_PROXIES` becomes nginx's address, and the port must be firewalled
|
to, `TRUSTED_PROXIES` becomes nginx's address, and the port must be firewalled
|
||||||
to that address — the traffic between them is cleartext.
|
to that address — the traffic between them is cleartext.
|
||||||
|
|
||||||
### HSTS is always sent, and is not configurable
|
### HSTS
|
||||||
|
|
||||||
Every response carries
|
Every response carries a `Strict-Transport-Security` header unless
|
||||||
`Strict-Transport-Security: max-age=63072000; includeSubDomains; preload`. Two
|
`HSTS_MAX_AGE` is `0`. With the default `HSTS_MAX_AGE` of `87600h` (ten years)
|
||||||
years, every subdomain, and a `preload` token. There is no setting that changes
|
it is:
|
||||||
or suppresses it.
|
|
||||||
|
|
||||||
This is worth knowing before the first request reaches a browser: a client that
|
```
|
||||||
sees it once will refuse plaintext HTTP to that hostname — and to every
|
Strict-Transport-Security: max-age=315360000; includeSubDomains; preload
|
||||||
subdomain of it — for two years, whatever else is served there. Terminate TLS on
|
```
|
||||||
a hostname you are prepared to keep on HTTPS.
|
|
||||||
|
|
||||||
## Backup, Restore, and Upgrades
|
`HSTS_MAX_AGE` sets `max-age`, in seconds. `includeSubDomains` and `preload` are
|
||||||
|
always part of the header. A browser that receives the header over HTTPS refuses
|
||||||
|
plaintext HTTP to that hostname until `max-age` runs out, whatever else is
|
||||||
|
served there, so terminate TLS on a hostname you are prepared to keep on HTTPS.
|
||||||
|
Browsers ignore the header when it arrives over plain HTTP.
|
||||||
|
|
||||||
|
`includeSubDomains` extends this to every subdomain of the hostname webhooker is
|
||||||
|
served at. Served at `hooks.example.com`, it covers every name under
|
||||||
|
`hooks.example.com`; served at `example.com`, every name under `example.com`.
|
||||||
|
|
||||||
|
`preload` invites browsers to put the domain on their built-in HSTS preload
|
||||||
|
lists. The token alone puts nothing there. A domain is listed after someone
|
||||||
|
submits it at hstspreload.org, which accepts only a whole registered domain such
|
||||||
|
as `example.com`, not `hooks.example.com`. A listed domain is HTTPS-only in
|
||||||
|
every browser that ships the list, even for users who have never visited it, and
|
||||||
|
removal takes months to reach those browsers, so treat a listing as permanent.
|
||||||
|
|
||||||
|
Setting `HSTS_MAX_AGE=0` stops the header being sent, but a browser that already
|
||||||
|
received it over HTTPS does not forget it: it keeps refusing plaintext HTTP
|
||||||
|
until the `max-age` it saw runs out. Each header a browser receives over HTTPS
|
||||||
|
replaces the `max-age` it stored, so to end the policy sooner, set a short
|
||||||
|
`HSTS_MAX_AGE` such as `1s` and keep it until returning browsers have picked it
|
||||||
|
up, or have a reverse proxy send `max-age=0`.
|
||||||
|
|
||||||
|
## Backup and Restore
|
||||||
|
|
||||||
### What to back up
|
### What to back up
|
||||||
|
|
||||||
@@ -940,6 +962,13 @@ with an error naming it: for `webhooker.db` the server and `webhooker resetpw`
|
|||||||
stop, and an event or archive database fails as a damaged one does (see
|
stop, and an event or archive database fails as a damaged one does (see
|
||||||
[Database Architecture](#database-architecture)).
|
[Database Architecture](#database-architecture)).
|
||||||
|
|
||||||
|
**`DATA_DIR` must be on a local filesystem.** Under WAL, SQLite maps each
|
||||||
|
database's `-shm` file into memory, which network filesystems (NFS, SMB/CIFS) do
|
||||||
|
not support reliably, so they are not supported. webhooker does not check which
|
||||||
|
filesystem `DATA_DIR` is on. An error reading `disk I/O error (522)`
|
||||||
|
(`SQLITE_IOERR_SHORT_READ`) points at the storage under `DATA_DIR`, not at
|
||||||
|
locking; moving `DATA_DIR` to a plain local disk is the first thing to try.
|
||||||
|
|
||||||
Configuration is **not** in `DATA_DIR` — it comes from the environment and from
|
Configuration is **not** in `DATA_DIR` — it comes from the environment and from
|
||||||
a `.env` file read out of the process working directory. Back that up with your
|
a `.env` file read out of the process working directory. Back that up with your
|
||||||
deployment config, separately.
|
deployment config, separately.
|
||||||
@@ -1039,63 +1068,20 @@ beside it, or wait until there are none.
|
|||||||
|
|
||||||
4. Start the service. The container gives the directory and the restored files
|
4. Start the service. The container gives the directory and the restored files
|
||||||
to the `webhooker` user before the app starts, whoever restored them (see
|
to the `webhooker` user before the app starts, whoever restored them (see
|
||||||
[Running with Docker](#running-with-docker)). `AutoMigrate` runs against each
|
[Running with Docker](#running-with-docker)).
|
||||||
restored database as it is opened.
|
|
||||||
|
|
||||||
### Upgrades
|
|
||||||
|
|
||||||
**Back up before every upgrade.** Every start runs GORM `AutoMigrate`
|
|
||||||
unconditionally against whatever files it finds:
|
|
||||||
|
|
||||||
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
|
|
||||||
`Entrypoint`, `Target`
|
|
||||||
- each event database when it is lazily opened — `Event`, `Delivery`,
|
|
||||||
`DeliveryResult`, `EventTotals`, `TargetTotals`, `EntrypointTotals`
|
|
||||||
- each archive database on every open and reopen
|
|
||||||
|
|
||||||
There is no schema version table, no migration ledger, and no down migrations.
|
|
||||||
Nothing in the files records which version wrote them, and no code path undoes a
|
|
||||||
migration.
|
|
||||||
|
|
||||||
Upgrade procedure:
|
|
||||||
|
|
||||||
1. Stop the service.
|
|
||||||
2. Back up `DATA_DIR` using one of the procedures above.
|
|
||||||
3. Pull the new image and start it.
|
|
||||||
4. Confirm `database migrations completed` in the logs before putting traffic
|
|
||||||
back on it.
|
|
||||||
5. Confirm the new build is the one running:
|
|
||||||
`curl -s http://host:8080/.well-known/healthcheck` reports the version it was
|
|
||||||
stamped with (see [Version stamping](#version-stamping)).
|
|
||||||
|
|
||||||
**Upgrading past the introduction of `BIND_ADDRESS`:** earlier versions always
|
|
||||||
bound every interface. **Container deployments are unaffected** — the image
|
|
||||||
ships `ENV BIND_ADDRESS=0.0.0.0`, so a `docker run` or Compose service that
|
|
||||||
worked before still works with nothing changed.
|
|
||||||
|
|
||||||
A **bare binary** is the case that changes: the listener now binds `127.0.0.1`
|
|
||||||
unless `BIND_ADDRESS` says otherwise, so a deployment that relied on reaching it
|
|
||||||
from another host becomes unreachable until it sets the address the proxy
|
|
||||||
connects to. Check the `bindAddress` field of the startup log to see what a
|
|
||||||
running process bound. See [Bind address](#bind-address).
|
|
||||||
|
|
||||||
**Downgrade is unsupported.** Once a newer binary has migrated the files there
|
|
||||||
is no way to move them back. `AutoMigrate` is additive — it adds tables, columns
|
|
||||||
and indexes and never drops or rewrites them — so an older binary will generally
|
|
||||||
open migrated files and appear to work while writing against a schema it does
|
|
||||||
not know about. The failure mode is silent divergence, not a startup error. The
|
|
||||||
only supported way back to an older version is restoring the pre-upgrade backup,
|
|
||||||
which discards everything received since that backup was taken.
|
|
||||||
|
|
||||||
### Version stamping
|
### Version stamping
|
||||||
|
|
||||||
The binary reports its version at `/.well-known/healthcheck` (the `version`
|
The binary reports its version at `/.well-known/healthcheck` (the `version`
|
||||||
field), in the UI footer, and in the startup log line (`msg=starting`,
|
field), in the UI footer, and in the startup log line (`msg=starting`,
|
||||||
`version=...`). It is also the Sentry release name, as `webhooker-{version}`.
|
`version=...`). It is also the Sentry release name, as `webhooker-{version}`.
|
||||||
|
`curl -s http://host:8080/.well-known/healthcheck` shows which build is running.
|
||||||
The value is stamped in at build time by the linker; it is not read from a file
|
The value is stamped in at build time by the linker; it is not read from a file
|
||||||
at runtime, so it identifies the build itself.
|
at runtime, so it identifies the build itself.
|
||||||
|
|
||||||
`script/version` produces the value and both build paths use it:
|
`script/version` produces the value for `make build`, and the image build runs
|
||||||
|
`make build` too; `script/docker` and `script/cibuild` run the same
|
||||||
|
`git describe --tags --always --dirty` on the host. All of them report:
|
||||||
|
|
||||||
| Build | What it reports |
|
| Build | What it reports |
|
||||||
| ----------------------- | --------------------------------------------------- |
|
| ----------------------- | --------------------------------------------------- |
|
||||||
@@ -1110,18 +1096,19 @@ carries, so any `docker build .` of a clone, with no build arguments, stamps the
|
|||||||
commit it was built from; a shallow clone of one branch has no tags and stamps
|
commit it was built from; a shallow clone of one branch has no tags and stamps
|
||||||
the short SHA. `.dockerignore` must therefore leave out neither `.git` nor any
|
the short SHA. `.dockerignore` must therefore leave out neither `.git` nor any
|
||||||
tracked file, which git in the build would see as deleted, marking the version
|
tracked file, which git in the build would see as deleted, marking the version
|
||||||
`-dirty`. It does leave `.git/config`, which can hold a remote URL carrying a
|
`-dirty`. It does leave out every git `config` (`**/.git/config`,
|
||||||
credential and which `git describe` does not need, out of a directory context. A
|
`**/.git/modules/**/config`), which can hold a remote URL carrying a credential
|
||||||
context sent as a tar is not filtered by `.dockerignore`, so it carries
|
and which `git describe` does not need, from a directory context. A context sent
|
||||||
`.git/config` unless its sender leaves it out; for upaas, that is
|
as a tar is not filtered by `.dockerignore`, so it carries `.git/config` unless
|
||||||
|
its sender leaves it out; for upaas, that is
|
||||||
https://git.eeqj.de/sneak/upaas/issues/274. git in the build reads the checkout
|
https://git.eeqj.de/sneak/upaas/issues/274. git in the build reads the checkout
|
||||||
whoever owns its files, since a context sent as a tar archive keeps the sender's
|
whoever owns its files, since a context sent as a tar archive keeps the sender's
|
||||||
owners and git otherwise refuses a checkout owned by another user. A `VERSION`
|
owners and git otherwise refuses a checkout owned by another user. A `VERSION`
|
||||||
build arg (`--build-arg VERSION=...`) takes precedence; `script/docker` (and so
|
build arg (`--build-arg VERSION=...`) takes precedence; `script/docker` (and so
|
||||||
`make docker`) passes the one `script/version` resolves on the host. The image
|
`make docker`) and `script/cibuild` pass the one they resolve on the host, or
|
||||||
build fails if its context carries `.git` and the version still comes out
|
`unknown` where git gives none. The image build fails if its context carries
|
||||||
`unknown`, which means git is missing from the build or could not read the
|
`.git` and the version still comes out empty, `dev` or `unknown`, which means
|
||||||
checkout.
|
git is missing from the build or could not read the checkout.
|
||||||
|
|
||||||
`unknown` is what a source tarball, or a `docker build` with no `.git` in its
|
`unknown` is what a source tarball, or a `docker build` with no `.git` in its
|
||||||
context and no `VERSION` build arg, reports. A build that reports `unknown` is a
|
context and no `VERSION` build arg, reports. A build that reports `unknown` is a
|
||||||
@@ -1135,7 +1122,9 @@ to a commit.
|
|||||||
|
|
||||||
Nothing that varies between two builds of the same commit is stamped — no
|
Nothing that varies between two builds of the same commit is stamped — no
|
||||||
timestamp, no hostname, no builder identity — so two builds of one commit still
|
timestamp, no hostname, no builder identity — so two builds of one commit still
|
||||||
produce a byte-identical binary.
|
produce a byte-identical binary. `make build` passes `-trimpath`, so the
|
||||||
|
directory it builds in is not recorded either, and `-s -w`, which leave out the
|
||||||
|
symbol table and debug information.
|
||||||
|
|
||||||
### Backups contain secrets
|
### Backups contain secrets
|
||||||
|
|
||||||
@@ -1145,26 +1134,6 @@ at rest and restrict who can read them.
|
|||||||
- `events-{uuid}.db` and `archive-….db` hold the **full payload body and
|
- `events-{uuid}.db` and `archive-….db` hold the **full payload body and
|
||||||
headers** of every event as received, including whatever the sending service
|
headers** of every event as received, including whatever the sending service
|
||||||
put in them — tokens, signatures, personal data.
|
put in them — tokens, signatures, personal data.
|
||||||
- Event databases written before
|
|
||||||
[issue #206](https://git.eeqj.de/sneak/webhooker/issues/206) was fixed **also
|
|
||||||
contain target credentials**: a GORM association upsert on the delivery and
|
|
||||||
retry write path copied `targets` rows, `config` included, into the
|
|
||||||
per-webhook database. For a Slack target the `webhookUrl` _is_ the bearer
|
|
||||||
credential, and an `http` target's URL can embed userinfo. This version never
|
|
||||||
writes those rows; the first time it opens such a file it deletes them and
|
|
||||||
vacuums the file, which removes the credential bytes rather than only
|
|
||||||
unlinking the rows. Deleting alone would not: the bytes stay readable in the
|
|
||||||
file's free pages until it is rewritten. The sweep is recorded in the file's
|
|
||||||
`user_version` only once the vacuum returns, so a sweep that fails or is
|
|
||||||
interrupted fails the open and is retried on the next one, and a file this
|
|
||||||
version has opened without error holds no leaked rows and no recoverable bytes
|
|
||||||
from them. On upgrade this rewrites each existing `events-{uuid}.db` once, on
|
|
||||||
its first open. Two cases still hand over live delivery destinations: a backup
|
|
||||||
taken from an older build, and a backup of a file this version has not yet
|
|
||||||
opened successfully. Copies already made stay affected — the sweep only
|
|
||||||
rewrites the file it opens, and freed blocks may persist in filesystem
|
|
||||||
snapshots and on the underlying storage. Rotate any target credential that was
|
|
||||||
in a backup you cannot account for.
|
|
||||||
- `webhooker.db` stores target config **unencrypted**, tracked at
|
- `webhooker.db` stores target config **unencrypted**, tracked at
|
||||||
[issue #212](https://git.eeqj.de/sneak/webhooker/issues/212), next to the
|
[issue #212](https://git.eeqj.de/sneak/webhooker/issues/212), next to the
|
||||||
session encryption key and the Argon2id password hashes.
|
session encryption key and the Argon2id password hashes.
|
||||||
@@ -1213,11 +1182,14 @@ commands with no script behind them, though `build`, `run` and `dev` first run
|
|||||||
`script/assets`, and `build` and `version` both take their value from
|
`script/assets`, and `build` and `version` both take their value from
|
||||||
`script/version`.
|
`script/version`.
|
||||||
|
|
||||||
`script/test`, `make build` and `make dev` each run `script/assets` first, which
|
`make build` and `make dev` each run `script/assets` first, which writes the
|
||||||
writes the ignored `static/js/alpine.min.js` (see
|
uncommitted `static/js/alpine.min.js` (see
|
||||||
[Third-party browser assets](#third-party-browser-assets)), so `make test`,
|
[Third-party browser assets](#third-party-browser-assets)), so they work on a
|
||||||
`make check` and the pre-commit hook work on a fresh clone without a separate
|
fresh clone without a separate step. The Docker stages that compile the code run
|
||||||
step.
|
it themselves.
|
||||||
|
|
||||||
|
Every `docker build` in `script/` passes `--no-cache`: a check served from the
|
||||||
|
build cache is a check that did not run.
|
||||||
|
|
||||||
We provide:
|
We provide:
|
||||||
|
|
||||||
@@ -1227,10 +1199,12 @@ We provide:
|
|||||||
- `script/projectname` — output the project name ("webhooker")
|
- `script/projectname` — output the project name ("webhooker")
|
||||||
- `script/assets` — extract Alpine.js from its tarball in `3p/` (see
|
- `script/assets` — extract Alpine.js from its tarball in `3p/` (see
|
||||||
[Third-party browser assets](#third-party-browser-assets))
|
[Third-party browser assets](#third-party-browser-assets))
|
||||||
- `script/test` — run the test suite
|
- `script/test` — run the test suite: builds the Dockerfile's `test` phase,
|
||||||
|
tagged `webhooker-test`
|
||||||
- `script/test-browser` — run the browser test in Docker (see
|
- `script/test-browser` — run the browser test in Docker (see
|
||||||
[Third-party browser assets](#third-party-browser-assets))
|
[Third-party browser assets](#third-party-browser-assets))
|
||||||
- `script/lint` — run golangci-lint and ESLint in Docker (see Linting below)
|
- `script/lint` — run the `gofmt` check, golangci-lint and ESLint: builds the
|
||||||
|
Dockerfile's `lint` phase, tagged `webhooker-lint` (see Linting below)
|
||||||
- `script/fmt` — format the Go code and, in Docker, the Markdown (writes)
|
- `script/fmt` — format the Go code and, in Docker, the Markdown (writes)
|
||||||
- `script/fmt-check` — check formatting (read-only)
|
- `script/fmt-check` — check formatting (read-only)
|
||||||
- `script/css` — regenerate `static/css/tailwind.css` in Docker (writes; see
|
- `script/css` — regenerate `static/css/tailwind.css` in Docker (writes; see
|
||||||
@@ -1241,11 +1215,11 @@ We provide:
|
|||||||
- `script/version` — output the version to stamp into the binary (see
|
- `script/version` — output the version to stamp into the binary (see
|
||||||
[Version stamping](#version-stamping))
|
[Version stamping](#version-stamping))
|
||||||
- `script/docker` — build the Docker image tagged via `script/projectname`,
|
- `script/docker` — build the Docker image tagged via `script/projectname`,
|
||||||
passing `script/version`'s output in as the `VERSION` build arg
|
passing the version `git describe` gives on the host in as the `VERSION` build
|
||||||
- `script/cibuild` — CI entrypoint: `docker build .` (the Dockerfile runs the
|
arg
|
||||||
checks, so a green build implies a green repo)
|
- `script/cibuild` — CI entrypoint: `script/bootstrap`, then `script/check`,
|
||||||
- `script/ci-mark-superseded` — CI helper: mark the commits whose run a newer
|
then the same image build as `script/docker`, whose gate phases run again (see
|
||||||
push cancelled (see [CI gate honesty](#ci-gate-honesty))
|
[CI gate honesty](#ci-gate-honesty))
|
||||||
- `script/precommit` — pre-commit checks (`go mod tidy` guard, then
|
- `script/precommit` — pre-commit checks (`go mod tidy` guard, then
|
||||||
`script/check`)
|
`script/check`)
|
||||||
- `script/install-precommit` — install the git pre-commit hook that runs
|
- `script/install-precommit` — install the git pre-commit hook that runs
|
||||||
@@ -1298,12 +1272,12 @@ apply. The directory is `3p/` rather than `vendor/` because Go treats a root
|
|||||||
|
|
||||||
`script/assets` (`make assets`) extracts the browser build,
|
`script/assets` (`make assets`) extracts the browser build,
|
||||||
`package/dist/cdn.min.js`, from the tarball to `static/js/alpine.min.js`, where
|
`package/dist/cdn.min.js`, from the tarball to `static/js/alpine.min.js`, where
|
||||||
`go:embed` picks it up. `script/test`, `make build` and `make dev` run it first,
|
`go:embed` picks it up. `make build` and `make dev` run it first, and so do the
|
||||||
and the Dockerfile builds through `make test` and `make build`, so nothing
|
Dockerfile's lint, test and build stages, so nothing downloads Alpine.js. The
|
||||||
downloads Alpine.js. The extracted file is not committed, and `.dockerignore`
|
extracted file is not committed, and `.dockerignore` keeps any host copy out of
|
||||||
keeps any host copy out of the build context. `static/static.go` names every
|
the build context. `static/static.go` names every file it embeds, so a build
|
||||||
file it embeds, so a build that skips the extraction, such as a bare `go build`,
|
that skips the extraction, such as a bare `go build`, fails with an error naming
|
||||||
fails with an error naming `js/alpine.min.js`.
|
`js/alpine.min.js`.
|
||||||
|
|
||||||
To move to a new version: download
|
To move to a new version: download
|
||||||
`https://registry.npmjs.org/@alpinejs/csp/-/csp-<version>.tgz`, check it against
|
`https://registry.npmjs.org/@alpinejs/csp/-/csp-<version>.tgz`, check it against
|
||||||
@@ -1560,8 +1534,7 @@ The cap is not arbitrary: the reaper computes its cutoff as a `time.Duration`,
|
|||||||
an int64 nanosecond count, and a longer period overflows it. An overflowed
|
an int64 nanosecond count, and a longer period overflows it. An overflowed
|
||||||
cutoff lands in the future, where it matches every row, so the sweep would
|
cutoff lands in the future, where it matches every row, so the sweep would
|
||||||
delete every event the webhook has instead of none. The reaper also clamps the
|
delete every event the webhook has instead of none. The reaper also clamps the
|
||||||
value it is given, so a row written by an older version cannot trigger that
|
value it reads to the cap, so no stored value can trigger that either.
|
||||||
either.
|
|
||||||
|
|
||||||
#### Entrypoint
|
#### Entrypoint
|
||||||
|
|
||||||
@@ -1877,8 +1850,9 @@ Deliveries still `pending` or `retrying` count in neither.
|
|||||||
|
|
||||||
#### Event-tier indexes
|
#### Event-tier indexes
|
||||||
|
|
||||||
These indexes on the per-webhook event databases are declared in the model tags,
|
These indexes on the per-webhook event databases are created by
|
||||||
so `AutoMigrate` creates them on a fresh database:
|
`internal/db/migrations/events/001_schema.sql`, and declared in the model tags
|
||||||
|
to match:
|
||||||
|
|
||||||
| Table | Columns | Serves |
|
| Table | Columns | Serves |
|
||||||
| ------------------ | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
| ------------------ | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
@@ -1942,8 +1916,9 @@ and application state:
|
|||||||
- **Targets** — delivery destination configurations
|
- **Targets** — delivery destination configurations
|
||||||
- **APIKeys** — programmatic access credentials
|
- **APIKeys** — programmatic access credentials
|
||||||
|
|
||||||
On first startup the main database is auto-migrated, a session encryption key is
|
On first startup the main database is created from its SQL files (see
|
||||||
generated and stored, and an `admin` user is created.
|
[Schema](#schema)), a session encryption key is generated and stored, and an
|
||||||
|
`admin` user is created.
|
||||||
|
|
||||||
**Per-Webhook Event Databases** (`{DATA_DIR}/events-{webhook_uuid}.db`) — each
|
**Per-Webhook Event Databases** (`{DATA_DIR}/events-{webhook_uuid}.db`) — each
|
||||||
webhook gets its own dedicated SQLite file containing:
|
webhook gets its own dedicated SQLite file containing:
|
||||||
@@ -1955,9 +1930,10 @@ webhook gets its own dedicated SQLite file containing:
|
|||||||
the above, the deliveries per target, and each entrypoint's last event, kept
|
the above, the deliveries per target, and each entrypoint's last event, kept
|
||||||
through retention
|
through retention
|
||||||
|
|
||||||
Per-webhook databases are created automatically when a webhook is created. They
|
Per-webhook databases are created automatically when a webhook is created, from
|
||||||
are managed by the `WebhookDBManager` component, which handles connection
|
the SQL files in `internal/db/migrations/events/`. They hold no copy of the main
|
||||||
pooling, lazy opening, migrations, and cleanup.
|
database's tables. They are managed by the `WebhookDBManager` component, which
|
||||||
|
handles connection pooling, lazy opening, migrations, and cleanup.
|
||||||
|
|
||||||
A per-webhook database that is missing or zero-length later means its webhook's
|
A per-webhook database that is missing or zero-length later means its webhook's
|
||||||
events and pending deliveries are gone. The next time it is opened, an empty one
|
events and pending deliveries are gone. The next time it is opened, an empty one
|
||||||
@@ -2041,14 +2017,15 @@ the service again.
|
|||||||
After each write the archive handle is closed and reopened, debounced to at most
|
After each write the archive handle is closed and reopened, debounced to at most
|
||||||
once per second, so an operator can move the archive file away for offline
|
once per second, so an operator can move the archive file away for offline
|
||||||
archiving without stopping the service; a moved or removed archive file is
|
archiving without stopping the service; a moved or removed archive file is
|
||||||
recreated automatically on the next write. A zero-length archive file is written
|
recreated automatically on the next write, from the SQL files in
|
||||||
to as a new archive: SQLite opens it as an empty database, so it holds nothing
|
`internal/db/migrations/archive/`. A zero-length archive file is written to as a
|
||||||
to lose. An optional `expiry` in the target's config JSON (e.g.
|
new archive: SQLite opens it as an empty database, so it holds nothing to lose.
|
||||||
`{"expiry":"720h"}`) is validated when the target is created — the default
|
An optional `expiry` in the target's config JSON (e.g. `{"expiry":"720h"}`) is
|
||||||
(unset or the literal `never`) keeps rows forever — and rows older than the
|
validated when the target is created — the default (unset or the literal
|
||||||
expiry are pruned each time the archive is (re)opened. An archive write failure
|
`never`) keeps rows forever — and rows older than the expiry are pruned each
|
||||||
is never silent success: the delivery records a failed attempt with the error
|
time the archive is (re)opened. An archive write failure is never silent
|
||||||
and is marked failed.
|
success: the delivery records a failed attempt with the error and is marked
|
||||||
|
failed.
|
||||||
|
|
||||||
Because reopens only happen on writes, an archive whose target has stopped
|
Because reopens only happen on writes, an archive whose target has stopped
|
||||||
receiving events would never be pruned. A background **archive sweeper** closes
|
receiving events would never be pruned. A background **archive sweeper** closes
|
||||||
@@ -2137,6 +2114,21 @@ The database uses the
|
|||||||
though CGO is required at build time due to the transitive `mattn/go-sqlite3`
|
though CGO is required at build time due to the transitive `mattn/go-sqlite3`
|
||||||
dependency from `gorm.io/driver/sqlite`.
|
dependency from `gorm.io/driver/sqlite`.
|
||||||
|
|
||||||
|
#### Schema
|
||||||
|
|
||||||
|
Each kind of database file has its schema in SQL files built into the binary,
|
||||||
|
one directory per kind under `internal/db/migrations/`. In each,
|
||||||
|
`000_migration.sql` creates the `schema_migrations` table, which records the
|
||||||
|
number of every file the database has had, and `001_schema.sql` holds the whole
|
||||||
|
schema. Opening a database runs, in one transaction, every file in its directory
|
||||||
|
that `schema_migrations` does not list:
|
||||||
|
|
||||||
|
- `main/` for the main database, on connect — settings, users, API keys,
|
||||||
|
webhooks, entrypoints and targets
|
||||||
|
- `events/` for each event database, when it is lazily opened — events,
|
||||||
|
deliveries, delivery results and their running totals
|
||||||
|
- `archive/` for each archive database, on every open and reopen
|
||||||
|
|
||||||
### Request Flow
|
### Request Flow
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -2509,10 +2501,23 @@ Where the pattern is out of reach — the transaction dispatch, an event capture
|
|||||||
outside the router, or a request that matched no route — the fallback is never
|
outside the router, or a request that matched no route — the fallback is never
|
||||||
the concrete path. The path becomes the literal `/(redacted)`, so the URL reads
|
the concrete path. The path becomes the literal `/(redacted)`, so the URL reads
|
||||||
`http://host/(redacted)`; a URL the rewrite cannot parse into a scheme is
|
`http://host/(redacted)`; a URL the rewrite cannot parse into a scheme is
|
||||||
withheld whole. A transaction event additionally carries the SDK's own
|
withheld whole.
|
||||||
`METHOD /path` name, built from the concrete path as well; it is rewritten on
|
|
||||||
the same terms, to `POST /h/{uuid}` where the pattern is known and
|
A transaction event also carries a name, which the SDK builds as `METHOD /path`
|
||||||
`POST /(redacted)` where it is not.
|
from the concrete path and gives the transaction source `url`. Sentry groups
|
||||||
|
transactions by name, so withholding it would put every request in the service
|
||||||
|
into one group per method, such as `POST /(redacted)`. Inside each Sentry
|
||||||
|
handler, a second middleware renames the transaction once the request has been
|
||||||
|
routed, to the method and the route pattern (`POST /h/{uuid}`), and sets its
|
||||||
|
source to `route`. The hook keeps a name whose source is `route` and rewrites
|
||||||
|
any other to `METHOD /(redacted)`. A path that matches nothing at the top level
|
||||||
|
has no pattern, so its transaction keeps the SDK's name and the hook withholds
|
||||||
|
it. Inside a route group, a path that matches no route in the group, and a
|
||||||
|
request the group's own middleware turns away before routing (a CSRF refusal,
|
||||||
|
the redirect to `/pages/login`), are named after the group's pattern, such as
|
||||||
|
`GET /pages/*` or `POST /hook/{sourceID}/*`, and the hook keeps that name. The
|
||||||
|
URL on a transaction event still falls back to `http://host/(redacted)`, because
|
||||||
|
the hook has no request there.
|
||||||
|
|
||||||
The headers are an allowlist for the same reason the rules above are
|
The headers are an allowlist for the same reason the rules above are
|
||||||
unconditional: the SDK's own filter removes four names and passes everything
|
unconditional: the SDK's own filter removes four names and passes everything
|
||||||
@@ -2969,17 +2974,14 @@ webhooker/
|
|||||||
├── internal/
|
├── internal/
|
||||||
│ ├── banner/
|
│ ├── banner/
|
||||||
│ │ └── banner.go # Ruled block for the one credential shown in the clear
|
│ │ └── banner.go # Ruled block for the one credential shown in the clear
|
||||||
│ ├── ciscript/
|
|
||||||
│ │ └── doc.go # Tests for the CI shell scripts in script/; no runtime code
|
|
||||||
│ ├── resetpw/
|
│ ├── resetpw/
|
||||||
│ │ └── resetpw.go # `webhooker resetpw`: set an account's password, stopped deployments only
|
│ │ └── resetpw.go # `webhooker resetpw`: set an account's password, stopped deployments only
|
||||||
│ ├── config/
|
│ ├── config/
|
||||||
│ │ ├── config.go # Configuration loading from environment variables
|
│ │ ├── config.go # Configuration loading from environment variables
|
||||||
│ │ └── testing.go # ClearEnvForTest: an empty environment for one test
|
│ │ └── configtest/ # Test support: ClearEnv, an empty environment for one test
|
||||||
│ ├── database/
|
│ ├── database/
|
||||||
│ │ ├── base_model.go # BaseModel with UUID primary keys
|
│ │ ├── base_model.go # BaseModel with UUID primary keys
|
||||||
│ │ ├── database.go # GORM connection, migrations, admin seed
|
│ │ ├── database.go # GORM connection, migrations, admin seed
|
||||||
│ │ ├── models.go # AutoMigrate for config-tier models
|
|
||||||
│ │ ├── model_setting.go # Setting entity (key-value app config)
|
│ │ ├── model_setting.go # Setting entity (key-value app config)
|
||||||
│ │ ├── model_user.go # User entity
|
│ │ ├── model_user.go # User entity
|
||||||
│ │ ├── model_webhook.go # Webhook entity
|
│ │ ├── model_webhook.go # Webhook entity
|
||||||
@@ -2992,10 +2994,15 @@ webhooker/
|
|||||||
│ │ ├── model_apikey.go # APIKey entity
|
│ │ ├── model_apikey.go # APIKey entity
|
||||||
│ │ ├── password.go # Argon2id hashing and verification
|
│ │ ├── password.go # Argon2id hashing and verification
|
||||||
│ │ ├── retention.go # Retention reaper (per-webhook event expiry)
|
│ │ ├── retention.go # Retention reaper (per-webhook event expiry)
|
||||||
│ │ ├── testing.go # NewTestDatabase: wrapper for tests, no fx lifecycle
|
│ │ ├── webhook_db_manager.go # Per-webhook DB lifecycle manager
|
||||||
│ │ └── webhook_db_manager.go # Per-webhook DB lifecycle manager
|
│ │ └── databasetest/ # Test support: a WebhookDBManager for tests in other packages
|
||||||
│ ├── datadir/
|
│ ├── datadir/
|
||||||
│ │ └── lock.go # Exclusive advisory lock on DATA_DIR (one instance)
|
│ │ └── lock.go # Exclusive advisory lock on DATA_DIR (one instance)
|
||||||
|
│ ├── db/migrations/
|
||||||
|
│ │ ├── migrations.go # Applies a database file's SQL files when it is opened
|
||||||
|
│ │ ├── main/ # 000_migration.sql and 001_schema.sql for webhooker.db
|
||||||
|
│ │ ├── events/ # The same for each events-{uuid}.db
|
||||||
|
│ │ └── archive/ # The same for each archive file
|
||||||
│ ├── globals/
|
│ ├── globals/
|
||||||
│ │ └── globals.go # Build-time variables (appname, version, arch)
|
│ │ └── globals.go # Build-time variables (appname, version, arch)
|
||||||
│ ├── gormlog/
|
│ ├── gormlog/
|
||||||
@@ -3054,7 +3061,7 @@ webhooker/
|
|||||||
│ │ ├── csrf.go # CSRF protection middleware (gorilla/csrf)
|
│ │ ├── csrf.go # CSRF protection middleware (gorilla/csrf)
|
||||||
│ │ ├── ratelimit.go # Per-IP rate limiting middleware (go-chi/httprate)
|
│ │ ├── ratelimit.go # Per-IP rate limiting middleware (go-chi/httprate)
|
||||||
│ │ ├── loginguard.go # Login failure counters and the Argon2id verification semaphore
|
│ │ ├── loginguard.go # Login failure counters and the Argon2id verification semaphore
|
||||||
│ │ └── testing.go # NewForTest: Middleware without the fx lifecycle
|
│ │ └── middlewaretest/ # Test support: a Middleware for tests in other packages
|
||||||
│ ├── reqtls/
|
│ ├── reqtls/
|
||||||
│ │ └── reqtls.go # IsTLS: the one TLS predicate, r.TLS or X-Forwarded-Proto
|
│ │ └── reqtls.go # IsTLS: the one TLS predicate, r.TLS or X-Forwarded-Proto
|
||||||
│ ├── server/
|
│ ├── server/
|
||||||
@@ -3062,8 +3069,7 @@ webhooker/
|
|||||||
│ │ ├── http.go # HTTP server setup with timeouts
|
│ │ ├── http.go # HTTP server setup with timeouts
|
||||||
│ │ └── routes.go # All route definitions
|
│ │ └── routes.go # All route definitions
|
||||||
│ ├── session/
|
│ ├── session/
|
||||||
│ │ ├── session.go # Cookie-based session management
|
│ │ └── session.go # Cookie-based session management
|
||||||
│ │ └── testing.go # NewForTest: Session without the fx lifecycle
|
|
||||||
│ └── versionscript/
|
│ └── versionscript/
|
||||||
│ └── doc.go # Tests for script/version and the build files that use it
|
│ └── doc.go # Tests for script/version and the build files that use it
|
||||||
├── static/
|
├── static/
|
||||||
@@ -3075,8 +3081,7 @@ webhooker/
|
|||||||
│ └── js/alpine.min.js # Alpine.js CSP build, extracted from 3p/ by make assets, not committed
|
│ └── js/alpine.min.js # Alpine.js CSP build, extracted from 3p/ by make assets, not committed
|
||||||
├── templates/ # Go HTML templates (base, login, sources, etc.)
|
├── templates/ # Go HTML templates (base, login, sources, etc.)
|
||||||
├── script/ # Scripts to Rule Them All entrypoints
|
├── script/ # Scripts to Rule Them All entrypoints
|
||||||
├── Dockerfile # Stages: lint, stylesheet, JavaScript lint, Markdown, test+build, Alpine runtime
|
├── Dockerfile # Stages: stylesheet, JavaScript lint, Markdown, lint, test, build, Alpine runtime
|
||||||
├── Dockerfile.lint # Lint-only image built by script/lint
|
|
||||||
├── Dockerfile.browser # Browser test image built by script/test-browser
|
├── Dockerfile.browser # Browser test image built by script/test-browser
|
||||||
├── Makefile # 13 of 19 targets shim script/; 6 are inline
|
├── Makefile # 13 of 19 targets shim script/; 6 are inline
|
||||||
├── go.mod / go.sum
|
├── go.mod / go.sum
|
||||||
@@ -3084,6 +3089,7 @@ webhooker/
|
|||||||
├── .yarnrc.yml # yarn settings: install into node_modules/
|
├── .yarnrc.yml # yarn settings: install into node_modules/
|
||||||
├── eslint.config.mjs # ESLint configuration for static/js/
|
├── eslint.config.mjs # ESLint configuration for static/js/
|
||||||
├── .prettierrc # prettier settings for the Markdown
|
├── .prettierrc # prettier settings for the Markdown
|
||||||
|
├── .prettierignore # Files prettier skips
|
||||||
└── .golangci.yml # golangci-lint configuration
|
└── .golangci.yml # golangci-lint configuration
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -3124,9 +3130,9 @@ any database — it only writes to record results.
|
|||||||
Applied to all routes in this order:
|
Applied to all routes in this order:
|
||||||
|
|
||||||
1. **RequestID** — Generate unique request IDs (chi built-in)
|
1. **RequestID** — Generate unique request IDs (chi built-in)
|
||||||
2. **SecurityHeaders** — Production security headers on every response (HSTS,
|
2. **SecurityHeaders** — Production security headers on every response (HSTS
|
||||||
X-Content-Type-Options, X-Frame-Options, CSP, Referrer-Policy,
|
unless `HSTS_MAX_AGE` is `0`, X-Content-Type-Options, X-Frame-Options, CSP,
|
||||||
Permissions-Policy)
|
Referrer-Policy, Permissions-Policy)
|
||||||
3. **Logging** — Structured request logging (method, URL, status, latency,
|
3. **Logging** — Structured request logging (method, URL, status, latency,
|
||||||
remote IP, client IP, user agent, request ID)
|
remote IP, client IP, user agent, request ID)
|
||||||
4. **Metrics** — Prometheus HTTP metrics (if `METRICS_USERNAME` and
|
4. **Metrics** — Prometheus HTTP metrics (if `METRICS_USERNAME` and
|
||||||
@@ -3136,7 +3142,8 @@ Applied to all routes in this order:
|
|||||||
7. **Recoverer** — Panic recovery: one `ERROR` record through `internal/logger`
|
7. **Recoverer** — Panic recovery: one `ERROR` record through `internal/logger`
|
||||||
and a `500`
|
and a `500`
|
||||||
8. **Sentry** — Error reporting to Sentry (if `SENTRY_DSN` is set; configured
|
8. **Sentry** — Error reporting to Sentry (if `SENTRY_DSN` is set; configured
|
||||||
with `Repanic: true` so panics still reach Recoverer)
|
with `Repanic: true` so panics still reach Recoverer), and inside it the
|
||||||
|
naming of the request's Sentry transaction after its route pattern
|
||||||
|
|
||||||
Recoverer sits seventh rather than first, and both neighbours are the reason. It
|
Recoverer sits seventh rather than first, and both neighbours are the reason. It
|
||||||
runs **inside** everything that observes the response, so the `500` it writes
|
runs **inside** everything that observes the response, so the `500` it writes
|
||||||
@@ -3151,9 +3158,9 @@ set a header or start a timer.
|
|||||||
|
|
||||||
Each admin page route group (`/pages`, `/user/*`, `/settings`, `/hooks`,
|
Each admin page route group (`/pages`, `/user/*`, `/settings`, `/hooks`,
|
||||||
`/hook/*`) starts with its own **Recoverer** and, if `SENTRY_DSN` is set, its
|
`/hook/*`) starts with its own **Recoverer** and, if `SENTRY_DSN` is set, its
|
||||||
own **Sentry** error reporting. That Recoverer answers a panic with the `500`
|
own **Sentry** error reporting and transaction naming. That Recoverer answers a
|
||||||
error page in the normal layout; the global one keeps the plain-text `500` for
|
panic with the `500` error page in the normal layout; the global one keeps the
|
||||||
every other route.
|
plain-text `500` for every other route.
|
||||||
|
|
||||||
Additionally, form endpoints (`/pages`, `/user/*`, `/settings`, `/hooks`,
|
Additionally, form endpoints (`/pages`, `/user/*`, `/settings`, `/hooks`,
|
||||||
`/hook/*`) apply a **MaxBodySize** middleware that limits POST/PUT/PATCH request
|
`/hook/*`) apply a **MaxBodySize** middleware that limits POST/PUT/PATCH request
|
||||||
@@ -3207,9 +3214,9 @@ credential check, see [The login endpoint](#the-login-endpoint).
|
|||||||
- Session regeneration on login to prevent session fixation attacks
|
- Session regeneration on login to prevent session fixation attacks
|
||||||
- Session key is a 32-byte value auto-generated on first startup and stored in
|
- Session key is a 32-byte value auto-generated on first startup and stored in
|
||||||
the database
|
the database
|
||||||
- Production security headers on all responses: HSTS, X-Content-Type-Options
|
- Production security headers on all responses: HSTS (unless `HSTS_MAX_AGE` is
|
||||||
(`nosniff`), X-Frame-Options (`DENY`), Content-Security-Policy,
|
`0`), X-Content-Type-Options (`nosniff`), X-Frame-Options (`DENY`),
|
||||||
Referrer-Policy, and Permissions-Policy
|
Content-Security-Policy, Referrer-Policy, and Permissions-Policy
|
||||||
- Request body size limits (1 MB) on all form POST endpoints, enforced by
|
- Request body size limits (1 MB) on all form POST endpoints, enforced by
|
||||||
middleware that runs before CSRF parses the form
|
middleware that runs before CSRF parses the form
|
||||||
- **CSRF protection** via [gorilla/csrf](https://github.com/gorilla/csrf) on all
|
- **CSRF protection** via [gorilla/csrf](https://github.com/gorilla/csrf) on all
|
||||||
@@ -3339,47 +3346,36 @@ Two operational consequences follow from bounding the sequence:
|
|||||||
|
|
||||||
### Linting
|
### Linting
|
||||||
|
|
||||||
golangci-lint never runs on the host. `script/lint` builds `Dockerfile.lint`,
|
golangci-lint never runs on the host. `script/lint` builds the Dockerfile's
|
||||||
which copies the repo into the digest-pinned golangci-lint image and lints as a
|
`lint` phase, which copies the repo into the digest-pinned golangci-lint image
|
||||||
build step, so a successful build is a clean lint. A host binary would share one
|
and lints as a build step, so a successful build is a clean lint. A host binary
|
||||||
cache and one lock with every other checkout on the machine, which has produced
|
would share one cache and one lock with every other checkout on the machine,
|
||||||
both invented findings attributed to other worktrees and unearned passes.
|
which has produced both invented findings attributed to other worktrees and
|
||||||
|
unearned passes.
|
||||||
|
|
||||||
Three properties are load-bearing:
|
Two properties are load-bearing:
|
||||||
|
|
||||||
- `script/lint` passes `--no-cache-filter=lint`. Without it an unchanged tree
|
- `script/lint` passes `--no-cache`. Without it an unchanged tree replays the
|
||||||
replays the lint layer from cache and the build exits 0 in under a second
|
lint layer from cache and the build exits 0 in under a second having linted
|
||||||
having linted nothing. The `deps` stage stays cacheable, so module downloads
|
nothing. Never prune the shared build cache instead.
|
||||||
are not repeated. Invalidation is scoped to the one stage; never prune the
|
- Both golangci-lint steps use `RUN --network=none`.
|
||||||
shared build cache.
|
`golangci-lint config verify` is documented as fetching its JSON schema over
|
||||||
- `script/lint` does not trust that flag. Docker silently ignores
|
HTTPS, which would be an unpinned remote dependency; the pinned image resolves
|
||||||
`--no-cache-filter` for a stage name that does not match, so a stage rename or
|
the schema without network access, and `--network=none` enforces that instead
|
||||||
a one-character typo would restore the cached false green with no warning and
|
of trusting it. Verify is worth keeping because `golangci-lint run` silently
|
||||||
a fast exit 0. The script therefore tees the build output and treats a run as
|
ignores config keys it does not recognize, so a typo would disable a setting
|
||||||
a pass only if golangci-lint's own summary line (`N issues.` / `N issues:`)
|
with no warning.
|
||||||
appears in it: no summary, no lint, whatever the exit code says.
|
|
||||||
- Both lint steps use `RUN --network=none`. `golangci-lint config verify` is
|
|
||||||
documented as fetching its JSON schema over HTTPS, which would be an unpinned
|
|
||||||
remote dependency; the pinned image resolves the schema without network
|
|
||||||
access, and `--network=none` enforces that instead of trusting it. Verify is
|
|
||||||
worth keeping because `golangci-lint run` silently ignores config keys it does
|
|
||||||
not recognize, so a typo would disable a setting with no warning.
|
|
||||||
|
|
||||||
ESLint never runs on the host either. It lints `static/js/` (not the extracted
|
ESLint never runs on the host either. It lints `static/js/` (not the extracted
|
||||||
Alpine.js) in the Dockerfile's `js-lint` stage, which `script/lint` builds after
|
Alpine.js) in the Dockerfile's `js-lint` stage. The `lint` phase copies a file
|
||||||
`Dockerfile.lint` and the image build runs before the builder stage. Its version
|
from it, so `make lint` and the image build both run ESLint. Its version is
|
||||||
is pinned in `package.json` and every package's hash in `yarn.lock`. The
|
pinned in `package.json` and every package's hash in `yarn.lock`. The `js-deps`
|
||||||
`js-deps` stage before it installs ESLint with `yarn install --immutable`, which
|
stage before it installs ESLint with `yarn install --immutable`, which fails
|
||||||
fails rather than change `yarn.lock`. The yarn it runs is the one the
|
rather than change `yarn.lock`. The yarn it runs is the one the `packageManager`
|
||||||
`packageManager` field in `package.json` pins by version and hash, which the
|
field in `package.json` pins by version and hash, which the node image's own
|
||||||
node image's own corepack fetches and checks. The stage stays cached until
|
corepack fetches and checks. `eslint.config.mjs` turns on the rules of the
|
||||||
`package.json`, `yarn.lock` or `.yarnrc.yml` changes, so only the lint step
|
JavaScript styleguide `REPO_POLICIES.md` links to that a linter can check:
|
||||||
re-runs and ESLint is not downloaded again. `eslint.config.mjs` turns on the
|
`no-var` and `prefer-const`.
|
||||||
rules of the JavaScript styleguide `REPO_POLICIES.md` links to that a linter can
|
|
||||||
check: `no-var` and `prefer-const`. ESLint prints nothing on a pass, so
|
|
||||||
`script/lint` has no summary line to look for; it names the stage once for both
|
|
||||||
`--target` and `--no-cache-filter`, and `--target` fails on a name that matches
|
|
||||||
no stage.
|
|
||||||
|
|
||||||
prettier formats the Markdown, and it never runs on the host either. It is
|
prettier formats the Markdown, and it never runs on the host either. It is
|
||||||
pinned in `package.json` and `yarn.lock` beside ESLint, installed by the same
|
pinned in `package.json` and `yarn.lock` beside ESLint, installed by the same
|
||||||
@@ -3391,103 +3387,83 @@ on any Markdown file prettier would change.
|
|||||||
### Docker
|
### Docker
|
||||||
|
|
||||||
The Dockerfile uses a multi-stage build. Each stage is pinned by digest, and the
|
The Dockerfile uses a multi-stage build. Each stage is pinned by digest, and the
|
||||||
lint and builder stages are separate images so the linter's version is fixed
|
lint phase and the Go stages are separate images so the linter's version is
|
||||||
independently of the compiler's:
|
fixed independently of the compiler's:
|
||||||
|
|
||||||
1. **Lint stage** (`golangci/golangci-lint:v2.12.2`, Debian-based) — downloads
|
1. **Stylesheet stages** (`debian:bookworm-slim`, with the Tailwind standalone
|
||||||
dependencies, copies the source, and runs the `gofmt` check, then
|
|
||||||
`script/assets` to extract Alpine.js from `3p/`, then
|
|
||||||
`golangci-lint config verify` and `golangci-lint run`, both with
|
|
||||||
`--network=none`.
|
|
||||||
2. **Stylesheet stages** (`debian:bookworm-slim`, with the Tailwind standalone
|
|
||||||
CLI pinned by version and sha256, one binary per architecture) — generate
|
CLI pinned by version and sha256, one binary per architecture) — generate
|
||||||
`static/css/tailwind.css` from `static/css/input.css` and the files its
|
`static/css/tailwind.css` from `static/css/input.css` and the files its
|
||||||
`@source` lines name. `css-check` fails when the committed file differs from
|
`@source` lines name. `css-check` fails when the committed file differs from
|
||||||
the generated one, and `make css` writes the generated file out from
|
the generated one, and `make css` writes the generated file out from
|
||||||
`css-output` (see [Stylesheet](#stylesheet)).
|
`css-output` (see [Stylesheet](#stylesheet)).
|
||||||
3. **JavaScript lint stages** (`node:24.21.0-alpine`, with the yarn
|
2. **JavaScript lint stages** (`node:24.21.0-alpine`, with the yarn
|
||||||
`package.json` pins, run through the image's corepack) — `js-deps` installs
|
`package.json` pins, run through the image's corepack) — `js-deps` installs
|
||||||
ESLint and prettier from `yarn.lock` and `js-lint` runs ESLint over
|
ESLint and prettier from `yarn.lock` and `js-lint` runs ESLint over
|
||||||
`static/js/` (see [Linting](#linting)).
|
`static/js/` (see [Linting](#linting)).
|
||||||
4. **Markdown stages** (on `js-deps`) — `markdown-check` runs prettier over the
|
3. **Markdown stages** (on `js-deps`) — `markdown-check` runs prettier over the
|
||||||
Markdown and fails on any file it would change, and `make fmt` writes the
|
Markdown and fails on any file it would change, and `make fmt` writes the
|
||||||
formatted files out from `markdown-output`.
|
formatted files out from `markdown-output`.
|
||||||
5. **Builder stage** (`golang:1.26.1-bookworm`) — depends on the lint,
|
4. **Lint phase** (`lint`, `golangci/golangci-lint:v2.14.0`, Debian-based) —
|
||||||
`css-check`, `js-lint` and `markdown-check` stages passing (it copies a file
|
downloads dependencies, copies the source, and runs the `gofmt` check, then
|
||||||
from each), runs `make test` and `make build` (both extract Alpine.js from
|
`script/assets` to extract Alpine.js from `3p/`, then
|
||||||
`3p/` first), and finally rebuilds the binary with `CGO_ENABLED=1` and static
|
`golangci-lint config verify` and `golangci-lint run`, both with
|
||||||
linking so it runs on musl. Both builds go through `make build`, the relink
|
`--network=none`, and depends on `js-lint` (it copies a file from it).
|
||||||
adding its `-extldflags` via `GO_LDFLAGS`, so neither can drop the `-X` that
|
`make lint` builds this stage alone.
|
||||||
stamps the version. The version is the `VERSION` build arg if one is given,
|
5. **Test phase** (`test`, `golang:1.26.1-bookworm`, whose C compiler `-race`
|
||||||
otherwise derived from the `.git` in the context, and the stage fails if a
|
needs) — extracts Alpine.js, then runs `go test -race -cover` at most four
|
||||||
context with `.git` would stamp `unknown` (see
|
packages and eight tests at a time, with a 90-second timeout per package. On
|
||||||
[Version stamping](#version-stamping)).
|
a failure it runs only the failed tests again with `-v`, since verbose output
|
||||||
6. **Runtime stage** (`alpine:3.21`) — copies the static binary and
|
from the whole suite would pass the 2 MiB at which the Docker build cuts off
|
||||||
|
a step's log, and then fails. `make test` builds this stage alone.
|
||||||
|
6. **Builder stage** (`golang:1.26.1-bookworm`) — depends on the lint and test
|
||||||
|
phases and the `css-check` and `markdown-check` stages passing (it copies a
|
||||||
|
file from each), runs `make build` (which extracts Alpine.js from `3p/`
|
||||||
|
first), and then rebuilds the binary with `CGO_ENABLED=1` and static linking
|
||||||
|
so it runs on musl, with the build tags `netgo`, `osusergo` and
|
||||||
|
`sqlite_omit_load_extension` so it needs none of glibc's shared libraries at
|
||||||
|
run time. Both builds go through `make build`, the relink adding its
|
||||||
|
`-extldflags` via `GO_LDFLAGS` and its tags via `GO_TAGS`, so neither can
|
||||||
|
drop the `-X` that stamps the version. The version is the `VERSION` build arg
|
||||||
|
if one is given, otherwise derived from the `.git` in the context, and the
|
||||||
|
stage fails if a context with `.git` would stamp an empty version, `dev` or
|
||||||
|
`unknown` (see [Version stamping](#version-stamping)).
|
||||||
|
7. **Runtime stage** (`alpine:3.21`) — copies the static binary and
|
||||||
`deploy/docker-entrypoint.sh`, creates the `/var/lib/webhooker` directory for
|
`deploy/docker-entrypoint.sh`, creates the `/var/lib/webhooker` directory for
|
||||||
all SQLite databases, exposes port 8080, and includes a health check against
|
all SQLite databases, exposes port 8080, and includes a health check against
|
||||||
`/.well-known/healthcheck`. It sets no `USER`: the `ENTRYPOINT` script starts
|
`/.well-known/healthcheck`. It sets no `USER`: the `ENTRYPOINT` script starts
|
||||||
as root, sets the data directory's owner and mode, and runs the app as the
|
as root, sets the data directory's owner and mode, and runs the app as the
|
||||||
non-root `webhooker` user (UID 1000) through `su-exec`.
|
non-root `webhooker` user (UID 1000) through `su-exec`.
|
||||||
|
|
||||||
The lint stage invokes `gofmt` and `golangci-lint` directly rather than
|
The lint and test phases invoke `gofmt`, `golangci-lint` and `go test` directly
|
||||||
`make fmt-check` and `make lint`: it is already the pinned linter image, and
|
rather than `make fmt-check`, `make lint` and `make test`: those targets build
|
||||||
both targets build docker stages, which would need a docker daemon inside this
|
docker stages, which would need a docker daemon inside this build.
|
||||||
build.
|
|
||||||
|
|
||||||
The lint and builder stages use Debian rather than Alpine because
|
The lint phase and the Go stages use Debian rather than Alpine because
|
||||||
`gorm.io/driver/sqlite` pulls in `mattn/go-sqlite3`, which needs CGO and does
|
`gorm.io/driver/sqlite` pulls in `mattn/go-sqlite3`, which needs CGO and does
|
||||||
not compile against musl. Only the final binary is statically linked, which is
|
not compile against musl. Only the final binary is statically linked, which is
|
||||||
what lets it run on the Alpine runtime image.
|
what lets it run on the Alpine runtime image.
|
||||||
|
|
||||||
`script/cibuild` — `docker build .` — is the CI gate: the checks run inside the
|
`script/cibuild` is the CI gate: it runs `script/bootstrap`, then
|
||||||
image, so a build that succeeds is a repo that is formatted, linted, tested and
|
`script/check`, then builds the image, whose build runs the lint and test phases
|
||||||
compiled, with a current stylesheet. `script/lint` also uses Docker
|
and the stylesheet and Markdown checks again. A build that succeeds is a repo
|
||||||
(`Dockerfile.lint` and the `js-lint` stage, see Linting above), so `make lint`
|
that is formatted, linted, tested and compiled, with a current stylesheet.
|
||||||
and `make check` run the same pinned linter versions the gate does; of the steps
|
`make check` runs the same stages the gate does; of its steps, only the `gofmt`
|
||||||
`make check` runs, only `script/test` and the `gofmt` check in
|
check in `script/fmt-check` runs on the host.
|
||||||
`script/fmt-check` run on the host.
|
|
||||||
|
|
||||||
#### CI gate honesty
|
#### CI gate honesty
|
||||||
|
|
||||||
A layer cache lets `docker build .` exit 0 in seconds with the lint and test
|
A layer cache lets `docker build .` exit 0 in seconds with the lint and test
|
||||||
stages replayed rather than executed, which would make a green check
|
stages replayed rather than executed, which would make a green check
|
||||||
meaningless. The `check` workflow therefore writes `.ci-fingerprint` into the
|
meaningless. Every `docker build` in `script/` therefore passes `--no-cache`, so
|
||||||
build context before building. Its value is the hash of the commit being
|
on every run the `gofmt` check, `golangci-lint`, ESLint, the stylesheet check,
|
||||||
checked, so every commit, docs-only ones and a squash merge whose tree matches
|
the Markdown check, `go test` and `make build` really execute. A run that
|
||||||
an already-built branch included, gets a new fingerprint, invalidates the
|
reports success ran them. A bare `docker build .` carries no such guarantee.
|
||||||
`COPY . .` layer of every check stage, and really runs the `gofmt` check,
|
|
||||||
`golangci-lint`, the stylesheet check, ESLint, the Markdown check, `make test`,
|
|
||||||
and `make build`. A run that reports success ran them.
|
|
||||||
|
|
||||||
The module download layer sits above `COPY . .` and stays cached.
|
The `check` workflow is the shared one from `REPO_POLICIES.md`: it checks out
|
||||||
|
the repository and runs `script/cibuild`, nothing else. Gitea cancels an
|
||||||
A separate workflow step, run before the fingerprint is written, covers a second
|
in-flight run when a newer commit lands on the same branch and records that as
|
||||||
way the gate lied: Gitea cancels an in-flight run when a newer commit lands on
|
`failure` / `Has been cancelled`: nothing was verified about that commit, so
|
||||||
the same branch and records that cancellation as a `failure` status, so a commit
|
test the commit itself before concluding anything about it.
|
||||||
nothing ever tested reads as a test result. Cancellation is unconditional
|
|
||||||
server-side for push events, so the superseding run calls
|
|
||||||
`script/ci-mark-superseded`, which rewrites that exact status to `failure` /
|
|
||||||
`Superseded by a newer commit; never tested`.
|
|
||||||
|
|
||||||
The state stays `failure` on purpose: Gitea's combined status folds `skipped`
|
|
||||||
into `success`, so marking a never-tested commit `skipped` made the status API
|
|
||||||
report green for it, indistinguishable from a commit that passed. Reading a
|
|
||||||
commit's status on this repo therefore goes:
|
|
||||||
|
|
||||||
- `success` / `Successful in ...` — the checks ran and passed.
|
|
||||||
- `failure` / `Failing after ...` — the checks ran and failed.
|
|
||||||
- `failure` / `Superseded by a newer commit; never tested` — the run was
|
|
||||||
cancelled, by a newer push or by hand, and nothing was verified about this
|
|
||||||
commit. Test the commit itself before concluding anything about it.
|
|
||||||
|
|
||||||
Genuine failures and successes are never touched, and no status is left
|
|
||||||
`pending`, which would block the commit indefinitely. The step derives its
|
|
||||||
context string from the workflow name, the job **id** and the event. That is
|
|
||||||
deliberately not byte-identical to Gitea's own rule, which uses the job's
|
|
||||||
display `name:` where the runner exports the id, so giving the job a `name:` —
|
|
||||||
or renaming the workflow — makes the derived context stop matching. The step
|
|
||||||
fails loudly when no status on the commit carries that context, so no rename can
|
|
||||||
silently disable the rewrite.
|
|
||||||
|
|
||||||
## TODO
|
## TODO
|
||||||
|
|
||||||
|
|||||||
+353
-90
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Repository Policies
|
title: Repository Policies
|
||||||
last_modified: 2026-08-07
|
last_modified: 2026-10-04
|
||||||
---
|
---
|
||||||
|
|
||||||
This document covers repository structure, tooling, and workflow standards. Code
|
This document covers repository structure, tooling, and workflow standards. Code
|
||||||
@@ -60,17 +60,28 @@ style conventions are in separate documents:
|
|||||||
prerequisite since nvm requires bash. yarn is then pinned via
|
prerequisite since nvm requires bash. yarn is then pinned via
|
||||||
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
|
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
|
||||||
always exact versions. `script/cibuild` runs the CI build: it changes to the
|
always exact versions. `script/cibuild` runs the CI build: it changes to the
|
||||||
repo root and runs `docker build .`; the Gitea workflow calls it. Four further
|
repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
|
||||||
scripts are our own extensions to the standard: `script/check` runs
|
with the version; the Gitea workflow calls it. **`script/cibuild` runs
|
||||||
`script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is
|
`script/bootstrap` first**, because the workflow checks out the repo and runs
|
||||||
what the git pre-commit hook runs, and it calls `script/check`;
|
nothing else, while `script/fmt-check` runs the formatter on the host: on a
|
||||||
`script/install-precommit` installs the git pre-commit hook (the `make hooks`
|
pristine checkout with nothing installed the run dies there, after the
|
||||||
target shims to it); and `script/projectname` (literally that filename) simply
|
containerised gates have passed. **The bootstrap alone is not enough**:
|
||||||
outputs the project's name. Scripts that need the name call
|
`script/bootstrap` installs node and yarn under nvm and leaves neither on the
|
||||||
`script/projectname` — e.g. `script/docker` assembles its image tag from it —
|
`PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
|
||||||
so those scripts stay byte-identical across all repos. Repo-type-specific
|
entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
|
||||||
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
|
source nvm for the pinned node version before invoking it, exactly as
|
||||||
`script/precommit`, not in the hook itself. Model scripts are at
|
`script/bootstrap`'s own install step does. A runner carrying nothing but
|
||||||
|
docker and git then gets through `script/check`. Four further scripts are our
|
||||||
|
own extensions to the standard: `script/check` runs `script/test`,
|
||||||
|
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
|
||||||
|
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
|
||||||
|
installs the git pre-commit hook (the `make hooks` target shims to it); and
|
||||||
|
`script/projectname` (literally that filename) simply outputs the project's
|
||||||
|
name. Scripts that need the name call `script/projectname` — e.g.
|
||||||
|
`script/docker` assembles its image tag from it — so those scripts stay
|
||||||
|
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
|
||||||
|
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
|
||||||
|
the hook itself. Model scripts are at
|
||||||
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
|
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
|
||||||
must document the provided scripts in an **Entrypoints** section (see the
|
must document the provided scripts in an **Entrypoints** section (see the
|
||||||
README requirements below).
|
README requirements below).
|
||||||
@@ -89,87 +100,198 @@ style conventions are in separate documents:
|
|||||||
contributor should be able to understand the entire development workflow by
|
contributor should be able to understand the entire development workflow by
|
||||||
reading the Makefile.
|
reading the Makefile.
|
||||||
|
|
||||||
- Every repo should have a `Dockerfile`. All Dockerfiles must run `make check`
|
- Every repo should have a `Dockerfile`, and it carries the repo's gates: a
|
||||||
as a build step so the build fails if the branch is not green. For non-server
|
`lint` phase and a `test` phase, with the final stage depending on both so the
|
||||||
repos, the Dockerfile should bring up a development environment and run
|
image cannot be built unless they pass. For non-server repos the final stage
|
||||||
`make check`. For server repos, `make check` should run as an early build
|
brings up a development environment; for server repos it is the runtime image.
|
||||||
stage before the final image is assembled. Dockerfiles install development
|
The gate phases and the build stage start from their pinned base images and
|
||||||
prerequisites by running `script/bootstrap` rather than duplicating installs
|
install what those images lack either inline, as the canonical Go `Dockerfile`
|
||||||
inline; COPY `script/` and the dependency manifests (`package.json` +
|
below does for `git`, or by running `script/bootstrap`, as the `prompts`
|
||||||
`yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap
|
repo's own `Dockerfile` does for its yarn packages. The development
|
||||||
layer stays cached until dependencies change.
|
environment stage installs development prerequisites by running
|
||||||
|
`script/bootstrap` rather than duplicating its installs inline. A stage that
|
||||||
|
runs `script/bootstrap` COPYs `script/` and the dependency manifests
|
||||||
|
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
|
||||||
|
|
||||||
- **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go
|
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
|
||||||
repos use a multistage build where linting runs in an independent stage based
|
no separate lint file. `script/lint` and `script/test` each build one phase
|
||||||
on the `golangci/golangci-lint` image (pinned by hash). This stage runs
|
and nothing else:
|
||||||
`make fmt-check` and `make lint` before the full build begins. The build stage
|
|
||||||
then declares an explicit dependency on the lint stage via
|
|
||||||
`COPY --from=lint /src/go.sum /dev/null`, which forces BuildKit to complete
|
|
||||||
linting before proceeding to compilation and tests. This ensures lint failures
|
|
||||||
surface in seconds rather than minutes, without blocking on dependency
|
|
||||||
download or compilation in the build stage.
|
|
||||||
|
|
||||||
The standard pattern for a Go repo Dockerfile is:
|
```sh
|
||||||
|
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
|
||||||
|
docker build --no-cache --target test -t "$(script/projectname)-test" .
|
||||||
|
```
|
||||||
|
|
||||||
|
**A stage that is not the last one in the file is built only when the final
|
||||||
|
stage's chain depends on it, or when `--target` names it.** That is why the
|
||||||
|
two gates are always invoked by name here, and why the final stage carries a
|
||||||
|
`COPY --from=` of a harmless file from each of them: without that edge a
|
||||||
|
plain `docker build .` builds the last stage alone and exits 0 having linted
|
||||||
|
and tested nothing.
|
||||||
|
|
||||||
|
**Every `docker build` in `script/` is tagged**, here and in
|
||||||
|
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
|
||||||
|
image behind on every invocation, on every developer host and every CI
|
||||||
|
runner; a tagged one replaces the previous image.
|
||||||
|
|
||||||
|
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
|
||||||
|
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
|
||||||
|
themselves a `docker build` and would recurse into a daemon that does not
|
||||||
|
exist in a build step. Formatting is the exception and stays on the host:
|
||||||
|
`script/fmt` writes the working tree, and `script/fmt-check` is its
|
||||||
|
read-only twin.
|
||||||
|
|
||||||
|
**No lint verdict may come from a host invocation of the linter.** On a
|
||||||
|
shared host golangci-lint reads a result cache keyed on file content rather
|
||||||
|
than location, so a second checkout of the same content is served the first
|
||||||
|
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
|
||||||
|
exit non-zero with `parallel golangci-lint is running` — a status a caller
|
||||||
|
cannot tell from real findings. Both have produced wrong verdicts in this
|
||||||
|
org, in both directions. A container has its own cache, its own `TMPDIR` and
|
||||||
|
a digest-pinned binary, so neither is reachable.
|
||||||
|
|
||||||
|
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
|
||||||
|
a `COPY` layer only when the copied content changes, so on an unchanged tree
|
||||||
|
the check `RUN` is served from cache, nothing executes, and the build still
|
||||||
|
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
|
||||||
|
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
|
||||||
|
four, and there is no fifth — `script/check` runs the two gate phases and
|
||||||
|
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
|
||||||
|
not evidence that anything ran: a sub-second build reporting success is a
|
||||||
|
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
|
||||||
|
and friends destroy a build cache shared with every other build on the host.
|
||||||
|
When a check is added or changed, prove it works by planting a defect it must
|
||||||
|
catch and watching the run fail on it, then revert the defect. A green run
|
||||||
|
alone shows neither that the check ran nor that it covers what it should.
|
||||||
|
|
||||||
|
- **The gate phases are separate stages, and the build stage depends on both.**
|
||||||
|
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
||||||
|
hash), so lint failures surface in seconds rather than after a full compile,
|
||||||
|
and the test phase is based on the Debian Go image. The canonical Go repo
|
||||||
|
`Dockerfile`:
|
||||||
|
|
||||||
```dockerfile
|
```dockerfile
|
||||||
# Lint stage — fast feedback on formatting and lint issues
|
# Lint phase
|
||||||
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
|
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
|
||||||
FROM golangci/golangci-lint@sha256:... AS lint
|
FROM golangci/golangci-lint@sha256:... AS lint
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN make fmt-check
|
RUN golangci-lint run --config .golangci.yml ./...
|
||||||
RUN make lint
|
|
||||||
|
|
||||||
# Build stage
|
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
|
||||||
# golang:1.x-alpine, YYYY-MM-DD
|
# image ships and the alpine one does not.
|
||||||
FROM golang@sha256:... AS builder
|
# golang:1.x, YYYY-MM-DD
|
||||||
|
FROM golang@sha256:... AS test
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
|
|
||||||
# Force BuildKit to run the lint stage before proceeding
|
|
||||||
COPY --from=lint /src/go.sum /dev/null
|
|
||||||
|
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN make test
|
RUN go test -timeout 90s -race -cover ./... || \
|
||||||
|
{ echo "--- Rerunning with -v for details ---"; \
|
||||||
|
go test -timeout 90s -race -v ./...; exit 1; }
|
||||||
|
|
||||||
ARG VERSION=dev
|
# Build stage. Nothing is wanted from either phase above; the copies
|
||||||
RUN CGO_ENABLED=0 go build -trimpath \
|
# are what make BuildKit build them first, so this stage cannot run
|
||||||
-ldflags="-s -w -X main.Version=${VERSION}" \
|
# unless lint and test passed.
|
||||||
-o /app ./cmd/app/
|
# golang:1.x-alpine, YYYY-MM-DD
|
||||||
|
FROM golang@sha256:... AS builder
|
||||||
|
COPY --from=lint /src/go.sum /dev/null
|
||||||
|
COPY --from=test /src/go.sum /dev/null
|
||||||
|
RUN apk add --no-cache git
|
||||||
|
# A tar-stream context keeps the sender's file owners, which git refuses.
|
||||||
|
RUN git config --system --add safe.directory /src
|
||||||
|
WORKDIR /src
|
||||||
|
COPY go.mod go.sum ./
|
||||||
|
RUN go mod download
|
||||||
|
COPY . .
|
||||||
|
|
||||||
# Runtime stage
|
# The VERSION build arg when one is given, otherwise
|
||||||
|
# `git describe --tags --always` on the .git in the build context. With
|
||||||
|
# .git present, a version that is still empty, dev or unknown fails the
|
||||||
|
# build: git is missing or could not read the checkout.
|
||||||
|
ARG VERSION
|
||||||
|
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
|
||||||
|
if [ -e .git ]; then \
|
||||||
|
case "$VERSION" in ""|dev|unknown) \
|
||||||
|
echo "version is '$VERSION' although .git is present" >&2; \
|
||||||
|
exit 1 ;; \
|
||||||
|
esac; \
|
||||||
|
fi; \
|
||||||
|
CGO_ENABLED=0 go build -trimpath \
|
||||||
|
-ldflags="-s -w -X main.Version=${VERSION}" \
|
||||||
|
-o /app ./cmd/app/
|
||||||
|
|
||||||
|
# Runtime stage, and the last one
|
||||||
FROM alpine@sha256:...
|
FROM alpine@sha256:...
|
||||||
COPY --from=builder /app /usr/local/bin/app
|
COPY --from=builder /app /usr/local/bin/app
|
||||||
ENTRYPOINT ["app"]
|
ENTRYPOINT ["app"]
|
||||||
```
|
```
|
||||||
|
|
||||||
Key points:
|
Key points:
|
||||||
- The lint stage uses the `golangci/golangci-lint` image directly (it
|
- The lint phase uses the `golangci/golangci-lint` image directly (it has
|
||||||
includes both Go and the linter), so there is no need to install the
|
both Go and the linter), so nothing needs installing.
|
||||||
linter separately.
|
- `COPY --from=<phase> /src/go.sum /dev/null` is a no-op copy whose only
|
||||||
- `COPY --from=lint /src/go.sum /dev/null` is a no-op file copy that creates
|
purpose is the ordering edge. BuildKit runs stages in parallel by default,
|
||||||
a stage dependency. BuildKit runs stages in parallel by default; without
|
and a stage nothing depends on is not built at all, so without these two
|
||||||
this line, the build stage would not wait for lint to finish and a lint
|
lines a red gate would not fail the build.
|
||||||
failure might not fail the overall build.
|
- Keep the runtime stage last, and if you add a stage after it, give it the
|
||||||
|
same two copies. A plain `docker build .` builds the last stage's chain
|
||||||
|
and nothing else.
|
||||||
- If the project uses `//go:embed` directives that reference build artifacts
|
- If the project uses `//go:embed` directives that reference build artifacts
|
||||||
(e.g. a web frontend compiled in a separate stage), the lint stage must
|
(e.g. a web frontend compiled in a separate stage), the lint phase must
|
||||||
create placeholder files so the embed directives resolve. Example:
|
create placeholder files so the embed directives resolve. Example:
|
||||||
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
|
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
|
||||||
The lint stage should not depend on the actual build output — it exists to
|
- If the project requires CGO or system libraries for linting, install them
|
||||||
fail fast.
|
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
|
||||||
- If the project requires CGO or system libraries for linting (e.g.
|
has no `apk`, so install with `apt-get` under the Debian package name
|
||||||
`vips-dev`), install them in the lint stage with `apk add`.
|
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
|
||||||
- The build stage runs `make test` after compilation setup. Tests run in the
|
lists in the same `RUN`, so the layer does not keep them:
|
||||||
build stage, not the lint stage, because they may require compiled
|
|
||||||
artifacts or heavier dependencies.
|
```dockerfile
|
||||||
|
RUN apt-get update \
|
||||||
|
&& apt-get install -y --no-install-recommends libvips-dev \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
```
|
||||||
|
|
||||||
|
- `.dockerignore` lets `.git` into the build context. It keeps out every git
|
||||||
|
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
|
||||||
|
repository's own, each submodule's under `.git/modules/`, and that of a
|
||||||
|
submodule keeping its own `.git` directory. `git describe` does not need
|
||||||
|
them, and each can hold a credential: a password in a remote URL, or the
|
||||||
|
token the CI checkout step stores there. A submodule whose name has a
|
||||||
|
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
|
||||||
|
git directory to `**/.git/modules/**/config`, and Go's version stamping
|
||||||
|
then fails the build: give it a name without that segment
|
||||||
|
(`git submodule add --name`). The stage that compiles has `git` (the
|
||||||
|
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
|
||||||
|
takes the version from the `VERSION` build argument when one is given,
|
||||||
|
otherwise from `git describe --tags --always`. That gives the tag on a
|
||||||
|
tagged commit; on a later commit, the tag, the number of commits since it
|
||||||
|
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
|
||||||
|
tag is reachable. The stage that compiles also marks its working directory
|
||||||
|
safe for git (`git config --system --add safe.directory /src`): a context
|
||||||
|
sent as a tar stream keeps the sender's file owners, and git refuses a
|
||||||
|
checkout owned by another user, so the version would come out empty.
|
||||||
|
`ARG VERSION` has no default, and the build fails if the context carries
|
||||||
|
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
|
||||||
|
`docker build .` with no build arguments must succeed; a Dockerfile that
|
||||||
|
refuses an empty build argument drops that refusal and keeps the argument.
|
||||||
|
|
||||||
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
|
||||||
runs `script/cibuild` (which runs `docker build .`) on push. Since the
|
runs `script/cibuild` on push, and checks out the repo as its only other step.
|
||||||
Dockerfile already runs `make check`, a successful build implies all checks
|
That script bootstraps, runs the gate phases, and then builds the image, so a
|
||||||
pass.
|
successful run means every check passed; a bare `docker build .` does not
|
||||||
|
carry the same guarantee, because its gate phases may come from the cache. The
|
||||||
|
image build is uncached and so runs the gate phases a second time. That is the
|
||||||
|
price of the rule above, and it is worth paying: the image that ships is built
|
||||||
|
from a run of its own gates rather than from a cache entry. A separate
|
||||||
|
workflow limited to `main` by a `branches` list under `on: push` cannot be
|
||||||
|
checked by review: to try a change to it, add the feature branch to that list
|
||||||
|
and push, then remove the branch from the list again before merging. Keep any
|
||||||
|
job in it that publishes behind `if: github.ref_name == 'main'`, so the run
|
||||||
|
from the feature branch publishes nothing.
|
||||||
|
|
||||||
- Use platform-standard formatters: `black` for Python, `prettier` for
|
- Use platform-standard formatters: `black` for Python, `prettier` for
|
||||||
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
|
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
|
||||||
@@ -193,15 +315,17 @@ style conventions are in separate documents:
|
|||||||
suite that exceeds it fails. Under 20 seconds is the target. A suite between
|
suite that exceeds it fails. Under 20 seconds is the target. A suite between
|
||||||
20 and 60 seconds is still green, but the overage must be filed as an
|
20 and 60 seconds is still green, but the overage must be filed as an
|
||||||
improvement bug against that repo. Add a 90-second timeout to the test
|
improvement bug against that repo. Add a 90-second timeout to the test
|
||||||
invocation in the Makefile (`go test -timeout 90s`). The backstop deliberately
|
invocation (`go test -timeout 90s`). The backstop deliberately sits above the
|
||||||
sits above the hard cap so that it catches a genuinely hung test rather than a
|
hard cap so that it catches a genuinely hung test rather than a merely slow
|
||||||
merely slow one.
|
one.
|
||||||
|
|
||||||
- **`make test` should use the conditional verbose rerun pattern.** Run tests
|
- **The test command should use the conditional verbose rerun pattern.** Run
|
||||||
without `-v` (verbose) first. If tests fail, automatically rerun with `-v` to
|
tests without `-v` (verbose) first. If tests fail, automatically rerun with
|
||||||
show full output. This keeps CI logs and `docker build` output clean on
|
`-v` to show full output. This keeps CI logs and `docker build` output clean
|
||||||
success (just package/suite summaries) while providing full diagnostic detail
|
on success (just package/suite summaries) while providing full diagnostic
|
||||||
on failure (every test case, every assertion). The general shell pattern:
|
detail on failure (every test case, every assertion). The command lives in the
|
||||||
|
`test` phase of the `Dockerfile`, since `script/test` builds that phase; the
|
||||||
|
Makefile form below is the same pattern for any repo-local invocation:
|
||||||
|
|
||||||
```makefile
|
```makefile
|
||||||
test:
|
test:
|
||||||
@@ -214,11 +338,26 @@ style conventions are in separate documents:
|
|||||||
|
|
||||||
```makefile
|
```makefile
|
||||||
test:
|
test:
|
||||||
@go test -timeout 90s -race -cover ./... || \
|
@go test -count=1 -timeout 90s -race -cover ./... || \
|
||||||
{ echo "--- Rerunning with -v for details ---"; \
|
{ echo "--- Rerunning with -v for details ---"; \
|
||||||
go test -timeout 90s -race -v ./...; exit 1; }
|
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`-count=1` is required on both invocations: it defeats Go's test _result_
|
||||||
|
cache, so neither run can report a stored pass in place of running the
|
||||||
|
tests. It leaves the build cache alone, so it costs the runtime of the suite
|
||||||
|
and no recompilation.
|
||||||
|
|
||||||
|
That cache is Go's own, separate from Docker's layer cache. Go stores a
|
||||||
|
passing result in its cache directory (`GOCACHE`), and when the same tests
|
||||||
|
run again on unchanged code it prints that result, marked `(cached)`,
|
||||||
|
without running them. That matters on a developer's machine, where this
|
||||||
|
target runs and the directory lasts from one run to the next. The `test`
|
||||||
|
phase of the `Dockerfile` needs no `-count=1`: its base image holds no
|
||||||
|
result for this repo's tests and nothing before its `go test` step runs a
|
||||||
|
test, so there is nothing to replay. `--no-cache` (above) is what makes that
|
||||||
|
step run on an unchanged tree.
|
||||||
|
|
||||||
Python example:
|
Python example:
|
||||||
|
|
||||||
```makefile
|
```makefile
|
||||||
@@ -244,10 +383,84 @@ style conventions are in separate documents:
|
|||||||
must be in `.gitignore`. No exceptions.
|
must be in `.gitignore`. No exceptions.
|
||||||
|
|
||||||
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
|
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
|
||||||
editor files (`.swp`, `*~`), language build artifacts, and `node_modules/`.
|
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
|
||||||
Fetch the standard `.gitignore` from
|
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
|
||||||
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
|
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
|
||||||
a new repo.
|
setting up a new repo. These patterns are written to `.gitignore`'s own
|
||||||
|
semantics, in which an unanchored pattern already matches at every depth; they
|
||||||
|
are not a `.dockerignore` and must not be transplanted into one unmodified.
|
||||||
|
|
||||||
|
- **`.dockerignore` does not use `.gitignore` semantics, and copying patterns
|
||||||
|
across unmodified leaves secrets in the build context.** Docker matches with
|
||||||
|
`moby/patternmatcher`: `filepath.Match` semantics plus a `**` extension, so
|
||||||
|
`*` does not cross `/` and a pattern without a leading `**/` is anchored at
|
||||||
|
the build-context root. A `.dockerignore` listing `.env`, `*.pem` and `*.key`
|
||||||
|
therefore excludes only the copies at the repository root, while `config/.env`
|
||||||
|
and `certs/server.key` still reach the context and can land in an image layer
|
||||||
|
— which is more dangerous than a short file with no secret patterns at all,
|
||||||
|
because it reads as solved and stops anyone looking. Give every
|
||||||
|
depth-independent pattern the `**/` prefix and leave only genuinely
|
||||||
|
root-anchored entries unprefixed: `.claude`, and the repo's own host-built
|
||||||
|
binary, written `/myapp` and never `**/myapp`, which would also match
|
||||||
|
`cmd/myapp/` and delete the package directory from the context. Matching is
|
||||||
|
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
|
||||||
|
secret names use character ranges — `**/*.[kK][eE][yY]`, `**/*.[pP][eE][mM]`,
|
||||||
|
and likewise for `.envrc` and the extensionless SSH keys. Where such a pattern
|
||||||
|
also catches something the build needs, re-include it with a negation
|
||||||
|
(`!docs/example.env`); deleting the pattern reopens the exposure for every
|
||||||
|
other file it covers. Fetch the standard `.dockerignore` from
|
||||||
|
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.dockerignore` and extend
|
||||||
|
it with the repo's own artifacts.
|
||||||
|
|
||||||
|
- **In-repo agent scratch belongs in both files, written to each file's own
|
||||||
|
semantics.** `.claude/` holds one worktree per in-flight agent — an entire
|
||||||
|
additional checkout of the repo — so under `COPY . .` the build context
|
||||||
|
inflates by a multiple of the repo and another session's unreviewed work can
|
||||||
|
be copied into an image layer. In `.gitignore` the entry is `.claude/`,
|
||||||
|
unanchored. In `.dockerignore` it is `.claude`, anchored and with **no** `**/`
|
||||||
|
prefix, because the prefixed form would also delete any nested directory of
|
||||||
|
that name from the build. Anchoring carries a known gap that the canonical
|
||||||
|
`.dockerignore` states in its own comment, since consuming repos receive the
|
||||||
|
file and not the tracker: the directory is created in the agent's working
|
||||||
|
directory, so a repo running agents in subdirectories still ships
|
||||||
|
`services/api/.claude/` and must add its own anchored entry there.
|
||||||
|
|
||||||
|
- **A plain `docker build .` of a clone stamps the version that
|
||||||
|
`git describe --tags --always` gives**, derived from the `.git` in the build
|
||||||
|
context as the canonical `Dockerfile` above shows. Without its failure check,
|
||||||
|
a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
|
||||||
|
and the build would still exit 0. `script/docker` and `script/cibuild` pass
|
||||||
|
the version they compute on the host; it takes precedence. They do this
|
||||||
|
byte-identically across repos:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# Own line: a failing command substitution inside an argument does not
|
||||||
|
# trip `set -e`, so the inline form degrades to an empty constant.
|
||||||
|
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
|
||||||
|
[ -n "$version" ] || version="unknown"
|
||||||
|
docker build --no-cache \
|
||||||
|
--build-arg VERSION="$version" \
|
||||||
|
-t "$(script/projectname)" .
|
||||||
|
```
|
||||||
|
|
||||||
|
`--always` makes an untagged repo yield an abbreviated commit hash rather
|
||||||
|
than failing, and the `[ -n "$version" ]` line is the single place the
|
||||||
|
fallback is applied — a live check that fires on a build from an export with
|
||||||
|
no `.git` and on a repository with no commits yet. Do not fold it into the
|
||||||
|
substitution as `|| echo unknown`, which makes the guard unreachable. The
|
||||||
|
Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
|
||||||
|
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
|
||||||
|
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
|
||||||
|
the scripts stay byte-identical. One consequence for CI: the standard
|
||||||
|
checkout action clones shallow and fetches no tags, so a repo that embeds a
|
||||||
|
tag-derived version must set `fetch-depth: 0` on its checkout step.
|
||||||
|
|
||||||
|
- **Verify `.dockerignore` by enumerating the image, not by reading the
|
||||||
|
patterns.** Plant files at the root _and_ at least two directories deep, build
|
||||||
|
a probe image that does `COPY . .`, and list what actually landed
|
||||||
|
(`docker run --rm --entrypoint find IMAGE /app`). The `transferring context`
|
||||||
|
size is not a substitute: a nested secret is a few bytes, and BuildKit
|
||||||
|
transfers only the delta from the previous build.
|
||||||
|
|
||||||
- **No build artifacts in version control.** Code-derived data (compiled
|
- **No build artifacts in version control.** Code-derived data (compiled
|
||||||
bundles, minified output, generated assets) must never be committed to the
|
bundles, minified output, generated assets) must never be committed to the
|
||||||
@@ -263,12 +476,56 @@ style conventions are in separate documents:
|
|||||||
- Make all changes on a feature branch. You can do whatever you want on a
|
- Make all changes on a feature branch. You can do whatever you want on a
|
||||||
feature branch.
|
feature branch.
|
||||||
|
|
||||||
- `.golangci.yml` is standardized and must _NEVER_ be modified by an agent, only
|
- `.golangci.yml` is standardized. The vendored copy in a consuming repo must
|
||||||
manually by the user. Fetch from
|
_NEVER_ be modified by an agent: fetch it from
|
||||||
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`. The
|
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it
|
||||||
canonical golangci-lint version is v2.12.2 (released 2026-05-06), installed
|
byte-identical, so that no repo can quietly loosen its own linting. Linter
|
||||||
commit-pinned via
|
configuration changes are made to the canonical copy in the `prompts` repo and
|
||||||
`go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@c0d3ddc9cf3faa61a4e378e879ece580256d76e5`.
|
reach consuming repos by re-vendoring; an agent may open a PR against
|
||||||
|
canonical, which only the user merges. One list is exempt from byte-identity,
|
||||||
|
because it cannot be written once for every repo: the `deny` list of the
|
||||||
|
`test-support` depguard rule, where a repo names its own test-support packages
|
||||||
|
by full import path. A repo adds entries there and changes nothing else, and a
|
||||||
|
re-vendor carries its entries forward. The canonical golangci-lint version is
|
||||||
|
v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
|
||||||
|
image
|
||||||
|
(`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
|
||||||
|
which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
|
||||||
|
directive must not name a newer Go minor version than the one golangci-lint
|
||||||
|
was built with, or golangci-lint refuses to lint it: this release lints
|
||||||
|
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
|
||||||
|
installs golangci-lint on the host. A repo sets the lint phase digest to the
|
||||||
|
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
|
||||||
|
the two prompted the change: the canonical copy can name linters that an older
|
||||||
|
golangci-lint rejects, and a newer golangci-lint can add linters that
|
||||||
|
`default: all` switches on until the canonical copy disables them.
|
||||||
|
|
||||||
|
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
|
||||||
|
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
|
||||||
|
`PATH` only, so on an already-provisioned machine the pin is inert and a
|
||||||
|
version bump is a silent no-op — while the Dockerfile, installing into a clean
|
||||||
|
image, gets the pinned version, so a local `make check` and `make docker` can
|
||||||
|
disagree about what the tool even is. The canonical form:
|
||||||
|
- compares the installed version against the pin over the **whole** version
|
||||||
|
token; a parser that stops at the first `-` reports `2.12.2` for a host
|
||||||
|
running `2.12.2-rc1` and skips the install;
|
||||||
|
- treats absent, non-zero, empty or unrecognised `--version` output as a
|
||||||
|
mismatch, so the failure direction is a redundant install and never a
|
||||||
|
skipped one;
|
||||||
|
- after installing, re-resolves the binary the way callers do — `hash -r`,
|
||||||
|
then through `PATH`, not through the directory the installer wrote to —
|
||||||
|
and fails naming the resolved path, since an install that a shadowing
|
||||||
|
binary hides succeeds while changing nothing any caller sees;
|
||||||
|
- is actually called, and prints the version on both success paths: a
|
||||||
|
function defined and never invoked has the same exit status and the same
|
||||||
|
empty output as one that worked.
|
||||||
|
|
||||||
|
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
|
||||||
|
|
||||||
|
A Go tool a repo needs on the host is installed with `go install` pinned to
|
||||||
|
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
|
||||||
|
a `go.mod` tool dependency or through a `tools.go` file, either of which
|
||||||
|
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
|
||||||
|
|
||||||
- When pinning images or packages by hash, add a comment above the reference
|
- When pinning images or packages by hash, add a comment above the reference
|
||||||
with the version and date (YYYY-MM-DD).
|
with the version and date (YYYY-MM-DD).
|
||||||
@@ -382,12 +639,14 @@ style conventions are in separate documents:
|
|||||||
settings.
|
settings.
|
||||||
|
|
||||||
- Avoid putting files in the repo root unless necessary. Root should contain
|
- Avoid putting files in the repo root unless necessary. Root should contain
|
||||||
only project-level config files (`README.md`, `Makefile`, `Dockerfile`,
|
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
|
||||||
`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and
|
`Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
|
||||||
language-specific config). Everything else goes in a subdirectory. Canonical
|
and language-specific config). Everything else goes in a subdirectory.
|
||||||
subdirectory names:
|
Canonical subdirectory names:
|
||||||
- `bin/` — executable scripts and tools
|
- `bin/` — executable scripts and tools
|
||||||
- `cmd/` — Go command entrypoints
|
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
|
||||||
|
body is a single call into `internal/` or `pkg/`, no project logic in
|
||||||
|
`cmd/`
|
||||||
- `configs/` — configuration templates and examples
|
- `configs/` — configuration templates and examples
|
||||||
- `deploy/` — deployment manifests (k8s, compose, terraform)
|
- `deploy/` — deployment manifests (k8s, compose, terraform)
|
||||||
- `docs/` — documentation and markdown (README.md stays in root)
|
- `docs/` — documentation and markdown (README.md stays in root)
|
||||||
@@ -414,3 +673,7 @@ style conventions are in separate documents:
|
|||||||
- Go: `go.mod`, `go.sum`, `.golangci.yml`
|
- Go: `go.mod`, `go.sum`, `.golangci.yml`
|
||||||
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
|
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
|
||||||
- Python: `pyproject.toml`
|
- Python: `pyproject.toml`
|
||||||
|
|
||||||
|
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
|
||||||
|
is never committed under a file or directory named after one agent tool, such
|
||||||
|
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
|
||||||
|
|||||||
@@ -12,24 +12,14 @@ One issue per unit of work, one branch and one PR per issue:
|
|||||||
`main` without notice. One `next` -> `main` PR accumulates the milestone;
|
`main` without notice. One `next` -> `main` PR accumulates the milestone;
|
||||||
releases are cut from `main` separately.
|
releases are cut from `main` separately.
|
||||||
|
|
||||||
Issue branches do NOT touch this file — the manager maintains it on `next`.
|
The tracker (https://git.eeqj.de/sneak/webhooker/issues) is the record of what
|
||||||
Every branch editing `TODO.md` conflicts with every other (#112).
|
is done and what is under way. This file keeps no list of either.
|
||||||
|
|
||||||
# Status
|
# Status
|
||||||
|
|
||||||
The milestone (https://git.eeqj.de/sneak/webhooker/milestone/9) is the
|
What is still open for 1.0.0 is in its milestone
|
||||||
authoritative list, and the only place to read a count or a state of play from.
|
(https://git.eeqj.de/sneak/webhooker/milestone/9). `v1.0.0` is not tagged, and
|
||||||
This file records where the project is, not what is in flight: a sentence whose
|
only @sneak tags it.
|
||||||
truth depends on a branch being unmerged is wrong the moment it merges, and this
|
|
||||||
file has been wrong that way before.
|
|
||||||
|
|
||||||
The durability defect that held the tag has landed
|
|
||||||
(https://git.eeqj.de/sneak/webhooker/issues/256, commit `8d64259`). Every SQLite
|
|
||||||
handle opens with WAL journaling and a busy timeout, a bookkeeping write that
|
|
||||||
fails leaves its delivery in a recoverable state rather than a lying one, and
|
|
||||||
recovery skips a delivery that already has a successful result row. Final
|
|
||||||
pre-tag verification exercised it and confirmed it holds. Whatever the milestone
|
|
||||||
still shows open is what remains before `v1.0.0`.
|
|
||||||
|
|
||||||
Delivery is at-least-once by design, not by accident: a send whose result row
|
Delivery is at-least-once by design, not by accident: a send whose result row
|
||||||
does not land is attempted again, so a receiver can see a duplicate. That is
|
does not land is attempted again, so a receiver can see a duplicate. That is
|
||||||
@@ -38,338 +28,31 @@ under Rationale. It is not a defect to re-file.
|
|||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
Clear the rest of the open 1.0.0 milestone
|
@sneak runs webhooker in limited production, then tags `v1.0.0`. Merging `next`
|
||||||
(https://git.eeqj.de/sneak/webhooker/milestone/9) and tag `v1.0.0`. Merging
|
into `main` is a separate act from tagging and waits on neither of those: `next`
|
||||||
`next` into `main` is a separate act from tagging and waits on neither of those:
|
is kept mergeable at all times, which is the point of the branch.
|
||||||
`next` is kept mergeable at all times, which is the point of the branch.
|
|
||||||
|
|
||||||
# Completed Steps
|
|
||||||
|
|
||||||
- 2026-08-24 Bind the plaintext HTTP listener deliberately, via `BIND_ADDRESS`
|
|
||||||
defaulting to `127.0.0.1`, and document the reverse-proxy deployment. A
|
|
||||||
hostname, an empty value or a value carrying a port is a startup error, and
|
|
||||||
the `Dockerfile` sets `0.0.0.0` because a loopback bind inside a container is
|
|
||||||
unreachable (https://git.eeqj.de/sneak/webhooker/issues/268). The same commit
|
|
||||||
removed the shutdown race: `httpServer` is built in the constructor rather
|
|
||||||
than assigned from the serving goroutine, which orders the write before every
|
|
||||||
fx hook and rules out the nil dereference a SIGTERM arriving first would have
|
|
||||||
caused, and `sentryEnabled` is an `atomic.Bool`
|
|
||||||
(https://git.eeqj.de/sneak/webhooker/issues/226)
|
|
||||||
- 2026-08-24 Remove inbound request signature verification. The entrypoint UUID
|
|
||||||
is the authentication secret, so the per-entrypoint shared secret, the
|
|
||||||
`internal/signature` package, the receiver check, the model fields and the
|
|
||||||
forms are all gone. This reverses the feature that landed earlier in the same
|
|
||||||
milestone (https://git.eeqj.de/sneak/webhooker/issues/67,
|
|
||||||
https://git.eeqj.de/sneak/webhooker/issues/279)
|
|
||||||
- 2026-08-24 Stamp the build version into the binary and render it in the UI
|
|
||||||
footer. `script/version` is the single source — `$VERSION`, else
|
|
||||||
`git describe --tags --always --dirty`, else `unknown` — so a `make build`
|
|
||||||
binary and a `make docker` image from one checkout report the same thing, and
|
|
||||||
nothing in it varies between two builds of the same commit, which the release
|
|
||||||
gate's byte-identical assertion would catch
|
|
||||||
(https://git.eeqj.de/sneak/webhooker/issues/253)
|
|
||||||
- 2026-08-24 Derive cookie `Secure` and CSRF strictness from the request
|
|
||||||
transport rather than from `WEBHOOKER_ENVIRONMENT`. Behind a real TLS proxy
|
|
||||||
with the environment left at its `dev` default, the session cookie silently
|
|
||||||
lost `Secure` while the CSRF cookie on the same response kept it.
|
|
||||||
`X-Forwarded-Proto` is now matched case-insensitively on its first
|
|
||||||
comma-separated element, so `HTTPS` and `https, http` no longer fall to the
|
|
||||||
relaxed CSRF path (https://git.eeqj.de/sneak/webhooker/issues/269)
|
|
||||||
- 2026-08-24 Roll back a failed webhook deletion instead of committing it. A
|
|
||||||
failing delete committed whatever had already succeeded, hard-deleted the
|
|
||||||
per-webhook event database anyway, and redirected as though it had worked —
|
|
||||||
orphaned config plus permanently destroyed history, reported as success. All
|
|
||||||
three delete positions now roll back with the event database intact
|
|
||||||
(https://git.eeqj.de/sneak/webhooker/issues/262)
|
|
||||||
- 2026-08-24 Name a deleted target on its historical deliveries, marked
|
|
||||||
`(deleted)`, rather than leaving the event log unable to say where a delivery
|
|
||||||
went. A deleted target's credentials stay masked exactly as a live one's, and
|
|
||||||
it cannot become deliverable again through the receiver, resubmit, replay, the
|
|
||||||
edit form or the toggle (https://git.eeqj.de/sneak/webhooker/issues/211)
|
|
||||||
- 2026-08-24 Bound both request-controlled `/metrics` label dimensions, so the
|
|
||||||
unauthenticated receiver is no longer a memory-exhaustion vector: `handler`
|
|
||||||
carries the chi route pattern, and `method` folds anything chi cannot route
|
|
||||||
onto a single `(unmatched)` sentinel. Both were reproduced before the fix —
|
|
||||||
300 random method tokens took the series count from 106 to 7,631, and path
|
|
||||||
flooding reached 62,532 — and a label audit across a live scrape found no
|
|
||||||
third unbounded dimension (https://git.eeqj.de/sneak/webhooker/issues/254,
|
|
||||||
https://git.eeqj.de/sneak/webhooker/issues/261)
|
|
||||||
- 2026-08-24 Validate `max_retries` on both target forms. `abc`, `2.7` and `-5`
|
|
||||||
silently became 0 — fire-and-forget — including on the edit path, where it
|
|
||||||
destroyed a working value, and `999999999` stored verbatim. The ceiling of 20
|
|
||||||
is the `max` both templates already declared
|
|
||||||
(https://git.eeqj.de/sneak/webhooker/issues/221)
|
|
||||||
- 2026-08-24 Resubmit a stored event as a new undelivered event, so a backend
|
|
||||||
under development can be tested against real captured traffic. Per-delivery
|
|
||||||
replay cannot serve that: it re-sends one finished delivery to its own
|
|
||||||
original target, and a target created for a dev backend has no prior delivery
|
|
||||||
to replay. Resubmit re-injects the stored event at the top of the receiver
|
|
||||||
path and fans it out to whatever targets are active now
|
|
||||||
(https://git.eeqj.de/sneak/webhooker/issues/250)
|
|
||||||
- 2026-08-20 Take an exclusive lock on `DATA_DIR` at startup, so two instances
|
|
||||||
on one directory cannot both deliver
|
|
||||||
(https://git.eeqj.de/sneak/webhooker/issues/201)
|
|
||||||
- 2026-08-20 Shut down the app when the HTTP listener fails. The `OnStart` hook
|
|
||||||
returned as soon as the serving goroutine was spawned, so a failed listen left
|
|
||||||
fx reporting RUNNING and a live process with nothing bound — invisible to
|
|
||||||
systemd and Docker restart policies
|
|
||||||
(https://git.eeqj.de/sneak/webhooker/issues/200)
|
|
||||||
- 2026-08-20 Stop target credentials leaking into the per-webhook event
|
|
||||||
databases (https://git.eeqj.de/sneak/webhooker/issues/206), log SQL with
|
|
||||||
placeholders rather than bound values
|
|
||||||
(https://git.eeqj.de/sneak/webhooker/issues/207), and fail loudly on half-set
|
|
||||||
metrics auth credentials (https://git.eeqj.de/sneak/webhooker/issues/205)
|
|
||||||
- 2026-08-20 Read queue depths with `Find`, not `Scan`. `Scan` swaps GORM's own
|
|
||||||
trace recorder in for the logging adapter, and that recorder does not
|
|
||||||
implement `gorm.ParamsFilter`, so those statements logged their bound values
|
|
||||||
interpolated and bypassed the suppression above. The two units gated green
|
|
||||||
against a `next` that lacked the other, and `next` went red when both landed
|
|
||||||
(https://git.eeqj.de/sneak/webhooker/issues/234)
|
|
||||||
- 2026-08-20 Render per-attempt delivery detail in the event log
|
|
||||||
(https://git.eeqj.de/sneak/webhooker/issues/202) and add replay of a
|
|
||||||
terminally failed delivery (https://git.eeqj.de/sneak/webhooker/issues/203)
|
|
||||||
- 2026-08-20 Expose delivery metrics on `/metrics`
|
|
||||||
(https://git.eeqj.de/sneak/webhooker/issues/209) and document the backup,
|
|
||||||
restore and upgrade procedures
|
|
||||||
(https://git.eeqj.de/sneak/webhooker/issues/210)
|
|
||||||
- 2026-08-20 Add a `webhooker resetpw` subcommand and a bootstrap banner. The
|
|
||||||
admin bootstrap password was printed once among roughly 45 fx lines, and under
|
|
||||||
`docker run -d` went to container logs subject to rotation; there was no reset
|
|
||||||
path at all, so recovery meant hand-deleting the users row, documented
|
|
||||||
nowhere. The password is read from stdin or generated, never from argv where
|
|
||||||
`/proc` would publish it (https://git.eeqj.de/sneak/webhooker/issues/208)
|
|
||||||
- 2026-08-20 Add `ALLOWED_EGRESS_CIDRS`, an allowlist-only escape hatch for the
|
|
||||||
SSRF guard, so a self-hosted proxy can forward into the operator's own
|
|
||||||
network. The guard's always-blocked set cannot be reopened by configuration
|
|
||||||
(https://git.eeqj.de/sneak/webhooker/issues/204)
|
|
||||||
- 2026-08-20 Harden operator-set target headers, which were carried unsafely
|
|
||||||
across a redirect (https://git.eeqj.de/sneak/webhooker/issues/233)
|
|
||||||
- 2026-08-20 Add a target edit form with headers and timeout fields
|
|
||||||
(https://git.eeqj.de/sneak/webhooker/issues/127)
|
|
||||||
- 2026-08-18 Raise `script/test`'s per-package timeout from 30s to 90s, matching
|
|
||||||
the org-wide backstop. `go test` applies `-timeout` per package, and
|
|
||||||
`internal/handlers` had grown past the old budget: a cache-defeated build
|
|
||||||
failed outright at `GOMAXPROCS=4`, and every run under deliberate host load
|
|
||||||
breached 30s. The measurement table lives in the script (#194)
|
|
||||||
- 2026-08-18 Re-sync `REPO_POLICIES.md` from `prompts`. The local copy was stale
|
|
||||||
and still mandated a 20s test target with a 30s timeout, which the org
|
|
||||||
replaced with a 60s cap and a 90s backstop. A synced copy is not a source;
|
|
||||||
reading it as one nearly produced a PR against `prompts` proposing a change
|
|
||||||
already merged there (#196)
|
|
||||||
- 2026-08-18 Report handler panics through the logger and answer 500. chi
|
|
||||||
v1.5.5's `Recoverer` scans for a `panic(0x` frame the runtime no longer emits,
|
|
||||||
then indexes `pkg[-1:]`, so it panicked inside its own stack printer before
|
|
||||||
writing a byte: the recovery never ran, the client got a dropped connection
|
|
||||||
instead of a 500, and the original panic was lost. A local middleware replaces
|
|
||||||
it, bounded by `MaxPanicLogLineBytes` (#187)
|
|
||||||
- 2026-08-18 Route GORM's logger through `slog` and bound it. Every `gorm.Open`
|
|
||||||
left `logger.Default` in place at `Warn` with `IgnoreRecordNotFoundError`
|
|
||||||
false, so **every record-not-found printed the fully interpolated SQL to
|
|
||||||
stdout** — including the client-chosen path on `/webhook/{uuid}` and the
|
|
||||||
submitted username on the login form, at no level the operator set and outside
|
|
||||||
`internal/logger` entirely. Three call sites, not the two the issue named
|
|
||||||
(#178)
|
|
||||||
- 2026-08-18 Bound every `slog` line against client-chosen text. Eight sites
|
|
||||||
reachable unauthenticated, found by reading every `slog` call in the tree
|
|
||||||
rather than only the one reported; the budget moved to a shared
|
|
||||||
`internal/logfield` so no second truncation exists. `DEBUG` being off by
|
|
||||||
default is not a bound and is not treated as one (#176)
|
|
||||||
- 2026-08-18 Stop a slow host turning a login-guard test into a segfault. A
|
|
||||||
non-fatal `assert` on an acquire result was dereferenced on the next line, so
|
|
||||||
one timing miss killed the whole `internal/middleware` binary and reddened CI
|
|
||||||
for unrelated PRs. The fix also removed a real production race — `acquire`
|
|
||||||
could shed a request with a slot standing free, because Go picks uniformly
|
|
||||||
among ready `select` cases (#186)
|
|
||||||
- 2026-08-18 Send the chi route pattern to Sentry rather than the concrete path.
|
|
||||||
The receiver's path carries the entrypoint capability token, so every Sentry
|
|
||||||
event from `/webhook/{uuid}` shipped a live credential to a third party.
|
|
||||||
Request `Data`, `QueryString`, `Cookies` and `Env` are dropped and headers
|
|
||||||
reduced to an allowlist (#179)
|
|
||||||
- 2026-08-18 Read form fields from the POST body only. `r.FormValue` merges the
|
|
||||||
query string, so a login could be driven by URL parameters — putting the
|
|
||||||
password somewhere that lands in access logs, proxy logs and browser history
|
|
||||||
(#160)
|
|
||||||
- 2026-08-18 Verify login credentials before spending rate-limit budget, so a
|
|
||||||
flood of wrong passwords cannot lock out the account it is guessing at. The
|
|
||||||
manager took this decision rather than stall the queue; it is flagged on the
|
|
||||||
issue for reversal (#150)
|
|
||||||
- 2026-08-18 Run all linting in Docker via `Dockerfile.lint`. Host lint was
|
|
||||||
wrong in both directions from version skew and shared caches. `script/lint`
|
|
||||||
asserts the summary line, because `--no-cache-filter` silently ignores a stage
|
|
||||||
name it does not match — the flag that makes the gate meaningful fails open
|
|
||||||
(#109)
|
|
||||||
- 2026-08-18 Serve an event's full stored body over HTTP. The list query
|
|
||||||
truncates for rendering, and that truncated value was the only way to read a
|
|
||||||
body, so the full payload was unreachable (#157)
|
|
||||||
- 2026-08-18 Bound the access log line against client-chosen text.
|
|
||||||
`internal/logfield` budgets by _encoded_ bytes, not runes, so a handler's JSON
|
|
||||||
escaping cannot multiply a field past its allowance (#146)
|
|
||||||
- 2026-08-18 Mark superseded CI commits `failure` rather than `skipped`. A
|
|
||||||
skipped run rolls up green, so a commit that was never tested reported success
|
|
||||||
(#152)
|
|
||||||
- 2026-08-18 Set `fx.StopTimeout` inside the container stop grace, so shutdown
|
|
||||||
hooks are bounded by a deadline the orchestrator will actually honour rather
|
|
||||||
than being killed mid-flush (#134)
|
|
||||||
- 2026-08-17 Bucket IPv6 rate-limit keys by `/64`. A single allocation hands out
|
|
||||||
2^64 addresses, so per-address keying let one client mint unlimited buckets.
|
|
||||||
Manager decision, recorded on the issue (#125)
|
|
||||||
- 2026-08-17 Correct release-blocking README and startup-warning inaccuracies,
|
|
||||||
including claims about behaviour the code does not have (#151)
|
|
||||||
- 2026-08-17 Fetch and verify Alpine.js at build time against
|
|
||||||
`static/vendor.sha256` instead of committing the minified blob, so the
|
|
||||||
dependency is pinned by hash rather than by trust (#145)
|
|
||||||
- 2026-08-17 Bound the event log's rendered bodies in the query itself, so a
|
|
||||||
large stored payload cannot be read into memory just to be truncated for
|
|
||||||
display (#135)
|
|
||||||
- 2026-08-17 Mask the `http` target's destination URL in the UI: it can carry a
|
|
||||||
bearer credential in its path or query, and was rendered verbatim. Manager
|
|
||||||
decision to mask unconditionally (#115)
|
|
||||||
- 2026-08-14 Bound shutdown hooks by their stop context, so a hook that hangs
|
|
||||||
cannot hold the process past its grace period (#102)
|
|
||||||
- 2026-08-14 Render templates via a buffer rather than the `ResponseWriter`, so
|
|
||||||
a template error part-way through cannot commit a 200 and then fail — the
|
|
||||||
response is written only once it is whole (#123)
|
|
||||||
- 2026-08-14 Align the session codec's max-age with the 7-day absolute cap. The
|
|
||||||
codec accepted cookies the session layer considered expired, so the cap was
|
|
||||||
enforced in one place and not the other (#108)
|
|
||||||
- 2026-08-12 Warn when `TRUSTED_PROXIES` is empty in production, where the safe
|
|
||||||
default silently discards forwarded headers and every client rate-limits as
|
|
||||||
the proxy's address (#149)
|
|
||||||
- 2026-08-12 Bound the receiver rate limit per client IP across the whole
|
|
||||||
`/webhook/*` route. The existing limiter keyed on the request path and
|
|
||||||
`/webhook/{uuid}` matches any single segment, so a client that invented a
|
|
||||||
fresh path per request minted a fresh bucket per request: the limit on the
|
|
||||||
only unauthenticated endpoint bounded nothing in aggregate, and every request
|
|
||||||
still cost an entrypoint lookup before it 404ed. An outer limiter keyed on the
|
|
||||||
client address alone now bounds that, chained in front of the unchanged
|
|
||||||
per-entrypoint limiter (#139)
|
|
||||||
- 2026-08-12 Correct release-blocking documentation inaccuracies: the README
|
|
||||||
promised manual redelivery in the present tense in three places when nothing
|
|
||||||
implements it (the same false claim also sat in the doc comment that was its
|
|
||||||
source text), the env table omitted `RETENTION_SWEEP_INTERVAL`, and `TODO.md`
|
|
||||||
itself omitted five landed units (#141)
|
|
||||||
- 2026-08-12 Make the CI gate execute the checks it reports on. The workflow now
|
|
||||||
writes a build-context fingerprint before calling `script/cibuild`, so a code
|
|
||||||
commit invalidates the `COPY` layer of the lint and builder stages while a
|
|
||||||
docs-only commit still replays from cache; a superseding run also rewrites the
|
|
||||||
`failure` status Gitea leaves on commits it cancelled and never tested.
|
|
||||||
Verified by pushing a deliberately broken test and watching CI go red (#119)
|
|
||||||
- 2026-08-12 Require a positive `RETENTION_SWEEP_INTERVAL`: a non-positive value
|
|
||||||
reached `time.NewTicker` in both the retention reaper and the archive sweeper,
|
|
||||||
panicking two goroutines with no recover after startup had already reported
|
|
||||||
success (#140)
|
|
||||||
- 2026-08-12 Bound the `X-Forwarded-For` scan's allocation to the hop cap: the
|
|
||||||
reverse walk cuts entries with `strings.LastIndexByte` instead of joining and
|
|
||||||
splitting, so a 1 MB header allocates 16 bytes rather than 1.6 MB per request
|
|
||||||
on the unauthenticated receiver. Semantics proven unchanged by differential
|
|
||||||
testing against the previous implementation (#133)
|
|
||||||
- 2026-08-12 Cap the `X-Forwarded-For` hop walk at 64 entries, so an
|
|
||||||
attacker-supplied chain cannot burn unbounded CPU in the rate-limit key
|
|
||||||
function; running off the end falls back to the peer address (#124)
|
|
||||||
- 2026-08-12 Gate forwarded-header trust behind a `TRUSTED_PROXIES` CIDR list:
|
|
||||||
all three rate limiters key on the connection's own address unless the direct
|
|
||||||
peer is a configured proxy, in which case `X-Forwarded-For` is walked right to
|
|
||||||
left for the first non-proxy hop. Default trusts nothing, and a
|
|
||||||
set-but-unparseable value aborts startup. Before this, any client could mint a
|
|
||||||
fresh bucket or drain another's by rotating a spoofed header (#88)
|
|
||||||
- 2026-08-11 Web UI cleanup: nav terminology unified on Webhooks, the Profile
|
|
||||||
settings placeholder removed, a progressive-enhancement copy button for the
|
|
||||||
entrypoint URL, and retention form copy that states the actual policy
|
|
||||||
(deletion by the reaper, 0 retains forever) (#57)
|
|
||||||
- 2026-08-11 Mask the webhook credential in delivery errors and logs: Go embeds
|
|
||||||
the request URL in `*url.Error`, so every transport failure persisted the full
|
|
||||||
Slack webhook URL into the per-webhook event database via
|
|
||||||
`DeliveryResult.Error`, a field a future REST API would have served.
|
|
||||||
`maskURLError` drops path, query and userinfo while preserving the wrapped
|
|
||||||
cause, so `errors.Is`/`As` and `Timeout()` still work and DNS, TLS and timeout
|
|
||||||
failures still read differently (#118)
|
|
||||||
- 2026-08-11 Rate-limit the public webhook receiver endpoint
|
|
||||||
(`RECEIVER_RATE_LIMIT`, default 120/min), keyed on client IP plus entrypoint
|
|
||||||
path so one entrypoint cannot exhaust another's budget; over-limit requests
|
|
||||||
get 429 with `Retry-After`. It was the one unauthenticated, internet-facing
|
|
||||||
endpoint with no limit at all (#64)
|
|
||||||
- 2026-08-11 Enforce the body size limit before CSRF parses the form:
|
|
||||||
`MaxBodySize` is now first in all four form-parsing route groups, so an
|
|
||||||
oversized request is rejected with 413 instead of being read in full by the
|
|
||||||
CSRF middleware before any cap applied (#90)
|
|
||||||
- 2026-08-11 Mask target config on the source detail page, which rendered the
|
|
||||||
stored blob verbatim and so exposed the Slack incoming-webhook URL — a bearer
|
|
||||||
credential that cannot be revoked per-holder. Config reaches the template only
|
|
||||||
as a `TargetView` of labelled fields, and header values are rendered as a
|
|
||||||
count (#113)
|
|
||||||
- 2026-08-11 Allow `retention_days` of 0 to mean retain forever, via a sentinel
|
|
||||||
written in `BeforeSave` so the GORM column default cannot win the race. Also
|
|
||||||
bounds the reaper's cutoff arithmetic: day counts above 106751 overflowed
|
|
||||||
`time.Duration` and wrapped the cutoff into the future, where every row
|
|
||||||
matched and the sweep deleted everything (#79)
|
|
||||||
- 2026-08-09 Inactivity-based session timeout: sliding idle expiry
|
|
||||||
(`SESSION_IDLE_TIMEOUT`, default `24h`) refreshed on authenticated requests,
|
|
||||||
with the 7-day absolute cap kept as an independent backstop that activity
|
|
||||||
never extends (#66)
|
|
||||||
- 2026-08-09 Restart recovery and the 60s retry sweep terminally fail an
|
|
||||||
orphaned `retrying` delivery whose target type no longer supports retries,
|
|
||||||
recording a `DeliveryResult` with the reason instead of leaving the delivery
|
|
||||||
stuck forever (#82)
|
|
||||||
- 2026-08-09 Root the delivery engine's worker pool and the retention reaper's
|
|
||||||
sweep loop at `context.Background()` rather than the fx `OnStart` hook context
|
|
||||||
(#97), which carries fx's 15s start timeout and killed both roughly fifteen
|
|
||||||
seconds after boot: the proxy silently stopped delivering webhooks entirely,
|
|
||||||
and the reaper never ran a single sweep under its default one-hour interval
|
|
||||||
- 2026-08-09 Archive writer lifecycle (#89): deleting a webhook (or its last
|
|
||||||
`database` target) evicts the cached archive writer and closes its handle
|
|
||||||
while deliberately leaving `archive-{webhookID}.db` on disk, and a new
|
|
||||||
`ArchiveSweeper` prunes idle archives on the existing
|
|
||||||
`RETENTION_SWEEP_INTERVAL` without ever creating an archive file
|
|
||||||
- 2026-08-09 Configuration parsing fails loudly on set-but-unparseable
|
|
||||||
environment values: `envInt` removed in favour of `envPositiveInt` plus a
|
|
||||||
`PORT` range check, `envBool` now parses with `strconv.ParseBool`, and
|
|
||||||
defaults apply only to unset variables (#80)
|
|
||||||
- 2026-08-07 Automatic event retention cleanup based on `retention_days`,
|
|
||||||
deleting expired events, deliveries, and delivery results from each
|
|
||||||
per-webhook event database (#63)
|
|
||||||
- 2026-08-07 Update golangci-lint to v2.12.2 (Docker image digest in
|
|
||||||
`Dockerfile`, release-archive sha256 pins in `script/bootstrap`), adopt the
|
|
||||||
canonical `.golangci.yml` (v2 `linters.settings` layout so
|
|
||||||
`lll`/`funlen`/`cyclop`/`dupl` thresholds actually apply), and fix all newly
|
|
||||||
surfaced lint findings
|
|
||||||
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
|
|
||||||
shims, README Entrypoints section
|
|
||||||
- 2026-03-25 pin golangci-lint Docker image for linting (#55)
|
|
||||||
- 2026-03-18 CSRF middleware detects TLS per-request, fixing login over plain
|
|
||||||
HTTP and behind reverse proxies (#54)
|
|
||||||
- 2026-03-17 root path redirects based on auth state (#52)
|
|
||||||
- 2026-03-17 CSRF protection, SSRF prevention for HTTP delivery targets with DNS
|
|
||||||
rebinding defense, and per-IP login rate limiting (#42)
|
|
||||||
- 2026-03-17 Slack target type for incoming webhook notifications (#47)
|
|
||||||
- 2026-03-17 Dockerfile absolute paths and static linking (#49); absolute dev
|
|
||||||
DATA_DIR default and clarified env docs (#46)
|
|
||||||
- 2026-03-05 security headers middleware, session regeneration on login, request
|
|
||||||
body size limits (#41)
|
|
||||||
- 2026-03-04 tests for delivery, middleware, and session packages (#32); removed
|
|
||||||
the build-architecture global (#31)
|
|
||||||
- 2026-03-04 1.0 MVP merge: Webhook/Entrypoint/Target rename, core delivery
|
|
||||||
engine with bounded worker pool and circuit breaker, parallel fan-out,
|
|
||||||
per-webhook event databases, management UI (#16)
|
|
||||||
- 2026-03-01 repo brought to REPO_POLICIES standards; TODO.md folded into README
|
|
||||||
(#6)
|
|
||||||
|
|
||||||
# Future Steps
|
# Future Steps
|
||||||
|
|
||||||
- Delivery status and retry management UI. Replay of a terminally failed
|
- Per-webhook rate limits, set in the web UI and enforced in the receiver
|
||||||
delivery and per-attempt detail already landed
|
handler, on top of the receiver's existing limits, which `RECEIVER_RATE_LIMIT`
|
||||||
(https://git.eeqj.de/sneak/webhooker/issues/203,
|
sets for the whole server (see Rate Limiting in the README)
|
||||||
https://git.eeqj.de/sneak/webhooker/issues/202)
|
- API key authentication for programmatic access (the `APIKey` model exists, but
|
||||||
- Per-webhook rate limiting in the receiver handler (per-webhook config plus
|
nothing creates a key and no middleware checks one)
|
||||||
handler enforcement; global limits must not apply to receiver endpoints)
|
|
||||||
- API key authentication for programmatic access (APIKey model exists; Bearer
|
|
||||||
token middleware does not)
|
|
||||||
- REST API v1
|
- REST API v1
|
||||||
- CRUD for webhooks, entrypoints, targets
|
- CRUD for webhooks, entrypoints, targets
|
||||||
- event viewing and filtering endpoints
|
- event viewing and filtering endpoints
|
||||||
- event redelivery endpoint
|
- event redelivery endpoint
|
||||||
- OpenAPI specification
|
- OpenAPI specification
|
||||||
- Analytics dashboard: success rates, response times, volume
|
- Analytics in the web UI: response times summarised per webhook or per target,
|
||||||
- A remember-me option at login
|
and figures for all webhooks together. The webhook page already shows event,
|
||||||
|
delivery and failure counts and failure percentages, the target list counts
|
||||||
|
each target's deliveries, the event log and each event's page show every
|
||||||
|
attempt's duration, and `/metrics` carries server-wide counts and delivery
|
||||||
|
durations
|
||||||
|
- A remember-me option at login. Every session already survives a browser
|
||||||
|
restart: it lasts up to 7 days, and ends sooner after `SESSION_IDLE_TIMEOUT`
|
||||||
|
(default 24 hours) without activity
|
||||||
- Password reset flow for a forgotten password over the web. The authenticated
|
- Password reset flow for a forgotten password over the web. The authenticated
|
||||||
password _change_ flow already landed, and a lost password is recoverable from
|
password _change_ flow already landed, and a lost password is recoverable from
|
||||||
the console with `webhooker resetpw`
|
the console with `webhooker resetpw`
|
||||||
@@ -377,7 +60,13 @@ Clear the rest of the open 1.0.0 milestone
|
|||||||
- Later, nice to have
|
- Later, nice to have
|
||||||
- email delivery target type
|
- email delivery target type
|
||||||
- SNS and S3 delivery targets
|
- SNS and S3 delivery targets
|
||||||
- data transformations (e.g. webhook to Slack message formatting)
|
- data transformations (e.g. a configurable Slack message format; the Slack
|
||||||
- JSONL file delivery with periodic S3 upload
|
target sends one fixed format today)
|
||||||
- webhook event search and filtering
|
- JSONL file delivery with periodic S3 upload (the `database` target already
|
||||||
- multi-user with role-based access control
|
archives events to SQLite files, rotated hourly, daily or monthly if set,
|
||||||
|
and the webhook page offers them as one gzipped JSON download)
|
||||||
|
- webhook event search, and filtering beyond the event log's failed and
|
||||||
|
pending views
|
||||||
|
- multi-user with role-based access control (each webhook already belongs to
|
||||||
|
one account, but there is only the account made at first boot and no way
|
||||||
|
to add another)
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ package main
|
|||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
|
"log/slog"
|
||||||
"os"
|
"os"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -187,6 +188,9 @@ func newApp() *fx.App {
|
|||||||
fx.Provide(
|
fx.Provide(
|
||||||
globals.New,
|
globals.New,
|
||||||
logger.New,
|
logger.New,
|
||||||
|
// The plain logger the session, the middleware and the
|
||||||
|
// webhook database manager take.
|
||||||
|
func(l *logger.Logger) *slog.Logger { return l.Get() },
|
||||||
config.New,
|
config.New,
|
||||||
database.New,
|
database.New,
|
||||||
database.NewWebhookDBManager,
|
database.NewWebhookDBManager,
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ import (
|
|||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config/configtest"
|
||||||
"sneak.berlin/go/webhooker/internal/datadir"
|
"sneak.berlin/go/webhooker/internal/datadir"
|
||||||
"sneak.berlin/go/webhooker/internal/resetpw"
|
"sneak.berlin/go/webhooker/internal/resetpw"
|
||||||
"sneak.berlin/go/webhooker/internal/server"
|
"sneak.berlin/go/webhooker/internal/server"
|
||||||
@@ -37,7 +37,7 @@ const dockerStopGrace = 10 * time.Second
|
|||||||
// fx.New applies options before it executes invokes, so the timeout
|
// fx.New applies options before it executes invokes, so the timeout
|
||||||
// is set whether or not the graph itself can be constructed here.
|
// is set whether or not the graph itself can be constructed here.
|
||||||
func TestNewApp_StopTimeout(t *testing.T) {
|
func TestNewApp_StopTimeout(t *testing.T) {
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
t.Setenv("DATA_DIR", t.TempDir())
|
t.Setenv("DATA_DIR", t.TempDir())
|
||||||
|
|
||||||
got := newApp().StopTimeout()
|
got := newApp().StopTimeout()
|
||||||
@@ -75,7 +75,7 @@ func freePort(t *testing.T) int {
|
|||||||
// anything is built, and the run of logger.New, which happens before
|
// anything is built, and the run of logger.New, which happens before
|
||||||
// the configuration sets the level.
|
// the configuration sets the level.
|
||||||
func TestNewApp_SendsFxEventsToTheLogger(t *testing.T) {
|
func TestNewApp_SendsFxEventsToTheLogger(t *testing.T) {
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
t.Setenv("DATA_DIR", t.TempDir())
|
t.Setenv("DATA_DIR", t.TempDir())
|
||||||
t.Setenv("PORT", strconv.Itoa(freePort(t)))
|
t.Setenv("PORT", strconv.Itoa(freePort(t)))
|
||||||
t.Setenv("DEBUG", "true")
|
t.Setenv("DEBUG", "true")
|
||||||
|
|||||||
@@ -22,7 +22,6 @@ require (
|
|||||||
github.com/stretchr/testify v1.11.1
|
github.com/stretchr/testify v1.11.1
|
||||||
go.uber.org/fx v1.24.0
|
go.uber.org/fx v1.24.0
|
||||||
golang.org/x/crypto v0.38.0
|
golang.org/x/crypto v0.38.0
|
||||||
gopkg.in/yaml.v3 v3.0.1
|
|
||||||
gorm.io/driver/sqlite v1.5.4
|
gorm.io/driver/sqlite v1.5.4
|
||||||
gorm.io/gorm v1.25.5
|
gorm.io/gorm v1.25.5
|
||||||
modernc.org/sqlite v1.28.0
|
modernc.org/sqlite v1.28.0
|
||||||
@@ -59,6 +58,7 @@ require (
|
|||||||
golang.org/x/text v0.25.0 // indirect
|
golang.org/x/text v0.25.0 // indirect
|
||||||
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
|
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
|
||||||
google.golang.org/protobuf v1.31.0 // indirect
|
google.golang.org/protobuf v1.31.0 // indirect
|
||||||
|
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||||
lukechampine.com/uint128 v1.2.0 // indirect
|
lukechampine.com/uint128 v1.2.0 // indirect
|
||||||
modernc.org/cc/v3 v3.40.0 // indirect
|
modernc.org/cc/v3 v3.40.0 // indirect
|
||||||
modernc.org/ccgo/v3 v3.16.13 // indirect
|
modernc.org/ccgo/v3 v3.16.13 // indirect
|
||||||
|
|||||||
@@ -1,387 +0,0 @@
|
|||||||
package ciscript_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"maps"
|
|
||||||
"os"
|
|
||||||
"os/exec"
|
|
||||||
"path/filepath"
|
|
||||||
"slices"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
"gopkg.in/yaml.v3"
|
|
||||||
)
|
|
||||||
|
|
||||||
const (
|
|
||||||
// supersededDesc is the description script/ci-mark-superseded
|
|
||||||
// writes, and the one an earlier revision of it wrote alongside a
|
|
||||||
// `skipped` state.
|
|
||||||
supersededDesc = "Superseded by a newer commit; never tested"
|
|
||||||
|
|
||||||
// liveContext is the commit-status context Gitea uses for this
|
|
||||||
// repository's runs, as seen in its API. The script derives it from
|
|
||||||
// the workflow and job names rather than hardcoding it; the
|
|
||||||
// derivation is checked against this value below.
|
|
||||||
liveContext = "check / check (push)"
|
|
||||||
|
|
||||||
scriptPath = "../../script/ci-mark-superseded"
|
|
||||||
workflow = "../../.gitea/workflows/check.yml"
|
|
||||||
|
|
||||||
// failure is the only state that neither folds into a combined
|
|
||||||
// `success` (as `skipped` does) nor blocks the commit forever (as
|
|
||||||
// `pending` does).
|
|
||||||
failure = "failure"
|
|
||||||
)
|
|
||||||
|
|
||||||
// repo is a throwaway git history: parent is the commit a run would be
|
|
||||||
// cancelled on, head the commit that superseded it.
|
|
||||||
type repo struct {
|
|
||||||
dir string
|
|
||||||
head string
|
|
||||||
parent string
|
|
||||||
}
|
|
||||||
|
|
||||||
// scriptEnv is the run identity the Gitea runner exports and the script
|
|
||||||
// builds its context string from.
|
|
||||||
type scriptEnv struct {
|
|
||||||
workflow string
|
|
||||||
job string
|
|
||||||
event string
|
|
||||||
}
|
|
||||||
|
|
||||||
func defaultEnv() scriptEnv {
|
|
||||||
return scriptEnv{workflow: "check", job: "check", event: "push"}
|
|
||||||
}
|
|
||||||
|
|
||||||
func cancelled() commitStatus {
|
|
||||||
return commitStatus{
|
|
||||||
Context: liveContext,
|
|
||||||
Status: failure,
|
|
||||||
Description: "Has been cancelled",
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func running() commitStatus {
|
|
||||||
return commitStatus{
|
|
||||||
Context: liveContext,
|
|
||||||
Status: "pending",
|
|
||||||
Description: "Has started running",
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestMarkSuperseded(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cases := map[string]struct {
|
|
||||||
parent commitStatus
|
|
||||||
wantMark bool
|
|
||||||
}{
|
|
||||||
"a cancelled run is marked": {
|
|
||||||
parent: cancelled(),
|
|
||||||
wantMark: true,
|
|
||||||
},
|
|
||||||
"a laundered skipped status is marked": {
|
|
||||||
parent: commitStatus{
|
|
||||||
Context: liveContext,
|
|
||||||
Status: "skipped",
|
|
||||||
Description: supersededDesc,
|
|
||||||
},
|
|
||||||
wantMark: true,
|
|
||||||
},
|
|
||||||
"a genuine failure is left alone": {
|
|
||||||
parent: commitStatus{
|
|
||||||
Context: liveContext,
|
|
||||||
Status: failure,
|
|
||||||
Description: "Failing after 3m1s",
|
|
||||||
},
|
|
||||||
wantMark: false,
|
|
||||||
},
|
|
||||||
"a passing run is left alone": {
|
|
||||||
parent: commitStatus{
|
|
||||||
Context: liveContext,
|
|
||||||
Status: "success",
|
|
||||||
Description: "Successful in 2m52s",
|
|
||||||
},
|
|
||||||
wantMark: false,
|
|
||||||
},
|
|
||||||
"another context is left alone": {
|
|
||||||
parent: commitStatus{
|
|
||||||
Context: "other / other (push)",
|
|
||||||
Status: failure,
|
|
||||||
Description: "Has been cancelled",
|
|
||||||
},
|
|
||||||
wantMark: false,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for name, tc := range cases {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
requireTools(t)
|
|
||||||
|
|
||||||
history := newRepo(t)
|
|
||||||
fake, api := newFakeGitea(t)
|
|
||||||
fake.setStatus(history.head, running())
|
|
||||||
fake.setStatus(history.parent, tc.parent)
|
|
||||||
|
|
||||||
out, err := runScript(t, history, api, defaultEnv())
|
|
||||||
require.NoError(t, err, out)
|
|
||||||
|
|
||||||
posted := fake.postedFor(history.parent)
|
|
||||||
if !tc.wantMark {
|
|
||||||
require.Empty(t, posted)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
require.Equal(t, []postedStatus{{
|
|
||||||
Context: liveContext,
|
|
||||||
// Not `skipped`: Gitea's combined status folds
|
|
||||||
// that into `success`, which is what made a
|
|
||||||
// never-tested commit read green.
|
|
||||||
State: failure,
|
|
||||||
Description: supersededDesc,
|
|
||||||
}}, posted)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// A second run must not rewrite what the first one wrote, or every
|
|
||||||
// later push would post a duplicate status.
|
|
||||||
func TestMarkSupersededIsIdempotent(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
requireTools(t)
|
|
||||||
|
|
||||||
history := newRepo(t)
|
|
||||||
fake, api := newFakeGitea(t)
|
|
||||||
fake.setStatus(history.head, running())
|
|
||||||
fake.setStatus(history.parent, cancelled())
|
|
||||||
|
|
||||||
for range 2 {
|
|
||||||
out, err := runScript(t, history, api, defaultEnv())
|
|
||||||
require.NoError(t, err, out)
|
|
||||||
}
|
|
||||||
|
|
||||||
require.Len(t, fake.postedFor(history.parent), 1)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Renaming the workflow or the job changes the context string Gitea
|
|
||||||
// uses. The script must say so instead of quietly matching nothing.
|
|
||||||
func TestMarkSupersededRejectsAnUnknownContext(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
requireTools(t)
|
|
||||||
|
|
||||||
history := newRepo(t)
|
|
||||||
fake, api := newFakeGitea(t)
|
|
||||||
fake.setStatus(history.head, running())
|
|
||||||
fake.setStatus(history.parent, cancelled())
|
|
||||||
|
|
||||||
env := defaultEnv()
|
|
||||||
env.job = "renamed"
|
|
||||||
|
|
||||||
out, err := runScript(t, history, api, env)
|
|
||||||
require.Error(t, err)
|
|
||||||
require.Contains(t, out, "renamed")
|
|
||||||
require.Contains(t, out, liveContext)
|
|
||||||
require.Empty(t, fake.postedFor(history.parent))
|
|
||||||
}
|
|
||||||
|
|
||||||
// ANCESTOR_LIMIT is a documented knob. A value that is set but unusable
|
|
||||||
// must abort: handing it to git and discarding the exit status left the
|
|
||||||
// walk empty and the step green, marking nothing.
|
|
||||||
func TestMarkSupersededRejectsAnUnparseableAncestorLimit(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
requireTools(t)
|
|
||||||
|
|
||||||
history := newRepo(t)
|
|
||||||
fake, api := newFakeGitea(t)
|
|
||||||
fake.setStatus(history.head, running())
|
|
||||||
fake.setStatus(history.parent, cancelled())
|
|
||||||
|
|
||||||
out, err := runScript(
|
|
||||||
t, history, api, defaultEnv(), "ANCESTOR_LIMIT=twenty",
|
|
||||||
)
|
|
||||||
require.Error(t, err)
|
|
||||||
require.Contains(t, out, "ANCESTOR_LIMIT")
|
|
||||||
require.Contains(t, out, "twenty")
|
|
||||||
require.Empty(t, fake.postedFor(history.parent))
|
|
||||||
}
|
|
||||||
|
|
||||||
// A status read that fails is not the same as a commit with nothing to
|
|
||||||
// do. Losing curl's exit status through a pipe made the two identical
|
|
||||||
// and left a laundered commit laundered with no signal.
|
|
||||||
func TestMarkSupersededFailsOnAnUnreadableAncestorStatus(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
requireTools(t)
|
|
||||||
|
|
||||||
history := newRepo(t)
|
|
||||||
fake, api := newFakeGitea(t)
|
|
||||||
fake.setStatus(history.head, running())
|
|
||||||
fake.setStatus(history.parent, cancelled())
|
|
||||||
fake.failStatusRead(history.parent)
|
|
||||||
|
|
||||||
out, err := runScript(t, history, api, defaultEnv())
|
|
||||||
require.Error(t, err)
|
|
||||||
require.Contains(t, out, history.parent)
|
|
||||||
require.Contains(t, out, "cannot read commit statuses")
|
|
||||||
require.Empty(t, fake.postedFor(history.parent))
|
|
||||||
}
|
|
||||||
|
|
||||||
// A shallow clone cannot resolve the parent, so it is indistinguishable
|
|
||||||
// from a root commit to rev-parse and the walk would exit 0 having
|
|
||||||
// marked nothing. It must abort instead: dropping `fetch-depth: 0` from
|
|
||||||
// the checkout step is one edit, and a silent no-op there restores the
|
|
||||||
// false-green bug this script exists to prevent.
|
|
||||||
func TestMarkSupersededRejectsAShallowRepository(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
requireTools(t)
|
|
||||||
|
|
||||||
history := shallowClone(t, newRepo(t))
|
|
||||||
fake, api := newFakeGitea(t)
|
|
||||||
fake.setStatus(history.head, running())
|
|
||||||
fake.setStatus(history.parent, cancelled())
|
|
||||||
|
|
||||||
out, err := runScript(t, history, api, defaultEnv())
|
|
||||||
require.Error(t, err)
|
|
||||||
require.Contains(t, out, "shallow repository")
|
|
||||||
require.Empty(t, fake.postedFor(history.parent))
|
|
||||||
require.Empty(t, fake.postedFor(history.head))
|
|
||||||
}
|
|
||||||
|
|
||||||
// shallowClone returns the same history as a depth-1 clone. The `file://`
|
|
||||||
// URL is required: git ignores --depth for a plain local path.
|
|
||||||
func shallowClone(t *testing.T, history repo) repo {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
|
|
||||||
//nolint:gosec // fixed argv, arguments are test-local paths
|
|
||||||
cmd := exec.CommandContext(t.Context(), "git", "clone", "-q",
|
|
||||||
"--depth=1", "file://"+history.dir, dir)
|
|
||||||
|
|
||||||
out, err := cmd.CombinedOutput()
|
|
||||||
require.NoError(t, err, string(out))
|
|
||||||
|
|
||||||
return repo{dir: dir, head: history.head, parent: history.parent}
|
|
||||||
}
|
|
||||||
|
|
||||||
// The derived context must equal the one Gitea actually uses, which is
|
|
||||||
// built from the same workflow and job names.
|
|
||||||
func TestDerivedContextMatchesGitea(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
requireTools(t)
|
|
||||||
|
|
||||||
name, job := workflowIdentity(t)
|
|
||||||
|
|
||||||
history := newRepo(t)
|
|
||||||
fake, api := newFakeGitea(t)
|
|
||||||
fake.setStatus(history.head, running())
|
|
||||||
fake.setStatus(history.parent, cancelled())
|
|
||||||
|
|
||||||
out, err := runScript(t, history, api, scriptEnv{
|
|
||||||
workflow: name,
|
|
||||||
job: job,
|
|
||||||
event: "push",
|
|
||||||
})
|
|
||||||
require.NoError(t, err, out)
|
|
||||||
|
|
||||||
posted := fake.postedFor(history.parent)
|
|
||||||
require.Len(t, posted, 1)
|
|
||||||
require.Equal(t, liveContext, posted[0].Context)
|
|
||||||
}
|
|
||||||
|
|
||||||
// workflowIdentity reads the workflow name and its single job id out of
|
|
||||||
// the checked-in workflow file.
|
|
||||||
func workflowIdentity(t *testing.T) (string, string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
raw, err := os.ReadFile(workflow)
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
var parsed struct {
|
|
||||||
Name string `yaml:"name"`
|
|
||||||
Jobs map[string]any `yaml:"jobs"`
|
|
||||||
}
|
|
||||||
|
|
||||||
require.NoError(t, yaml.Unmarshal(raw, &parsed))
|
|
||||||
|
|
||||||
jobs := slices.Collect(maps.Keys(parsed.Jobs))
|
|
||||||
require.Len(t, jobs, 1)
|
|
||||||
|
|
||||||
return parsed.Name, jobs[0]
|
|
||||||
}
|
|
||||||
|
|
||||||
func runScript(
|
|
||||||
t *testing.T, history repo, api string, env scriptEnv,
|
|
||||||
extra ...string,
|
|
||||||
) (string, error) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
script, err := filepath.Abs(scriptPath)
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
//nolint:gosec // fixed argv, repo-local script under test
|
|
||||||
cmd := exec.CommandContext(t.Context(), "sh", script)
|
|
||||||
cmd.Dir = history.dir
|
|
||||||
cmd.Env = append(os.Environ(),
|
|
||||||
"GITHUB_API_URL="+api,
|
|
||||||
"GITHUB_REPOSITORY=sneak/webhooker",
|
|
||||||
"GITHUB_SHA="+history.head,
|
|
||||||
"GITHUB_WORKFLOW="+env.workflow,
|
|
||||||
"GITHUB_JOB="+env.job,
|
|
||||||
"GITHUB_EVENT_NAME="+env.event,
|
|
||||||
"GITEA_TOKEN=test-token",
|
|
||||||
)
|
|
||||||
cmd.Env = append(cmd.Env, extra...)
|
|
||||||
|
|
||||||
out, err := cmd.CombinedOutput()
|
|
||||||
|
|
||||||
return string(out), err
|
|
||||||
}
|
|
||||||
|
|
||||||
func newRepo(t *testing.T) repo {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
|
|
||||||
git := func(args ...string) string {
|
|
||||||
//nolint:gosec // fixed argv, arguments are test constants
|
|
||||||
cmd := exec.CommandContext(t.Context(), "git", args...)
|
|
||||||
cmd.Dir = dir
|
|
||||||
|
|
||||||
out, err := cmd.CombinedOutput()
|
|
||||||
require.NoError(t, err, string(out))
|
|
||||||
|
|
||||||
return strings.TrimSpace(string(out))
|
|
||||||
}
|
|
||||||
|
|
||||||
commit := func(message string) string {
|
|
||||||
git(
|
|
||||||
"-c", "user.email=ci@example.invalid",
|
|
||||||
"-c", "user.name=ci",
|
|
||||||
"-c", "commit.gpgsign=false",
|
|
||||||
"commit", "-q", "--allow-empty", "-m", message,
|
|
||||||
)
|
|
||||||
|
|
||||||
return git("rev-parse", "HEAD")
|
|
||||||
}
|
|
||||||
|
|
||||||
git("init", "-q", "-b", "main")
|
|
||||||
|
|
||||||
parent := commit("parent")
|
|
||||||
head := commit("head")
|
|
||||||
|
|
||||||
return repo{dir: dir, head: head, parent: parent}
|
|
||||||
}
|
|
||||||
|
|
||||||
func requireTools(t *testing.T) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
for _, tool := range []string{"sh", "git", "curl", "jq"} {
|
|
||||||
_, err := exec.LookPath(tool)
|
|
||||||
if err != nil {
|
|
||||||
t.Skipf("%s is not installed: %v", tool, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,10 +0,0 @@
|
|||||||
// Package ciscript holds the tests for the repository's CI shell
|
|
||||||
// scripts in script/. It carries no runtime code: the scripts run on
|
|
||||||
// the CI runner, not inside the binary, but their behaviour still has
|
|
||||||
// to be verified by the test suite.
|
|
||||||
//
|
|
||||||
// The scripts under test are outside the Go build graph, so `go test`'s
|
|
||||||
// result cache serves a stale PASS when only a script changed: run the
|
|
||||||
// container build, or GOFLAGS=-count=1, to trust a result here after
|
|
||||||
// editing script/.
|
|
||||||
package ciscript
|
|
||||||
@@ -1,162 +0,0 @@
|
|||||||
package ciscript_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"encoding/json"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"sync"
|
|
||||||
"testing"
|
|
||||||
)
|
|
||||||
|
|
||||||
// commitStatus is the part of an entry in Gitea's combined-status
|
|
||||||
// response that script/ci-mark-superseded reads.
|
|
||||||
type commitStatus struct {
|
|
||||||
Context string `json:"context"`
|
|
||||||
Status string `json:"status"`
|
|
||||||
Description string `json:"description"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// postedStatus is the part of a create-status request body the script
|
|
||||||
// writes.
|
|
||||||
type postedStatus struct {
|
|
||||||
Context string `json:"context"`
|
|
||||||
State string `json:"state"`
|
|
||||||
Description string `json:"description"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// fakeGitea serves the two endpoints the script talks to. Like Gitea,
|
|
||||||
// the newest status for a context replaces the previous one, so a
|
|
||||||
// second run of the script sees what the first one wrote.
|
|
||||||
type fakeGitea struct {
|
|
||||||
mu sync.Mutex
|
|
||||||
statuses map[string][]commitStatus
|
|
||||||
posted map[string][]postedStatus
|
|
||||||
// failRead is a commit whose combined-status read answers HTTP
|
|
||||||
// 500, standing in for a status API that is down.
|
|
||||||
failRead string
|
|
||||||
}
|
|
||||||
|
|
||||||
// newFakeGitea returns the fake and the base URL to hand the script as
|
|
||||||
// GITHUB_API_URL.
|
|
||||||
func newFakeGitea(t *testing.T) (*fakeGitea, string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
fake := &fakeGitea{
|
|
||||||
mu: sync.Mutex{},
|
|
||||||
statuses: map[string][]commitStatus{},
|
|
||||||
posted: map[string][]postedStatus{},
|
|
||||||
failRead: "",
|
|
||||||
}
|
|
||||||
|
|
||||||
srv := httptest.NewServer(fake.routes())
|
|
||||||
t.Cleanup(srv.Close)
|
|
||||||
|
|
||||||
return fake, srv.URL
|
|
||||||
}
|
|
||||||
|
|
||||||
func (f *fakeGitea) routes() http.Handler {
|
|
||||||
mux := http.NewServeMux()
|
|
||||||
|
|
||||||
mux.HandleFunc(
|
|
||||||
"GET /repos/{owner}/{repo}/commits/{sha}/status",
|
|
||||||
f.handleCombined,
|
|
||||||
)
|
|
||||||
mux.HandleFunc(
|
|
||||||
"POST /repos/{owner}/{repo}/statuses/{sha}",
|
|
||||||
f.handleCreate,
|
|
||||||
)
|
|
||||||
|
|
||||||
return mux
|
|
||||||
}
|
|
||||||
|
|
||||||
func (f *fakeGitea) handleCombined(
|
|
||||||
w http.ResponseWriter, r *http.Request,
|
|
||||||
) {
|
|
||||||
f.mu.Lock()
|
|
||||||
defer f.mu.Unlock()
|
|
||||||
|
|
||||||
sha := r.PathValue("sha")
|
|
||||||
if f.failRead != "" && f.failRead == sha {
|
|
||||||
http.Error(w, "boom", http.StatusInternalServerError)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
body := struct {
|
|
||||||
Statuses []commitStatus `json:"statuses"`
|
|
||||||
}{Statuses: f.statuses[sha]}
|
|
||||||
|
|
||||||
payload, err := json.Marshal(body)
|
|
||||||
if err != nil {
|
|
||||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
w.Header().Set("Content-Type", "application/json")
|
|
||||||
|
|
||||||
_, _ = w.Write(payload)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (f *fakeGitea) handleCreate(w http.ResponseWriter, r *http.Request) {
|
|
||||||
var got postedStatus
|
|
||||||
|
|
||||||
err := json.NewDecoder(r.Body).Decode(&got)
|
|
||||||
if err != nil {
|
|
||||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
sha := r.PathValue("sha")
|
|
||||||
|
|
||||||
f.mu.Lock()
|
|
||||||
defer f.mu.Unlock()
|
|
||||||
|
|
||||||
f.posted[sha] = append(f.posted[sha], got)
|
|
||||||
f.replaceLocked(sha, commitStatus{
|
|
||||||
Context: got.Context,
|
|
||||||
Status: got.State,
|
|
||||||
Description: got.Description,
|
|
||||||
})
|
|
||||||
|
|
||||||
w.WriteHeader(http.StatusCreated)
|
|
||||||
}
|
|
||||||
|
|
||||||
// failStatusRead makes the combined-status read for one commit answer
|
|
||||||
// HTTP 500.
|
|
||||||
func (f *fakeGitea) failStatusRead(sha string) {
|
|
||||||
f.mu.Lock()
|
|
||||||
defer f.mu.Unlock()
|
|
||||||
|
|
||||||
f.failRead = sha
|
|
||||||
}
|
|
||||||
|
|
||||||
// setStatus gives a commit its latest status for a context.
|
|
||||||
func (f *fakeGitea) setStatus(sha string, status commitStatus) {
|
|
||||||
f.mu.Lock()
|
|
||||||
defer f.mu.Unlock()
|
|
||||||
|
|
||||||
f.replaceLocked(sha, status)
|
|
||||||
}
|
|
||||||
|
|
||||||
// postedFor returns the statuses the script created for a commit.
|
|
||||||
func (f *fakeGitea) postedFor(sha string) []postedStatus {
|
|
||||||
f.mu.Lock()
|
|
||||||
defer f.mu.Unlock()
|
|
||||||
|
|
||||||
return append([]postedStatus(nil), f.posted[sha]...)
|
|
||||||
}
|
|
||||||
|
|
||||||
// replaceLocked requires f.mu.
|
|
||||||
func (f *fakeGitea) replaceLocked(sha string, status commitStatus) {
|
|
||||||
for i, existing := range f.statuses[sha] {
|
|
||||||
if existing.Context == status.Context {
|
|
||||||
f.statuses[sha][i] = status
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
f.statuses[sha] = append(f.statuses[sha], status)
|
|
||||||
}
|
|
||||||
@@ -7,21 +7,22 @@ import (
|
|||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
|
"sneak.berlin/go/webhooker/internal/config/configtest"
|
||||||
)
|
)
|
||||||
|
|
||||||
// TestClearEnvForTest_RemovesAddedVariables pins that a variable set
|
// TestClearEnv_RemovesAddedVariables pins that a variable set
|
||||||
// after the clear other than through t.Setenv, as a test's .env file
|
// after the clear other than through t.Setenv, as a test's .env file
|
||||||
// sets one, is gone once the test ends, so it cannot reach the tests
|
// sets one, is gone once the test ends, so it cannot reach the tests
|
||||||
// that run after it.
|
// that run after it.
|
||||||
//
|
//
|
||||||
//nolint:paralleltest // ClearEnvForTest uses t.Setenv.
|
//nolint:paralleltest // ClearEnv uses t.Setenv.
|
||||||
func TestClearEnvForTest_RemovesAddedVariables(t *testing.T) {
|
func TestClearEnv_RemovesAddedVariables(t *testing.T) {
|
||||||
// The outer clear keeps a value of the key exported in the shell
|
// The outer clear keeps a value of the key exported in the shell
|
||||||
// from making it a variable the inner clear has to put back.
|
// from making it a variable the inner clear has to put back.
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
|
|
||||||
t.Run("loads a .env file after the clear", func(t *testing.T) {
|
t.Run("loads a .env file after the clear", func(t *testing.T) {
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
|
|
||||||
path := writeDotEnv(t, dotEnvKey+"=from-dot-env\n")
|
path := writeDotEnv(t, dotEnvKey+"=from-dot-env\n")
|
||||||
require.NoError(t, config.LoadDotEnvFileForTest(path))
|
require.NoError(t, config.LoadDotEnvFileForTest(path))
|
||||||
@@ -68,6 +68,10 @@ const (
|
|||||||
// authenticated activity before it expires.
|
// authenticated activity before it expires.
|
||||||
defaultSessionIdleTimeout = 24 * time.Hour
|
defaultSessionIdleTimeout = 24 * time.Hour
|
||||||
|
|
||||||
|
// defaultHSTSMaxAge is the max-age of the Strict-Transport-Security
|
||||||
|
// header when HSTS_MAX_AGE is unset: ten years.
|
||||||
|
defaultHSTSMaxAge = 87600 * time.Hour
|
||||||
|
|
||||||
// defaultReceiverRateLimit is the default number of requests
|
// defaultReceiverRateLimit is the default number of requests
|
||||||
// per minute each client IP may send to a single webhook
|
// per minute each client IP may send to a single webhook
|
||||||
// receiver entrypoint. Generous for legitimate webhook
|
// receiver entrypoint. Generous for legitimate webhook
|
||||||
@@ -120,6 +124,10 @@ var ErrInvalidBindAddress = errors.New("invalid bind address")
|
|||||||
// the Sentry SDK cannot parse as a DSN.
|
// the Sentry SDK cannot parse as a DSN.
|
||||||
var ErrInvalidSentryDSN = errors.New("invalid Sentry DSN")
|
var ErrInvalidSentryDSN = errors.New("invalid Sentry DSN")
|
||||||
|
|
||||||
|
// ErrInvalidHSTSMaxAge is returned when HSTS_MAX_AGE is set to a
|
||||||
|
// negative duration or one that is not a whole number of seconds.
|
||||||
|
var ErrInvalidHSTSMaxAge = errors.New("invalid HSTS max-age")
|
||||||
|
|
||||||
// ErrDotEnvUnreadable is returned when the optional .env file exists
|
// ErrDotEnvUnreadable is returned when the optional .env file exists
|
||||||
// but cannot be read or parsed. A file that is not there is not an
|
// but cannot be read or parsed. A file that is not there is not an
|
||||||
// error; a file that is there and broken is.
|
// error; a file that is there and broken is.
|
||||||
@@ -169,6 +177,10 @@ type Config struct {
|
|||||||
// which a session expires. Non-positive disables idle expiry.
|
// which a session expires. Non-positive disables idle expiry.
|
||||||
SessionIdleTimeout time.Duration
|
SessionIdleTimeout time.Duration
|
||||||
|
|
||||||
|
// HSTSMaxAge is the max-age of the Strict-Transport-Security
|
||||||
|
// header. Zero means the header is not sent at all.
|
||||||
|
HSTSMaxAge time.Duration
|
||||||
|
|
||||||
// ReceiverRateLimit is the number of requests per minute each
|
// ReceiverRateLimit is the number of requests per minute each
|
||||||
// client IP may send to a single webhook receiver entrypoint.
|
// client IP may send to a single webhook receiver entrypoint.
|
||||||
ReceiverRateLimit int
|
ReceiverRateLimit int
|
||||||
@@ -624,6 +636,27 @@ func resolveEnvironment() (string, error) {
|
|||||||
return environment, nil
|
return environment, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// resolveHSTSMaxAge reads HSTS_MAX_AGE. Zero is allowed and turns the
|
||||||
|
// Strict-Transport-Security header off. A fraction of a second is an
|
||||||
|
// error rather than rounded, because the header carries whole seconds
|
||||||
|
// and would otherwise say something other than what was set.
|
||||||
|
func resolveHSTSMaxAge() (time.Duration, error) {
|
||||||
|
maxAge, err := envDuration("HSTS_MAX_AGE", defaultHSTSMaxAge)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
|
||||||
|
if maxAge < 0 || maxAge%time.Second != 0 {
|
||||||
|
return 0, fmt.Errorf(
|
||||||
|
"%w: HSTS_MAX_AGE must be zero or a positive whole "+
|
||||||
|
"number of seconds, got %s",
|
||||||
|
ErrInvalidHSTSMaxAge, maxAge,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
return maxAge, nil
|
||||||
|
}
|
||||||
|
|
||||||
// resolveListener reads the two variables that describe the HTTP
|
// resolveListener reads the two variables that describe the HTTP
|
||||||
// listener: which port it claims and which address it claims it on.
|
// listener: which port it claims and which address it claims it on.
|
||||||
// They are read together because neither is meaningful alone, and
|
// They are read together because neither is meaningful alone, and
|
||||||
@@ -684,6 +717,11 @@ func loadFromEnv() (*Config, error) {
|
|||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
hstsMaxAge, err := resolveHSTSMaxAge()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
receiverRateLimit, err := envPositiveInt(
|
receiverRateLimit, err := envPositiveInt(
|
||||||
"RECEIVER_RATE_LIMIT",
|
"RECEIVER_RATE_LIMIT",
|
||||||
defaultReceiverRateLimit,
|
defaultReceiverRateLimit,
|
||||||
@@ -723,6 +761,7 @@ func loadFromEnv() (*Config, error) {
|
|||||||
SentryDSN: sentryDSN,
|
SentryDSN: sentryDSN,
|
||||||
RetentionSweepInterval: retentionSweepInterval,
|
RetentionSweepInterval: retentionSweepInterval,
|
||||||
SessionIdleTimeout: sessionIdleTimeout,
|
SessionIdleTimeout: sessionIdleTimeout,
|
||||||
|
HSTSMaxAge: hstsMaxAge,
|
||||||
ReceiverRateLimit: receiverRateLimit,
|
ReceiverRateLimit: receiverRateLimit,
|
||||||
TrustedProxies: trustedProxies,
|
TrustedProxies: trustedProxies,
|
||||||
AllowedEgressCIDRs: allowedEgressCIDRs,
|
AllowedEgressCIDRs: allowedEgressCIDRs,
|
||||||
@@ -807,6 +846,7 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
|
|||||||
// disables idle expiry entirely, and that is worth showing
|
// disables idle expiry entirely, and that is worth showing
|
||||||
// back to the operator.
|
// back to the operator.
|
||||||
"sessionIdleTimeout", s.SessionIdleTimeout.String(),
|
"sessionIdleTimeout", s.SessionIdleTimeout.String(),
|
||||||
|
"hstsMaxAge", s.HSTSMaxAge.String(),
|
||||||
"receiverRateLimit", s.ReceiverRateLimit,
|
"receiverRateLimit", s.ReceiverRateLimit,
|
||||||
"trustedProxies", len(s.TrustedProxies),
|
"trustedProxies", len(s.TrustedProxies),
|
||||||
"allowedEgressCIDRs", len(s.AllowedEgressCIDRs),
|
"allowedEgressCIDRs", len(s.AllowedEgressCIDRs),
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
"go.uber.org/fx"
|
"go.uber.org/fx"
|
||||||
"go.uber.org/fx/fxtest"
|
"go.uber.org/fx/fxtest"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
|
"sneak.berlin/go/webhooker/internal/config/configtest"
|
||||||
"sneak.berlin/go/webhooker/internal/globals"
|
"sneak.berlin/go/webhooker/internal/globals"
|
||||||
"sneak.berlin/go/webhooker/internal/logger"
|
"sneak.berlin/go/webhooker/internal/logger"
|
||||||
)
|
)
|
||||||
@@ -70,7 +71,7 @@ func TestEnvironmentConfig(t *testing.T) {
|
|||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
// Cannot use t.Parallel() here because t.Setenv
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
// is incompatible with parallel subtests.
|
// is incompatible with parallel subtests.
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
|
|
||||||
if tt.envValue != "" {
|
if tt.envValue != "" {
|
||||||
t.Setenv(
|
t.Setenv(
|
||||||
@@ -196,7 +197,7 @@ func TestRetentionSweepInterval(t *testing.T) {
|
|||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
// Cannot use t.Parallel() here because t.Setenv
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
// is incompatible with parallel subtests.
|
// is incompatible with parallel subtests.
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
|
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
|
||||||
|
|
||||||
if tt.set {
|
if tt.set {
|
||||||
@@ -335,7 +336,7 @@ func TestSessionIdleTimeout(t *testing.T) {
|
|||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
// Cannot use t.Parallel() here because t.Setenv
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
// is incompatible with parallel subtests.
|
// is incompatible with parallel subtests.
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
|
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
|
||||||
|
|
||||||
if tt.set {
|
if tt.set {
|
||||||
@@ -378,6 +379,83 @@ func testSessionIdleTimeoutSuccess(
|
|||||||
assert.Equal(t, expected, cfg.SessionIdleTimeout)
|
assert.Equal(t, expected, cfg.SessionIdleTimeout)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestHSTSMaxAge(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
set bool
|
||||||
|
value string
|
||||||
|
expectError bool
|
||||||
|
// sentinel, when set, must be wrapped by the startup
|
||||||
|
// error; every error case must additionally name the
|
||||||
|
// variable in its message.
|
||||||
|
sentinel error
|
||||||
|
expected time.Duration
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: caseUnsetUsesDefault,
|
||||||
|
set: false,
|
||||||
|
expected: 87600 * time.Hour,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: caseValidValueParsed,
|
||||||
|
set: true,
|
||||||
|
value: "720h",
|
||||||
|
expected: 720 * time.Hour,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// Zero turns the header off; it is not an error.
|
||||||
|
name: "zero is accepted",
|
||||||
|
set: true,
|
||||||
|
value: "0",
|
||||||
|
expected: 0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: caseUnparseableFails,
|
||||||
|
set: true,
|
||||||
|
value: "10 years",
|
||||||
|
expectError: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// A whole number of seconds, so only the sign is wrong.
|
||||||
|
name: "negative whole seconds fail startup",
|
||||||
|
set: true,
|
||||||
|
value: "-1s",
|
||||||
|
expectError: true,
|
||||||
|
sentinel: config.ErrInvalidHSTSMaxAge,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "fraction of a second fails startup",
|
||||||
|
set: true,
|
||||||
|
value: "1500ms",
|
||||||
|
expectError: true,
|
||||||
|
sentinel: config.ErrInvalidHSTSMaxAge,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
|
// is incompatible with parallel subtests.
|
||||||
|
configtest.ClearEnv(t)
|
||||||
|
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
|
||||||
|
|
||||||
|
if tt.set {
|
||||||
|
t.Setenv("HSTS_MAX_AGE", tt.value)
|
||||||
|
}
|
||||||
|
|
||||||
|
if tt.expectError {
|
||||||
|
expectStartupErrorFor(t, "HSTS_MAX_AGE", tt.sentinel)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg, err := buildConfig(t)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, tt.expected, cfg.HSTSMaxAge)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestDefaultDataDir(t *testing.T) {
|
func TestDefaultDataDir(t *testing.T) {
|
||||||
for _, env := range []string{"", "dev", "prod"} {
|
for _, env := range []string{"", "dev", "prod"} {
|
||||||
name := env
|
name := env
|
||||||
@@ -388,7 +466,7 @@ func TestDefaultDataDir(t *testing.T) {
|
|||||||
t.Run("env="+name, func(t *testing.T) {
|
t.Run("env="+name, func(t *testing.T) {
|
||||||
// Cannot use t.Parallel() here because t.Setenv
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
// is incompatible with parallel subtests.
|
// is incompatible with parallel subtests.
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
|
|
||||||
if env != "" {
|
if env != "" {
|
||||||
t.Setenv("WEBHOOKER_ENVIRONMENT", env)
|
t.Setenv("WEBHOOKER_ENVIRONMENT", env)
|
||||||
@@ -433,7 +511,7 @@ func TestDataDirHelper(t *testing.T) {
|
|||||||
t.Run(name, func(t *testing.T) {
|
t.Run(name, func(t *testing.T) {
|
||||||
// Cannot use t.Parallel() here because t.Setenv
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
// is incompatible with parallel subtests.
|
// is incompatible with parallel subtests.
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
|
|
||||||
if set != "" {
|
if set != "" {
|
||||||
t.Setenv("DATA_DIR", set)
|
t.Setenv("DATA_DIR", set)
|
||||||
@@ -498,7 +576,7 @@ func TestReceiverRateLimit(t *testing.T) {
|
|||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
// Cannot use t.Parallel() here because t.Setenv
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
// is incompatible with parallel subtests.
|
// is incompatible with parallel subtests.
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
|
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
|
||||||
|
|
||||||
if tt.set {
|
if tt.set {
|
||||||
@@ -614,7 +692,7 @@ func TestTrustedProxies(t *testing.T) {
|
|||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
// Cannot use t.Parallel() here because t.Setenv
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
// is incompatible with parallel subtests.
|
// is incompatible with parallel subtests.
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
|
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
|
||||||
|
|
||||||
if tt.set {
|
if tt.set {
|
||||||
@@ -725,7 +803,7 @@ func TestAllowedEgressCIDRs(t *testing.T) {
|
|||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
// Cannot use t.Parallel() here because t.Setenv
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
// is incompatible with parallel subtests.
|
// is incompatible with parallel subtests.
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
|
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
|
||||||
|
|
||||||
if tt.set {
|
if tt.set {
|
||||||
@@ -797,7 +875,7 @@ func TestEgressAllowlistWarning(t *testing.T) {
|
|||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
// Cannot use t.Parallel() here because t.Setenv
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
// is incompatible with parallel subtests.
|
// is incompatible with parallel subtests.
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
t.Setenv("WEBHOOKER_ENVIRONMENT", config.EnvironmentDev)
|
t.Setenv("WEBHOOKER_ENVIRONMENT", config.EnvironmentDev)
|
||||||
|
|
||||||
if tt.allowed != "" {
|
if tt.allowed != "" {
|
||||||
@@ -933,7 +1011,7 @@ func TestMetricsAuthConfig(t *testing.T) {
|
|||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
// Cannot use t.Parallel() here because t.Setenv
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
// is incompatible with parallel subtests.
|
// is incompatible with parallel subtests.
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
|
|
||||||
if tt.username.set {
|
if tt.username.set {
|
||||||
t.Setenv("METRICS_USERNAME", tt.username.value)
|
t.Setenv("METRICS_USERNAME", tt.username.value)
|
||||||
|
|||||||
@@ -1,4 +1,6 @@
|
|||||||
package config
|
// Package configtest holds test support for code that reads the
|
||||||
|
// process environment.
|
||||||
|
package configtest
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"os"
|
"os"
|
||||||
@@ -6,12 +8,12 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ClearEnvForTest unsets every variable in the process environment
|
// ClearEnv unsets every variable in the process environment
|
||||||
// for the rest of the test, so a test sees only the variables it sets
|
// for the rest of the test, so a test sees only the variables it sets
|
||||||
// itself, not whatever the developer's shell exports. When the test
|
// itself, not whatever the developer's shell exports. When the test
|
||||||
// ends it leaves the environment exactly as it found it: each variable
|
// ends it leaves the environment exactly as it found it: each variable
|
||||||
// it unset is put back, and any variable added since is removed.
|
// it unset is put back, and any variable added since is removed.
|
||||||
func ClearEnvForTest(t *testing.T) {
|
func ClearEnv(t *testing.T) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
present := make(map[string]bool)
|
present := make(map[string]bool)
|
||||||
@@ -8,6 +8,7 @@ import (
|
|||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
|
"sneak.berlin/go/webhooker/internal/config/configtest"
|
||||||
)
|
)
|
||||||
|
|
||||||
// dotEnvKey is a throwaway variable name the .env tests write and
|
// dotEnvKey is a throwaway variable name the .env tests write and
|
||||||
@@ -39,9 +40,9 @@ func writeDotEnv(t *testing.T, contents string) string {
|
|||||||
// normally rather than be refused for a file it was never meant to
|
// normally rather than be refused for a file it was never meant to
|
||||||
// have.
|
// have.
|
||||||
//
|
//
|
||||||
//nolint:paralleltest // ClearEnvForTest uses t.Setenv.
|
//nolint:paralleltest // ClearEnv uses t.Setenv.
|
||||||
func TestLoadDotEnv_MissingFileIsFine(t *testing.T) {
|
func TestLoadDotEnv_MissingFileIsFine(t *testing.T) {
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
|
|
||||||
absent := filepath.Join(t.TempDir(), config.DotEnvPath)
|
absent := filepath.Join(t.TempDir(), config.DotEnvPath)
|
||||||
require.NoError(t, config.LoadDotEnvFileForTest(absent))
|
require.NoError(t, config.LoadDotEnvFileForTest(absent))
|
||||||
@@ -54,9 +55,9 @@ func TestLoadDotEnv_MissingFileIsFine(t *testing.T) {
|
|||||||
// reaches the environment, which is the whole reason the file is read
|
// reaches the environment, which is the whole reason the file is read
|
||||||
// at all.
|
// at all.
|
||||||
//
|
//
|
||||||
//nolint:paralleltest // ClearEnvForTest uses t.Setenv.
|
//nolint:paralleltest // ClearEnv uses t.Setenv.
|
||||||
func TestLoadDotEnv_AppliesValues(t *testing.T) {
|
func TestLoadDotEnv_AppliesValues(t *testing.T) {
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
|
|
||||||
path := writeDotEnv(t, "# a comment\n"+dotEnvKey+"=from-dot-env\n")
|
path := writeDotEnv(t, "# a comment\n"+dotEnvKey+"=from-dot-env\n")
|
||||||
|
|
||||||
@@ -82,9 +83,9 @@ func TestLoadDotEnv_RealEnvironmentWins(t *testing.T) {
|
|||||||
// reverts to its default; the process used to start that way with no
|
// reverts to its default; the process used to start that way with no
|
||||||
// log line naming the file at all.
|
// log line naming the file at all.
|
||||||
//
|
//
|
||||||
//nolint:paralleltest // ClearEnvForTest uses t.Setenv.
|
//nolint:paralleltest // ClearEnv uses t.Setenv.
|
||||||
func TestLoadDotEnv_MalformedFileAborts(t *testing.T) {
|
func TestLoadDotEnv_MalformedFileAborts(t *testing.T) {
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
|
|
||||||
path := writeDotEnv(
|
path := writeDotEnv(
|
||||||
t, malformedDotEnv+dotEnvKey+"=from-dot-env\n",
|
t, malformedDotEnv+dotEnvKey+"=from-dot-env\n",
|
||||||
@@ -132,7 +133,7 @@ func TestLoadDotEnv_UnreadableFileAborts(t *testing.T) {
|
|||||||
//
|
//
|
||||||
//nolint:paralleltest // t.Chdir moves the whole process.
|
//nolint:paralleltest // t.Chdir moves the whole process.
|
||||||
func TestLoadDotEnv_ReadsTheWorkingDirectory(t *testing.T) {
|
func TestLoadDotEnv_ReadsTheWorkingDirectory(t *testing.T) {
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
|
|
||||||
dir := t.TempDir()
|
dir := t.TempDir()
|
||||||
require.NoError(t, os.WriteFile(
|
require.NoError(t, os.WriteFile(
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"go.uber.org/fx"
|
"go.uber.org/fx"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
|
"sneak.berlin/go/webhooker/internal/config/configtest"
|
||||||
"sneak.berlin/go/webhooker/internal/globals"
|
"sneak.berlin/go/webhooker/internal/globals"
|
||||||
"sneak.berlin/go/webhooker/internal/logger"
|
"sneak.berlin/go/webhooker/internal/logger"
|
||||||
)
|
)
|
||||||
@@ -120,7 +121,7 @@ func TestEnvBool(t *testing.T) {
|
|||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
// Cannot use t.Parallel() here because t.Setenv
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
// is incompatible with parallel subtests.
|
// is incompatible with parallel subtests.
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
|
|
||||||
if tt.set {
|
if tt.set {
|
||||||
t.Setenv(testEnvKey, tt.value)
|
t.Setenv(testEnvKey, tt.value)
|
||||||
@@ -169,7 +170,7 @@ func runEnvIntCases(
|
|||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
// Cannot use t.Parallel() here because t.Setenv
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
// is incompatible with parallel subtests.
|
// is incompatible with parallel subtests.
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
|
|
||||||
if tt.set {
|
if tt.set {
|
||||||
t.Setenv(testEnvKey, tt.value)
|
t.Setenv(testEnvKey, tt.value)
|
||||||
@@ -310,7 +311,7 @@ func TestEnvBindAddress(t *testing.T) {
|
|||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
// Cannot use t.Parallel() here because t.Setenv
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
// is incompatible with parallel subtests.
|
// is incompatible with parallel subtests.
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
|
|
||||||
if tt.set {
|
if tt.set {
|
||||||
t.Setenv(testEnvKey, tt.value)
|
t.Setenv(testEnvKey, tt.value)
|
||||||
@@ -476,7 +477,7 @@ func TestNewRejectsBadEnvValues(t *testing.T) {
|
|||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
// Cannot use t.Parallel() here because t.Setenv
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
// is incompatible with parallel subtests.
|
// is incompatible with parallel subtests.
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
|
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
|
||||||
t.Setenv(tt.key, tt.value)
|
t.Setenv(tt.key, tt.value)
|
||||||
|
|
||||||
@@ -638,7 +639,7 @@ func sentryEnvValueCases() []badEnvValueCase {
|
|||||||
// break the legitimate unset case: absent variables still get their
|
// break the legitimate unset case: absent variables still get their
|
||||||
// documented defaults.
|
// documented defaults.
|
||||||
func TestNewUsesDefaultsWhenUnset(t *testing.T) {
|
func TestNewUsesDefaultsWhenUnset(t *testing.T) {
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
|
t.Setenv("WEBHOOKER_ENVIRONMENT", "dev")
|
||||||
|
|
||||||
cfg, err := buildConfig(t)
|
cfg, err := buildConfig(t)
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ import (
|
|||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
|
"sneak.berlin/go/webhooker/internal/config/configtest"
|
||||||
)
|
)
|
||||||
|
|
||||||
// envKeySentryDSN is the variable envSentryDSN reads in production.
|
// envKeySentryDSN is the variable envSentryDSN reads in production.
|
||||||
@@ -100,7 +101,7 @@ func TestEnvSentryDSN(t *testing.T) {
|
|||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
// Cannot use t.Parallel() here because t.Setenv
|
// Cannot use t.Parallel() here because t.Setenv
|
||||||
// is incompatible with parallel subtests.
|
// is incompatible with parallel subtests.
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
|
|
||||||
if tt.set {
|
if tt.set {
|
||||||
t.Setenv(envKeySentryDSN, tt.value)
|
t.Setenv(envKeySentryDSN, tt.value)
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ import (
|
|||||||
"sneak.berlin/go/webhooker/internal/banner"
|
"sneak.berlin/go/webhooker/internal/banner"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
"sneak.berlin/go/webhooker/internal/datadir"
|
"sneak.berlin/go/webhooker/internal/datadir"
|
||||||
|
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||||
"sneak.berlin/go/webhooker/internal/logger"
|
"sneak.berlin/go/webhooker/internal/logger"
|
||||||
)
|
)
|
||||||
@@ -249,12 +250,21 @@ func (d *Database) connectTo(dataDir string) error {
|
|||||||
return fmt.Errorf("migrating %s: %w", dbPath, err)
|
return fmt.Errorf("migrating %s: %w", dbPath, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
err = parseModels(
|
||||||
|
db, &Setting{}, &User{}, &APIKey{},
|
||||||
|
&Webhook{}, &Entrypoint{}, &Target{},
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("connecting to %s: %w", dbPath, err)
|
||||||
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// migrate applies the main database's SQL files. See
|
||||||
|
// internal/db/migrations.
|
||||||
func (d *Database) migrate() error {
|
func (d *Database) migrate() error {
|
||||||
// Run GORM auto-migrations
|
err := migrations.Apply(d.db, migrations.Main)
|
||||||
err := d.Migrate()
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
d.log.Error(
|
d.log.Error(
|
||||||
"failed to run database migrations",
|
"failed to run database migrations",
|
||||||
@@ -269,6 +279,21 @@ func (d *Database) migrate() error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// parseModels has GORM parse each model for db before the handle is
|
||||||
|
// shared. GORM parses a model the first time a handle uses it, and two
|
||||||
|
// goroutines doing that at once for models that refer to each other,
|
||||||
|
// as Webhook and Target do, race inside GORM.
|
||||||
|
func parseModels(db *gorm.DB, models ...any) error {
|
||||||
|
for _, model := range models {
|
||||||
|
err := (&gorm.Statement{DB: db}).Parse(model)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("parsing model %T: %w", model, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// ensureAdminUser creates the bootstrap admin account when the
|
// ensureAdminUser creates the bootstrap admin account when the
|
||||||
// deployment has no users at all.
|
// deployment has no users at all.
|
||||||
func (d *Database) ensureAdminUser() error {
|
func (d *Database) ensureAdminUser() error {
|
||||||
|
|||||||
@@ -0,0 +1,55 @@
|
|||||||
|
// Package databasetest builds a WebhookDBManager for tests in other
|
||||||
|
// packages.
|
||||||
|
package databasetest
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log/slog"
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"go.uber.org/fx/fxtest"
|
||||||
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
)
|
||||||
|
|
||||||
|
// NewWebhookDBManager creates a WebhookDBManager backed by the given
|
||||||
|
// data directory, logging at DEBUG to standard error.
|
||||||
|
func NewWebhookDBManager(
|
||||||
|
t *testing.T, dataDir string,
|
||||||
|
) *database.WebhookDBManager {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
return NewWebhookDBManagerWithLogger(
|
||||||
|
t,
|
||||||
|
dataDir,
|
||||||
|
slog.New(slog.NewTextHandler(
|
||||||
|
os.Stderr,
|
||||||
|
&slog.HandlerOptions{Level: slog.LevelDebug},
|
||||||
|
)),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewWebhookDBManagerWithLogger is NewWebhookDBManager with the
|
||||||
|
// logger supplied by the caller. The per-webhook databases this manager
|
||||||
|
// opens hand that logger to gormlog, so a test that needs to see the SQL
|
||||||
|
// the service emits can capture it.
|
||||||
|
//
|
||||||
|
// It is built through database.NewWebhookDBManager on a lifecycle that
|
||||||
|
// is never started, so nothing closes its databases but the caller.
|
||||||
|
func NewWebhookDBManagerWithLogger(
|
||||||
|
t *testing.T, dataDir string, log *slog.Logger,
|
||||||
|
) *database.WebhookDBManager {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
mgr, err := database.NewWebhookDBManager(
|
||||||
|
fxtest.NewLifecycle(t),
|
||||||
|
database.WebhookDBManagerParams{
|
||||||
|
Config: &config.Config{DataDir: dataDir},
|
||||||
|
Logger: log,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
return mgr
|
||||||
|
}
|
||||||
@@ -2,7 +2,6 @@ package database
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
"log/slog"
|
|
||||||
|
|
||||||
"gorm.io/gorm"
|
"gorm.io/gorm"
|
||||||
"gorm.io/gorm/clause"
|
"gorm.io/gorm/clause"
|
||||||
@@ -18,14 +17,14 @@ const omitAssociationsCallback = "webhooker:omit_associations"
|
|||||||
//
|
//
|
||||||
// A per-webhook database holds the event tier only, but Delivery
|
// A per-webhook database holds the event tier only, but Delivery
|
||||||
// declares belongs-to Event and Target and the delivery engine fills
|
// declares belongs-to Event and Target and the delivery engine fills
|
||||||
// both in memory before writing. Without this guard GORM upserts
|
// both in memory before writing. Without this guard GORM upserts those
|
||||||
// those parent rows here on the delivery and retry write paths,
|
// parent rows here on the delivery and retry write paths, and the
|
||||||
// copying targets.config, which holds destination URLs and bearer
|
// target, whose config holds destination URLs and bearer credentials,
|
||||||
// credentials, into the file most likely to be backed up or handed
|
// goes to a targets table this file does not have, which fails the
|
||||||
// to someone else. Registering the guard on the connection covers
|
// write. Registering the guard on the connection
|
||||||
// every write path, including writes inside a transaction and write
|
// covers every write path, including writes inside a transaction and
|
||||||
// paths added later. Every event-tier row this file holds is written
|
// write paths added later. Every event-tier row this file holds is
|
||||||
// explicitly, so nothing depends on the automatic save.
|
// written explicitly, so nothing depends on the automatic save.
|
||||||
func omitAssociations(db *gorm.DB) error {
|
func omitAssociations(db *gorm.DB) error {
|
||||||
omit := func(tx *gorm.DB) {
|
omit := func(tx *gorm.DB) {
|
||||||
tx.Statement.Omits = append(
|
tx.Statement.Omits = append(
|
||||||
@@ -53,107 +52,3 @@ func omitAssociations(db *gorm.DB) error {
|
|||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// eventDBSweptVersion is the PRAGMA user_version purgeTargetRows
|
|
||||||
// stamps into a per-webhook database once it has removed any leaked
|
|
||||||
// target rows *and* the VACUUM that removes their bytes has returned.
|
|
||||||
// Nothing else in the tree uses user_version, so 0 means "not swept
|
|
||||||
// by this build".
|
|
||||||
//
|
|
||||||
// The stamp, not the DELETE, is what records that a file is done. A
|
|
||||||
// DELETE commits on its own, so a sweep that is interrupted or whose
|
|
||||||
// VACUUM fails leaves a file whose rows are gone but whose credential
|
|
||||||
// bytes are still in the free pages -- indistinguishable, by row
|
|
||||||
// count, from a file that never leaked. Both leave the stamp unset,
|
|
||||||
// so the next open sweeps again.
|
|
||||||
const eventDBSweptVersion = 1
|
|
||||||
|
|
||||||
// purgeTargetRows deletes target rows that an earlier build's
|
|
||||||
// association upsert wrote into a per-webhook database, and rewrites
|
|
||||||
// the file so their bytes are gone with them. AutoMigrate creates a
|
|
||||||
// targets table in every one of these files because Delivery declares
|
|
||||||
// a belongs-to Target, but nothing in the event tier may put rows in
|
|
||||||
// it. The rows it did put there are junk, not history: they carry an
|
|
||||||
// empty webhook_id, and delivery rows resolve their target against
|
|
||||||
// the main database, so nothing here refers to them.
|
|
||||||
//
|
|
||||||
// The DELETE only unlinks the rows: modernc.org/sqlite leaves
|
|
||||||
// secure_delete at SQLite's default of off, so the credential bytes
|
|
||||||
// stay readable in the file's free pages and a backup of a swept file
|
|
||||||
// would still hand them over. VACUUM rewrites the file without them.
|
|
||||||
//
|
|
||||||
// This runs before every migration and is gated on
|
|
||||||
// eventDBSweptVersion, so a file pays for the rewrite once, on the
|
|
||||||
// first open that finds it unstamped, and every open after that is a
|
|
||||||
// PRAGMA read. A file this build created is stamped before its
|
|
||||||
// targets table exists, so it never vacuums at all. A failure here
|
|
||||||
// fails the open with the stamp left unset, so the sweep is retried
|
|
||||||
// rather than skipped -- a webhook whose file cannot be swept stays
|
|
||||||
// unusable instead of quietly serving from a file that still holds
|
|
||||||
// recoverable credentials.
|
|
||||||
func purgeTargetRows(
|
|
||||||
db *gorm.DB, log *slog.Logger, webhookID string,
|
|
||||||
) error {
|
|
||||||
var version int
|
|
||||||
|
|
||||||
// Row().Scan, not (*gorm.DB).Scan: see internal/gormlog.
|
|
||||||
err := db.Raw("PRAGMA user_version").Row().Scan(&version)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf(
|
|
||||||
"reading sweep marker of webhook database %s: %w",
|
|
||||||
webhookID, err,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
if version >= eventDBSweptVersion {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
var purged int64
|
|
||||||
|
|
||||||
if db.Migrator().HasTable("targets") {
|
|
||||||
res := db.Exec("DELETE FROM targets")
|
|
||||||
if res.Error != nil {
|
|
||||||
return fmt.Errorf(
|
|
||||||
"purging target rows from webhook database %s: %w",
|
|
||||||
webhookID, res.Error,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
purged = res.RowsAffected
|
|
||||||
|
|
||||||
// Unconditional: a zero row count here does not mean there is
|
|
||||||
// nothing to remove, only that no *live* row is left. See
|
|
||||||
// eventDBSweptVersion.
|
|
||||||
err = db.Exec("VACUUM").Error
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf(
|
|
||||||
"purged %d leaked target rows from webhook database "+
|
|
||||||
"%s but vacuuming it failed, so the deleted "+
|
|
||||||
"target credentials are still recoverable from "+
|
|
||||||
"the file; it stays marked unswept and the next "+
|
|
||||||
"open retries: %w",
|
|
||||||
purged, webhookID, err,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
err = db.Exec(fmt.Sprintf(
|
|
||||||
"PRAGMA user_version = %d", eventDBSweptVersion,
|
|
||||||
)).Error
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf(
|
|
||||||
"marking webhook database %s swept: %w", webhookID, err,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
if purged > 0 {
|
|
||||||
log.Warn(
|
|
||||||
"purged leaked target rows from per-webhook database",
|
|
||||||
"webhook_id", webhookID,
|
|
||||||
"rows", purged,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,391 +1,15 @@
|
|||||||
package database_test
|
package database_test
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
|
||||||
"database/sql"
|
|
||||||
"fmt"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
_ "modernc.org/sqlite"
|
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||||
)
|
)
|
||||||
|
|
||||||
// testDataDirPerm is the mode the test data directory is created
|
|
||||||
// with.
|
|
||||||
const testDataDirPerm = 0o750
|
|
||||||
|
|
||||||
// eventDBDataDir returns a data directory that a WebhookDBManager
|
|
||||||
// can be pointed at.
|
|
||||||
func eventDBDataDir(t *testing.T) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
dir := filepath.Join(t.TempDir(), "events")
|
|
||||||
require.NoError(t, os.MkdirAll(dir, testDataDirPerm))
|
|
||||||
|
|
||||||
return dir
|
|
||||||
}
|
|
||||||
|
|
||||||
// openRawEventDB opens the per-webhook database file directly,
|
|
||||||
// without the manager, so a test can put a file on disk in a state
|
|
||||||
// the manager has to cope with, or inspect one afterwards.
|
|
||||||
func openRawEventDB(
|
|
||||||
t *testing.T, dataDir, webhookID string,
|
|
||||||
) *sql.DB {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
path := filepath.Join(
|
|
||||||
dataDir, fmt.Sprintf("events-%s.db", webhookID),
|
|
||||||
)
|
|
||||||
|
|
||||||
sqlDB, err := sql.Open(
|
|
||||||
"sqlite",
|
|
||||||
fmt.Sprintf("file:%s?mode=rwc", path),
|
|
||||||
)
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = sqlDB.Close() })
|
|
||||||
|
|
||||||
return sqlDB
|
|
||||||
}
|
|
||||||
|
|
||||||
// eventDBFileBytes reads a per-webhook database file off disk, so a
|
|
||||||
// test can assert on what the file itself still holds rather than on
|
|
||||||
// what a query returns.
|
|
||||||
func eventDBFileBytes(t *testing.T, dataDir, webhookID string) []byte {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
//nolint:gosec // reads a file the test just created under t.TempDir()
|
|
||||||
raw, err := os.ReadFile(filepath.Join(
|
|
||||||
dataDir, fmt.Sprintf("events-%s.db", webhookID),
|
|
||||||
))
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
return raw
|
|
||||||
}
|
|
||||||
|
|
||||||
// eventDBUserVersion returns the PRAGMA user_version of a per-webhook
|
|
||||||
// database file, which is the marker purgeTargetRows stamps once it
|
|
||||||
// has swept and vacuumed.
|
|
||||||
func eventDBUserVersion(t *testing.T, sqlDB *sql.DB) int {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
var version int
|
|
||||||
|
|
||||||
require.NoError(t, sqlDB.QueryRowContext(
|
|
||||||
t.Context(), "PRAGMA user_version",
|
|
||||||
).Scan(&version))
|
|
||||||
|
|
||||||
return version
|
|
||||||
}
|
|
||||||
|
|
||||||
// clearEventDBSweptMarker resets the sweep marker to 0, which is what
|
|
||||||
// a file written by a build without the sweep looks like. Tests that
|
|
||||||
// seed a leaked row have to create the file through the manager to
|
|
||||||
// get the real targets table shape, and that stamps it.
|
|
||||||
func clearEventDBSweptMarker(t *testing.T, sqlDB *sql.DB) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
_, err := sqlDB.ExecContext(t.Context(), "PRAGMA user_version = 0")
|
|
||||||
require.NoError(t, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// countTargetRows returns the number of rows in the targets table of
|
|
||||||
// a per-webhook database file, or -1 if the table does not exist.
|
|
||||||
func countTargetRows(t *testing.T, sqlDB *sql.DB) int {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
var tables int
|
|
||||||
|
|
||||||
require.NoError(t, sqlDB.QueryRowContext(
|
|
||||||
t.Context(),
|
|
||||||
"SELECT count(*) FROM sqlite_master "+
|
|
||||||
"WHERE type = 'table' AND name = 'targets'",
|
|
||||||
).Scan(&tables))
|
|
||||||
|
|
||||||
if tables == 0 {
|
|
||||||
return -1
|
|
||||||
}
|
|
||||||
|
|
||||||
var rows int
|
|
||||||
|
|
||||||
require.NoError(t, sqlDB.QueryRowContext(
|
|
||||||
t.Context(), "SELECT count(*) FROM targets",
|
|
||||||
).Scan(&rows))
|
|
||||||
|
|
||||||
return rows
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOpenPurgesLeakedTargetRows covers the sweep for event
|
|
||||||
// databases written by a build that let GORM upsert target rows
|
|
||||||
// into them: opening the database clears them, and opening it again
|
|
||||||
// is a no-op.
|
|
||||||
func TestOpenPurgesLeakedTargetRows(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dataDir := eventDBDataDir(t)
|
|
||||||
webhookID := uuid.New().String()
|
|
||||||
|
|
||||||
// Create the file the way the application does, so the targets
|
|
||||||
// table has exactly the shape AutoMigrate gives it, then write
|
|
||||||
// a leaked row into it the way the association upsert did.
|
|
||||||
initial := database.NewTestWebhookDBManager(dataDir)
|
|
||||||
|
|
||||||
_, err := initial.GetDB(webhookID)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NoError(t, initial.CloseAll())
|
|
||||||
|
|
||||||
seed := openRawEventDB(t, dataDir, webhookID)
|
|
||||||
|
|
||||||
_, err = seed.ExecContext(
|
|
||||||
t.Context(),
|
|
||||||
"INSERT INTO targets "+
|
|
||||||
"(id, webhook_id, name, type, config) "+
|
|
||||||
"VALUES (?, '', ?, ?, ?)",
|
|
||||||
uuid.New().String(),
|
|
||||||
"leaked-target",
|
|
||||||
"slack",
|
|
||||||
`{"webhookUrl":"https://hooks.example/T000/B000/secret"}`,
|
|
||||||
)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.Equal(t, 1, countTargetRows(t, seed))
|
|
||||||
clearEventDBSweptMarker(t, seed)
|
|
||||||
require.NoError(t, seed.Close())
|
|
||||||
|
|
||||||
mgr := database.NewTestWebhookDBManager(dataDir)
|
|
||||||
|
|
||||||
_, err = mgr.GetDB(webhookID)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NoError(t, mgr.CloseAll())
|
|
||||||
|
|
||||||
check := openRawEventDB(t, dataDir, webhookID)
|
|
||||||
assert.Zero(t, countTargetRows(t, check))
|
|
||||||
assert.Equal(
|
|
||||||
t, 1, eventDBUserVersion(t, check),
|
|
||||||
"a completed sweep must mark the file so later opens skip it",
|
|
||||||
)
|
|
||||||
require.NoError(t, check.Close())
|
|
||||||
|
|
||||||
// Idempotent: a second open leaves it at zero and does not
|
|
||||||
// error.
|
|
||||||
again := database.NewTestWebhookDBManager(dataDir)
|
|
||||||
|
|
||||||
_, err = again.GetDB(webhookID)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NoError(t, again.CloseAll())
|
|
||||||
|
|
||||||
recheck := openRawEventDB(t, dataDir, webhookID)
|
|
||||||
assert.Zero(t, countTargetRows(t, recheck))
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOpenPurgeRemovesCredentialBytes covers the sweep at the level
|
|
||||||
// that matters for a backup handed to someone else: the leaked
|
|
||||||
// credential must be gone from the raw bytes of the file, not merely
|
|
||||||
// unreachable by query. A bare DELETE unlinks the row and leaves the
|
|
||||||
// bytes readable in the free pages, so this fails without the VACUUM
|
|
||||||
// in purgeTargetRows.
|
|
||||||
func TestOpenPurgeRemovesCredentialBytes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dataDir := eventDBDataDir(t)
|
|
||||||
webhookID := uuid.New().String()
|
|
||||||
credential := "T00000000/B00000000/" + uuid.New().String()
|
|
||||||
|
|
||||||
initial := database.NewTestWebhookDBManager(dataDir)
|
|
||||||
|
|
||||||
_, err := initial.GetDB(webhookID)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NoError(t, initial.CloseAll())
|
|
||||||
|
|
||||||
seed := openRawEventDB(t, dataDir, webhookID)
|
|
||||||
|
|
||||||
_, err = seed.ExecContext(
|
|
||||||
t.Context(),
|
|
||||||
"INSERT INTO targets "+
|
|
||||||
"(id, webhook_id, name, type, config) "+
|
|
||||||
"VALUES (?, '', ?, ?, ?)",
|
|
||||||
uuid.New().String(),
|
|
||||||
"leaked-target",
|
|
||||||
"slack",
|
|
||||||
fmt.Sprintf(
|
|
||||||
`{"webhookUrl":"https://hooks.example/%s"}`, credential,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
require.NoError(t, err)
|
|
||||||
clearEventDBSweptMarker(t, seed)
|
|
||||||
require.NoError(t, seed.Close())
|
|
||||||
|
|
||||||
// The seed has to be in the file for its absence later to mean
|
|
||||||
// anything.
|
|
||||||
require.True(
|
|
||||||
t,
|
|
||||||
bytes.Contains(
|
|
||||||
eventDBFileBytes(t, dataDir, webhookID),
|
|
||||||
[]byte(credential),
|
|
||||||
),
|
|
||||||
"seeded credential is not in the file, so this test proves nothing",
|
|
||||||
)
|
|
||||||
|
|
||||||
mgr := database.NewTestWebhookDBManager(dataDir)
|
|
||||||
|
|
||||||
_, err = mgr.GetDB(webhookID)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NoError(t, mgr.CloseAll())
|
|
||||||
|
|
||||||
assert.NotContains(
|
|
||||||
t,
|
|
||||||
string(eventDBFileBytes(t, dataDir, webhookID)),
|
|
||||||
credential,
|
|
||||||
"leaked credential is still recoverable from the raw file",
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOpenRevacuumsAfterIncompleteSweep covers the case a row count
|
|
||||||
// cannot see: the rows are already deleted but the file was never
|
|
||||||
// vacuumed, because an earlier sweep died between the two or its
|
|
||||||
// VACUUM failed. The credential bytes are still recoverable, and the
|
|
||||||
// unset marker is the only thing that says so, so the next open must
|
|
||||||
// vacuum rather than conclude from the empty table that there is
|
|
||||||
// nothing to do.
|
|
||||||
func TestOpenRevacuumsAfterIncompleteSweep(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dataDir := eventDBDataDir(t)
|
|
||||||
webhookID := uuid.New().String()
|
|
||||||
credential := "T00000000/B00000000/" + uuid.New().String()
|
|
||||||
|
|
||||||
initial := database.NewTestWebhookDBManager(dataDir)
|
|
||||||
|
|
||||||
_, err := initial.GetDB(webhookID)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NoError(t, initial.CloseAll())
|
|
||||||
|
|
||||||
seed := openRawEventDB(t, dataDir, webhookID)
|
|
||||||
|
|
||||||
_, err = seed.ExecContext(
|
|
||||||
t.Context(),
|
|
||||||
"INSERT INTO targets "+
|
|
||||||
"(id, webhook_id, name, type, config) "+
|
|
||||||
"VALUES (?, '', ?, ?, ?)",
|
|
||||||
uuid.New().String(),
|
|
||||||
"leaked-target",
|
|
||||||
"slack",
|
|
||||||
fmt.Sprintf(
|
|
||||||
`{"webhookUrl":"https://hooks.example/%s"}`, credential,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
// Exactly the state an interrupted sweep leaves: rows gone,
|
|
||||||
// marker unset, bytes still in the free pages.
|
|
||||||
_, err = seed.ExecContext(t.Context(), "DELETE FROM targets")
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.Zero(t, countTargetRows(t, seed))
|
|
||||||
clearEventDBSweptMarker(t, seed)
|
|
||||||
require.NoError(t, seed.Close())
|
|
||||||
|
|
||||||
require.True(
|
|
||||||
t,
|
|
||||||
bytes.Contains(
|
|
||||||
eventDBFileBytes(t, dataDir, webhookID),
|
|
||||||
[]byte(credential),
|
|
||||||
),
|
|
||||||
"the deleted row's bytes must still be in the file, or this "+
|
|
||||||
"test proves nothing",
|
|
||||||
)
|
|
||||||
|
|
||||||
mgr := database.NewTestWebhookDBManager(dataDir)
|
|
||||||
|
|
||||||
_, err = mgr.GetDB(webhookID)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NoError(t, mgr.CloseAll())
|
|
||||||
|
|
||||||
assert.NotContains(
|
|
||||||
t,
|
|
||||||
string(eventDBFileBytes(t, dataDir, webhookID)),
|
|
||||||
credential,
|
|
||||||
"an interrupted sweep was not retried, so the credential is "+
|
|
||||||
"still recoverable from the raw file",
|
|
||||||
)
|
|
||||||
|
|
||||||
check := openRawEventDB(t, dataDir, webhookID)
|
|
||||||
assert.Equal(t, 1, eventDBUserVersion(t, check))
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOpenSkipsSweptDatabase covers the other half of the marker: a
|
|
||||||
// file this build created is marked without ever being vacuumed, and
|
|
||||||
// a marked file is not swept again.
|
|
||||||
func TestOpenSkipsSweptDatabase(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dataDir := eventDBDataDir(t)
|
|
||||||
webhookID := uuid.New().String()
|
|
||||||
|
|
||||||
mgr := database.NewTestWebhookDBManager(dataDir)
|
|
||||||
|
|
||||||
_, err := mgr.GetDB(webhookID)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NoError(t, mgr.CloseAll())
|
|
||||||
|
|
||||||
marked := openRawEventDB(t, dataDir, webhookID)
|
|
||||||
assert.Equal(t, 1, eventDBUserVersion(t, marked))
|
|
||||||
|
|
||||||
// A marked file is left alone, so a row written into it survives
|
|
||||||
// a reopen. Nothing writes target rows any more; this stands in
|
|
||||||
// for the sweep having run.
|
|
||||||
_, err = marked.ExecContext(
|
|
||||||
t.Context(),
|
|
||||||
"INSERT INTO targets "+
|
|
||||||
"(id, webhook_id, name, type, config) "+
|
|
||||||
"VALUES (?, '', ?, ?, ?)",
|
|
||||||
uuid.New().String(), "sentinel", "slack", `{}`,
|
|
||||||
)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NoError(t, marked.Close())
|
|
||||||
|
|
||||||
again := database.NewTestWebhookDBManager(dataDir)
|
|
||||||
|
|
||||||
_, err = again.GetDB(webhookID)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NoError(t, again.CloseAll())
|
|
||||||
|
|
||||||
check := openRawEventDB(t, dataDir, webhookID)
|
|
||||||
assert.Equal(
|
|
||||||
t, 1, countTargetRows(t, check),
|
|
||||||
"a marked file must not be swept again",
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestOpenSucceedsWithoutTargetsTable covers an existing event
|
|
||||||
// database that never grew a targets table. The sweep must not fail
|
|
||||||
// startup on it.
|
|
||||||
func TestOpenSucceedsWithoutTargetsTable(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dataDir := eventDBDataDir(t)
|
|
||||||
webhookID := uuid.New().String()
|
|
||||||
|
|
||||||
seed := openRawEventDB(t, dataDir, webhookID)
|
|
||||||
|
|
||||||
_, err := seed.ExecContext(
|
|
||||||
t.Context(),
|
|
||||||
"CREATE TABLE events (id text PRIMARY KEY)",
|
|
||||||
)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NoError(t, seed.Close())
|
|
||||||
|
|
||||||
mgr := database.NewTestWebhookDBManager(dataDir)
|
|
||||||
|
|
||||||
db, err := mgr.GetDB(webhookID)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.NotNil(t, db)
|
|
||||||
require.NoError(t, mgr.CloseAll())
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestEventDBCreateOmitsAssociations covers the connection-level
|
// TestEventDBCreateOmitsAssociations covers the connection-level
|
||||||
// guard directly: a Delivery carrying its Event and Target in
|
// guard directly: a Delivery carrying its Event and Target in
|
||||||
// memory, written through the manager's handle, must store only the
|
// memory, written through the manager's handle, must store only the
|
||||||
@@ -393,10 +17,9 @@ func TestOpenSucceedsWithoutTargetsTable(t *testing.T) {
|
|||||||
func TestEventDBCreateOmitsAssociations(t *testing.T) {
|
func TestEventDBCreateOmitsAssociations(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
dataDir := eventDBDataDir(t)
|
|
||||||
webhookID := uuid.New().String()
|
webhookID := uuid.New().String()
|
||||||
|
|
||||||
mgr := database.NewTestWebhookDBManager(dataDir)
|
mgr := databasetest.NewWebhookDBManager(t, t.TempDir())
|
||||||
|
|
||||||
db, err := mgr.GetDB(webhookID)
|
db, err := mgr.GetDB(webhookID)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
@@ -431,8 +54,9 @@ func TestEventDBCreateOmitsAssociations(t *testing.T) {
|
|||||||
require.NoError(t, db.Model(d).
|
require.NoError(t, db.Model(d).
|
||||||
Update("status", database.DeliveryStatusDelivered).
|
Update("status", database.DeliveryStatusDelivered).
|
||||||
Error)
|
Error)
|
||||||
require.NoError(t, mgr.CloseAll())
|
|
||||||
|
|
||||||
check := openRawEventDB(t, dataDir, webhookID)
|
// The file has no targets table, so the create and update above
|
||||||
assert.Zero(t, countTargetRows(t, check))
|
// would have failed had GORM tried to save the target.
|
||||||
|
assert.False(t, db.Migrator().HasTable(&database.Target{}))
|
||||||
|
require.NoError(t, mgr.CloseAll())
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -13,10 +13,8 @@ import (
|
|||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
)
|
)
|
||||||
|
|
||||||
// TestWebhookDBManager_OpenAddsEventTierIndexes verifies that opening a
|
// TestWebhookDBManager_OpenAddsEventTierIndexes verifies that a new
|
||||||
// per-webhook database that predates these indexes creates them. It
|
// per-webhook database has these indexes.
|
||||||
// stands in for an older database file by dropping the indexes
|
|
||||||
// AutoMigrate just created, then reopening the same file.
|
|
||||||
func TestWebhookDBManager_OpenAddsEventTierIndexes(t *testing.T) {
|
func TestWebhookDBManager_OpenAddsEventTierIndexes(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -42,27 +40,9 @@ func TestWebhookDBManager_OpenAddsEventTierIndexes(t *testing.T) {
|
|||||||
db, err := mgr.GetDB(webhookID)
|
db, err := mgr.GetDB(webhookID)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
// A fresh database has them.
|
|
||||||
for _, ix := range indexes {
|
|
||||||
require.True(t, db.Migrator().HasIndex(ix.model, ix.name))
|
|
||||||
}
|
|
||||||
|
|
||||||
// Stand in for a database file created before the indexes existed.
|
|
||||||
for _, ix := range indexes {
|
|
||||||
require.NoError(t, db.Migrator().DropIndex(ix.model, ix.name))
|
|
||||||
require.False(t, db.Migrator().HasIndex(ix.model, ix.name))
|
|
||||||
}
|
|
||||||
|
|
||||||
// Drop the cached connection so the next open reopens the file and
|
|
||||||
// runs AutoMigrate against it, as a restart would.
|
|
||||||
require.NoError(t, mgr.CloseAll())
|
|
||||||
|
|
||||||
db, err = mgr.GetDB(webhookID)
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
for _, ix := range indexes {
|
for _, ix := range indexes {
|
||||||
assert.True(t, db.Migrator().HasIndex(ix.model, ix.name),
|
assert.True(t, db.Migrator().HasIndex(ix.model, ix.name),
|
||||||
"opening the existing database should create %s", ix.name)
|
"a new database should have %s", ix.name)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -45,8 +45,7 @@ type Event struct {
|
|||||||
|
|
||||||
// ResubmittedFromID names the event this one was copied from by
|
// ResubmittedFromID names the event this one was copied from by
|
||||||
// an operator resubmit. It is nil for an event that arrived on
|
// an operator resubmit. It is nil for an event that arrived on
|
||||||
// the receiver, which is every event created before the column
|
// the receiver. It is not a foreign key: the source event can be
|
||||||
// existed. It is not a foreign key: the source event can be
|
|
||||||
// reaped by retention while its copies remain, and the id is
|
// reaped by retention while its copies remain, and the id is
|
||||||
// kept as the record of where the copy came from either way.
|
// kept as the record of where the copy came from either way.
|
||||||
ResubmittedFromID *string `gorm:"type:uuid;index:idx_events_resubmitted_from_id,priority:1;index:idx_events_entrypoint_id,priority:3" json:"resubmittedFromId,omitempty"`
|
ResubmittedFromID *string `gorm:"type:uuid;index:idx_events_resubmitted_from_id,priority:1;index:idx_events_entrypoint_id,priority:3" json:"resubmittedFromId,omitempty"`
|
||||||
|
|||||||
@@ -82,7 +82,7 @@ type Webhook struct {
|
|||||||
// value while building the insert statement, which runs after
|
// value while building the insert statement, which runs after
|
||||||
// BeforeSave; rewriting any later than this loses that race and the
|
// BeforeSave; rewriting any later than this loses that race and the
|
||||||
// row lands at 30 days. Living on the model also means a future call
|
// row lands at 30 days. Living on the model also means a future call
|
||||||
// site — a REST API, a fixture, a migration — cannot bypass it.
|
// site, such as a REST API or a fixture, cannot bypass it.
|
||||||
func (w *Webhook) BeforeSave(_ *gorm.DB) error {
|
func (w *Webhook) BeforeSave(_ *gorm.DB) error {
|
||||||
if w.RetentionDays <= 0 {
|
if w.RetentionDays <= 0 {
|
||||||
w.RetentionDays = RetentionForeverDays
|
w.RetentionDays = RetentionForeverDays
|
||||||
@@ -98,8 +98,8 @@ func (w *Webhook) BeforeSave(_ *gorm.DB) error {
|
|||||||
// are exempt from reaping.
|
// are exempt from reaping.
|
||||||
//
|
//
|
||||||
// It accepts the RetentionForeverDays sentinel written by BeforeSave
|
// It accepts the RetentionForeverDays sentinel written by BeforeSave
|
||||||
// and, defensively, the non-positive values that rows written before
|
// and, defensively, any non-positive value, which BeforeSave treats
|
||||||
// the sentinel existed may still carry.
|
// as a request to retain forever.
|
||||||
func retainsForever(retentionDays int) bool {
|
func retainsForever(retentionDays int) bool {
|
||||||
return retentionDays <= 0 ||
|
return retentionDays <= 0 ||
|
||||||
retentionDays >= RetentionForeverDays
|
retentionDays >= RetentionForeverDays
|
||||||
|
|||||||
@@ -1,17 +0,0 @@
|
|||||||
package database
|
|
||||||
|
|
||||||
// Migrate runs database migrations for the main application database.
|
|
||||||
// Only configuration-tier models are stored in the main database.
|
|
||||||
// Event-tier models (Event, Delivery, DeliveryResult, EventTotals,
|
|
||||||
// TargetTotals, EntrypointTotals) live in
|
|
||||||
// per-webhook dedicated databases managed by WebhookDBManager.
|
|
||||||
func (d *Database) Migrate() error {
|
|
||||||
return d.db.AutoMigrate(
|
|
||||||
&Setting{},
|
|
||||||
&User{},
|
|
||||||
&APIKey{},
|
|
||||||
&Webhook{},
|
|
||||||
&Entrypoint{},
|
|
||||||
&Target{},
|
|
||||||
)
|
|
||||||
}
|
|
||||||
@@ -253,7 +253,7 @@ func (r *RetentionReaper) reapWebhook(
|
|||||||
// row in the database: the sweep then deletes every event, delivery,
|
// row in the database: the sweep then deletes every event, delivery,
|
||||||
// and delivery result, including ones created seconds ago. Rejecting
|
// and delivery result, including ones created seconds ago. Rejecting
|
||||||
// out-of-range input at the form is the primary guard; saturating here
|
// out-of-range input at the form is the primary guard; saturating here
|
||||||
// means an old row, a migration, or a future call site cannot turn a
|
// means a value that reaches the row by any other path cannot turn a
|
||||||
// too-large retention into total data loss.
|
// too-large retention into total data loss.
|
||||||
func retentionCutoff(
|
func retentionCutoff(
|
||||||
now time.Time,
|
now time.Time,
|
||||||
|
|||||||
@@ -51,7 +51,7 @@ func setupRetentionTest(t *testing.T) *retentionTestEnv {
|
|||||||
|
|
||||||
mgr, err := database.NewWebhookDBManager(
|
mgr, err := database.NewWebhookDBManager(
|
||||||
lc,
|
lc,
|
||||||
database.WebhookDBManagerParams{Config: cfg, Logger: l},
|
database.WebhookDBManagerParams{Config: cfg, Logger: l.Get()},
|
||||||
)
|
)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,168 @@
|
|||||||
|
package database_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"gorm.io/gorm"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||||
|
)
|
||||||
|
|
||||||
|
// withFixedNow returns db with GORM's clock stopped at a time in UTC,
|
||||||
|
// so that the created and updated times it writes come back from
|
||||||
|
// SQLite equal to the row written, which holds them in UTC.
|
||||||
|
func withFixedNow(db *gorm.DB) (*gorm.DB, time.Time) {
|
||||||
|
at := time.Date(2026, 10, 13, 12, 30, 45, 123456789, time.UTC)
|
||||||
|
|
||||||
|
return db.Session(&gorm.Session{
|
||||||
|
NowFunc: func() time.Time { return at },
|
||||||
|
}), at
|
||||||
|
}
|
||||||
|
|
||||||
|
// newBaseModel returns a BaseModel with a new ID.
|
||||||
|
func newBaseModel() database.BaseModel {
|
||||||
|
return database.BaseModel{ID: uuid.New().String()}
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertRoundTrip writes row, which must be the only row of its
|
||||||
|
// table, and reads back the same row.
|
||||||
|
func assertRoundTrip[T any](t *testing.T, db *gorm.DB, row *T) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
require.NoError(t, db.Create(row).Error)
|
||||||
|
|
||||||
|
var got T
|
||||||
|
|
||||||
|
require.NoError(t, db.First(&got).Error)
|
||||||
|
assert.Equal(t, *row, got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestMainSchemaHoldsEveryModel writes a row of each main database
|
||||||
|
// model, every field set, into a database made from its SQL files.
|
||||||
|
func TestMainSchemaHoldsEveryModel(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
mainDB, err := database.Open(
|
||||||
|
t.TempDir(), slog.New(slog.DiscardHandler),
|
||||||
|
)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
t.Cleanup(func() { _ = mainDB.Close() })
|
||||||
|
|
||||||
|
db, at := withFixedNow(mainDB.DB())
|
||||||
|
|
||||||
|
assertRoundTrip(t, db, &database.Setting{
|
||||||
|
Key: "session_key", Value: "value",
|
||||||
|
})
|
||||||
|
assertRoundTrip(t, db, &database.User{
|
||||||
|
BaseModel: newBaseModel(),
|
||||||
|
Username: "user",
|
||||||
|
Password: "password hash",
|
||||||
|
})
|
||||||
|
assertRoundTrip(t, db, &database.APIKey{
|
||||||
|
BaseModel: newBaseModel(),
|
||||||
|
UserID: uuid.New().String(),
|
||||||
|
Key: "key",
|
||||||
|
Description: "API key description",
|
||||||
|
LastUsedAt: &at,
|
||||||
|
})
|
||||||
|
assertRoundTrip(t, db, &database.Webhook{
|
||||||
|
BaseModel: newBaseModel(),
|
||||||
|
UserID: uuid.New().String(),
|
||||||
|
Name: "webhook",
|
||||||
|
Description: "webhook description",
|
||||||
|
RetentionDays: 7,
|
||||||
|
})
|
||||||
|
assertRoundTrip(t, db, &database.Entrypoint{
|
||||||
|
BaseModel: newBaseModel(),
|
||||||
|
WebhookID: uuid.New().String(),
|
||||||
|
Path: uuid.New().String(),
|
||||||
|
Description: "entrypoint description",
|
||||||
|
Active: true,
|
||||||
|
})
|
||||||
|
assertRoundTrip(t, db, &database.Target{
|
||||||
|
BaseModel: newBaseModel(),
|
||||||
|
WebhookID: uuid.New().String(),
|
||||||
|
Name: "target",
|
||||||
|
Type: database.TargetTypeHTTP,
|
||||||
|
Active: true,
|
||||||
|
Config: `{"url":"https://example.com/"}`,
|
||||||
|
MaxRetries: 3,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestEventSchemaHoldsEveryModel does the same for a webhook's event
|
||||||
|
// database, which the manager makes from its SQL files.
|
||||||
|
func TestEventSchemaHoldsEveryModel(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
mgr := databasetest.NewWebhookDBManager(t, t.TempDir())
|
||||||
|
|
||||||
|
t.Cleanup(func() { _ = mgr.CloseAll() })
|
||||||
|
|
||||||
|
events, err := mgr.GetDB(uuid.New().String())
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
db, at := withFixedNow(events)
|
||||||
|
source := uuid.New().String()
|
||||||
|
|
||||||
|
assertRoundTrip(t, db, &database.Event{
|
||||||
|
BaseModel: newBaseModel(),
|
||||||
|
WebhookID: uuid.New().String(),
|
||||||
|
EntrypointID: uuid.New().String(),
|
||||||
|
Method: http.MethodPost,
|
||||||
|
RawQuery: "a=1",
|
||||||
|
Headers: `{"X":"Y"}`,
|
||||||
|
Body: "body",
|
||||||
|
ContentType: "text/plain",
|
||||||
|
BodyBytes: 4,
|
||||||
|
ResubmittedFromID: &source,
|
||||||
|
})
|
||||||
|
assertRoundTrip(t, db, &database.Delivery{
|
||||||
|
BaseModel: newBaseModel(),
|
||||||
|
EventID: uuid.New().String(),
|
||||||
|
TargetID: uuid.New().String(),
|
||||||
|
Status: database.DeliveryStatusDelivered,
|
||||||
|
FinishedAt: &at,
|
||||||
|
Replay: true,
|
||||||
|
})
|
||||||
|
assertRoundTrip(t, db, &database.DeliveryResult{
|
||||||
|
BaseModel: newBaseModel(),
|
||||||
|
DeliveryID: uuid.New().String(),
|
||||||
|
AttemptNum: 2,
|
||||||
|
Success: true,
|
||||||
|
StatusCode: 200,
|
||||||
|
ResponseBody: "ok",
|
||||||
|
Error: "error",
|
||||||
|
Duration: 15,
|
||||||
|
})
|
||||||
|
assertRoundTrip(t, db, &database.TargetTotals{
|
||||||
|
TargetID: uuid.New().String(),
|
||||||
|
Deliveries: 5,
|
||||||
|
Delivered: 3,
|
||||||
|
Failed: 1,
|
||||||
|
DeliveriesRemoved: 2,
|
||||||
|
FailedRemoved: 1,
|
||||||
|
})
|
||||||
|
assertRoundTrip(t, db, &database.EntrypointTotals{
|
||||||
|
EntrypointID: uuid.New().String(), LastEventAt: at,
|
||||||
|
})
|
||||||
|
|
||||||
|
// A new event database has its one row of event totals already.
|
||||||
|
require.NoError(t, database.AddEventTotals(db, database.EventTotals{
|
||||||
|
Events: 3, EventsRemoved: 1, LastEventAt: &at,
|
||||||
|
}))
|
||||||
|
|
||||||
|
var totals []database.EventTotals
|
||||||
|
|
||||||
|
require.NoError(t, db.Find(&totals).Error)
|
||||||
|
assert.Equal(t, []database.EventTotals{{
|
||||||
|
ID: 1, Events: 3, EventsRemoved: 1, LastEventAt: &at,
|
||||||
|
}}, totals)
|
||||||
|
}
|
||||||
@@ -118,9 +118,8 @@ const SQLiteFilePerm fs.FileMode = 0o600
|
|||||||
// SQLite still reports the absence rather than this function
|
// SQLite still reports the absence rather than this function
|
||||||
// materializing an empty database the caller asked not to create.
|
// materializing an empty database the caller asked not to create.
|
||||||
//
|
//
|
||||||
// Chmod of a file that already exists is what tightens a data
|
// The mode is set on every open, whoever created the file, so a
|
||||||
// directory an earlier build left at 0644 — including a developer's
|
// database or sidecar found at a wider mode is tightened too.
|
||||||
// own scratch directory — without any migration machinery.
|
|
||||||
func reserveSQLiteFile(path string, create bool) error {
|
func reserveSQLiteFile(path string, create bool) error {
|
||||||
if create {
|
if create {
|
||||||
// gosec G304: the path is the database file the caller asked
|
// gosec G304: the path is the database file the caller asked
|
||||||
@@ -242,7 +241,19 @@ func SQLiteDSN(path, mode string) string {
|
|||||||
)
|
)
|
||||||
q.Add("_pragma", "journal_mode(WAL)")
|
q.Add("_pragma", "journal_mode(WAL)")
|
||||||
|
|
||||||
return "file:" + path + "?" + q.Encode()
|
// SQLite reads a file: URI's path only up to the first ? or #, and
|
||||||
|
// decodes %XX escapes in it, so url.URL escapes the path. OmitHost
|
||||||
|
// keeps a relative path as "file:data/x.db"; written
|
||||||
|
// "file://data/x.db", SQLite would reject "data" as the URI's
|
||||||
|
// authority.
|
||||||
|
dsn := url.URL{
|
||||||
|
Scheme: "file",
|
||||||
|
Path: path,
|
||||||
|
OmitHost: true,
|
||||||
|
RawQuery: q.Encode(),
|
||||||
|
}
|
||||||
|
|
||||||
|
return dsn.String()
|
||||||
}
|
}
|
||||||
|
|
||||||
// OpenSQLite opens the SQLite file at path with the service's
|
// OpenSQLite opens the SQLite file at path with the service's
|
||||||
|
|||||||
@@ -2,6 +2,8 @@ package database_test
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -112,6 +114,79 @@ func TestMainDBAppliesPragmasOnALiveHandle(t *testing.T) {
|
|||||||
assert.Equal(t, "10000", busy)
|
assert.Equal(t, "10000", busy)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestOpenSQLiteOpensThePathAskedFor uses a data directory whose name
|
||||||
|
// holds ?, # and %. SQLite reads a file: URI's path only up to a ? or
|
||||||
|
// #, and decodes %41 to A, so an unescaped path sent every database in
|
||||||
|
// that directory to one file outside it.
|
||||||
|
func TestOpenSQLiteOpensThePathAskedFor(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := filepath.Join(t.TempDir(), "hooks#1?x%41")
|
||||||
|
require.NoError(t, os.Mkdir(dir, 0o700))
|
||||||
|
|
||||||
|
for _, name := range []string{
|
||||||
|
database.MainDBFileName,
|
||||||
|
"events-" + uuid.New().String() + ".db",
|
||||||
|
} {
|
||||||
|
path := filepath.Join(dir, name)
|
||||||
|
|
||||||
|
sqlDB, err := database.OpenSQLite(
|
||||||
|
path, database.SQLiteModeCreate,
|
||||||
|
)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
t.Cleanup(func() { assert.NoError(t, sqlDB.Close()) })
|
||||||
|
|
||||||
|
assertOpenedFile(t, sqlDB, path)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestOpenSQLiteOpensARelativePath opens a database at a path relative
|
||||||
|
// to the working directory, which is what a relative DATA_DIR such as
|
||||||
|
// the README's DATA_DIR=../webhooker-data gives OpenSQLite.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // t.Chdir moves the whole process.
|
||||||
|
func TestOpenSQLiteOpensARelativePath(t *testing.T) {
|
||||||
|
t.Chdir(t.TempDir())
|
||||||
|
require.NoError(t, os.Mkdir("data", 0o700))
|
||||||
|
|
||||||
|
path := filepath.Join("data", database.MainDBFileName)
|
||||||
|
|
||||||
|
sqlDB, err := database.OpenSQLite(path, database.SQLiteModeCreate)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
defer func() { require.NoError(t, sqlDB.Close()) }()
|
||||||
|
|
||||||
|
assertOpenedFile(t, sqlDB, path)
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertOpenedFile checks that sqlDB is the SQLite file at path.
|
||||||
|
// database_list names the file SQLite actually opened, as an absolute
|
||||||
|
// path with symlinks resolved, so the files are compared, not the names.
|
||||||
|
func assertOpenedFile(t *testing.T, sqlDB *sql.DB, path string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var (
|
||||||
|
seq int
|
||||||
|
schema, file string
|
||||||
|
)
|
||||||
|
|
||||||
|
require.NoError(t, sqlDB.
|
||||||
|
QueryRowContext(t.Context(), "pragma database_list").
|
||||||
|
Scan(&seq, &schema, &file))
|
||||||
|
|
||||||
|
asked, err := os.Stat(path)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
opened, err := os.Stat(file)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
assert.True(
|
||||||
|
t, os.SameFile(asked, opened),
|
||||||
|
"asked for %s, SQLite opened %s", path, file,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
// TestConcurrentReaderDoesNotBlockWrites is the unit-scale form of the
|
// TestConcurrentReaderDoesNotBlockWrites is the unit-scale form of the
|
||||||
// reproduction in
|
// reproduction in
|
||||||
// https://git.eeqj.de/sneak/webhooker/issues/256: an operator's
|
// https://git.eeqj.de/sneak/webhooker/issues/256: an operator's
|
||||||
|
|||||||
@@ -1,47 +0,0 @@
|
|||||||
package database
|
|
||||||
|
|
||||||
import (
|
|
||||||
"log/slog"
|
|
||||||
"os"
|
|
||||||
|
|
||||||
"gorm.io/gorm"
|
|
||||||
)
|
|
||||||
|
|
||||||
// NewTestDatabase creates a Database wrapper around a pre-opened *gorm.DB.
|
|
||||||
// Intended for use in tests that need a *database.Database without the
|
|
||||||
// full fx lifecycle. The caller is responsible for closing the underlying
|
|
||||||
// sql.DB connection.
|
|
||||||
func NewTestDatabase(db *gorm.DB) *Database {
|
|
||||||
return &Database{
|
|
||||||
db: db,
|
|
||||||
log: slog.New(slog.NewTextHandler(
|
|
||||||
os.Stderr,
|
|
||||||
&slog.HandlerOptions{Level: slog.LevelDebug},
|
|
||||||
)),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// NewTestWebhookDBManager creates a WebhookDBManager backed by the given
|
|
||||||
// data directory. Intended for use in tests without the fx lifecycle.
|
|
||||||
func NewTestWebhookDBManager(dataDir string) *WebhookDBManager {
|
|
||||||
return NewTestWebhookDBManagerWithLogger(
|
|
||||||
dataDir,
|
|
||||||
slog.New(slog.NewTextHandler(
|
|
||||||
os.Stderr,
|
|
||||||
&slog.HandlerOptions{Level: slog.LevelDebug},
|
|
||||||
)),
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
// NewTestWebhookDBManagerWithLogger is NewTestWebhookDBManager with the
|
|
||||||
// logger supplied by the caller. The per-webhook databases this manager
|
|
||||||
// opens hand that logger to gormlog, so a test that needs to see the SQL
|
|
||||||
// the service emits can capture it.
|
|
||||||
func NewTestWebhookDBManagerWithLogger(
|
|
||||||
dataDir string, log *slog.Logger,
|
|
||||||
) *WebhookDBManager {
|
|
||||||
return &WebhookDBManager{
|
|
||||||
dataDir: dataDir,
|
|
||||||
log: log,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -14,8 +14,8 @@ import (
|
|||||||
"gorm.io/gorm"
|
"gorm.io/gorm"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
"sneak.berlin/go/webhooker/internal/datadir"
|
"sneak.berlin/go/webhooker/internal/datadir"
|
||||||
|
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||||
"sneak.berlin/go/webhooker/internal/logger"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// WebhookDBManagerParams holds the fx dependencies for
|
// WebhookDBManagerParams holds the fx dependencies for
|
||||||
@@ -24,7 +24,7 @@ type WebhookDBManagerParams struct {
|
|||||||
fx.In
|
fx.In
|
||||||
|
|
||||||
Config *config.Config
|
Config *config.Config
|
||||||
Logger *logger.Logger
|
Logger *slog.Logger
|
||||||
}
|
}
|
||||||
|
|
||||||
// errInvalidCachedDBType indicates a type assertion failure
|
// errInvalidCachedDBType indicates a type assertion failure
|
||||||
@@ -70,7 +70,7 @@ func NewWebhookDBManager(
|
|||||||
) (*WebhookDBManager, error) {
|
) (*WebhookDBManager, error) {
|
||||||
m := &WebhookDBManager{
|
m := &WebhookDBManager{
|
||||||
dataDir: params.Config.DataDir,
|
dataDir: params.Config.DataDir,
|
||||||
log: params.Logger.Get(),
|
log: params.Logger,
|
||||||
}
|
}
|
||||||
|
|
||||||
// Create data directory if it doesn't exist. datadir.DirPerm is the
|
// Create data directory if it doesn't exist. datadir.DirPerm is the
|
||||||
@@ -371,18 +371,7 @@ func (m *WebhookDBManager) openDB(
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
err = purgeTargetRows(db, m.log, webhookID)
|
err = migrations.Apply(db, migrations.Events)
|
||||||
if err != nil {
|
|
||||||
_ = sqlDB.Close()
|
|
||||||
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
// Run migrations for event-tier models only
|
|
||||||
err = db.AutoMigrate(
|
|
||||||
&Event{}, &Delivery{}, &DeliveryResult{},
|
|
||||||
&EventTotals{}, &TargetTotals{}, &EntrypointTotals{},
|
|
||||||
)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
_ = sqlDB.Close()
|
_ = sqlDB.Close()
|
||||||
|
|
||||||
@@ -392,14 +381,15 @@ func (m *WebhookDBManager) openDB(
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// A new database gets its row of event totals, all zero. Target
|
err = parseModels(
|
||||||
// totals rows are created by the first delivery to each target.
|
db, &Event{}, &Delivery{}, &DeliveryResult{},
|
||||||
err = db.FirstOrCreate(&EventTotals{}).Error
|
&EventTotals{}, &TargetTotals{}, &EntrypointTotals{},
|
||||||
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
_ = sqlDB.Close()
|
_ = sqlDB.Close()
|
||||||
|
|
||||||
return nil, fmt.Errorf(
|
return nil, fmt.Errorf(
|
||||||
"creating event totals for webhook database %s: %w",
|
"connecting to webhook database %s: %w",
|
||||||
webhookID, err,
|
webhookID, err,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ import (
|
|||||||
"gorm.io/gorm"
|
"gorm.io/gorm"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||||
"sneak.berlin/go/webhooker/internal/globals"
|
"sneak.berlin/go/webhooker/internal/globals"
|
||||||
"sneak.berlin/go/webhooker/internal/logger"
|
"sneak.berlin/go/webhooker/internal/logger"
|
||||||
)
|
)
|
||||||
@@ -50,7 +51,7 @@ func setupTestWebhookDBManager(
|
|||||||
lc,
|
lc,
|
||||||
database.WebhookDBManagerParams{
|
database.WebhookDBManagerParams{
|
||||||
Config: cfg,
|
Config: cfg,
|
||||||
Logger: l,
|
Logger: l.Get(),
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
@@ -117,7 +118,8 @@ func TestWebhookDBManager_ConcurrentFirstTouchOpensOnce(t *testing.T) {
|
|||||||
|
|
||||||
var logs bytes.Buffer
|
var logs bytes.Buffer
|
||||||
|
|
||||||
mgr := database.NewTestWebhookDBManagerWithLogger(
|
mgr := databasetest.NewWebhookDBManagerWithLogger(
|
||||||
|
t,
|
||||||
t.TempDir(),
|
t.TempDir(),
|
||||||
slog.New(slog.NewTextHandler(&logs, nil)),
|
slog.New(slog.NewTextHandler(&logs, nil)),
|
||||||
)
|
)
|
||||||
@@ -307,7 +309,8 @@ func TestWebhookDBManager_LostDatabaseIsLogged(t *testing.T) {
|
|||||||
|
|
||||||
var logs bytes.Buffer
|
var logs bytes.Buffer
|
||||||
|
|
||||||
mgr := database.NewTestWebhookDBManagerWithLogger(
|
mgr := databasetest.NewWebhookDBManagerWithLogger(
|
||||||
|
t,
|
||||||
t.TempDir(),
|
t.TempDir(),
|
||||||
slog.New(slog.NewTextHandler(&logs, nil)),
|
slog.New(slog.NewTextHandler(&logs, nil)),
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -0,0 +1,9 @@
|
|||||||
|
-- Migration 000: the table recording which migrations this file has
|
||||||
|
-- had. Applied before the others; it records itself.
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||||
|
version INTEGER PRIMARY KEY,
|
||||||
|
applied_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
INSERT OR IGNORE INTO schema_migrations (version) VALUES (0);
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
-- Migration 001: the schema of a database target's archive file. The
|
||||||
|
-- archivedEvent model in internal/delivery must agree with it.
|
||||||
|
|
||||||
|
CREATE TABLE archived_events (
|
||||||
|
id integer PRIMARY KEY AUTOINCREMENT,
|
||||||
|
event_id text,
|
||||||
|
webhook_id text,
|
||||||
|
entrypoint_id text,
|
||||||
|
method text,
|
||||||
|
raw_query text,
|
||||||
|
headers text,
|
||||||
|
body text,
|
||||||
|
content_type text,
|
||||||
|
archived_at datetime
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX idx_archived_events_event_id ON archived_events (event_id);
|
||||||
|
CREATE INDEX idx_archived_events_archived_at ON archived_events (archived_at);
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
-- Migration 000: the table recording which migrations this file has
|
||||||
|
-- had. Applied before the others; it records itself.
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||||
|
version INTEGER PRIMARY KEY,
|
||||||
|
applied_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
INSERT OR IGNORE INTO schema_migrations (version) VALUES (0);
|
||||||
@@ -0,0 +1,98 @@
|
|||||||
|
-- Migration 001: the schema of a webhook's event database. The models
|
||||||
|
-- in internal/database must agree with it. The webhook, entrypoint and
|
||||||
|
-- target ids here name rows in the main database, which has those
|
||||||
|
-- tables; this file does not.
|
||||||
|
|
||||||
|
CREATE TABLE events (
|
||||||
|
id uuid,
|
||||||
|
created_at datetime,
|
||||||
|
updated_at datetime,
|
||||||
|
deleted_at datetime,
|
||||||
|
webhook_id uuid NOT NULL,
|
||||||
|
entrypoint_id uuid NOT NULL,
|
||||||
|
method text NOT NULL,
|
||||||
|
raw_query text,
|
||||||
|
headers text,
|
||||||
|
body text,
|
||||||
|
content_type text,
|
||||||
|
body_bytes integer NOT NULL,
|
||||||
|
resubmitted_from_id uuid,
|
||||||
|
PRIMARY KEY (id)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX idx_events_created_at ON events (created_at);
|
||||||
|
CREATE INDEX idx_events_deleted_at ON events (deleted_at);
|
||||||
|
CREATE INDEX idx_events_deleted_at_created_at
|
||||||
|
ON events (deleted_at, created_at);
|
||||||
|
CREATE INDEX idx_events_resubmitted_from_id
|
||||||
|
ON events (resubmitted_from_id, deleted_at);
|
||||||
|
CREATE INDEX idx_events_entrypoint_id
|
||||||
|
ON events (entrypoint_id, deleted_at, resubmitted_from_id, created_at);
|
||||||
|
|
||||||
|
CREATE TABLE deliveries (
|
||||||
|
id uuid,
|
||||||
|
created_at datetime,
|
||||||
|
updated_at datetime,
|
||||||
|
deleted_at datetime,
|
||||||
|
event_id uuid NOT NULL,
|
||||||
|
target_id uuid NOT NULL,
|
||||||
|
status text NOT NULL DEFAULT 'pending',
|
||||||
|
finished_at datetime,
|
||||||
|
replay numeric NOT NULL DEFAULT false,
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
CONSTRAINT fk_events_deliveries
|
||||||
|
FOREIGN KEY (event_id) REFERENCES events (id)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX idx_deliveries_deleted_at ON deliveries (deleted_at);
|
||||||
|
CREATE INDEX idx_deliveries_event_id ON deliveries (event_id, deleted_at);
|
||||||
|
CREATE INDEX idx_deliveries_status
|
||||||
|
ON deliveries (status, deleted_at, finished_at, target_id);
|
||||||
|
|
||||||
|
CREATE TABLE delivery_results (
|
||||||
|
id uuid,
|
||||||
|
created_at datetime,
|
||||||
|
updated_at datetime,
|
||||||
|
deleted_at datetime,
|
||||||
|
delivery_id uuid NOT NULL,
|
||||||
|
attempt_num integer NOT NULL,
|
||||||
|
success numeric,
|
||||||
|
status_code integer,
|
||||||
|
response_body text,
|
||||||
|
error text,
|
||||||
|
duration integer,
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
CONSTRAINT fk_deliveries_delivery_results
|
||||||
|
FOREIGN KEY (delivery_id) REFERENCES deliveries (id)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX idx_delivery_results_deleted_at
|
||||||
|
ON delivery_results (deleted_at);
|
||||||
|
CREATE INDEX idx_delivery_results_delivery_id
|
||||||
|
ON delivery_results (delivery_id, deleted_at);
|
||||||
|
|
||||||
|
CREATE TABLE event_totals (
|
||||||
|
id integer PRIMARY KEY AUTOINCREMENT,
|
||||||
|
events integer NOT NULL,
|
||||||
|
events_removed integer NOT NULL,
|
||||||
|
last_event_at datetime
|
||||||
|
);
|
||||||
|
|
||||||
|
-- The one row of event totals, all zero.
|
||||||
|
INSERT INTO event_totals (events, events_removed) VALUES (0, 0);
|
||||||
|
|
||||||
|
CREATE TABLE target_totals (
|
||||||
|
target_id uuid,
|
||||||
|
deliveries integer NOT NULL,
|
||||||
|
delivered integer NOT NULL,
|
||||||
|
failed integer NOT NULL,
|
||||||
|
deliveries_removed integer NOT NULL,
|
||||||
|
failed_removed integer NOT NULL,
|
||||||
|
PRIMARY KEY (target_id)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE entrypoint_totals (
|
||||||
|
entrypoint_id uuid,
|
||||||
|
last_event_at datetime NOT NULL,
|
||||||
|
PRIMARY KEY (entrypoint_id)
|
||||||
|
);
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
-- Migration 000: the table recording which migrations this file has
|
||||||
|
-- had. Applied before the others; it records itself.
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||||
|
version INTEGER PRIMARY KEY,
|
||||||
|
applied_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
INSERT OR IGNORE INTO schema_migrations (version) VALUES (0);
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
-- Migration 001: the main database's schema. The models in
|
||||||
|
-- internal/database must agree with it.
|
||||||
|
|
||||||
|
CREATE TABLE settings (
|
||||||
|
"key" text,
|
||||||
|
value text NOT NULL,
|
||||||
|
PRIMARY KEY ("key")
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE users (
|
||||||
|
id uuid,
|
||||||
|
created_at datetime,
|
||||||
|
updated_at datetime,
|
||||||
|
deleted_at datetime,
|
||||||
|
username text NOT NULL,
|
||||||
|
password text NOT NULL,
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
CONSTRAINT chk_users_username
|
||||||
|
CHECK (length(CAST(username AS BLOB)) <= 1024)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX idx_users_deleted_at ON users (deleted_at);
|
||||||
|
CREATE UNIQUE INDEX idx_users_username ON users (username);
|
||||||
|
|
||||||
|
CREATE TABLE api_keys (
|
||||||
|
id uuid,
|
||||||
|
created_at datetime,
|
||||||
|
updated_at datetime,
|
||||||
|
deleted_at datetime,
|
||||||
|
user_id uuid NOT NULL,
|
||||||
|
"key" text NOT NULL,
|
||||||
|
description text,
|
||||||
|
last_used_at datetime,
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
CONSTRAINT fk_users_api_keys FOREIGN KEY (user_id) REFERENCES users (id)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX idx_api_keys_deleted_at ON api_keys (deleted_at);
|
||||||
|
CREATE UNIQUE INDEX idx_api_keys_key ON api_keys ("key");
|
||||||
|
|
||||||
|
CREATE TABLE webhooks (
|
||||||
|
id uuid,
|
||||||
|
created_at datetime,
|
||||||
|
updated_at datetime,
|
||||||
|
deleted_at datetime,
|
||||||
|
user_id uuid NOT NULL,
|
||||||
|
name text NOT NULL,
|
||||||
|
description text,
|
||||||
|
retention_days integer DEFAULT 30,
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
CONSTRAINT fk_users_webhooks FOREIGN KEY (user_id) REFERENCES users (id)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX idx_webhooks_deleted_at ON webhooks (deleted_at);
|
||||||
|
|
||||||
|
CREATE TABLE entrypoints (
|
||||||
|
id uuid,
|
||||||
|
created_at datetime,
|
||||||
|
updated_at datetime,
|
||||||
|
deleted_at datetime,
|
||||||
|
webhook_id uuid NOT NULL,
|
||||||
|
path text NOT NULL,
|
||||||
|
description text,
|
||||||
|
active numeric DEFAULT true,
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
CONSTRAINT fk_webhooks_entrypoints
|
||||||
|
FOREIGN KEY (webhook_id) REFERENCES webhooks (id)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX idx_entrypoints_deleted_at ON entrypoints (deleted_at);
|
||||||
|
CREATE UNIQUE INDEX idx_entrypoints_path ON entrypoints (path);
|
||||||
|
|
||||||
|
CREATE TABLE targets (
|
||||||
|
id uuid,
|
||||||
|
created_at datetime,
|
||||||
|
updated_at datetime,
|
||||||
|
deleted_at datetime,
|
||||||
|
webhook_id uuid NOT NULL,
|
||||||
|
name text NOT NULL,
|
||||||
|
type text NOT NULL,
|
||||||
|
active numeric DEFAULT true,
|
||||||
|
config text,
|
||||||
|
max_retries integer,
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
CONSTRAINT fk_webhooks_targets
|
||||||
|
FOREIGN KEY (webhook_id) REFERENCES webhooks (id)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX idx_targets_deleted_at ON targets (deleted_at);
|
||||||
@@ -0,0 +1,135 @@
|
|||||||
|
// Package migrations holds the schema of each kind of SQLite file
|
||||||
|
// webhooker writes, as numbered SQL files embedded in the binary, and
|
||||||
|
// applies it when a file is opened.
|
||||||
|
//
|
||||||
|
// Each kind of file has its own directory. In each, 000_migration.sql
|
||||||
|
// creates only the schema_migrations table, and 001_schema.sql holds
|
||||||
|
// the whole schema. Before 1.0 a schema change is an edit to
|
||||||
|
// 001_schema.sql, never a new file.
|
||||||
|
package migrations
|
||||||
|
|
||||||
|
import (
|
||||||
|
"embed"
|
||||||
|
"fmt"
|
||||||
|
"io/fs"
|
||||||
|
"path"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gorm.io/gorm"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The directories, one for each kind of database file.
|
||||||
|
const (
|
||||||
|
// Main is the main database, webhooker.db.
|
||||||
|
Main = "main"
|
||||||
|
|
||||||
|
// Events is a webhook's event database, events-{uuid}.db.
|
||||||
|
Events = "events"
|
||||||
|
|
||||||
|
// Archive is a database target's archive file.
|
||||||
|
Archive = "archive"
|
||||||
|
)
|
||||||
|
|
||||||
|
//go:embed main/*.sql events/*.sql archive/*.sql
|
||||||
|
var files embed.FS
|
||||||
|
|
||||||
|
// Apply runs each SQL file in dir, which is Main, Events or Archive,
|
||||||
|
// that the database has not had yet, in the order of the number its
|
||||||
|
// name starts with, and records that number in schema_migrations.
|
||||||
|
// 000_migration.sql creates that table and records itself, so it runs
|
||||||
|
// first, and only when the table is missing.
|
||||||
|
//
|
||||||
|
// It all runs in one transaction, so a failure leaves the file as it
|
||||||
|
// was.
|
||||||
|
func Apply(db *gorm.DB, dir string) error {
|
||||||
|
return db.Transaction(func(tx *gorm.DB) error {
|
||||||
|
var tables int
|
||||||
|
|
||||||
|
// Row().Scan, not (*gorm.DB).Scan: see internal/gormlog.
|
||||||
|
err := tx.Raw(
|
||||||
|
"SELECT count(*) FROM sqlite_master "+
|
||||||
|
"WHERE type = 'table' AND name = ?",
|
||||||
|
"schema_migrations",
|
||||||
|
).Row().Scan(&tables)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("looking for schema_migrations: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if tables == 0 {
|
||||||
|
err = run(tx, path.Join(dir, "000_migration.sql"))
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// fs.Glob lists the files in name order, which is number order.
|
||||||
|
names, err := fs.Glob(files, path.Join(dir, "*.sql"))
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("listing migrations: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, name := range names {
|
||||||
|
err = applyOnce(tx, name)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// applyOnce runs the migration file name and records its number,
|
||||||
|
// unless schema_migrations already has that number.
|
||||||
|
func applyOnce(tx *gorm.DB, name string) error {
|
||||||
|
number, _, _ := strings.Cut(path.Base(name), "_")
|
||||||
|
|
||||||
|
version, err := strconv.Atoi(number)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("numbering migration %s: %w", name, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var applied int
|
||||||
|
|
||||||
|
err = tx.Raw(
|
||||||
|
"SELECT count(*) FROM schema_migrations WHERE version = ?",
|
||||||
|
version,
|
||||||
|
).Row().Scan(&applied)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("checking migration %s: %w", name, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if applied > 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
err = run(tx, name)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
err = tx.Exec(
|
||||||
|
"INSERT INTO schema_migrations (version) VALUES (?)", version,
|
||||||
|
).Error
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("recording migration %s: %w", name, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// run executes every statement in the migration file name.
|
||||||
|
func run(tx *gorm.DB, name string) error {
|
||||||
|
script, err := files.ReadFile(name)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("reading migration %s: %w", name, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = tx.Exec(string(script)).Error
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("applying migration %s: %w", name, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
package migrations_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"database/sql"
|
||||||
|
"log/slog"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"gorm.io/driver/sqlite"
|
||||||
|
"gorm.io/gorm"
|
||||||
|
_ "modernc.org/sqlite"
|
||||||
|
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||||
|
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestApplyRunsEachFileOnce applies each kind of database file's
|
||||||
|
// migrations to a new file twice. The first time records every file.
|
||||||
|
// The second finds them all recorded and runs none, which is the only
|
||||||
|
// way it can succeed: 001_schema.sql creates tables that then exist.
|
||||||
|
func TestApplyRunsEachFileOnce(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, dir := range []string{
|
||||||
|
migrations.Main, migrations.Events, migrations.Archive,
|
||||||
|
} {
|
||||||
|
t.Run(dir, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
sqlDB, err := sql.Open(
|
||||||
|
"sqlite", filepath.Join(t.TempDir(), "test.db"),
|
||||||
|
)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
t.Cleanup(func() { _ = sqlDB.Close() })
|
||||||
|
|
||||||
|
db, err := gorm.Open(
|
||||||
|
sqlite.Dialector{Conn: sqlDB},
|
||||||
|
&gorm.Config{
|
||||||
|
Logger: gormlog.New(slog.New(slog.DiscardHandler)),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
require.NoError(t, migrations.Apply(db, dir))
|
||||||
|
require.NoError(t, migrations.Apply(db, dir))
|
||||||
|
|
||||||
|
var versions string
|
||||||
|
|
||||||
|
require.NoError(t, sqlDB.QueryRowContext(t.Context(),
|
||||||
|
"SELECT group_concat(version) FROM schema_migrations",
|
||||||
|
).Scan(&versions))
|
||||||
|
assert.Equal(t, "0,1", versions)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -201,8 +201,7 @@ func (s *ArchiveSweeper) sweep(ctx context.Context) {
|
|||||||
|
|
||||||
// sweepTarget prunes the archive of a single database target.
|
// sweepTarget prunes the archive of a single database target.
|
||||||
// A missing, empty, or "never" expiry parses as a zero duration
|
// A missing, empty, or "never" expiry parses as a zero duration
|
||||||
// and is skipped entirely, so those archives keep exactly the
|
// and is skipped entirely, so those archives are never pruned.
|
||||||
// behaviour they had before the sweep existed.
|
|
||||||
func (s *ArchiveSweeper) sweepTarget(target *database.Target) {
|
func (s *ArchiveSweeper) sweepTarget(target *database.Target) {
|
||||||
expiry, err := parseArchiveExpiry(target.Config)
|
expiry, err := parseArchiveExpiry(target.Config)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -21,6 +21,8 @@ import (
|
|||||||
"gorm.io/gorm/clause"
|
"gorm.io/gorm/clause"
|
||||||
_ "modernc.org/sqlite" // Pure Go SQLite driver.
|
_ "modernc.org/sqlite" // Pure Go SQLite driver.
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||||
|
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||||
"sneak.berlin/go/webhooker/internal/delivery"
|
"sneak.berlin/go/webhooker/internal/delivery"
|
||||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||||
)
|
)
|
||||||
@@ -59,29 +61,14 @@ func setupArchiveTest(t *testing.T) *archiveEnv {
|
|||||||
dataDir := t.TempDir()
|
dataDir := t.TempDir()
|
||||||
log := archiveTestLogger()
|
log := archiveTestLogger()
|
||||||
|
|
||||||
sqlDB, err := sql.Open(
|
mainDB, err := database.Open(dataDir, slog.New(slog.DiscardHandler))
|
||||||
"sqlite",
|
|
||||||
fmt.Sprintf(
|
|
||||||
"file:%s?mode=rwc",
|
|
||||||
filepath.Join(dataDir, "main.db"),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
t.Cleanup(func() { _ = sqlDB.Close() })
|
t.Cleanup(func() { _ = mainDB.Close() })
|
||||||
|
|
||||||
gdb, err := gorm.Open(
|
|
||||||
sqlite.Dialector{Conn: sqlDB},
|
|
||||||
&gorm.Config{Logger: gormlog.New(slog.New(slog.DiscardHandler))},
|
|
||||||
)
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
mainDB := database.NewTestDatabase(gdb)
|
|
||||||
require.NoError(t, mainDB.Migrate())
|
|
||||||
|
|
||||||
eng := delivery.NewTestEngineWithDB(
|
eng := delivery.NewTestEngineWithDB(
|
||||||
mainDB,
|
mainDB,
|
||||||
database.NewTestWebhookDBManager(dataDir),
|
databasetest.NewWebhookDBManager(t, dataDir),
|
||||||
log,
|
log,
|
||||||
&http.Client{Timeout: 5 * time.Second},
|
&http.Client{Timeout: 5 * time.Second},
|
||||||
1,
|
1,
|
||||||
@@ -186,9 +173,7 @@ func seedArchiveFile(
|
|||||||
)
|
)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
require.NoError(
|
require.NoError(t, migrations.Apply(gdb, migrations.Archive))
|
||||||
t, gdb.AutoMigrate(&delivery.ExportArchivedEvent{}),
|
|
||||||
)
|
|
||||||
|
|
||||||
for i, at := range archivedAt {
|
for i, at := range archivedAt {
|
||||||
row := delivery.ExportArchivedEvent{
|
row := delivery.ExportArchivedEvent{
|
||||||
@@ -795,9 +780,9 @@ func TestArchiveSweep_NeverExpiryUntouched(t *testing.T) {
|
|||||||
//
|
//
|
||||||
// The spec is stronger than that — a "never" archive is skipped
|
// The spec is stronger than that — a "never" archive is skipped
|
||||||
// before any file is touched — so the archive here exists but has
|
// before any file is touched — so the archive here exists but has
|
||||||
// never been migrated. Opening it at all would run AutoMigrate
|
// never been migrated. Opening it at all would apply the archive
|
||||||
// and create the archive table, which is exactly what must not
|
// schema and create the archive table, which is exactly what must
|
||||||
// happen.
|
// not happen.
|
||||||
func TestArchiveSweep_NeverExpirySkipsBeforeOpening(
|
func TestArchiveSweep_NeverExpirySkipsBeforeOpening(
|
||||||
t *testing.T,
|
t *testing.T,
|
||||||
) {
|
) {
|
||||||
@@ -821,7 +806,7 @@ func TestArchiveSweep_NeverExpirySkipsBeforeOpening(
|
|||||||
|
|
||||||
// seedUnmigratedArchive creates an archive file that exists but
|
// seedUnmigratedArchive creates an archive file that exists but
|
||||||
// carries no archive schema, so any open of it is observable: the
|
// carries no archive schema, so any open of it is observable: the
|
||||||
// archive table appears only if something ran AutoMigrate.
|
// archive table appears only if something applied the schema.
|
||||||
func seedUnmigratedArchive(t *testing.T, path string) {
|
func seedUnmigratedArchive(t *testing.T, path string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
|
|||||||
@@ -10,7 +10,6 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
"strings"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -19,12 +18,11 @@ import (
|
|||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"gorm.io/driver/sqlite"
|
|
||||||
"gorm.io/gorm"
|
"gorm.io/gorm"
|
||||||
_ "modernc.org/sqlite"
|
_ "modernc.org/sqlite"
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||||
"sneak.berlin/go/webhooker/internal/delivery"
|
"sneak.berlin/go/webhooker/internal/delivery"
|
||||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// iSetup holds common integration test dependencies.
|
// iSetup holds common integration test dependencies.
|
||||||
@@ -45,12 +43,12 @@ func newISetup(t *testing.T) iSetup {
|
|||||||
wDB := iSeedWebhookDB(t, dbMgr, wID)
|
wDB := iSeedWebhookDB(t, dbMgr, wID)
|
||||||
|
|
||||||
return iSetup{
|
return iSetup{
|
||||||
MainDB: mainDB,
|
MainDB: mainDB.DB(),
|
||||||
DBMgr: dbMgr,
|
DBMgr: dbMgr,
|
||||||
WebhookID: wID,
|
WebhookID: wID,
|
||||||
WebhookDB: wDB,
|
WebhookDB: wDB,
|
||||||
Engine: delivery.NewTestEngineWithDB(
|
Engine: delivery.NewTestEngineWithDB(
|
||||||
database.NewTestDatabase(mainDB),
|
mainDB,
|
||||||
dbMgr,
|
dbMgr,
|
||||||
slog.New(slog.NewTextHandler(
|
slog.New(slog.NewTextHandler(
|
||||||
os.Stderr,
|
os.Stderr,
|
||||||
@@ -64,35 +62,16 @@ func newISetup(t *testing.T) iSetup {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func iMainDB(t *testing.T) *gorm.DB {
|
// iMainDB opens a main database through database.Open, the way the
|
||||||
|
// service opens it, so these tests cannot pass against journal and
|
||||||
|
// locking settings production does not use.
|
||||||
|
func iMainDB(t *testing.T) *database.Database {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
dbPath := filepath.Join(
|
db, err := database.Open(t.TempDir(), slog.New(slog.DiscardHandler))
|
||||||
t.TempDir(), "main-test.db",
|
|
||||||
)
|
|
||||||
|
|
||||||
// Opened the way the service opens the main database, so these
|
|
||||||
// tests cannot pass against journal and locking settings
|
|
||||||
// production does not use.
|
|
||||||
sqlDB, err := database.OpenSQLite(
|
|
||||||
dbPath, database.SQLiteModeCreate,
|
|
||||||
)
|
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
t.Cleanup(func() { _ = sqlDB.Close() })
|
t.Cleanup(func() { _ = db.Close() })
|
||||||
|
|
||||||
db, err := gorm.Open(
|
|
||||||
sqlite.Dialector{Conn: sqlDB},
|
|
||||||
&gorm.Config{Logger: gormlog.New(slog.New(slog.DiscardHandler))},
|
|
||||||
)
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
require.NoError(t, db.AutoMigrate(
|
|
||||||
&database.Webhook{},
|
|
||||||
&database.Target{},
|
|
||||||
&database.User{},
|
|
||||||
&database.Setting{},
|
|
||||||
))
|
|
||||||
|
|
||||||
return db
|
return db
|
||||||
}
|
}
|
||||||
@@ -102,7 +81,7 @@ func iDBManager(
|
|||||||
) *database.WebhookDBManager {
|
) *database.WebhookDBManager {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
return database.NewTestWebhookDBManager(t.TempDir())
|
return databasetest.NewWebhookDBManager(t, t.TempDir())
|
||||||
}
|
}
|
||||||
|
|
||||||
func iSeedWebhookDB(
|
func iSeedWebhookDB(
|
||||||
@@ -984,29 +963,12 @@ func TestFailUnretryableRetry_WritesNoTargetRow(
|
|||||||
database.DeliveryStatusFailed,
|
database.DeliveryStatusFailed,
|
||||||
)
|
)
|
||||||
|
|
||||||
// The table exists in the per-webhook database because GORM
|
// The per-webhook database has no targets table, so writing a
|
||||||
// migrates the Delivery relation's model alongside it. It must
|
// target row there would have failed the status write above.
|
||||||
// stay empty.
|
assert.False(t, s.WebhookDB.Migrator().HasTable("targets"),
|
||||||
var targetRows int64
|
|
||||||
|
|
||||||
require.NoError(t, s.WebhookDB.
|
|
||||||
Table("targets").
|
|
||||||
Count(&targetRows).Error)
|
|
||||||
|
|
||||||
assert.Zero(t, targetRows,
|
|
||||||
"orphaned-retry terminal failure wrote a target row "+
|
"orphaned-retry terminal failure wrote a target row "+
|
||||||
"into the per-webhook event database",
|
"into the per-webhook event database",
|
||||||
)
|
)
|
||||||
|
|
||||||
var configs []string
|
|
||||||
|
|
||||||
require.NoError(t, s.WebhookDB.
|
|
||||||
Table("targets").
|
|
||||||
Pluck("config", &configs).Error)
|
|
||||||
|
|
||||||
assert.NotContains(
|
|
||||||
t, strings.Join(configs, " "), hookURL,
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRecoverSingleRetry_UnknownTargetType(
|
func TestRecoverSingleRetry_UnknownTargetType(
|
||||||
@@ -1154,17 +1116,17 @@ func TestRecoverInFlight_ReportsAMissingWebhookDatabase(t *testing.T) {
|
|||||||
|
|
||||||
mainDB := iMainDB(t)
|
mainDB := iMainDB(t)
|
||||||
webhookID := uuid.New().String()
|
webhookID := uuid.New().String()
|
||||||
iCreateWebhook(t, mainDB, webhookID, "lost-database")
|
iCreateWebhook(t, mainDB.DB(), webhookID, "lost-database")
|
||||||
|
|
||||||
var logs bytes.Buffer
|
var logs bytes.Buffer
|
||||||
|
|
||||||
dbMgr := database.NewTestWebhookDBManagerWithLogger(
|
dbMgr := databasetest.NewWebhookDBManagerWithLogger(
|
||||||
t.TempDir(), slog.New(slog.NewTextHandler(&logs, nil)),
|
t, t.TempDir(), slog.New(slog.NewTextHandler(&logs, nil)),
|
||||||
)
|
)
|
||||||
t.Cleanup(func() { _ = dbMgr.CloseAll() })
|
t.Cleanup(func() { _ = dbMgr.CloseAll() })
|
||||||
|
|
||||||
engine := delivery.NewTestEngineWithDB(
|
engine := delivery.NewTestEngineWithDB(
|
||||||
database.NewTestDatabase(mainDB), dbMgr,
|
mainDB, dbMgr,
|
||||||
slog.New(slog.DiscardHandler),
|
slog.New(slog.DiscardHandler),
|
||||||
&http.Client{Timeout: 5 * time.Second}, 1,
|
&http.Client{Timeout: 5 * time.Second}, 1,
|
||||||
)
|
)
|
||||||
@@ -1188,14 +1150,14 @@ func TestRecoverInFlight_SkipsAWebhookDeletedAfterTheListIsRead(
|
|||||||
|
|
||||||
mainDB := iMainDB(t)
|
mainDB := iMainDB(t)
|
||||||
webhookID := uuid.New().String()
|
webhookID := uuid.New().String()
|
||||||
iCreateWebhook(t, mainDB, webhookID, "deleted-during-recovery")
|
iCreateWebhook(t, mainDB.DB(), webhookID, "deleted-during-recovery")
|
||||||
|
|
||||||
// The first query to return is recovery's read of the list of
|
// The first query to return is recovery's read of the list of
|
||||||
// webhooks. Deleting the webhook right after it puts the delete
|
// webhooks. Deleting the webhook right after it puts the delete
|
||||||
// between that read and the opening of the webhook's database.
|
// between that read and the opening of the webhook's database.
|
||||||
deleted := false
|
deleted := false
|
||||||
|
|
||||||
require.NoError(t, mainDB.Callback().Query().After("gorm:query").
|
require.NoError(t, mainDB.DB().Callback().Query().After("gorm:query").
|
||||||
Register("delete-after-list", func(*gorm.DB) {
|
Register("delete-after-list", func(*gorm.DB) {
|
||||||
if deleted {
|
if deleted {
|
||||||
return
|
return
|
||||||
@@ -1203,16 +1165,16 @@ func TestRecoverInFlight_SkipsAWebhookDeletedAfterTheListIsRead(
|
|||||||
|
|
||||||
deleted = true
|
deleted = true
|
||||||
|
|
||||||
require.NoError(t, mainDB.Delete(
|
require.NoError(t, mainDB.DB().Delete(
|
||||||
&database.Webhook{}, "id = ?", webhookID,
|
&database.Webhook{}, "id = ?", webhookID,
|
||||||
).Error)
|
).Error)
|
||||||
}))
|
}))
|
||||||
|
|
||||||
dbMgr := database.NewTestWebhookDBManager(t.TempDir())
|
dbMgr := databasetest.NewWebhookDBManager(t, t.TempDir())
|
||||||
t.Cleanup(func() { _ = dbMgr.CloseAll() })
|
t.Cleanup(func() { _ = dbMgr.CloseAll() })
|
||||||
|
|
||||||
engine := delivery.NewTestEngineWithDB(
|
engine := delivery.NewTestEngineWithDB(
|
||||||
database.NewTestDatabase(mainDB), dbMgr,
|
mainDB, dbMgr,
|
||||||
slog.New(slog.DiscardHandler),
|
slog.New(slog.DiscardHandler),
|
||||||
&http.Client{Timeout: 5 * time.Second}, 1,
|
&http.Client{Timeout: 5 * time.Second}, 1,
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -10,7 +10,6 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
@@ -22,12 +21,11 @@ import (
|
|||||||
"github.com/prometheus/client_golang/prometheus"
|
"github.com/prometheus/client_golang/prometheus"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"gorm.io/driver/sqlite"
|
|
||||||
"gorm.io/gorm"
|
"gorm.io/gorm"
|
||||||
_ "modernc.org/sqlite"
|
_ "modernc.org/sqlite"
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||||
"sneak.berlin/go/webhooker/internal/delivery"
|
"sneak.berlin/go/webhooker/internal/delivery"
|
||||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
|
||||||
"sneak.berlin/go/webhooker/internal/metrics"
|
"sneak.berlin/go/webhooker/internal/metrics"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -37,35 +35,18 @@ const testContentType = "application/json"
|
|||||||
func testWebhookDB(t *testing.T) *gorm.DB {
|
func testWebhookDB(t *testing.T) *gorm.DB {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
dbPath := filepath.Join(
|
// Opened by the manager the service uses, so these tests cannot
|
||||||
t.TempDir(), "events-test.db",
|
// pass against a schema, journal and locking settings or write
|
||||||
|
// guard production does not use.
|
||||||
|
mgr := databasetest.NewWebhookDBManagerWithLogger(
|
||||||
|
t, t.TempDir(), slog.New(slog.DiscardHandler),
|
||||||
)
|
)
|
||||||
|
|
||||||
// Opened the way the service opens a per-webhook database, so
|
t.Cleanup(func() { _ = mgr.CloseAll() })
|
||||||
// these tests cannot pass against journal and locking settings
|
|
||||||
// production does not use.
|
db, err := mgr.GetDB(uuid.New().String())
|
||||||
sqlDB, err := database.OpenSQLite(
|
|
||||||
dbPath, database.SQLiteModeCreate,
|
|
||||||
)
|
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
t.Cleanup(func() { _ = sqlDB.Close() })
|
|
||||||
|
|
||||||
db, err := gorm.Open(
|
|
||||||
sqlite.Dialector{Conn: sqlDB},
|
|
||||||
&gorm.Config{Logger: gormlog.New(slog.New(slog.DiscardHandler))},
|
|
||||||
)
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
require.NoError(t, db.AutoMigrate(
|
|
||||||
&database.Event{},
|
|
||||||
&database.Delivery{},
|
|
||||||
&database.DeliveryResult{},
|
|
||||||
&database.EventTotals{},
|
|
||||||
&database.TargetTotals{},
|
|
||||||
))
|
|
||||||
require.NoError(t, db.Create(&database.EventTotals{}).Error)
|
|
||||||
|
|
||||||
return db
|
return db
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -48,8 +48,9 @@ func fSweepSetup(
|
|||||||
//
|
//
|
||||||
// Every caller drives the dispatch paths synchronously and has already
|
// Every caller drives the dispatch paths synchronously and has already
|
||||||
// waited for them to return, so anything they queued is in the channel
|
// waited for them to return, so anything they queued is in the channel
|
||||||
// by now. The short grace covers nothing but scheduler jitter, and is
|
// by now, and nothing is waited for. A timer here would race the queued
|
||||||
// kept small because one of these tests runs the drain forty times.
|
// tasks: on a busy host it can be due by the time select looks, and
|
||||||
|
// select picks at random among the cases that are ready.
|
||||||
func fDrain(e *delivery.Engine) []delivery.Task {
|
func fDrain(e *delivery.Engine) []delivery.Task {
|
||||||
var out []delivery.Task
|
var out []delivery.Task
|
||||||
|
|
||||||
@@ -59,7 +60,7 @@ func fDrain(e *delivery.Engine) []delivery.Task {
|
|||||||
out = append(out, task)
|
out = append(out, task)
|
||||||
case task := <-e.ExportRetryCh():
|
case task := <-e.ExportRetryCh():
|
||||||
out = append(out, task)
|
out = append(out, task)
|
||||||
case <-time.After(25 * time.Millisecond):
|
default:
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,7 +5,6 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -14,11 +13,9 @@ import (
|
|||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"gorm.io/driver/sqlite"
|
|
||||||
"gorm.io/gorm"
|
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||||
"sneak.berlin/go/webhooker/internal/delivery"
|
"sneak.berlin/go/webhooker/internal/delivery"
|
||||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// qdAggregateMarker identifies the queue-depth aggregate in the
|
// qdAggregateMarker identifies the queue-depth aggregate in the
|
||||||
@@ -49,27 +46,13 @@ func (q *qdSyncBuf) String() string {
|
|||||||
|
|
||||||
// qdMainDB opens a main database whose GORM logger is the service's
|
// qdMainDB opens a main database whose GORM logger is the service's
|
||||||
// adapter, writing through log.
|
// adapter, writing through log.
|
||||||
func qdMainDB(t *testing.T, log *slog.Logger) *gorm.DB {
|
func qdMainDB(t *testing.T, log *slog.Logger) *database.Database {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
sqlDB, err := database.OpenSQLite(
|
db, err := database.Open(t.TempDir(), log)
|
||||||
filepath.Join(t.TempDir(), "main-gormlog.db"),
|
|
||||||
database.SQLiteModeCreate,
|
|
||||||
)
|
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
t.Cleanup(func() { _ = sqlDB.Close() })
|
t.Cleanup(func() { _ = db.Close() })
|
||||||
|
|
||||||
db, err := gorm.Open(
|
|
||||||
sqlite.Dialector{Conn: sqlDB},
|
|
||||||
&gorm.Config{Logger: gormlog.New(log)},
|
|
||||||
)
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
require.NoError(t, db.AutoMigrate(
|
|
||||||
&database.Webhook{},
|
|
||||||
&database.Target{},
|
|
||||||
))
|
|
||||||
|
|
||||||
return db
|
return db
|
||||||
}
|
}
|
||||||
@@ -106,18 +89,18 @@ func TestQueueDepthSample_LogsNoBoundValue(t *testing.T) {
|
|||||||
))
|
))
|
||||||
|
|
||||||
mainDB := qdMainDB(t, log)
|
mainDB := qdMainDB(t, log)
|
||||||
dbMgr := database.NewTestWebhookDBManagerWithLogger(
|
dbMgr := databasetest.NewWebhookDBManagerWithLogger(
|
||||||
t.TempDir(), log,
|
t, t.TempDir(), log,
|
||||||
)
|
)
|
||||||
|
|
||||||
webhookID := uuid.New().String()
|
webhookID := uuid.New().String()
|
||||||
webhookDB := iSeedWebhookDB(t, dbMgr, webhookID)
|
webhookDB := iSeedWebhookDB(t, dbMgr, webhookID)
|
||||||
|
|
||||||
iCreateWebhook(t, mainDB, webhookID, "queue-depth-gormlog")
|
iCreateWebhook(t, mainDB.DB(), webhookID, "queue-depth-gormlog")
|
||||||
|
|
||||||
targetID := uuid.New().String()
|
targetID := uuid.New().String()
|
||||||
|
|
||||||
iCreateTarget(t, mainDB, targetID, webhookID,
|
iCreateTarget(t, mainDB.DB(), targetID, webhookID,
|
||||||
"queue-depth-gormlog-target", database.TargetTypeHTTP,
|
"queue-depth-gormlog-target", database.TargetTypeHTTP,
|
||||||
iHTTPConfig("https://example.com/hook"), 3,
|
iHTTPConfig("https://example.com/hook"), 3,
|
||||||
)
|
)
|
||||||
@@ -136,7 +119,7 @@ func TestQueueDepthSample_LogsNoBoundValue(t *testing.T) {
|
|||||||
)
|
)
|
||||||
|
|
||||||
engine := delivery.NewTestEngineWithDB(
|
engine := delivery.NewTestEngineWithDB(
|
||||||
database.NewTestDatabase(mainDB),
|
mainDB,
|
||||||
dbMgr,
|
dbMgr,
|
||||||
log,
|
log,
|
||||||
&http.Client{Timeout: 5 * time.Second},
|
&http.Client{Timeout: 5 * time.Second},
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ import (
|
|||||||
"gorm.io/driver/sqlite"
|
"gorm.io/driver/sqlite"
|
||||||
"gorm.io/gorm"
|
"gorm.io/gorm"
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/db/migrations"
|
||||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -335,7 +336,7 @@ func (w *archiveWriter) openMode(
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
err = gdb.AutoMigrate(&archivedEvent{})
|
err = migrations.Apply(gdb, migrations.Archive)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
_ = sqlDB.Close()
|
_ = sqlDB.Close()
|
||||||
|
|
||||||
|
|||||||
@@ -460,8 +460,13 @@ func TestArchiveExport_OneFileOpenAtATime(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// heapPeak is an io.Writer that discards what it is given and records
|
// heapPeak is an io.Writer that discards what it is given and records
|
||||||
// the largest heap it saw at a write. It collects garbage before each
|
// the largest heap it saw at a write. It collects garbage twice before
|
||||||
// reading, so the heap it reads is what is still held.
|
// each reading, so the heap it reads is what is still held. Once is not
|
||||||
|
// enough: the libraries the export calls (regexp, under GORM's table
|
||||||
|
// names, and encoding/json among them) cache buffers in a sync.Pool,
|
||||||
|
// which keeps them through one collection, so after one the reading
|
||||||
|
// counts however many happen to be cached. That varies from run to run
|
||||||
|
// by about as much as the limit in TestArchiveExport_Streams.
|
||||||
type heapPeak struct {
|
type heapPeak struct {
|
||||||
max uint64
|
max uint64
|
||||||
}
|
}
|
||||||
@@ -469,6 +474,7 @@ type heapPeak struct {
|
|||||||
func (p *heapPeak) Write(b []byte) (int, error) {
|
func (p *heapPeak) Write(b []byte) (int, error) {
|
||||||
var m runtime.MemStats
|
var m runtime.MemStats
|
||||||
|
|
||||||
|
runtime.GC()
|
||||||
runtime.GC()
|
runtime.GC()
|
||||||
runtime.ReadMemStats(&m)
|
runtime.ReadMemStats(&m)
|
||||||
p.max = max(p.max, m.HeapAlloc)
|
p.max = max(p.max, m.HeapAlloc)
|
||||||
@@ -498,6 +504,8 @@ func exportHeapGrowth(t *testing.T, rows, bodySize int) uint64 {
|
|||||||
|
|
||||||
export := listExport(t, path)
|
export := listExport(t, path)
|
||||||
|
|
||||||
|
// Twice, for the reason heapPeak gives.
|
||||||
|
runtime.GC()
|
||||||
runtime.GC()
|
runtime.GC()
|
||||||
|
|
||||||
var start runtime.MemStats
|
var start runtime.MemStats
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ import (
|
|||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database/databasetest"
|
||||||
"sneak.berlin/go/webhooker/internal/delivery"
|
"sneak.berlin/go/webhooker/internal/delivery"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -335,7 +336,7 @@ func TestArchivePathAt(t *testing.T) {
|
|||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
dataDir := t.TempDir()
|
dataDir := t.TempDir()
|
||||||
dbMgr := database.NewTestWebhookDBManager(dataDir)
|
dbMgr := databasetest.NewWebhookDBManager(t, dataDir)
|
||||||
webhook := &database.Webhook{
|
webhook := &database.Webhook{
|
||||||
BaseModel: database.BaseModel{ID: "wh-id"}, Name: "Orders",
|
BaseModel: database.BaseModel{ID: "wh-id"}, Name: "Orders",
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -82,8 +82,8 @@ func isReservedTargetHeader(name string) bool {
|
|||||||
// one pair silently overwriting the other.
|
// one pair silently overwriting the other.
|
||||||
//
|
//
|
||||||
// An input with no pairs yields an empty map, which omitempty drops
|
// An input with no pairs yields an empty map, which omitempty drops
|
||||||
// from the stored config: a target configured with no headers keeps
|
// from the stored config, so a target configured with no headers
|
||||||
// the same config JSON it had before this field existed.
|
// stores no headers key at all.
|
||||||
func ParseTargetHeaders(raw string) (map[string]string, error) {
|
func ParseTargetHeaders(raw string) (map[string]string, error) {
|
||||||
headers := make(map[string]string)
|
headers := make(map[string]string)
|
||||||
|
|
||||||
|
|||||||
@@ -13,9 +13,9 @@ import (
|
|||||||
const RedactionMarker = "(redacted)"
|
const RedactionMarker = "(redacted)"
|
||||||
|
|
||||||
// Redactor removes one target's own credential material from
|
// Redactor removes one target's own credential material from
|
||||||
// text that target's remote peer chose: a delivery response
|
// a delivery response body, whose text the target's remote
|
||||||
// body, or a delivery error stored before the delivery path
|
// peer chose, and from a stored delivery error as a second
|
||||||
// learned to mask the URLs it embeds.
|
// line behind the delivery path's own URL masking.
|
||||||
//
|
//
|
||||||
// It removes byte-identical echoes of strings taken from the
|
// It removes byte-identical echoes of strings taken from the
|
||||||
// target's stored configuration, and nothing else. Anything
|
// target's stored configuration, and nothing else. Anything
|
||||||
|
|||||||
@@ -312,13 +312,9 @@ func TestFailMissingTarget_WritesNoTargetRow(
|
|||||||
database.DeliveryStatusFailed,
|
database.DeliveryStatusFailed,
|
||||||
)
|
)
|
||||||
|
|
||||||
var configs []string
|
// The per-webhook database has no targets table, so writing a
|
||||||
|
// target row there would have failed the status write above.
|
||||||
require.NoError(t, s.WebhookDB.
|
assert.False(t, s.WebhookDB.Migrator().HasTable("targets"),
|
||||||
Table("targets").
|
|
||||||
Pluck("config", &configs).Error)
|
|
||||||
|
|
||||||
assert.Empty(t, configs,
|
|
||||||
"the deleted-target terminal path wrote a target row "+
|
"the deleted-target terminal path wrote a target row "+
|
||||||
"into the per-webhook event database",
|
"into the per-webhook event database",
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package gormlog_test
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"database/sql"
|
"database/sql"
|
||||||
|
"log/slog"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -13,6 +14,7 @@ import (
|
|||||||
"go.uber.org/fx/fxtest"
|
"go.uber.org/fx/fxtest"
|
||||||
_ "modernc.org/sqlite" // Pure Go SQLite driver.
|
_ "modernc.org/sqlite" // Pure Go SQLite driver.
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
|
"sneak.berlin/go/webhooker/internal/config/configtest"
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
"sneak.berlin/go/webhooker/internal/globals"
|
"sneak.berlin/go/webhooker/internal/globals"
|
||||||
"sneak.berlin/go/webhooker/internal/logger"
|
"sneak.berlin/go/webhooker/internal/logger"
|
||||||
@@ -128,7 +130,7 @@ func readFirstBootSecrets(
|
|||||||
func bootAtDebug(t *testing.T, dataDir string) string {
|
func bootAtDebug(t *testing.T, dataDir string) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
config.ClearEnvForTest(t)
|
configtest.ClearEnv(t)
|
||||||
t.Setenv("DEBUG", "true")
|
t.Setenv("DEBUG", "true")
|
||||||
t.Setenv("DATA_DIR", dataDir)
|
t.Setenv("DATA_DIR", dataDir)
|
||||||
|
|
||||||
@@ -145,6 +147,7 @@ func bootAtDebug(t *testing.T, dataDir string) string {
|
|||||||
fx.Provide(
|
fx.Provide(
|
||||||
globals.New,
|
globals.New,
|
||||||
logger.New,
|
logger.New,
|
||||||
|
func(l *logger.Logger) *slog.Logger { return l.Get() },
|
||||||
config.New,
|
config.New,
|
||||||
database.New,
|
database.New,
|
||||||
session.New,
|
session.New,
|
||||||
|
|||||||
@@ -135,9 +135,11 @@ func openDB(
|
|||||||
)
|
)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
require.NoError(t, gdb.AutoMigrate(&thing{}))
|
require.NoError(t, gdb.Exec(
|
||||||
|
"CREATE TABLE things (id text PRIMARY KEY, name text)",
|
||||||
|
).Error)
|
||||||
|
|
||||||
// Migration chatter is not what any of these cases is about.
|
// Creating the table is not what any of these cases is about.
|
||||||
buf.Reset()
|
buf.Reset()
|
||||||
|
|
||||||
return gdb
|
return gdb
|
||||||
|
|||||||
@@ -115,8 +115,8 @@ func skipDir(name string) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// walkNonTestGo parses every non-test .go file under root. It returns
|
// walkNonTestGo parses every non-test .go file under root. It returns
|
||||||
// the directories, relative to root, it parsed a file in, along with
|
// the directories, relative to root, it parsed a file in or below,
|
||||||
// every unguarded Scan it found.
|
// along with every unguarded Scan it found.
|
||||||
func walkNonTestGo(t *testing.T, root string) (map[string]bool, []string) {
|
func walkNonTestGo(t *testing.T, root string) (map[string]bool, []string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -155,7 +155,12 @@ func walkNonTestGo(t *testing.T, root string) (map[string]bool, []string) {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
walked[dir] = true
|
// The directories above count too, so that one holding
|
||||||
|
// only a nested package, as internal/db does, is not
|
||||||
|
// taken for one the walk skipped.
|
||||||
|
for ; dir != "."; dir = filepath.Dir(dir) {
|
||||||
|
walked[dir] = true
|
||||||
|
}
|
||||||
|
|
||||||
for _, pos := range unguardedScans(fset, file) {
|
for _, pos := range unguardedScans(fset, file) {
|
||||||
hits = append(hits, relPosition(root, pos))
|
hits = append(hits, relPosition(root, pos))
|
||||||
@@ -205,9 +210,9 @@ func TestGormScanIsNeverCalledOutsideTests(t *testing.T) {
|
|||||||
walked, offenders := walkNonTestGo(t, root)
|
walked, offenders := walkNonTestGo(t, root)
|
||||||
|
|
||||||
// The module's packages are static, templates, and every directory
|
// The module's packages are static, templates, and every directory
|
||||||
// directly under cmd and internal. Each holds non-test code, so one
|
// directly under cmd and internal. Each holds non-test code, itself
|
||||||
// the walk parsed nothing in was skipped, and a Scan there would
|
// or in a package below it, so one the walk parsed nothing in or
|
||||||
// pass unseen.
|
// below was skipped, and a Scan there would pass unseen.
|
||||||
packages := []string{"static", "templates"}
|
packages := []string{"static", "templates"}
|
||||||
|
|
||||||
for _, parent := range []string{"cmd", "internal"} {
|
for _, parent := range []string{"cmd", "internal"} {
|
||||||
|
|||||||
@@ -43,7 +43,7 @@ func TestDeliveryAttempts_ReadInTheTargetTypesOwnTerms(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
for _, tc := range cases {
|
for _, tc := range cases {
|
||||||
t.Run(string(tc.targetType), func(t *testing.T) {
|
t.Run(string(tc.targetType)+" "+tc.outcome, func(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
f := newRecentEventsFixture(t)
|
f := newRecentEventsFixture(t)
|
||||||
|
|||||||
@@ -314,21 +314,15 @@ func assertReplayTask(
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// assertNoLeakedTarget proves the per-webhook database holds no target
|
// assertNoLeakedTarget proves the per-webhook database holds no
|
||||||
// rows. AutoMigrate creates the table there because Delivery declares
|
// target: an association write would have upserted the whole target,
|
||||||
// the relation, so it is a ROW that signals a leak: an association
|
// plaintext config and all, into the event database. See
|
||||||
// write would have upserted the whole target, plaintext config and
|
|
||||||
// all, into the event database. See
|
|
||||||
// https://git.eeqj.de/sneak/webhooker/issues/206.
|
// https://git.eeqj.de/sneak/webhooker/issues/206.
|
||||||
func assertNoLeakedTarget(t *testing.T, webhookDB *gorm.DB) {
|
func assertNoLeakedTarget(t *testing.T, webhookDB *gorm.DB) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
var leaked int64
|
assert.False(
|
||||||
|
t, webhookDB.Migrator().HasTable(&database.Target{}),
|
||||||
require.NoError(t, webhookDB.Unscoped().
|
|
||||||
Model(&database.Target{}).Count(&leaked).Error)
|
|
||||||
assert.Zero(
|
|
||||||
t, leaked,
|
|
||||||
"replay must not write the target into the event database",
|
"replay must not write the target into the event database",
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -12,13 +12,12 @@ import (
|
|||||||
//
|
//
|
||||||
// The bound is enforced here and in SQL, because this page's
|
// The bound is enforced here and in SQL, because this page's
|
||||||
// memory profile must not depend on a constant in another
|
// memory profile must not depend on a constant in another
|
||||||
// package staying where it is, and because rows predating the
|
// package staying where it is, or on every stored row having
|
||||||
// delivery engine's own cap or restored from an archive are
|
// been written by the delivery engine under its own cap.
|
||||||
// not covered by it at all.
|
|
||||||
//
|
//
|
||||||
// It happens to equal that engine cap, so a row written by the
|
// It happens to equal that engine cap, so a row written by the
|
||||||
// current engine reaches this one exactly and is never cut
|
// engine reaches this one exactly and is never cut twice.
|
||||||
// twice. Nothing here may assume the two differ: see view.
|
// Nothing here may assume the two differ: see view.
|
||||||
const maxRenderedResponseBytes = 4096
|
const maxRenderedResponseBytes = 4096
|
||||||
|
|
||||||
// deliveryResultColumns is the delivery attempt projection.
|
// deliveryResultColumns is the delivery attempt projection.
|
||||||
@@ -40,9 +39,8 @@ const deliveryResultColumns = "delivery_id, attempt_num, created_at, " +
|
|||||||
//
|
//
|
||||||
// Both Error and ResponseBody have been through the target's
|
// Both Error and ResponseBody have been through the target's
|
||||||
// Redactor. The engine already masks the URL out of the
|
// Redactor. The engine already masks the URL out of the
|
||||||
// errors it stores, so for errors this is a second line
|
// errors it stores, so for errors this is a second line; for
|
||||||
// covering rows written before it did; for response bodies it
|
// response bodies it is the only line, and its reach is what
|
||||||
// is the only line, and its reach is what
|
|
||||||
// delivery.Redactor documents.
|
// delivery.Redactor documents.
|
||||||
type DeliveryResultView struct {
|
type DeliveryResultView struct {
|
||||||
AttemptNum int
|
AttemptNum int
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"html/template"
|
"html/template"
|
||||||
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"sync"
|
"sync"
|
||||||
@@ -249,6 +250,7 @@ func newTestAppWithConfig(
|
|||||||
fx.Provide(
|
fx.Provide(
|
||||||
globals.New,
|
globals.New,
|
||||||
logger.New,
|
logger.New,
|
||||||
|
func(l *logger.Logger) *slog.Logger { return l.Get() },
|
||||||
func() *config.Config { return cfg },
|
func() *config.Config { return cfg },
|
||||||
database.New,
|
database.New,
|
||||||
database.NewWebhookDBManager,
|
database.NewWebhookDBManager,
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ import (
|
|||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
"sneak.berlin/go/webhooker/internal/handlers"
|
"sneak.berlin/go/webhooker/internal/handlers"
|
||||||
"sneak.berlin/go/webhooker/internal/logger"
|
"sneak.berlin/go/webhooker/internal/logger"
|
||||||
"sneak.berlin/go/webhooker/internal/middleware"
|
"sneak.berlin/go/webhooker/internal/middleware/middlewaretest"
|
||||||
"sneak.berlin/go/webhooker/internal/session"
|
"sneak.berlin/go/webhooker/internal/session"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -135,7 +135,7 @@ func TestUserRoute_Unauthenticated_RedirectedByMiddleware(t *testing.T) {
|
|||||||
|
|
||||||
t.Cleanup(app.RequireStop)
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
mw := middleware.NewForTest(log.Get(), cfg, sess)
|
mw := middlewaretest.New(t, log.Get(), cfg, sess)
|
||||||
|
|
||||||
var handlerReached bool
|
var handlerReached bool
|
||||||
|
|
||||||
|
|||||||
@@ -86,6 +86,13 @@ func settingRows(cfg *config.Config) []settingRow {
|
|||||||
"Idle session timeout (Go duration)",
|
"Idle session timeout (Go duration)",
|
||||||
cfg.SessionIdleTimeout.String(),
|
cfg.SessionIdleTimeout.String(),
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"HSTS_MAX_AGE",
|
||||||
|
"max-age of the Strict-Transport-Security header, which " +
|
||||||
|
"also carries includeSubDomains and preload (Go " +
|
||||||
|
"duration, whole seconds). 0 sends no header",
|
||||||
|
cfg.HSTSMaxAge.String(),
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"RECEIVER_RATE_LIMIT",
|
"RECEIVER_RATE_LIMIT",
|
||||||
"Receiver requests/minute per IP per entrypoint " +
|
"Receiver requests/minute per IP per entrypoint " +
|
||||||
|
|||||||
@@ -76,6 +76,7 @@ func TestSettingsPageShowsLoadedConfiguration(t *testing.T) {
|
|||||||
BindAddress: "192.0.2.10",
|
BindAddress: "192.0.2.10",
|
||||||
RetentionSweepInterval: 17 * time.Minute,
|
RetentionSweepInterval: 17 * time.Minute,
|
||||||
SessionIdleTimeout: 3 * time.Hour,
|
SessionIdleTimeout: 3 * time.Hour,
|
||||||
|
HSTSMaxAge: 8760 * time.Hour,
|
||||||
ReceiverRateLimit: 77,
|
ReceiverRateLimit: 77,
|
||||||
TrustedProxies: []netip.Prefix{
|
TrustedProxies: []netip.Prefix{
|
||||||
netip.MustParsePrefix("10.1.0.0/16"),
|
netip.MustParsePrefix("10.1.0.0/16"),
|
||||||
@@ -99,6 +100,7 @@ func TestSettingsPageShowsLoadedConfiguration(t *testing.T) {
|
|||||||
"SENTRY_DSN": "not set",
|
"SENTRY_DSN": "not set",
|
||||||
"RETENTION_SWEEP_INTERVAL": "17m0s",
|
"RETENTION_SWEEP_INTERVAL": "17m0s",
|
||||||
"SESSION_IDLE_TIMEOUT": "3h0m0s",
|
"SESSION_IDLE_TIMEOUT": "3h0m0s",
|
||||||
|
"HSTS_MAX_AGE": "8760h0m0s",
|
||||||
"RECEIVER_RATE_LIMIT": "77",
|
"RECEIVER_RATE_LIMIT": "77",
|
||||||
"TRUSTED_PROXIES": "10.1.0.0/16",
|
"TRUSTED_PROXIES": "10.1.0.0/16",
|
||||||
"ALLOWED_EGRESS_CIDRS": "192.168.5.0/24, fd00::/8",
|
"ALLOWED_EGRESS_CIDRS": "192.168.5.0/24, fd00::/8",
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ import (
|
|||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
"sneak.berlin/go/webhooker/internal/middleware"
|
"sneak.berlin/go/webhooker/internal/middleware/middlewaretest"
|
||||||
)
|
)
|
||||||
|
|
||||||
// targetSecretSegments are the path segments of an incoming-webhook
|
// targetSecretSegments are the path segments of an incoming-webhook
|
||||||
@@ -65,7 +65,8 @@ func postTargetCreate(
|
|||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
logBuf := new(bytes.Buffer)
|
logBuf := new(bytes.Buffer)
|
||||||
mw := middleware.NewForTest(
|
mw := middlewaretest.New(
|
||||||
|
t,
|
||||||
slog.New(slog.NewJSONHandler(
|
slog.New(slog.NewJSONHandler(
|
||||||
logBuf, &slog.HandlerOptions{Level: slog.LevelInfo},
|
logBuf, &slog.HandlerOptions{Level: slog.LevelInfo},
|
||||||
)),
|
)),
|
||||||
|
|||||||
@@ -24,7 +24,7 @@ import (
|
|||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
"sneak.berlin/go/webhooker/internal/delivery"
|
"sneak.berlin/go/webhooker/internal/delivery"
|
||||||
"sneak.berlin/go/webhooker/internal/middleware"
|
"sneak.berlin/go/webhooker/internal/middleware/middlewaretest"
|
||||||
)
|
)
|
||||||
|
|
||||||
// errClientGone is the write failure of a client that has gone away.
|
// errClientGone is the write failure of a client that has gone away.
|
||||||
@@ -310,7 +310,8 @@ func limitedServer(
|
|||||||
const sendBuffer = 4 << 10
|
const sendBuffer = 4 << 10
|
||||||
|
|
||||||
logBuf := new(bytes.Buffer)
|
logBuf := new(bytes.Buffer)
|
||||||
mw := middleware.NewForTest(
|
mw := middlewaretest.New(
|
||||||
|
t,
|
||||||
slog.New(slog.NewJSONHandler(logBuf, nil)),
|
slog.New(slog.NewJSONHandler(logBuf, nil)),
|
||||||
&config.Config{Environment: config.EnvironmentDev},
|
&config.Config{Environment: config.EnvironmentDev},
|
||||||
nil,
|
nil,
|
||||||
|
|||||||
@@ -49,13 +49,6 @@ var (
|
|||||||
// working retry configuration over a typo. A default answers a
|
// working retry configuration over a typo. A default answers a
|
||||||
// question that was not asked; it never answers one that was asked
|
// question that was not asked; it never answers one that was asked
|
||||||
// badly.
|
// badly.
|
||||||
//
|
|
||||||
// A target stored with a count above the ceiling before this
|
|
||||||
// validation existed keeps rendering and keeps delivering — nothing
|
|
||||||
// clamps the row. Re-saving it from the edit form does have to bring
|
|
||||||
// it into range, because the form submits the pre-filled value back
|
|
||||||
// and accepting it would be the ceiling not applying to the edit
|
|
||||||
// path. The 400 names the ceiling, so the fix is one field.
|
|
||||||
func parseMaxRetries(raw string, fallback int) (int, error) {
|
func parseMaxRetries(raw string, fallback int) (int, error) {
|
||||||
raw = strings.TrimSpace(raw)
|
raw = strings.TrimSpace(raw)
|
||||||
if raw == "" {
|
if raw == "" {
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ import (
|
|||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
"sneak.berlin/go/webhooker/internal/middleware"
|
"sneak.berlin/go/webhooker/internal/middleware"
|
||||||
|
"sneak.berlin/go/webhooker/internal/middleware/middlewaretest"
|
||||||
)
|
)
|
||||||
|
|
||||||
// floodRequests is the number of distinct invented paths each flood
|
// floodRequests is the number of distinct invented paths each flood
|
||||||
@@ -83,7 +84,7 @@ func capturingMiddleware(t *testing.T) (*middleware.Middleware, *bytes.Buffer) {
|
|||||||
TrustedProxies: trustedProxies("192.0.2.1/32"),
|
TrustedProxies: trustedProxies("192.0.2.1/32"),
|
||||||
}
|
}
|
||||||
|
|
||||||
return middleware.NewForTest(log, cfg, nil), buf
|
return middlewaretest.New(t, log, cfg, nil), buf
|
||||||
}
|
}
|
||||||
|
|
||||||
// capturingTextMiddleware is capturingMiddleware for the other handler
|
// capturingTextMiddleware is capturingMiddleware for the other handler
|
||||||
@@ -107,7 +108,7 @@ func capturingTextMiddleware(
|
|||||||
TrustedProxies: trustedProxies("192.0.2.1/32"),
|
TrustedProxies: trustedProxies("192.0.2.1/32"),
|
||||||
}
|
}
|
||||||
|
|
||||||
return middleware.NewForTest(log, cfg, nil), buf
|
return middlewaretest.New(t, log, cfg, nil), buf
|
||||||
}
|
}
|
||||||
|
|
||||||
// accessLogRouter mirrors the production route shapes that an
|
// accessLogRouter mirrors the production route shapes that an
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ import (
|
|||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
"sneak.berlin/go/webhooker/internal/middleware"
|
"sneak.berlin/go/webhooker/internal/middleware"
|
||||||
|
"sneak.berlin/go/webhooker/internal/middleware/middlewaretest"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -133,8 +134,8 @@ func clientLogLines(
|
|||||||
TrustedProxies: trustedProxies(trustedProxyCIDR),
|
TrustedProxies: trustedProxies(trustedProxyCIDR),
|
||||||
}
|
}
|
||||||
|
|
||||||
m := middleware.NewForTest(
|
m := middlewaretest.New(
|
||||||
log, cfg, newTestSessionManager(cfg, log, nil),
|
t, log, cfg, newTestSessionManager(t, cfg),
|
||||||
)
|
)
|
||||||
handler := m.Logging()(site.build(m))
|
handler := m.Logging()(site.build(m))
|
||||||
|
|
||||||
|
|||||||
@@ -41,6 +41,7 @@ import (
|
|||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
"sneak.berlin/go/webhooker/internal/middleware"
|
"sneak.berlin/go/webhooker/internal/middleware"
|
||||||
|
"sneak.berlin/go/webhooker/internal/middleware/middlewaretest"
|
||||||
)
|
)
|
||||||
|
|
||||||
// bodyLimitBytes is the MaxBodySize cap these tests install. Any
|
// bodyLimitBytes is the MaxBodySize cap these tests install. Any
|
||||||
@@ -155,9 +156,9 @@ func capturingBoundMiddleware(
|
|||||||
ReceiverRateLimit: receiverLimitPerMinute,
|
ReceiverRateLimit: receiverLimitPerMinute,
|
||||||
}
|
}
|
||||||
|
|
||||||
sess := newTestSessionManager(cfg, log, nil)
|
sess := newTestSessionManager(t, cfg)
|
||||||
|
|
||||||
return middleware.NewForTest(log, cfg, sess), buf
|
return middlewaretest.New(t, log, cfg, sess), buf
|
||||||
}
|
}
|
||||||
|
|
||||||
// unreachable is a next-handler that fails the test if the middleware
|
// unreachable is a next-handler that fails the test if the middleware
|
||||||
|
|||||||
@@ -236,9 +236,9 @@ func TestLoginGuard_SemaphoreBoundsConcurrentVerifications(
|
|||||||
|
|
||||||
// rendezvousDeadlock is the deadlock guard described below.
|
// rendezvousDeadlock is the deadlock guard described below.
|
||||||
// It is orders of magnitude longer than any scheduling delay,
|
// It is orders of magnitude longer than any scheduling delay,
|
||||||
// so it never decides the result, and well inside script/test's
|
// so it never decides the result, and well inside the 90s
|
||||||
// 30s timeout, so a wedge fails on the assertion instead of
|
// package timeout of the Dockerfile's test phase, so a wedge
|
||||||
// blowing the package timeout.
|
// fails on the assertion instead of blowing that timeout.
|
||||||
rendezvousDeadlock = 5 * time.Second
|
rendezvousDeadlock = 5 * time.Second
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -151,7 +151,7 @@ var _ httpmetrics.Recorder = boundedLabelRecorder{}
|
|||||||
|
|
||||||
// Metrics returns middleware that records Prometheus HTTP metrics
|
// Metrics returns middleware that records Prometheus HTTP metrics
|
||||||
// with the Middleware's one recorder, which New builds on the registry
|
// with the Middleware's one recorder, which New builds on the registry
|
||||||
// the /metrics route serves and NewForTest on a registry of its own.
|
// it is given: in the application, the one the /metrics route serves.
|
||||||
// Every call reuses that recorder, so any number of routers can
|
// Every call reuses that recorder, so any number of routers can
|
||||||
// install it.
|
// install it.
|
||||||
func (s *Middleware) Metrics() func(http.Handler) http.Handler {
|
func (s *Middleware) Metrics() func(http.Handler) http.Handler {
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ import (
|
|||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
"sneak.berlin/go/webhooker/internal/middleware"
|
"sneak.berlin/go/webhooker/internal/middleware"
|
||||||
|
"sneak.berlin/go/webhooker/internal/middleware/middlewaretest"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -70,8 +71,8 @@ func metricsTestRouter(
|
|||||||
Environment: "prod",
|
Environment: "prod",
|
||||||
ReceiverRateLimit: receiverLimit,
|
ReceiverRateLimit: receiverLimit,
|
||||||
}
|
}
|
||||||
m := middleware.NewForTest(
|
m := middlewaretest.New(
|
||||||
log, cfg, newTestSessionManager(cfg, log, nil),
|
t, log, cfg, newTestSessionManager(t, cfg),
|
||||||
)
|
)
|
||||||
|
|
||||||
reg := prometheus.NewRegistry()
|
reg := prometheus.NewRegistry()
|
||||||
@@ -455,11 +456,11 @@ func TestMetrics_StatusAndSizeStillRecorded(t *testing.T) {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestMetrics_WorksOnNewForTestMiddleware pins that a Middleware built
|
// TestMetrics_WorksOnMiddlewaretestNew pins that a Middleware built
|
||||||
// by NewForTest has a recorder of its own: its Metrics() serves a
|
// by middlewaretest.New has a recorder of its own: its Metrics()
|
||||||
// request instead of panicking, and a second one does not collide
|
// serves a request instead of panicking, and a second one does not
|
||||||
// with the first.
|
// collide with the first.
|
||||||
func TestMetrics_WorksOnNewForTestMiddleware(t *testing.T) {
|
func TestMetrics_WorksOnMiddlewaretestNew(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
log := slog.New(slog.DiscardHandler)
|
log := slog.New(slog.DiscardHandler)
|
||||||
@@ -469,7 +470,7 @@ func TestMetrics_WorksOnNewForTestMiddleware(t *testing.T) {
|
|||||||
})
|
})
|
||||||
|
|
||||||
for range 2 {
|
for range 2 {
|
||||||
h := middleware.NewForTest(log, cfg, nil).Metrics()(ok)
|
h := middlewaretest.New(t, log, cfg, nil).Metrics()(ok)
|
||||||
|
|
||||||
req := httptest.NewRequestWithContext(
|
req := httptest.NewRequestWithContext(
|
||||||
t.Context(), http.MethodGet, okRoute, nil,
|
t.Context(), http.MethodGet, okRoute, nil,
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import (
|
|||||||
"net"
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/url"
|
"net/url"
|
||||||
|
"strconv"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -22,7 +23,6 @@ import (
|
|||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
"sneak.berlin/go/webhooker/internal/globals"
|
"sneak.berlin/go/webhooker/internal/globals"
|
||||||
"sneak.berlin/go/webhooker/internal/logfield"
|
"sneak.berlin/go/webhooker/internal/logfield"
|
||||||
"sneak.berlin/go/webhooker/internal/logger"
|
|
||||||
"sneak.berlin/go/webhooker/internal/session"
|
"sneak.berlin/go/webhooker/internal/session"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -155,7 +155,7 @@ const (
|
|||||||
type MiddlewareParams struct {
|
type MiddlewareParams struct {
|
||||||
fx.In
|
fx.In
|
||||||
|
|
||||||
Logger *logger.Logger
|
Logger *slog.Logger
|
||||||
Globals *globals.Globals
|
Globals *globals.Globals
|
||||||
Config *config.Config
|
Config *config.Config
|
||||||
Session *session.Session
|
Session *session.Session
|
||||||
@@ -169,12 +169,10 @@ type Middleware struct {
|
|||||||
params *MiddlewareParams
|
params *MiddlewareParams
|
||||||
session *session.Session
|
session *session.Session
|
||||||
|
|
||||||
// metricsRecorder records the inbound HTTP metrics. New builds
|
// metricsRecorder records the inbound HTTP metrics on
|
||||||
// it on the registry /metrics serves, NewForTest on a registry
|
// params.Registry. It is built once per Middleware and Metrics
|
||||||
// of its own. Either way it is built once per Middleware and
|
// reuses it, because building it registers its collectors, and a
|
||||||
// Metrics reuses it, because building it registers its
|
// second registration on the same registry panics.
|
||||||
// collectors, and a second registration on the same registry
|
|
||||||
// panics.
|
|
||||||
metricsRecorder httpmetrics.Recorder
|
metricsRecorder httpmetrics.Recorder
|
||||||
|
|
||||||
// loginGuard counts failed credential verifications and bounds
|
// loginGuard counts failed credential verifications and bounds
|
||||||
@@ -193,7 +191,7 @@ func New(
|
|||||||
) (*Middleware, error) {
|
) (*Middleware, error) {
|
||||||
s := new(Middleware)
|
s := new(Middleware)
|
||||||
s.params = ¶ms
|
s.params = ¶ms
|
||||||
s.log = params.Logger.Get()
|
s.log = params.Logger
|
||||||
s.session = params.Session
|
s.session = params.Session
|
||||||
s.metricsRecorder = prommetrics.NewRecorder(
|
s.metricsRecorder = prommetrics.NewRecorder(
|
||||||
prommetrics.Config{Registry: params.Registry},
|
prommetrics.Config{Registry: params.Registry},
|
||||||
@@ -538,18 +536,23 @@ func (s *Middleware) MetricsAuth() func(http.Handler) http.Handler {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// SecurityHeaders returns middleware that sets production security
|
// SecurityHeaders returns middleware that sets production security
|
||||||
// headers on every response: HSTS, X-Content-Type-Options,
|
// headers on every response: HSTS (unless HSTS_MAX_AGE is zero),
|
||||||
// X-Frame-Options, CSP, Referrer-Policy, and Permissions-Policy.
|
// X-Content-Type-Options, X-Frame-Options, CSP, Referrer-Policy, and
|
||||||
|
// Permissions-Policy.
|
||||||
func (s *Middleware) SecurityHeaders() func(http.Handler) http.Handler {
|
func (s *Middleware) SecurityHeaders() func(http.Handler) http.Handler {
|
||||||
|
maxAge := s.params.Config.HSTSMaxAge
|
||||||
|
hsts := "max-age=" + strconv.FormatInt(int64(maxAge/time.Second), 10) +
|
||||||
|
"; includeSubDomains; preload"
|
||||||
|
|
||||||
return func(next http.Handler) http.Handler {
|
return func(next http.Handler) http.Handler {
|
||||||
return http.HandlerFunc(func(
|
return http.HandlerFunc(func(
|
||||||
w http.ResponseWriter,
|
w http.ResponseWriter,
|
||||||
r *http.Request,
|
r *http.Request,
|
||||||
) {
|
) {
|
||||||
w.Header().Set(
|
if maxAge > 0 {
|
||||||
"Strict-Transport-Security",
|
w.Header().Set("Strict-Transport-Security", hsts)
|
||||||
"max-age=63072000; includeSubDomains; preload",
|
}
|
||||||
)
|
|
||||||
w.Header().Set(
|
w.Header().Set(
|
||||||
"X-Content-Type-Options", "nosniff",
|
"X-Content-Type-Options", "nosniff",
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -14,36 +14,34 @@ import (
|
|||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
|
"go.uber.org/fx/fxtest"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
"sneak.berlin/go/webhooker/internal/middleware"
|
"sneak.berlin/go/webhooker/internal/middleware"
|
||||||
|
"sneak.berlin/go/webhooker/internal/middleware/middlewaretest"
|
||||||
"sneak.berlin/go/webhooker/internal/session"
|
"sneak.berlin/go/webhooker/internal/session"
|
||||||
)
|
)
|
||||||
|
|
||||||
const testKeySize = 32
|
const testKeySize = 32
|
||||||
|
|
||||||
// testMiddleware creates a Middleware with minimal dependencies
|
// testMiddleware creates a Middleware with minimal dependencies
|
||||||
// for testing. It uses a real session.Session backed by an
|
// for testing. It uses a real session.Session.
|
||||||
// in-memory cookie store.
|
|
||||||
func testMiddleware(
|
func testMiddleware(
|
||||||
t *testing.T,
|
t *testing.T,
|
||||||
env string,
|
env string,
|
||||||
) (*middleware.Middleware, *session.Session) {
|
) (*middleware.Middleware, *session.Session) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
m, s, _ := testMiddlewareWithSessionClock(t, env, 0, nil)
|
return testMiddlewareWithIdleTimeout(t, env, 0)
|
||||||
|
|
||||||
return m, s
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// testMiddlewareWithSessionClock is testMiddleware with a
|
// testMiddlewareWithIdleTimeout is testMiddleware with a
|
||||||
// configurable session idle timeout and a manually advanced clock,
|
// configurable session idle timeout, for the session-expiry tests.
|
||||||
// for the session-expiry tests. A nil clock uses the real one.
|
func testMiddlewareWithIdleTimeout(
|
||||||
func testMiddlewareWithSessionClock(
|
|
||||||
t *testing.T,
|
t *testing.T,
|
||||||
env string,
|
env string,
|
||||||
idleTimeout time.Duration,
|
idleTimeout time.Duration,
|
||||||
clock *fakeClock,
|
) (*middleware.Middleware, *session.Session) {
|
||||||
) (*middleware.Middleware, *session.Session, *fakeClock) {
|
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
log := slog.New(slog.NewTextHandler(
|
log := slog.New(slog.NewTextHandler(
|
||||||
@@ -56,59 +54,44 @@ func testMiddlewareWithSessionClock(
|
|||||||
SessionIdleTimeout: idleTimeout,
|
SessionIdleTimeout: idleTimeout,
|
||||||
}
|
}
|
||||||
|
|
||||||
sessManager := newTestSessionManager(cfg, log, clock)
|
sessManager := newTestSessionManager(t, cfg)
|
||||||
|
|
||||||
m := middleware.NewForTest(log, cfg, sessManager)
|
m := middlewaretest.New(t, log, cfg, sessManager)
|
||||||
|
|
||||||
return m, sessManager, clock
|
return m, sessManager
|
||||||
}
|
}
|
||||||
|
|
||||||
// newTestSessionManager builds the real session.Session the
|
// newTestSessionManager builds the real session.Session the
|
||||||
// middleware tests run against: an in-memory cookie store with a
|
// middleware tests run against, through session.New, with its key
|
||||||
// known key, and optionally a manually advanced clock.
|
// in a main database of its own.
|
||||||
func newTestSessionManager(
|
func newTestSessionManager(
|
||||||
|
t *testing.T,
|
||||||
cfg *config.Config,
|
cfg *config.Config,
|
||||||
log *slog.Logger,
|
|
||||||
clock *fakeClock,
|
|
||||||
) *session.Session {
|
) *session.Session {
|
||||||
key := make([]byte, testKeySize)
|
t.Helper()
|
||||||
|
|
||||||
for i := range key {
|
discard := slog.New(slog.DiscardHandler)
|
||||||
key[i] = byte(i)
|
|
||||||
}
|
|
||||||
|
|
||||||
store := session.NewStore(key)
|
db, err := database.Open(t.TempDir(), discard)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
var now func() time.Time
|
t.Cleanup(func() { _ = db.Close() })
|
||||||
|
|
||||||
if clock != nil {
|
lc := fxtest.NewLifecycle(t)
|
||||||
now = clock.Now
|
|
||||||
}
|
|
||||||
|
|
||||||
return session.NewForTest(store, cfg, log, key, now)
|
sessManager, err := session.New(lc, session.Params{
|
||||||
}
|
Config: cfg,
|
||||||
|
Database: db,
|
||||||
|
Logger: discard,
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
// fakeClock is a manually advanced clock, so session expiry can be
|
// The start hook reads the key from db and builds the cookie
|
||||||
// tested without sleeping.
|
// store.
|
||||||
type fakeClock struct {
|
lc.RequireStart()
|
||||||
t time.Time
|
t.Cleanup(lc.RequireStop)
|
||||||
}
|
|
||||||
|
|
||||||
func (c *fakeClock) Now() time.Time {
|
return sessManager
|
||||||
return c.t
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c *fakeClock) Advance(d time.Duration) {
|
|
||||||
c.t = c.t.Add(d)
|
|
||||||
}
|
|
||||||
|
|
||||||
// newFakeClock returns a clock started at a fixed instant.
|
|
||||||
func newFakeClock() *fakeClock {
|
|
||||||
return &fakeClock{
|
|
||||||
t: time.Date(
|
|
||||||
2026, time.January, 2, 3, 4, 5, 0, time.UTC,
|
|
||||||
),
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// --- Logging Middleware Tests ---
|
// --- Logging Middleware Tests ---
|
||||||
@@ -583,6 +566,40 @@ func sessionCookies(
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// aged re-issues the session cookie in cookies with both of its
|
||||||
|
// timestamps moved back by d: the cookie as it stands once d has
|
||||||
|
// passed, so session expiry can be tested without sleeping.
|
||||||
|
func aged(
|
||||||
|
t *testing.T,
|
||||||
|
sessManager *session.Session,
|
||||||
|
cookies []*http.Cookie,
|
||||||
|
d time.Duration,
|
||||||
|
) []*http.Cookie {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(
|
||||||
|
context.Background(), http.MethodGet, "/", nil)
|
||||||
|
|
||||||
|
for _, c := range cookies {
|
||||||
|
req.AddCookie(c)
|
||||||
|
}
|
||||||
|
|
||||||
|
sess, err := sessManager.Get(req)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
for _, key := range []string{session.CreatedAtKey, session.LastSeenKey} {
|
||||||
|
at, ok := sess.Values[key].(int64)
|
||||||
|
require.True(t, ok, "the session has no %s", key)
|
||||||
|
|
||||||
|
sess.Values[key] = at - int64(d/time.Second)
|
||||||
|
}
|
||||||
|
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
require.NoError(t, sessManager.Save(req, w, sess))
|
||||||
|
|
||||||
|
return sessionCookies(w)
|
||||||
|
}
|
||||||
|
|
||||||
func TestRequireAuth_IdleExpiredSession_RedirectsToLogin(
|
func TestRequireAuth_IdleExpiredSession_RedirectsToLogin(
|
||||||
t *testing.T,
|
t *testing.T,
|
||||||
) {
|
) {
|
||||||
@@ -590,13 +607,11 @@ func TestRequireAuth_IdleExpiredSession_RedirectsToLogin(
|
|||||||
|
|
||||||
idle := time.Hour
|
idle := time.Hour
|
||||||
|
|
||||||
m, sessManager, clock := testMiddlewareWithSessionClock(
|
m, sessManager := testMiddlewareWithIdleTimeout(
|
||||||
t, config.EnvironmentDev, idle, newFakeClock(),
|
t, config.EnvironmentDev, idle,
|
||||||
)
|
)
|
||||||
|
|
||||||
cookies := loginCookies(t, sessManager)
|
cookies := aged(t, sessManager, loginCookies(t, sessManager), idle)
|
||||||
|
|
||||||
clock.Advance(idle)
|
|
||||||
|
|
||||||
called, w := runAuthed(t, m, cookies)
|
called, w := runAuthed(t, m, cookies)
|
||||||
|
|
||||||
@@ -621,14 +636,12 @@ func TestRequireAuth_RefreshesIdleDeadlineOnActivity(
|
|||||||
|
|
||||||
idle := time.Hour
|
idle := time.Hour
|
||||||
|
|
||||||
m, sessManager, clock := testMiddlewareWithSessionClock(
|
m, sessManager := testMiddlewareWithIdleTimeout(
|
||||||
t, config.EnvironmentDev, idle, newFakeClock(),
|
t, config.EnvironmentDev, idle,
|
||||||
)
|
)
|
||||||
|
|
||||||
cookies := loginCookies(t, sessManager)
|
|
||||||
|
|
||||||
// Activity halfway through the idle window.
|
// Activity halfway through the idle window.
|
||||||
clock.Advance(idle / 2)
|
cookies := aged(t, sessManager, loginCookies(t, sessManager), idle/2)
|
||||||
|
|
||||||
called, w := runAuthed(t, m, cookies)
|
called, w := runAuthed(t, m, cookies)
|
||||||
require.True(t, called, "handler should run while valid")
|
require.True(t, called, "handler should run while valid")
|
||||||
@@ -640,16 +653,22 @@ func TestRequireAuth_RefreshesIdleDeadlineOnActivity(
|
|||||||
)
|
)
|
||||||
|
|
||||||
// Past the original deadline. The refreshed cookie is still
|
// Past the original deadline. The refreshed cookie is still
|
||||||
// good; the original one is not.
|
// good; the original one is not. A minute short of the idle
|
||||||
clock.Advance(idle - time.Second)
|
// window leaves room for the real clock, which the session
|
||||||
|
// reads, to tick on while the test runs.
|
||||||
|
later := idle - time.Minute
|
||||||
|
|
||||||
calledRefreshed, _ := runAuthed(t, m, refreshed)
|
calledRefreshed, _ := runAuthed(
|
||||||
|
t, m, aged(t, sessManager, refreshed, later),
|
||||||
|
)
|
||||||
assert.True(
|
assert.True(
|
||||||
t, calledRefreshed,
|
t, calledRefreshed,
|
||||||
"refreshed session should outlive the original deadline",
|
"refreshed session should outlive the original deadline",
|
||||||
)
|
)
|
||||||
|
|
||||||
calledStale, staleW := runAuthed(t, m, cookies)
|
calledStale, staleW := runAuthed(
|
||||||
|
t, m, aged(t, sessManager, cookies, later),
|
||||||
|
)
|
||||||
assert.False(
|
assert.False(
|
||||||
t, calledStale,
|
t, calledStale,
|
||||||
"the pre-refresh cookie carries the old idle deadline",
|
"the pre-refresh cookie carries the old idle deadline",
|
||||||
@@ -662,8 +681,8 @@ func TestRequireAuth_UnauthenticatedRequestDoesNotRefresh(
|
|||||||
) {
|
) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
m, sessManager, _ := testMiddlewareWithSessionClock(
|
m, sessManager := testMiddlewareWithIdleTimeout(
|
||||||
t, config.EnvironmentDev, time.Hour, newFakeClock(),
|
t, config.EnvironmentDev, time.Hour,
|
||||||
)
|
)
|
||||||
|
|
||||||
// A session cookie that exists but was never authenticated.
|
// A session cookie that exists but was never authenticated.
|
||||||
@@ -726,6 +745,70 @@ func TestNoCache_SetsHeaders(t *testing.T) {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// --- SecurityHeaders Middleware Tests ---
|
||||||
|
|
||||||
|
// hstsHeaders serves one request through SecurityHeaders built over
|
||||||
|
// a Config with the given HSTSMaxAge, and returns every
|
||||||
|
// Strict-Transport-Security value on the response.
|
||||||
|
func hstsHeaders(t *testing.T, maxAge time.Duration) []string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
cfg := &config.Config{HSTSMaxAge: maxAge}
|
||||||
|
m := middlewaretest.New(
|
||||||
|
t, slog.New(slog.DiscardHandler), cfg,
|
||||||
|
newTestSessionManager(t, cfg),
|
||||||
|
)
|
||||||
|
|
||||||
|
handler := m.SecurityHeaders()(http.HandlerFunc(
|
||||||
|
func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
},
|
||||||
|
))
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(
|
||||||
|
context.Background(), http.MethodGet, "/", nil,
|
||||||
|
)
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
|
||||||
|
handler.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
return w.Header().Values("Strict-Transport-Security")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSecurityHeaders_HSTS(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
maxAge time.Duration
|
||||||
|
expected []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "default of ten years",
|
||||||
|
maxAge: 87600 * time.Hour,
|
||||||
|
expected: []string{"max-age=315360000; includeSubDomains; preload"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "custom value is sent in seconds",
|
||||||
|
maxAge: 90 * time.Minute,
|
||||||
|
expected: []string{"max-age=5400; includeSubDomains; preload"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "zero sends no header",
|
||||||
|
maxAge: 0,
|
||||||
|
expected: nil,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
assert.Equal(t, tt.expected, hstsHeaders(t, tt.maxAge))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// --- MaxBodySize Middleware Tests ---
|
// --- MaxBodySize Middleware Tests ---
|
||||||
|
|
||||||
const testBodyLimit int64 = 64
|
const testBodyLimit int64 = 64
|
||||||
@@ -924,12 +1007,9 @@ func metricsAuthMiddleware(
|
|||||||
MetricsPassword: "secret",
|
MetricsPassword: "secret",
|
||||||
}
|
}
|
||||||
|
|
||||||
key := make([]byte, testKeySize)
|
return middlewaretest.New(
|
||||||
store := session.NewStore(key)
|
t, log, cfg, newTestSessionManager(t, cfg),
|
||||||
|
)
|
||||||
sessManager := session.NewForTest(store, cfg, log, key, nil)
|
|
||||||
|
|
||||||
return middleware.NewForTest(log, cfg, sessManager)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// runMetricsAuthRequest sends a GET /metrics request with the
|
// runMetricsAuthRequest sends a GET /metrics request with the
|
||||||
|
|||||||
@@ -0,0 +1,42 @@
|
|||||||
|
// Package middlewaretest builds a Middleware for tests in other
|
||||||
|
// packages.
|
||||||
|
package middlewaretest
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log/slog"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/prometheus/client_golang/prometheus"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"go.uber.org/fx/fxtest"
|
||||||
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
|
"sneak.berlin/go/webhooker/internal/middleware"
|
||||||
|
"sneak.berlin/go/webhooker/internal/session"
|
||||||
|
)
|
||||||
|
|
||||||
|
// New builds a Middleware through middleware.New, on a
|
||||||
|
// lifecycle that is never started.
|
||||||
|
//
|
||||||
|
// Its metrics recorder writes to a fresh registry of its own, so
|
||||||
|
// Metrics() works on it and two of them never collide.
|
||||||
|
func New(
|
||||||
|
t *testing.T,
|
||||||
|
log *slog.Logger,
|
||||||
|
cfg *config.Config,
|
||||||
|
sess *session.Session,
|
||||||
|
) *middleware.Middleware {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
m, err := middleware.New(
|
||||||
|
fxtest.NewLifecycle(t),
|
||||||
|
middleware.MiddlewareParams{
|
||||||
|
Logger: log,
|
||||||
|
Config: cfg,
|
||||||
|
Session: sess,
|
||||||
|
Registry: prometheus.NewRegistry(),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
return m
|
||||||
|
}
|
||||||
@@ -18,6 +18,7 @@ import (
|
|||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
"sneak.berlin/go/webhooker/internal/middleware"
|
"sneak.berlin/go/webhooker/internal/middleware"
|
||||||
|
"sneak.berlin/go/webhooker/internal/middleware/middlewaretest"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestPostRateLimit_AllowsGET(t *testing.T) {
|
func TestPostRateLimit_AllowsGET(t *testing.T) {
|
||||||
@@ -198,7 +199,7 @@ func rateLimitMiddleware(
|
|||||||
&slog.HandlerOptions{Level: slog.LevelDebug},
|
&slog.HandlerOptions{Level: slog.LevelDebug},
|
||||||
))
|
))
|
||||||
|
|
||||||
return middleware.NewForTest(log, cfg, nil)
|
return middlewaretest.New(t, log, cfg, nil)
|
||||||
}
|
}
|
||||||
|
|
||||||
// trustedProxies parses CIDR strings for a test Config.
|
// trustedProxies parses CIDR strings for a test Config.
|
||||||
|
|||||||
@@ -1,32 +0,0 @@
|
|||||||
package middleware
|
|
||||||
|
|
||||||
import (
|
|
||||||
"log/slog"
|
|
||||||
|
|
||||||
"github.com/prometheus/client_golang/prometheus"
|
|
||||||
prommetrics "github.com/slok/go-http-metrics/metrics/prometheus"
|
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
|
||||||
"sneak.berlin/go/webhooker/internal/session"
|
|
||||||
)
|
|
||||||
|
|
||||||
// NewForTest creates a Middleware with the minimum dependencies
|
|
||||||
// needed for testing. This bypasses the fx lifecycle.
|
|
||||||
//
|
|
||||||
// Its metrics recorder writes to a fresh registry of its own, so
|
|
||||||
// Metrics() works on it and two of them never collide.
|
|
||||||
func NewForTest(
|
|
||||||
log *slog.Logger,
|
|
||||||
cfg *config.Config,
|
|
||||||
sess *session.Session,
|
|
||||||
) *Middleware {
|
|
||||||
return &Middleware{
|
|
||||||
log: log,
|
|
||||||
params: &MiddlewareParams{
|
|
||||||
Config: cfg,
|
|
||||||
},
|
|
||||||
session: sess,
|
|
||||||
metricsRecorder: prommetrics.NewRecorder(
|
|
||||||
prommetrics.Config{Registry: prometheus.NewRegistry()},
|
|
||||||
),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -174,6 +174,7 @@ func newServerApp(
|
|||||||
fx.Provide(
|
fx.Provide(
|
||||||
globals.New,
|
globals.New,
|
||||||
logger.New,
|
logger.New,
|
||||||
|
func(l *logger.Logger) *slog.Logger { return l.Get() },
|
||||||
func() *config.Config {
|
func() *config.Config {
|
||||||
return &config.Config{DataDir: dir}
|
return &config.Config{DataDir: dir}
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -280,6 +280,23 @@ func click(ctx context.Context, t *testing.T, xpath string) {
|
|||||||
))
|
))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// clickAndLoad clicks the link or button matching an XPath expression
|
||||||
|
// and waits, as loadPage does, for the page the click opens to load and
|
||||||
|
// for Alpine.js to start on it. Reading earlier, a check can find an
|
||||||
|
// element of the page being left, gone by the time its value is read;
|
||||||
|
// and the wait in shown is too short for a page load on a busy host.
|
||||||
|
func clickAndLoad(ctx context.Context, t *testing.T, xpath string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
_, err := chromedp.RunResponse(
|
||||||
|
ctx, chromedp.Click(xpath, chromedp.BySearch),
|
||||||
|
)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, chromedp.Run(
|
||||||
|
ctx, chromedp.WaitNotPresent("[x-cloak]", chromedp.ByQuery),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
|
||||||
// checkAddEntrypoint loads a webhook page and checks that the add
|
// checkAddEntrypoint loads a webhook page and checks that the add
|
||||||
// entrypoint form stays hidden until the Add button beside its heading
|
// entrypoint form stays hidden until the Add button beside its heading
|
||||||
// is clicked. The click looks for a button element there, so it also
|
// is clicked. The click looks for a button element there, so it also
|
||||||
@@ -422,7 +439,7 @@ func checkAddTarget(
|
|||||||
)))
|
)))
|
||||||
}
|
}
|
||||||
|
|
||||||
click(ctx, t, saveButton)
|
clickAndLoad(ctx, t, saveButton)
|
||||||
assert.Truef(t, shown(ctx, `//span[text()="`+name+
|
assert.Truef(t, shown(ctx, `//span[text()="`+name+
|
||||||
`"]/following-sibling::div/span[text()="`+badge+`"]`),
|
`"]/following-sibling::div/span[text()="`+badge+`"]`),
|
||||||
"%s: the added target is not listed as %s", targetType, badge)
|
"%s: the added target is not listed as %s", targetType, badge)
|
||||||
@@ -485,10 +502,9 @@ func checkArchiveChoices(ctx context.Context, t *testing.T, url string) {
|
|||||||
`/following-sibling::span[text()="daily"]`),
|
`/following-sibling::span[text()="daily"]`),
|
||||||
"a database target added with daily is not listed as daily")
|
"a database target added with daily is not listed as daily")
|
||||||
|
|
||||||
click(ctx, t, row+`//a[text()="Edit"]`)
|
clickAndLoad(ctx, t, row+`//a[text()="Edit"]`)
|
||||||
require.NoError(t, chromedp.Run(
|
require.NoError(t, chromedp.Run(
|
||||||
ctx,
|
ctx,
|
||||||
chromedp.WaitReady("#expiry", chromedp.ByQuery),
|
|
||||||
chromedp.Value("#expiry", &editedExpiry, chromedp.ByQuery),
|
chromedp.Value("#expiry", &editedExpiry, chromedp.ByQuery),
|
||||||
chromedp.Value("#rotation", &editedRotation, chromedp.ByQuery),
|
chromedp.Value("#rotation", &editedRotation, chromedp.ByQuery),
|
||||||
))
|
))
|
||||||
@@ -523,7 +539,7 @@ func checkRefusedTarget(ctx context.Context, t *testing.T, url string) {
|
|||||||
chromedp.Click(forwardQuery, chromedp.ByQuery),
|
chromedp.Click(forwardQuery, chromedp.ByQuery),
|
||||||
))
|
))
|
||||||
|
|
||||||
click(ctx, t, saveButton)
|
clickAndLoad(ctx, t, saveButton)
|
||||||
|
|
||||||
assert.True(t, shown(ctx, reason),
|
assert.True(t, shown(ctx, reason),
|
||||||
"a refused target does not show the reason")
|
"a refused target does not show the reason")
|
||||||
@@ -636,7 +652,7 @@ func checkRefusedEdits(
|
|||||||
))
|
))
|
||||||
}
|
}
|
||||||
|
|
||||||
click(ctx, t, `//button[text()="Save Changes"]`)
|
clickAndLoad(ctx, t, `//button[text()="Save Changes"]`)
|
||||||
|
|
||||||
assert.Truef(t, shown(ctx, reason),
|
assert.Truef(t, shown(ctx, reason),
|
||||||
"%s: a refused save does not show the reason", edit.url)
|
"%s: a refused save does not show the reason", edit.url)
|
||||||
@@ -760,7 +776,7 @@ func checkEntrypointEdit(
|
|||||||
require.NoError(t, chromedp.Run(
|
require.NoError(t, chromedp.Run(
|
||||||
ctx, chromedp.SendKeys(input, "Billing sender", chromedp.ByQuery),
|
ctx, chromedp.SendKeys(input, "Billing sender", chromedp.ByQuery),
|
||||||
))
|
))
|
||||||
click(ctx, t, saveEdit)
|
clickAndLoad(ctx, t, saveEdit)
|
||||||
|
|
||||||
assert.True(t, shown(ctx, `//span[text()="Billing sender"]`),
|
assert.True(t, shown(ctx, `//span[text()="Billing sender"]`),
|
||||||
"saving the edit form does not change the description")
|
"saving the edit form does not change the description")
|
||||||
@@ -798,7 +814,7 @@ func checkRecentEvents(ctx context.Context, t *testing.T, url string) {
|
|||||||
"clicking the newest event does not collapse it")
|
"clicking the newest event does not collapse it")
|
||||||
|
|
||||||
require.NoError(t, chromedp.Run(ctx, loadPage(url)))
|
require.NoError(t, chromedp.Run(ctx, loadPage(url)))
|
||||||
click(ctx, t, newest+`/ancestor::div[@x-data][1]//a[text()="Open"]`)
|
clickAndLoad(ctx, t, newest+`/ancestor::div[@x-data][1]//a[text()="Open"]`)
|
||||||
|
|
||||||
assert.True(t, shown(ctx, `//h2[text()="Body"]`),
|
assert.True(t, shown(ctx, `//h2[text()="Body"]`),
|
||||||
"Open does not lead to the event's own page")
|
"Open does not lead to the event's own page")
|
||||||
@@ -1199,7 +1215,7 @@ func checkNewWebhookTargets(
|
|||||||
`","rotation":"none"}`
|
`","rotation":"none"}`
|
||||||
}
|
}
|
||||||
|
|
||||||
click(ctx, t, createButton)
|
clickAndLoad(ctx, t, createButton)
|
||||||
require.Truef(t, shown(ctx, `//h1[text()="`+name+`"]`),
|
require.Truef(t, shown(ctx, `//h1[text()="`+name+`"]`),
|
||||||
"%s: the new webhook's page does not open", name)
|
"%s: the new webhook's page does not open", name)
|
||||||
|
|
||||||
@@ -1260,7 +1276,7 @@ func checkRefusedNewWebhook(ctx context.Context, t *testing.T, url string) {
|
|||||||
chromedp.SetValue(pruningChoice, "2160h", chromedp.BySearch),
|
chromedp.SetValue(pruningChoice, "2160h", chromedp.BySearch),
|
||||||
chromedp.SetValue("#archive_rotation", "monthly", chromedp.ByQuery),
|
chromedp.SetValue("#archive_rotation", "monthly", chromedp.ByQuery),
|
||||||
))
|
))
|
||||||
click(ctx, t, createButton)
|
clickAndLoad(ctx, t, createButton)
|
||||||
|
|
||||||
assert.True(t, shown(ctx, `//div[@class="alert-error"]`),
|
assert.True(t, shown(ctx, `//div[@class="alert-error"]`),
|
||||||
"a refused webhook does not show the reason")
|
"a refused webhook does not show the reason")
|
||||||
|
|||||||
@@ -91,6 +91,24 @@ func NewRouterForTest(
|
|||||||
return s.router
|
return s.router
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// NewRouterWithSentryForTest is NewRouterForTest with the sentryhttp
|
||||||
|
// handler registered, as a set SENTRY_DSN does in production.
|
||||||
|
func NewRouterWithSentryForTest(
|
||||||
|
t *testing.T,
|
||||||
|
log *logger.Logger,
|
||||||
|
cfg *config.Config,
|
||||||
|
mw *middleware.Middleware,
|
||||||
|
h *handlers.Handlers,
|
||||||
|
) http.Handler {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
s := newServerForTest(t, log, cfg, mw, h)
|
||||||
|
s.sentryEnabled.Store(true)
|
||||||
|
s.SetupRoutes()
|
||||||
|
|
||||||
|
return s.router
|
||||||
|
}
|
||||||
|
|
||||||
// ListenAddrForTest exposes the address the HTTP listener binds for
|
// ListenAddrForTest exposes the address the HTTP listener binds for
|
||||||
// a given Config, so the rendering of host and port — IPv6
|
// a given Config, so the rendering of host and port — IPv6
|
||||||
// bracketing above all — can be pinned without standing up a
|
// bracketing above all — can be pinned without standing up a
|
||||||
|
|||||||
@@ -90,8 +90,8 @@ func (s *Server) setupGlobalMiddleware() {
|
|||||||
|
|
||||||
// recoverPanics installs on r the recoverer, answering a panic with
|
// recoverPanics installs on r the recoverer, answering a panic with
|
||||||
// page (a plain 500 when page is nil), and inside it the Sentry error
|
// page (a plain 500 when page is nil), and inside it the Sentry error
|
||||||
// reporting (if SENTRY_DSN is set). Repanic is true so panics still
|
// reporting and transaction naming (if SENTRY_DSN is set). Repanic is
|
||||||
// bubble up to the recoverer.
|
// true so panics still bubble up to the recoverer.
|
||||||
//
|
//
|
||||||
// Each admin page route group installs its own, with the error page,
|
// Each admin page route group installs its own, with the error page,
|
||||||
// as its first middleware. A panic there is logged, reported and
|
// as its first middleware. A panic there is logged, reported and
|
||||||
@@ -105,6 +105,11 @@ func (s *Server) recoverPanics(r chi.Router, page http.Handler) {
|
|||||||
Repanic: true,
|
Repanic: true,
|
||||||
})
|
})
|
||||||
r.Use(sentryHandler.Handle)
|
r.Use(sentryHandler.Handle)
|
||||||
|
// Installed with every sentryhttp handler rather than once
|
||||||
|
// globally: in an admin page route group, the group's own
|
||||||
|
// handler finishes the transaction as the group returns, so
|
||||||
|
// a rename further out would come too late.
|
||||||
|
r.Use(nameSentryTransaction)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package server_test
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"html"
|
"html"
|
||||||
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"net/url"
|
"net/url"
|
||||||
@@ -130,6 +131,7 @@ func newTestEnvWithConfig(
|
|||||||
fx.Provide(
|
fx.Provide(
|
||||||
globals.New,
|
globals.New,
|
||||||
logger.New,
|
logger.New,
|
||||||
|
func(l *logger.Logger) *slog.Logger { return l.Get() },
|
||||||
func() *config.Config { return cfg },
|
func() *config.Config { return cfg },
|
||||||
database.New,
|
database.New,
|
||||||
database.NewWebhookDBManager,
|
database.NewWebhookDBManager,
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package server
|
package server
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/url"
|
"net/url"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -71,10 +72,15 @@ func scrubSentryRequest(
|
|||||||
|
|
||||||
pattern := sentryRoutePattern(hint)
|
pattern := sentryRoutePattern(hint)
|
||||||
|
|
||||||
// Only transaction events carry a Transaction name, and the SDK
|
// Only transaction events carry a Transaction name. sentryhttp
|
||||||
// builds it from the concrete path too (sentryhttp.go:105 via
|
// builds it from the concrete path too (sentryhttp.go:105 via
|
||||||
// tracing.go:553). Rewritten on the same terms.
|
// tracing.go:553) and marks it SourceURL, so it is rewritten on
|
||||||
if event.Transaction != "" {
|
// the same terms. Only a name nameSentryTransaction set from the
|
||||||
|
// route pattern, which it marks SourceRoute, is kept.
|
||||||
|
namedByRoute := event.TransactionInfo != nil &&
|
||||||
|
event.TransactionInfo.Source == sentry.SourceRoute
|
||||||
|
|
||||||
|
if event.Transaction != "" && !namedByRoute {
|
||||||
event.Transaction = sentryTransactionName(
|
event.Transaction = sentryTransactionName(
|
||||||
event.Transaction, pattern,
|
event.Transaction, pattern,
|
||||||
)
|
)
|
||||||
@@ -120,7 +126,8 @@ func scrubSentryRequest(
|
|||||||
// The transaction dispatch has no such request: Span.doFinish calls
|
// The transaction dispatch has no such request: Span.doFinish calls
|
||||||
// hub.CaptureEvent (tracing.go:356), which passes a nil hint that the
|
// hub.CaptureEvent (tracing.go:356), which passes a nil hint that the
|
||||||
// client replaces with an empty one (client.go:620-622). The pattern
|
// client replaces with an empty one (client.go:620-622). The pattern
|
||||||
// is therefore always "" there, and the callers fall back.
|
// is therefore always "" there, and the callers fall back. A routed
|
||||||
|
// transaction is named by nameSentryTransaction instead.
|
||||||
func sentryRoutePattern(hint *sentry.EventHint) string {
|
func sentryRoutePattern(hint *sentry.EventHint) string {
|
||||||
if hint == nil || hint.Context == nil {
|
if hint == nil || hint.Context == nil {
|
||||||
return ""
|
return ""
|
||||||
@@ -142,6 +149,52 @@ func sentryRoutePattern(hint *sentry.EventHint) string {
|
|||||||
return rctx.RoutePattern()
|
return rctx.RoutePattern()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// nameSentryTransaction renames the request's Sentry transaction to
|
||||||
|
// "METHOD <route pattern>", such as "POST /h/{uuid}". sentryhttp names
|
||||||
|
// it after the concrete path, which scrubSentryRequest withholds;
|
||||||
|
// since Sentry groups transactions by name, every transaction in the
|
||||||
|
// service would otherwise land in one group per method, such as
|
||||||
|
// "POST /(redacted)".
|
||||||
|
//
|
||||||
|
// It has to run inside the sentryhttp handler, where the transaction
|
||||||
|
// is on the request context. It renames on the way out, once chi has
|
||||||
|
// routed the request, and does so in a defer so that a request whose
|
||||||
|
// handler panics is named too. The context is taken when the defer is
|
||||||
|
// set up, before routing, but the chi routing context on it is a
|
||||||
|
// pointer that chi fills in as it routes.
|
||||||
|
func nameSentryTransaction(next http.Handler) http.Handler {
|
||||||
|
return http.HandlerFunc(
|
||||||
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
defer setSentryTransactionName(r.Context(), r.Method)
|
||||||
|
|
||||||
|
next.ServeHTTP(w, r)
|
||||||
|
},
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// setSentryTransactionName does the rename for nameSentryTransaction.
|
||||||
|
// A path that matches nothing at the top level has no pattern; its
|
||||||
|
// name is left as the SDK set it, for scrubSentryRequest to withhold.
|
||||||
|
// Inside a route group, a path that matches no route in the group, or
|
||||||
|
// a request the group's own middleware turns away before routing, has
|
||||||
|
// the group's pattern, such as "/pages/*", and is named after it.
|
||||||
|
func setSentryTransactionName(ctx context.Context, method string) {
|
||||||
|
transaction := sentry.TransactionFromContext(ctx)
|
||||||
|
rctx := chi.RouteContext(ctx)
|
||||||
|
|
||||||
|
if transaction == nil || rctx == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
pattern := rctx.RoutePattern()
|
||||||
|
if pattern == "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
transaction.Name = method + " " + pattern
|
||||||
|
transaction.Source = sentry.SourceRoute
|
||||||
|
}
|
||||||
|
|
||||||
// sentryRouteURL rebuilds an event's request URL with the route
|
// sentryRouteURL rebuilds an event's request URL with the route
|
||||||
// pattern in place of the concrete path.
|
// pattern in place of the concrete path.
|
||||||
//
|
//
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ import (
|
|||||||
"github.com/go-chi/chi"
|
"github.com/go-chi/chi"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
"sneak.berlin/go/webhooker/internal/server"
|
"sneak.berlin/go/webhooker/internal/server"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -130,11 +131,15 @@ func (c sentryCase) capture(t *testing.T) []*sentry.Event {
|
|||||||
// router mirrors the one ordering these tests depend on, over the two
|
// router mirrors the one ordering these tests depend on, over the two
|
||||||
// route patterns they need: a recovering middleware outside, then the
|
// route patterns they need: a recovering middleware outside, then the
|
||||||
// sentryhttp handler registered with Use and Repanic set, exactly as
|
// sentryhttp handler registered with Use and Repanic set, exactly as
|
||||||
// routes.go orders the two. The bare recover stands in for
|
// routes.go orders the two. It leaves out nameSentryTransaction,
|
||||||
// Middleware.Recoverer, which holds that outer slot in production; it
|
// which routes.go installs inside the sentryhttp handler, so a
|
||||||
// is here only to keep panic stacks out of the test output. That the
|
// transaction captured here keeps the SDK's name and shows what the
|
||||||
// production one really does catch what sentryhttp re-raises is
|
// hook does with a transaction that missed the rename. The bare
|
||||||
// pinned separately, by TestSentryStillSeesAPanic.
|
// recover stands in for Middleware.Recoverer, which holds that outer
|
||||||
|
// slot in production; it is here only to keep panic stacks out of the
|
||||||
|
// test output. That the production one really does catch what
|
||||||
|
// sentryhttp re-raises is pinned separately, by
|
||||||
|
// TestSentryStillSeesAPanic.
|
||||||
func (c sentryCase) router() http.Handler {
|
func (c sentryCase) router() http.Handler {
|
||||||
handler := func(_ http.ResponseWriter, r *http.Request) {
|
handler := func(_ http.ResponseWriter, r *http.Request) {
|
||||||
// This call is what drains the body tee and fills the
|
// This call is what drains the body tee and fills the
|
||||||
@@ -353,8 +358,9 @@ func TestSentryScrub_KeepsTheRoutingContext(t *testing.T) {
|
|||||||
// Span.doFinish captures with a nil hint, so BeforeSendTransaction
|
// Span.doFinish captures with a nil hint, so BeforeSendTransaction
|
||||||
// gets one with no context and no request: the route pattern is out of
|
// gets one with no context and no request: the route pattern is out of
|
||||||
// reach and both the URL and the SDK-built transaction name have to
|
// reach and both the URL and the SDK-built transaction name have to
|
||||||
// fall back. Tracing is off in this service, so no transaction event
|
// fall back. This router has no nameSentryTransaction, so this is the
|
||||||
// is produced today; the hook is a floor against that changing.
|
// floor under a transaction that missed the rename;
|
||||||
|
// TestSentryTransaction_NamedByRoute covers the routes as they ship.
|
||||||
func TestSentryScrub_RedactsTheTransactionDispatch(t *testing.T) {
|
func TestSentryScrub_RedactsTheTransactionDispatch(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -376,6 +382,83 @@ func TestSentryScrub_RedactsTheTransactionDispatch(t *testing.T) {
|
|||||||
assert.Equal(t, "POST /(redacted)", event.Transaction)
|
assert.Equal(t, "POST /(redacted)", event.Transaction)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestSentryTransaction_NamedByRoute serves requests through the real
|
||||||
|
// route tree with tracing on, which the service leaves off. A routed
|
||||||
|
// request's transaction is named after its route pattern, so Sentry
|
||||||
|
// can group transactions by route, and the entrypoint UUID in the
|
||||||
|
// receiver path appears nowhere in the event. A path that matches
|
||||||
|
// nothing at the top level has no pattern, and its name is still
|
||||||
|
// withheld; one that matches no route inside a route group is named
|
||||||
|
// after the group's pattern.
|
||||||
|
func TestSentryTransaction_NamedByRoute(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// newTestEnv leaves the receiver rate limit at zero, which
|
||||||
|
// refuses every request before it reaches the receiver.
|
||||||
|
env := newTestEnvWithConfig(t, &config.Config{
|
||||||
|
DataDir: t.TempDir(),
|
||||||
|
Environment: config.EnvironmentDev,
|
||||||
|
ReceiverRateLimit: 10,
|
||||||
|
})
|
||||||
|
|
||||||
|
router := server.NewRouterWithSentryForTest(
|
||||||
|
t, env.log, env.cfg, env.mw, env.hnd,
|
||||||
|
)
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
method, path, want string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
http.MethodPost,
|
||||||
|
"/h/" + sentryReceiverUUID,
|
||||||
|
"POST /h/{uuid}",
|
||||||
|
},
|
||||||
|
// An admin page route group, which installs its own
|
||||||
|
// sentryhttp handler.
|
||||||
|
{http.MethodGet, "/pages/login", "GET /pages/login"},
|
||||||
|
{http.MethodGet, "/pages/unrouted", "GET /pages/*"},
|
||||||
|
{
|
||||||
|
http.MethodPost,
|
||||||
|
"/h/" + sentryReceiverUUID + "/unrouted",
|
||||||
|
"POST /(redacted)",
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(tc.want, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
transport := &captureTransport{}
|
||||||
|
|
||||||
|
opts := server.SentryClientOptionsForTest(
|
||||||
|
"https://public@sentry.invalid/1",
|
||||||
|
"webhooker-test",
|
||||||
|
)
|
||||||
|
opts.Transport = transport
|
||||||
|
opts.EnableTracing = true
|
||||||
|
opts.TracesSampleRate = 1.0
|
||||||
|
|
||||||
|
client, err := sentry.NewClient(opts)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(
|
||||||
|
sentry.SetHubOnContext(
|
||||||
|
context.Background(),
|
||||||
|
sentry.NewHub(client, sentry.NewScope()),
|
||||||
|
),
|
||||||
|
tc.method, tc.path, nil,
|
||||||
|
)
|
||||||
|
router.ServeHTTP(httptest.NewRecorder(), req)
|
||||||
|
|
||||||
|
event := onlyEvent(t, transport.events)
|
||||||
|
require.Equal(t, "transaction", event.Type)
|
||||||
|
|
||||||
|
assert.Equal(t, tc.want, event.Transaction)
|
||||||
|
assert.NotContains(
|
||||||
|
t, marshalEvent(t, event), sentryReceiverUUID,
|
||||||
|
)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestSentryScrub_TransactionDispatchIsUnscrubbedWithoutTheHook pins
|
// TestSentryScrub_TransactionDispatchIsUnscrubbedWithoutTheHook pins
|
||||||
// that dispatch's premise the same way, since it is the one the
|
// that dispatch's premise the same way, since it is the one the
|
||||||
// service does not exercise today.
|
// service does not exercise today.
|
||||||
|
|||||||
@@ -16,8 +16,7 @@ func NewStore(key []byte) *sessions.CookieStore {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// NewForTest creates a Session with a pre-configured cookie store for use
|
// NewForTest creates a Session with a pre-configured cookie store for use
|
||||||
// in tests. This bypasses the fx lifecycle and database dependency, allowing
|
// in tests. This bypasses the fx lifecycle and database dependency. The key
|
||||||
// middleware and handler tests to use real session functionality. The key
|
|
||||||
// parameter is the raw 32-byte authentication key used for session encryption
|
// parameter is the raw 32-byte authentication key used for session encryption
|
||||||
// and CSRF cookie signing.
|
// and CSRF cookie signing.
|
||||||
//
|
//
|
||||||
@@ -16,7 +16,6 @@ import (
|
|||||||
"go.uber.org/fx"
|
"go.uber.org/fx"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
"sneak.berlin/go/webhooker/internal/logger"
|
|
||||||
"sneak.berlin/go/webhooker/internal/reqtls"
|
"sneak.berlin/go/webhooker/internal/reqtls"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -80,7 +79,7 @@ type Params struct {
|
|||||||
|
|
||||||
Config *config.Config
|
Config *config.Config
|
||||||
Database *database.Database
|
Database *database.Database
|
||||||
Logger *logger.Logger
|
Logger *slog.Logger
|
||||||
}
|
}
|
||||||
|
|
||||||
// Session manages encrypted session storage.
|
// Session manages encrypted session storage.
|
||||||
@@ -180,7 +179,7 @@ func New(
|
|||||||
params Params,
|
params Params,
|
||||||
) (*Session, error) {
|
) (*Session, error) {
|
||||||
s := &Session{
|
s := &Session{
|
||||||
log: params.Logger.Get(),
|
log: params.Logger,
|
||||||
idleTimeout: params.Config.SessionIdleTimeout,
|
idleTimeout: params.Config.SessionIdleTimeout,
|
||||||
now: time.Now,
|
now: time.Now,
|
||||||
}
|
}
|
||||||
@@ -442,10 +441,9 @@ func (s *Session) Regenerate(
|
|||||||
//
|
//
|
||||||
// Whichever comes first ends the session.
|
// Whichever comes first ends the session.
|
||||||
//
|
//
|
||||||
// A session that claims to be authenticated but carries no
|
// A session that claims to be authenticated but lacks the timestamp
|
||||||
// timestamps predates this check; it is treated as expired so the
|
// a deadline is measured from is treated as expired, so the user
|
||||||
// user re-authenticates rather than being granted an unbounded
|
// re-authenticates rather than being granted an unbounded session.
|
||||||
// session.
|
|
||||||
func (s *Session) expired(sess *sessions.Session) bool {
|
func (s *Session) expired(sess *sessions.Session) bool {
|
||||||
now := s.now()
|
now := s.now()
|
||||||
|
|
||||||
|
|||||||
@@ -156,7 +156,8 @@ func TestVersion_IsStableAcrossInvocations(t *testing.T) {
|
|||||||
// The build target is the only place composing linker flags. If a
|
// The build target is the only place composing linker flags. If a
|
||||||
// future edit drops either half, the binary silently reports "dev"
|
// future edit drops either half, the binary silently reports "dev"
|
||||||
// again (the defect this package exists for) or fails to link on
|
// again (the defect this package exists for) or fails to link on
|
||||||
// Alpine.
|
// Alpine. If it stops passing GO_TAGS, the image's static binary links
|
||||||
|
// glibc's getaddrinfo and dlopen again.
|
||||||
func TestMakefile_BuildComposesVersionAndExtraFlags(t *testing.T) {
|
func TestMakefile_BuildComposesVersionAndExtraFlags(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -164,6 +165,7 @@ func TestMakefile_BuildComposesVersionAndExtraFlags(t *testing.T) {
|
|||||||
|
|
||||||
require.Contains(t, makefile, "-X main.version=$(VERSION)")
|
require.Contains(t, makefile, "-X main.version=$(VERSION)")
|
||||||
require.Contains(t, makefile, "$(GO_LDFLAGS)")
|
require.Contains(t, makefile, "$(GO_LDFLAGS)")
|
||||||
|
require.Contains(t, makefile, "-tags '$(GO_TAGS)'")
|
||||||
require.Contains(t, makefile, "VERSION ?= $(shell script/version)")
|
require.Contains(t, makefile, "VERSION ?= $(shell script/version)")
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -233,7 +235,9 @@ func TestDockerfile_BuildsThroughTheMakeTarget(t *testing.T) {
|
|||||||
"a raw go build bypasses the Makefile's -X flag")
|
"a raw go build bypasses the Makefile's -X flag")
|
||||||
require.Contains(t, dockerfile, "ARG VERSION")
|
require.Contains(t, dockerfile, "ARG VERSION")
|
||||||
require.Contains(t, dockerfile,
|
require.Contains(t, dockerfile,
|
||||||
`make build VERSION="$VERSION" GO_LDFLAGS='-extldflags "-static"'`)
|
`make build VERSION="$VERSION"`+
|
||||||
|
` GO_TAGS=netgo,osusergo,sqlite_omit_load_extension`+
|
||||||
|
` GO_LDFLAGS='-extldflags "-static"'`)
|
||||||
}
|
}
|
||||||
|
|
||||||
func read(t *testing.T, path string) string {
|
func read(t *testing.T, path string) string {
|
||||||
|
|||||||
+2
-2
@@ -2,8 +2,8 @@
|
|||||||
# script/assets: extract Alpine.js from its npm package tarball, committed
|
# script/assets: extract Alpine.js from its npm package tarball, committed
|
||||||
# in 3p/, to static/js/alpine.min.js, where go:embed reads it. The package
|
# in 3p/, to static/js/alpine.min.js, where go:embed reads it. The package
|
||||||
# is @alpinejs/csp, Alpine's build for pages whose Content-Security-Policy
|
# is @alpinejs/csp, Alpine's build for pages whose Content-Security-Policy
|
||||||
# forbids eval. The extracted file is not committed. script/test, make
|
# forbids eval. The extracted file is not committed. make build, make dev
|
||||||
# build and make dev run this first.
|
# and the Dockerfile's lint, test and build stages run this first.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|||||||
+12
-4
@@ -11,6 +11,7 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|||||||
|
|
||||||
PKGMGR=""
|
PKGMGR=""
|
||||||
SUDO=""
|
SUDO=""
|
||||||
|
APT_UPDATED=""
|
||||||
|
|
||||||
detect_pkgmgr() {
|
detect_pkgmgr() {
|
||||||
[ -n "$PKGMGR" ] && return 0
|
[ -n "$PKGMGR" ] && return 0
|
||||||
@@ -39,7 +40,14 @@ pkg_install() {
|
|||||||
detect_pkgmgr
|
detect_pkgmgr
|
||||||
case "$PKGMGR" in
|
case "$PKGMGR" in
|
||||||
nix) nix-env -iA "nixpkgs.$1" ;;
|
nix) nix-env -iA "nixpkgs.$1" ;;
|
||||||
apt) $SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y "$2" ;;
|
apt)
|
||||||
|
# Package lists may be empty (fresh images); refresh once per run.
|
||||||
|
if [ -z "$APT_UPDATED" ]; then
|
||||||
|
$SUDO env DEBIAN_FRONTEND=noninteractive apt-get update
|
||||||
|
APT_UPDATED=1
|
||||||
|
fi
|
||||||
|
$SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y "$2"
|
||||||
|
;;
|
||||||
brew) brew install "$3" ;;
|
brew) brew install "$3" ;;
|
||||||
apk) apk add --no-cache "$4" ;;
|
apk) apk add --no-cache "$4" ;;
|
||||||
esac
|
esac
|
||||||
@@ -60,10 +68,10 @@ main() {
|
|||||||
if missing go; then pkg_install go golang go go; fi
|
if missing go; then pkg_install go golang go go; fi
|
||||||
|
|
||||||
# Not installed here: docker is platform-specific and out of scope for a
|
# Not installed here: docker is platform-specific and out of scope for a
|
||||||
# package-manager bootstrap, but script/lint, script/fmt and script/css
|
# package-manager bootstrap, but script/test, script/lint, script/fmt and
|
||||||
# need it.
|
# script/css need it.
|
||||||
if missing docker; then
|
if missing docker; then
|
||||||
echo "bootstrap: docker not found; script/lint, script/fmt and script/css require it" >&2
|
echo "bootstrap: docker not found; script/test, script/lint, script/fmt and script/css require it" >&2
|
||||||
fi
|
fi
|
||||||
|
|
||||||
go mod download
|
go mod download
|
||||||
|
|||||||
+2
-2
@@ -1,7 +1,7 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/check: run all checks (test, lint, fmt-check, css-check). Our own
|
# script/check: run all checks (test, lint, fmt-check, css-check). Our own
|
||||||
# extension to scripts-to-rule-them-all.
|
# extension to scripts-to-rule-them-all. test, lint and css-check are
|
||||||
# Writes only the ignored static/js/alpine.min.js, through script/test.
|
# Docker builds; fmt-check runs gofmt on the host. Writes nothing.
|
||||||
# Generic, apart from css-check.
|
# Generic, apart from css-check.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
|
|||||||
@@ -1,152 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
# script/ci-mark-superseded: record an honest status on commits whose CI
|
|
||||||
# run Gitea cancelled because a newer commit landed on the same branch.
|
|
||||||
# Gitea writes `failure` / "Has been cancelled" for such a run, which
|
|
||||||
# reads as a test result on a commit nothing ever tested. Cancellation is
|
|
||||||
# unconditional server-side for push events, so the superseding run
|
|
||||||
# rewrites those statuses to `failure` with a description that says the
|
|
||||||
# commit was never tested. `skipped` cannot be used: Gitea's combined
|
|
||||||
# status folds `skipped` into `success`, so a never-tested commit would
|
|
||||||
# report green. Genuine failures and successes are never touched.
|
|
||||||
#
|
|
||||||
# Called by the Gitea Actions workflow, which supplies GITHUB_API_URL,
|
|
||||||
# GITHUB_REPOSITORY, GITHUB_SHA, GITHUB_WORKFLOW, GITHUB_JOB,
|
|
||||||
# GITHUB_EVENT_NAME and GITEA_TOKEN. ANCESTOR_LIMIT (default 20) caps how
|
|
||||||
# far back the walk looks; a value that is set but not a positive integer
|
|
||||||
# aborts rather than silently disabling the walk.
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
SUPERSEDED_DESC='Superseded by a newer commit; never tested'
|
|
||||||
|
|
||||||
# Gitea builds the commit-status context as
|
|
||||||
# "<workflow name> / <job name> (<event>)", so derive it rather than
|
|
||||||
# hardcoding the result.
|
|
||||||
#
|
|
||||||
# The derivation is deliberately not byte-exact with Gitea's own rule and
|
|
||||||
# must not be "fixed" into a silent fallback. Gitea uses the job's `name:`
|
|
||||||
# (falling back to the job id) and the workflow's `name:` (falling back to
|
|
||||||
# the workflow filename), while the runner exports GITHUB_JOB as the job
|
|
||||||
# *id* and GITHUB_WORKFLOW as the parsed workflow `name:`. So giving the
|
|
||||||
# job a display `name:`, or dropping the workflow's `name:`, makes the
|
|
||||||
# derived context stop matching --- and require_own_context below then
|
|
||||||
# turns every push red with a message. That loud failure is the point
|
|
||||||
# (https://git.eeqj.de/sneak/webhooker/issues/147 item 2); guessing at a
|
|
||||||
# fallback would restore the silent no-op it replaced.
|
|
||||||
context() {
|
|
||||||
printf '%s / %s (%s)' \
|
|
||||||
"$GITHUB_WORKFLOW" "$GITHUB_JOB" "$GITHUB_EVENT_NAME"
|
|
||||||
}
|
|
||||||
|
|
||||||
# ANCESTOR_LIMIT is a documented knob, so a value that is set but
|
|
||||||
# unusable must fail loudly instead of defaulting
|
|
||||||
# (https://git.eeqj.de/sneak/webhooker/issues/80). Passing it straight to
|
|
||||||
# git would print `fatal: not an integer` into a discarded exit status
|
|
||||||
# and mark nothing.
|
|
||||||
ancestor_limit() {
|
|
||||||
# `-` and not `:-`: an explicitly empty value is set-but-unusable
|
|
||||||
# config, so it aborts like any other bad value rather than silently
|
|
||||||
# running at the default.
|
|
||||||
_limit="${ANCESTOR_LIMIT-20}"
|
|
||||||
case "$_limit" in
|
|
||||||
'' | *[!0-9]* | 0*)
|
|
||||||
echo "ANCESTOR_LIMIT must be a positive integer," \
|
|
||||||
"got '${_limit}'" >&2
|
|
||||||
|
|
||||||
return 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
printf '%s' "$_limit"
|
|
||||||
}
|
|
||||||
|
|
||||||
# The status Gitea created for this very job proves which context string
|
|
||||||
# it uses. If the derived one is missing, the workflow or the job was
|
|
||||||
# renamed and the match below would silently stop firing, restoring the
|
|
||||||
# false-red bug with no signal. Fail loudly instead.
|
|
||||||
require_own_context() {
|
|
||||||
if ! _body="$(curl -sf --retry 3 --retry-delay 2 --max-time 30 \
|
|
||||||
"${1}/commits/${GITHUB_SHA}/status")"; then
|
|
||||||
echo "cannot read commit statuses for ${GITHUB_SHA}" >&2
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
_found="$(printf '%s' "$_body" | jq -r '(.statuses // [])[].context')"
|
|
||||||
if printf '%s\n' "$_found" | grep -qxF "$2"; then
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
echo "no commit status with context '${2}' on ${GITHUB_SHA}:" >&2
|
|
||||||
echo "workflow or job renamed? contexts present:" >&2
|
|
||||||
printf '%s\n' "$_found" >&2
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
|
|
||||||
# Latest status for our context on a commit, as "state|description".
|
|
||||||
# The read is retried and bounded, and a read that still fails aborts the
|
|
||||||
# step: a laundered commit that cannot be read is not the same as one
|
|
||||||
# with nothing to do, and piping curl into jq would discard the
|
|
||||||
# difference.
|
|
||||||
status_of() {
|
|
||||||
if ! _sbody="$(curl -sf --retry 3 --retry-delay 2 --max-time 30 \
|
|
||||||
"${1}/commits/${2}/status")"; then
|
|
||||||
echo "cannot read commit statuses for ${2}" >&2
|
|
||||||
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
printf '%s' "$_sbody" | jq -r --arg c "$3" \
|
|
||||||
'[(.statuses // [])[] | select(.context == $c)][0] // empty
|
|
||||||
| "\(.status)|\(.description)"'
|
|
||||||
}
|
|
||||||
|
|
||||||
mark_superseded() {
|
|
||||||
curl -sf -X POST "${1}/statuses/${2}" \
|
|
||||||
-H "Authorization: token ${GITEA_TOKEN}" \
|
|
||||||
-H 'Content-Type: application/json' \
|
|
||||||
-d "$(jq -nc --arg c "$3" --arg d "$SUPERSEDED_DESC" \
|
|
||||||
'{context: $c, state: "failure", description: $d}')" \
|
|
||||||
>/dev/null
|
|
||||||
}
|
|
||||||
|
|
||||||
main() {
|
|
||||||
_api="${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}"
|
|
||||||
_ctx="$(context)"
|
|
||||||
|
|
||||||
_limit="$(ancestor_limit)"
|
|
||||||
|
|
||||||
require_own_context "$_api" "$_ctx"
|
|
||||||
|
|
||||||
# A shallow clone cannot resolve the parent, so it looks exactly like
|
|
||||||
# a root commit to rev-parse below and would exit 0 having walked
|
|
||||||
# nothing (or, at depth > 1, only the ancestors that happen to be
|
|
||||||
# present). The workflow checks out with `fetch-depth: 0`; verify
|
|
||||||
# that here rather than depend on it silently.
|
|
||||||
if [ "$(git rev-parse --is-shallow-repository)" = 'true' ]; then
|
|
||||||
echo "shallow repository: the ancestor walk needs full history" >&2
|
|
||||||
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# A root commit legitimately has no ancestors and is not an error.
|
|
||||||
# A SHA this repository does not have lands here too, since its
|
|
||||||
# parent is equally unresolvable, but require_own_context above has
|
|
||||||
# already aborted on the 404 for it. The walk itself carries no
|
|
||||||
# `|| true`, so a rev-list failure aborts.
|
|
||||||
if ! git rev-parse -q --verify "${GITHUB_SHA}^" >/dev/null; then
|
|
||||||
echo "no ancestor of ${GITHUB_SHA} to check"
|
|
||||||
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
_walk="$(git rev-list --max-count="$_limit" "${GITHUB_SHA}^")"
|
|
||||||
|
|
||||||
for _sha in $_walk; do
|
|
||||||
_latest="$(status_of "$_api" "$_sha" "$_ctx")"
|
|
||||||
# A run that was cancelled, or one an earlier revision of this
|
|
||||||
# script laundered into `skipped`. Anything else stands.
|
|
||||||
case "$_latest" in
|
|
||||||
'failure|Has been cancelled' | "skipped|${SUPERSEDED_DESC}") ;;
|
|
||||||
*) continue ;;
|
|
||||||
esac
|
|
||||||
mark_superseded "$_api" "$_sha" "$_ctx"
|
|
||||||
echo "marked superseded: ${_sha}"
|
|
||||||
done
|
|
||||||
}
|
|
||||||
|
|
||||||
main "$@"
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user