16 Commits
Author SHA1 Message Date
clawbot fe69d20a12 Re-vendor the canonical files from sneak/prompts at dd4027b (closes #65)
check / check (push) Successful in 5m33s
The vendored files are fetched from sneak/prompts commit dd4027b. This
repository's own entries come after the canonical content, at the end of
each file: /bin in .dockerignore, the Go lines of .gitignore and [*.go]
in .editorconfig; the test-support deny list has no entries of its own.
The lint phase moves to golangci-lint v2.14.0. The build stage now takes
the version from git describe on the .git the build context carries,
unless VERSION is passed, and fails when .git is present but no version
comes out. The test phase drops -count=1, which the policy says it does
not need, and keeps its tmpfs build cache. One test calls Header.Get
with X-Real-IP, as canonicalheader asks.

Model: opus-5-5
2026-10-06 01:48:54 +00:00
clawbot 7f6f89cd83 Network lists: always allowed, exempt from rate limits, always refused (closes #19)
check / check (push) Successful in 3m47s
Adds SWWAF_ALLOW_NETS, SWWAF_RATE_LIMIT_EXEMPT_NETS and SWWAF_DENY_NETS,
read like SWWAF_TRUSTED_PROXIES and empty by default, and checked against
the client's own address before its country is looked up. A client in
SWWAF_ALLOW_NETS skips the country lists and the rate limits and is not
looked up. One in SWWAF_DENY_NETS is refused with 403, logged as denied
and not counted. One in SWWAF_RATE_LIMIT_EXEMPT_NETS is neither counted
nor refused by the rate limits. SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES now
refuses a private, loopback or link-local client unless SWWAF_ALLOW_NETS
lists it.

Judgement call: an address in both SWWAF_ALLOW_NETS and SWWAF_DENY_NETS is let through.
Judgement call: the size and time limits still apply to SWWAF_ALLOW_NETS.

Model: opus-5-5
2026-10-06 02:36:27 +02:00
clawbot df4cf769e0 Derive the proxy tests' short timeout setting from the timeout (closes #61)
check / check (push) Successful in 4m14s
shortTimeoutSetting is now shortTimeout.String(), so changing waitLimit
changes both together.

Model: opus-5-5
2026-10-04 11:50:08 +02:00
clawbot 8410cf05e6 Quiet the useradd and Perl warnings in the image build (closes #58)
check / check (push) Failing after 3s
useradd --system warns when the uid it is given is above SYS_UID_MAX,
999 on Ubuntu; --key raises that limit for this one call, so the uid
stays 65532.

minsysusers, which runit's install runs to create its _runit-log user,
prints a Perl warning because runit's sysusers line leaves out the
shell. It reads /etc/sysusers.d/runit.conf in place of runit's file, so
the image writes a copy of that line there, naming the shell minsysusers
gives anyway; the user it creates is unchanged.

The runsvinit warning stays, since it needs a change to runsvinit: its
reaper and its own wait on runsvdir race for the same exited process.

Model: opus-5-5
2026-10-04 11:38:14 +02:00
clawbot e7fb88af9e Spend less of make test writing the image and waiting (closes #56)
check / check (push) Failing after 45s
The test phase spends most of its time compiling with the race
detector from an empty build cache; then come writing the test image
and the internal/proxy tests.

- Go's build cache is on a tmpfs in the test phase, so its 137 MB are
  no longer written into the test image.
- TestUpgradedConnectionOutlastsTheTimeouts waits until just past
  shortTimeout after the answer to the upgrade was read, by when every
  timeout has started, rather than 7.5 s, so it ends with the other
  timing tests.
- shortTimeout is written as waitLimit / 2, as its comment says it is.

Model: opus-5-5
2026-10-04 11:13:32 +02:00
clawbot d4f90dba37 The image apps build FROM, with its health check (closes #45)
check / check (push) Failing after 3s
The Dockerfile's last stage is now the image of "Deployment" in SPEC.md:
Ubuntu 26.04 with ca-certificates, nix-bin and runit from a dated
snapshot whose InRelease files are checked by hash, nixpkgs from its
release file checked by SHA-256, runsvinit built at a fixed commit, and
smallwebwaf as a runit service. smallwebwaf answers
/_smallwebwaf/healthz, and `smallwebwaf healthcheck`, which takes no
further argument, is the image's HEALTHCHECK. script/example-app builds
an app on the image and checks it end to end.

The Nix profile comes last on the PATH: first, busybox from nixpkgs
replaced runit's own runsvdir and sv. SPEC.md is corrected to match
what was built.

Model: opus-5-5
2026-10-04 10:05:30 +02:00
clawbot 0750879e58 Country allow and deny lists, looked up through GeoJS (closes #44)
check / check (push) Failing after 3s
SWWAF_DENIED_COUNTRIES and SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES refuse a
request with 403 before its body is read or rate-limited, logged as
country_denied. internal/lookup asks GeoJS only while a list is set, 200
clients per request, one at a time, keeping answers 7 days. Failures, a
redirect or an answer leaving an address out included, are logged without
addresses; GeoJS is then left alone a second, doubling to five minutes.
Private, loopback and link-local clients are never sent.

Deviation, per the issue: no SWWAF_LOOKUP_SOURCE or SWWAF_LOOKUP_TIMEOUT; 403, not SWWAF_BAN_RESPONSE.
Deviation: GeoJS's country endpoint, not geo.json.
Judgement call: an IPv6 /64 is asked about by its first address; at most 10,000 clients wait.
Judgement call: config.go lists the ISO 3166-1 codes; no widely used library holds them.

Model: opus-5-5
2026-10-04 08:29:41 +02:00
clawbot 6977ff73df Proxy timing tests outlast a hold-up of the test process (closes #53)
check / check (push) Successful in 1m53s
smallwebwaf starts each timeout as the request arrives, before the step
a test needs first: the upgrade answered, the app's buffers full, the
first part of an answer passed on. A hold-up of the test process longer
than the 300 ms timeout ran it out before that step. No test can make
that step come first, and in the "waiting on the app" cases it cannot
see which side smallwebwaf was waiting on, so the timeout is now 5 s,
the hold-up wantTimedOut already allows. The timeout tests set when it
must not run out goes from 10 s to 1 m to stay clear of it. The upgrade
test waits 7.5 s past the upgrade.

Judgement call: one shared value; the proxy tests take about 8 s, not 2 s.

Model: opus-5-5
2026-10-04 07:53:18 +02:00
clawbot d730fcb57d Point the unreachable-app test at port 1 (closes #51)
check / check (push) Successful in 2m21s
TestAnswers502WhenTheAppCannotBeReached closed a listener and pointed
smallwebwaf at its port, which another test running in parallel could
open in between, so the test sometimes got that server's answer instead
of a 502. It now uses 127.0.0.1:1: nothing listens there, and a test
listening on port 0 is always given a port from 32768 up, so no test
can take it. No other test reuses a closed listener's port.

Model: opus-5-5
2026-10-04 06:39:07 +02:00
clawbot 6bd5f620f6 TestRequestTimeouts expects 504 when none of the body reached the app (closes #49)
check / check (push) Successful in 2m18s
In the cases where the app reads and the client stops sending halfway,
smallwebwaf waits on the client only once it has passed the first bytes
of the body to the app. A test process held up for the whole 300 ms
timeout before then rightly gets 504, as SPEC.md asks, so the test was
wrong to expect 408 every time.

The app in those cases now records whether it received any of the body.
Once the app has finished with the request, the case expects 408 and its
log line if it did, and 504 and its log line if not.

Model: opus-5-5
2026-10-04 06:09:22 +02:00
clawbot f51459fbfe Per-client request rate limits over a minute, an hour and a day (closes #43)
check / check (push) Successful in 2m50s
Each client, one IPv4 address or one IPv6 /64, is counted in two buckets
per window, the earlier weighted by how much of it the window covers; at
most 20,000 clients are kept, least recently seen dropped first. A
request over SWWAF_RATE_LIMIT_PER_MINUTE, _HOUR or _DAY (1000, 10000,
50000, or off) gets 429 before reaching the app. Refused requests count,
413s included. A clock set back over a second behind a bucket's start
restarts that window. The log line gains limit_hit and the action
rate_limited.

Deviation from SPEC.md, per the issue: the 20,000 bound and /64 are fixed.
Judgement call: golang-lru/v2 holds the table; httprate does not count refused requests.
Deviation: go.mod and go.sum hand-written; no make target tidies them.

Model: opus-5-5
2026-10-04 04:24:34 +02:00
clawbot bedd324f3c Add the MIT licence (closes #15)
check / check (push) Successful in 3m8s
LICENSE with the MIT text, the README naming MIT in its opening line and a
License section, and the licence field in package.json.

Model: opus-5-5
2026-10-04 03:07:52 +02:00
clawbot a399601c77 Settle the spec's open points after milestones 1 and 2 (closes #36)
check / check (push) Successful in 1m43s
The header size and the idle time stay fixed at their defaults through
milestone 2 and become settings in the first stage after it, which also
brings SWWAF_ALLOW_NETS and with it the allow-only country list's refusal
of private addresses. While a request body is on its way, a request
timeout answers 408 or 504 by the side smallwebwaf was waiting on, as
milestone 1's code does. The README says the state files, the GeoJS
answers among them, come in milestone 3 or later, and points at the
build order.

Model: opus-5-5
2026-10-04 02:57:24 +02:00
clawbot 2e35528c62 Settle the open points of the Ubuntu and nixpkgs image (closes #38)
check / check (push) Successful in 2m47s
ca-certificates, nix-bin and runit come from a dated Ubuntu snapshot no
older than the pinned Ubuntu image. The Dockerfile names the SHA-256 hash
of each snapshot InRelease file apt uses, and the build checks them before
apt-get install, so every package is checked against hashed files. That
install uses the Go image's CA certificate file. ca-certificates is
installed by name. The image writes build-users-group = to
/etc/nix/nix.conf so root can build without a daemon. nixpkgs comes from
its release file on releases.nixos.org, checked by SHA-256, and takes about
500 MiB of disk. runsvinit is archived upstream and is built at a fixed
commit with a go.mod written for the build. The example run scripts put
their code in a main function.

Model: opus-5-5
2026-10-04 02:42:56 +02:00
clawbot 983192ace3 Deploy model: listen port, token files, state directory owner (closes #33)
check / check (push) Successful in 2m37s
SWWAF_LISTEN_ADDR may set another port: the health check takes its port
from it, and traefik's port label must name the same one. Its address
part stays empty (:9000), so smallwebwaf keeps listening on every
address, where traefik and the health check on 127.0.0.1 both reach it.
A token file is made on the host owned by uid 65532 with mode 0400 and
its directory mounted read-only; through upaas, that directory is one of
the app's volume mounts. The run script of smallwebwaf makes the state
directory and every file in it belong to the smallwebwaf user.

Model: opus-5-5
2026-10-04 01:42:43 +02:00
clawbot d76715b0df Pass-through proxy with timeouts, size limits and a request log (closes #13)
check / check (push) Successful in 1m29s
Milestone 1, the repo's first code. smallwebwaf passes each request to the app and the answer back unchanged, streaming bodies and WebSocket upgrades, within four timeouts (client and app, request and response) and two size limits, and writes one JSON line per request to stdout. Every setting has an SWWAF_ name and a default, and an invalid value stops the start. The repo gets the standard layout: script/ entrypoints, make targets that call them, a Dockerfile that runs the checks, and the Gitea workflow.

Disclosure: SPEC.md changed. Go's server reads the request line and headers before smallwebwaf sees the request, so slow headers are closed without an answer, and neither slow nor oversized headers get a log line.
Disclosure: standard library only.

Model: opus-5-5
2026-10-03 17:24:34 +02:00
64 changed files with 8127 additions and 156 deletions
+75
View File
@@ -0,0 +1,75 @@
# .dockerignore does NOT use .gitignore semantics. Docker matches with
# moby/patternmatcher: filepath.Match plus `**`, so `*` does not cross
# `/` and an unprefixed pattern is anchored at the context root. Every
# depth-independent pattern therefore needs `**/`, or `config/.env` and
# `certs/server.key` still ship while this file reads as solved. Only
# genuinely root-anchored entries go unprefixed. Never transplant these
# into .gitignore, where `**/` is wrong.
#
# Matching is case-sensitive, so secrets use character ranges rather
# than an ALL-CAPS twin, which would still miss `Server.Key`.
#
# Extend with this repo's own host-built artifacts, written anchored:
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
# deletes the package directory from the context.
# .git is sent without its config. Without a VERSION build argument the
# stage that compiles runs `git describe --tags --always` on .git, which
# does not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there.
# Each submodule keeps a config with the same exposure in its git directory
# under .git/modules/, nested again for a submodule's own submodules, or in
# its own .git directory when it keeps one.
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
# `deploy/config`, `config/lib`) loses its whole git directory, because
# `**/.git/modules/**/config` also matches that segment's directory
# under .git/modules/. Go's version stamping then fails the build;
# nothing leaks. Name such a submodule without that segment:
# `git submodule add --name`.
**/.git/config
**/.git/modules/**/config
# Agent scratch: one full checkout of the repo per in-flight agent.
# Anchored because it occurs once where agents run at the repo root.
# KNOWN GAP: a repo running agents in subdirectories still ships
# `services/api/.claude/` and must add its own anchored entry.
.claude
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Re-include a committed template with a negation if the
# build needs one: `!docs/example.env`.
**/*.[eE][nN][vV]
**/.[eE][nN][vV].*
**/.[eE][nN][vV][rR][cC]
# Private keys and the bundles carrying them. Public certificates
# (*.crt, *.cer) are deliberately absent: they are legitimate inputs.
**/*.[pP][eE][mM]
**/*.[kK][eE][yY]
**/*.[pP]12
**/*.[pP][fF][xX]
**/[iI][dD]_[rR][sS][aA]
**/[iI][dD]_[dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
**/[iI][dD]_[eE][dD]25519
**/[iI][dD]_[eE][dD]25519_[sS][kK]
# Dependencies: restored inside the image, never copied in.
**/node_modules
# OS metadata.
**/.DS_Store
**/Thumbs.db
# Editor state: never a build input, and it churns COPY.
**/*.swp
**/*.swo
**/*~
**/*.bak
**/.idea
**/.vscode
**/*.sublime-*
# The binary `make build` writes on the host; the image builds its own.
/bin
+15
View File
@@ -0,0 +1,15 @@
root = true
[*]
indent_style = space
indent_size = 4
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
[Makefile]
indent_style = tab
[*.go]
indent_style = tab
+9
View File
@@ -0,0 +1,9 @@
name: check
on: [push]
jobs:
check:
runs-on: ubuntu-latest
steps:
# actions/checkout v4.2.2, 2026-02-22
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- run: script/cibuild
+53
View File
@@ -0,0 +1,53 @@
# OS
.DS_Store
Thumbs.db
# Editors
*.swp
*.swo
*~
*.bak
.idea/
.vscode/
*.sublime-*
# Agent scratch (worktrees of this repo, created and destroyed by
# in-flight tooling). Unanchored: .gitignore patterns already match at
# every depth, so no prefix is wanted here. This is not a .dockerignore
# entry and must not be given a `**/` prefix on the way into one.
.claude/
# Node
node_modules/
# Secrets. Unanchored like every entry above, so each matches at every
# depth. Matching is case-sensitive on Linux, so names use character
# ranges rather than a lowercase form that misses `Server.Key`.
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds
# its own negation after these lines, for example `!.env.example`.
*.[eE][nN][vV]
.[eE][nN][vV].*
.[eE][nN][vV][rR][cC]
!example.env
!sample.env
# Private keys and the bundles carrying them.
*.[pP][eE][mM]
*.[kK][eE][yY]
*.[pP]12
*.[pP][fF][xX]
[iI][dD]_[rR][sS][aA]
[iI][dD]_[dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# Go: the binary `make build` writes, test binaries, profiles and logs
/bin/
*.test
*.out
*.log
+99
View File
@@ -0,0 +1,99 @@
version: "2"
# Config schema uses the golangci-lint v2 layout (settings live under
# linters.settings, not top-level linters-settings) so that the
# thresholds below are actually applied by golangci-lint >= v2.
run:
timeout: 5m
modules-download-mode: readonly
linters:
default: all
enable:
# Successor to the deprecated gomodguard. Named explicitly, rather than
# left to `default: all`, because it carries the module policy below.
- gomodguard_v2
disable:
# Genuinely incompatible with project patterns
- exhaustruct # Requires all struct fields
- exhaustruct_v5 # Requires all struct fields (successor to exhaustruct)
- godot # Requires comments to end with periods
- wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go
# Deprecated: the warning is attached to the old name, so it is
# silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2
settings:
lll:
line-length: 88
funlen:
lines: 80
statements: 50
cyclop:
max-complexity: 15
dupl:
threshold: 100
depguard:
# Test-support code must not be compiled into the shipped binary. A
# test-support package exists to hand a test privileges the program
# itself must never have, so a file that is not a test must not import
# one. Test files, and the files inside a package whose directory name
# ends in `test`, are where that code belongs, and are exempt.
#
# The deny list below is the one part of this file a repository is
# expected to extend, and the only part it may. depguard matches an
# import path against a list of prefixes, so it cannot be told "any path
# whose last segment ends in test"; a repository's own test-support
# packages have to be named here one at a time, by full import path,
# under a module path that differs from repository to repository. Add
# them; change nothing else.
rules:
test-support:
list-mode: lax
files:
- "$all"
- "!$test"
- "!**/*test/**"
deny:
- pkg: net/http/httptest
desc: >-
Test-support code belongs in test files and in packages whose
directory name ends in test, not in the shipped binary.
# Only decisions already recorded in the Go package defaults are
# listed here. Every entry matches the module path exactly.
gomodguard_v2:
blocked:
- module: github.com/rs/zerolog
recommendations:
- log/slog
reason: "Structured logging is stdlib log/slog."
# One entry per pre-fork module path, because the later releases
# are separate paths. A prefix match would be shorter but would
# also reach github.com/go-redis/redismock, the test double for
# the successor these entries recommend.
- module: github.com/go-redis/redis
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v7
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v8
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/sergi/go-diff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "No unified diff output; use go-udiff."
- module: github.com/hexops/gotextdiff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "Unmaintained fork; use go-udiff."
issues:
max-issues-per-linter: 0
max-same-issues: 0
+2
View File
@@ -0,0 +1,2 @@
node_modules/
yarn.lock
+4
View File
@@ -0,0 +1,4 @@
{
"tabWidth": 4,
"proseWrap": "always"
}
+177
View File
@@ -0,0 +1,177 @@
# Lint phase. The linter is invoked directly rather than through `make
# lint` or `script/lint`, which are themselves a docker build and would
# recurse into a daemon that does not exist in a build step.
#
# golangci/golangci-lint v2.14.0 (built with go1.27.0), 2026-09-24
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN golangci-lint run --config .golangci.yml ./...
# Test phase, same shape and for the same reason. The go directive in
# go.mod is a minimum, so this Go may be newer than the linter's. The
# Debian image rather than the Alpine one, because the race detector
# needs the C compiler it carries.
#
# golang 1.27.1-trixie, 2026-09-19
FROM golang@sha256:3b77fc618ec235a1ab412de7737f120dd507c57e8d87de4cbb7994fb94275ed5 AS test
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# Go's build cache is kept on a tmpfs, out of the image: nothing uses it
# after this step, and writing it into the image takes seconds.
RUN --mount=type=tmpfs,target=/root/.cache/go-build \
go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from the two phases above; the copies
# are what make BuildKit build them first, so the image, which needs this
# stage, cannot be produced unless lint and test passed.
#
# golang 1.27.1-trixie, 2026-09-19
FROM golang@sha256:3b77fc618ec235a1ab412de7737f120dd507c57e8d87de4cbb7994fb94275ed5 AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
# This image has git. A tar-stream context keeps the sender's file
# owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context. With
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /usr/local/bin/smallwebwaf ./cmd/smallwebwaf
# runsvinit, the image's entrypoint, built at the last commit of its
# archived repository. It has no go.mod, and `go build` of its directory
# needs one; it uses only the standard library, so the one written here
# names nothing else.
#
# golang 1.27.1-trixie, 2026-09-19
FROM golang@sha256:3b77fc618ec235a1ab412de7737f120dd507c57e8d87de4cbb7994fb94275ed5 AS runsvinit
RUN git clone --quiet https://github.com/peterbourgon/runsvinit /src
WORKDIR /src
# runsvinit v2.0.0-8-gb4b2c78, 2015-10-07
RUN git checkout --quiet --detach b4b2c785308b1ce785b6155c7fe5f16879080193 \
&& go mod init github.com/peterbourgon/runsvinit \
&& CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" \
-o /usr/local/bin/runsvinit .
# The image an app's Dockerfile builds FROM, described under "Deployment"
# in SPEC.md. It is the last stage, so a plain `docker build .` builds it.
#
# ubuntu 26.04, 2026-09-27
FROM ubuntu@sha256:f144425ff09be612d6d9ad965196e9cdc23dae1f42110a8a11a3e9a8198759f7
# runit's install creates its _runit-log user with minsysusers, which
# reads this file in place of runit's /usr/lib/sysusers.d/runit.conf.
# runit's line leaves out the shell, and minsysusers prints a Perl
# warning for that; this copy of it names /sbin/nologin, the shell
# minsysusers gives when none is named.
RUN mkdir /etc/sysusers.d \
&& echo 'u _runit-log - "runit svlogd user" /nonexistent /sbin/nologin' \
> /etc/sysusers.d/runit.conf
# ca-certificates, nix-bin and runit, from Ubuntu's archive as it was at
# the snapshot moment, which is never earlier than the Ubuntu image above.
# apt checks every package against the snapshot's InRelease files, and
# this step checks those against the hashes named here, which are those
# of the amd64 archive: other architectures use Ubuntu's ports archive.
# apt also fetches the live archive's InRelease files, which change daily
# and which the install does not use. The snapshot service is HTTPS only
# and this image has no CA certificates yet, so this step uses the Go
# image's.
RUN --mount=type=bind,from=builder,source=/etc/ssl/certs/ca-certificates.crt,target=/tmp/go-image-ca.crt \
apt-get update --snapshot 20261001T000000Z \
-o Acquire::https::CaInfo=/tmp/go-image-ca.crt \
&& printf '%s\n' \
'45f95ce276cdba3e41870516a130e03c58b8b7a79e9546b0efe9e526d255740c snapshot.ubuntu.com_ubuntu_20261001T000000Z_dists_resolute_InRelease' \
'802e675dd9de4c7f3916434a95e7c1d8eec0e82886622d7805ab19a2c6fe0365 snapshot.ubuntu.com_ubuntu_20261001T000000Z_dists_resolute-updates_InRelease' \
'64b3353f0bd4970b4f7271962245bcea9ff24d4cc7bea16b433f8a60e42ca3dd snapshot.ubuntu.com_ubuntu_20261001T000000Z_dists_resolute-backports_InRelease' \
'1d5041572116a8b23aabf79ac7439ad8af83d57ad3fb0f9aa0d4523ec10c5908 snapshot.ubuntu.com_ubuntu_20261001T000000Z_dists_resolute-security_InRelease' \
| (cd /var/lib/apt/lists && sha256sum --check --strict) \
&& DEBIAN_FRONTEND=noninteractive apt-get install --yes --no-install-recommends \
--snapshot 20261001T000000Z \
-o Acquire::https::CaInfo=/tmp/go-image-ca.crt \
ca-certificates nix-bin runit \
&& rm -rf /var/lib/apt/lists/*
# Nix run by root expects a group of build users, which nix-bin does not
# create; with the setting empty, root's builds run without them.
RUN mkdir /etc/nix && echo 'build-users-group =' > /etc/nix/nix.conf
# nixpkgs, from its release file, checked by SHA-256, and set up for root
# as `nixpkgs`, so that an app's Dockerfile installs a package with
# `nix-env -iA nixpkgs.<name>`. curl and xz come with nix-bin.
#
# nixpkgs nixos-26.05.11045.774debe7a0d1, 2026-10-02
RUN curl -fsSL -o /tmp/nixexprs.tar.xz \
https://releases.nixos.org/nixos/26.05/nixos-26.05.11045.774debe7a0d1/nixexprs.tar.xz \
&& echo 'b2994104605601690023a5a6a3bb5a07b2bd1716b4e3b208cba1056dacd2ab08 /tmp/nixexprs.tar.xz' \
| sha256sum --check --strict \
&& mkdir -p /root/.nix-defexpr/nixpkgs \
&& tar -xJf /tmp/nixexprs.tar.xz -C /root/.nix-defexpr/nixpkgs --strip-components=1 \
&& rm /tmp/nixexprs.tar.xz
# What root installs with nix-env lands in root's profile. This path to
# it works for every user, unlike /root/.nix-profile: only root can
# enter /root. It comes last, so that no package shadows the image's
# own tools: busybox, for one, brings an sv that looks for services
# elsewhere.
ENV PATH=${PATH}:/nix/var/nix/profiles/default/bin
COPY --from=runsvinit /usr/local/bin/runsvinit /usr/local/bin/runsvinit
COPY --from=builder /usr/local/bin/smallwebwaf /usr/local/bin/smallwebwaf
# 65532 is above the uids Ubuntu keeps for system users, which end at
# 999; useradd warns about it unless --key raises that end for this call.
RUN groupadd --system --gid 65532 smallwebwaf \
&& useradd --system --key SYS_UID_MAX=65532 --uid 65532 \
--gid smallwebwaf --no-create-home --shell /usr/sbin/nologin \
smallwebwaf
# runsvinit starts runit's runsvdir on /etc/service, where Ubuntu's sv
# looks too.
COPY --chmod=755 share/smallwebwaf.run /etc/service/smallwebwaf/run
EXPOSE 8080
# traefik sends a container no requests until it is healthy, so the
# check runs every second from the start until it first passes, for up
# to a minute, and every 30 seconds after that.
HEALTHCHECK --start-period=1m --start-interval=1s \
CMD ["/usr/local/bin/smallwebwaf", "healthcheck"]
ENTRYPOINT ["/usr/local/bin/runsvinit"]
+51 -54
View File
@@ -17,8 +17,8 @@ configured only by environment variables, that provides:
- R5: temporary blocks for abusers, permanent bans for repeat offenders - R5: temporary blocks for abusers, permanent bans for repeat offenders
- R6: one or more RBLs or IP reputation APIs - R6: one or more RBLs or IP reputation APIs
- R7: AS number lookup - R7: AS number lookup
- R8: thresholds biased by AS number or country (for example, listed AS - R8: thresholds biased by AS number or country (for example, listed AS numbers
numbers get 50 percent of the normal limit) get 50 percent of the normal limit)
- R9: WAF-style attack detection and prevention - R9: WAF-style attack detection and prevention
- R10: runs as a plain env-var-configured sidecar between traefik and one app - R10: runs as a plain env-var-configured sidecar between traefik and one app
@@ -37,19 +37,19 @@ Closest existing options, and why each still falls short:
- BunkerWeb is the closest single product. - BunkerWeb is the closest single product.
- Meets: R1 (rates in requests per second, minute, hour or day), R2 - Meets: R1 (rates in requests per second, minute, hour or day), R2
(`LIMIT_IGNORE_IP`, also by AS number and reverse DNS), R3 partly (webhook, (`LIMIT_IGNORE_IP`, also by AS number and reverse DNS), R3 partly
Slack, Discord, Matrix plugins, fired only on denied requests; ntfy only (webhook, Slack, Discord, Matrix plugins, fired only on denied requests;
through the generic webhook, payload format unverified), R5 partly ntfy only through the generic webhook, payload format unverified), R5
(`BAD_BEHAVIOR_BAN_TIME`, `0` means permanent; no escalation for repeat partly (`BAD_BEHAVIOR_BAN_TIME`, `0` means permanent; no escalation for
offenders, the ban length is one fixed value), R6 (DNSBL plugin, external repeat offenders, the ban length is one fixed value), R6 (DNSBL plugin,
blacklist URLs, optional CrowdSec), R7 partly (AS number used for external blacklist URLs, optional CrowdSec), R7 partly (AS number used for
blacklist and whitelist decisions), R9 (ModSecurity with the Core Rule blacklist and whitelist decisions), R9 (ModSecurity with the Core Rule
Set, or Coraza plugin), env-var settings. Set, or Coraza plugin), env-var settings.
- Fails: R8 (AS number and country can only allow or deny, never scale a - Fails: R8 (AS number and country can only allow or deny, never scale a
limit), R4 (no volume or byte threshold alerts in the free edition; limit), R4 (no volume or byte threshold alerts in the free edition;
reporting is a paid feature), R5 escalation, and R10 in spirit: since reporting is a paid feature), R5 escalation, and R10 in spirit: since 1.6
1.6 it needs a `bunkerweb` container plus a `bw-scheduler` container and it needs a `bunkerweb` container plus a `bw-scheduler` container and a
a database, or the all-in-one image that bundles nginx, scheduler, UI and database, or the all-in-one image that bundles nginx, scheduler, UI and
Redis in one container. It is designed to be the front door for many Redis in one container. It is designed to be the front door for many
sites, not a per-app sidecar. AGPL-3.0. sites, not a per-app sidecar. AGPL-3.0.
- Unverified: whether several rates (minute, hour, day) can be stacked on - Unverified: whether several rates (minute, hour, day) can be stacked on
@@ -61,15 +61,14 @@ Closest existing options, and why each still falls short:
(community blocklist, further blocklists, reputation API), R7 (alerts are (community blocklist, further blocklists, reputation API), R7 (alerts are
enriched with AS number and country), R9 (AppSec component with virtual enriched with AS number and country), R9 (AppSec component with virtual
patching and ModSecurity-syntax rules), R2 (allowlists). patching and ModSecurity-syntax rules), R2 (allowlists).
- Partly: R1 and R8. Detection is by leaky-bucket scenarios over logs, and - Partly: R1 and R8. Detection is by leaky-bucket scenarios over logs, and a
a scenario can filter on AS number or country, so a stricter bucket for scenario can filter on AS number or country, so a stricter bucket for
listed AS numbers is possible, but each is a hand-written YAML scenario, listed AS numbers is possible, but each is a hand-written YAML scenario,
it reacts after the fact by banning, and it is not an inline limiter that it reacts after the fact by banning, and it is not an inline limiter that
answers 429. answers 429.
- Fails: R10 (needs the security engine container with persistent state, - Fails: R10 (needs the security engine container with persistent state, log
log acquisition from traefik, a bouncer such as the traefik plugin, and acquisition from traefik, a bouncer such as the traefik plugin, and YAML
YAML for acquisition, profiles, scenarios and notifications), bytes half for acquisition, profiles, scenarios and notifications), bytes half of R4.
of R4.
- CrowdSec plus traefik's own `rateLimit` middleware is the best combination - CrowdSec plus traefik's own `rateLimit` middleware is the best combination
with no new code. It gives inline limiting (one window per middleware, keyed with no new code. It gives inline limiting (one window per middleware, keyed
by IP, with `sourceCriterion` exclusions), bans with escalation, alerts and by IP, with `sourceCriterion` exclusions), bans with escalation, alerts and
@@ -89,19 +88,19 @@ Closest existing options, and why each still falls short:
central API, or AppSec only. Supports captcha remediation. central API, or AppSec only. Supports captcha remediation.
- Covers: R5, R6, R9, R3 and R7 through the engine (see Verdict). - Covers: R5, R6, R9, R3 and R7 through the engine (see Verdict).
- Misses: the plugin itself does no rate limiting; R8; R4 bytes. - Misses: the plugin itself does no rate limiting; R8; R4 bytes.
- Configuration: traefik static config to load the plugin, dynamic config - Configuration: traefik static config to load the plugin, dynamic config or
or labels for the middleware, CrowdSec YAML for everything else. labels for the middleware, CrowdSec YAML for everything else.
- Sidecar fit: no. It lives inside traefik, plus a separate engine - Sidecar fit: no. It lives inside traefik, plus a separate engine
container. container.
- Maturity: widely used (about 900 stars, listed in the traefik plugin - Maturity: widely used (about 900 stars, listed in the traefik plugin
catalog, documented by CrowdSec itself), actively maintained. Traefik catalog, documented by CrowdSec itself), actively maintained. Traefik
plugins run in an interpreter inside traefik, which costs some plugins run in an interpreter inside traefik, which costs some per-request
per-request time. time.
- CrowdSec generic bouncers (nginx, Caddy, firewall) - CrowdSec generic bouncers (nginx, Caddy, firewall)
- Same engine, different enforcement point. The firewall bouncer blocks at - Same engine, different enforcement point. The firewall bouncer blocks at
nftables level on the host, which is cheap and covers every service on nftables level on the host, which is cheap and covers every service on the
the host at once; worth considering fleet-wide regardless of this host at once; worth considering fleet-wide regardless of this project. Not
project. Not a sidecar, same misses as above. a sidecar, same misses as above.
- BunkerWeb 1.6.14 (`bunkerity/bunkerweb`) - BunkerWeb 1.6.14 (`bunkerity/bunkerweb`)
- What it is: nginx with Lua plugins, ModSecurity and the Core Rule Set, - What it is: nginx with Lua plugins, ModSecurity and the Core Rule Set,
configured by settings that are passed as env vars to its scheduler configured by settings that are passed as env vars to its scheduler
@@ -123,23 +122,22 @@ Closest existing options, and why each still falls short:
edition is capped at 10 applications. edition is capped at 10 applications.
- Configuration: web UI backed by PostgreSQL. No env-var configuration. - Configuration: web UI backed by PostgreSQL. No env-var configuration.
- Sidecar fit: no. Seven containers (postgres, management, detector, - Sidecar fit: no. Seven containers (postgres, management, detector,
tengine, and three helpers); the proxy container uses host networking. tengine, and three helpers); the proxy container uses host networking. The
The detection engine is closed source. detection engine is closed source.
- Maturity: very active, vendor-driven. - Maturity: very active, vendor-driven.
- Coraza (`corazawaf/coraza`) and Coraza-based proxies - Coraza (`corazawaf/coraza`) and Coraza-based proxies
- What it is: a Go library that implements the ModSecurity rule language - What it is: a Go library that implements the ModSecurity rule language and
and runs the OWASP Core Rule Set. OWASP project, actively maintained, the runs the OWASP Core Rule Set. OWASP project, actively maintained, the
successor path now that ModSecurity is in maintenance only. successor path now that ModSecurity is in maintenance only.
- Packagings: `coraza-caddy` (Caddy module), `coraza-spoa` (HAProxy), - Packagings: `coraza-caddy` (Caddy module), `coraza-spoa` (HAProxy),
`coraza-proxy-wasm` (Envoy), a traefik WASM plugin, and `coraza-proxy-wasm` (Envoy), a traefik WASM plugin, and
`coreruleset/coraza-crs-docker` (Caddy plus Coraza plus the Core Rule `coreruleset/coraza-crs-docker` (Caddy plus Coraza plus the Core Rule Set,
Set, with env vars for backend address, engine mode and rule-set with env vars for backend address, engine mode and rule-set tuning).
tuning).
- Covers: R9 only. `coraza-crs-docker` fits R10 well: one container, env - Covers: R9 only. `coraza-crs-docker` fits R10 well: one container, env
vars, backend address. vars, backend address.
- Misses: R1 to R8. ModSecurity-language rules can count requests per IP - Misses: R1 to R8. ModSecurity-language rules can count requests per IP in
in a persistent collection, but Coraza's support for persistent a persistent collection, but Coraza's support for persistent collections
collections is limited and this is not a practical rate limiter. is limited and this is not a practical rate limiter.
- Value here: the right library to embed for R9 in a purpose-built sidecar. - Value here: the right library to embed for R9 in a purpose-built sidecar.
- ModSecurity Core Rule Set containers (`owasp/modsecurity-crs`) - ModSecurity Core Rule Set containers (`owasp/modsecurity-crs`)
- What it is: official images of Apache or nginx with ModSecurity and the - What it is: official images of Apache or nginx with ModSecurity and the
@@ -149,21 +147,21 @@ Closest existing options, and why each still falls short:
- Covers: R9, and R10 (single container, env vars, one backend). - Covers: R9, and R10 (single container, env vars, one backend).
- Misses: R1 to R8. - Misses: R1 to R8.
- Maturity: rule set is very actively maintained; the ModSecurity engine - Maturity: rule set is very actively maintained; the ModSecurity engine
itself is in maintenance under OWASP after Trustwave ended support in itself is in maintenance under OWASP after Trustwave ended support
2024. in 2024.
- Anubis (`TecharoHQ/anubis`), 1.27 current - Anubis (`TecharoHQ/anubis`), 1.27 current
- What it is: a single-binary reverse proxy that makes browsers solve a - What it is: a single-binary reverse proxy that makes browsers solve a
proof-of-work challenge before passing them to `TARGET`. Aimed at proof-of-work challenge before passing them to `TARGET`. Aimed at
scrapers, which is likely a large share of unwanted traffic on a public scrapers, which is likely a large share of unwanted traffic on a public
gitea. gitea.
- Covers: R10 well (one container, env vars for listener, target, - Covers: R10 well (one container, env vars for listener, target,
difficulty, cookies). Policy rules can match on path, user agent, difficulty, cookies). Policy rules can match on path, user agent, headers,
headers, IP ranges, and with the vendor's hosted data service also AS IP ranges, and with the vendor's hosted data service also AS number and
number and country, and can weigh a request toward a harder challenge. country, and can weigh a request toward a harder challenge.
- Misses: R1, R3, R4, R5, R6, R9. Bot policy needs a YAML file, not env - Misses: R1, R3, R4, R5, R6, R9. Bot policy needs a YAML file, not env
vars. AS number and country matching depend on the vendor's hosted vars. AS number and country matching depend on the vendor's hosted
service. Breaks non-browser clients unless paths are exempted; for service. Breaks non-browser clients unless paths are exempted; for gitea,
gitea, git-over-HTTP and API paths must be allowed through by rule. git-over-HTTP and API paths must be allowed through by rule.
- Maturity: very active, widely deployed on code forges since 2025. - Maturity: very active, widely deployed on code forges since 2025.
- Value here: complementary. It can be chained (traefik, then the sidecar, - Value here: complementary. It can be chained (traefik, then the sidecar,
then Anubis, then the app) if challenge pages are wanted. then Anubis, then the app) if challenge pages are wanted.
@@ -187,29 +185,28 @@ Closest existing options, and why each still falls short:
- Sidecar fit: no; they live inside traefik. - Sidecar fit: no; they live inside traefik.
- Traefik built-in middlewares - Traefik built-in middlewares
- `rateLimit` (one average-and-burst window per middleware, optional Redis - `rateLimit` (one average-and-burst window per middleware, optional Redis
in traefik 3.x), `inFlightReq`, `ipAllowList`. No bans, alerts, in traefik 3.x), `inFlightReq`, `ipAllowList`. No bans, alerts, reputation
reputation or AS number awareness. or AS number awareness.
- caddy-waf (`fabriziosalmi/caddy-waf`), 0.4.x - caddy-waf (`fabriziosalmi/caddy-waf`), 0.4.x
- What it is: a Caddy module with regex rules and anomaly scoring, per-IP - What it is: a Caddy module with regex rules and anomaly scoring, per-IP
and per-path rate limiting with one configurable window, IP and DNS and per-path rate limiting with one configurable window, IP and DNS
blacklists, Tor exit list fetch, country and AS number allow or deny blacklists, Tor exit list fetch, country and AS number allow or deny from
from MaxMind databases. MaxMind databases.
- Misses: R8 (allow or deny only), R3, R4, R5 (no documented ban state or - Misses: R8 (allow or deny only), R3, R4, R5 (no documented ban state or
alerting), R1's three windows. Caddyfile configuration. One maintainer, alerting), R1's three windows. Caddyfile configuration. One maintainer,
pre-1.0, AGPL-3.0. pre-1.0, AGPL-3.0.
- open-appsec (Check Point) - open-appsec (Check Point)
- Machine-learning WAF agent attached to nginx, Kong, Envoy or similar, - Machine-learning WAF agent attached to nginx, Kong, Envoy or similar, with
with a declarative policy file or the vendor's cloud console. Covers R9 a declarative policy file or the vendor's cloud console. Covers R9 only;
only; rate limiting and richer features are in paid tiers. Not a sidecar rate limiting and richer features are in paid tiers. Not a sidecar in the
in the required sense. required sense.
- iocaine - iocaine
- Serves generated garbage pages to clients the fronting proxy classifies - Serves generated garbage pages to clients the fronting proxy classifies as
as scrapers. Not a limiter, WAF or ban tool; out of scope except as a scrapers. Not a limiter, WAF or ban tool; out of scope except as a
curiosity for scraper traffic. curiosity for scraper traffic.
- Pangolin - Pangolin
- A tunnelled access platform that bundles traefik and optionally - A tunnelled access platform that bundles traefik and optionally CrowdSec.
CrowdSec. Replaces the ingress rather than adding a sidecar; out of Replaces the ingress rather than adding a sidecar; out of scope.
scope.
## Requirement by requirement, across the field ## Requirement by requirement, across the field
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 sneak
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+43
View File
@@ -0,0 +1,43 @@
.PHONY: bootstrap setup test lint fmt fmt-check check docker hooks build run \
example-app
# Makefile targets are thin shims; the implementations live in script/
# per the scripts-to-rule-them-all pattern (see the Entrypoints section
# of README.md). build and run are for working on the code by hand;
# example-app checks the image with an app built on it.
bootstrap:
@script/bootstrap
setup:
@script/setup
test:
@script/test
lint:
@script/lint
fmt:
@script/fmt
fmt-check:
@script/fmt-check
check:
@script/check
docker:
@script/docker
hooks:
@script/install-precommit
build:
@script/build
run:
@script/run
example-app:
@script/example-app
+339 -41
View File
@@ -1,18 +1,202 @@
# smallwebwaf # smallwebwaf
`smallwebwaf` is a simple, fast, logging web application firewall for people who `smallwebwaf` is a simple, fast, logging web application firewall, MIT-licensed
host their own services. It runs inside the container of the one application it and written in Go by [@sneak](https://sneak.berlin), for people who host their
protects, between your reverse proxy (traefik) and the app: the app's Dockerfile own services. It runs inside the container of the one application it protects,
builds `FROM` the `smallwebwaf` image, traefik sends the app's requests to between your reverse proxy (traefik) and the app: the app's Dockerfile builds
`FROM` the `smallwebwaf` image, traefik sends the app's requests to
`smallwebwaf` on port 8080, and `smallwebwaf` passes them on to the app on `smallwebwaf` on port 8080, and `smallwebwaf` passes them on to the app on
`127.0.0.1:8081`. It needs no setting, and protects the app from the first `127.0.0.1:8081`. It needs no setting, and protects the app from the first
request with defaults chosen for a service on the open internet. It keeps its request with defaults chosen for a service on the open internet. It keeps its
state in memory and in JSON files you can read and edit, and writes a detailed state in memory and in JSON files you can read and edit, and writes a detailed
JSON log line for every request. JSON log line for every request.
Status: design stage. This repository currently holds the documents only; no Status: the first two milestones are built
code has been written. The design is in [`SPEC.md`](SPEC.md), and the survey of (https://git.eeqj.de/sneak/smallwebwaf/issues/13 and
existing tools that led to it is in [`EVALUATION.md`](EVALUATION.md). https://git.eeqj.de/sneak/smallwebwaf/issues/14), and so are the static lists,
which come next in the build order. `smallwebwaf` passes each request to the app
and the app's answer back, unchanged, within its timeouts and size limits, works
out each client's address, refuses a client that sends too many requests, comes
from a country you refuse or from a network you refuse, lets the networks you
choose through, and writes a JSON log line for every request. It comes as the
image the app's own image is built on. The rest of the design comes after that,
in the order of the build order in [`SPEC.md`](SPEC.md). The survey of existing
tools that led to the design is in [`EVALUATION.md`](EVALUATION.md).
## Getting started
Build the `smallwebwaf` image from a clone:
```sh
git clone https://git.eeqj.de/sneak/smallwebwaf.git
cd smallwebwaf
make docker
```
`make docker` runs the tests and the linter, then builds the image, tagged
`smallwebwaf`, for amd64 and only on an amd64 host: the hashes the `Dockerfile`
checks Ubuntu's package lists against are those of Ubuntu's amd64 archive. Push
it to a registry your hosts pull from, and build each app's image on it, pinned
by digest, as "How it works, in short" below shows. `make example-app` builds a
small app on the image, the one in `deploy/example-app`, and checks that it
works.
To work on the code, `make build` builds the binary alone, with Go installed,
and `make run` builds and runs it, listening on port 8080 in front of an app at
`SWWAF_UPSTREAM_URL`, by default `http://127.0.0.1:8081`.
## What it does so far
- Passes each request to the app and the app's answer back unchanged: method,
path, query, headers, body and status. Bodies stream through in both
directions and are never held whole in memory. A WebSocket, or any other
upgraded connection, passes through, and the timeouts do not cut it.
- Works out the client's address. A TCP peer outside `SWWAF_TRUSTED_PROXIES` is
the client, and the forwarded headers it sends are replaced, not passed on.
For a peer inside it, `X-Forwarded-For` is read from the right, and the first
address outside `SWWAF_TRUSTED_PROXIES` is the client; if every address in it
is inside, the leftmost is, and with no header the peer is. The app sees what
it would see from traefik directly: the same `Host`, the same
`X-Forwarded-Proto`, and `X-Forwarded-For` with the peer added at the end.
- Enforces the four timeouts and the two size limits below. A limit passed
before the response has started gets `smallwebwaf`'s own answer: `408` for a
client too slow to send its request, `413` for a request body that is too
large, `504` for an app too slow to answer, and `502` for a response that is
too large or an app that cannot be reached. A request that announces a body
over the limit is refused before anything reaches the app. While a request
body is still on its way, a request timeout that runs out answers `408` if
`smallwebwaf` was waiting for the client to send more, and `504` if it was
waiting for the app to take what it had. Once the response has started, a
limit can only cut the connection.
- Counts each client's requests over a minute, an hour and a day. A request that
takes the client over one of the rate limits below is refused with `429`
before anything reaches the app, and so is each request after it until the
client is back under every limit. A client is one IPv4 address, or one IPv6
/64, since one abuser usually holds a whole /64. Refused requests count too,
so a client that keeps sending too fast stays refused until it slows down.
Each window is counted in two fixed buckets, the earlier one weighted by how
much of it the window still covers. At most 20,000 clients are kept, the least
recently seen dropped first, and only in memory: a restart starts every client
afresh.
- Refuses a request from a country you refuse with `403`, as soon as the
client's country is known and before its body is read; such a request is not
counted for the rate limits. While one of the country lists below is set, each
client's country is looked up through GeoJS (see "Country and AS number
lookup" below); with neither set, no visitor's address leaves the host. A
client on a private, loopback or link-local address has no country and is
never looked up: `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless it is
in `SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it.
- Checks the client's own address against the static lists, the three netblock
settings below, before anything else, its country included. A client in
`SWWAF_ALLOW_NETS` skips the country lists and the rate limits, and is not
looked up; the timeouts and size limits still apply. A client in
`SWWAF_DENY_NETS` is refused with `403` before its body is read, and the
request is not counted for the rate limits; an address in `SWWAF_ALLOW_NETS`
too is let through. A client in `SWWAF_RATE_LIMIT_EXEMPT_NETS` is neither
counted nor refused by the rate limits; the country lists still apply to it.
- Answers `GET /_smallwebwaf/healthz` itself with `200` and `ok`, before any
check and without asking the app, for the image's health check.
- Writes a line in the request log for each request (see "Request log" below).
## Settings
Each setting is an environment variable, and each has a default, so none has to
be set. A setting that is set but invalid stops the start with a message naming
it, and the effective settings are logged at start.
- `SWWAF_LISTEN_ADDR` (default `:8080`): where `smallwebwaf` listens.
- `SWWAF_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app, as `http` or
`https`, a host and an optional port, and nothing more.
- `SWWAF_TRUSTED_PROXIES` (default `10.0.0.0/8,172.16.0.0/12,192.168.0.0/16`,
the private address ranges): the netblocks whose `X-Forwarded-For` is
believed. A list given replaces the default; set but empty, it trusts nothing.
- `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take to
send its request line and headers, and then, from the end of the headers, its
body.
- `SWWAF_CLIENT_RESPONSE_TIMEOUT` (default `30m`): how long the response may
take to reach the client, from the end of the request to the last byte.
- `SWWAF_UPSTREAM_REQUEST_TIMEOUT` (default `60s`): how long connecting to the
app and sending it the whole request may take.
- `SWWAF_UPSTREAM_RESPONSE_TIMEOUT` (default `30m`): how long the app may take
to send its whole answer, from the end of the request to the last byte.
- `SWWAF_REQUEST_MAX_BYTES` (default `100M`): the largest request body.
- `SWWAF_RESPONSE_MAX_BYTES` (default `5G`): the largest response body.
- `SWWAF_ALLOW_NETS` (default empty): netblocks whose clients skip the country
lists and the rate limits, such as your monitoring or your own networks.
- `SWWAF_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients the
rate limits do not apply to, such as a machine that talks to the app all day.
- `SWWAF_DENY_NETS` (default empty): netblocks whose clients are always refused.
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `1000`), `SWWAF_RATE_LIMIT_PER_HOUR`
(default `10000`) and `SWWAF_RATE_LIMIT_PER_DAY` (default `50000`): the most
requests a client may make in a minute, an hour and a day. The defaults are
several times what one busy person produces, since a browser loading a heavy
page makes a few hundred requests and several people often share one address.
- `SWWAF_DENIED_COUNTRIES` (default empty): countries whose clients are refused,
for example `cn,ru,kp`.
- `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` (default empty): when set, the only
countries whose clients get through, for example `us,de`. A client whose
country cannot be found is refused too, so that new clients are not let in
whenever GeoJS stops answering.
Durations are in Go's syntax, with `d` for days (`90s`, `15m`, `7d`). Sizes are
bytes, with an optional `K`, `M` or `G`, which are powers of 1024 (`1K` is 1024
bytes). Rate limits are whole numbers of requests. Netblocks are in CIDR form,
and a bare address stands for itself alone. Countries are the two-letter codes
ISO 3166-1 assigns today, and `xk` for Kosovo, in either case (`de` and `DE` are
the same); any other code, such as `nk` (North Korea is `kp`) or the withdrawn
`su`, stops the start, and so does a code on both country lists. `off` switches
a timeout, a size limit or a rate limit off.
Several limits are fixed rather than settings. The request line and headers may
take up to 32 KiB, above which the answer is `431` and nothing reaches the app.
A kept-open connection that sends nothing for 120 seconds is closed. That is
longer than the 90 seconds after which traefik closes a connection it is not
using, so traefik never sends a request on a connection `smallwebwaf` is
closing. At most 20,000 clients are kept for the rate limits, and an IPv6 client
is counted by its /64. A new client waits at most a second for its country, and
at most 100,000 answers from GeoJS are kept, for 7 days each.
## Request log
`smallwebwaf` writes one JSON object per line on stdout for every request,
refused ones included:
```
{"type":"request","time":"2026-10-03T12:00:00.123Z","client_ip":"203.0.113.9","peer_ip":"172.18.0.2","country":"DE","method":"GET","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"upstream_status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","action":"forward","duration_total":3.217,"duration_upstream_total":3.104}
```
- `time` is when the request arrived, in UTC. `peer_ip` is the TCP peer,
normally traefik. `path` and `query` are as the client sent them.
- `country` is the client's country as GeoJS places it, and empty when it is not
known: with neither country list set, for a client in `SWWAF_ALLOW_NETS` or
`SWWAF_DENY_NETS`, for a client on a private, loopback or link-local address,
and when GeoJS cannot place the client or has not answered in time.
- `status` is what the client was sent, `0` if nothing was; `upstream_status` is
what the app answered, and is left out when the app did not answer.
- `request_bytes` and `response_bytes` count body bytes.
- `action` is `forward` for a request passed to the app, `denied` for one
refused because its client is in `SWWAF_DENY_NETS`, `country_denied` for one
refused for its client's country, `rate_limited` for one refused for a rate
limit, `too_large` for a request or response over its size limit, `timed_out`
for one that ran out of time, `upstream_error` when the app could not be
reached or its answer broke off, and `admin` for one `smallwebwaf` answered at
its own endpoint.
- `limit_hit` is there for a request refused for a rate limit, and names the
window whose limit it went over: `minute`, `hour` or `day`, the shortest if it
went over several.
- `aborted` is there, and true, when the client went away early.
- `duration_total` and `duration_upstream_total` are in milliseconds.
No body and no other header is logged. `smallwebwaf`'s own messages (start, the
settings, stop, errors) share the stream as JSON lines marked
`"type":"process"`.
Go's HTTP server, on which `smallwebwaf` is built, reads a request's line and
headers before `smallwebwaf` sees the request, and some requests end there,
without a line in the log: headers over 32 KiB, which it answers `431`, headers
slower than `SWWAF_CLIENT_REQUEST_TIMEOUT`, whose connection it closes without
an answer, and requests it cannot read at all, which it answers itself, mostly
with `400`.
## Why ## Why
@@ -114,7 +298,10 @@ goes through the candidates one by one.
answers, the reputation cache, the alerting state) held in memory and kept in answers, the reputation cache, the alerting state) held in memory and kept in
readable JSON files, written regularly and at every stop, so a restart loses readable JSON files, written regularly and at every stop, so a restart loses
nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks nothing. Edit a file, or add a rule file, and the running `smallwebwaf` picks
up the change. Nothing is read from disk while serving a request. up the change. Nothing is read from disk while serving a request. The files
come in milestone 3 or later (see the build order in [`SPEC.md`](SPEC.md));
until then the rate counters and the GeoJS answers are kept in memory only,
and a restart loses them.
- Health checks, the metrics, and listing, adding and lifting bans or asking why - Health checks, the metrics, and listing, adding and lifting bans or asking why
a given address was refused, all on the one port every request uses: under a given address was refused, all on the one port every request uses: under
`/_smallwebwaf/` on the app's own address, through traefik like any other `/_smallwebwaf/` on the app's own address, through traefik like any other
@@ -175,19 +362,24 @@ stand for the app's own options:
```bash ```bash
#!/usr/bin/env bash #!/usr/bin/env bash
set -euo pipefail set -euo pipefail
sleep 1
exec chpst -u app:app /usr/local/bin/app \ main() {
--listen 127.0.0.1:8081 \ sleep 1
--trusted-proxies 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1/32,::1/128 exec chpst -u app:app /usr/local/bin/app \
--listen 127.0.0.1:8081 \
--trusted-proxies 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1/32,::1/128
}
main "$@"
``` ```
- The image's entrypoint, `runsvinit`, has runit start `smallwebwaf` and the app - The image's entrypoint, `runsvinit`, has runit start `smallwebwaf` and the app
side by side, each as its own user, and start either again a second after it side by side, each as its own user, and start either again a second after it
exits. Leave out `ENTRYPOINT` and `USER` from the app's Dockerfile. exits. Leave out `ENTRYPOINT` and `USER` from the app's Dockerfile.
- `nix-env -iA nixpkgs.<name>` installs a package from the nixpkgs in the image, - `nix-env -iA nixpkgs.<name>` installs a package from the nixpkgs in the image,
and the app finds it on its `PATH`. That nixpkgs is fixed at one commit, so and the app finds it on its `PATH`, after Ubuntu's own commands. That nixpkgs
the same `smallwebwaf` image always gives the app the same packages; newer is fixed at one commit, so the same `smallwebwaf` image always gives the app
ones come with a newer `smallwebwaf` image. the same packages; newer ones come with a newer `smallwebwaf` image.
- Deploy it as you deploy any app, with traefik's labels on this one container - Deploy it as you deploy any app, with traefik's labels on this one container
pointing at port 8080. upaas needs no change for this. pointing at port 8080. upaas needs no change for this.
- The app has to trust `127.0.0.1` and `::1` for forwarded headers, besides the - The app has to trust `127.0.0.1` and `::1` for forwarded headers, besides the
@@ -197,10 +389,18 @@ exec chpst -u app:app /usr/local/bin/app \
- Port 8080 is the only one the app must leave free: the health check, the - Port 8080 is the only one the app must leave free: the health check, the
metrics and ban management are all on it, under `/_smallwebwaf/`. The image's metrics and ban management are all on it, under `/_smallwebwaf/`. The image's
health check passes while `smallwebwaf` answers and the app accepts health check passes while `smallwebwaf` answers and the app accepts
connections. connections. `SWWAF_LISTEN_ADDR` can move `smallwebwaf` to another port, which
the app then leaves free instead; the health check follows it, and traefik's
labels must point at it. The address part of `SWWAF_LISTEN_ADDR` stays empty
(for example `:9000`, never `127.0.0.1:9000`), so `smallwebwaf` keeps
listening on every address: traefik reaches it on the container's address, and
the health check on `127.0.0.1`.
- `smallwebwaf` keeps its state files in `/var/lib/smallwebwaf`. Mount a volume - `smallwebwaf` keeps its state files in `/var/lib/smallwebwaf`. Mount a volume
there to keep bans and client history when a deploy replaces the container; there to keep bans and client history when a deploy replaces the container;
without one, it still starts. without one, it still starts. The state files come in milestone 3 or later;
until then it writes nothing to disk and needs no volume.
- `docker stop` has runit stop both processes. `smallwebwaf` then stops taking
requests and gives those in progress five seconds to finish.
A rule file is one rule per line: a name, what to match against, what to do, and A rule file is one rule per line: a name, what to match against, what to do, and
a regex. a regex.
@@ -216,41 +416,139 @@ the metrics, failure behaviour and the build order.
## Country and AS number lookup ## Country and AS number lookup
`smallwebwaf` looks up the AS number and country of every client, for the So far `smallwebwaf` looks up only the country, only through GeoJS, and only
request log, the metrics and the ban notes, and for the country lists and biased while `SWWAF_DENIED_COUNTRIES` or `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` is set:
limits when you set them. It works with no setup: by default it asks the free then the address of every new visitor outside `SWWAF_ALLOW_NETS` and
GeoJS web service, which needs no account and no file. This means that, by `SWWAF_DENY_NETS` is sent to GeoJS, and with neither set, none is. An IPv6
default, the address of every new visitor is sent to GeoJS. Each answer is kept visitor is asked about by the first address of its /64. A new visitor waits at
in memory for seven days, and many addresses are asked about in one request; most a second for its answer, and without one counts as coming from an unknown
writing the answers to disk, so that they survive a restart, comes in milestone country until the answer arrives. The addresses waiting are asked about
3 or later. GeoJS publishes no rate limit but may block a caller it thinks asks together, up to 200 in one request, one request at a time; at most 10,000
too much; while it is not answering, new visitors count as coming from an visitors wait, and one more counts as coming from an unknown country until there
unknown country, which `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses. is room. While GeoJS fails, visitors with a kept answer are unaffected and new
ones count as coming from an unknown country. GeoJS is then left alone for a
second, twice as long after each further failure up to five minutes, and asked
again by the next request that needs it.
In the full design, `smallwebwaf` looks up the AS number and country of every
client, for the request log, the metrics and the ban notes, and for the country
lists and biased limits when you set them. It works with no setup: by default it
asks the free GeoJS web service, which needs no account and no file. This means
that, by default, the address of every new visitor is sent to GeoJS. Each answer
is kept in memory for seven days, and many addresses are asked about in one
request; writing the answers to disk, so that they survive a restart, comes in
milestone 3 or later (see the build order in [`SPEC.md`](SPEC.md)). GeoJS
publishes no rate limit but may block a caller it thinks asks too much; while it
is not answering, new visitors count as coming from an unknown country, which
`SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses.
To keep your visitors' addresses on your own host, set To keep your visitors' addresses on your own host, set
`SWWAF_LOOKUP_SOURCE=off`, or use the database file instead of GeoJS: `SWWAF_LOOKUP_SOURCE=off`, or use the database file instead of GeoJS:
`SWWAF_LOOKUP_SOURCE=file` reads the free IPinfo Lite database `SWWAF_LOOKUP_SOURCE=file` reads the free IPinfo Lite database
(`ipinfo_lite.mmdb`). You download it with your own IPinfo account, mount the (`ipinfo_lite.mmdb`). `SWWAF_LOOKUP_SOURCE` comes in milestone 3 or later (see
directory that holds it into the container, point `SWWAF_LOOKUP_DB_PATH` at the the build order in [`SPEC.md`](SPEC.md)); until then GeoJS is asked only while a
file and refresh it when you choose; `smallwebwaf` never downloads it itself, country list is set. You download the database with your own IPinfo account,
and reads it again when you replace it. It has to be the directory rather than mount the directory that holds it into the container, point
the file itself: docker does not show a single mounted file being replaced, so a `SWWAF_LOOKUP_DB_PATH` at the file and refresh it when you choose; `smallwebwaf`
refresh would go unseen. IPinfo releases it under the Creative Commons never downloads it itself, and reads it again when you replace it. It has to be
Attribution-ShareAlike 4.0 International License and asks for attribution, in the directory rather than the file itself: docker does not show a single mounted
its own words on https://ipinfo.io/lite: "The attribution requirements can be file being replaced, so a refresh would go unseen. IPinfo releases it under the
met by giving our service credit as your data source. Simply place a link to Creative Commons Attribution-ShareAlike 4.0 International License and asks for
IPinfo on the website, application, or social media account that uses our data." attribution, in its own words on https://ipinfo.io/lite: "The attribution
Its example of such a credit is a link mentioning "IP address data is powered by requirements can be met by giving our service credit as your data source. Simply
IPinfo". A service that uses the database through `smallwebwaf` should carry place a link to IPinfo on the website, application, or social media account that
that link. uses our data." Its example of such a credit is a link mentioning "IP address
data is powered by IPinfo". A service that uses the database through
`smallwebwaf` should carry that link.
Neither source can place a private address, so a client on one, such as a Neither source can place a private address, so a client on one, such as a
visitor on your local network, another container or your monitoring, has no visitor on your local network, another container or your monitoring, has no
country: `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless you list it in country: `SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses it unless you list it in
`SWWAF_ALLOW_NETS`. Such addresses are never sent to GeoJS. `SWWAF_ALLOW_NETS`, and `SWWAF_DENIED_COUNTRIES` does not refuse it. Such
addresses are never sent to GeoJS.
## How the code is laid out
- `cmd/smallwebwaf`: the binary, which only calls `internal/smallwebwaf`.
- `internal/smallwebwaf`: the process: it reads the settings, listens, serves
requests until `SIGTERM` or `SIGINT`, and stops. Run as
`smallwebwaf healthcheck`, it is the image's health check instead.
- `internal/config`: reads the settings, the one place they are read.
- `internal/proxy`: what happens to each request: it works out the client, runs
the checks, passes the request to the app and the answer back with the
standard library's `httputil.ReverseProxy` within the timeouts and size
limits, and writes the request's log line. Its `check` method is where a
request is refused before anything reaches the app: for `SWWAF_DENY_NETS`, for
the country lists, for a rate limit, and for an announced body over the size
limit.
- `internal/lookup`: looks up each client's country through GeoJS, and keeps the
answers.
- `internal/ratelimit`: counts each client's requests and tells when one takes
it over a rate limit.
- `internal/requestlog`: the lines on stdout: the request log line and the
process's own messages.
- `Dockerfile`: the lint and test phases, then the image, whose last stage
installs Ubuntu's packages, nixpkgs, `runsvinit` and `smallwebwaf`, with
`share/smallwebwaf.run` as runit's `run` script for `smallwebwaf`.
- `deploy/example-app`: an app built on the image, which `script/example-app`
checks.
Besides the Go standard library, `github.com/hashicorp/golang-lru/v2` keeps the
table of clients to 20,000 and the GeoJS answers to 100,000, dropping the least
recently seen. The country codes are the list in `internal/config/config.go`.
## Entrypoints
This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: the scripts in `script/` are the entrypoints for working on it, and
the `Makefile` targets are thin shims that call them. The scripts are POSIX sh,
so that they run in minimal containers.
- `script/bootstrap`: installs what the other scripts need on the host: `make`,
`git`, `curl`, Go for `gofmt`, node and yarn, and prettier.
- `script/setup`: readies a fresh clone: runs `script/bootstrap`, then
`script/install-precommit`.
- `script/projectname`: prints the project's name, `smallwebwaf`, which
`script/docker` and the others tag their images with.
- `script/test`: runs the tests, as the `test` phase of the `Dockerfile`.
- `script/lint`: runs golangci-lint, as the `lint` phase of the `Dockerfile`.
- `script/fmt`: formats the Go code with `gofmt` and the Markdown with prettier.
- `script/fmt-check`: checks the formatting, and changes nothing.
- `script/check`: runs `script/test`, `script/lint` and `script/fmt-check`.
- `script/docker`: builds the image, whose build runs the tests and the linter
first.
- `script/cibuild`: what CI runs: `script/bootstrap`, `script/check`, then the
image build.
- `script/precommit`: run by the git pre-commit hook; runs `script/check`.
- `script/install-precommit`: installs that hook; `make hooks` runs it.
- `script/build`: builds `bin/smallwebwaf` on the host, with Go installed, for
working on the code by hand; `make build` runs it.
- `script/run`: builds `bin/smallwebwaf` with `script/build` and runs it;
`make run` runs it.
- `script/example-app`: builds the image and, on it, the example app in
`deploy/example-app`, runs it, and checks that the health check passes, that a
request reaches the app through `smallwebwaf`, and that `sv stop` and
`docker stop` stop it in order; then removes the container and both images. It
needs network access, for nixpkgs' binary cache, and `script/check` does not
run it; `make example-app` does.
## TODO
- The rest of milestone 3, after the static lists, and the rest of the design,
in the order of the build order in [`SPEC.md`](SPEC.md).
## Documents ## Documents
- [`SPEC.md`](SPEC.md): the design. - [`SPEC.md`](SPEC.md): the design.
- [`EVALUATION.md`](EVALUATION.md): what already exists, what each tool covers - [`EVALUATION.md`](EVALUATION.md): what already exists, what each tool covers
and misses, and why none was adopted. and misses, and why none was adopted.
- [`REPO_POLICIES.md`](REPO_POLICIES.md): the policies this repository follows.
## License
MIT. See [`LICENSE`](LICENSE).
## Author
[@sneak](https://sneak.berlin)
+679
View File
@@ -0,0 +1,679 @@
---
title: Repository Policies
last_modified: 2026-10-04
---
This document covers repository structure, tooling, and workflow standards. Code
style conventions are in separate documents:
- [Code Styleguide](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE.md)
(general, bash, Docker)
- [Go](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_GO.md)
- [JavaScript](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_JS.md)
- [Python](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_PYTHON.md)
- [Go HTTP Server Conventions](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/GO_HTTP_SERVER_CONVENTIONS.md)
---
- Cross-project documentation (such as this file) must include
`last_modified: YYYY-MM-DD` in the YAML front matter so it can be kept in sync
with the authoritative source as policies evolve.
- **ALL external references must be pinned by cryptographic hash.** This
includes Docker base images, Go modules, npm packages, GitHub Actions, and
anything else fetched from a remote source. Version tags (`@v4`, `@latest`,
`:3.21`, etc.) are server-mutable and therefore remote code execution
vulnerabilities. The ONLY acceptable way to reference an external dependency
is by its content hash (Docker `@sha256:...`, Go module hash in `go.sum`, npm
integrity hash in lockfile, GitHub Actions `@<commit-sha>`). No exceptions.
This also means never `curl | bash` to install tools like pyenv, nvm, rustup,
etc. Instead, download a specific release archive from GitHub, verify its hash
(hardcoded in the Dockerfile or script), and only then install. Unverified
install scripts are arbitrary remote code execution. This is the single most
important rule in this document. Double-check every external reference in
every file before committing. There are zero exceptions to this rule.
- Every repo with software must have a root `Makefile` with these targets:
`make bootstrap`, `make setup`, `make test`, `make lint`, `make fmt` (writes),
`make fmt-check` (read-only), `make check` (runs `test`, `lint`, `fmt-check`),
`make docker`, and `make hooks` (installs pre-commit hook). A model Makefile
is at `https://git.eeqj.de/sneak/prompts/raw/branch/main/Makefile`.
- Repos follow the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
pattern: the implementation of each Makefile target lives in an executable
script in `script/` (`script/bootstrap`, `script/setup`, `script/test`,
`script/lint`, `script/fmt`, `script/fmt-check`, `script/check`,
`script/docker`), and the Makefile targets are thin shims that call them. The
scripts must be POSIX sh (`#!/bin/sh`, `set -eu`, no bashisms) so they run in
minimal containers (e.g. alpine images have no bash); locate the repo root
with `$(cd "$(dirname "$0")/.." && pwd -P)` and `cd` there before acting. From
the standard's canonical set we use `bootstrap`, `setup` (make the repo ready
for development after a fresh clone: runs `bootstrap`, then
`install-precommit`, plus any repo-specific initialization), `test`, and
`cibuild`. `script/bootstrap` installs all dependencies idempotently and
assumes nothing is present: base tools come from nix, apt, brew, or apk
(detected in that order; apt runs noninteractive). For node it uses the
installed node if present; otherwise it installs a PINNED node version via
nvm, first installing nvm itself if missing — from a hash-verified GitHub
release archive (never `curl | sh`), with bash installed as an explicit
prerequisite since nvm requires bash. yarn is then pinned via
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
always exact versions. `script/cibuild` runs the CI build: it changes to the
repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
with the version; the Gitea workflow calls it. **`script/cibuild` runs
`script/bootstrap` first**, because the workflow checks out the repo and runs
nothing else, while `script/fmt-check` runs the formatter on the host: on a
pristine checkout with nothing installed the run dies there, after the
containerised gates have passed. **The bootstrap alone is not enough**:
`script/bootstrap` installs node and yarn under nvm and leaves neither on the
`PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
source nvm for the pinned node version before invoking it, exactly as
`script/bootstrap`'s own install step does. A runner carrying nothing but
docker and git then gets through `script/check`. Four further scripts are our
own extensions to the standard: `script/check` runs `script/test`,
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
installs the git pre-commit hook (the `make hooks` target shims to it); and
`script/projectname` (literally that filename) simply outputs the project's
name. Scripts that need the name call `script/projectname` — e.g.
`script/docker` assembles its image tag from it — so those scripts stay
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the
README requirements below).
- Always use Makefile targets (`make fmt`, `make test`, `make lint`, etc.)
instead of invoking the underlying tools directly. The Makefile is the single
source of truth for how these operations are run.
- The Makefile is authoritative documentation for how the repo is used. Beyond
the required targets above, it should have targets for every common operation:
running a local development server (`make run`, `make dev`), re-initializing
or migrating the database (`make db-reset`, `make migrate`), building
artifacts (`make build`), generating code, seeding data, or anything else a
developer would do regularly. If someone checks out the repo and types
`make<tab>`, they should see every meaningful operation available. A new
contributor should be able to understand the entire development workflow by
reading the Makefile.
- Every repo should have a `Dockerfile`, and it carries the repo's gates: a
`lint` phase and a `test` phase, with the final stage depending on both so the
image cannot be built unless they pass. For non-server repos the final stage
brings up a development environment; for server repos it is the runtime image.
The gate phases and the build stage start from their pinned base images and
install what those images lack either inline, as the canonical Go `Dockerfile`
below does for `git`, or by running `script/bootstrap`, as the `prompts`
repo's own `Dockerfile` does for its yarn packages. The development
environment stage installs development prerequisites by running
`script/bootstrap` rather than duplicating its installs inline. A stage that
runs `script/bootstrap` COPYs `script/` and the dependency manifests
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
no separate lint file. `script/lint` and `script/test` each build one phase
and nothing else:
```sh
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
docker build --no-cache --target test -t "$(script/projectname)-test" .
```
**A stage that is not the last one in the file is built only when the final
stage's chain depends on it, or when `--target` names it.** That is why the
two gates are always invoked by name here, and why the final stage carries a
`COPY --from=` of a harmless file from each of them: without that edge a
plain `docker build .` builds the last stage alone and exits 0 having linted
and tested nothing.
**Every `docker build` in `script/` is tagged**, here and in
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
image behind on every invocation, on every developer host and every CI
runner; a tagged one replaces the previous image.
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
themselves a `docker build` and would recurse into a daemon that does not
exist in a build step. Formatting is the exception and stays on the host:
`script/fmt` writes the working tree, and `script/fmt-check` is its
read-only twin.
**No lint verdict may come from a host invocation of the linter.** On a
shared host golangci-lint reads a result cache keyed on file content rather
than location, so a second checkout of the same content is served the first
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own cache, its own `TMPDIR` and
a digest-pinned binary, so neither is reachable.
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
a `COPY` layer only when the copied content changes, so on an unchanged tree
the check `RUN` is served from cache, nothing executes, and the build still
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
four, and there is no fifth — `script/check` runs the two gate phases and
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
not evidence that anything ran: a sub-second build reporting success is a
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
and friends destroy a build cache shared with every other build on the host.
When a check is added or changed, prove it works by planting a defect it must
catch and watching the run fail on it, then revert the defect. A green run
alone shows neither that the check ran nor that it covers what it should.
- **The gate phases are separate stages, and the build stage depends on both.**
The lint phase is based on the `golangci/golangci-lint` image (pinned by
hash), so lint failures surface in seconds rather than after a full compile,
and the test phase is based on the Debian Go image. The canonical Go repo
`Dockerfile`:
```dockerfile
# Lint phase
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN golangci-lint run --config .golangci.yml ./...
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
# image ships and the alpine one does not.
# golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed.
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context. With
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# Runtime stage, and the last one
FROM alpine@sha256:...
COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"]
```
Key points:
- The lint phase uses the `golangci/golangci-lint` image directly (it has
both Go and the linter), so nothing needs installing.
- `COPY --from=<phase> /src/go.sum /dev/null` is a no-op copy whose only
purpose is the ordering edge. BuildKit runs stages in parallel by default,
and a stage nothing depends on is not built at all, so without these two
lines a red gate would not fail the build.
- Keep the runtime stage last, and if you add a stage after it, give it the
same two copies. A plain `docker build .` builds the last stage's chain
and nothing else.
- If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint phase must
create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
- If the project requires CGO or system libraries for linting, install them
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
has no `apk`, so install with `apt-get` under the Debian package name
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
lists in the same `RUN`, so the layer does not keep them:
```dockerfile
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/*
```
- `.dockerignore` lets `.git` into the build context. It keeps out every git
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
repository's own, each submodule's under `.git/modules/`, and that of a
submodule keeping its own `.git` directory. `git describe` does not need
them, and each can hold a credential: a password in a remote URL, or the
token the CI checkout step stores there. A submodule whose name has a
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
git directory to `**/.git/modules/**/config`, and Go's version stamping
then fails the build: give it a name without that segment
(`git submodule add --name`). The stage that compiles has `git` (the
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
takes the version from the `VERSION` build argument when one is given,
otherwise from `git describe --tags --always`. That gives the tag on a
tagged commit; on a later commit, the tag, the number of commits since it
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
tag is reachable. The stage that compiles also marks its working directory
safe for git (`git config --system --add safe.directory /src`): a context
sent as a tar stream keeps the sender's file owners, and git refuses a
checkout owned by another user, so the version would come out empty.
`ARG VERSION` has no default, and the build fails if the context carries
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
`docker build .` with no build arguments must succeed; a Dockerfile that
refuses an empty build argument drops that refusal and keeps the argument.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` on push, and checks out the repo as its only other step.
That script bootstraps, runs the gate phases, and then builds the image, so a
successful run means every check passed; a bare `docker build .` does not
carry the same guarantee, because its gate phases may come from the cache. The
image build is uncached and so runs the gate phases a second time. That is the
price of the rule above, and it is worth paying: the image that ships is built
from a run of its own gates rather than from a cache entry. A separate
workflow limited to `main` by a `branches` list under `on: push` cannot be
checked by review: to try a change to it, add the feature branch to that list
and push, then remove the branch from the list again before merging. Keep any
job in it that publishes behind `if: github.ref_name == 'main'`, so the run
from the feature branch publishes nothing.
- Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
two exceptions: four-space indents (except Go), and `proseWrap: always` for
Markdown (hard-wrap at 80 columns). Documentation and writing repos (Markdown,
HTML, CSS) should also have `.prettierrc` and `.prettierignore`.
- Pre-commit hook: runs `script/precommit`, which calls `script/check`. If local
testing is not possible in the repo, `script/precommit` may skip `script/test`
and run only `script/lint` and `script/fmt-check`. The hook is installed by
`script/install-precommit`; the Makefile must provide a `make hooks` target
that shims to it.
- All repos with software must have tests that run via the platform-standard
test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). If no meaningful
tests exist yet, add the most minimal test possible — e.g. importing the
module under test to verify it compiles/parses. There is no excuse for
`make test` to be a no-op.
- `make test` must complete in under 60 seconds. That is the hard cap, and a
suite that exceeds it fails. Under 20 seconds is the target. A suite between
20 and 60 seconds is still green, but the overage must be filed as an
improvement bug against that repo. Add a 90-second timeout to the test
invocation (`go test -timeout 90s`). The backstop deliberately sits above the
hard cap so that it catches a genuinely hung test rather than a merely slow
one.
- **The test command should use the conditional verbose rerun pattern.** Run
tests without `-v` (verbose) first. If tests fail, automatically rerun with
`-v` to show full output. This keeps CI logs and `docker build` output clean
on success (just package/suite summaries) while providing full diagnostic
detail on failure (every test case, every assertion). The command lives in the
`test` phase of the `Dockerfile`, since `script/test` builds that phase; the
Makefile form below is the same pattern for any repo-local invocation:
```makefile
test:
@<test-command> || \
{ echo "--- Rerunning with -v for details ---"; \
<test-command-with-v>; exit 1; }
```
Go example:
```makefile
test:
@go test -count=1 -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
```
`-count=1` is required on both invocations: it defeats Go's test _result_
cache, so neither run can report a stored pass in place of running the
tests. It leaves the build cache alone, so it costs the runtime of the suite
and no recompilation.
That cache is Go's own, separate from Docker's layer cache. Go stores a
passing result in its cache directory (`GOCACHE`), and when the same tests
run again on unchanged code it prints that result, marked `(cached)`,
without running them. That matters on a developer's machine, where this
target runs and the directory lasts from one run to the next. The `test`
phase of the `Dockerfile` needs no `-count=1`: its base image holds no
result for this repo's tests and nothing before its `go test` step runs a
test, so there is nothing to replay. `--no-cache` (above) is what makes that
step run on an unchanged tree.
Python example:
```makefile
test:
@python -m pytest || \
{ echo "--- Rerunning with -v for details ---"; \
python -m pytest -v; exit 1; }
```
The `exit 1` ensures the target always fails after a rerun — the first run
already proved the tests are broken, so the build must not pass even if a
flaky test happens to succeed on the second attempt. The rerun exists solely
for diagnostic output.
- Docker builds must complete in under 5 minutes.
- `make check` must not modify any files in the repo. Tests may use temporary
directories.
- `main` must always pass `make check`, no exceptions.
- Never commit secrets. `.env` files, credentials, API keys, and private keys
must be in `.gitignore`. No exceptions.
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
setting up a new repo. These patterns are written to `.gitignore`'s own
semantics, in which an unanchored pattern already matches at every depth; they
are not a `.dockerignore` and must not be transplanted into one unmodified.
- **`.dockerignore` does not use `.gitignore` semantics, and copying patterns
across unmodified leaves secrets in the build context.** Docker matches with
`moby/patternmatcher`: `filepath.Match` semantics plus a `**` extension, so
`*` does not cross `/` and a pattern without a leading `**/` is anchored at
the build-context root. A `.dockerignore` listing `.env`, `*.pem` and `*.key`
therefore excludes only the copies at the repository root, while `config/.env`
and `certs/server.key` still reach the context and can land in an image layer
— which is more dangerous than a short file with no secret patterns at all,
because it reads as solved and stops anyone looking. Give every
depth-independent pattern the `**/` prefix and leave only genuinely
root-anchored entries unprefixed: `.claude`, and the repo's own host-built
binary, written `/myapp` and never `**/myapp`, which would also match
`cmd/myapp/` and delete the package directory from the context. Matching is
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
secret names use character ranges — `**/*.[kK][eE][yY]`, `**/*.[pP][eE][mM]`,
and likewise for `.envrc` and the extensionless SSH keys. Where such a pattern
also catches something the build needs, re-include it with a negation
(`!docs/example.env`); deleting the pattern reopens the exposure for every
other file it covers. Fetch the standard `.dockerignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.dockerignore` and extend
it with the repo's own artifacts.
- **In-repo agent scratch belongs in both files, written to each file's own
semantics.** `.claude/` holds one worktree per in-flight agent — an entire
additional checkout of the repo — so under `COPY . .` the build context
inflates by a multiple of the repo and another session's unreviewed work can
be copied into an image layer. In `.gitignore` the entry is `.claude/`,
unanchored. In `.dockerignore` it is `.claude`, anchored and with **no** `**/`
prefix, because the prefixed form would also delete any nested directory of
that name from the build. Anchoring carries a known gap that the canonical
`.dockerignore` states in its own comment, since consuming repos receive the
file and not the tracker: the directory is created in the agent's working
directory, so a repo running agents in subdirectories still ships
`services/api/.claude/` and must add its own anchored entry there.
- **A plain `docker build .` of a clone stamps the version that
`git describe --tags --always` gives**, derived from the `.git` in the build
context as the canonical `Dockerfile` above shows. Without its failure check,
a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
and the build would still exit 0. `script/docker` and `script/cibuild` pass
the version they compute on the host; it takes precedence. They do this
byte-identically across repos:
```sh
# Own line: a failing command substitution inside an argument does not
# trip `set -e`, so the inline form degrades to an empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$(script/projectname)" .
```
`--always` makes an untagged repo yield an abbreviated commit hash rather
than failing, and the `[ -n "$version" ]` line is the single place the
fallback is applied — a live check that fires on a build from an export with
no `.git` and on a repository with no commits yet. Do not fold it into the
substitution as `|| echo unknown`, which makes the guard unreachable. The
Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
the scripts stay byte-identical. One consequence for CI: the standard
checkout action clones shallow and fetches no tags, so a repo that embeds a
tag-derived version must set `fetch-depth: 0` on its checkout step.
- **Verify `.dockerignore` by enumerating the image, not by reading the
patterns.** Plant files at the root _and_ at least two directories deep, build
a probe image that does `COPY . .`, and list what actually landed
(`docker run --rm --entrypoint find IMAGE /app`). The `transferring context`
size is not a substitute: a nested secret is a few bytes, and BuildKit
transfers only the delta from the previous build.
- **No build artifacts in version control.** Code-derived data (compiled
bundles, minified output, generated assets) must never be committed to the
repository if it can be avoided. The build process (e.g. Dockerfile, Makefile)
should generate these at build time. Notable exception: Go protobuf generated
files (`.pb.go`) ARE committed because repos need to work with `go get`, which
downloads code but does not execute code generation.
- Never use `git add -A` or `git add .`. Always stage files explicitly by name.
- Never force-push to `main`.
- Make all changes on a feature branch. You can do whatever you want on a
feature branch.
- `.golangci.yml` is standardized. The vendored copy in a consuming repo must
_NEVER_ be modified by an agent: fetch it from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it
byte-identical, so that no repo can quietly loosen its own linting. Linter
configuration changes are made to the canonical copy in the `prompts` repo and
reach consuming repos by re-vendoring; an agent may open a PR against
canonical, which only the user merges. One list is exempt from byte-identity,
because it cannot be written once for every repo: the `deny` list of the
`test-support` depguard rule, where a repo names its own test-support packages
by full import path. A repo adds entries there and changes nothing else, and a
re-vendor carries its entries forward. The canonical golangci-lint version is
v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
image
(`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
directive must not name a newer Go minor version than the one golangci-lint
was built with, or golangci-lint refuses to lint it: this release lints
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
installs golangci-lint on the host. A repo sets the lint phase digest to the
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
the two prompted the change: the canonical copy can name linters that an older
golangci-lint rejects, and a newer golangci-lint can add linters that
`default: all` switches on until the canonical copy disables them.
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
`PATH` only, so on an already-provisioned machine the pin is inert and a
version bump is a silent no-op — while the Dockerfile, installing into a clean
image, gets the pinned version, so a local `make check` and `make docker` can
disagree about what the tool even is. The canonical form:
- compares the installed version against the pin over the **whole** version
token; a parser that stops at the first `-` reports `2.12.2` for a host
running `2.12.2-rc1` and skips the install;
- treats absent, non-zero, empty or unrecognised `--version` output as a
mismatch, so the failure direction is a redundant install and never a
skipped one;
- after installing, re-resolves the binary the way callers do — `hash -r`,
then through `PATH`, not through the directory the installer wrote to —
and fails naming the resolved path, since an install that a shadowing
binary hides succeeds while changing nothing any caller sees;
- is actually called, and prints the version on both success paths: a
function defined and never invoked has the same exit status and the same
empty output as one that worked.
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
A Go tool a repo needs on the host is installed with `go install` pinned to
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
a `go.mod` tool dependency or through a `tools.go` file, either of which
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
- When pinning images or packages by hash, add a comment above the reference
with the version and date (YYYY-MM-DD).
- Use `yarn`, not `npm`.
- Write all dates as YYYY-MM-DD (ISO 8601).
- Simple projects should be configured with environment variables.
- Dockerized web services listen on port 8080 by default, overridable with
`PORT`.
- **HTTP/web services must be hardened for production internet exposure before
tagging 1.0.** This means full compliance with security best practices
including, without limitation, all of the following:
- **Security headers** on every response:
- `Strict-Transport-Security` (HSTS) with `max-age` of at least one year
and `includeSubDomains`.
- `Content-Security-Policy` (CSP) with a restrictive default policy
(`default-src 'self'` as a baseline, tightened per-resource as
needed). Never use `unsafe-inline` or `unsafe-eval` unless
unavoidable, and document the reason.
- `X-Frame-Options: DENY` (or `SAMEORIGIN` if framing is required).
Prefer the `frame-ancestors` CSP directive as the primary control.
- `X-Content-Type-Options: nosniff`.
- `Referrer-Policy: strict-origin-when-cross-origin` (or stricter).
- `Permissions-Policy` restricting access to browser features the
application does not use (camera, microphone, geolocation, etc.).
- **Request and response limits:**
- Maximum request body size enforced on all endpoints (e.g. Go
`http.MaxBytesReader`). Choose a sane default per-route; never accept
unbounded input.
- Maximum response body size where applicable (e.g. paginated APIs).
- `ReadTimeout` and `ReadHeaderTimeout` on the `http.Server` to defend
against slowloris attacks.
- `WriteTimeout` on the `http.Server`.
- `IdleTimeout` on the `http.Server`.
- Per-handler execution time limits via `context.WithTimeout` or
chi/stdlib `middleware.Timeout`.
- **Authentication and session security:**
- Rate limiting on password-based authentication endpoints. API keys are
high-entropy and not susceptible to brute force, so they are exempt.
- CSRF tokens on all state-mutating HTML forms. API endpoints
authenticated via `Authorization` header (Bearer token, API key) are
exempt because the browser does not attach these automatically.
- Passwords stored using bcrypt, scrypt, or argon2 — never plain-text,
MD5, or SHA.
- Session cookies set with `HttpOnly`, `Secure`, and `SameSite=Lax` (or
`Strict`) attributes.
- **Reverse proxy awareness:**
- True client IP detection when behind a reverse proxy
(`X-Forwarded-For`, `X-Real-IP`). The application must accept
forwarded headers only from a configured set of trusted proxy
addresses — never trust `X-Forwarded-For` unconditionally.
- **CORS:**
- Authenticated endpoints must restrict `Access-Control-Allow-Origin` to
an explicit allowlist of known origins. Wildcard (`*`) is acceptable
only for public, unauthenticated read-only APIs.
- **Error handling:**
- Internal errors must never leak stack traces, SQL queries, file paths,
or other implementation details to the client. Return generic error
messages in production; detailed errors only when `DEBUG` is enabled.
- **TLS:**
- Services never terminate TLS directly. They are always deployed behind
a TLS-terminating reverse proxy. The service itself listens on plain
HTTP. However, HSTS headers and `Secure` cookie flags must still be
set by the application so that the browser enforces HTTPS end-to-end.
This list is non-exhaustive. Apply defense-in-depth: if a standard security
hardening measure exists for HTTP services and is not listed here, it is
still expected. When in doubt, harden.
- `README.md` is the primary documentation. Required sections:
- **Description**: First line must include the project name, purpose,
category (web server, SPA, CLI tool, etc.), license, and author. Example:
"µPaaS is an MIT-licensed Go web application by @sneak that receives
git-frontend webhooks and deploys applications via Docker in realtime."
- **Getting Started**: Copy-pasteable install/usage code block.
- **Entrypoints**: Opens by stating that the repo adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard (with that link), then documents each provided `script/`
entrypoint and its purpose.
- **Rationale**: Why does this exist?
- **Design**: How is the program structured?
- **TODO**: Update meticulously, even between commits. When planning, put
the todo list in the README so a new agent can pick up where the last one
left off.
- **License**: MIT, GPL, or WTFPL. Ask the user for new projects. Include a
`LICENSE` file in the repo root and a License section in the README.
- **Author**: [@sneak](https://sneak.berlin).
- First commit of a new repo should contain only `README.md`.
- Go module root: `sneak.berlin/go/<name>`. Always run `go mod tidy` before
committing.
- Use SemVer.
- Database migrations live in `internal/db/migrations/` and must be embedded in
the binary.
- `000_migration.sql` — contains ONLY the creation of the migrations
tracking table itself. Nothing else.
- `001_schema.sql` — the full application schema.
- **Pre-1.0.0:** never add additional migration files (002, 003, etc.).
There is no installed base to migrate. Edit `001_schema.sql` directly.
- **Post-1.0.0:** add new numbered migration files for each schema change.
Never edit existing migrations after release.
- All repos should have an `.editorconfig` enforcing the project's indentation
settings.
- Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
`Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
and language-specific config). Everything else goes in a subdirectory.
Canonical subdirectory names:
- `bin/` — executable scripts and tools
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
body is a single call into `internal/` or `pkg/`, no project logic in
`cmd/`
- `configs/` — configuration templates and examples
- `deploy/` — deployment manifests (k8s, compose, terraform)
- `docs/` — documentation and markdown (README.md stays in root)
- `internal/` — Go internal packages
- `internal/db/migrations/` — database migrations
- `pkg/` — Go library packages
- `share/` — systemd units, data files
- `static/` — static assets (images, fonts, etc.)
- `web/` — web frontend source
- When setting up a new repo, files from the `prompts` repo may be used as
templates. Fetch them from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/<path>`.
- New repos must contain at minimum:
- `README.md`, `.git`, `.gitignore`, `.editorconfig`
- `LICENSE`, `REPO_POLICIES.md` (copy from the `prompts` repo)
- `Makefile`
- `script/` entrypoints (`bootstrap`, `setup`, `projectname`, `test`,
`lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
`install-precommit`)
- `Dockerfile`, `.dockerignore`
- `.gitea/workflows/check.yml`
- Go: `go.mod`, `go.sum`, `.golangci.yml`
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
- Python: `pyproject.toml`
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
is never committed under a file or directory named after one agent tool, such
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
+159 -61
View File
@@ -1,8 +1,8 @@
# smallwebwaf SPEC (draft): protective reverse proxy for one app # smallwebwaf SPEC (draft): protective reverse proxy for one app
Status: fourth draft, with the owner's rulings to date applied. Nothing has been Status: fourth draft, with the owner's rulings to date applied. Milestone 1 of
built yet. `EVALUATION.md` beside this file explains why no existing tool was the build order is built. `EVALUATION.md` beside this file explains why no
chosen. existing tool was chosen.
## Purpose ## Purpose
@@ -409,7 +409,8 @@ The settings, by group:
Bodies stream straight through, so a request body reaches the app while the Bodies stream straight through, so a request body reaches the app while the
client is still sending it. client is still sending it.
- `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take - `SWWAF_CLIENT_REQUEST_TIMEOUT` (default `60s`): how long a client may take
to send its whole request, headers and body. to send its request line and headers, and then, from the end of the
headers, its body.
- `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest - `SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` (default `32K`): the largest
request line and headers a client may send. Over it, `smallwebwaf` answers request line and headers a client may send. Over it, `smallwebwaf` answers
`431` and closes the connection, and nothing reaches the app. `431` and closes the connection, and nothing reaches the app.
@@ -436,6 +437,19 @@ The settings, by group:
an app that is too slow. A request that announces a body larger than its an app that is too slow. A request that announces a body larger than its
limit is refused before anything reaches the app. Once the response has limit is refused before anything reaches the app. Once the response has
started it can only be cut off, and the connection is closed. started it can only be cut off, and the connection is closed.
- Since a request body streams through, each side can hold up the other: a
slow client slows the send to the app, and an app slow to take the body
slows the client's send. So while a request body is still on its way, a
request timeout that runs out, `SWWAF_CLIENT_REQUEST_TIMEOUT` or
`SWWAF_UPSTREAM_REQUEST_TIMEOUT`, answers `408` if `smallwebwaf` was
waiting for the client to send more at that moment, and `504` if it was
waiting for the app to take what it had.
- Go's HTTP server, on which `smallwebwaf` is built, reads a request's line
and headers before `smallwebwaf` sees the request. A client that takes
longer than `SWWAF_CLIENT_REQUEST_TIMEOUT` to send them gets no answer:
the server closes its connection. Headers over
`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES` are answered `431` by the server
itself. Neither request gets a line in the request log.
- A WebSocket connection leaves these limits behind once it is upgraded: it - A WebSocket connection leaves these limits behind once it is upgraded: it
stays open until either side closes it. stays open until either side closes it.
- Lookup of AS number and country (R7). On by default through GeoJS, which needs - Lookup of AS number and country (R7). On by default through GeoJS, which needs
@@ -1012,10 +1026,12 @@ and the running `smallwebwaf` takes the edit in.
## Request log ## Request log
One JSON object per line on stdout for every request, including refused ones. One JSON object per line on stdout for every request, including refused ones,
stdout is always on. When `SWWAF_LOG_REMOTE_URL` is set the same lines are also apart from those Go's HTTP server ends before `smallwebwaf` sees them (see
sent to the remote endpoint, so a deployment can stop depending on docker's log "Configuration surface", size and time limits). stdout is always on. When
handling while `docker logs` keeps working. `SWWAF_LOG_REMOTE_URL` is set the same lines are also sent to the remote
endpoint, so a deployment can stop depending on docker's log handling while
`docker logs` keeps working.
- Standard web log fields: `time` (RFC 3339 with milliseconds), `instance`, - Standard web log fields: `time` (RFC 3339 with milliseconds), `instance`,
`client_ip`, `method`, `scheme`, `host`, `path`, `query`, `protocol`, `client_ip`, `method`, `scheme`, `host`, `path`, `query`, `protocol`,
@@ -1146,31 +1162,46 @@ The image holds:
- Ubuntu 26.04 LTS, the newest long-term support release of Ubuntu, pinned by - Ubuntu 26.04 LTS, the newest long-term support release of Ubuntu, pinned by
digest. The image moves to the next LTS release when that ships. digest. The image moves to the next LTS release when that ships.
- Three packages from Ubuntu, `ca-certificates`, `nix-bin` and `runit`,
installed from a dated snapshot of Ubuntu's archive and checked by hash (see
"Packages from Ubuntu" below). The Ubuntu image has no CA certificates, and
they come with `nix-bin` only because a library it uses recommends them, so
`ca-certificates` is installed by name. Without it Nix cannot download
packages, and `smallwebwaf`, unable to reach GeoJS, would count every visitor
as coming from an unknown country.
- Nix, the package manager, from Ubuntu's own `nix-bin` package, and nixpkgs, - Nix, the package manager, from Ubuntu's own `nix-bin` package, and nixpkgs,
the collection of packages Nix installs from, fixed at one commit (see the collection of packages Nix installs from, fixed at one commit (see
"Packages from nixpkgs" below). Root uses Nix directly, and no Nix daemon runs "Packages from nixpkgs" below). Root uses Nix directly, and no Nix daemon runs
in the container. in the container. Nix run by root expects a group of build users, `nixbld`,
- runit, from Ubuntu's own `runit` package, and `runsvinit` as the entrypoint. which `nix-bin` does not create, so the image writes `build-users-group =` to
Neither Ubuntu nor nixpkgs packages `runsvinit`, so the image builds it from `/etc/nix/nix.conf`, and root's builds then run without build users.
its source (`github.com/peterbourgon/runsvinit`) at a version fixed by hash. - runit, from Ubuntu's own `runit` package, and `runsvinit` as the entrypoint,
`runsvinit` starts runit's `runsvdir`, which starts a `runsv` for each as the owner's code style guide asks for service containers. Neither Ubuntu
directory under `/etc/service`; each `runsv` runs the `run` script in its nor nixpkgs packages `runsvinit`, so the image builds it from its source
directory, and runs it again whenever it exits. Ubuntu's runit looks for (`github.com/peterbourgon/runsvinit`) at a fixed commit hash. Its repository
is archived and has not changed since 2015, and its last tag is `v2.0.0`. It
has no `go.mod`, and `go build` of its directory needs one, so the build
writes one; since `runsvinit` uses only Go's standard library, that file names
nothing else. `runsvinit` starts runit's `runsvdir`, which starts a `runsv`
for each directory under `/etc/service`; each `runsv` runs the `run` script in
its directory, and runs it again whenever it exits. Ubuntu's runit looks for
services in `/etc/service` too, so when `docker stop` has `runsvinit` stop services in `/etc/service` too, so when `docker stop` has `runsvinit` stop
each service with runit's `sv`, `sv` finds it. each service with runit's `sv`, `sv` finds it.
- The `smallwebwaf` binary, and a user of its own, `smallwebwaf` (uid and gid - The `smallwebwaf` binary, and a user of its own, `smallwebwaf` (uid and gid
65532). 65532).
- The service directory `/etc/service/smallwebwaf`, whose `run` script waits one - The service directory `/etc/service/smallwebwaf`, whose `run` script waits one
second (`sleep 1`), makes `SWWAF_STATE_DIR` belong to the `smallwebwaf` user, second (`sleep 1`), makes `SWWAF_STATE_DIR` and every file in it belong to the
and starts `smallwebwaf` as that user with runit's `chpst`. `smallwebwaf` user, and starts `smallwebwaf` as that user with runit's
`chpst`.
- The directory `/var/lib/smallwebwaf` for the state files, and - The directory `/var/lib/smallwebwaf` for the state files, and
`/etc/smallwebwaf/rules.d` with the default rule file (see "Rule files"). `/etc/smallwebwaf/rules.d` with the default rule file (see "Rule files").
- Port 8080 declared (`EXPOSE 8080`), and the health check described below - Port 8080 declared (`EXPOSE 8080`), and the health check described below
(`HEALTHCHECK`). (`HEALTHCHECK`).
Every `run` script, the app's included, is a bash script that starts with Every `run` script, the app's included, is a bash script that starts with
`#!/usr/bin/env bash` and `set -euo pipefail`, as the owner's code style guide `#!/usr/bin/env bash` and `set -euo pipefail` and puts its code in a `main`
asks; Ubuntu ships bash. function, called on its last line, as the owner's code style guide asks; Ubuntu
ships bash.
Beyond its `FROM` line, the app's Dockerfile adds: Beyond its `FROM` line, the app's Dockerfile adds:
@@ -1209,23 +1240,64 @@ with `app.run` beside the Dockerfile:
```bash ```bash
#!/usr/bin/env bash #!/usr/bin/env bash
set -euo pipefail set -euo pipefail
sleep 1
exec chpst -u app:app /usr/local/bin/app \ main() {
--listen 127.0.0.1:8081 \ sleep 1
--trusted-proxies 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1/32,::1/128 exec chpst -u app:app /usr/local/bin/app \
--listen 127.0.0.1:8081 \
--trusted-proxies 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,127.0.0.1/32,::1/128
}
main "$@"
``` ```
Packages from Ubuntu: the image installs `ca-certificates`, `nix-bin` and
`runit` from Ubuntu's snapshot service, which serves the archive as it was at a
given moment, rather than from the archive itself, whose packages change with
every update. The image's Dockerfile names that moment, in apt's
`--snapshot 20261001T000000Z` form, on both `apt-get update` and
`apt-get install`. That moment is never earlier than the date of the pinned
Ubuntu image, since packages from an older snapshot can need older versions of
packages the Ubuntu image already holds, and it moves forward whenever that
image's digest does. `apt-get update` keeps the snapshot's `InRelease` files,
which apt checks against the archive's signature, in `/var/lib/apt/lists/`; each
lists the SHA-256 hash of the package lists it covers, and each package list the
hash of every package in it. The Dockerfile also names the SHA-256 hash of each
of the snapshot's `InRelease` files, and the build checks them after
`apt-get update` and before `apt-get install`, so every package apt installs is
checked, through those files, against hashes the Dockerfile names.
`apt-get update` also fetches the live archive's `InRelease` files into the same
directory; their hashes change whenever the archive does, and the install does
not use them, so the check leaves them out. The hashes are those of the archive
for amd64, and so the image is built for amd64: other architectures use Ubuntu's
ports archive, whose `InRelease` files differ. The snapshot service is reached
over HTTPS, and the Ubuntu image has no CA certificates of its own, so this one
install uses those of the Go image that `smallwebwaf` is built in, which is
pinned by digest too: apt's `Acquire::https::CaInfo` option names that image's
CA certificate file, `/etc/ssl/certs/ca-certificates.crt`, mounted for that one
step.
Packages from nixpkgs: nixpkgs is fixed at one commit of its newest release Packages from nixpkgs: nixpkgs is fixed at one commit of its newest release
branch, `nixos-26.05` today. The image's Dockerfile names the commit and the branch, `nixos-26.05` today. For each commit of the branch that has passed its
hash of its contents, and the build checks that hash. nixpkgs is set up for root tests, the Nix project publishes a release on `releases.nixos.org`, such as
under the name `nixpkgs`, so the app's Dockerfile installs a package with `nixos-26.05.11045.774debe7a0d1`, and the image takes nixpkgs from that
`nix-env -iA nixpkgs.<name>`, and whatever it installs is on the `PATH` of every release's file `nixexprs.tar.xz`, not from a GitHub archive of the commit, whose
service. Because nixpkgs stays at that commit, an app built on the same bytes can change. The image's Dockerfile names the release and the SHA-256 hash
`smallwebwaf` image gets the same packages each time it is built. A newer commit of that file, which the release's page lists, and the build checks the hash
of the branch, with its security fixes, comes with a newer `smallwebwaf` image, before unpacking it. nixpkgs is set up for root under the name `nixpkgs`, so the
as do Ubuntu's own fixes; an app takes them by changing the digest in its `FROM` app's Dockerfile installs a package with `nix-env -iA nixpkgs.<name>`, and
line. When nixpkgs makes its next release, every six months, the image moves to whatever it installs is on the `PATH` of every service: the image adds root's
that release's branch. Nix profile, `/nix/var/nix/profiles/default/bin`, at the end of the `PATH`,
after Ubuntu's own directories, so that no package hides the image's own
commands. busybox, for one, brings its own `sv`, which looks for services
elsewhere. Because nixpkgs stays at that commit, an app built on the same
`smallwebwaf` image gets the same packages each time it is built. Unpacked,
nixpkgs takes about 500 MiB of disk, more on some filesystems such as ZFS, and
each package an app installs from it adds its own size, with everything it
depends on. A newer commit of the branch, with its security fixes, comes with a
newer `smallwebwaf` image, as do Ubuntu's own fixes; an app takes them by
changing the digest in its `FROM` line. When nixpkgs makes its next release,
every six months, the image moves to that release's branch.
The two processes: The two processes:
@@ -1251,19 +1323,28 @@ The two processes:
- The container's root filesystem stays writable: runit writes each service's - The container's root filesystem stays writable: runit writes each service's
status into its directory under `/etc/service`. status into its directory under `/etc/service`.
The health check: the image's `HEALTHCHECK` passes while `smallwebwaf` answers The health check: the image's `HEALTHCHECK` runs `smallwebwaf healthcheck`,
`GET /_smallwebwaf/healthz` on `127.0.0.1:8080` and the app accepts connections which passes while `smallwebwaf` answers `GET /_smallwebwaf/healthz` on
`127.0.0.1`, at the port in `SWWAF_LISTEN_ADDR`, and the app accepts connections
at the address in `SWWAF_UPSTREAM_URL`, and fails when either does not. The at the address in `SWWAF_UPSTREAM_URL`, and fails when either does not. The
container therefore shows as healthy only while both processes are up. An app container therefore shows as healthy only while both processes are up. traefik
with a health check of its own can replace the image's `HEALTHCHECK` with one sends a container no requests until it shows as healthy, so the check runs every
that checks both. second from the container's start until it first passes, for up to a minute, and
every 30 seconds after that. An app with a health check of its own can replace
the image's `HEALTHCHECK` with one that checks both.
Ports: `smallwebwaf` listens on port 8080 on every address and on no other port; Ports: `smallwebwaf` listens on port 8080 on every address and on no other port;
its health check, metrics and ban management are all on that listener, under its health check, metrics and ban management are all on that listener, under
`/_smallwebwaf/` (see "Admin endpoints"). The app must leave port 8080 free. It `/_smallwebwaf/` (see "Admin endpoints"). `SWWAF_LISTEN_ADDR` may set another
listens on `127.0.0.1:8081` only, so that nothing outside the container reaches port: the image's health check takes its port from that setting, and traefik's
it except through `smallwebwaf`: an app that listens on every address can be port label (`traefik.http.services.<name>.loadbalancer.server.port`) must name
reached around `smallwebwaf` by anything that reaches the container. the same port, and the app must leave that port free. The address part of
`SWWAF_LISTEN_ADDR` stays empty (for example `:9000`, never `127.0.0.1:9000`),
so `smallwebwaf` keeps listening on every address: traefik reaches it on the
container's address, and the health check on `127.0.0.1`. The app listens on
`127.0.0.1:8081` only, so that nothing outside the container reaches it except
through `smallwebwaf`: an app that listens on every address can be reached
around `smallwebwaf` by anything that reaches the container.
State: `smallwebwaf` keeps its state files in `/var/lib/smallwebwaf` State: `smallwebwaf` keeps its state files in `/var/lib/smallwebwaf`
(`SWWAF_STATE_DIR`), a directory of its own beside the app's data, which the app (`SWWAF_STATE_DIR`), a directory of its own beside the app's data, which the app
@@ -1271,13 +1352,21 @@ keeps in directories of its own, such as `/var/lib/app`. Without a volume there,
the files live in the container: they survive a restart of the container and are the files live in the container: they survive a restart of the container and are
lost when a deploy replaces it. A volume mounted at `/var/lib/smallwebwaf`, lost when a deploy replaces it. A volume mounted at `/var/lib/smallwebwaf`,
named or a host directory, keeps them across deploys; the `run` script of named or a host directory, keeps them across deploys; the `run` script of
`smallwebwaf` makes it belong to the `smallwebwaf` user, so a host directory `smallwebwaf` makes it and every file in it belong to the `smallwebwaf` user, so
mounted there needs no change of owner. It is a volume of its own, separate from a host directory mounted there needs no change of owner, and files an earlier
the app's, holds a few tens of MiB at most with the defaults (see "Persistent owner left in it can be read and replaced. It is a volume of its own, separate
state"), and needs no backup beyond whatever the host already does. The image from the app's, holds a few tens of MiB at most with the defaults (see
declares no volume, since every app image built on it would inherit it. "Persistent state"), and needs no backup beyond whatever the host already does.
Milestone 2 (https://git.eeqj.de/sneak/smallwebwaf/issues/14) writes no state The image declares no volume, since every app image built on it would inherit
files and needs no volume. it. Milestone 2 (https://git.eeqj.de/sneak/smallwebwaf/issues/14) writes no
state files and needs no volume.
Tokens: a token given as a file (`SWWAF_ADMIN_TOKEN_FILE`,
`SWWAF_METRICS_TOKEN_FILE`) is out of the app's reach only while the
`smallwebwaf` user alone can read the file. The operator makes the file on the
host, owned by uid 65532, the `smallwebwaf` user, with mode `0400`, and mounts
the directory that holds it into the container read-only; the container sees the
same owner and mode.
Forwarded headers: the app's TCP peer is `smallwebwaf` on `127.0.0.1`, and the Forwarded headers: the app's TCP peer is `smallwebwaf` on `127.0.0.1`, and the
`X-Forwarded-For` the app receives ends with traefik's address, which `X-Forwarded-For` the app receives ends with traefik's address, which
@@ -1296,7 +1385,8 @@ and runs the one container as it runs any app, with the app's traefik labels,
environment variables and volumes. The labels route to port 8080 environment variables and volumes. The labels route to port 8080
(`traefik.http.services.<name>.loadbalancer.server.port=8080`), any `SWWAF_` (`traefik.http.services.<name>.loadbalancer.server.port=8080`), any `SWWAF_`
settings go with the app's environment variables, and the volume for settings go with the app's environment variables, and the volume for
`/var/lib/smallwebwaf` goes beside the app's own. `/var/lib/smallwebwaf` goes beside the app's own, as does the directory that
holds any token file.
- The endpoints of `smallwebwaf` are reached through traefik like any other - The endpoints of `smallwebwaf` are reached through traefik like any other
request, for example `https://app.example.invalid/_smallwebwaf/metrics` for a request, for example `https://app.example.invalid/_smallwebwaf/metrics` for a
@@ -1320,12 +1410,13 @@ settings go with the app's environment variables, and the volume for
taking more than 60 seconds is cut off, whether it is a git push, an LFS taking more than 60 seconds is cut off, whether it is a git push, an LFS
object, a container image layer, a package file or a release attachment. object, a container image layer, a package file or a release attachment.
The client is answered `413` for a body that is too large, before anything The client is answered `413` for a body that is too large, before anything
reaches gitea when the request announces its size, or `408` for one that reaches gitea when the request announces its size, or `408` for one the
is too slow; the upload fails, and no one is banned for it. A gitea that client sends too slowly (`504` if gitea is too slow to take it); the
takes large uploads needs `SWWAF_REQUEST_MAX_BYTES`, upload fails, and no one is banned for it. A gitea that takes large
`SWWAF_CLIENT_REQUEST_TIMEOUT` and `SWWAF_UPSTREAM_REQUEST_TIMEOUT` raised uploads needs `SWWAF_REQUEST_MAX_BYTES`, `SWWAF_CLIENT_REQUEST_TIMEOUT`
to fit. The Core Rule Set does not read an upload's body, which streams and `SWWAF_UPSTREAM_REQUEST_TIMEOUT` raised to fit. The Core Rule Set does
through without being held in memory. not read an upload's body, which streams through without being held in
memory.
- At the defaults (see "Configuration surface", attack detection), the Core - At the defaults (see "Configuration surface", attack detection), the Core
Rule Set lets gitea's ordinary use through, apart from the refusals in the Rule Set lets gitea's ordinary use through, apart from the refusals in the
next note: browsing and views of files in a repository, with their next note: browsing and views of files in a repository, with their
@@ -1495,7 +1586,7 @@ settings go with the app's environment variables, and the volume for
stops the start. The app starts with the same environment variables as stops the start. The app starts with the same environment variables as
`smallwebwaf`, so it can read a token given as one; a token given as a file `smallwebwaf`, so it can read a token given as one; a token given as a file
that only the `smallwebwaf` user can read (`SWWAF_ADMIN_TOKEN_FILE`, that only the `smallwebwaf` user can read (`SWWAF_ADMIN_TOKEN_FILE`,
`SWWAF_METRICS_TOKEN_FILE`) is out of the app's reach. `SWWAF_METRICS_TOKEN_FILE`) is out of the app's reach (see "Deployment").
- GeoJS, the default lookup source: every new visitor's address goes to a third - GeoJS, the default lookup source: every new visitor's address goes to a third
party, and a swarm of fresh addresses, when lookups peak, is when GeoJS may party, and a swarm of fresh addresses, when lookups peak, is when GeoJS may
slow down or block `smallwebwaf`. Keeping answers for 7 days and asking about slow down or block `smallwebwaf`. Keeping answers for 7 days and asking about
@@ -1524,16 +1615,23 @@ settings go with the app's environment variables, and the volume for
only while a list is set. A client on a private, loopback or link-local only while a list is set. A client on a private, loopback or link-local
address has no country, and neither list checks it. address has no country, and neither list checks it.
- Like milestone 1, it writes nothing to disk: the GeoJS answers and the - Like milestone 1, it writes nothing to disk: the GeoJS answers and the
rate counters are kept in memory only, and a restart loses them. rate counters are kept in memory only, and a restart loses them. The
header size and the idle time stay fixed at their defaults.
- The container image described under "Deployment", with runit and the - The container image described under "Deployment", with runit and the
container's health check. The health check calls `/_smallwebwaf/healthz`, container's health check. The health check calls `/_smallwebwaf/healthz`,
so milestone 2 answers that path, although the other admin endpoints come so milestone 2 answers that path, although the other admin endpoints come
later. later. The image's `/var/lib/smallwebwaf`, which the `run` script gives to
- After milestone 2, the rest of the design, in this order: the `smallwebwaf` user, and `/etc/smallwebwaf/rules.d` come with the state
files and the rule files.
- Milestone 3 and later: the rest of the design, in this order:
- static lists, the bans that broken request limits lead to, the ban ledger - static lists, the bans that broken request limits lead to, the ban ledger
and the JSON state files with edits taken in while running, exemptions, and the JSON state files with edits taken in while running, exemptions,
`observe` mode, the rest of the request log's fields, the metrics `observe` mode, the rest of the request log's fields, the metrics
endpoint; endpoint, and the header size and the idle time as settings
(`SWWAF_CLIENT_REQUEST_HEADER_MAX_BYTES`, `SWWAF_CLIENT_IDLE_TIMEOUT`).
With the static lists comes `SWWAF_ALLOW_NETS`, and from then on
`SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES` refuses a client on a private,
loopback or link-local address unless `SWWAF_ALLOW_NETS` lists it;
- rule files, the other admin endpoints, alerting to all three destinations, - rule files, the other admin endpoints, alerting to all three destinations,
remote log sending; remote log sending;
- AS number and country lookup for every client, from the file or GeoJS, - AS number and country lookup for every client, from the file or GeoJS,
+18
View File
@@ -0,0 +1,18 @@
// Command smallwebwaf is a web application firewall for one app: it runs
// between traefik and the app, and passes requests through within its
// limits. Everything it does is in internal/smallwebwaf.
package main
import (
"os"
"sneak.berlin/go/smallwebwaf/internal/smallwebwaf"
)
// Version is the version of the binary, set when it is built with
// -ldflags "-X main.Version=...".
var Version = "dev" //nolint:gochecknoglobals // the linker sets it
func main() {
os.Exit(smallwebwaf.Main(Version))
}
+20
View File
@@ -0,0 +1,20 @@
# An app built on the smallwebwaf image, as under "Deployment" in
# SPEC.md, which script/example-app builds and checks. The app is
# busybox's web server, from the nixpkgs in the image, serving one page;
# a real app copies in its own binary instead.
#
# A real app names the smallwebwaf image by digest. This one takes the
# image script/example-app has just built, or else the one `make docker`
# builds.
ARG SMALLWEBWAF_IMAGE=smallwebwaf
FROM ${SMALLWEBWAF_IMAGE}
# Packages the app needs, from the nixpkgs in the image.
RUN nix-env -iA nixpkgs.busybox
# The app's page, and a user of its own to run it.
RUN mkdir /var/www && echo 'hello from the example app' > /var/www/index.html
RUN useradd --system --no-create-home --shell /usr/sbin/nologin app
# The app's runit service.
COPY --chmod=755 app.run /etc/service/app/run
+9
View File
@@ -0,0 +1,9 @@
#!/usr/bin/env bash
set -euo pipefail
main() {
sleep 1
exec chpst -u app:app busybox httpd -f -p 127.0.0.1:8081 -h /var/www
}
main "$@"
+5
View File
@@ -0,0 +1,5 @@
module sneak.berlin/go/smallwebwaf
go 1.26.0
require github.com/hashicorp/golang-lru/v2 v2.0.7
+2
View File
@@ -0,0 +1,2 @@
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
+488
View File
@@ -0,0 +1,488 @@
// Package config reads smallwebwaf's settings. Every setting is an
// environment variable whose name starts with SWWAF_, every setting has a
// default, and this package is the one place they are read.
package config
import (
"errors"
"fmt"
"log/slog"
"math"
"net"
"net/netip"
"net/url"
"slices"
"strconv"
"strings"
"time"
)
// Config is smallwebwaf's settings. A timeout, size or rate limit of zero
// is off.
type Config struct {
// ListenAddr is where smallwebwaf listens (SWWAF_LISTEN_ADDR).
ListenAddr string
// UpstreamURL is the app (SWWAF_UPSTREAM_URL).
UpstreamURL *url.URL
// TrustedProxies are the netblocks whose X-Forwarded-For is
// believed (SWWAF_TRUSTED_PROXIES).
TrustedProxies []netip.Prefix
// ClientRequestTimeout bounds reading the whole request from the
// client (SWWAF_CLIENT_REQUEST_TIMEOUT).
ClientRequestTimeout time.Duration
// ClientResponseTimeout bounds writing the whole response to the
// client (SWWAF_CLIENT_RESPONSE_TIMEOUT).
ClientResponseTimeout time.Duration
// UpstreamRequestTimeout bounds connecting to the app and writing
// the whole request to it (SWWAF_UPSTREAM_REQUEST_TIMEOUT).
UpstreamRequestTimeout time.Duration
// UpstreamResponseTimeout bounds reading the whole response from
// the app (SWWAF_UPSTREAM_RESPONSE_TIMEOUT).
UpstreamResponseTimeout time.Duration
// RequestMaxBytes is the largest request body
// (SWWAF_REQUEST_MAX_BYTES).
RequestMaxBytes int64
// ResponseMaxBytes is the largest response body
// (SWWAF_RESPONSE_MAX_BYTES).
ResponseMaxBytes int64
// AllowNets are the netblocks whose clients skip every check
// (SWWAF_ALLOW_NETS). RateLimitExemptNets are those whose clients the
// rate limits neither count nor refuse (SWWAF_RATE_LIMIT_EXEMPT_NETS).
// DenyNets are those whose clients are always refused
// (SWWAF_DENY_NETS).
AllowNets []netip.Prefix
RateLimitExemptNets []netip.Prefix
DenyNets []netip.Prefix
// RateLimitPerMinute, RateLimitPerHour and RateLimitPerDay are the
// most requests a client may make in a minute, an hour and a day
// (SWWAF_RATE_LIMIT_PER_MINUTE, SWWAF_RATE_LIMIT_PER_HOUR and
// SWWAF_RATE_LIMIT_PER_DAY).
RateLimitPerMinute int64
RateLimitPerHour int64
RateLimitPerDay int64
// DeniedCountries are the countries whose clients are refused
// (SWWAF_DENIED_COUNTRIES). ExclusivelyAllowedCountries, when not
// empty, are the only countries whose clients are let through
// (SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES). Both hold two-letter codes in
// capitals, as GeoJS gives them.
DeniedCountries []string
ExclusivelyAllowedCountries []string
// settings are the values read, as given or by default, for the
// log line at start.
settings []slog.Attr
}
// off is the value that switches a timeout, a size limit or a rate limit
// off.
const off = "off"
const (
day = 24 * time.Hour
kibibyte = 1 << 10
mebibyte = 1 << 20
gibibyte = 1 << 30
)
var (
errNotDuration = errors.New(
"is not a duration such as 90s, 15m or 7d, or off")
errNotSize = errors.New(
"is not a size such as 512K, 100M or 5G, or off")
errNotCount = errors.New(
"is not a whole number of requests such as 1000, or off")
errNotPositive = errors.New("must be more than zero, or off")
errEmptyItem = errors.New("has an empty item in its list")
errNotNetblock = errors.New(
"is not a netblock such as 10.0.0.0/8, or an address")
errNotListenAddr = errors.New(
"is not an address to listen on, such as :8080")
errNotUpstreamURL = errors.New(
"is not a URL with only a scheme, a host and an optional port, " +
"such as http://127.0.0.1:8081")
errNotCountry = errors.New(
"is not a two-letter country code such as de or kp")
errOnBothLists = errors.New("is in SWWAF_DENIED_COUNTRIES too")
)
// FromEnvironment reads the settings with lookupEnv, normally
// os.LookupEnv. A setting that is not set takes its default. A setting
// that is set but invalid is an error that names it.
func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
env := &environment{lookupEnv: lookupEnv}
cfg := &Config{
ListenAddr: env.address("SWWAF_LISTEN_ADDR", ":8080"),
UpstreamURL: env.appURL("SWWAF_UPSTREAM_URL", "http://127.0.0.1:8081"),
TrustedProxies: env.netblocks("SWWAF_TRUSTED_PROXIES", privateRanges),
ClientRequestTimeout: env.duration("SWWAF_CLIENT_REQUEST_TIMEOUT", "60s"),
ClientResponseTimeout: env.duration("SWWAF_CLIENT_RESPONSE_TIMEOUT", "30m"),
UpstreamRequestTimeout: env.duration("SWWAF_UPSTREAM_REQUEST_TIMEOUT", "60s"),
UpstreamResponseTimeout: env.duration("SWWAF_UPSTREAM_RESPONSE_TIMEOUT", "30m"),
RequestMaxBytes: env.size("SWWAF_REQUEST_MAX_BYTES", "100M"),
ResponseMaxBytes: env.size("SWWAF_RESPONSE_MAX_BYTES", "5G"),
AllowNets: env.netblocks("SWWAF_ALLOW_NETS", ""),
RateLimitExemptNets: env.netblocks("SWWAF_RATE_LIMIT_EXEMPT_NETS", ""),
DenyNets: env.netblocks("SWWAF_DENY_NETS", ""),
RateLimitPerMinute: env.count("SWWAF_RATE_LIMIT_PER_MINUTE", "1000"),
RateLimitPerHour: env.count("SWWAF_RATE_LIMIT_PER_HOUR", "10000"),
RateLimitPerDay: env.count("SWWAF_RATE_LIMIT_PER_DAY", "50000"),
DeniedCountries: env.countries("SWWAF_DENIED_COUNTRIES", ""),
ExclusivelyAllowedCountries: env.countries(
"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES", ""),
}
for _, country := range cfg.ExclusivelyAllowedCountries {
if slices.Contains(cfg.DeniedCountries, country) {
env.check("SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES",
fmt.Errorf("%q %w", country, errOnBothLists))
}
}
if env.err != nil {
return nil, env.err
}
cfg.settings = env.settings
return cfg, nil
}
// privateRanges are the private address ranges, the default trusted
// proxies.
const privateRanges = "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"
// LogValue makes a Config log as each setting's name with the value it was
// given, or its default.
func (c *Config) LogValue() slog.Value {
return slog.GroupValue(c.settings...)
}
// environment is where FromEnvironment reads the settings: it notes each
// value for the log, and keeps the first error.
type environment struct {
lookupEnv func(string) (string, bool)
settings []slog.Attr
err error
}
// value returns a setting's value, or its default when it is not set,
// and notes it for the log.
func (e *environment) value(name, defaultValue string) string {
value, ok := e.lookupEnv(name)
if !ok {
value = defaultValue
}
e.settings = append(e.settings, slog.String(name, value))
return value
}
// check keeps the first error, naming the setting it is about.
func (e *environment) check(name string, err error) {
if err != nil && e.err == nil {
e.err = fmt.Errorf("%s: %w", name, err)
}
}
// address reads a setting that is an address to listen on.
func (e *environment) address(name, defaultValue string) string {
address, err := parseListenAddr(e.value(name, defaultValue))
e.check(name, err)
return address
}
// appURL reads a setting that is the app's URL.
func (e *environment) appURL(name, defaultValue string) *url.URL {
upstream, err := parseUpstreamURL(e.value(name, defaultValue))
e.check(name, err)
return upstream
}
// netblocks reads a setting that is a list of netblocks.
func (e *environment) netblocks(name, defaultValue string) []netip.Prefix {
netblocks, err := parseNetblocks(e.value(name, defaultValue))
e.check(name, err)
return netblocks
}
// duration reads a setting that is a duration.
func (e *environment) duration(name, defaultValue string) time.Duration {
duration, err := parseDuration(e.value(name, defaultValue))
e.check(name, err)
return duration
}
// size reads a setting that is a number of bytes.
func (e *environment) size(name, defaultValue string) int64 {
size, err := parseSize(e.value(name, defaultValue))
e.check(name, err)
return size
}
// count reads a setting that is a number of requests.
func (e *environment) count(name, defaultValue string) int64 {
count, err := parseCount(e.value(name, defaultValue))
e.check(name, err)
return count
}
// countries reads a setting that is a list of countries.
func (e *environment) countries(name, defaultValue string) []string {
countries, err := parseCountries(e.value(name, defaultValue))
e.check(name, err)
return countries
}
// parseDuration reads a duration in Go's syntax, such as 90s or 15m, a
// whole number of days such as 7d, or off.
func parseDuration(value string) (time.Duration, error) {
if value == off {
return 0, nil
}
duration, err := durationOrDays(value)
if err != nil {
return 0, fmt.Errorf("%q %w", value, errNotDuration)
}
if duration <= 0 {
return 0, fmt.Errorf("%q %w", value, errNotPositive)
}
return duration, nil
}
// durationOrDays reads Go's duration syntax, or a whole number of days.
func durationOrDays(value string) (time.Duration, error) {
days, isDays := strings.CutSuffix(value, "d")
if !isDays {
return time.ParseDuration(value)
}
n, err := strconv.ParseInt(days, 10, 64)
if err != nil || n < 0 || n > math.MaxInt64/int64(day) {
return 0, errNotDuration
}
return time.Duration(n) * day, nil
}
// parseSize reads a number of bytes with an optional K, M or G suffix, in
// powers of 1024 (1K is 1024 bytes), or off.
func parseSize(value string) (int64, error) {
if value == off {
return 0, nil
}
number, unit := splitUnit(value)
n, err := strconv.ParseInt(number, 10, 64)
if err != nil || n > math.MaxInt64/unit {
return 0, fmt.Errorf("%q %w", value, errNotSize)
}
if n <= 0 {
return 0, fmt.Errorf("%q %w", value, errNotPositive)
}
return n * unit, nil
}
// splitUnit splits a size into its number and the bytes its suffix
// stands for.
func splitUnit(value string) (string, int64) {
switch {
case strings.HasSuffix(value, "K"):
return strings.TrimSuffix(value, "K"), kibibyte
case strings.HasSuffix(value, "M"):
return strings.TrimSuffix(value, "M"), mebibyte
case strings.HasSuffix(value, "G"):
return strings.TrimSuffix(value, "G"), gibibyte
default:
return value, 1
}
}
// parseCount reads a whole number of requests, or off.
func parseCount(value string) (int64, error) {
if value == off {
return 0, nil
}
n, err := strconv.ParseInt(value, 10, 64)
if err != nil {
return 0, fmt.Errorf("%q %w", value, errNotCount)
}
if n <= 0 {
return 0, fmt.Errorf("%q %w", value, errNotPositive)
}
return n, nil
}
// parseList splits a comma-separated list and trims the spaces around
// each item. An empty value is an empty list.
func parseList(value string) ([]string, error) {
if strings.TrimSpace(value) == "" {
return []string{}, nil
}
items := strings.Split(value, ",")
for i, item := range items {
items[i] = strings.TrimSpace(item)
if items[i] == "" {
return nil, fmt.Errorf("%q %w", value, errEmptyItem)
}
}
return items, nil
}
// parseNetblocks reads a comma-separated list of netblocks.
func parseNetblocks(value string) ([]netip.Prefix, error) {
items, err := parseList(value)
if err != nil {
return nil, err
}
netblocks := make([]netip.Prefix, 0, len(items))
for _, item := range items {
netblock, err := parseNetblock(item)
if err != nil {
return nil, err
}
netblocks = append(netblocks, netblock)
}
return netblocks, nil
}
// parseNetblock reads a netblock in CIDR form, such as 10.0.0.0/8. A bare
// address is a netblock of that address alone, a /32 or a /128.
func parseNetblock(value string) (netip.Prefix, error) {
if strings.Contains(value, "/") {
netblock, err := netip.ParsePrefix(value)
if err != nil {
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
}
return netblock.Masked(), nil
}
addr, err := netip.ParseAddr(value)
if err != nil || addr.Zone() != "" {
return netip.Prefix{}, fmt.Errorf("%q %w", value, errNotNetblock)
}
return netip.PrefixFrom(addr, addr.BitLen()), nil
}
// countryCodes are the two-letter codes ISO 3166-1 assigns today, and XK,
// the code in common use for Kosovo. golang.org/x/text/language cannot
// check them: it also takes withdrawn codes such as su, and reserved ones
// such as ac, as countries.
const countryCodes = `
AD AE AF AG AI AL AM AO AQ AR AS AT AU AW AX AZ
BA BB BD BE BF BG BH BI BJ BL BM BN BO BQ BR BS BT BV BW BY BZ
CA CC CD CF CG CH CI CK CL CM CN CO CR CU CV CW CX CY CZ
DE DJ DK DM DO DZ
EC EE EG EH ER ES ET
FI FJ FK FM FO FR
GA GB GD GE GF GG GH GI GL GM GN GP GQ GR GS GT GU GW GY
HK HM HN HR HT HU
ID IE IL IM IN IO IQ IR IS IT
JE JM JO JP
KE KG KH KI KM KN KP KR KW KY KZ
LA LB LC LI LK LR LS LT LU LV LY
MA MC MD ME MF MG MH MK ML MM MN MO MP MQ MR MS MT MU MV MW MX MY MZ
NA NC NE NF NG NI NL NO NP NR NU NZ
OM
PA PE PF PG PH PK PL PM PN PR PS PT PW PY
QA
RE RO RS RU RW
SA SB SC SD SE SG SH SI SJ SK SL SM SN SO SR SS ST SV SX SY SZ
TC TD TF TG TH TJ TK TL TM TN TO TR TT TV TW TZ
UA UG UM US UY UZ
VA VC VE VG VI VN VU
WF WS
XK
YE YT
ZA ZM ZW
`
// parseCountries reads a comma-separated list of country codes in either
// case, and returns them in capitals.
func parseCountries(value string) ([]string, error) {
items, err := parseList(value)
if err != nil {
return nil, err
}
known := strings.Fields(countryCodes)
countries := make([]string, 0, len(items))
for _, item := range items {
country := strings.ToUpper(item)
if !slices.Contains(known, country) {
return nil, fmt.Errorf("%q %w", item, errNotCountry)
}
countries = append(countries, country)
}
return countries, nil
}
// parseListenAddr checks an address to listen on: an optional host and a
// port number.
func parseListenAddr(value string) (string, error) {
_, port, err := net.SplitHostPort(value)
if err != nil {
return "", fmt.Errorf("%q %w", value, errNotListenAddr)
}
_, err = strconv.ParseUint(port, 10, 16)
if err != nil {
return "", fmt.Errorf("%q %w", value, errNotListenAddr)
}
return value, nil
}
// parseUpstreamURL reads the app's URL: http or https, a host and an
// optional port from 1 to 65535, and nothing else, since the request's
// own path and query go to the app unchanged.
func parseUpstreamURL(value string) (*url.URL, error) {
upstream, err := url.Parse(value)
if err != nil {
return nil, fmt.Errorf("%q %w", value, errNotUpstreamURL)
}
onlySchemeAndHost := (upstream.Scheme == "http" || upstream.Scheme == "https") &&
upstream.Hostname() != "" && upstream.User == nil && upstream.Opaque == "" &&
(upstream.Path == "" || upstream.Path == "/") &&
upstream.RawQuery == "" && upstream.Fragment == ""
if !onlySchemeAndHost {
return nil, fmt.Errorf("%q %w", value, errNotUpstreamURL)
}
if upstream.Port() != "" {
port, err := strconv.ParseUint(upstream.Port(), 10, 16)
if err != nil || port == 0 {
return nil, fmt.Errorf("%q %w", value, errNotUpstreamURL)
}
}
return upstream, nil
}
+350
View File
@@ -0,0 +1,350 @@
package config_test
import (
"bytes"
"encoding/json"
"log/slog"
"maps"
"net/netip"
"slices"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
)
// The settings, by name.
const (
listenAddr = "SWWAF_LISTEN_ADDR"
upstreamURL = "SWWAF_UPSTREAM_URL"
trustedProxies = "SWWAF_TRUSTED_PROXIES"
clientRequestTimeout = "SWWAF_CLIENT_REQUEST_TIMEOUT"
clientResponseTimeout = "SWWAF_CLIENT_RESPONSE_TIMEOUT"
upstreamRequestTimeout = "SWWAF_UPSTREAM_REQUEST_TIMEOUT"
upstreamResponseTimeout = "SWWAF_UPSTREAM_RESPONSE_TIMEOUT"
requestMaxBytes = "SWWAF_REQUEST_MAX_BYTES"
responseMaxBytes = "SWWAF_RESPONSE_MAX_BYTES"
allowNets = "SWWAF_ALLOW_NETS"
rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS"
denyNets = "SWWAF_DENY_NETS"
rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE"
rateLimitPerHour = "SWWAF_RATE_LIMIT_PER_HOUR"
rateLimitPerDay = "SWWAF_RATE_LIMIT_PER_DAY"
deniedCountries = "SWWAF_DENIED_COUNTRIES"
allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
)
// off switches a timeout, a size limit or a rate limit off.
const off = "off"
// environment is a set of environment variables, for FromEnvironment.
type environment map[string]string
// lookupEnv reads one of the variables, as os.LookupEnv does.
func (e environment) lookupEnv(name string) (string, bool) {
value, ok := e[name]
return value, ok
}
// fromEnvironment reads the settings from env, which must be valid.
func fromEnvironment(t *testing.T, env environment) *config.Config {
t.Helper()
cfg, err := config.FromEnvironment(env.lookupEnv)
if err != nil {
t.Fatalf("settings %v: %v", env, err)
}
return cfg
}
func TestDefaults(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{})
wantSettings(t, cfg, config.Config{
ListenAddr: ":8080",
ClientRequestTimeout: time.Minute,
ClientResponseTimeout: 30 * time.Minute,
UpstreamRequestTimeout: time.Minute,
UpstreamResponseTimeout: 30 * time.Minute,
RequestMaxBytes: 100 << 20,
ResponseMaxBytes: 5 << 30,
RateLimitPerMinute: 1000,
RateLimitPerHour: 10000,
RateLimitPerDay: 50000,
})
if cfg.UpstreamURL.String() != "http://127.0.0.1:8081" {
t.Errorf("%s is %s", upstreamURL, cfg.UpstreamURL)
}
wantNetblocks(t, cfg.TrustedProxies,
"10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")
wantNetblocks(t, cfg.AllowNets)
wantNetblocks(t, cfg.RateLimitExemptNets)
wantNetblocks(t, cfg.DenyNets)
wantCountries(t, deniedCountries, cfg.DeniedCountries)
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries)
}
func TestValuesAsSet(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{
listenAddr: "127.0.0.1:9000",
upstreamURL: "https://app.internal:8443/",
trustedProxies: " 192.0.2.1, 10.1.2.3/8 ,2001:db8::/32",
clientRequestTimeout: "90s",
clientResponseTimeout: "7d",
upstreamRequestTimeout: "1h30m",
upstreamResponseTimeout: off,
requestMaxBytes: "512K",
responseMaxBytes: "1234",
allowNets: "192.0.2.7",
rateLimitExemptNets: "2001:db8::/48, 10.9.8.7",
denyNets: "198.51.100.0/24",
rateLimitPerMinute: "60",
rateLimitPerHour: "600",
rateLimitPerDay: "6000",
deniedCountries: "cn, RU,kp,Xk",
allowedCountries: "de",
})
wantSettings(t, cfg, config.Config{
ListenAddr: "127.0.0.1:9000",
ClientRequestTimeout: 90 * time.Second,
ClientResponseTimeout: 7 * 24 * time.Hour,
UpstreamRequestTimeout: 90 * time.Minute,
UpstreamResponseTimeout: 0,
RequestMaxBytes: 512 << 10,
ResponseMaxBytes: 1234,
RateLimitPerMinute: 60,
RateLimitPerHour: 600,
RateLimitPerDay: 6000,
})
if cfg.UpstreamURL.String() != "https://app.internal:8443/" {
t.Errorf("%s is %s", upstreamURL, cfg.UpstreamURL)
}
wantNetblocks(t, cfg.TrustedProxies, "192.0.2.1/32", "10.0.0.0/8", "2001:db8::/32")
wantNetblocks(t, cfg.AllowNets, "192.0.2.7/32")
wantNetblocks(t, cfg.RateLimitExemptNets, "2001:db8::/48", "10.9.8.7/32")
wantNetblocks(t, cfg.DenyNets, "198.51.100.0/24")
wantCountries(t, deniedCountries, cfg.DeniedCountries, "CN", "RU", "KP", "XK")
wantCountries(t, allowedCountries, cfg.ExclusivelyAllowedCountries, "DE")
}
func TestCodeOnBothCountryListsStopsTheStart(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{
deniedCountries: "cn,ru",
allowedCountries: "de,RU",
}.lookupEnv)
if err == nil {
t.Fatal("ru on both country lists was accepted")
}
if !strings.HasPrefix(err.Error(), allowedCountries+": ") ||
!strings.Contains(err.Error(), `"RU"`) {
t.Errorf("error %q does not name %s and RU", err, allowedCountries)
}
}
func TestSizesAndOff(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{
requestMaxBytes: "3G",
responseMaxBytes: off,
clientRequestTimeout: off,
})
if cfg.RequestMaxBytes != 3<<30 || cfg.ResponseMaxBytes != 0 ||
cfg.ClientRequestTimeout != 0 {
t.Errorf("3G, off and off read as %d, %d and %s",
cfg.RequestMaxBytes, cfg.ResponseMaxBytes, cfg.ClientRequestTimeout)
}
}
func TestRateLimitsOff(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{
rateLimitPerMinute: off,
rateLimitPerHour: off,
rateLimitPerDay: off,
})
if cfg.RateLimitPerMinute != 0 || cfg.RateLimitPerHour != 0 ||
cfg.RateLimitPerDay != 0 {
t.Errorf("off read as %d, %d and %d",
cfg.RateLimitPerMinute, cfg.RateLimitPerHour, cfg.RateLimitPerDay)
}
}
func TestTrustedProxiesSetButEmptyTrustNothing(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{trustedProxies: ""})
if len(cfg.TrustedProxies) != 0 {
t.Errorf("trusted proxies %v, want none", cfg.TrustedProxies)
}
}
func TestInvalidValueStopsTheStart(t *testing.T) {
t.Parallel()
for _, tc := range []struct{ name, value string }{
{listenAddr, "8080"},
{listenAddr, ":http"},
{listenAddr, ":65536"},
{upstreamURL, "127.0.0.1:8081"},
{upstreamURL, "ftp://127.0.0.1:8081"},
{upstreamURL, "http://"},
{upstreamURL, "http://:8081"},
{upstreamURL, "http://127.0.0.1:0"},
{upstreamURL, "http://127.0.0.1:99999"},
{upstreamURL, "http://127.0.0.1:8081/app"},
{upstreamURL, "http://127.0.0.1:8081/?a=1"},
{upstreamURL, "http://user:secret@127.0.0.1:8081"},
{trustedProxies, "10.0.0.0/33"},
{trustedProxies, "traefik"},
{trustedProxies, "10.0.0.0/8,,192.168.0.0/16"},
{trustedProxies, "fe80::1%eth0"},
{allowNets, "192.0.2.0/24,monitoring"},
{rateLimitExemptNets, "2001:db8::/129"},
{denyNets, "198.51.100.0/24,"},
{clientRequestTimeout, "60"},
{clientRequestTimeout, ""},
{clientResponseTimeout, "1y"},
{upstreamRequestTimeout, "-1s"},
{upstreamResponseTimeout, "0s"},
{upstreamResponseTimeout, "1.5d"},
{requestMaxBytes, "100MB"},
{requestMaxBytes, "100m"},
{requestMaxBytes, "1.5M"},
{responseMaxBytes, "0"},
{responseMaxBytes, "-5"},
{responseMaxBytes, "99999999999G"},
{rateLimitPerMinute, ""},
{rateLimitPerMinute, "1K"},
{rateLimitPerHour, "0"},
{rateLimitPerHour, "1.5"},
{rateLimitPerDay, "-1"},
{rateLimitPerDay, "lots"},
{deniedCountries, "nk"},
{deniedCountries, "kp,,ir"},
{deniedCountries, "prk"},
{deniedCountries, "408"},
{deniedCountries, "k"},
{deniedCountries, "eu"},
{deniedCountries, "un"},
{deniedCountries, "su"},
{allowedCountries, "ac"},
{allowedCountries, "uk"},
{allowedCountries, "zz"},
{allowedCountries, "de,germany"},
} {
t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
if err == nil {
t.Fatalf("%s=%q was accepted", tc.name, tc.value)
}
if !strings.HasPrefix(err.Error(), tc.name+": ") {
t.Errorf("error %q does not name %s", err, tc.name)
}
})
}
}
func TestLogsEachSettingWithItsValue(t *testing.T) {
t.Parallel()
cfg := fromEnvironment(t, environment{clientRequestTimeout: "45s"})
var out bytes.Buffer
slog.New(slog.NewJSONHandler(&out, nil)).Info("starting", "settings", cfg)
var line struct {
Settings map[string]string `json:"settings"`
}
err := json.Unmarshal(out.Bytes(), &line)
if err != nil {
t.Fatalf("decode %s: %v", out.Bytes(), err)
}
want := map[string]string{
listenAddr: ":8080",
upstreamURL: "http://127.0.0.1:8081",
trustedProxies: "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
clientRequestTimeout: "45s",
clientResponseTimeout: "30m",
upstreamRequestTimeout: "60s",
upstreamResponseTimeout: "30m",
requestMaxBytes: "100M",
responseMaxBytes: "5G",
allowNets: "",
rateLimitExemptNets: "",
denyNets: "",
rateLimitPerMinute: "1000",
rateLimitPerHour: "10000",
rateLimitPerDay: "50000",
deniedCountries: "",
allowedCountries: "",
}
if !maps.Equal(line.Settings, want) {
t.Errorf("logged settings\n%v\nwant\n%v", line.Settings, want)
}
}
// wantSettings checks the settings that are plain values.
func wantSettings(t *testing.T, got *config.Config, want config.Config) {
t.Helper()
if got.ListenAddr != want.ListenAddr ||
got.ClientRequestTimeout != want.ClientRequestTimeout ||
got.ClientResponseTimeout != want.ClientResponseTimeout ||
got.UpstreamRequestTimeout != want.UpstreamRequestTimeout ||
got.UpstreamResponseTimeout != want.UpstreamResponseTimeout ||
got.RequestMaxBytes != want.RequestMaxBytes ||
got.ResponseMaxBytes != want.ResponseMaxBytes ||
got.RateLimitPerMinute != want.RateLimitPerMinute ||
got.RateLimitPerHour != want.RateLimitPerHour ||
got.RateLimitPerDay != want.RateLimitPerDay {
t.Errorf("settings\n%+v\nwant\n%+v", got, want)
}
}
// wantNetblocks checks a list of netblocks.
func wantNetblocks(t *testing.T, got []netip.Prefix, want ...string) {
t.Helper()
gotText := make([]string, 0, len(got))
for _, netblock := range got {
gotText = append(gotText, netblock.String())
}
if !slices.Equal(gotText, want) {
t.Errorf("netblocks %v, want %v", gotText, want)
}
}
// wantCountries checks the list of countries the setting name gave.
func wantCountries(t *testing.T, name string, got []string, want ...string) {
t.Helper()
if !slices.Equal(got, want) {
t.Errorf("%s gave %v, want %v", name, got, want)
}
}
+385
View File
@@ -0,0 +1,385 @@
// Package lookup looks up each client's country through the GeoJS web
// service, and keeps the answers in memory, for at most 100,000 clients
// and for 7 days each.
package lookup
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"net/netip"
"strings"
"sync"
"time"
"github.com/hashicorp/golang-lru/v2/simplelru"
)
// URL is GeoJS's country endpoint. Asked about several addresses at once,
// comma separated in its ip parameter, it answers with a list.
const URL = "https://get.geojs.io/v1/ip/country.json"
const (
// keepFor is how long an answer is used instead of asking GeoJS again.
keepFor = 7 * 24 * time.Hour
// maxAnswers is how many answers are kept. Past it, the one used
// longest ago is dropped.
maxAnswers = 100000
// maxWaiting is how many clients may wait to be asked about. Past it,
// a new client counts as not found and is not asked about until there
// is room, so that a swarm of new addresses while GeoJS is down cannot
// fill the memory.
maxWaiting = 10000
// maxPerRequest is how many addresses one request to GeoJS asks about.
maxPerRequest = 200
// timeout is how long a new client waits for its answer, and how long
// a request to GeoJS may take before it is abandoned.
timeout = time.Second
// After a failure GeoJS is not asked again for a second, and for
// retryDelayFactor times as long after each further failure in a row,
// up to five minutes.
firstRetryDelay = time.Second
retryDelayFactor = 2
maxRetryDelay = 5 * time.Minute
// maxResponseBytes is the most of GeoJS's answer that is read.
maxResponseBytes = 1 << 20
)
var (
errStatus = errors.New("GeoJS answered")
errLeftOut = errors.New("GeoJS's answer left out")
)
// Params are what New needs.
type Params struct {
// URL is where GeoJS is asked, normally URL.
URL string
// Now tells the time, normally time.Now.
Now func() time.Time
// ProcessLog receives GeoJS's failures.
ProcessLog *slog.Logger
}
// GeoJS looks up clients' countries through GeoJS. At most one request
// to GeoJS is under way at a time, and it asks about every client waiting,
// up to maxPerRequest. It is safe for concurrent use.
type GeoJS struct {
url string
now func() time.Time
processLog *slog.Logger
// httpClient follows no redirect, so that visitors' addresses go to
// GeoJS alone: a redirect is a failure.
httpClient *http.Client
mu sync.Mutex
answers *simplelru.LRU[netip.Prefix, answer]
// waiting are the clients without an answer: those to ask GeoJS about,
// and those it is being asked about.
waiting map[netip.Prefix]*wait
// asking is true while a request to GeoJS is under way.
asking bool
// retryDelay is how long GeoJS is left alone after its last failure,
// zero after an answer; retryAt is when it may be asked again.
retryDelay time.Duration
retryAt time.Time
}
// answer is what GeoJS said about a client: its country, "" when GeoJS
// cannot place it, and when GeoJS said so.
type answer struct {
country string
received time.Time
}
// wait is a client waiting for its answer.
type wait struct {
// asked is closed when the client gets its answer, and closed and
// replaced each time GeoJS fails before then.
asked chan struct{}
// late is true once the client has gone without an answer, for a
// whole timeout or because GeoJS failed: its requests no longer wait.
late bool
}
// New returns a GeoJS with no answer kept yet.
func New(params Params) *GeoJS {
answers, err := simplelru.NewLRU[netip.Prefix, answer](maxAnswers, nil)
if err != nil {
panic(err) // NewLRU fails only for a size below one
}
return &GeoJS{
url: params.URL,
now: params.Now,
processLog: params.ProcessLog,
httpClient: &http.Client{
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
},
},
answers: answers,
waiting: map[netip.Prefix]*wait{},
}
}
// Country returns the country GeoJS places client in, as a two-letter
// code in capitals, or "" when the country cannot be found: GeoJS cannot
// place the client, or has not answered in time. An answer is kept for 7
// days. Without one, a client waits up to timeout for it, unless it has
// gone without one before; until GeoJS answers, the client is asked about
// again in the background. ctx is the context of the client's request,
// and ends the wait when it ends.
//
// GeoJS is asked about the client's first address, which is the client's
// own address for IPv4, and an address in the same place for an IPv6 /64.
func (g *GeoJS) Country(ctx context.Context, client netip.Prefix) string {
country, asked := g.answerOrWait(ctx, client)
if asked == nil {
return country
}
timer := time.NewTimer(timeout)
defer timer.Stop()
select {
case <-asked:
case <-timer.C:
case <-ctx.Done():
}
g.mu.Lock()
defer g.mu.Unlock()
country, found := g.kept(client)
w, waiting := g.waiting[client]
if !found && waiting {
w.late = true
}
return country
}
// answerOrWait returns client's kept answer if it has one. Otherwise it
// puts the client among those waiting if there is room, has GeoJS asked
// about them if it can be, and returns what to wait on for the answer, or
// nil when there is nothing to wait for.
func (g *GeoJS) answerOrWait(
ctx context.Context, client netip.Prefix,
) (string, <-chan struct{}) {
g.mu.Lock()
defer g.mu.Unlock()
country, found := g.kept(client)
if found {
return country, nil
}
w, waiting := g.waiting[client]
if !waiting && len(g.waiting) < maxWaiting {
w = &wait{asked: make(chan struct{})}
g.waiting[client] = w
}
g.ask(ctx)
if w == nil {
return "", nil // too many clients wait already
}
if !g.asking {
// GeoJS is left alone after a failure, so no answer can come.
w.late = true
}
if w.late {
return "", nil
}
return "", w.asked
}
// kept returns client's answer, if one was received less than keepFor
// ago.
func (g *GeoJS) kept(client netip.Prefix) (string, bool) {
kept, found := g.answers.Get(client)
if !found || g.now().Sub(kept.received) >= keepFor {
return "", false
}
return kept.country, true
}
// ask starts asking GeoJS about the waiting clients, unless a request to
// it is under way or it is left alone after a failure. The requests to
// GeoJS are for every client waiting, so they go on when the client's
// request whose ctx is given ends.
func (g *GeoJS) ask(ctx context.Context) {
if g.asking || g.now().Before(g.retryAt) {
return
}
g.asking = true
go g.askAboutWaiting(context.WithoutCancel(ctx))
}
// askAboutWaiting asks GeoJS about the waiting clients, one request at a
// time, until none is left or GeoJS fails.
func (g *GeoJS) askAboutWaiting(ctx context.Context) {
for {
clients := g.nextClients()
if len(clients) == 0 {
return
}
countries, err := g.request(ctx, clients)
if !g.keep(clients, countries, err) {
return
}
}
}
// nextClients returns up to maxPerRequest of the waiting clients. When
// none is waiting, it returns none and notes that no request to GeoJS is
// under way.
func (g *GeoJS) nextClients() []netip.Prefix {
g.mu.Lock()
defer g.mu.Unlock()
if len(g.waiting) == 0 {
g.asking = false
return nil
}
clients := make([]netip.Prefix, 0, min(len(g.waiting), maxPerRequest))
for client := range g.waiting {
if len(clients) == maxPerRequest {
break
}
clients = append(clients, client)
}
return clients
}
// keep notes how a request to GeoJS about clients ended, and reports
// whether GeoJS answered about all of them. Each client whose address
// GeoJS's answer names gets its answer, with no country when GeoJS gave
// none. An answer that leaves an address out is a failure. After a
// failure GeoJS is left alone for a while, and every client still waiting
// stops waiting and is asked about once GeoJS is asked again.
func (g *GeoJS) keep(
clients []netip.Prefix, countries map[netip.Addr]string, err error,
) bool {
g.mu.Lock()
defer g.mu.Unlock()
now := g.now()
leftOut := 0
for _, client := range clients {
country, named := countries[client.Addr()]
if !named {
leftOut++
continue
}
g.answers.Add(client, answer{country: country, received: now})
close(g.waiting[client].asked)
delete(g.waiting, client)
}
if err == nil && leftOut > 0 {
err = fmt.Errorf("%w %d of %d addresses", errLeftOut, leftOut, len(clients))
}
if err != nil {
g.retryDelay = min(max(retryDelayFactor*g.retryDelay, firstRetryDelay),
maxRetryDelay)
g.retryAt = now.Add(g.retryDelay)
g.asking = false
for _, w := range g.waiting {
close(w.asked)
w.asked = make(chan struct{})
w.late = true
}
g.processLog.Warn("asking GeoJS failed",
"error", err.Error(), "asking_again_in", g.retryDelay.String())
return false
}
g.retryDelay = 0
return true
}
// request asks GeoJS about clients in one request, and returns the
// country it gave, in capitals, for each address its answer names.
func (g *GeoJS) request(
ctx context.Context, clients []netip.Prefix,
) (map[netip.Addr]string, error) {
addrs := make([]string, 0, len(clients))
for _, client := range clients {
addrs = append(addrs, client.Addr().String())
}
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, g.url, http.NoBody)
if err != nil {
return nil, fmt.Errorf("make the request to GeoJS: %w", err)
}
req.URL.RawQuery = "ip=" + strings.Join(addrs, ",")
res, err := g.httpClient.Do(req)
if err != nil {
// Do's error names the URL, and so the visitors' addresses, which
// are not to be logged: only what went wrong is kept.
return nil, fmt.Errorf("ask GeoJS: %w", errors.Unwrap(err))
}
defer func() {
_ = res.Body.Close()
}()
if res.StatusCode != http.StatusOK {
return nil, fmt.Errorf("%w %s", errStatus, res.Status)
}
var answers []struct {
IP string `json:"ip"`
Country string `json:"country"`
}
err = json.NewDecoder(io.LimitReader(res.Body, maxResponseBytes)).Decode(&answers)
if err != nil {
return nil, fmt.Errorf("read GeoJS's answer: %w", err)
}
countries := make(map[netip.Addr]string, len(answers))
for _, item := range answers {
addr, err := netip.ParseAddr(item.IP)
if err == nil {
countries[addr] = strings.ToUpper(item.Country)
}
}
return countries, nil
}
+553
View File
@@ -0,0 +1,553 @@
package lookup_test
import (
"encoding/json"
"log/slog"
"net/http"
"net/http/httptest"
"net/netip"
"slices"
"strings"
"sync"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/lookup"
)
const (
// germany is where the stand-in for GeoJS places every address but
// unplaced.
germany = "DE"
// unplaced is the address it cannot place.
unplaced = "192.0.2.1"
// leftOut is the address it leaves out of its answer when
// answeringWithoutLeftOut.
leftOut = "203.0.113.7"
// timeout is how long a new client waits for its answer.
timeout = time.Second
// waitLimit bounds how long a test waits for what should happen.
waitLimit = 10 * time.Second
// pollInterval is how often a test looks again.
pollInterval = 10 * time.Millisecond
// week is how long an answer is kept.
week = 7 * 24 * time.Hour
)
func TestKeptAnswerIsUsedFor7DaysThenAskedAgain(t *testing.T) {
t.Parallel()
geojs, clock, g := start(t)
placed := netip.MustParsePrefix("203.0.113.9/32")
notPlaced := netip.MustParsePrefix(unplaced + "/32")
wantCountry(t, g, placed, germany)
wantCountry(t, g, notPlaced, "")
wantRequests(t, geojs, 2)
// An answer without a country is kept too.
clock.advance(week - time.Second)
wantCountry(t, g, placed, germany)
wantCountry(t, g, notPlaced, "")
wantRequests(t, geojs, 2)
clock.advance(time.Second)
wantCountry(t, g, placed, germany)
wantRequests(t, geojs, 3)
wantAsked(t, geojs, 2, "203.0.113.9")
}
func TestNewClientWaitsAtMostOneSecondThenCountsAsNotFound(t *testing.T) {
t.Parallel()
geojs, clock, g := start(t)
client := netip.MustParsePrefix("203.0.113.9/32")
// The client comes while GeoJS is asked about an earlier client, which
// it answers most of a second later. It is then asked about the client
// and does not answer: that request is abandoned a second after it
// began, well after the client's wait is over.
geojs.set(answeringSlowly)
var earlier sync.WaitGroup
earlier.Go(func() { g.Country(t.Context(), netip.MustParsePrefix("203.0.113.1/32")) })
defer earlier.Wait()
waitForRequests(t, geojs, 1)
geojs.set(hanging)
began := time.Now()
wantCountry(t, g, client, "")
took := time.Since(began)
if took < timeout || took > timeout+timeout/2 {
t.Errorf("waited %s for the answer, want %s", took, timeout)
}
// Its next request does not wait.
began = time.Now()
wantCountry(t, g, client, "")
took = time.Since(began)
if took > timeout/2 {
t.Errorf("waited %s again, want no wait", took)
}
// Once GeoJS answers, the client is asked about again in the
// background, and has its country.
geojs.set(answering)
waitForCountry(t, g, clock, client, germany)
}
func TestAddressLeftOutOfAnAnswerIsAskedAboutAgain(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
answers int
// named is whether the answer names the other client asked about.
named bool
}{
{"null", answeringNull, false},
{"empty list", answeringEmptyList, false},
{"list without " + leftOut, answeringWithoutLeftOut, true},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
geojs, clock, g := start(t)
other := netip.MustParsePrefix("203.0.113.1/32")
client := netip.MustParsePrefix(leftOut + "/32")
// GeoJS fails, and is left alone for a second while the client
// comes too, so that the next request asks about both.
geojs.set(failing)
wantCountry(t, g, other, "")
wantCountry(t, g, client, "")
geojs.set(tc.answers)
clock.advance(time.Second)
wantCountry(t, g, other, "")
waitForRequests(t, geojs, 2)
// The answer counts as a failure, and the client is asked about
// again, with the other client only if the answer left it out too.
geojs.set(answering)
waitForCountry(t, g, clock, client, germany)
wantCountry(t, g, other, germany)
wantRequests(t, geojs, 3)
if tc.named {
wantAsked(t, geojs, 2, leftOut)
} else {
wantAsked(t, geojs, 2, leftOut, "203.0.113.1")
}
})
}
}
func TestRedirectCountsAsFailure(t *testing.T) {
t.Parallel()
geojs, _, g := start(t)
geojs.set(redirecting)
wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "")
wantRequests(t, geojs, 1)
}
func TestCountryIsKeptInCapitals(t *testing.T) {
t.Parallel()
geojs, _, g := start(t)
geojs.set(answeringInLowerCase)
wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), germany)
}
func TestFailureIsLoggedWithoutTheAddressesAskedAbout(t *testing.T) {
t.Parallel()
var log strings.Builder
// Nothing listens on port 1, so asking GeoJS fails.
g := lookup.New(lookup.Params{
URL: "http://127.0.0.1:1",
Now: time.Now,
ProcessLog: slog.New(slog.NewTextHandler(&log, nil)),
})
wantCountry(t, g, netip.MustParsePrefix("203.0.113.9/32"), "")
logged := log.String()
if !strings.Contains(logged, "asking GeoJS failed") ||
strings.Contains(logged, "203.0.113.9") {
t.Errorf("logged %q, want the failure without the address asked about", logged)
}
}
func TestWaitingClientsAreAskedAboutInOneRequest(t *testing.T) {
t.Parallel()
geojs, clock, g := start(t)
// GeoJS fails, and is then left alone for a second, while three more
// clients come. An IPv6 client is a /64, and GeoJS is asked about its
// first address.
geojs.set(failing)
wantCountry(t, g, netip.MustParsePrefix("203.0.113.1/32"), "")
wantCountry(t, g, netip.MustParsePrefix("203.0.113.2/32"), "")
wantCountry(t, g, netip.MustParsePrefix("2001:db8:1:2::/64"), "")
wantRequests(t, geojs, 1)
geojs.set(answering)
clock.advance(time.Second)
wantCountry(t, g, netip.MustParsePrefix("203.0.113.3/32"), germany)
wantRequests(t, geojs, 2)
wantAsked(t, geojs, 1, "203.0.113.1", "203.0.113.2", "2001:db8:1:2::", "203.0.113.3")
}
func TestKeptAnswersUnaffectedWhileGeoJSFailsAndAskedAgainWithBackoff(t *testing.T) {
t.Parallel()
geojs, clock, g := start(t)
clients := newClients()
kept := clients()
wantCountry(t, g, kept, germany)
geojs.set(failing)
wantCountry(t, g, kept, germany)
wantRequests(t, geojs, 1)
// Each failure leaves GeoJS alone twice as long as the one before, up
// to five minutes. New clients meanwhile count as not found, and the
// client with a kept answer still gets its country, without GeoJS being
// asked.
requests := 1
for _, delay := range []time.Duration{
time.Second, 2 * time.Second, 4 * time.Second, 8 * time.Second,
16 * time.Second, 32 * time.Second, 64 * time.Second, 128 * time.Second,
256 * time.Second, 5 * time.Minute, 5 * time.Minute,
} {
wantCountry(t, g, clients(), "")
requests++
wantRequests(t, geojs, requests)
clock.advance(delay - time.Millisecond)
wantCountry(t, g, clients(), "")
wantCountry(t, g, kept, germany)
wantRequests(t, geojs, requests)
clock.advance(time.Millisecond)
}
// Once GeoJS answers again, it is asked about every client waiting.
geojs.set(answering)
wantCountry(t, g, clients(), germany)
wantRequests(t, geojs, requests+1)
asked := waitForRequests(t, geojs, requests+1)
if len(asked[requests]) != 23 {
t.Errorf("GeoJS was asked about %d clients, want 23", len(asked[requests]))
}
}
func TestAtMost200AddressesInOneRequest(t *testing.T) {
t.Parallel()
geojs, clock, g := start(t)
clients := newClients()
first := clients()
// 201 clients wait while GeoJS is left alone after a failure.
geojs.set(failing)
wantCountry(t, g, first, "")
for range 200 {
wantCountry(t, g, clients(), "")
}
// The first one's next request has GeoJS asked again.
geojs.set(answering)
clock.advance(time.Second)
wantCountry(t, g, first, "")
asked := waitForRequests(t, geojs, 3)
if len(asked[1]) != 200 || len(asked[2]) != 1 {
t.Errorf("GeoJS was asked about %d and then %d clients, want 200 and 1",
len(asked[1]), len(asked[2]))
}
}
func TestAtMost10000ClientsWait(t *testing.T) {
t.Parallel()
geojs, clock, g := start(t)
clients := newClients()
first := clients()
// 10,000 clients wait while GeoJS is left alone after a failure, and
// one more cannot join them.
geojs.set(failing)
wantCountry(t, g, first, "")
for range 9999 {
wantCountry(t, g, clients(), "")
}
extra := clients()
wantCountry(t, g, extra, "")
// The first one's next request has GeoJS asked about the 10,000, 200
// at a time, and not about the one more.
geojs.set(answering)
clock.advance(time.Second)
wantCountry(t, g, first, "")
asked := waitForRequests(t, geojs, 51)
for i, request := range asked {
if slices.Contains(request, extra.Addr().String()) {
t.Errorf("request %d asked about %s", i, extra.Addr())
}
}
// With room among those waiting, it is asked about.
wantCountry(t, g, extra, germany)
}
// How the stand-in for GeoJS answers.
const (
answering = iota
answeringSlowly // most of a second later
answeringInLowerCase // with each country in lower case
answeringWithoutLeftOut // with a list that leaves leftOut out
answeringEmptyList // with []
answeringNull // with null
failing // with 503
hanging // not at all, until the request is abandoned
redirecting // with a redirect to itself
)
// standIn is a stand-in for GeoJS. It notes the addresses each request
// asks about.
type standIn struct {
server *httptest.Server
mu sync.Mutex
answers int
requests [][]string
}
// ServeHTTP answers a request about the addresses in its ip parameter.
func (s *standIn) ServeHTTP(w http.ResponseWriter, r *http.Request) {
addrs := strings.Split(r.URL.Query().Get("ip"), ",")
s.mu.Lock()
s.requests = append(s.requests, addrs)
answers := s.answers
s.mu.Unlock()
switch answers {
case failing:
w.WriteHeader(http.StatusServiceUnavailable)
return
case hanging:
<-r.Context().Done()
return
case redirecting:
http.Redirect(w, r, "/", http.StatusFound)
return
case answeringSlowly:
select {
case <-time.After(timeout * 4 / 5):
case <-r.Context().Done():
return
}
}
list := make([]map[string]string, 0, len(addrs))
for _, addr := range addrs {
country := germany
switch {
case addr == unplaced:
country = ""
case addr == leftOut && answers == answeringWithoutLeftOut:
continue
case answers == answeringInLowerCase:
country = strings.ToLower(germany)
}
list = append(list, map[string]string{"ip": addr, "country": country})
}
var answer any = list
switch answers {
case answeringEmptyList:
answer = []string{}
case answeringNull:
answer = nil
}
err := json.NewEncoder(w).Encode(answer)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
// set sets how the stand-in answers.
func (s *standIn) set(answers int) {
s.mu.Lock()
defer s.mu.Unlock()
s.answers = answers
}
// asked returns the addresses each request has asked about so far.
func (s *standIn) asked() [][]string {
s.mu.Lock()
defer s.mu.Unlock()
return slices.Clone(s.requests)
}
// testClock is a clock the test sets.
type testClock struct {
mu sync.Mutex
now time.Time
}
// Now tells the time.
func (c *testClock) Now() time.Time {
c.mu.Lock()
defer c.mu.Unlock()
return c.now
}
// advance moves the clock on by d.
func (c *testClock) advance(d time.Duration) {
c.mu.Lock()
defer c.mu.Unlock()
c.now = c.now.Add(d)
}
// start starts a stand-in for GeoJS that answers, and returns it, a
// clock, and a GeoJS asking it by that clock.
func start(t *testing.T) (*standIn, *testClock, *lookup.GeoJS) {
t.Helper()
geojs := &standIn{}
geojs.server = httptest.NewServer(geojs)
t.Cleanup(geojs.server.Close)
clock := &testClock{now: time.Date(2026, 10, 4, 0, 0, 0, 0, time.UTC)}
g := lookup.New(lookup.Params{
URL: geojs.server.URL,
Now: clock.Now,
ProcessLog: slog.New(slog.DiscardHandler),
})
return geojs, clock, g
}
// newClients returns what returns a new IPv4 client each time it is
// called.
func newClients() func() netip.Prefix {
addr := netip.MustParseAddr("10.0.0.0")
return func() netip.Prefix {
addr = addr.Next()
return netip.PrefixFrom(addr, addr.BitLen())
}
}
// wantCountry checks the country g gives client.
func wantCountry(t *testing.T, g *lookup.GeoJS, client netip.Prefix, want string) {
t.Helper()
got := g.Country(t.Context(), client)
if got != want {
t.Errorf("%s is in %q, want %q", client, got, want)
}
}
// wantRequests checks how many requests GeoJS has had.
func wantRequests(t *testing.T, geojs *standIn, want int) {
t.Helper()
got := len(geojs.asked())
if got != want {
t.Errorf("GeoJS had %d requests, want %d", got, want)
}
}
// wantAsked checks the addresses request i asked about, in any order.
func wantAsked(t *testing.T, geojs *standIn, i int, want ...string) {
t.Helper()
asked := geojs.asked()
if len(asked) <= i {
t.Fatalf("GeoJS had %d requests, want more than %d", len(asked), i)
}
got := slices.Sorted(slices.Values(asked[i]))
slices.Sort(want)
if !slices.Equal(got, want) {
t.Errorf("request %d asked about %v, want %v", i, got, want)
}
}
// waitForRequests waits for GeoJS to have had count requests, and returns
// the addresses each asked about.
func waitForRequests(t *testing.T, geojs *standIn, count int) [][]string {
t.Helper()
deadline := time.Now().Add(waitLimit)
for time.Now().Before(deadline) {
asked := geojs.asked()
if len(asked) >= count {
return asked
}
time.Sleep(pollInterval)
}
t.Fatalf("fewer than %d requests to GeoJS after %s", count, waitLimit)
return nil
}
// waitForCountry waits for g to give client the country want, moving the
// clock on a minute at a time, so that GeoJS is asked again after a
// failure.
func waitForCountry(
t *testing.T, g *lookup.GeoJS, clock *testClock, client netip.Prefix, want string,
) {
t.Helper()
deadline := time.Now().Add(waitLimit)
for g.Country(t.Context(), client) != want {
if time.Now().After(deadline) {
t.Fatalf("%s is not in %q after %s", client, want, waitLimit)
}
clock.advance(time.Minute)
time.Sleep(pollInterval)
}
}
+171
View File
@@ -0,0 +1,171 @@
package proxy
import (
"errors"
"io"
"net/http"
"sync/atomic"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// errResponseTooLarge ends an app's response body that is longer than
// SWWAF_RESPONSE_MAX_BYTES.
var errResponseTooLarge = errors.New(
"the response body is over SWWAF_RESPONSE_MAX_BYTES")
// requestBody is the client's request body on its way to the app. The
// transport reads it on a goroutine of its own.
type requestBody struct {
// body is the client's body, ending in an *http.MaxBytesError past
// SWWAF_REQUEST_MAX_BYTES.
body io.ReadCloser
rq *request
// waiting is true while a Read waits for the client to send more.
waiting atomic.Bool
// received is true once the client has sent the whole body.
received atomic.Bool
// bytes is how much of the body has been read.
bytes atomic.Int64
}
// Read reads from the client's body.
func (b *requestBody) Read(p []byte) (int, error) {
b.waiting.Store(true)
n, err := b.body.Read(p)
b.waiting.Store(false)
b.bytes.Add(int64(n))
var tooLarge *http.MaxBytesError
switch {
case errors.Is(err, io.EOF):
b.received.Store(true)
b.rq.bodyReceived()
case errors.As(err, &tooLarge):
b.rq.refuse(refusal{
status: http.StatusRequestEntityTooLarge,
action: requestlog.ActionTooLarge,
})
}
return n, err
}
// Close closes the client's body.
func (b *requestBody) Close() error {
return b.body.Close()
}
// responseBody is the app's response body on its way to the client.
type responseBody struct {
// body is the app's body, ending in an *http.MaxBytesError past
// SWWAF_RESPONSE_MAX_BYTES.
body io.ReadCloser
rq *request
}
// Read reads from the app's body.
func (b *responseBody) Read(p []byte) (int, error) {
n, err := b.body.Read(p)
if err == nil {
return n, nil
}
var tooLarge *http.MaxBytesError
switch {
case errors.Is(err, io.EOF):
b.rq.responseReceived()
case errors.As(err, &tooLarge):
b.rq.refuse(refusal{
status: http.StatusBadGateway,
action: requestlog.ActionTooLarge,
})
return n, errResponseTooLarge
case b.rq.in.Context().Err() == nil:
// The answer broke off, not because the client went away. If a
// timeout cut it, that refusal came first and is the one kept.
b.rq.refuse(refusal{
status: http.StatusBadGateway,
action: requestlog.ActionUpstreamError,
})
}
return n, err
}
// Close closes the app's body.
func (b *responseBody) Close() error {
return b.body.Close()
}
// limitBody returns body, cut off with an *http.MaxBytesError after
// maxBytes, or unchanged if maxBytes is zero, which is off.
func limitBody(body io.ReadCloser, maxBytes int64) io.ReadCloser {
if maxBytes == 0 {
return body
}
// Without a ResponseWriter, MaxBytesReader only counts and cuts off.
return http.MaxBytesReader(nil, body, maxBytes)
}
// responseWriter is the response to the client. It notes the status and
// size for the log line, and the first error writing to the client.
type responseWriter struct {
http.ResponseWriter
// status is the final status sent, or zero before one is.
status int
bytes int64
err error
}
// WriteHeader sends the status and headers. An informational 1xx status
// is passed on and the final status still comes later.
func (w *responseWriter) WriteHeader(status int) {
if status >= http.StatusOK && w.status == 0 {
w.status = status
}
w.ResponseWriter.WriteHeader(status)
}
// Write sends part of the body.
func (w *responseWriter) Write(p []byte) (int, error) {
if w.status == 0 {
w.status = http.StatusOK
}
n, err := w.ResponseWriter.Write(p)
w.bytes += int64(n)
w.noteError(err)
return n, err
}
// FlushError sends what has been written so far.
// http.ResponseController calls it, as ReverseProxy does after each
// write.
func (w *responseWriter) FlushError() error {
err := http.NewResponseController(w.ResponseWriter).Flush()
w.noteError(err)
return err
}
// Unwrap lets http.ResponseController reach net/http's own
// ResponseWriter, which is how ReverseProxy takes over the connection of
// an upgraded request.
func (w *responseWriter) Unwrap() http.ResponseWriter {
return w.ResponseWriter
}
// noteError keeps the first error writing to the client.
func (w *responseWriter) noteError(err error) {
if w.err == nil {
w.err = err
}
}
+116
View File
@@ -0,0 +1,116 @@
package proxy
import (
"net/http"
"net/netip"
"slices"
"strings"
)
// peerAddress is the address of the request's TCP peer, normally traefik.
func peerAddress(r *http.Request) netip.Addr {
addrPort, err := netip.ParseAddrPort(r.RemoteAddr)
if err != nil {
return netip.Addr{}
}
return addrPort.Addr().Unmap()
}
// clientAddress works out who the client is. A peer outside the trusted
// proxies is the client, and what it says in X-Forwarded-For is ignored.
// For a peer inside them, X-Forwarded-For is read from the right, and the
// first address outside them is the client; if every address in it is
// inside, the leftmost is, and with no header, the peer. An entry that is
// not an address ends the reading, since nothing to its left can be
// believed.
func clientAddress(
peer netip.Addr, forwardedFor []string, trusted []netip.Prefix,
) netip.Addr {
client := peer
if !isInside(peer, trusted) {
return client
}
entries := strings.Split(strings.Join(forwardedFor, ","), ",")
for _, entry := range slices.Backward(entries) {
addr, err := netip.ParseAddr(strings.TrimSpace(entry))
if err != nil {
break
}
client = addr.Unmap()
if !isInside(client, trusted) {
break
}
}
return client
}
// ipv6GroupPrefix is the length of the IPv6 netblock that is one client.
const ipv6GroupPrefix = 64
// clientGroup is the client a request is counted toward: its IPv4
// address, or the /64 its IPv6 address is in, since one abuser usually
// holds a whole /64. An IPv4 address in IPv6 form counts as IPv4.
func clientGroup(addr netip.Addr) netip.Prefix {
addr = addr.Unmap()
if addr.Is6() {
return netip.PrefixFrom(addr, ipv6GroupPrefix).Masked()
}
return netip.PrefixFrom(addr, addr.BitLen())
}
// isInside reports whether addr is in one of the netblocks.
func isInside(addr netip.Addr, netblocks []netip.Prefix) bool {
return slices.ContainsFunc(netblocks, func(netblock netip.Prefix) bool {
return netblock.Contains(addr)
})
}
// setForwardedHeaders sets the headers in which the app learns about the
// client, so that it sees what it would see from traefik directly. A
// trusted proxy's forwarded headers pass on, with the proxy's own address
// added to X-Forwarded-For. Those of any other peer are its own claims and
// are replaced: X-Forwarded-For names the peer, X-Forwarded-Host the host
// it asked for, and X-Forwarded-Proto plain http, which is how it reached
// smallwebwaf.
func setForwardedHeaders(in, out *http.Request, peer netip.Addr, trusted bool) {
forwardedFor := peer.String()
if trusted {
// ReverseProxy removes these from out before Rewrite.
for _, name := range []string{"Forwarded", "X-Forwarded-Host", "X-Forwarded-Proto"} {
values, ok := in.Header[name]
if ok {
out.Header[name] = values
}
}
prior := in.Header.Values("X-Forwarded-For")
if len(prior) > 0 {
forwardedFor = strings.Join(prior, ", ") + ", " + forwardedFor
}
out.Header.Set("X-Forwarded-For", forwardedFor)
return
}
// ReverseProxy has removed Forwarded and the three set below; these
// are the other headers in which traefik tells the app about the
// client and its request.
for _, name := range []string{
"X-Forwarded-Port", "X-Forwarded-Server", "X-Forwarded-Uri",
"X-Forwarded-Method", "X-Forwarded-Prefix", "X-Forwarded-Tls-Client-Cert",
"X-Forwarded-Tls-Client-Cert-Info", "X-Real-Ip",
} {
out.Header.Del(name)
}
out.Header.Set("X-Forwarded-For", forwardedFor)
out.Header.Set("X-Forwarded-Host", in.Host)
out.Header.Set("X-Forwarded-Proto", "http")
}
+161
View File
@@ -0,0 +1,161 @@
package proxy_test
import (
"encoding/json"
"net/http"
"testing"
)
const (
// trustLocalhost trusts the address every test connects from, and a
// network for proxies in front of it.
trustLocalhost = localhost + "/32,10.0.0.0/8"
// appHost is the host every test asks for.
appHost = "app.example"
// client is the client's address, as a proxy names it.
client = "203.0.113.9"
// forwardedFor is the header that lists the client and its proxies.
forwardedFor = "X-Forwarded-For"
// secure is the scheme a client reached traefik with.
secure = "https"
)
// appHeaders is what the app tells about the headers it received.
type appHeaders struct {
Host string `json:"host"`
ForwardedFor string `json:"forwardedFor"`
ForwardedHost string `json:"forwardedHost"`
ForwardedProto string `json:"forwardedProto"`
RealIP string `json:"realIp"`
}
// clientAddressCase is a request and what smallwebwaf makes of it.
type clientAddressCase struct {
name string
env map[string]string
header http.Header
wantClient string
wantApp appHeaders
}
func TestClientAddressAndForwardedHeaders(t *testing.T) {
t.Parallel()
for _, tc := range clientAddressCases() {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
got, line := requestWithHeaders(t, tc.env, tc.header)
tc.wantApp.Host = appHost
if got != tc.wantApp {
t.Errorf("app received %+v, want %+v", got, tc.wantApp)
}
if line.ClientIP != tc.wantClient || line.PeerIP != localhost {
t.Errorf("log line has client_ip %q and peer_ip %q, want %q and %q",
line.ClientIP, line.PeerIP, tc.wantClient, localhost)
}
})
}
}
// clientAddressCases are the requests TestClientAddressAndForwardedHeaders
// sends, from 127.0.0.1, which the default trusted proxies leave out.
func clientAddressCases() []clientAddressCase {
trusted := map[string]string{trustedProxies: trustLocalhost}
forged := http.Header{
forwardedFor: {client},
"X-Forwarded-Host": {"forged.example"},
"X-Forwarded-Proto": {secure},
"X-Real-Ip": {client},
}
replaced := appHeaders{
ForwardedFor: localhost, ForwardedHost: appHost, ForwardedProto: "http",
}
return []clientAddressCase{{
name: "a peer outside the trusted proxies is the client, " +
"and its forwarded headers are replaced",
header: forged, wantClient: localhost, wantApp: replaced,
}, {
name: "set but empty, the trusted proxies trust nothing",
env: map[string]string{trustedProxies: ""},
header: forged, wantClient: localhost, wantApp: replaced,
}, {
name: "behind a trusted peer, the client is the first address " +
"outside the trusted proxies from the right",
env: trusted,
header: http.Header{
forwardedFor: {"198.51.100.7, " + client + ", 10.0.0.2"},
"X-Forwarded-Host": {appHost},
"X-Forwarded-Proto": {secure},
"X-Real-Ip": {client},
},
wantClient: client,
wantApp: appHeaders{
ForwardedFor: "198.51.100.7, " + client + ", 10.0.0.2, " + localhost,
ForwardedHost: appHost, ForwardedProto: secure, RealIP: client,
},
}, {
name: "when every address is a trusted proxy, the leftmost is the client",
env: trusted,
header: http.Header{forwardedFor: {"10.0.0.5, 10.0.0.2"}},
wantClient: "10.0.0.5",
wantApp: appHeaders{ForwardedFor: "10.0.0.5, 10.0.0.2, " + localhost},
}, {
name: "with no header, a trusted peer is the client",
env: trusted,
wantClient: localhost,
wantApp: appHeaders{ForwardedFor: localhost},
}, {
name: "an entry that is not an address ends the reading",
env: trusted,
header: http.Header{forwardedFor: {client + ", unknown, 10.0.0.2"}},
wantClient: "10.0.0.2",
wantApp: appHeaders{
ForwardedFor: client + ", unknown, 10.0.0.2, " + localhost,
},
}, {
name: "several header lines are read as one list",
env: trusted,
header: http.Header{forwardedFor: {"2001:db8::7", "10.0.0.2"}},
wantClient: "2001:db8::7",
wantApp: appHeaders{ForwardedFor: "2001:db8::7, 10.0.0.2, " + localhost},
}}
}
// requestWithHeaders sends a request for appHost with header through
// smallwebwaf, with the settings in env, and returns the headers the app
// received and the request's log line.
func requestWithHeaders(
t *testing.T, env map[string]string, header http.Header,
) (appHeaders, logLine) {
t.Helper()
app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
_ = json.NewEncoder(w).Encode(appHeaders{
Host: r.Host,
ForwardedFor: r.Header.Get(forwardedFor),
ForwardedHost: r.Header.Get("X-Forwarded-Host"),
ForwardedProto: r.Header.Get("X-Forwarded-Proto"),
RealIP: r.Header.Get("X-Real-IP"),
})
})
addr, out := startProxy(t, app.URL, env)
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
req.Host = appHost
req.Header = header.Clone()
answered := do(t, req)
var got appHeaders
err := json.Unmarshal(answered.body, &got)
if err != nil {
t.Fatalf("decode the app's answer %q: %v", answered.body, err)
}
return got, out.requestLine(t)
}
+41
View File
@@ -0,0 +1,41 @@
package proxy
import (
"context"
"net/netip"
"slices"
)
// countryDenied reports whether the country lists refuse the request.
// The client's country is looked up only while a list is set, and never
// for a client on a private, loopback or link-local address, which has
// no country. A client without a country, or whose country cannot be
// found, is refused only by SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES. ctx is
// the request's own context.
func (rq *request) countryDenied(ctx context.Context) bool {
denied := rq.h.config.DeniedCountries
allowed := rq.h.config.ExclusivelyAllowedCountries
if len(denied) == 0 && len(allowed) == 0 {
return false
}
var country string
if hasCountry(rq.client) {
country = rq.h.geojs.Country(ctx, clientGroup(rq.client))
}
rq.line.Country = country
if slices.Contains(denied, country) {
return true
}
return len(allowed) > 0 && !slices.Contains(allowed, country)
}
// hasCountry reports whether addr can be placed in a country: private,
// loopback and link-local addresses cannot.
func hasCountry(addr netip.Addr) bool {
return !addr.IsPrivate() && !addr.IsLoopback() && !addr.IsLinkLocalUnicast()
}
+308
View File
@@ -0,0 +1,308 @@
package proxy_test
import (
"encoding/json"
"maps"
"net/http"
"net/http/httptest"
"slices"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// The clients the stand-in for GeoJS knows about.
const (
// fromDE is placed in Germany.
fromDE = client
// fromKP is placed in North Korea.
fromKP = "198.51.100.7"
// unplaced cannot be placed in any country.
unplaced = "192.0.2.1"
)
func TestCountryLists(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
env map[string]string
refused []string
}{
{"denied", map[string]string{deniedCountries: "kp"}, []string{fromKP}},
{
"exclusively allowed", map[string]string{allowedCountries: "DE"},
[]string{fromKP, unplaced},
},
{
"both", map[string]string{deniedCountries: "kp", allowedCountries: "de,fr"},
[]string{fromKP, unplaced},
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var calls atomic.Int32
app := startApp(t, func(http.ResponseWriter, *http.Request) {
calls.Add(1)
})
geojsURL, _ := startGeoJS(t)
env := map[string]string{trustedProxies: trustLocalhost}
maps.Copy(env, tc.env)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, env)
for i, sent := range []struct{ client, country string }{
{fromDE, "DE"}, {fromKP, "KP"}, {unplaced, ""},
} {
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
req.Header.Set(forwardedFor, sent.client)
got := do(t, req)
line := out.requestLines(t, i+1)[i]
if line.Country != sent.country {
t.Errorf("log line has country %q, want %q", line.Country, sent.country)
}
if slices.Contains(tc.refused, sent.client) {
wantStatus(t, got, http.StatusForbidden)
wantLine(t, line, http.StatusForbidden, requestlog.ActionCountryDenied)
} else {
wantStatus(t, got, http.StatusOK)
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
}
}
if int(calls.Load()) != 3-len(tc.refused) {
t.Errorf("the app was called %d times, want %d",
calls.Load(), 3-len(tc.refused))
}
})
}
}
func TestCountryRefusalComesBeforeTheBody(t *testing.T) {
t.Parallel()
var calls atomic.Int32
app := startApp(t, func(http.ResponseWriter, *http.Request) {
calls.Add(1)
})
geojsURL, _ := startGeoJS(t)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{
trustedProxies: trustLocalhost,
deniedCountries: "kp",
})
req := newRequest(t, http.MethodPost, addr, "/", strings.NewReader("a body"))
req.Header.Set(forwardedFor, fromKP)
wantStatus(t, do(t, req), http.StatusForbidden)
line := out.requestLine(t)
wantLine(t, line, http.StatusForbidden, requestlog.ActionCountryDenied)
if line.RequestBytes != 0 {
t.Errorf("log line has request_bytes %d, want 0", line.RequestBytes)
}
if calls.Load() != 0 {
t.Errorf("the app was called %d times, want none", calls.Load())
}
}
func TestRequestRefusedByCountryIsNotCounted(t *testing.T) {
t.Parallel()
// The stand-in for GeoJS fails until placing is set, and then places
// every address in Germany.
var placing atomic.Bool
geojs := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if !placing.Load() {
w.WriteHeader(http.StatusServiceUnavailable)
return
}
answer := []map[string]string{{"ip": r.URL.Query().Get("ip"), "country": "DE"}}
err := json.NewEncoder(w).Encode(answer)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}))
t.Cleanup(geojs.Close)
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
addr, _ := startProxyWithGeoJS(t, app.URL, geojs.URL, map[string]string{
trustedProxies: trustLocalhost,
allowedCountries: "de",
rateLimitPerMinute: "1",
})
request := func() answer {
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
req.Header.Set(forwardedFor, fromDE)
return do(t, req)
}
// While GeoJS fails, the client's country cannot be found, and its
// request is refused.
wantStatus(t, request(), http.StatusForbidden)
// Once GeoJS places it, a second after the failure, its requests are let
// through. No refused one was counted, so the first let through is
// within the limit of one a minute.
placing.Store(true)
deadline := time.Now().Add(waitLimit)
got := request()
for got.status == http.StatusForbidden && time.Now().Before(deadline) {
time.Sleep(pollInterval)
got = request()
}
wantStatus(t, got, http.StatusOK)
}
func TestCountryNotLookedUpWithoutAListOrForAPrivateAddress(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
env map[string]string
clients []string // "" sends no X-Forwarded-For: the client is 127.0.0.1
}{
{"no country list is set", nil, []string{fromKP, fromDE}},
{
"private, loopback and link-local addresses",
map[string]string{deniedCountries: "kp"},
[]string{"10.0.0.5", "192.168.1.9", "fd00::5", "", "169.254.0.9", "fe80::9"},
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
geojsURL, asked := startGeoJS(t)
env := map[string]string{trustedProxies: trustLocalhost}
maps.Copy(env, tc.env)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, env)
for i, sent := range tc.clients {
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
if sent != "" {
req.Header.Set(forwardedFor, sent)
}
wantStatus(t, do(t, req), http.StatusOK)
line := out.requestLines(t, i+1)[i]
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
country, present := line.fields["country"]
if !present || country != "" {
t.Errorf("log line for %q has country %v, want an empty one",
line.ClientIP, country)
}
}
if len(asked()) != 0 {
t.Errorf("GeoJS was asked about %v, want nothing", asked())
}
})
}
}
func TestExclusiveListRefusesAPrivateAddressUnlessAllowed(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
allowNets string
status int
action string
}{
{
"not in SWWAF_ALLOW_NETS", "",
http.StatusForbidden, requestlog.ActionCountryDenied,
},
{
"in SWWAF_ALLOW_NETS", "10.0.0.7,fd00::/8",
http.StatusOK, requestlog.ActionForward,
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
geojsURL, asked := startGeoJS(t)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{
trustedProxies: trustLocalhost,
allowedCountries: "de",
allowNets: tc.allowNets,
})
wantAnswers(t, addr, out, []sentRequest{
{"10.0.0.7", tc.status, tc.action},
{"fd00::5", tc.status, tc.action},
})
if len(asked()) != 0 {
t.Errorf("GeoJS was asked about %v, want nothing", asked())
}
})
}
}
// startGeoJS starts a stand-in for GeoJS, which places fromDE and fromKP
// and no other address. It returns its URL, and what returns the
// addresses it has been asked about.
func startGeoJS(t *testing.T) (string, func() []string) {
t.Helper()
places := map[string]string{fromDE: "DE", fromKP: "KP"}
var asked struct {
mu sync.Mutex
addrs []string
}
geojs := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
addrs := strings.Split(r.URL.Query().Get("ip"), ",")
asked.mu.Lock()
asked.addrs = append(asked.addrs, addrs...)
asked.mu.Unlock()
answers := make([]map[string]string, 0, len(addrs))
for _, addr := range addrs {
answers = append(answers, map[string]string{
"ip": addr, "country": places[addr],
})
}
err := json.NewEncoder(w).Encode(answers)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}))
t.Cleanup(geojs.Close)
return geojs.URL, func() []string {
asked.mu.Lock()
defer asked.mu.Unlock()
return slices.Clone(asked.addrs)
}
}
+47
View File
@@ -0,0 +1,47 @@
package proxy_test
import (
"net/http"
"sync/atomic"
"testing"
"sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
func TestHealthEndpointIsAnsweredBeforeAnyCheck(t *testing.T) {
t.Parallel()
var calls atomic.Int32
app := startApp(t, func(http.ResponseWriter, *http.Request) {
calls.Add(1)
})
// With a limit of one request a minute, any request counted before
// the last one would have it refused.
addr, out := startProxy(t, app.URL, map[string]string{rateLimitPerMinute: "1"})
const healthChecks = 3
for range healthChecks {
got := get(t, addr, proxy.HealthPath)
wantStatus(t, got, http.StatusOK)
if string(got.body) != "ok\n" {
t.Errorf("health endpoint answered %q, want ok", got.body)
}
}
wantStatus(t, get(t, addr, "/"), http.StatusOK)
lines := out.requestLines(t, healthChecks+1)
for _, line := range lines[:healthChecks] {
wantLine(t, line, http.StatusOK, requestlog.ActionAdmin)
}
wantLine(t, lines[healthChecks], http.StatusOK, requestlog.ActionForward)
if calls.Load() != 1 {
t.Errorf("the app was called %d times, want once", calls.Load())
}
}
+146
View File
@@ -0,0 +1,146 @@
package proxy_test
import (
"bytes"
"errors"
"io"
"net/http"
"strconv"
"sync/atomic"
"testing"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// sizeLimit is the size limit the tests set, 1K as a setting.
const (
sizeLimit = 1 << 10
sizeLimitSetting = "1K"
)
func TestRequestBodyLimit(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
size int
// announced sends the size in Content-Length; otherwise the body
// is sent in chunks with no length given.
announced bool
want int
action string
// refusedBeforeApp is a refusal before anything reaches the app.
// A body over the limit with no length given has already partly
// reached the app when it is refused.
refusedBeforeApp bool
}{
{"announced, over the limit", 2 * sizeLimit, true,
http.StatusRequestEntityTooLarge, requestlog.ActionTooLarge, true},
{"announced, at the limit", sizeLimit, true,
http.StatusOK, requestlog.ActionForward, false},
{"not announced, over the limit", 4 * sizeLimit, false,
http.StatusRequestEntityTooLarge, requestlog.ActionTooLarge, false},
{"not announced, at the limit", sizeLimit, false,
http.StatusOK, requestlog.ActionForward, false},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var calls atomic.Int32
app := startApp(t, func(_ http.ResponseWriter, r *http.Request) {
calls.Add(1)
_, _ = io.Copy(io.Discard, r.Body)
})
addr, out := startProxy(t, app.URL, map[string]string{
requestMaxBytes: sizeLimitSetting,
})
var body io.Reader = bytes.NewReader(make([]byte, tc.size))
if !tc.announced {
body = io.MultiReader(body) // hides the length
}
wantStatus(t, do(t, newRequest(t, http.MethodPost, addr, "/upload", body)),
tc.want)
wantLine(t, out.requestLine(t), tc.want, tc.action)
if tc.refusedBeforeApp && calls.Load() != 0 {
t.Errorf("the app was called %d times, want never", calls.Load())
}
})
}
}
func TestResponseBodyLimit(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
size int
// announced sends the size in Content-Length; otherwise the body
// is sent in chunks with no length given.
announced bool
want int
action string
// received is how much of a body the client gets, and cutOff
// whether the connection is then cut.
received int
cutOff bool
}{
{"announced, over the limit", 2 * sizeLimit, true, http.StatusBadGateway,
requestlog.ActionTooLarge, len("Bad Gateway\n"), false},
{"announced, at the limit", sizeLimit, true, http.StatusOK,
requestlog.ActionForward, sizeLimit, false},
{"not announced, over the limit", 4 * sizeLimit, false, http.StatusOK,
requestlog.ActionTooLarge, sizeLimit, true},
{"not announced, at the limit", sizeLimit, false, http.StatusOK,
requestlog.ActionForward, sizeLimit, false},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
answerWithSize(w, tc.size, tc.announced)
})
addr, out := startProxy(t, app.URL, map[string]string{
responseMaxBytes: sizeLimitSetting,
})
got := get(t, addr, "/download")
wantStatus(t, got, tc.want)
if len(got.body) != tc.received ||
errors.Is(got.err, io.ErrUnexpectedEOF) != tc.cutOff {
t.Errorf("client got %d bytes (%v), want %d",
len(got.body), got.err, tc.received)
}
line := out.requestLine(t)
wantLine(t, line, tc.want, tc.action)
if line.UpstreamStatus != http.StatusOK {
t.Errorf("log line has upstream_status %d", line.UpstreamStatus)
}
})
}
}
// answerWithSize answers with a body of size bytes, announced in
// Content-Length or sent in chunks with no length given.
func answerWithSize(w http.ResponseWriter, size int, announced bool) {
body := make([]byte, size)
if announced {
w.Header().Set("Content-Length", strconv.Itoa(size))
_, _ = w.Write(body)
return
}
// Sending part of it before the end keeps Go's server from working
// out the length.
_, _ = w.Write(body[:size/2])
_ = http.NewResponseController(w).Flush()
_, _ = w.Write(body[size/2:])
}
+421
View File
@@ -0,0 +1,421 @@
package proxy_test
import (
"bufio"
"bytes"
"errors"
"io"
"net/http"
"slices"
"strings"
"sync/atomic"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// A request target with an escaped slash and space in its path, and a
// query with a parameter ReverseProxy cannot parse.
const (
rawPath = "/some%2Fpath/with%20space"
rawQuery = "b=2&a=1&bad=%zz;x"
)
// chunkSize is the size of each part of a body a test sends in parts.
const chunkSize = 1 << 10
var errNotStreamed = errors.New("the first part never reached the app")
// appSaw is what the app received.
type appSaw struct {
method string
target string
header http.Header
body []byte
}
func TestPassesRequestAndAnswerUnchanged(t *testing.T) {
t.Parallel()
requestBody := bytes.Repeat([]byte("request body "), 8000)
answerBody := bytes.Repeat([]byte("answer body "), 8000)
saw := make(chan appSaw, 1)
app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
saw <- appSaw{r.Method, r.RequestURI, r.Header.Clone(), body}
w.Header().Set("X-App", "yes")
w.Header().Add("Set-Cookie", "a=1")
w.Header().Add("Set-Cookie", "b=2")
w.WriteHeader(http.StatusTeapot)
_, _ = w.Write(answerBody)
})
addr, out := startProxy(t, app.URL, nil)
req := newRequest(t, http.MethodPatch, addr, rawPath+"?"+rawQuery,
bytes.NewReader(requestBody))
req.Header.Add("X-Test", "one")
req.Header.Add("X-Test", "two")
req.Header.Set("User-Agent", "test-agent")
got := do(t, req)
wantAppSaw(t, <-saw, requestBody)
wantAnswer(t, got, answerBody)
line := out.requestLine(t)
wantLine(t, line, http.StatusTeapot, requestlog.ActionForward)
wantRequestFields(t, line, addr, len(requestBody), len(answerBody))
}
// wantAppSaw checks that the app received the test's request unchanged.
func wantAppSaw(t *testing.T, saw appSaw, body []byte) {
t.Helper()
if saw.method != http.MethodPatch || saw.target != rawPath+"?"+rawQuery {
t.Errorf("app saw %s %s, want %s %s", saw.method, saw.target,
http.MethodPatch, rawPath+"?"+rawQuery)
}
if !slices.Equal(saw.header.Values("X-Test"), []string{"one", "two"}) {
t.Errorf("app saw X-Test %q", saw.header.Values("X-Test"))
}
if saw.header.Get("User-Agent") != "test-agent" {
t.Errorf("app saw User-Agent %q", saw.header.Get("User-Agent"))
}
if !bytes.Equal(saw.body, body) {
t.Errorf("app saw a body of %d bytes, want the %d sent",
len(saw.body), len(body))
}
}
// wantAnswer checks that the client received the app's answer unchanged.
func wantAnswer(t *testing.T, got answer, body []byte) {
t.Helper()
wantStatus(t, got, http.StatusTeapot)
if got.header.Get("X-App") != "yes" {
t.Errorf("client got X-App %q", got.header.Get("X-App"))
}
if !slices.Equal(got.header.Values("Set-Cookie"), []string{"a=1", "b=2"}) {
t.Errorf("client got Set-Cookie %q", got.header.Values("Set-Cookie"))
}
if got.err != nil || !bytes.Equal(got.body, body) {
t.Errorf("client got %d bytes (%v), want the %d the app sent",
len(got.body), got.err, len(body))
}
}
// wantRequestFields checks the log line's fields about the request.
func wantRequestFields(t *testing.T, line logLine, host string, sent, received int) {
t.Helper()
want := requestlog.Line{
Type: "request", Time: line.Time, ClientIP: localhost, PeerIP: localhost,
Method: http.MethodPatch, Host: host, Path: rawPath, Query: rawQuery,
Protocol: "HTTP/1.1", Status: http.StatusTeapot,
UpstreamStatus: http.StatusTeapot, RequestBytes: int64(sent),
ResponseBytes: int64(received), UserAgent: "test-agent",
Action: requestlog.ActionForward, DurationTotal: line.DurationTotal,
DurationUpstreamTotal: line.DurationUpstreamTotal,
}
if line.Line != want {
t.Errorf("log line\n%+v\nwant\n%+v", line.Line, want)
}
_, err := time.Parse(time.RFC3339, line.Time)
if err != nil || line.DurationTotal <= 0 || line.DurationUpstreamTotal <= 0 {
t.Errorf("log line has time %q and durations %v and %v",
line.Time, line.DurationTotal, line.DurationUpstreamTotal)
}
}
func TestStreamsTheRequestBody(t *testing.T) {
t.Parallel()
chunk := bytes.Repeat([]byte("x"), chunkSize)
firstArrived := make(chan struct{})
app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
first := make([]byte, len(chunk))
_, err := io.ReadFull(r.Body, first)
if err != nil {
return
}
close(firstArrived)
rest, _ := io.ReadAll(r.Body)
_, _ = w.Write(rest)
})
addr, _ := startProxy(t, app.URL, nil)
body, writer := io.Pipe()
go func() {
_, _ = writer.Write(chunk)
select {
case <-firstArrived:
_, _ = writer.Write(chunk)
_ = writer.Close()
case <-time.After(waitLimit):
_ = writer.CloseWithError(errNotStreamed)
}
}()
got := do(t, newRequest(t, http.MethodPost, addr, "/upload", body))
if got.err != nil || !bytes.Equal(got.body, chunk) {
t.Errorf("app read %d bytes after the first part (%v), want %d",
len(got.body), got.err, len(chunk))
}
}
func TestStreamsTheAnswerBody(t *testing.T) {
t.Parallel()
chunk := bytes.Repeat([]byte("y"), chunkSize)
firstArrived := make(chan struct{})
app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write(chunk)
_ = http.NewResponseController(w).Flush()
select {
case <-firstArrived:
_, _ = w.Write(chunk)
case <-time.After(waitLimit):
}
})
addr, _ := startProxy(t, app.URL, nil)
req := newRequest(t, http.MethodGet, addr, "/download", http.NoBody)
res, err := newClient(t).Do(req)
if err != nil {
t.Fatalf("request: %v", err)
}
first := make([]byte, len(chunk))
_, err = io.ReadFull(res.Body, first)
close(firstArrived)
got := readAnswer(res)
if err != nil || got.err != nil || !bytes.Equal(got.body, chunk) {
t.Errorf("client read %d bytes after the first part (%v, %v), want %d",
len(got.body), err, got.err, len(chunk))
}
}
func TestUpgradedConnectionOutlastsTheTimeouts(t *testing.T) {
t.Parallel()
app := startApp(t, echoAfterUpgrade)
addr, out := startProxy(t, app.URL, map[string]string{
clientRequestTimeout: shortTimeoutSetting,
clientResponseTimeout: shortTimeoutSetting,
upstreamRequestTimeout: shortTimeoutSetting,
upstreamResponseTimeout: shortTimeoutSetting,
})
conn := dial(t, addr)
send(t, conn, "GET /socket HTTP/1.1\r\nHost: app\r\n"+
"Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n")
reader := bufio.NewReader(conn)
res, err := http.ReadResponse(reader, nil)
if err != nil {
t.Fatalf("read the answer to the upgrade: %v", err)
}
answered := time.Now()
_ = res.Body.Close()
if res.StatusCode != http.StatusSwitchingProtocols {
t.Fatalf("status %d, want %d", res.StatusCode, http.StatusSwitchingProtocols)
}
// Every timeout started before the upgrade was answered, the response
// timeouts last, at the end of the request: wait until just past
// shortTimeout after the answer was read, then use the connection.
time.Sleep(time.Until(answered.Add(shortTimeout + 100*time.Millisecond)))
send(t, conn, "still here\n")
echoed, err := reader.ReadString('\n')
if err != nil || echoed != "still here\n" {
t.Errorf("echo %q (%v), want %q", echoed, err, "still here\n")
}
_ = conn.Close()
wantLine(t, out.requestLine(t), http.StatusSwitchingProtocols,
requestlog.ActionForward)
}
// echoAfterUpgrade is an app that switches protocols on request, and then
// sends back each line it receives.
func echoAfterUpgrade(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Upgrade") != "websocket" {
http.Error(w, "not an upgrade", http.StatusBadRequest)
return
}
conn, buffered, err := http.NewResponseController(w).Hijack()
if err != nil {
return
}
defer func() {
_ = conn.Close()
}()
_, _ = buffered.WriteString("HTTP/1.1 101 Switching Protocols\r\n" +
"Connection: Upgrade\r\nUpgrade: websocket\r\n\r\n")
_ = buffered.Flush()
for {
line, err := buffered.ReadString('\n')
if err != nil {
return
}
_, _ = buffered.WriteString(line)
_ = buffered.Flush()
}
}
func TestServerHasTheFixedLimits(t *testing.T) {
t.Parallel()
cfg, err := config.FromEnvironment(func(string) (string, bool) { return "", false })
if err != nil {
t.Fatalf("default settings: %v", err)
}
server := proxy.New(proxy.Params{
Config: cfg,
RequestLog: io.Discard,
ProcessLog: requestlog.NewProcessLogger(io.Discard),
})
if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 ||
server.IdleTimeout != 2*time.Minute || server.ReadHeaderTimeout != time.Minute {
t.Errorf("server listens on %q with header limit %d, idle time %s and "+
"header timeout %s", server.Addr, server.MaxHeaderBytes,
server.IdleTimeout, server.ReadHeaderTimeout)
}
}
func TestRefusesHeadersOver32KiB(t *testing.T) {
t.Parallel()
var calls atomic.Int32
app := startApp(t, func(http.ResponseWriter, *http.Request) {
calls.Add(1)
})
addr, _ := startProxy(t, app.URL, nil)
// size counts every byte of the request: the request line, the
// headers and the blank line that ends them.
const (
start = "GET / HTTP/1.1\r\nHost: app\r\nX-Large: "
end = "\r\n\r\n"
)
for _, tc := range []struct {
size int
want int
}{
{size: 32 << 10, want: http.StatusOK},
{size: 32<<10 + 1, want: http.StatusRequestHeaderFieldsTooLarge},
} {
conn := dial(t, addr)
send(t, conn, start+strings.Repeat("a", tc.size-len(start)-len(end))+end)
wantStatus(t, readResponse(t, conn), tc.want)
}
if calls.Load() != 1 {
t.Errorf("the app was called %d times, want once", calls.Load())
}
}
func TestAnswers502WhenTheAppCannotBeReached(t *testing.T) {
t.Parallel()
// No test can listen on port 1: listening on port 0 gets one from 32768 up.
addr, out := startProxy(t, "http://"+localhost+":1", nil)
wantStatus(t, get(t, addr, "/"), http.StatusBadGateway)
wantLine(t, out.requestLine(t), http.StatusBadGateway,
requestlog.ActionUpstreamError)
logged := slices.ContainsFunc(out.lines(t), func(line map[string]any) bool {
return line["type"] == "process" && line["msg"] == "request to the app failed"
})
if !logged {
t.Errorf("no process line says the request to the app failed")
}
}
func TestLogsAnAnswerThatBrokeOff(t *testing.T) {
t.Parallel()
app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
_, _ = io.WriteString(w, "the first part")
_ = http.NewResponseController(w).Flush()
panic(http.ErrAbortHandler) // drops the connection mid-answer
})
addr, out := startProxy(t, app.URL, nil)
got := get(t, addr, "/")
if string(got.body) != "the first part" || !errors.Is(got.err, io.ErrUnexpectedEOF) {
t.Errorf("client read %q (%v), want the first part cut off", got.body, got.err)
}
wantLine(t, out.requestLine(t), http.StatusOK, requestlog.ActionUpstreamError)
}
func TestLogsAClientThatWentAway(t *testing.T) {
t.Parallel()
arrived := make(chan struct{})
app := startApp(t, func(_ http.ResponseWriter, r *http.Request) {
close(arrived)
<-r.Context().Done()
})
addr, out := startProxy(t, app.URL, nil)
conn := dial(t, addr)
send(t, conn, "GET /slow HTTP/1.1\r\nHost: app\r\n\r\n")
select {
case <-arrived:
case <-time.After(waitLimit):
t.Fatal("the request never reached the app")
}
_ = conn.Close()
line := out.requestLine(t)
if !line.Aborted || line.Status != 0 || line.Action != requestlog.ActionForward {
t.Errorf("log line has aborted %v, status %d and action %q, "+
"want true, 0 and %q",
line.Aborted, line.Status, line.Action, requestlog.ActionForward)
}
}
+136
View File
@@ -0,0 +1,136 @@
// Package proxy passes each request to the app and the app's answer back,
// unchanged, within the size and time limits, and writes one request log
// line for each request.
package proxy
import (
"io"
"log"
"log/slog"
"net/http"
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// The request line and headers a client may send, and how long a
// kept-open client connection may wait for its next request, are fixed
// rather than settings. The limit on the request line and headers is
// 32 KiB, but Go's server reads 4 KiB past its MaxHeaderBytes before it
// refuses, so MaxHeaderBytes is set 4 KiB lower. The idle time is longer
// than the 90 seconds after which traefik closes a connection it is not
// using, so traefik never sends a request on a connection smallwebwaf is
// closing.
const (
requestHeaderMaxBytes = 32<<10 - 4<<10
clientIdleTimeout = 120 * time.Second
)
// How smallwebwaf keeps connections to the app open between requests.
const (
appIdleConns = 100
appIdleConnTimeout = 90 * time.Second
)
// HealthPath is smallwebwaf's health endpoint, which the container's
// health check asks.
const HealthPath = "/_smallwebwaf/healthz"
// Params are what New needs.
type Params struct {
Config *config.Config
// RequestLog receives one JSON line per request.
RequestLog io.Writer
// ProcessLog receives the process's own messages.
ProcessLog *slog.Logger
// GeoJSURL is where clients' countries are looked up, normally
// lookup.URL. GeoJS is asked only while a country list is set.
GeoJSURL string
}
// New returns the server smallwebwaf runs: each request it reads passes
// through the proxy. Go's server itself refuses headers over 32 KiB, with
// 431, closes a connection idle for 120 seconds, and applies
// SWWAF_CLIENT_REQUEST_TIMEOUT while the headers arrive; the proxy
// applies the timeouts and size limits from then on.
func New(params Params) *http.Server {
errorLog := slog.NewLogLogger(params.ProcessLog.Handler(), slog.LevelWarn)
return &http.Server{
Addr: params.Config.ListenAddr,
Handler: &handler{
config: params.Config,
requestLog: params.RequestLog,
processLog: params.ProcessLog,
errorLog: errorLog,
transport: newTransport(),
limiter: ratelimit.New(ratelimit.Limits{
PerMinute: params.Config.RateLimitPerMinute,
PerHour: params.Config.RateLimitPerHour,
PerDay: params.Config.RateLimitPerDay,
}),
geojs: lookup.New(lookup.Params{
URL: params.GeoJSURL,
Now: time.Now,
ProcessLog: params.ProcessLog,
}),
},
ReadHeaderTimeout: params.Config.ClientRequestTimeout,
IdleTimeout: clientIdleTimeout,
MaxHeaderBytes: requestHeaderMaxBytes,
ErrorLog: errorLog,
}
}
// handler is the proxy. It holds what every request shares; what belongs
// to one request is in a request.
type handler struct {
config *config.Config
requestLog io.Writer
processLog *slog.Logger
errorLog *log.Logger
transport http.RoundTripper
limiter *ratelimit.Limiter
geojs *lookup.GeoJS
}
// newTransport returns what carries requests to the app. It never goes
// through a proxy named in the environment, and leaves the app's answers
// compressed or not as the app sent them.
func newTransport() *http.Transport {
return &http.Transport{
MaxIdleConns: appIdleConns,
MaxIdleConnsPerHost: appIdleConns,
IdleConnTimeout: appIdleConnTimeout,
DisableCompression: true,
}
}
// ServeHTTP handles one request: it works out the client, runs the
// checks, passes the request to the app and the answer back within the
// limits, and writes the request's log line.
func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
rq := h.newRequest(w, r)
defer rq.finish()
// The health endpoint is answered at once, before any check, so that
// a health checker is never refused. It does not ask the app.
if r.Method == http.MethodGet && r.URL.Path == HealthPath {
rq.line.Action = requestlog.ActionAdmin
_, _ = io.WriteString(rq.out, "ok\n")
return
}
refused := rq.check(r.Context())
if refused != nil {
rq.answer(*refused)
return
}
rq.forward(r.Context())
}
+352
View File
@@ -0,0 +1,352 @@
package proxy_test
import (
"bufio"
"bytes"
"encoding/json"
"io"
"maps"
"net"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
const (
// shortTimeout is what a test sets a timeout to, to see it run out.
// It starts before the test has set up its case, such as an upgrade
// or the app's buffers filling, so it is as long as the hold-up of the
// test process that wantTimedOut allows: a shorter one can run out
// first on a busy host.
shortTimeout = waitLimit / 2
// longTimeoutSetting is a timeout that does not run out in a test.
longTimeoutSetting = "1m"
// waitLimit bounds how long a test waits for what should happen.
waitLimit = 10 * time.Second
// pollInterval is how often a test looks for a log line.
pollInterval = 10 * time.Millisecond
// localhost is where every test server listens, and so the address
// smallwebwaf sees each test's requests come from.
localhost = "127.0.0.1"
)
// shortTimeoutSetting is shortTimeout as a setting's value.
//
//nolint:gochecknoglobals // a constant cannot call String
var shortTimeoutSetting = shortTimeout.String()
// The settings the tests set.
const (
clientRequestTimeout = "SWWAF_CLIENT_REQUEST_TIMEOUT"
clientResponseTimeout = "SWWAF_CLIENT_RESPONSE_TIMEOUT"
upstreamRequestTimeout = "SWWAF_UPSTREAM_REQUEST_TIMEOUT"
upstreamResponseTimeout = "SWWAF_UPSTREAM_RESPONSE_TIMEOUT"
requestMaxBytes = "SWWAF_REQUEST_MAX_BYTES"
responseMaxBytes = "SWWAF_RESPONSE_MAX_BYTES"
trustedProxies = "SWWAF_TRUSTED_PROXIES"
allowNets = "SWWAF_ALLOW_NETS"
rateLimitExemptNets = "SWWAF_RATE_LIMIT_EXEMPT_NETS"
denyNets = "SWWAF_DENY_NETS"
rateLimitPerMinute = "SWWAF_RATE_LIMIT_PER_MINUTE"
deniedCountries = "SWWAF_DENIED_COUNTRIES"
allowedCountries = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
)
// output collects what smallwebwaf writes on stdout.
type output struct {
mu sync.Mutex
buf bytes.Buffer
}
// Write adds lines smallwebwaf writes.
func (o *output) Write(p []byte) (int, error) {
o.mu.Lock()
defer o.mu.Unlock()
return o.buf.Write(p)
}
// lines returns every line written so far, decoded.
func (o *output) lines(t *testing.T) []map[string]any {
t.Helper()
o.mu.Lock()
defer o.mu.Unlock()
var lines []map[string]any
for text := range strings.Lines(o.buf.String()) {
var line map[string]any
err := json.Unmarshal([]byte(text), &line)
if err != nil {
t.Fatalf("output line %q is not JSON: %v", text, err)
}
lines = append(lines, line)
}
return lines
}
// logLine is a request log line, as typed fields and as the JSON object
// it was written as.
type logLine struct {
requestlog.Line
fields map[string]any
}
// requestLines waits for count request log lines and returns them.
func (o *output) requestLines(t *testing.T, count int) []logLine {
t.Helper()
deadline := time.Now().Add(waitLimit)
for time.Now().Before(deadline) {
var found []logLine
for _, fields := range o.lines(t) {
if fields["type"] == "request" {
found = append(found, decodeLine(t, fields))
}
}
if len(found) >= count {
return found
}
time.Sleep(pollInterval)
}
t.Fatalf("fewer than %d request log lines after %s", count, waitLimit)
return nil
}
// requestLine waits for the request log line of a test's one request.
func (o *output) requestLine(t *testing.T) logLine {
t.Helper()
return o.requestLines(t, 1)[0]
}
// decodeLine reads a request log line's fields into a logLine.
func decodeLine(t *testing.T, fields map[string]any) logLine {
t.Helper()
encoded, err := json.Marshal(fields)
if err != nil {
t.Fatalf("encode %v: %v", fields, err)
}
line := logLine{fields: fields}
err = json.Unmarshal(encoded, &line.Line)
if err != nil {
t.Fatalf("decode %s: %v", encoded, err)
}
return line
}
// startApp starts app as the app smallwebwaf passes requests to.
func startApp(t *testing.T, app http.HandlerFunc) *httptest.Server {
t.Helper()
server := httptest.NewServer(app)
t.Cleanup(server.Close)
return server
}
// startProxy starts smallwebwaf in front of the app at appURL, with the
// settings in env on top of the defaults, and returns where it listens and
// what it writes.
func startProxy(t *testing.T, appURL string, env map[string]string) (string, *output) {
t.Helper()
return startProxyWithGeoJS(t, appURL, "", env)
}
// startProxyWithGeoJS is startProxy with clients' countries looked up at
// geojsURL.
func startProxyWithGeoJS(
t *testing.T, appURL, geojsURL string, env map[string]string,
) (string, *output) {
t.Helper()
settings := map[string]string{"SWWAF_UPSTREAM_URL": appURL}
maps.Copy(settings, env)
cfg, err := config.FromEnvironment(func(name string) (string, bool) {
value, ok := settings[name]
return value, ok
})
if err != nil {
t.Fatalf("settings %v: %v", settings, err)
}
out := &output{}
server := proxy.New(proxy.Params{
Config: cfg,
RequestLog: out,
ProcessLog: requestlog.NewProcessLogger(out),
GeoJSURL: geojsURL,
})
listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0")
if err != nil {
t.Fatalf("listen: %v", err)
}
go func() {
_ = server.Serve(listener)
}()
t.Cleanup(func() {
_ = server.Close()
})
return listener.Addr().String(), out
}
// newClient returns an HTTP client that sends requests as they are made,
// with no compression of its own.
func newClient(t *testing.T) *http.Client {
t.Helper()
transport := &http.Transport{DisableCompression: true}
t.Cleanup(transport.CloseIdleConnections)
return &http.Client{Transport: transport}
}
// answer is a response as a test reads it: the status, the headers, as
// much of the body as arrived, and the error that ended the reading, nil
// when the whole body arrived.
type answer struct {
status int
header http.Header
body []byte
err error
}
// readAnswer reads all of res, and closes its body.
func readAnswer(res *http.Response) answer {
body, err := io.ReadAll(res.Body)
_ = res.Body.Close()
return answer{status: res.StatusCode, header: res.Header, body: body, err: err}
}
// newRequest makes a request for path to smallwebwaf at addr.
func newRequest(t *testing.T, method, addr, path string, body io.Reader) *http.Request {
t.Helper()
req, err := http.NewRequestWithContext(t.Context(), method, "http://"+addr+path, body)
if err != nil {
t.Fatalf("new request: %v", err)
}
return req
}
// do sends req and reads the answer.
func do(t *testing.T, req *http.Request) answer {
t.Helper()
res, err := newClient(t).Do(req)
if err != nil {
t.Fatalf("%s %s: %v", req.Method, req.URL.Path, err)
}
return readAnswer(res)
}
// get sends a GET request for path to smallwebwaf at addr.
func get(t *testing.T, addr, path string) answer {
t.Helper()
return do(t, newRequest(t, http.MethodGet, addr, path, http.NoBody))
}
// dial opens a connection to smallwebwaf at addr, for requests the HTTP
// client cannot make, such as one that stops sending halfway.
func dial(t *testing.T, addr string) net.Conn {
t.Helper()
conn, err := (&net.Dialer{}).DialContext(t.Context(), "tcp", addr)
if err != nil {
t.Fatalf("dial %s: %v", addr, err)
}
t.Cleanup(func() {
_ = conn.Close()
})
return conn
}
// send writes text to conn.
func send(t *testing.T, conn net.Conn, text string) {
t.Helper()
_, err := io.WriteString(conn, text)
if err != nil {
t.Fatalf("send: %v", err)
}
}
// readResponse reads the answer to a request sent on conn.
func readResponse(t *testing.T, conn net.Conn) answer {
t.Helper()
err := conn.SetReadDeadline(time.Now().Add(waitLimit))
if err != nil {
t.Fatalf("set read deadline: %v", err)
}
res, err := http.ReadResponse(bufio.NewReader(conn), nil)
if err != nil {
t.Fatalf("read response: %v", err)
}
return readAnswer(res)
}
// wantLine checks the request log line's status and action.
func wantLine(t *testing.T, line logLine, status int, action string) {
t.Helper()
if line.Status != status || line.Action != action {
t.Errorf("log line has status %d and action %q, want %d and %q",
line.Status, line.Action, status, action)
}
}
// wantStatus checks an answer's status.
func wantStatus(t *testing.T, got answer, status int) {
t.Helper()
if got.status != status {
t.Errorf("status %d, want %d", got.status, status)
}
}
// wantTimedOut checks that what began at start ended once shortTimeout
// had run out, and not much later.
func wantTimedOut(t *testing.T, start time.Time) {
t.Helper()
took := time.Since(start)
if took < shortTimeout || took > shortTimeout+waitLimit/2 {
t.Errorf("took %s, want %s", took, shortTimeout)
}
}
+67
View File
@@ -0,0 +1,67 @@
package proxy_test
import (
"net/http"
"sync/atomic"
"testing"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
func TestRateLimitRefusesWith429BeforeTheApp(t *testing.T) {
t.Parallel()
var calls atomic.Int32
app := startApp(t, func(http.ResponseWriter, *http.Request) {
calls.Add(1)
})
addr, out := startProxy(t, app.URL, map[string]string{
trustedProxies: trustLocalhost,
rateLimitPerMinute: "1",
})
const otherClient = "203.0.113.10"
// With a limit of one request a minute, a client's second request is
// refused. A client is one IPv4 address, or one IPv6 /64; an IPv4
// address in IPv6 form is that IPv4 address.
requests := []struct {
client string // as X-Forwarded-For names it
logged string // as the log line's client_ip names it
want int
}{
{client, client, http.StatusOK},
{client, client, http.StatusTooManyRequests},
{otherClient, otherClient, http.StatusOK},
{"::ffff:" + otherClient, otherClient, http.StatusTooManyRequests},
{"2001:db8::1", "2001:db8::1", http.StatusOK},
{"2001:db8::8000:0:0:1", "2001:db8::8000:0:0:1", http.StatusTooManyRequests},
{"2001:db8:0:1::1", "2001:db8:0:1::1", http.StatusOK},
}
for i, sent := range requests {
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
req.Header.Set(forwardedFor, sent.client)
wantStatus(t, do(t, req), sent.want)
line := out.requestLines(t, i+1)[i]
if line.ClientIP != sent.logged {
t.Errorf("log line has client_ip %q, want %q", line.ClientIP, sent.logged)
}
if sent.want == http.StatusOK {
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
} else {
wantLine(t, line, http.StatusTooManyRequests, requestlog.ActionRateLimited)
if line.LimitHit != "minute" {
t.Errorf("log line has limit_hit %q, want minute", line.LimitHit)
}
}
}
if calls.Load() != 4 {
t.Errorf("the app was called %d times, want 4", calls.Load())
}
}
+488
View File
@@ -0,0 +1,488 @@
package proxy
import (
"context"
"errors"
"net/http"
"net/http/httptrace"
"net/http/httputil"
"net/netip"
"os"
"sync"
"sync/atomic"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// flushAfterEachWrite has ReverseProxy pass on each part of the app's
// answer as soon as it arrives.
const flushAfterEachWrite time.Duration = -1
// refusal is smallwebwaf refusing a request, or refusing to go on with it:
// the status the client is answered if the response has not started yet,
// and the action the log line names.
type refusal struct {
status int
action string
}
// request is one request on its way through smallwebwaf, from the moment
// its headers have been read to its log line.
type request struct {
h *handler
in *http.Request
// rc sets the deadlines of the connection to the client.
rc *http.ResponseController
out *responseWriter
body *requestBody // nil for a request without a body
line requestlog.Line
client netip.Addr
peer netip.Addr
peerTrusted bool
start time.Time
// upstreamStart is when the request was handed to the app.
upstreamStart time.Time
// cancel ends the request to the app.
cancel context.CancelFunc
// refused is the first refusal, from whichever goroutine meets it.
refused atomic.Pointer[refusal]
// complete is true once the app's whole answer has been passed on.
complete bool
// mu guards what follows. The timeouts run on goroutines of their
// own, and the transport starts and stops them from its own; once
// timersStopped is set, none of them acts any more.
mu sync.Mutex
timersStopped bool
clientRequestTimer *time.Timer
upstreamRequestTimer *time.Timer
upstreamResponseTimer *time.Timer
// requestSent is when the app had been sent the whole request.
requestSent time.Time
}
// newRequest starts handling r: it notes the time and works out the
// client.
func (h *handler) newRequest(w http.ResponseWriter, r *http.Request) *request {
start := time.Now()
peer := peerAddress(r)
trusted := h.config.TrustedProxies
client := clientAddress(peer, r.Header.Values("X-Forwarded-For"), trusted)
rq := &request{
h: h,
in: r,
rc: http.NewResponseController(w),
out: &responseWriter{ResponseWriter: w},
client: client,
peer: peer,
peerTrusted: isInside(peer, trusted),
start: start,
line: requestlog.Line{
Time: requestlog.FormatTime(start),
ClientIP: client.String(),
PeerIP: peer.String(),
Method: r.Method,
Host: r.Host,
Path: r.URL.EscapedPath(),
Query: r.URL.RawQuery,
Protocol: r.Proto,
Referer: r.Referer(),
UserAgent: r.UserAgent(),
Action: requestlog.ActionForward,
},
}
if r.Body != http.NoBody {
rq.body = &requestBody{body: limitBody(r.Body, h.config.RequestMaxBytes), rq: rq}
}
return rq
}
// check is the one place where a request can be refused once its client
// is known, before its body is read or anything reaches the app. It
// returns nil to let the request through. A client in SWWAF_ALLOW_NETS
// skips every check but the size limit. For any other client,
// SWWAF_DENY_NETS comes first, so that a client it refuses is not looked
// up, and then the country lists; a request either refuses is not counted
// for the rate limits. Then come the rate limits, unless the client is in
// SWWAF_RATE_LIMIT_EXEMPT_NETS, so that every other request is counted,
// one refused for its size too. ctx is the request's own context.
func (rq *request) check(ctx context.Context) *refusal {
cfg := rq.h.config
allowed := isInside(rq.client, cfg.AllowNets)
if !allowed && isInside(rq.client, cfg.DenyNets) {
return &refusal{
status: http.StatusForbidden,
action: requestlog.ActionDenied,
}
}
if !allowed && rq.countryDenied(ctx) {
return &refusal{
status: http.StatusForbidden,
action: requestlog.ActionCountryDenied,
}
}
if !allowed && !isInside(rq.client, cfg.RateLimitExemptNets) {
limitHit := rq.h.limiter.Count(clientGroup(rq.client), rq.start)
if limitHit != "" {
rq.line.LimitHit = limitHit
return &refusal{
status: http.StatusTooManyRequests,
action: requestlog.ActionRateLimited,
}
}
}
maxBytes := cfg.RequestMaxBytes
if maxBytes > 0 && rq.in.ContentLength > maxBytes {
return &refusal{
status: http.StatusRequestEntityTooLarge,
action: requestlog.ActionTooLarge,
}
}
return nil
}
// forward passes the request to the app and the app's answer back. ctx
// is the request's own context.
func (rq *request) forward(ctx context.Context) {
ctx, cancel := context.WithCancel(ctx)
defer cancel()
rq.cancel = cancel
ctx = httptrace.WithClientTrace(ctx, &httptrace.ClientTrace{
WroteRequest: rq.wroteRequest,
})
out := rq.in.WithContext(ctx)
if rq.body != nil {
out.Body = rq.body
}
reverseProxy := &httputil.ReverseProxy{
Rewrite: rq.rewrite,
Transport: rq.h.transport,
FlushInterval: flushAfterEachWrite,
ErrorLog: rq.h.errorLog,
ModifyResponse: rq.modifyResponse,
ErrorHandler: rq.answerError,
}
rq.startRequestTimers()
rq.upstreamStart = time.Now()
reverseProxy.ServeHTTP(rq.out, out)
}
// rewrite makes the request the app receives: the client's request,
// unchanged, sent to SWWAF_UPSTREAM_URL, with the forwarded headers set.
func (rq *request) rewrite(pr *httputil.ProxyRequest) {
upstream := rq.h.config.UpstreamURL
pr.Out.URL.Scheme = upstream.Scheme
pr.Out.URL.Host = upstream.Host
// ReverseProxy drops query parameters it cannot parse; the app gets
// the query as the client sent it.
pr.Out.URL.RawQuery = pr.In.URL.RawQuery
setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted)
}
// modifyResponse looks at the app's answer before ReverseProxy passes it
// on.
func (rq *request) modifyResponse(res *http.Response) error {
rq.line.UpstreamStatus = res.StatusCode
if res.StatusCode == http.StatusSwitchingProtocols {
// An upgraded connection, such as a WebSocket, is not cut by the
// timeouts. ReverseProxy writes this answer straight to the
// connection it takes over, not through rq.out.
rq.stopTimers()
rq.out.status = res.StatusCode
return nil
}
maxBytes := rq.h.config.ResponseMaxBytes
if maxBytes > 0 && res.Body != http.NoBody && res.ContentLength > maxBytes {
rq.refuse(refusal{status: http.StatusBadGateway, action: requestlog.ActionTooLarge})
return errResponseTooLarge
}
res.Body = &responseBody{body: limitBody(res.Body, maxBytes), rq: rq}
rq.startClientResponseTimeout()
return nil
}
// answerError is ReverseProxy's ErrorHandler: the request could not be
// passed to the app, or the app's answer cannot be passed on.
func (rq *request) answerError(_ http.ResponseWriter, _ *http.Request, err error) {
refused := rq.refused.Load()
if refused == nil {
if rq.in.Context().Err() != nil {
return // the client has gone, and there is no one to answer
}
rq.h.processLog.Warn("request to the app failed", "error", err.Error())
refused = &refusal{
status: http.StatusBadGateway,
action: requestlog.ActionUpstreamError,
}
}
rq.answer(*refused)
}
// answer sends smallwebwaf's own answer, unless the response has already
// started, and records the refusal for the log line.
func (rq *request) answer(r refusal) {
rq.refused.CompareAndSwap(nil, &r)
if rq.out.status != 0 {
return // too late to answer: the connection can only be cut
}
// A client found too slow is read no more; any other may go on
// sending until its time is up, so that Go's server can read the
// rest of the body and end the request cleanly.
deadline := rq.clientRequestDeadline()
if r.status == http.StatusRequestTimeout {
deadline = time.Now()
}
rq.stopReadingBody(deadline)
timeout := rq.h.config.ClientResponseTimeout
if timeout > 0 {
_ = rq.rc.SetWriteDeadline(time.Now().Add(timeout))
}
http.Error(rq.out, http.StatusText(r.status), r.status)
}
// refuse records r, unless an earlier refusal was, and ends the request
// to the app.
func (rq *request) refuse(r refusal) {
rq.refused.CompareAndSwap(nil, &r)
rq.cancel()
}
// finish ends the request's timeouts and writes its log line.
func (rq *request) finish() {
rq.stopTimers()
refused := rq.refused.Load()
if refused == nil {
rq.stopReadingBody(rq.clientRequestDeadline())
}
line := &rq.line
line.Status = rq.out.status
line.ResponseBytes = rq.out.bytes
if rq.body != nil {
line.RequestBytes = rq.body.bytes.Load()
}
switch {
case refused != nil:
line.Action = refused.action
case errors.Is(rq.out.err, os.ErrDeadlineExceeded):
// The client took longer than SWWAF_CLIENT_RESPONSE_TIMEOUT to
// take the response.
line.Action = requestlog.ActionTimedOut
case !rq.complete && (rq.out.err != nil || rq.in.Context().Err() != nil):
line.Aborted = true
}
now := time.Now()
line.DurationTotal = requestlog.Milliseconds(now.Sub(rq.start))
if !rq.upstreamStart.IsZero() {
line.DurationUpstreamTotal = requestlog.Milliseconds(now.Sub(rq.upstreamStart))
}
err := requestlog.Write(rq.h.requestLog, line)
if err != nil {
rq.h.processLog.Error("writing the request log failed", "error", err.Error())
}
}
// clientRequestDeadline is when the client must have sent its whole
// request, or zero when SWWAF_CLIENT_REQUEST_TIMEOUT is off.
func (rq *request) clientRequestDeadline() time.Time {
timeout := rq.h.config.ClientRequestTimeout
if timeout == 0 {
return time.Time{}
}
return rq.start.Add(timeout)
}
// stopReadingBody ends, at deadline, the reading of a client body that has
// not arrived whole: Go's server then reads no more of it, and closes the
// connection after the answer.
func (rq *request) stopReadingBody(deadline time.Time) {
if rq.body == nil || rq.body.received.Load() {
return
}
_ = rq.rc.SetReadDeadline(deadline)
}
// startRequestTimers starts the timeouts that run while the request goes
// to the app: SWWAF_CLIENT_REQUEST_TIMEOUT until the client has sent its
// whole body, and SWWAF_UPSTREAM_REQUEST_TIMEOUT until the app has been
// sent the whole request.
func (rq *request) startRequestTimers() {
rq.mu.Lock()
defer rq.mu.Unlock()
if rq.body != nil && rq.h.config.ClientRequestTimeout > 0 {
rq.clientRequestTimer = time.AfterFunc(
time.Until(rq.clientRequestDeadline()), rq.requestTimedOut)
}
timeout := rq.h.config.UpstreamRequestTimeout
if timeout > 0 {
rq.upstreamRequestTimer = time.AfterFunc(timeout, rq.requestTimedOut)
}
}
// requestTimedOut is called when a request timeout runs out while the
// request is still on its way to the app. The answer names the side
// smallwebwaf was waiting on at that moment: 408 when it was waiting for
// the client to send more of its body, 504 when it was waiting for the
// app to be reached or to take what it had.
func (rq *request) requestTimedOut() {
rq.mu.Lock()
defer rq.mu.Unlock()
if rq.timersStopped {
return
}
if rq.body == nil || !rq.body.waiting.Load() {
rq.refuse(refusal{
status: http.StatusGatewayTimeout,
action: requestlog.ActionTimedOut,
})
return
}
rq.refuse(refusal{
status: http.StatusRequestTimeout,
action: requestlog.ActionTimedOut,
})
// The transport gives up on the app only once its Read of the
// client's body returns, so that Read is ended now. The lock keeps
// this from reaching the connection after the request is handled.
_ = rq.rc.SetReadDeadline(time.Now())
}
// bodyReceived is called once the client has sent its whole body.
func (rq *request) bodyReceived() {
rq.mu.Lock()
defer rq.mu.Unlock()
stopTimer(rq.clientRequestTimer)
}
// wroteRequest is called once the app has been sent the whole request:
// the request timeouts end and SWWAF_UPSTREAM_RESPONSE_TIMEOUT starts.
func (rq *request) wroteRequest(info httptrace.WroteRequestInfo) {
if info.Err != nil {
return // the transport gives up, or tries again
}
rq.mu.Lock()
defer rq.mu.Unlock()
if rq.timersStopped {
return
}
stopTimer(rq.clientRequestTimer)
stopTimer(rq.upstreamRequestTimer)
rq.requestSent = time.Now()
timeout := rq.h.config.UpstreamResponseTimeout
if timeout > 0 {
rq.upstreamResponseTimer = time.AfterFunc(timeout, rq.responseTimedOut)
}
}
// responseTimedOut is called when SWWAF_UPSTREAM_RESPONSE_TIMEOUT runs out
// before the app has sent its whole answer.
func (rq *request) responseTimedOut() {
rq.mu.Lock()
defer rq.mu.Unlock()
if !rq.timersStopped {
rq.refuse(refusal{
status: http.StatusGatewayTimeout,
action: requestlog.ActionTimedOut,
})
}
}
// responseReceived is called once the app has sent its whole answer.
func (rq *request) responseReceived() {
rq.complete = true
rq.stopTimers()
}
// startClientResponseTimeout sets SWWAF_CLIENT_RESPONSE_TIMEOUT on the
// connection to the client: the response must reach the client within it
// of the end of the request, or of now if the app answers before it has
// the whole request.
func (rq *request) startClientResponseTimeout() {
timeout := rq.h.config.ClientResponseTimeout
if timeout == 0 {
return
}
from := rq.sentAt()
if from.IsZero() {
from = time.Now()
}
_ = rq.rc.SetWriteDeadline(from.Add(timeout))
}
// sentAt is when the app had been sent the whole request, or zero.
func (rq *request) sentAt() time.Time {
rq.mu.Lock()
defer rq.mu.Unlock()
return rq.requestSent
}
// stopTimers stops the request's timeouts and keeps any from starting
// later: the app's answer is complete, the connection upgraded, or the
// request handled.
func (rq *request) stopTimers() {
rq.mu.Lock()
defer rq.mu.Unlock()
rq.timersStopped = true
stopTimer(rq.clientRequestTimer)
stopTimer(rq.upstreamRequestTimer)
stopTimer(rq.upstreamResponseTimer)
}
// stopTimer stops t, which is nil when its timeout is off.
func stopTimer(t *time.Timer) {
if t != nil {
t.Stop()
}
}
+184
View File
@@ -0,0 +1,184 @@
package proxy_test
import (
"net/http"
"strings"
"sync/atomic"
"testing"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// The rate limits count an IPv6 client by its /64, so these two addresses
// are one client for them. The static lists match each address on its own,
// and the tests list listedAddr alone.
const (
listedAddr = "2001:db8::1"
unlistedAddr = "2001:db8::2"
)
func TestAllowNetsSkipEveryCheckButTheSizeLimit(t *testing.T) {
t.Parallel()
var calls atomic.Int32
app := startApp(t, func(http.ResponseWriter, *http.Request) {
calls.Add(1)
})
geojsURL, asked := startGeoJS(t)
// fromKP is in SWWAF_ALLOW_NETS, and in SWWAF_DENY_NETS too, which
// comes after it.
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{
trustedProxies: trustLocalhost,
allowNets: "198.51.100.0/24",
denyNets: fromKP,
deniedCountries: "kp",
rateLimitPerMinute: "1",
requestMaxBytes: "1K",
})
// Neither SWWAF_DENY_NETS, the country lists nor the limit of one
// request a minute refuses the client, and its country is not looked
// up.
wantAnswers(t, addr, out, []sentRequest{
{fromKP, http.StatusOK, requestlog.ActionForward},
{fromKP, http.StatusOK, requestlog.ActionForward},
})
if len(asked()) != 0 {
t.Errorf("GeoJS was asked about %v, want nothing", asked())
}
// The size limit still applies.
body := strings.NewReader(strings.Repeat("a", 2<<10))
req := newRequest(t, http.MethodPost, addr, "/", body)
req.Header.Set(forwardedFor, fromKP)
wantStatus(t, do(t, req), http.StatusRequestEntityTooLarge)
wantLine(t, out.requestLines(t, 3)[2],
http.StatusRequestEntityTooLarge, requestlog.ActionTooLarge)
if calls.Load() != 2 {
t.Errorf("the app was called %d times, want 2", calls.Load())
}
}
func TestRequestFromAllowNetsIsNotCounted(t *testing.T) {
t.Parallel()
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
addr, out := startProxy(t, app.URL, map[string]string{
trustedProxies: trustLocalhost,
allowNets: listedAddr,
rateLimitPerMinute: "1",
})
// listedAddr's requests are not counted, so the first request from
// unlistedAddr is within the limit of one a minute.
wantAnswers(t, addr, out, []sentRequest{
{listedAddr, http.StatusOK, requestlog.ActionForward},
{listedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusTooManyRequests, requestlog.ActionRateLimited},
})
}
func TestDenyNetsRefuseBeforeTheLookupAndTheBody(t *testing.T) {
t.Parallel()
var calls atomic.Int32
app := startApp(t, func(http.ResponseWriter, *http.Request) {
calls.Add(1)
})
geojsURL, asked := startGeoJS(t)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{
trustedProxies: trustLocalhost,
denyNets: "203.0.113.0/24",
deniedCountries: "kp",
})
req := newRequest(t, http.MethodPost, addr, "/", strings.NewReader("a body"))
req.Header.Set(forwardedFor, fromDE)
wantStatus(t, do(t, req), http.StatusForbidden)
line := out.requestLine(t)
wantLine(t, line, http.StatusForbidden, requestlog.ActionDenied)
if line.RequestBytes != 0 {
t.Errorf("log line has request_bytes %d, want 0", line.RequestBytes)
}
if len(asked()) != 0 {
t.Errorf("GeoJS was asked about %v, want nothing", asked())
}
if calls.Load() != 0 {
t.Errorf("the app was called %d times, want none", calls.Load())
}
}
func TestRequestRefusedByDenyNetsIsNotCounted(t *testing.T) {
t.Parallel()
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
addr, out := startProxy(t, app.URL, map[string]string{
trustedProxies: trustLocalhost,
denyNets: listedAddr,
rateLimitPerMinute: "1",
})
// listedAddr's refused requests are not counted, so the first request
// from unlistedAddr is within the limit of one a minute.
wantAnswers(t, addr, out, []sentRequest{
{listedAddr, http.StatusForbidden, requestlog.ActionDenied},
{listedAddr, http.StatusForbidden, requestlog.ActionDenied},
{unlistedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusTooManyRequests, requestlog.ActionRateLimited},
})
}
func TestRateLimitExemptNetsAreNeitherCountedNorRefused(t *testing.T) {
t.Parallel()
app := startApp(t, func(http.ResponseWriter, *http.Request) {})
geojsURL, _ := startGeoJS(t)
addr, out := startProxyWithGeoJS(t, app.URL, geojsURL, map[string]string{
trustedProxies: trustLocalhost,
rateLimitExemptNets: listedAddr + "," + fromKP,
deniedCountries: "kp",
rateLimitPerMinute: "1",
})
// listedAddr's requests are neither refused nor counted, so the first
// request from unlistedAddr is within the limit of one a minute. The
// country lists still refuse an exempt client.
wantAnswers(t, addr, out, []sentRequest{
{listedAddr, http.StatusOK, requestlog.ActionForward},
{listedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusOK, requestlog.ActionForward},
{unlistedAddr, http.StatusTooManyRequests, requestlog.ActionRateLimited},
{fromKP, http.StatusForbidden, requestlog.ActionCountryDenied},
})
}
// sentRequest is a GET request from client, as X-Forwarded-For names it,
// and the status and log line action it should get.
type sentRequest struct {
client string
status int
action string
}
// wantAnswers sends requests to smallwebwaf at addr one after another and
// checks each one's answer and log line. They must be the first requests
// smallwebwaf is sent, since the log lines are matched to them in order.
func wantAnswers(t *testing.T, addr string, out *output, requests []sentRequest) {
t.Helper()
for i, sent := range requests {
req := newRequest(t, http.MethodGet, addr, "/", http.NoBody)
req.Header.Set(forwardedFor, sent.client)
wantStatus(t, do(t, req), sent.status)
wantLine(t, out.requestLines(t, i+1)[i], sent.status, sent.action)
}
}
+275
View File
@@ -0,0 +1,275 @@
package proxy_test
import (
"errors"
"io"
"net"
"net/http"
"net/http/httptest"
"strconv"
"sync"
"sync/atomic"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// largeBodySize is more than the connections between the client,
// smallwebwaf and the app can hold while nobody reads, so that a sender
// soon waits.
const largeBodySize = 64 << 20
// writeSize is how much a test sender writes at a time.
const writeSize = 32 << 10
func TestRequestTimeouts(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
env map[string]string
// appTakesNothing has the app never read, while the client sends
// as fast as it can; otherwise the app reads, and the client
// stops sending halfway.
appTakesNothing bool
want int
}{
{
name: "client request timeout, waiting on the client",
env: map[string]string{clientRequestTimeout: shortTimeoutSetting},
want: http.StatusRequestTimeout,
},
{
name: "upstream request timeout, waiting on the client",
env: map[string]string{
upstreamRequestTimeout: shortTimeoutSetting,
clientRequestTimeout: longTimeoutSetting,
},
want: http.StatusRequestTimeout,
},
{
name: "upstream request timeout, waiting on the app",
env: map[string]string{upstreamRequestTimeout: shortTimeoutSetting},
appTakesNothing: true,
want: http.StatusGatewayTimeout,
},
{
name: "client request timeout, waiting on the app",
env: map[string]string{
clientRequestTimeout: shortTimeoutSetting,
upstreamRequestTimeout: longTimeoutSetting,
},
appTakesNothing: true,
want: http.StatusGatewayTimeout,
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var (
app *httptest.Server
appURL string
sendRequest func(*testing.T, string) net.Conn
appGotBody atomic.Bool
)
if tc.appTakesNothing {
appURL, sendRequest = startAppThatTakesNothing(t), sendLargeBody
} else {
app = startApp(t, func(_ http.ResponseWriter, r *http.Request) {
n, _ := io.Copy(io.Discard, r.Body)
appGotBody.Store(n > 0)
})
appURL, sendRequest = app.URL, sendPartOfBody
}
addr, out := startProxy(t, appURL, tc.env)
start := time.Now()
got := readResponse(t, sendRequest(t, addr))
wantTimedOut(t, start)
want := tc.want
if app != nil {
// Close returns once the app has finished with the request.
app.Close()
// Until some of the body has reached the app, smallwebwaf
// waits on the app, and SPEC.md asks for 504; the timeout
// runs out then only if the test process is held up.
if !appGotBody.Load() {
want = http.StatusGatewayTimeout
}
}
wantStatus(t, got, want)
wantLine(t, out.requestLine(t), want, requestlog.ActionTimedOut)
})
}
}
// startAppThatTakesNothing starts an app that accepts connections and
// never reads from them, and returns its URL.
func startAppThatTakesNothing(t *testing.T) string {
t.Helper()
listener, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0")
if err != nil {
t.Fatalf("listen: %v", err)
}
var (
mu sync.Mutex
held []net.Conn
)
hold := func(conn net.Conn) {
mu.Lock()
defer mu.Unlock()
held = append(held, conn)
}
go func() {
for {
conn, err := listener.Accept()
if err != nil {
return
}
hold(conn)
}
}()
t.Cleanup(func() {
_ = listener.Close()
mu.Lock()
defer mu.Unlock()
for _, conn := range held {
_ = conn.Close()
}
})
return "http://" + listener.Addr().String()
}
// sendPartOfBody sends a request that announces a large body, and only
// the first bytes of it.
func sendPartOfBody(t *testing.T, addr string) net.Conn {
t.Helper()
conn := dial(t, addr)
send(t, conn, "POST /upload HTTP/1.1\r\nHost: app\r\nContent-Length: "+
strconv.Itoa(largeBodySize)+"\r\n\r\nthe first bytes")
return conn
}
// sendLargeBody sends a request with a large body, as fast as smallwebwaf
// takes it, from a goroutine of its own.
func sendLargeBody(t *testing.T, addr string) net.Conn {
t.Helper()
conn := dial(t, addr)
send(t, conn, "POST /upload HTTP/1.1\r\nHost: app\r\nContent-Length: "+
strconv.Itoa(largeBodySize)+"\r\n\r\n")
go func() {
chunk := make([]byte, writeSize)
for range largeBodySize / writeSize {
_, err := conn.Write(chunk)
if err != nil {
return
}
}
}()
return conn
}
func TestAppTooSlowToAnswer(t *testing.T) {
t.Parallel()
app := startApp(t, func(_ http.ResponseWriter, r *http.Request) {
<-r.Context().Done()
})
addr, out := startProxy(t, app.URL, map[string]string{
upstreamResponseTimeout: shortTimeoutSetting,
})
start := time.Now()
wantStatus(t, get(t, addr, "/slow"), http.StatusGatewayTimeout)
wantTimedOut(t, start)
line := out.requestLine(t)
wantLine(t, line, http.StatusGatewayTimeout, requestlog.ActionTimedOut)
_, answered := line.fields["upstream_status"]
if answered {
t.Errorf("log line has upstream_status %v for an app that never answered",
line.fields["upstream_status"])
}
}
func TestAppTooSlowToFinishItsAnswer(t *testing.T) {
t.Parallel()
app := startApp(t, func(w http.ResponseWriter, r *http.Request) {
_, _ = io.WriteString(w, "the first part")
_ = http.NewResponseController(w).Flush()
<-r.Context().Done()
})
addr, out := startProxy(t, app.URL, map[string]string{
upstreamResponseTimeout: shortTimeoutSetting,
})
start := time.Now()
got := get(t, addr, "/slow")
wantStatus(t, got, http.StatusOK)
if string(got.body) != "the first part" || !errors.Is(got.err, io.ErrUnexpectedEOF) {
t.Errorf("client read %q (%v), want the first part cut off", got.body, got.err)
}
wantTimedOut(t, start)
line := out.requestLine(t)
wantLine(t, line, http.StatusOK, requestlog.ActionTimedOut)
if line.UpstreamStatus != http.StatusOK {
t.Errorf("log line has upstream_status %d, want %d",
line.UpstreamStatus, http.StatusOK)
}
}
func TestClientTooSlowToTakeTheAnswer(t *testing.T) {
t.Parallel()
app := startApp(t, func(w http.ResponseWriter, _ *http.Request) {
chunk := make([]byte, writeSize)
for range largeBodySize / writeSize {
_, err := w.Write(chunk)
if err != nil {
return
}
}
})
addr, out := startProxy(t, app.URL, map[string]string{
clientResponseTimeout: shortTimeoutSetting,
})
start := time.Now()
// The client asks, and never reads the answer.
conn := dial(t, addr)
send(t, conn, "GET /large HTTP/1.1\r\nHost: app\r\n\r\n")
line := out.requestLine(t)
wantTimedOut(t, start)
wantLine(t, line, http.StatusOK, requestlog.ActionTimedOut)
}
+132
View File
@@ -0,0 +1,132 @@
// Package ratelimit counts each client's requests over a minute, an hour
// and a day, as the "Counting method" section of SPEC.md describes, and
// tells when a request takes a client over a rate limit. The counts are
// kept in memory only, for at most 20,000 clients.
package ratelimit
import (
"net/netip"
"sync"
"time"
"github.com/hashicorp/golang-lru/v2/simplelru"
)
// maxClients is how many clients are kept. Past it, the least recently
// seen client is dropped, and starts afresh if it comes back.
const maxClients = 20000
const day = 24 * time.Hour
// Limits are the most requests a client may make in a minute, an hour and
// a day. Zero is no limit.
type Limits struct {
PerMinute int64
PerHour int64
PerDay int64
}
// Limiter counts each client's requests against the limits. It is safe
// for concurrent use.
type Limiter struct {
windows [3]window
mu sync.Mutex
// clients holds each client's buckets, one pair for each of windows,
// in the same order.
clients *simplelru.LRU[netip.Prefix, *[3]buckets]
}
// New returns a Limiter for limits, with no client counted yet.
func New(limits Limits) *Limiter {
clients, err := simplelru.NewLRU[netip.Prefix, *[3]buckets](maxClients, nil)
if err != nil {
panic(err) // NewLRU fails only for a size below one
}
return &Limiter{
windows: [3]window{
{name: "minute", length: time.Minute, limit: limits.PerMinute},
{name: "hour", length: time.Hour, limit: limits.PerHour},
{name: "day", length: day, limit: limits.PerDay},
},
clients: clients,
}
}
// Count counts a request from client at now, in every window, whether or
// not it is refused. It returns the window whose limit the request takes
// the client over, "minute", "hour" or "day", the shortest if it is over
// several, or "" if it is within every limit.
func (l *Limiter) Count(client netip.Prefix, now time.Time) string {
l.mu.Lock()
defer l.mu.Unlock()
counts, seen := l.clients.Get(client)
if !seen {
counts = &[3]buckets{}
l.clients.Add(client, counts)
}
limitHit := ""
for i, w := range l.windows {
requests := counts[i].add(now, w.length)
if limitHit == "" && w.limit > 0 && requests > float64(w.limit) {
limitHit = w.name
}
}
return limitHit
}
// window is a length of time over which requests are counted, and the
// most requests a client may make in it.
type window struct {
name string
length time.Duration
limit int64
}
// buckets are a client's two buckets in one window: the requests in the
// bucket under way, which began at start, and in the bucket before it.
type buckets struct {
start time.Time
current int64
previous int64
}
// add counts a request at now in a window of length, and returns the
// client's requests in the window that ends at now: those in the bucket
// under way, and those in the bucket before it weighted by how much of
// that bucket the window still covers.
//
// Concurrent requests can be counted out of order, so now can be a moment
// before the bucket under way began; such a request is counted in that
// bucket. A request dated more than a second before it means the clock
// was set back, and the buckets start afresh: otherwise the bucket before
// would keep its full weight until the clock caught up.
func (b *buckets) add(now time.Time, length time.Duration) float64 {
if now.Before(b.start.Add(-time.Second)) {
*b = buckets{}
}
start := now.Truncate(length)
if start.After(b.start) {
if start.Equal(b.start.Add(length)) {
b.previous = b.current
} else {
b.previous = 0
}
b.start = start
b.current = 0
}
b.current++
elapsed := max(now.Sub(b.start), 0)
covered := 1 - float64(elapsed)/float64(length)
return float64(b.previous)*covered + float64(b.current)
}
+200
View File
@@ -0,0 +1,200 @@
package ratelimit_test
import (
"net/netip"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
)
// limit is the limit the tests set.
const limit = 3
// The windows, as Count names them.
const (
minute = "minute"
hour = "hour"
)
func TestEachWindowRefusesAtItsLimitAndLetsTheClientBack(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
window string
limits ratelimit.Limits
length time.Duration
}{
{minute, ratelimit.Limits{PerMinute: limit}, time.Minute},
{hour, ratelimit.Limits{PerHour: limit}, time.Hour},
{"day", ratelimit.Limits{PerDay: limit}, 24 * time.Hour},
} {
t.Run(tc.window, func(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(tc.limits)
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
quarter := tc.length / 4
for range limit {
wantCount(t, limiter, client, start, "")
}
wantCount(t, limiter, client, start, tc.window)
// A quarter into the next bucket, the window still covers three
// quarters of the bucket before, with its four requests: 3 + 1
// is over the limit.
wantCount(t, limiter, client, start.Add(tc.length+quarter), tc.window)
// Three quarters into it, a quarter: 1 + 2 is within.
wantCount(t, limiter, client, start.Add(tc.length+3*quarter), "")
})
}
}
func TestClientBackAfterAWholeBucketIsWithinTheLimitAtOnce(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
for range limit {
wantCount(t, limiter, client, start, "")
}
wantCount(t, limiter, client, start, hour)
// No request in the whole next bucket, so a quarter into the one after
// it the window covers none of the four requests: 1 is within the
// limit. Were they counted as the bucket before, 3 + 1 would be over.
wantCount(t, limiter, client, start.Add(2*time.Hour+time.Hour/4), "")
}
func TestRefusedRequestsCount(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit, PerHour: 2 * limit})
refused := netip.MustParsePrefix("203.0.113.9/32")
within := netip.MustParsePrefix("203.0.113.10/32")
start := midnight()
for range limit {
wantCount(t, limiter, refused, start, "")
wantCount(t, limiter, within, start, "")
}
for range limit {
wantCount(t, limiter, refused, start, minute)
}
// Half a minute into the next bucket the window covers half of the
// bucket before: 3 + 1 is over the minute's limit for the client
// whose three refused requests count, and 1.5 + 1 within it for the
// other. The first is over the hour's limit too, and the shorter
// window is named.
halfway := start.Add(time.Minute + time.Minute/2)
wantCount(t, limiter, refused, halfway, minute)
wantCount(t, limiter, within, halfway, "")
// The refused requests count in the hour as well: 6 + 1 + 1 is over
// its limit, and 3 + 1 + 1 within it.
later := start.Add(10 * time.Minute)
wantCount(t, limiter, refused, later, hour)
wantCount(t, limiter, within, later, "")
}
func TestRequestCountedLateGoesInTheBucketUnderWay(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerMinute: limit})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
for range limit {
wantCount(t, limiter, client, start, "")
}
// A concurrent request dated a moment before the bucket under way, but
// counted after it began, is counted in it: 3 + 1 is over the limit.
wantCount(t, limiter, client, start.Add(-time.Millisecond), minute)
}
func TestClockSetBackStartsTheBucketsAfresh(t *testing.T) {
t.Parallel()
limiter := ratelimit.New(ratelimit.Limits{PerHour: limit})
client := netip.MustParsePrefix("203.0.113.9/32")
start := midnight()
for range limit {
wantCount(t, limiter, client, start, "")
}
// Half an hour into the next bucket: 3 / 2 + 1 is within the limit.
wantCount(t, limiter, client, start.Add(time.Hour+time.Hour/2), "")
// The clock is set back an hour. Counted in the bucket under way, the
// next request would find the bucket before it at full weight, 3 + 2,
// over the limit until the clock caught up. The buckets start afresh
// instead, and the client is refused only past the limit again.
setBack := start.Add(time.Hour / 2)
for range limit {
wantCount(t, limiter, client, setBack, "")
}
wantCount(t, limiter, client, setBack, hour)
}
func TestKeepsAtMost20000ClientsDroppingTheLeastRecentlySeen(t *testing.T) {
t.Parallel()
const maxClients = 20000
limiter := ratelimit.New(ratelimit.Limits{PerMinute: 1})
now := midnight()
clients := make([]netip.Prefix, maxClients+1)
addr := netip.MustParseAddr("10.0.0.0")
for i := range clients {
clients[i] = netip.PrefixFrom(addr, addr.BitLen())
addr = addr.Next()
}
for _, client := range clients[:maxClients] {
wantCount(t, limiter, client, now, "")
}
// The first client is seen again: its second request is over the
// limit of one, so it is still counted.
wantCount(t, limiter, clients[0], now, minute)
// One client more drops the least recently seen, the second, which
// starts afresh, while the first is kept.
wantCount(t, limiter, clients[maxClients], now, "")
wantCount(t, limiter, clients[1], now, "")
wantCount(t, limiter, clients[0], now, minute)
}
// midnight is the start of a bucket in every window.
func midnight() time.Time {
return time.Date(2026, 10, 4, 0, 0, 0, 0, time.UTC)
}
// wantCount counts a request from client at now, and checks the window
// whose limit it goes over, "" for none.
func wantCount(
t *testing.T, limiter *ratelimit.Limiter, client netip.Prefix, now time.Time,
want string,
) {
t.Helper()
got := limiter.Count(client, now)
if got != want {
t.Errorf("request from %s at %s is over %q, want %q",
client, now.Format(time.RFC3339), got, want)
}
}
+117
View File
@@ -0,0 +1,117 @@
// Package requestlog writes the lines smallwebwaf prints on stdout: one
// JSON object per request, marked "type":"request", and the process's own
// messages as JSON lines marked "type":"process".
package requestlog
import (
"encoding/json"
"fmt"
"io"
"log/slog"
"time"
)
// The action a request line names: what smallwebwaf did with the
// request.
const (
// ActionForward is a request passed to the app.
ActionForward = "forward"
// ActionTooLarge is a request or response over its size limit.
ActionTooLarge = "too_large"
// ActionTimedOut is a request or response that ran out of time.
ActionTimedOut = "timed_out"
// ActionUpstreamError is a request the app could not be reached
// for, or whose answer could not be passed on.
ActionUpstreamError = "upstream_error"
// ActionRateLimited is a request refused because it took its client
// over a rate limit, or came while the client was over one.
ActionRateLimited = "rate_limited"
// ActionDenied is a request refused because its client is in
// SWWAF_DENY_NETS.
ActionDenied = "denied"
// ActionCountryDenied is a request refused for its client's country.
ActionCountryDenied = "country_denied"
// ActionAdmin is a request smallwebwaf answered at one of its own
// endpoints, under /_smallwebwaf/.
ActionAdmin = "admin"
)
// timeLayout is RFC 3339 with milliseconds.
const timeLayout = "2006-01-02T15:04:05.000Z07:00"
// Line is one request's line in the request log. The field names are
// those of the "Request log" section of SPEC.md.
//
//nolint:tagliatelle // SPEC.md's request log names its fields in snake_case
type Line struct {
Type string `json:"type"`
Time string `json:"time"`
ClientIP string `json:"client_ip"`
PeerIP string `json:"peer_ip"`
Country string `json:"country"`
Method string `json:"method"`
Host string `json:"host"`
Path string `json:"path"`
Query string `json:"query"`
Protocol string `json:"protocol"`
Status int `json:"status"`
UpstreamStatus int `json:"upstream_status,omitempty"`
RequestBytes int64 `json:"request_bytes"`
ResponseBytes int64 `json:"response_bytes"`
Referer string `json:"referer"`
UserAgent string `json:"user_agent"`
Action string `json:"action"`
// LimitHit is the window whose rate limit the request went over:
// minute, hour or day.
LimitHit string `json:"limit_hit,omitempty"`
// Aborted is true when the client went away early.
Aborted bool `json:"aborted,omitempty"`
// DurationTotal and DurationUpstreamTotal are in milliseconds.
DurationTotal float64 `json:"duration_total"`
DurationUpstreamTotal float64 `json:"duration_upstream_total,omitempty"`
}
// Write writes line to w as one JSON line marked "type":"request".
func Write(w io.Writer, line *Line) error {
line.Type = "request"
encoded, err := json.Marshal(line)
if err != nil {
return fmt.Errorf("encode the request log line: %w", err)
}
_, err = w.Write(append(encoded, '\n'))
if err != nil {
return fmt.Errorf("write the request log line: %w", err)
}
return nil
}
// FormatTime formats t for a line's time field: RFC 3339 in UTC, with
// milliseconds.
func FormatTime(t time.Time) string {
return t.UTC().Format(timeLayout)
}
// Milliseconds is d in milliseconds, to the microsecond.
func Milliseconds(d time.Duration) float64 {
return float64(d.Microseconds()) / float64(time.Millisecond/time.Microsecond)
}
// NewProcessLogger returns the logger for the process's own messages:
// JSON lines on w, marked "type":"process", with the time in the same form
// as a request line's.
func NewProcessLogger(w io.Writer) *slog.Logger {
handler := slog.NewJSONHandler(w, &slog.HandlerOptions{
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
if attr.Key == slog.TimeKey && len(groups) == 0 {
return slog.String(slog.TimeKey, FormatTime(attr.Value.Time()))
}
return attr
},
})
return slog.New(handler).With("type", "process")
}
+90
View File
@@ -0,0 +1,90 @@
package requestlog_test
import (
"bytes"
"encoding/json"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
func TestWriteWritesOneJSONLineMarkedRequest(t *testing.T) {
t.Parallel()
var out bytes.Buffer
err := requestlog.Write(&out, &requestlog.Line{
Time: requestlog.FormatTime(time.Date(2026, 10, 3, 12, 0, 0, 0, time.UTC)),
ClientIP: "203.0.113.9",
Status: 200,
Action: requestlog.ActionForward,
DurationTotal: requestlog.Milliseconds(1500 * time.Microsecond),
})
if err != nil {
t.Fatalf("write: %v", err)
}
text := out.String()
if strings.Count(text, "\n") != 1 || !strings.HasSuffix(text, "\n") {
t.Fatalf("wrote %q, want one line", text)
}
var fields map[string]any
err = json.Unmarshal(out.Bytes(), &fields)
if err != nil {
t.Fatalf("decode %q: %v", text, err)
}
want := map[string]any{
"type": "request", "time": "2026-10-03T12:00:00.000Z",
"client_ip": "203.0.113.9", "status": 200.0, "action": "forward",
"duration_total": 1.5,
}
for name, value := range want {
if fields[name] != value {
t.Errorf("%s is %v, want %v", name, fields[name], value)
}
}
unset := []string{
"upstream_status", "limit_hit", "aborted", "duration_upstream_total",
}
for _, name := range unset {
_, present := fields[name]
if present {
t.Errorf("%s is there with no value to give", name)
}
}
}
func TestProcessLinesAreMarkedProcess(t *testing.T) {
t.Parallel()
var out bytes.Buffer
requestlog.NewProcessLogger(&out).Info("starting", "version", "v1")
var fields map[string]any
err := json.Unmarshal(out.Bytes(), &fields)
if err != nil {
t.Fatalf("decode %q: %v", out.String(), err)
}
if fields["type"] != "process" || fields["msg"] != "starting" ||
fields["level"] != "INFO" || fields["version"] != "v1" {
t.Errorf("process line %v", fields)
}
timeText, _ := fields["time"].(string)
logged, err := time.Parse(time.RFC3339, timeText)
if err != nil || !strings.HasSuffix(timeText, "Z") ||
len(timeText) != len("2006-01-02T15:04:05.000Z") ||
time.Since(logged) > time.Minute {
t.Errorf("process line time %q, want now in UTC with milliseconds", timeText)
}
}
+100
View File
@@ -0,0 +1,100 @@
package smallwebwaf
import (
"context"
"errors"
"fmt"
"io"
"net"
"net/http"
"net/url"
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/proxy"
)
// healthCheckTimeout bounds the whole health check.
const healthCheckTimeout = 5 * time.Second
var errHealthEndpoint = errors.New("smallwebwaf's health endpoint answered")
// HealthCheck is the container's health check. It returns 0 while
// smallwebwaf answers its health endpoint on 127.0.0.1, at the port in
// SWWAF_LISTEN_ADDR, and the app accepts connections at the address in
// SWWAF_UPSTREAM_URL. Otherwise it writes why to stderr and returns 1.
// args are the arguments after `healthcheck`; it takes none, and given
// one it names it on stderr and returns 1 without checking anything.
func HealthCheck(
ctx context.Context, args []string, lookupEnv func(string) (string, bool),
stderr io.Writer,
) int {
if len(args) > 0 {
_, _ = fmt.Fprintf(stderr,
"smallwebwaf healthcheck: unexpected argument %q\n", args[0])
return 1
}
err := healthCheck(ctx, lookupEnv)
if err != nil {
_, _ = fmt.Fprintln(stderr, "unhealthy:", err)
return 1
}
return 0
}
func healthCheck(ctx context.Context, lookupEnv func(string) (string, bool)) error {
ctx, cancel := context.WithTimeout(ctx, healthCheckTimeout)
defer cancel()
cfg, err := config.FromEnvironment(lookupEnv)
if err != nil {
return fmt.Errorf("invalid setting: %w", err)
}
// The settings have checked that the address has a port.
_, port, _ := net.SplitHostPort(cfg.ListenAddr)
health := "http://" + net.JoinHostPort("127.0.0.1", port) + proxy.HealthPath
req, err := http.NewRequestWithContext(ctx, http.MethodGet, health, http.NoBody)
if err != nil {
return fmt.Errorf("make the request: %w", err)
}
res, err := http.DefaultClient.Do(req)
if err != nil {
return fmt.Errorf("ask smallwebwaf: %w", err)
}
_ = res.Body.Close()
if res.StatusCode != http.StatusOK {
return fmt.Errorf("%w %s", errHealthEndpoint, res.Status)
}
conn, err := (&net.Dialer{}).DialContext(ctx, "tcp", appAddress(cfg.UpstreamURL))
if err != nil {
return fmt.Errorf("connect to the app: %w", err)
}
_ = conn.Close()
return nil
}
// appAddress is the host and port of the app's URL, the port being the
// scheme's own when the URL names none.
func appAddress(app *url.URL) string {
port := app.Port()
if port == "" {
port = "80"
if app.Scheme == "https" {
port = "443"
}
}
return net.JoinHostPort(app.Hostname(), port)
}
@@ -0,0 +1,27 @@
package smallwebwaf
import (
"net/url"
"testing"
)
func TestAppAddress(t *testing.T) {
t.Parallel()
for app, want := range map[string]string{
"http://127.0.0.1:8081": "127.0.0.1:8081",
"http://app": "app:80",
"https://app/": "app:443",
"https://[::1]": "[::1]:443",
} {
parsed, err := url.Parse(app)
if err != nil {
t.Fatalf("parse %q: %v", app, err)
}
got := appAddress(parsed)
if got != want {
t.Errorf("appAddress(%q) is %q, want %q", app, got, want)
}
}
}
+97
View File
@@ -0,0 +1,97 @@
package smallwebwaf_test
import (
"bytes"
"context"
"net"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/smallwebwaf"
)
func TestHealthCheck(t *testing.T) {
t.Parallel()
app := httptest.NewServer(http.NotFoundHandler())
defer app.Close()
ctx, stop := context.WithCancel(t.Context())
defer stop()
out := &output{}
exited := make(chan int, 1)
go func() {
exited <- run(ctx, map[string]string{
listenAddr: localhost + ":0",
upstreamURL: app.URL,
}, out)
}()
addr, _ := out.line(t, "msg", "starting")["address"].(string)
_, port, _ := net.SplitHostPort(addr)
// The container's settings: an address to listen on with an empty
// host part, which the health check asks at 127.0.0.1.
env := map[string]string{listenAddr: ":" + port, upstreamURL: app.URL}
wantHealthCheck(t, env, 0, "")
app.Close()
wantHealthCheck(t, env, 1, "unhealthy: connect to the app: ")
stop()
select {
case <-exited:
case <-time.After(waitLimit):
t.Fatal("still running after being told to stop")
}
wantHealthCheck(t, env, 1, "unhealthy: ask smallwebwaf: ")
wantHealthCheck(t, map[string]string{listenAddr: "8080"}, 1,
"unhealthy: invalid setting: SWWAF_LISTEN_ADDR: ")
}
func TestHealthCheckRefusesAnArgument(t *testing.T) {
t.Parallel()
var stderr bytes.Buffer
noSettings := func(string) (string, bool) {
return "", false
}
got := smallwebwaf.HealthCheck(t.Context(), []string{"now"}, noSettings, &stderr)
want := "smallwebwaf healthcheck: unexpected argument \"now\"\n"
if got != 1 || stderr.String() != want {
t.Errorf("health check returned %d and wrote %q, want 1 and %q",
got, stderr.String(), want)
}
}
// wantHealthCheck runs the health check with the settings in env, and
// checks its exit status and the start of what it writes to stderr,
// which is nothing when message is empty.
func wantHealthCheck(t *testing.T, env map[string]string, status int, message string) {
t.Helper()
var stderr bytes.Buffer
got := smallwebwaf.HealthCheck(t.Context(), nil, func(name string) (string, bool) {
value, ok := env[name]
return value, ok
}, &stderr)
wrote := stderr.String()
if got != status || !strings.HasPrefix(wrote, message) ||
(message == "" && wrote != "") {
t.Errorf("health check returned %d and wrote %q, want %d and %q",
got, wrote, status, message)
}
}
+137
View File
@@ -0,0 +1,137 @@
// Package smallwebwaf runs the smallwebwaf process: it reads the settings,
// serves requests until it is told to stop, and then stops in an orderly
// way.
package smallwebwaf
import (
"context"
"errors"
"io"
"log/slog"
"net"
"net/http"
"os"
"os/signal"
"syscall"
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/proxy"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// shutdownTimeout is how long requests in progress may take to finish
// once smallwebwaf is told to stop, before their connections are closed.
// runit and docker wait a little longer before they kill the process.
const shutdownTimeout = 5 * time.Second
// Params are what Run needs from the process.
type Params struct {
// Version is the version of the binary, set when it is built.
Version string
// LookupEnv reads an environment variable, normally os.LookupEnv.
LookupEnv func(string) (string, bool)
// Stdout receives the request log and the process's own messages.
Stdout io.Writer
}
// Main runs smallwebwaf until SIGTERM or SIGINT, and returns the
// process's exit status. Run as `smallwebwaf healthcheck`, it is the
// container's health check instead.
func Main(version string) int {
if len(os.Args) > 1 && os.Args[1] == "healthcheck" {
return HealthCheck(context.Background(), os.Args[2:], os.LookupEnv, os.Stderr)
}
ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGTERM, os.Interrupt)
defer stop()
return Run(ctx, Params{
Version: version,
LookupEnv: os.LookupEnv,
Stdout: os.Stdout,
})
}
// Run reads the settings, then serves requests until ctx is done. It
// returns the process's exit status, 1 when smallwebwaf cannot start.
func Run(ctx context.Context, params Params) int {
processLog := requestlog.NewProcessLogger(params.Stdout)
cfg, err := config.FromEnvironment(params.LookupEnv)
if err != nil {
processLog.Error("invalid setting", "error", err.Error())
return 1
}
listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", cfg.ListenAddr)
if err != nil {
processLog.Error("cannot listen on SWWAF_LISTEN_ADDR",
"error", err.Error())
return 1
}
server := proxy.New(proxy.Params{
Config: cfg,
RequestLog: params.Stdout,
ProcessLog: processLog,
GeoJSURL: lookup.URL,
})
processLog.Info("starting",
"version", params.Version,
"address", listener.Addr().String(),
"settings", cfg)
return serve(ctx, server, listener, processLog)
}
// serve serves requests on listener until ctx is done, then gives the
// requests in progress shutdownTimeout to finish.
func serve(
ctx context.Context, server *http.Server, listener net.Listener,
processLog *slog.Logger,
) int {
served := make(chan error, 1)
go func() {
served <- server.Serve(listener)
}()
select {
case err := <-served:
processLog.Error("serving failed", "error", err.Error())
return 1
case <-ctx.Done():
}
processLog.Info("stopping")
shutdownCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx),
shutdownTimeout)
defer cancel()
err := server.Shutdown(shutdownCtx)
if err != nil {
processLog.Warn("requests still in progress were cut off",
"error", err.Error())
_ = server.Close()
}
err = <-served
if !errors.Is(err, http.ErrServerClosed) {
processLog.Error("serving failed", "error", err.Error())
return 1
}
processLog.Info("stopped")
return 0
}
+234
View File
@@ -0,0 +1,234 @@
package smallwebwaf_test
import (
"bytes"
"context"
"encoding/json"
"io"
"net"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/smallwebwaf"
)
const (
// waitLimit bounds how long a test waits for what should happen.
waitLimit = 10 * time.Second
// pollInterval is how often a test looks for a line.
pollInterval = 10 * time.Millisecond
// testVersion is the version the tests give smallwebwaf.
testVersion = "test"
// localhost is where the tests listen.
localhost = "127.0.0.1"
listenAddr = "SWWAF_LISTEN_ADDR"
upstreamURL = "SWWAF_UPSTREAM_URL"
)
// output collects what smallwebwaf writes on stdout.
type output struct {
mu sync.Mutex
buf bytes.Buffer
}
// Write adds lines smallwebwaf writes.
func (o *output) Write(p []byte) (int, error) {
o.mu.Lock()
defer o.mu.Unlock()
return o.buf.Write(p)
}
// line returns the first line whose field key is value, waiting for it.
func (o *output) line(t *testing.T, key, value string) map[string]any {
t.Helper()
deadline := time.Now().Add(waitLimit)
for time.Now().Before(deadline) {
o.mu.Lock()
text := o.buf.String()
o.mu.Unlock()
for line := range strings.Lines(text) {
var fields map[string]any
err := json.Unmarshal([]byte(line), &fields)
if err != nil {
t.Fatalf("output line %q is not JSON: %v", line, err)
}
if fields[key] == value {
return fields
}
}
time.Sleep(pollInterval)
}
t.Fatalf("no line with %s %q in the output:\n%s", key, value, o.buf.String())
return nil
}
// run runs smallwebwaf with the settings in env until ctx is done, and
// returns its exit status.
func run(ctx context.Context, env map[string]string, out *output) int {
return smallwebwaf.Run(ctx, smallwebwaf.Params{
Version: testVersion,
LookupEnv: func(name string) (string, bool) {
value, ok := env[name]
return value, ok
},
Stdout: out,
})
}
func TestInvalidSettingStopsTheStart(t *testing.T) {
t.Parallel()
out := &output{}
status := run(t.Context(), map[string]string{"SWWAF_REQUEST_MAX_BYTES": "lots"}, out)
if status != 1 {
t.Errorf("exit status %d, want 1", status)
}
line := out.line(t, "msg", "invalid setting")
message, _ := line["error"].(string)
if line["type"] != "process" || line["level"] != "ERROR" ||
!strings.HasPrefix(message, "SWWAF_REQUEST_MAX_BYTES: ") {
t.Errorf("start refused with %v", line)
}
}
func TestAddressInUseStopsTheStart(t *testing.T) {
t.Parallel()
taken, err := (&net.ListenConfig{}).Listen(t.Context(), "tcp", localhost+":0")
if err != nil {
t.Fatalf("listen: %v", err)
}
defer func() {
_ = taken.Close()
}()
out := &output{}
status := run(t.Context(), map[string]string{listenAddr: taken.Addr().String()}, out)
if status != 1 {
t.Errorf("exit status %d, want 1", status)
}
out.line(t, "msg", "cannot listen on SWWAF_LISTEN_ADDR")
}
func TestServesUntilToldToStop(t *testing.T) {
t.Parallel()
app := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
_, _ = io.WriteString(w, "hello from the app")
}))
defer app.Close()
ctx, stop := context.WithCancel(t.Context())
out := &output{}
exited := make(chan int, 1)
go func() {
exited <- run(ctx, map[string]string{
listenAddr: localhost + ":0",
upstreamURL: app.URL,
}, out)
}()
starting := out.line(t, "msg", "starting")
wantStartingLine(t, starting, app.URL)
addr, _ := starting["address"].(string)
wantGreeting(t, "http://"+addr+"/")
out.line(t, "type", "request")
stop()
select {
case status := <-exited:
if status != 0 {
t.Errorf("exit status %d, want 0", status)
}
case <-time.After(waitLimit):
t.Fatal("still running after being told to stop")
}
out.line(t, "msg", "stopped")
}
// wantStartingLine checks that the line at start gives the version and
// every setting's value.
func wantStartingLine(t *testing.T, line map[string]any, appURL string) {
t.Helper()
settings, _ := line["settings"].(map[string]any)
want := map[string]any{
listenAddr: localhost + ":0",
upstreamURL: appURL,
"SWWAF_TRUSTED_PROXIES": "10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
"SWWAF_CLIENT_REQUEST_TIMEOUT": "60s",
"SWWAF_CLIENT_RESPONSE_TIMEOUT": "30m",
"SWWAF_UPSTREAM_REQUEST_TIMEOUT": "60s",
"SWWAF_UPSTREAM_RESPONSE_TIMEOUT": "30m",
"SWWAF_REQUEST_MAX_BYTES": "100M",
"SWWAF_RESPONSE_MAX_BYTES": "5G",
"SWWAF_ALLOW_NETS": "",
"SWWAF_RATE_LIMIT_EXEMPT_NETS": "",
"SWWAF_DENY_NETS": "",
"SWWAF_RATE_LIMIT_PER_MINUTE": "1000",
"SWWAF_RATE_LIMIT_PER_HOUR": "10000",
"SWWAF_RATE_LIMIT_PER_DAY": "50000",
"SWWAF_DENIED_COUNTRIES": "",
"SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES": "",
}
for name, value := range want {
if settings[name] != value {
t.Errorf("starting line gives %s=%v, want %v", name, settings[name], value)
}
}
if line["version"] != testVersion || line["type"] != "process" {
t.Errorf("starting line %v", line)
}
}
// wantGreeting checks that a request to url gets the app's answer.
func wantGreeting(t *testing.T, url string) {
t.Helper()
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, url,
http.NoBody)
if err != nil {
t.Fatalf("new request: %v", err)
}
transport := &http.Transport{}
defer transport.CloseIdleConnections()
res, err := (&http.Client{Transport: transport}).Do(req)
if err != nil {
t.Fatalf("request: %v", err)
}
body, err := io.ReadAll(res.Body)
_ = res.Body.Close()
if err != nil || string(body) != "hello from the app" {
t.Errorf("got %q (%v), want the app's answer", body, err)
}
}
+6
View File
@@ -0,0 +1,6 @@
{
"license": "MIT",
"devDependencies": {
"prettier": "3.8.1"
}
}
+144
View File
@@ -0,0 +1,144 @@
#!/bin/sh
# script/bootstrap: install all dependencies needed to build and develop
# this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes nothing is present. Node is
# used directly if installed; otherwise it is installed at a pinned
# version via nvm (installing nvm itself first, from a hash-verified
# release archive, never curl | sh). Go comes from the package manager,
# for gofmt in script/fmt and script/fmt-check: the tests and the linter
# run in docker and need no Go on the host.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Pinned versions, 2026-07-06
NODE_VERSION="22.17.0"
NVM_VERSION="0.40.3"
# sha256 of https://github.com/nvm-sh/nvm/archive/refs/tags/v0.40.3.tar.gz
NVM_SHA256="5f4d6aaa04a177dc93c985e31dbc411ab6b8c6e1e21d8015dbc1372625fcd1d0"
YARN_VERSION="1.22.22"
PKGMGR=""
SUDO=""
detect_pkgmgr() {
[ -n "$PKGMGR" ] && return 0
if command -v nix-env >/dev/null 2>&1; then
PKGMGR="nix"
elif command -v apt-get >/dev/null 2>&1; then
PKGMGR="apt"
elif command -v brew >/dev/null 2>&1; then
PKGMGR="brew"
elif command -v apk >/dev/null 2>&1; then
PKGMGR="apk"
else
echo "bootstrap: no supported package manager (nix, apt, brew, apk)" >&2
exit 1
fi
if [ "$PKGMGR" = "apt" ]; then
export DEBIAN_FRONTEND=noninteractive
if [ "$(id -u)" != "0" ]; then
SUDO="sudo"
fi
fi
}
# pkg_install <nix-attr> <apt-pkg> <brew-formula> <apk-pkg>
pkg_install() {
detect_pkgmgr
case "$PKGMGR" in
nix) nix-env -iA "nixpkgs.$1" ;;
apt)
# A fresh CI runner may carry no package lists at all.
$SUDO env DEBIAN_FRONTEND=noninteractive apt-get update -q
$SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y "$2"
;;
brew) brew install "$3" ;;
apk) apk add --no-cache "$4" ;;
esac
}
missing() {
! command -v "$1" >/dev/null 2>&1
}
# verify_sha256 <file> <expected-hash>
verify_sha256() {
if command -v sha256sum >/dev/null 2>&1; then
actual="$(sha256sum "$1" | cut -d' ' -f1)"
else
actual="$(shasum -a 256 "$1" | cut -d' ' -f1)"
fi
if [ "$actual" != "$2" ]; then
echo "bootstrap: sha256 mismatch for $1" >&2
echo " expected: $2" >&2
echo " actual: $actual" >&2
exit 1
fi
}
# nvm is a bash script; run a command in a bash with nvm loaded
nvm_sh() {
bash -c ". \"\$HOME/.nvm/nvm.sh\" && $*"
}
ensure_nvm() {
[ -s "$HOME/.nvm/nvm.sh" ] && return 0
# nvm prerequisites; nvm itself requires bash
if missing bash; then pkg_install bash bash bash bash; fi
if missing curl; then pkg_install curl curl curl curl; fi
if missing git; then pkg_install git git git git; fi
tmp="$(mktemp -d)"
curl -fsSL -o "$tmp/nvm.tar.gz" \
"https://github.com/nvm-sh/nvm/archive/refs/tags/v${NVM_VERSION}.tar.gz"
verify_sha256 "$tmp/nvm.tar.gz" "$NVM_SHA256"
mkdir -p "$HOME/.nvm"
tar -xzf "$tmp/nvm.tar.gz" -C "$HOME/.nvm" --strip-components=1
rm -rf "$tmp"
}
ensure_node() {
if ! missing node; then return 0; fi
ensure_nvm
nvm_sh "nvm install $NODE_VERSION"
}
ensure_yarn() {
if ! missing yarn; then return 0; fi
if ! missing corepack; then
corepack enable
corepack prepare "yarn@$YARN_VERSION" --activate
elif [ -s "$HOME/.nvm/nvm.sh" ]; then
nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
corepack prepare yarn@$YARN_VERSION --activate"
else
npm install -g "yarn@$YARN_VERSION"
fi
}
install_js_deps() {
if missing yarn && [ -s "$HOME/.nvm/nvm.sh" ]; then
nvm_sh "nvm use $NODE_VERSION >/dev/null && cd \"$ROOT\" && \
yarn install --frozen-lockfile"
else
yarn install --frozen-lockfile
fi
}
main() {
cd "$ROOT"
if missing make; then pkg_install gnumake make make make; fi
if missing git; then pkg_install git git git git; fi
if missing curl; then pkg_install curl curl curl curl; fi
if missing gofmt; then pkg_install go golang go go; fi
ensure_node
ensure_yarn
install_js_deps
echo "bootstrap complete"
}
main "$@"
Executable
+18
View File
@@ -0,0 +1,18 @@
#!/bin/sh
# script/build: build bin/smallwebwaf on the host, with Go installed, for
# working on the code by hand. The version it reports comes from git, as
# in script/docker.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
go build -trimpath -ldflags "-X main.Version=$version" \
-o bin/smallwebwaf ./cmd/smallwebwaf
}
main "$@"
Executable
+16
View File
@@ -0,0 +1,16 @@
#!/bin/sh
# script/check: run all checks (test, lint, fmt-check). Our own
# extension to scripts-to-rule-them-all. test and lint are Docker
# phases; fmt-check is native, because a formatter writes the working
# tree. Must not modify any files.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
main() {
"$SCRIPT_DIR/test"
"$SCRIPT_DIR/lint"
"$SCRIPT_DIR/fmt-check"
}
main "$@"
Executable
+28
View File
@@ -0,0 +1,28 @@
#!/bin/sh
# script/cibuild: run the CI build. It bootstraps first: a CI runner
# checks out and runs this and nothing else, and script/fmt-check runs
# the formatter on the host, which a pristine checkout cannot do.
# --no-cache for the same reason as script/docker: the gate phases the
# final stage depends on are RUN steps, and a cached one is a check that
# did not run.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
"$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/check"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
# the version a build stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
}
main "$@"
Executable
+24
View File
@@ -0,0 +1,24 @@
#!/bin/sh
# script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname.
# --no-cache because the gate phases the final stage depends on are RUN
# steps, and a cached one is a check that did not run.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
# the version a build stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
}
main "$@"
+88
View File
@@ -0,0 +1,88 @@
#!/bin/sh
# script/example-app: build the image, and on it the example app in
# deploy/example-app, then run the app's container and check that the
# health check passes, that a request is served through smallwebwaf,
# that `sv stop` stops smallwebwaf in order, and that `docker stop`
# stops the container without having to kill it. The container and both
# images are removed however the script ends. Building the app needs
# network access, for nixpkgs' binary cache. script/check does not run
# this.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
# Named after this run, so that runs in other clones on the same host
# never touch each other's.
NAME="$("$SCRIPT_DIR/projectname")-example-$$"
IMAGE="$NAME-base"
APP_IMAGE="$NAME-app"
CONTAINER="$NAME"
cleanup() {
docker rm --force "$CONTAINER" >/dev/null 2>&1 || true
docker rmi --force "$APP_IMAGE" "$IMAGE" >/dev/null 2>&1 || true
}
fail() {
echo "example-app: $*; the container's output:" >&2
docker logs "$CONTAINER" >&2 || true
exit 1
}
# wait_for <what fails> <command>...: run the command every second until
# it succeeds, for at most a minute.
wait_for() {
failure="$1"
shift
tries=0
until "$@"; do
tries=$((tries + 1))
[ "$tries" -lt 60 ] || fail "$failure"
sleep 1
done
}
healthy() {
status="$(docker inspect --format '{{.State.Health.Status}}' "$CONTAINER")"
[ "$status" = healthy ]
}
# logged <text>: the container's output holds text.
logged() {
docker logs "$CONTAINER" 2>&1 | grep -qF "$1"
}
main() {
cd "$ROOT"
trap cleanup EXIT
trap 'exit 1' HUP INT TERM
docker build --no-cache -t "$IMAGE" .
docker build --no-cache --build-arg SMALLWEBWAF_IMAGE="$IMAGE" \
-t "$APP_IMAGE" deploy/example-app
docker run --detach --name "$CONTAINER" --publish 127.0.0.1::8080 \
"$APP_IMAGE" >/dev/null
wait_for "the health check did not pass" healthy
echo "example-app: the health check passes"
address="$(docker port "$CONTAINER" 8080/tcp)"
page="$(curl --fail --silent --show-error --max-time 10 "http://$address/")" ||
fail "no answer on port 8080"
[ "$page" = "hello from the example app" ] || fail "port 8080 answered $page"
wait_for "smallwebwaf logged no request it forwarded" logged '"action":"forward"'
echo "example-app: smallwebwaf passes a request to the app and its answer back"
docker exec "$CONTAINER" sv stop smallwebwaf >/dev/null ||
fail "sv stop smallwebwaf failed"
wait_for "smallwebwaf did not stop in order" logged '"msg":"stopped"'
echo "example-app: sv stop stops smallwebwaf in order"
docker stop "$CONTAINER" >/dev/null
status="$(docker inspect --format '{{.State.ExitCode}}' "$CONTAINER")"
[ "$status" = 0 ] || fail "docker stop left exit status $status"
echo "example-app: docker stop stops the container in order"
}
main "$@"
Executable
+33
View File
@@ -0,0 +1,33 @@
#!/bin/sh
# script/fmt: format all files (writes): the Go code with gofmt, the
# Markdown with prettier.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Must match the pin in script/bootstrap.
NODE_VERSION="22.17.0"
# script/bootstrap installs node and yarn under nvm and leaves neither
# on the PATH of the shell that called it, so resolve the pinned
# toolchain here the way bootstrap's own install step does. nvm is a
# bash script, hence the subshell.
run_yarn() {
if command -v yarn >/dev/null 2>&1; then
exec yarn "$@"
fi
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
echo "fmt: no yarn; run script/bootstrap first" >&2
exit 1
fi
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
}
main() {
cd "$ROOT"
gofmt -w cmd internal
run_yarn run prettier --write '**/*.md' --tab-width 4 --prose-wrap always
}
main "$@"
+38
View File
@@ -0,0 +1,38 @@
#!/bin/sh
# script/fmt-check: check formatting (read-only): the Go code with
# gofmt, the Markdown with prettier.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Must match the pin in script/bootstrap.
NODE_VERSION="22.17.0"
# script/bootstrap installs node and yarn under nvm and leaves neither
# on the PATH of the shell that called it, so resolve the pinned
# toolchain here the way bootstrap's own install step does. nvm is a
# bash script, hence the subshell.
run_yarn() {
if command -v yarn >/dev/null 2>&1; then
exec yarn "$@"
fi
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
echo "fmt-check: no yarn; run script/bootstrap first" >&2
exit 1
fi
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
}
main() {
cd "$ROOT"
unformatted="$(gofmt -l cmd internal)"
if [ -n "$unformatted" ]; then
echo "fmt-check: gofmt would change:" >&2
echo "$unformatted" >&2
exit 1
fi
run_yarn run prettier --check '**/*.md' --tab-width 4 --prose-wrap always
}
main "$@"
+16
View File
@@ -0,0 +1,16 @@
#!/bin/sh
# script/install-precommit: install the git pre-commit hook that runs
# script/precommit. Our own extension to scripts-to-rule-them-all.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
hook=".git/hooks/pre-commit"
printf '#!/bin/sh\nset -e\nscript/precommit\n' > .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
echo "pre-commit hook installed: runs script/precommit"
}
main "$@"
Executable
+23
View File
@@ -0,0 +1,23 @@
#!/bin/sh
# script/lint: run the linter. Linting is a phase of the Dockerfile and
# this builds that phase alone; the linter is never installed or run on
# a developer host, where a shared result cache and a host-global lock
# make its answer untrustworthy.
#
# The phase is not the last stage in the file, so it is built only when
# --target names it. --no-cache because a cached lint layer is a lint
# that did not run. The tag makes each build replace the previous image
# instead of leaving a dangling one behind.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
docker build --no-cache \
--target lint \
-t "$("$SCRIPT_DIR/projectname")-lint" .
}
main "$@"
+12
View File
@@ -0,0 +1,12 @@
#!/bin/sh
# script/precommit: run by the git pre-commit hook; fails the commit if
# checks fail. Our own extension to scripts-to-rule-them-all.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
main() {
"$SCRIPT_DIR/check"
}
main "$@"
+12
View File
@@ -0,0 +1,12 @@
#!/bin/sh
# script/projectname: output the name of this project. Our own
# extension to scripts-to-rule-them-all. Other scripts that need the
# name (e.g. script/docker) call this, so they can stay identical
# across all repos.
set -eu
main() {
echo "smallwebwaf"
}
main "$@"
Executable
+14
View File
@@ -0,0 +1,14 @@
#!/bin/sh
# script/run: build bin/smallwebwaf with script/build and run it, with
# the settings in the environment.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
"$SCRIPT_DIR/build"
exec "$ROOT/bin/smallwebwaf"
}
main "$@"
Executable
+13
View File
@@ -0,0 +1,13 @@
#!/bin/sh
# script/setup: set up the repo for development after a fresh clone:
# installs dependencies and the git pre-commit hook.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
main() {
"$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/install-precommit"
}
main "$@"
Executable
+19
View File
@@ -0,0 +1,19 @@
#!/bin/sh
# script/test: run the test suite. Testing is a phase of the Dockerfile
# and this builds that phase alone, on the same terms as script/lint:
# --target because a phase that is not the last stage is built only when
# named, --no-cache because a cached test layer is a test that did not
# run, and a tag so each build replaces the previous image.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
docker build --no-cache \
--target test \
-t "$("$SCRIPT_DIR/projectname")-test" .
}
main "$@"
+12
View File
@@ -0,0 +1,12 @@
#!/usr/bin/env bash
set -euo pipefail
# runit's run script for smallwebwaf, run again whenever smallwebwaf
# exits; the wait spaces out the restarts. exec, so that the signal
# `sv stop` sends reaches smallwebwaf itself.
main() {
sleep 1
exec chpst -u smallwebwaf:smallwebwaf /usr/local/bin/smallwebwaf
}
main "$@"
+8
View File
@@ -0,0 +1,8 @@
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1
prettier@3.8.1:
version "3.8.1"
resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.8.1.tgz#edf48977cf991558f4fcbd8a3ba6015ba2a3a173"
integrity sha512-UOnG6LftzbdaHZcKoPFtOcCKztrQ57WkHDeRD9t/PTQtmT0NHSeWWepj6pS0z/N7+08BHFDQVUrfmfMRcZwbMg==