5 Commits
Author SHA1 Message Date
clawbot 7fed6b6d36 Remove what is about deploying behind Gitea (#210)
check / check (push) Successful in 3m30s
2026-11-07 03:52:02 +01:00
sneak f8676debda Merge branch 'main' into next
check / check (push) Successful in 5m27s
2026-11-07 02:52:20 +01:00
clawbot a41d61e9ad Install the image's Ubuntu packages from the normal mirrors (closes #190)
check / check (push) Successful in 3m14s
The image's `ca-certificates`, `nix-bin` and `runit` now come from Ubuntu's archive instead of a dated snapshot, whose service had failed image builds. apt fetches them over plain HTTP and checks each against the archive's signed package lists, so no CA certificates are borrowed from the Go image. The README, `SPEC.md` and `docs/image.md` say so; the amd64-only limit, which came from the removed hash check, is gone.

Rule cut against: `REPO_POLICIES.md` pins every external reference by hash; sneak's ruling at sneak/prompts#72 (comment) allows Ubuntu's own package index.

Model: deepseek-v4-pro (implementation), opus-5-5 (rework)
2026-11-06 19:15:28 +00:00
clawbot 84a9190d2b next into main
check / check (push) Successful in 4m1s
Merged by sneak's order of 6 November ("take the current state of smallwebwaf and get it on to main. i am going to deploy it soon in its alpha state").

Model: opus-5-5
2026-11-06 18:58:02 +01:00
sneak 1bdf66cd21 Merge pull request 'next into main' (#160) from next into main
check / check (push) Canceled after 0s
Reviewed-on: #160
2026-10-26 13:11:04 +01:00
53 changed files with 474 additions and 2246 deletions
+3 -3
View File
@@ -153,15 +153,15 @@ Closest existing options, and why each still falls short:
- What it is: a single-binary reverse proxy that makes browsers solve a - What it is: a single-binary reverse proxy that makes browsers solve a
proof-of-work challenge before passing them to `TARGET`. Aimed at proof-of-work challenge before passing them to `TARGET`. Aimed at
scrapers, which is likely a large share of unwanted traffic on a public scrapers, which is likely a large share of unwanted traffic on a public
gitea. code forge.
- Covers: R10 well (one container, env vars for listener, target, - Covers: R10 well (one container, env vars for listener, target,
difficulty, cookies). Policy rules can match on path, user agent, headers, difficulty, cookies). Policy rules can match on path, user agent, headers,
IP ranges, and with the vendor's hosted data service also AS number and IP ranges, and with the vendor's hosted data service also AS number and
country, and can weigh a request toward a harder challenge. country, and can weigh a request toward a harder challenge.
- Misses: R1, R3, R4, R5, R6, R9. Bot policy needs a YAML file, not env - Misses: R1, R3, R4, R5, R6, R9. Bot policy needs a YAML file, not env
vars. AS number and country matching depend on the vendor's hosted vars. AS number and country matching depend on the vendor's hosted
service. Breaks non-browser clients unless paths are exempted; for gitea, service. Breaks non-browser clients unless paths are exempted; for a code
git-over-HTTP and API paths must be allowed through by rule. forge, git-over-HTTP and API paths must be allowed through by rule.
- Maturity: very active, widely deployed on code forges since 2025. - Maturity: very active, widely deployed on code forges since 2025.
- Value here: complementary. It can be chained (traefik, then the sidecar, - Value here: complementary. It can be chained (traefik, then the sidecar,
then Anubis, then the app) if challenge pages are wanted. then Anubis, then the app) if challenge pages are wanted.
+2 -10
View File
@@ -1,13 +1,11 @@
.PHONY: bootstrap setup test lint fmt fmt-check tidy check docker hooks build \ .PHONY: bootstrap setup test lint fmt fmt-check tidy check docker hooks build \
run example-app gitea-requests gitea-setup run example-app
# Makefile targets are thin shims; the implementations live in script/ # Makefile targets are thin shims; the implementations live in script/
# per the scripts-to-rule-them-all pattern (see the Entrypoints section # per the scripts-to-rule-them-all pattern (see the Entrypoints section
# of README.md). tidy writes go.mod and go.sum as `go mod tidy` does, # of README.md). tidy writes go.mod and go.sum as `go mod tidy` does,
# which test checks. build and run are for working on the code by hand; # which test checks. build and run are for working on the code by hand;
# example-app checks the image with an app built on it; gitea-requests # example-app checks the image with an app built on it.
# reports the gitea requests the Core Rule Set refuses at the defaults;
# gitea-setup checks the setup in deploy/gitea.
bootstrap: bootstrap:
@script/bootstrap @script/bootstrap
@@ -47,9 +45,3 @@ run:
example-app: example-app:
@script/example-app @script/example-app
gitea-requests:
@script/gitea-requests
gitea-setup:
@script/gitea-setup
+4 -101
View File
@@ -100,8 +100,8 @@ Point traefik's labels at port 8080 of `rproxy`
of the app or of `servewaf`. Anything on that network can reach the app around of the app or of `servewaf`. Anything on that network can reach the app around
`rproxy`, and can ask `servewaf` questions, so the network holds only what you `rproxy`, and can ask `servewaf` questions, so the network holds only what you
trust. Each further app adds its own container and its own `rproxy`, whose trust. Each further app adds its own container and its own `rproxy`, whose
`SWRP_WAF_BACKEND` names the same `servewaf`. The "Smallest setup" and "In front `SWRP_WAF_BACKEND` names the same `servewaf`. The "Smallest setup" example below
of gitea" examples below show this shape. shows this shape.
### `rproxy` in the app's container ### `rproxy` in the app's container
@@ -338,9 +338,8 @@ full, and the limits that are fixed rather than settings.
Each example is a complete `compose.yaml`. `smallwebwaf` is the tag Each example is a complete `compose.yaml`. `smallwebwaf` is the tag
`make docker` gives the image; on another host, use the name you pushed it `make docker` gives the image; on another host, use the name you pushed it
under, by digest. Docker Compose puts the services on a private network of its under, by digest. Docker Compose puts the services on a private network of its
own, where each reaches the others by service name. Only "In front of gitea" own, where each reaches the others by service name. Each example publishes
uses traefik. The other three publish `rproxy`'s port on the host, for trying it `rproxy`'s port on the host, for trying it out.
out.
### Smallest setup ### Smallest setup
@@ -426,88 +425,6 @@ volumes:
servewaf-state: servewaf-state:
``` ```
### In front of gitea
gitea behind traefik, set up for a code forge:
- the upload limits raised, in size and in time;
- the response size limit raised, for clones over 1 GiB;
- the Actions runners exempt from counting;
- the default request and byte limits kept.
`traefik` is the network traefik is on, made outside this file.
```yaml
services:
gitea:
# gitea 1.27.3, 2026-08-29
image: gitea/gitea@sha256:87a67ee09d3ae0d1df5fda5dcda3e2a1f9236a45b0a59025d6e00e46adc43bef
environment:
GITEA__server__ROOT_URL: https://git.example/
# The host and the port of the SSH clone URLs gitea shows.
GITEA__server__SSH_DOMAIN: git.example
GITEA__server__SSH_PORT: "2222"
# gitea's TCP peer is rproxy, on the private network.
GITEA__security__REVERSE_PROXY_TRUSTED_PROXIES: 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
volumes:
- gitea-data:/data
networks:
- site
ports:
# SSH does not pass through smallwebwaf.
- "2222:22"
rproxy:
image: smallwebwaf
environment:
SWRP_WAF_BACKEND: http://servewaf:8080
SWRP_UPSTREAM_URL: http://gitea:3000
# Pushes, LFS objects and image layers over 100 MiB, or slower
# than a minute, are cut off at the defaults.
SWRP_MAX_REQUEST_SIZE: 2G
SWRP_MAX_REQUEST_TIME_MS: "600000"
# A clone larger than 1 GiB is cut off at the default.
SWRP_MAX_RESPONSE_SIZE: 4G
networks:
- site
- traefik
labels:
traefik.enable: "true"
# rproxy is on two networks; traefik reaches it on this one.
traefik.docker.network: traefik
traefik.http.routers.git.rule: Host(`git.example`)
traefik.http.routers.git.entrypoints: websecure
traefik.http.routers.git.tls.certresolver: letsencrypt
traefik.http.services.git.loadbalancer.server.port: "8080"
servewaf:
image: smallwebwaf
environment:
SMALLWEBWAF_COMMAND: servewaf
SWWAF_INSTANCE_NAME: fsn1/servewaf
# The Actions runners ask for work all day; never count them.
SWWAF_RATE_LIMIT_EXEMPT_NETS: 10.0.4.0/24
# gitea never serves these; a request for one bans the client.
SWWAF_TRAP_PATHS: /wp-login.php,/xmlrpc.php,/wp-admin/
volumes:
- servewaf-state:/var/lib/smallwebwaf
networks:
- site
networks:
site:
traefik:
external: true
volumes:
gitea-data:
servewaf-state:
```
`deploy/gitea` holds a complete setup of gitea, traefik included, as a Docker
Compose file. [`docs/gitea.md`](docs/gitea.md) explains it, and records what the
defaults do in front of gitea, request by request.
### `rproxy` beside the app ### `rproxy` beside the app
The app is built on the `smallwebwaf` image, with `rproxy` inside its container. The app is built on the `smallwebwaf` image, with `rproxy` inside its container.
@@ -771,18 +688,6 @@ on the host.
- orderly stops with `sv stop` and `docker stop`; - orderly stops with `sv stop` and `docker stop`;
- bans surviving a new container on the same volume; - bans surviving a new container on the same volume;
- a setting given as a file. - a setting given as a file.
- `script/gitea-requests`: builds the image and runs gitea 1.27.3 behind
`rproxy` and `servewaf`, three containers on one network. Every setting is at
its default, but for those that join the containers and
`SWWAF_LOG_REQUEST_HEADERS`, which adds the header that labels each request.
It sends through them a sample of gitea's ordinary use and the requests the
Core Rule Set is known to refuse, and prints what `servewaf` did with each.
Needs network access, and a docker daemon that pushes to a registry on
`localhost` over plain HTTP. `make gitea-requests`.
- `script/gitea-setup`: builds the image, brings up the setup in `deploy/gitea`,
and checks it as [`docs/gitea.md`](docs/gitea.md) describes. Then it removes
the containers, the network, the volumes and the image it built. Needs network
access. `make gitea-setup`.
## TODO ## TODO
@@ -806,8 +711,6 @@ on the host.
- [`docs/state-files.md`](docs/state-files.md): the state files and what each - [`docs/state-files.md`](docs/state-files.md): the state files and what each
holds. holds.
- [`docs/image.md`](docs/image.md): what the image holds and how it is pinned. - [`docs/image.md`](docs/image.md): what the image holds and how it is pinned.
- [`docs/gitea.md`](docs/gitea.md): `smallwebwaf` in front of gitea: the setup
in `deploy/gitea`, and what the defaults do there.
## License ## License
+46 -56
View File
@@ -123,8 +123,7 @@ runs steps 3 to 13, and does the counting of step 15. The first step that
produces a final answer ends the checks. Step 15 runs for each request produces a final answer ends the checks. Step 15 runs for each request
`servewaf` answers with `allow`, `deny` or `close`. `rproxy`'s size and time `servewaf` answers with `allow`, `deny` or `close`. `rproxy`'s size and time
limits apply to the whole exchange. Each step names the section that holds its limits apply to the whole exchange. Each step names the section that holds its
rules. Each check is a heuristic, whose name the request log's `heuristic` field rules.
gives, shown here in parentheses.
1. `GET /_smallwebwaf/healthz` is answered at once by `rproxy`, before any 1. `GET /_smallwebwaf/healthz` is answered at once by `rproxy`, before any
check, so a health checker never needs an exemption and is never refused. check, so a health checker never needs an exemption and is never refused.
@@ -142,9 +141,9 @@ gives, shown here in parentheses.
around its address, a /64 by default, since one abuser usually controls a around its address, a /64 by default, since one abuser usually controls a
whole /64. A client is therefore one IPv4 address or one IPv6 group. whole /64. A client is therefore one IPv4 address or one IPv6 group.
3. The static lists. A client in `SWWAF_ALLOW_NETS` skips steps 4 to 12, the 3. The static lists. A client in `SWWAF_ALLOW_NETS` skips steps 4 to 12, the
byte limits and the error burst. A client in `SWWAF_DENY_NETS` is refused byte limits and the error burst. A client in `SWWAF_DENY_NETS` is refused; a
(`deny_nets`); a client in both lists is let through. client in both lists is let through.
4. Bans. A client under a ban is refused (`active_ban`; see "Bans"). 4. Bans. A client under a ban is refused (see "Bans").
5. The lookup, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Lookup of AS number 5. The lookup, unless `SWWAF_LOOKUP_SOURCE` is `off` (see "Lookup of AS number
and country"). The request goes on at once. It waits for the first GeoJS and country"). The request goes on at once. It waits for the first GeoJS
answer, up to `SWWAF_LOOKUP_TIMEOUT`, only when one of these needs it: answer, up to `SWWAF_LOOKUP_TIMEOUT`, only when one of these needs it:
@@ -154,26 +153,22 @@ gives, shown here in parentheses.
A request that ends before the answer comes is logged without it. A request that ends before the answer comes is logged without it.
6. The country lists (`countries`; see "Country lists"). 6. The country lists (see "Country lists").
7. The reputation sources (see "Reputation"), in this order: 7. The reputation sources (see "Reputation"). A client on a fetched blocklist
- A client on a fetched blocklist gets `SWWAF_BLOCKLIST_ACTION` gets `SWWAF_BLOCKLIST_ACTION`. A client on the CrowdSec decision list is
(`blocklists`). banned. A client listed by a DNS blocklist, or scored at least
- A client on the CrowdSec decision list is banned (`crowdsec`). `SWWAF_ABUSEIPDB_MIN_SCORE` by AbuseIPDB, gets `SWWAF_REPUTATION_ACTION`.
- A client listed by a DNS blocklist gets `SWWAF_REPUTATION_ACTION`
(`dnsbl`).
- A client scored at least `SWWAF_ABUSEIPDB_MIN_SCORE` by AbuseIPDB gets
`SWWAF_REPUTATION_ACTION` (`abuseipdb`).
8. The client's limit percentage (see "Biased thresholds"). 8. The client's limit percentage (see "Biased thresholds").
9. Rate limiting (`rate_limits`; see "Request rate limits, per client"). A 9. Rate limiting (see "Request rate limits, per client"). A request over a limit
request over a limit times the percentage is refused and bans the client. times the percentage is refused and bans the client.
10. The trap paths (`trap_paths`; see "Attack detection"). A request for one is 10. The trap paths (see "Attack detection"). A request for one is refused and
refused and bans the client. bans the client.
11. The rule files (`rule_files`; see "Rule files"). Matching stops at the first 11. The rule files (see "Rule files"). Matching stops at the first rule that
rule that refuses. refuses.
12. The header limits (`header_limits`; see "Answering in time"), and then the 12. The header limits (see "Answering in time"), and then the Core Rule Set (see
Core Rule Set (`core_rule_set`; see "Attack detection"). A Core Rule Set "Attack detection"). A Core Rule Set match is refused with `403` while
match is refused with `403` while `SWWAF_WAF_MODE` is `block`, and only `SWWAF_WAF_MODE` is `block`, and only logged and alerted while it is
logged and alerted while it is `detect`. `detect`.
13. A path under `/_smallwebwaf/` is answered `404`, so that the app never sees 13. A path under `/_smallwebwaf/` is answered `404`, so that the app never sees
that prefix. that prefix.
14. `rproxy` passes the request to `SWRP_UPSTREAM_URL`, streaming (see "Passing 14. `rproxy` passes the request to `SWRP_UPSTREAM_URL`, streaming (see "Passing
@@ -187,14 +182,14 @@ gives, shown here in parentheses.
are added to the byte counters, as `SWWAF_BYTES_COUNT` says. For a are added to the byte counters, as `SWWAF_BYTES_COUNT` says. For a
WebSocket, what it carried each way counts once it closes. A client whose WebSocket, what it carried each way counts once it closes. A client whose
total passes a byte limit times the percentage has broken that limit and total passes a byte limit times the percentage has broken that limit and
is banned (`byte_limits`). The response in progress is not cut off. is banned. The response in progress is not cut off.
- For a request it allowed, the client's history and the anomaly counters - For a request it allowed, the client's history and the anomaly counters
are updated, and the alert thresholds are evaluated. are updated, and the alert thresholds are evaluated.
For a request it refused, `servewaf` adds nothing to the byte limits. It For a request it refused, `servewaf` adds nothing to the byte limits. It
updates the history and the anomaly counters at its decision. It counts the updates the history and the anomaly counters at its decision. It counts the
refusals the error burst counts at the decision too (`refusal_limit`; see refusals the error burst counts at the decision too (see "Attack
"Attack detection"). detection").
In `SWWAF_MODE=observe`, every refusal above is logged as what would have In `SWWAF_MODE=observe`, every refusal above is logged as what would have
happened, nothing is refused, and `servewaf` makes no ban of its own. A path happened, nothing is refused, and `servewaf` makes no ban of its own. A path
@@ -407,10 +402,10 @@ scrapers that send many requests slowly over hours or days. The hour limit is
the main one. the main one.
- Per minute: 120, 2 requests a second. It leaves room for a person opening two - Per minute: 120, 2 requests a second. It leaves room for a person opening two
pages at once with an empty cache. A gitea page opened with an empty cache pages at once with an empty cache. A web app's page opened with an empty cache
loads 20 to 60 files, and a handful once cached. loads 20 to 60 files, and a handful once cached.
- Per hour: 1,500. A person reading code in gitea for a whole hour, a page every - Per hour: 1,500. A person reading a site for a whole hour, a page every 20
20 seconds, makes about 900 requests. A scraper making one request every 2 seconds, makes about 900 requests. A scraper making one request every 2
seconds makes 1,800 an hour, more than the limit. seconds makes 1,800 an hour, more than the limit.
- Per day: 6,000, four hours at the hour limit. A scraper making one request - Per day: 6,000, four hours at the hour limit. A scraper making one request
every 10 seconds makes 8,640 a day, more than the limit. every 10 seconds makes 8,640 a day, more than the limit.
@@ -422,7 +417,7 @@ the blocklists. An abuser that trips none of them is not caught.
Some legitimate clients pass the limits: Some legitimate clients pass the limits:
- An Actions runner asks gitea for work all day and can pass the limits. - A CI runner asks its code forge for work all day and can pass the limits.
- Everyone behind one shared address, such as an office, shares one client's - Everyone behind one shared address, such as an office, shares one client's
limits. limits.
@@ -555,12 +550,11 @@ service's abuse comes from.
By default the Core Rule Set reads the method, the URL with its query string, By default the Core Rule Set reads the method, the URL with its query string,
and the headers of every request. That is where automated attacks show: and the headers of every request. That is where automated attacks show:
injection in query strings, path traversal, attacks carried in headers, injection in query strings, path traversal, attacks carried in headers,
scanners' user agents. It reads no request body. In front of a code forge such scanners' user agents. It reads no request body. In front of a code forge,
as gitea, bodies carry what people write and publish, such as issue text and bodies carry what people write and publish, such as issue text and files saved
files saved in the web editor. The Core Rule Set takes the shell commands and in the web editor. The Core Rule Set takes the shell commands and code in them
code in them for attacks. The default gives up refusing an attack carried in a for attacks. The default gives up refusing an attack carried in a body. The rule
body. The rule files, the limits and the bans still apply to the client that files, the limits and the bans still apply to the client that sends it.
sends it.
- The Core Rule Set skips a path under `SWWAF_WAF_EXEMPT_PATHS`. - The Core Rule Set skips a path under `SWWAF_WAF_EXEMPT_PATHS`.
- `SWWAF_WAF_PARANOIA_LEVEL` and `SWWAF_WAF_ANOMALY_THRESHOLD` are the Core Rule - `SWWAF_WAF_PARANOIA_LEVEL` and `SWWAF_WAF_ANOMALY_THRESHOLD` are the Core Rule
@@ -596,15 +590,14 @@ sends it.
data, where binary content such as a git push trips rules written for text. A data, where binary content such as a git push trips rules written for text. A
body the Core Rule Set cannot parse adds 5 to the score, and so does a body the Core Rule Set cannot parse adds 5 to the score, and so does a
multipart body cut by the limit in the middle of a part's header. Body multipart body cut by the limit in the middle of a part's header. Body
inspection suits apps whose forms carry no code; `docs/gitea.md` records what inspection suits apps whose forms carry no code.
it refuses in front of gitea.
- `SWWAF_TRAP_PATHS` names paths the app never serves and only scanners ask for, - `SWWAF_TRAP_PATHS` names paths the app never serves and only scanners ask for,
such as `/wp-login.php` in front of gitea. This is the short form of a `path` such as `/wp-login.php` in front of an app that is not WordPress. This is the
rule with the `ban` action, for deployments that mount no rule files. A trap short form of a `path` rule with the `ban` action, for deployments that mount
path is matched whole, without the query, against the path as sent and once no rule files. A trap path is matched whole, without the query, against the
percent-decoded: `/wp-login.php` matches `/wp-login.php?x=1` and path as sent and once percent-decoded: `/wp-login.php` matches
`/%77p-login.php`, and does not match `/wp-login.php/` or `/wp-login.php?x=1` and `/%77p-login.php`, and does not match `/wp-login.php/`
`/blog/wp-login.php`. or `/blog/wp-login.php`.
- `SWWAF_ERROR_BURST_THRESHOLD` counts a client's refusals per minute of these - `SWWAF_ERROR_BURST_THRESHOLD` counts a client's refusals per minute of these
kinds, and of no other: kinds, and of no other:
- a rule file match; - a rule file match;
@@ -614,8 +607,8 @@ sends it.
More than this breaks a limit (see "Bans"). A client trying one attack after More than this breaks a limit (see "Bans"). A client trying one attack after
another is refused many times a minute; a person rarely more than a few another is refused many times a minute; a person rarely more than a few
times. Answers the app gives are not counted, since they do not tell a times. Answers the app gives are not counted, since they do not tell a
scanner from an ordinary client. In front of gitea, registry and git clients scanner from an ordinary client. In front of a code forge, registry and git
get `404` and `401` by design, hundreds of times a minute. clients get `404` and `401` by design, hundreds of times a minute.
`smallwebwaf` changes the Core Rule Set 4.25.0 in six ways, since in front of `smallwebwaf` changes the Core Rule Set 4.25.0 in six ways, since in front of
gitea it would otherwise refuse ordinary requests. The changes hold in front of gitea it would otherwise refuse ordinary requests. The changes hold in front of
@@ -841,7 +834,7 @@ and the default file.
runs. It reads the files again once the directory has had no change for 2 runs. It reads the files again once the directory has had no change for 2
seconds, so that a file still being written is read whole. A file whose name seconds, so that a file still being written is read whole. A file whose name
starts with `.`, such as an editor's lock file, is not a rule file. starts with `.`, such as an editor's lock file, is not a rule file.
- Files are read in name order (`00-default.rules` before `50-gitea.rules`), - Files are read in name order (`00-default.rules` before `50-site.rules`),
rules in line order. rules in line order.
- A rule's regex is Go's RE2 syntax. It has no backreferences or lookaround; in - A rule's regex is Go's RE2 syntax. It has no backreferences or lookaround; in
exchange matching time is linear in the input, so no rule can be made to stall exchange matching time is linear in the input, so no rule can be made to stall
@@ -870,14 +863,12 @@ and the default file.
- The image ships a default file, kept short and limited to patterns that are - The image ships a default file, kept short and limited to patterns that are
wrong for every app, with its path rules anchored at the site root. In front wrong for every app, with its path rules anchored at the site root. In front
of gitea, `/.env` is a probe, while of a code forge, `/.env` is a probe, while
`/<owner>/<repo>/src/branch/main/.env.example` is a file in a repository that `/<owner>/<repo>/src/branch/main/.env.example` is a file in a repository that
any visitor may open. Anchor the paths of your own `ban` rules at the site any visitor may open. Anchor the paths of your own `ban` rules at the site
root too. Anything else, such as a rule banning WordPress probes in front of root too. A rule has one target, so a `path` rule applies on every site behind
gitea, belongs in a file the deployer adds to the image `servewaf` runs from, that `servewaf`. A directory mounted over `SWWAF_RULES_DIR` replaces the
or mounts. A rule has one target, so a `path` rule applies on every site default file; single files mounted into it add to it.
behind that `servewaf`. A directory mounted over `SWWAF_RULES_DIR` replaces
the default file; single files mounted into it add to it.
- Rules meant to be edited while `servewaf` runs belong in a mounted directory, - Rules meant to be edited while `servewaf` runs belong in a mounted directory,
not a mounted file. Many editors save a file by replacing it, and a container not a mounted file. Many editors save a file by replacing it, and a container
keeps seeing the old one when the file alone is mounted. The lookup database keeps seeing the old one when the file alone is mounted. The lookup database
@@ -1700,8 +1691,7 @@ with no change to upaas. upaas builds the app's Dockerfile and runs the one
container as it runs any app. The traefik labels route to port 8080, and the container as it runs any app. The traefik labels route to port 8080, and the
`SWRP_` settings go with the app's environment variables. `servewaf` runs in a `SWRP_` settings go with the app's environment variables. `servewaf` runs in a
container of its own, on a network the app's container can reach, with its state container of its own, on a network the app's container can reach, with its state
volume and the directory of any token file. `docs/gitea.md` records what the volume and the directory of any token file.
defaults do in front of gitea, and `script/gitea-requests` checks those notes.
## Planned, not built ## Planned, not built
-135
View File
@@ -1,135 +0,0 @@
# Gitea behind traefik, with rproxy and servewaf in front of it.
# docs/gitea.md says what each setting here does and why. Before starting it:
# - set SMALLWEBWAF_IMAGE to the smallwebwaf image, named by digest;
# - replace git.example.org with the site's host name;
# - replace each 10.99.0 with the first three numbers of a /24 subnet that
# no other network on the host uses. The /24 must lie inside 10.0.0.0/8,
# 172.16.0.0/12 or 192.168.0.0/16, because rproxy believes
# X-Forwarded-For only from those by default. Replace each fd10:99 with
# the first two hextets of an unused /64 subnet whose third and fourth
# hextets are 0, as in fd10:99::/64;
# - set GITEA_SECRET_KEY to the output of
# `docker run --rm <gitea's image below> gitea generate secret SECRET_KEY`,
# once per install, and keep it.
# Then run `docker compose up --detach` in this directory.
# script/gitea-setup checks this setup.
name: gitea
x-logging: &logging
driver: json-file
options:
max-size: 10m
max-file: "5"
services:
traefik:
# traefik v3.7.17, 2026-11-03
image: traefik@sha256:68ae5fe5b64ca166704b746a9e8afcd385e36b3f569a5691401bf652d57b3748
# traefik replaces the X-Forwarded-For header a visitor sends. Keep it
# so: otherwise a visitor could name an address on the shared network,
# and skip servewaf's checks.
command:
- --providers.docker.exposedbydefault=false
- --entrypoints.web.address=:80
- --entrypoints.web.http.redirections.entrypoint.to=websecure
- --entrypoints.websecure.address=:443
# Until a certificate resolver is added here, as for your other
# sites, traefik answers with a certificate it made itself.
- --entrypoints.websecure.http.tls=true
# traefik's own limit of 60 seconds for a whole request would cut
# off an upload long before rproxy's limit of 30 minutes.
- --entrypoints.websecure.transport.respondingTimeouts.readTimeout=30m
ports:
- 80:80
- 443:443
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
networks:
shared:
# Containers on the shared network find traefik under the
# site's name, and so reach it from their own address.
aliases:
- git.example.org
restart: unless-stopped
gitea:
# gitea 1.27.3, 2026-08-29
image: gitea/gitea@sha256:87a67ee09d3ae0d1df5fda5dcda3e2a1f9236a45b0a59025d6e00e46adc43bef
environment:
GITEA__server__ROOT_URL: https://git.example.org/
# Gitea starts at its defaults, with SQLite, instead of showing its
# install page to whoever opens it first.
GITEA__security__INSTALL_LOCK: "true"
GITEA__security__SECRET_KEY: ${GITEA_SECRET_KEY:?set GITEA_SECRET_KEY, see docs/gitea.md}
# Accounts are made with `gitea admin user create`.
GITEA__service__DISABLE_REGISTRATION: "true"
# rproxy's addresses. Gitea then takes each visitor's address from
# the X-Real-Ip header traefik sets and rproxy passes on.
GITEA__security__REVERSE_PROXY_TRUSTED_PROXIES: 10.99.0.2,fd10:99::2
volumes:
- gitea-data:/data
networks:
- shared
restart: unless-stopped
rproxy:
image: ${SMALLWEBWAF_IMAGE:?set SMALLWEBWAF_IMAGE to the smallwebwaf image}
environment:
SWRP_WAF_BACKEND: http://servewaf:8080
SWRP_UPSTREAM_URL: http://gitea:3000
# Pushes, LFS objects, packages and release attachments of up to
# 2 GiB, sent in up to 30 minutes.
SWRP_MAX_REQUEST_SIZE: 2G
SWRP_MAX_REQUEST_TIME_MS: "1800000"
# 15 minutes for a clone or a download, half the default.
SWRP_MAX_RESPONSE_TIME_MS: "900000"
labels:
traefik.enable: "true"
traefik.http.routers.gitea.rule: Host(`git.example.org`)
traefik.http.routers.gitea.entrypoints: websecure
traefik.http.services.gitea.loadbalancer.server.port: "8080"
networks:
shared:
ipv4_address: 10.99.0.2
ipv6_address: fd10:99::2
logging: *logging
restart: unless-stopped
servewaf:
image: ${SMALLWEBWAF_IMAGE:?set SMALLWEBWAF_IMAGE to the smallwebwaf image}
environment:
SMALLWEBWAF_COMMAND: servewaf
# The containers on the shared network skip the checks. Add the
# address of an automation fleet on another host here too; without
# that, it is counted like any client.
SWWAF_ALLOW_NETS: 10.99.0.128/25,fd10:99:0:0:8000::/65
# In block mode the Core Rule Set refuses everyday Gitea searches
# and file names, and 31 refusals in a minute ban the address.
SWWAF_WAF_MODE: detect
volumes:
- servewaf-state:/var/lib/smallwebwaf
networks:
- shared
logging: *logging
restart: unless-stopped
networks:
# The network traefik and Gitea share. Its containers get addresses in the
# upper half of each subnet, which servewaf lets skip the checks. Each
# subnet's gateway, the host's own address on it, stays out of that half:
# requests docker passes on from a published port can come from it.
shared:
enable_ipv6: true
ipam:
config:
- subnet: 10.99.0.0/24
ip_range: 10.99.0.128/25
gateway: 10.99.0.1
- subnet: fd10:99::/64
ip_range: fd10:99:0:0:8000::/65
gateway: fd10:99::1
volumes:
gitea-data:
servewaf-state:
-340
View File
@@ -1,340 +0,0 @@
# smallwebwaf in front of gitea
`deploy/gitea/compose.yaml` runs gitea behind traefik, with `rproxy` and
`servewaf` in front of it. This document says what that setup sets and why, and
what the defaults would do without it. [`SPEC.md`](../SPEC.md) says how
`smallwebwaf` behaves.
## The setup
Four containers share one Docker network, the shared network:
- traefik, which routes the site's HTTPS requests to `rproxy`, and redirects
plain HTTP to HTTPS;
- `rproxy`, in front of gitea;
- gitea 1.27.3, as its image comes;
- `servewaf`, with a volume for its state files.
To start it:
1. In `deploy/gitea/compose.yaml`, replace `git.example.org` with the site's
host name.
2. Replace each `10.99.0` with the first three numbers of a `/24` subnet that no
other network on the host uses. The `/24` must lie inside `10.0.0.0/8`,
`172.16.0.0/12` or `192.168.0.0/16`, because `rproxy` believes
`X-Forwarded-For` only from those by default. Replace each `fd10:99` with the
first two hextets of an unused `/64` subnet whose third and fourth hextets
are 0, as in `fd10:99::/64`.
3. Set `SMALLWEBWAF_IMAGE` to the `smallwebwaf` image, named by digest.
4. Set `GITEA_SECRET_KEY` to the output of
`docker run --rm <gitea's image> gitea generate secret SECRET_KEY`, with
gitea's image as `deploy/gitea/compose.yaml` names it. Do this once per
install, and keep the key.
5. Run `docker compose up --detach` in `deploy/gitea`.
6. Make gitea's first user with `gitea admin user create`, in gitea's container,
as the user `git`.
Sign-up is off: accounts are made with `gitea admin user create`. gitea starts
at its defaults, with SQLite. traefik answers with a certificate it made itself
until you add a certificate resolver, as for your other sites. SSH to gitea does
not pass through `smallwebwaf`; publish gitea's SSH port yourself if you use it.
`make gitea-setup` checks the setup. It builds the image, brings the setup up
under a name of its own, and checks:
- that a page load, a push and a clone sent through traefik pass through
`rproxy`, and that `servewaf` checks them;
- that gitea logs the visitor's address, not `rproxy`'s;
- that a request from a container on the shared network skips the checks;
- that with `servewaf` stopped, gitea still answers through `rproxy`.
## Uploads and downloads
An upload is a request:
- a git push;
- an LFS object;
- a container image layer;
- a package file;
- a release attachment.
The setup lets an upload of up to 2 GiB, sent in up to 30 minutes, through:
- `SWRP_MAX_REQUEST_SIZE` is `2G`. At the default, an upload over 100 MiB is cut
off.
- `SWRP_MAX_REQUEST_TIME_MS` is `1800000`, 30 minutes. At the default, an upload
that takes over 60 seconds is cut off.
- traefik's `readTimeout` is 30 minutes. traefik's own default of 60 seconds for
a whole request would cut off an upload first.
A clone or a download is a response:
- `SWRP_MAX_RESPONSE_TIME_MS` is `900000`, 15 minutes, half the default. A clone
or download that takes longer is cut off.
- `SWRP_MAX_RESPONSE_SIZE` stays at its default, 1 GiB. A repository whose clone
is larger needs it raised.
The byte limits stay at their defaults: no minute limit, 50 GiB an hour and 100
GiB a day.
## The checks
`SWWAF_WAF_MODE` is `detect`. The Core Rule Set then logs a match, with its rule
ids, and refuses nothing. In `block` mode, the default, it refuses everyday
gitea searches and file names. The section "What the Core Rule Set refuses in
`block` mode" below lists them. Each such refusal counts toward the error burst,
so 31 in one minute ban the address.
Every other check stays at its default. A client may make 120 requests a minute,
1,500 an hour and 6,000 a day. Archive downloads and blame and history pages are
what scrapers hammer, and these limits do most of the work there.
## The shared network
Containers on the shared network skip `servewaf`'s checks:
- The shared network's subnets are `10.99.0.0/24` and `fd10:99::/64`. Docker
gives its containers addresses in the upper half of each, `10.99.0.128/25` and
`fd10:99:0:0:8000::/65`, which `SWWAF_ALLOW_NETS` names.
- traefik is known on the shared network by the site's host name. So a container
there that asks for the site reaches traefik from its own address. An
automation fleet in such a container skips the checks.
- Each subnet's gateway, `10.99.0.1` and `fd10:99::1`, is the host's own address
on it. Requests docker passes on from a published port can come from the
gateway, so the gateways are left out of `SWWAF_ALLOW_NETS`. The network has
IPv6, so a visitor over IPv6 reaches traefik from its own address, and
`servewaf` checks it as itself.
- A container on the shared network that asks for the site at the host's public
address arrives from the gateway. It is checked like any visitor.
- A fleet on another host can have its address added to `SWWAF_ALLOW_NETS`.
Without that, it is counted like any client.
- traefik replaces the `X-Forwarded-For` a visitor sends, by default. Keep it
so: otherwise a visitor could name an address on the shared network, and skip
the checks.
- Anything on the shared network can reach gitea and `servewaf` around `rproxy`.
Put only containers you trust on it.
## gitea's settings
- `REVERSE_PROXY_TRUSTED_PROXIES` names `rproxy`'s fixed addresses, `10.99.0.2`
and `fd10:99::2`. gitea then takes each visitor's address from the `X-Real-Ip`
header, which traefik sets and `rproxy` passes on. Without it, gitea logs
every visitor as `rproxy`.
- `INSTALL_LOCK` is `true`, as "The setup" above says.
## Methods
`rproxy` passes on only these methods: `GET`, `HEAD`, `POST`, `OPTIONS`, `PUT`,
`DELETE`, `PATCH` and `QUERY`. gitea 1.27.3 needs no other. Its routes use:
- pages and the API: `GET`, `HEAD`, `POST`, `PUT`, `PATCH` and `DELETE`;
- git's push, clone and fetch over HTTP: `GET`, `POST` and `OPTIONS`;
- LFS: `GET`, `POST` and `PUT`;
- the container registry and the other package registries: `GET`, `HEAD`,
`POST`, `PUT`, `PATCH` and `DELETE`;
- Actions runners: `POST`.
## The log
`rproxy` and `servewaf` write each query string to the log whole. A token sent
in the query string, as `?token=`, is then in the log. Send it in the
`Authorization` header instead.
## Actions runners
A runner on the shared network skips the checks. One on another host is counted
like any client. A runner older than version 0.4 (April 2026) asks for work
every 2 seconds, 1,800 requests an hour. While it runs a job, it reports the
job's log and state every second instead, two requests a second.
So one idle runner makes 1,800 requests an hour, more than the hour limit of
1,500. A busy one, at two requests a second, passes the minute limit of 120
within two minutes. Its address is then banned, its running jobs fail, and each
repeat within a day makes the ban longer. Newer runners ask for work less often.
Put the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`.
## What the Core Rule Set refuses in `block` mode
These notes record what the Core Rule Set does in front of gitea in `block`
mode, at the other defaults. They are observations, not requirements.
`script/gitea-requests` runs gitea behind `rproxy` and `servewaf` at the
defaults. It sends through them a sample of gitea's ordinary use and the
requests named below. It reports each request the Core Rule Set refuses, so that
these notes can be checked again when gitea or the Core Rule Set changes
version. In `detect` mode, as the setup runs it, the same requests are logged
with their rule ids, and pass.
Each refusal answers only that request ("Bans" in `SPEC.md` says what it counts
toward). The request log names the rule in `waf_rule_ids`, which
`SWWAF_WAF_DISABLED_RULES` can switch off.
### What it lets through
The Core Rule Set lets gitea's ordinary use through, apart from the refusals in
the sections after this one:
- browsing and views of files in a repository, with their history, blame and the
file tree;
- diffs, including their hidden lines and large files, and pull request review;
- git's clone, fetch and push over HTTP;
- signing in, including the return to the page a visitor came from, and sign-in
with Git Credential Manager, git-credential-oauth or tea;
- the API's calls for a file and its commits;
- pushing and pulling container images and packages;
- Actions runners, and artifacts uploaded with `actions/upload-artifact@v4`;
- posting issues, pull requests, comments, wiki pages and files saved in the web
editor, code included, since no body is read;
- the page gitea shows after a change, with its message naming what was done.
### A query string that reads as an attack
Most often this is a search. A search term is a query parameter, which the Core
Rule Set checks against its lists of system files and commands (see the fourth
change under "Attack detection" in `SPEC.md`). A release attachment uploaded
through the API as `docker-compose.yml` is refused the same way. Examples:
- a name on those lists, such as `package.json`, `.gitignore` or
`docker-compose.yml`;
- text that starts with a command name, such as `python3` or `ssh key`, or has
one right after a `/` in its first word, such as `feat/docker-support`;
- a shell command with its options or a system path (`ls -la`, `sed -i`,
`/bin/sh`), or a command in backticks;
- script code (`fetch(`, `${VAR}`, `process.env`), HTML (`<img src=`) or an SQL
statement (`SELECT * FROM users WHERE`);
- a URL naming an IP address or `localhost`.
### A branch, tag or file name that starts with a command name
The Core Rule Set's list of commands (932260) holds names such as `docker-`,
`python3`, `ansible`, `base64` and `whoami`. A query parameter holding a branch,
tag or file name that starts with one is refused. So is one that has such a name
right after a `/`, such as `feat/docker-support`, `fix/docker-build` or
`renovate/docker-build-push-action-6.x`. gitea's own pages send the name in a
query parameter that 930120, 932160 and 932260 still check:
- `refSubUrl`, the branch or tag of a directory listing. A listing sends it when
it asks for its entries' last commits in a second request. It does so whenever
gitea takes more than a second to work them out;
- `name`, when a branch is deleted or restored on the branches page;
- `tag`, when a release is started from a tag;
- `template`, the file of an issue template, when an issue is opened from it;
- `rule_name`, when a branch protection rule is opened for editing in the
repository's settings or has just been saved.
What a visitor sees, for a branch named `docker-build`:
- A directory listing that makes that second request leaves those last commits
out and shows an error.
- The branches page shows an error instead of deleting or restoring the branch.
- A release started from such a tag gets the `403` answer of `smallwebwaf` in
place of its form. So does an issue opened from such a template, and such a
rule opened for editing.
- Saving such a rule stores it. gitea then returns to the branch settings page,
which gets the `403` answer, so the save looks as if it failed.
### A file name in a query parameter that trips the code rules
The Core Rule Set's rules for Java, script, PowerShell and SQL find what they
look for in some file names. This holds too in the ten query parameters that
930120, 932160 and 932260 do not check (see the fourth change under "Attack
detection" in `SPEC.md`). Examples:
- `runtime` in OpenJDK's `src/java.base/share/classes/java/lang/Runtime.java`
(944110);
- a Java class, as in `java.lang.String.md` (944130);
- `document.write.js` (941180);
- `process.env.d.ts` (934100);
- `__proto__` (934130);
- the PowerShell command `Install-Script`, as in `install-script.sh` (932120);
- a database's name, as in PostgreSQL's `information_schema.sql` (942140).
For such a file, these requests are refused:
- the API's list of its commits (`path`);
- the API's call for the contents of files that names it in its query (`body`);
- the commit graph of that file alone (`file`);
- the hidden lines of its diff (`path`), which stay hidden;
- the sign-in page, for a visitor who opens a page of the file that needs
signing in, such as its edit page (`redirect_to`). The visitor can still sign
in by opening the sign-in page itself.
### A path
- A file name ending in `~`, or an `.xhtml` file whose path holds a space.
- A path that holds what the Core Rule Set takes for a Java class (944130), as
`java.lang.String.md` and
`META-INF/services/org.apache.commons.logging.LogFactory` do.
- A path that holds what it takes for Node.js code (934100), as
`process.env.d.ts` does.
- A path with a directory named `...` (930100, 930110), such as `.../notes.txt`.
For such a file, these are refused:
- the page of the file;
- its raw file;
- its history and its blame;
- its edit page.
The directory that holds it lists it, and a clone carries it.
### A container image push that reuses a layer
A push can reuse a layer of an image docker has pushed to that gitea before.
docker then asks gitea to take the layer from that image, naming it in the query
parameter `from`, which 932260 checks. The request is refused when the image's
owner starts with an entry on the list of commands. It is refused too when the
image's name after the owner starts with one. Examples are `owner/python3-app`,
`owner/docker-base`, and any image of an owner named `ansible` or `docker-team`.
What happens next depends on docker's image store. A docker that keeps its
images in containerd's image store fails the push, and fails each new try the
same way. A new install of Docker Engine 29 does this. A docker that keeps them
in its own older image store uploads the layer instead. The push then succeeds
after one refusal for each layer it reuses, and each of those refusals counts
toward the error burst ("Attack detection" in `SPEC.md`).
### Artifact uploads from `actions/upload-artifact@v3`
This action sends the header `Content-Range`, which the Core Rule Set refuses
(920450). The action sends two files at a time, does not retry a `403`, and
sends no further file once one is refused. So an upload is refused once or
twice, however many files it holds, and the step fails. One upload does not ban
the runner. The error burst ("Attack detection" in `SPEC.md`) bans it after more
than 30 such refusals from its address in a minute, as when many jobs upload at
once. `actions/upload-artifact@v4` does not send the header.
### What body inspection refuses
With `SWWAF_WAF_BODY_LIMIT` set, the Core Rule Set reads request bodies too. In
front of gitea it then refuses these, until their rule ids, named in the request
log, are added to `SWWAF_WAF_DISABLED_RULES`:
- issue and comment text, wiki pages and files saved in the web editor that hold
shell commands or code (932125, 932235, 932250 and others);
- package descriptions that show code;
- PyPI uploads (922130);
- attachments named like `debug.log` or `config.yml` (932180).
### What `script/gitea-requests` sends
Of the requests above, the script sends:
- for each file named under "A file name in a query parameter that trips the
code rules", the API's list of its commits;
- for `Runtime.java` alone, the commit graph of that file, the hidden lines of
its diff, and the API's call for the contents of files that names it;
- the sign-in page for a visitor who opens the edit page of `document.write.js`;
- the pages of `java.lang.String.md`,
`META-INF/services/org.apache.commons.logging.LogFactory`, `process.env.d.ts`
and `.../notes.txt`;
- one container image push that reuses a layer, which it does not try again.
It does not send the raw file, history, blame or edit page of those four files,
or the pages of the directories `META-INF/services` and `...`. It also sends a
short sample of gitea's ordinary use:
- signing in;
- pages of files and commits, and a diff;
- the API;
- a push, and clone and fetch over HTTP;
- two container image pushes.
-25
View File
@@ -107,31 +107,6 @@ Every one of these lines has:
- `would_action`: in `observe` mode, what `enforce` mode would have done with a - `would_action`: in `observe` mode, what `enforce` mode would have done with a
request it would have refused. `action` then names what was done, normally request it would have refused. `action` then names what was done, normally
`forward`. `forward`.
- `heuristic`: the heuristic that refused the request or banned its client, in
`observe` mode too. It is left out when none did.
- `heuristic_figures`: an object of the figures that heuristic acted on. Each
heuristic and its figures:
- `deny_nets`: `netblock`, the entry of `SWWAF_DENY_NETS` the client is in;
- `active_ban`: `netblock`, the ban's netblock, and `ban_expires`;
- `countries`: `country`, and `setting`, the country list that refused it;
- `blocklists`: `lists`, the blocklist URLs that list the client;
- `crowdsec`: `scenario`, the decision's scenario, and `expires`, when the
decision ends;
- `dnsbl`: `zones`, the DNS blocklist zones that list the client, their keys
masked;
- `abuseipdb`: `score`, and `min_score`, `SWWAF_ABUSEIPDB_MIN_SCORE`;
- `rate_limits`: `window`, `count` and `limit`;
- `trap_paths`: `path`, the trap path asked for;
- `rule_files`: `rule_id` and `rule_action`, of the rule that refused it;
- `header_limits`: `fields`, `fields_limit`, `cookies` and `cookies_limit`;
- `core_rule_set`: `score`, and `threshold`, `SWWAF_WAF_ANOMALY_THRESHOLD`;
- `refusal_limit`: `window`, `count` and `limit`, of the error burst;
- `byte_limits`: `window`, `count` and `limit`, the count in bytes.
`refusal_limit` bans after a request refused by another heuristic has been
answered, so it replaces that heuristic on the line. `byte_limits` bans when
the report comes, so only the `report` line names it.
- `limit_percent` and `limit_percent_setting`: the client's percentage of the - `limit_percent` and `limit_percent_setting`: the client's percentage of the
rate limits and the setting that lowered it, when one did. `bytes_percent` and rate limits and the setting that lowered it, when one did. `bytes_percent` and
`bytes_percent_setting` do the same for the byte limits. `bytes_percent_setting` do the same for the byte limits.
+3 -3
View File
@@ -55,10 +55,10 @@ empty-agent user_agent log ^$
## A file of your own ## A file of your own
`servewaf` reads the rule files in its own container. Rules of your own go in a `servewaf` reads the rule files in its own container. Rules of your own go in a
file named to sort after the default, such as `50-gitea.rules`: file named to sort after the default, such as `50-site.rules`:
``` ```
# 50-gitea.rules: WordPress probes, which gitea never serves # 50-site.rules: WordPress probes, which this site never serves
wp-probe path ban (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/) wp-probe path ban (?i)^/(wp-login\.php|xmlrpc\.php|wp-admin/)
``` ```
@@ -66,7 +66,7 @@ A `Dockerfile` built on the `smallwebwaf` image, for `servewaf` to run from,
adds it: adds it:
```dockerfile ```dockerfile
COPY 50-gitea.rules /etc/smallwebwaf/rules.d/50-gitea.rules COPY 50-site.rules /etc/smallwebwaf/rules.d/50-site.rules
``` ```
Or mount it there. A `path` rule applies on every site behind that `servewaf`. Or mount it there. A `path` rule applies on every site behind that `servewaf`.
+6 -7
View File
@@ -101,19 +101,18 @@ chooses between the two (see "The image's command" below).
- `SWWAF_RATE_LIMIT_PER_MINUTE` (default `120`, 2 a second): the most requests a - `SWWAF_RATE_LIMIT_PER_MINUTE` (default `120`, 2 a second): the most requests a
client may make in a minute. A count or `off`. The default leaves room for a client may make in a minute. A count or `off`. The default leaves room for a
person opening two gitea pages at once with an empty cache, up to 60 files person opening two pages at once with an empty cache, up to 60 files each.
each.
- `SWWAF_RATE_LIMIT_PER_HOUR` (default `1500`): the most requests a client may - `SWWAF_RATE_LIMIT_PER_HOUR` (default `1500`): the most requests a client may
make in an hour. A count or `off`. It is the main limit against scrapers. A make in an hour. A count or `off`. It is the main limit against scrapers. A
person reading code in gitea for an hour, a page every 20 seconds, makes person reading a site for an hour, a page every 20 seconds, makes about 900. A
about 900. A scraper making one request every 2 seconds makes 1,800 an hour, scraper making one request every 2 seconds makes 1,800 an hour, more than the
more than the limit. limit.
- `SWWAF_RATE_LIMIT_PER_DAY` (default `6000`): the most requests a client may - `SWWAF_RATE_LIMIT_PER_DAY` (default `6000`): the most requests a client may
make in a day. A count or `off`. The default is four hours at the hour limit. make in a day. A count or `off`. The default is four hours at the hour limit.
A scraper making one request every 10 seconds makes 8,640 a day, more than the A scraper making one request every 10 seconds makes 8,640 a day, more than the
limit. limit.
- An Actions runner asks gitea for work all day and can pass the limits. Put the - A CI runner asks its code forge for work all day and can pass the limits. Put
runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. Do the same for a the runners' addresses in `SWWAF_RATE_LIMIT_EXEMPT_NETS`. Do the same for a
network where many people share one address, such as an office. network where many people share one address, such as an office.
- Counting cannot catch a client that stays under all three limits (see "Request - Counting cannot catch a client that stays under all three limits (see "Request
rate limits, per client" in [`SPEC.md`](../SPEC.md)). rate limits, per client" in [`SPEC.md`](../SPEC.md)).
+10 -10
View File
@@ -39,7 +39,7 @@ const (
ntfyURL = "https://ntfy.example/smallwebwaf-alerts" ntfyURL = "https://ntfy.example/smallwebwaf-alerts"
ntfyToken = "tk_0123456789abcdefghijklmnopq" ntfyToken = "tk_0123456789abcdefghijklmnopq"
// instance is the instance name every alert gives. // instance is the instance name every alert gives.
instance = "fsn1app1/gitea" instance = "fsn1app1/blog"
// started is when each test starts, as an alert gives it, and // started is when each test starts, as an alert gives it, and
// anHourOn an hour later. // anHourOn an hour later.
started = "2000-01-01T00:00:00Z" started = "2000-01-01T00:00:00Z"
@@ -965,8 +965,8 @@ func TestEachDestinationIsSentOnlyTheAlertsWaitingForIt(t *testing.T) {
synctest.Wait() synctest.Wait()
wantBodies(t, standIns[alerts.DestinationSlack], wantBodies(t, standIns[alerts.DestinationSlack],
`{"text":"*fsn1app1/gitea: file_error*\nsecond"}`, `{"text":"*fsn1app1/blog: file_error*\nsecond"}`,
`{"text":"*fsn1app1/gitea: file_error*\nthird"}`) `{"text":"*fsn1app1/blog: file_error*\nthird"}`)
wantBodies(t, standIns[alerts.DestinationNtfy], "fourth") wantBodies(t, standIns[alerts.DestinationNtfy], "fourth")
// alerts.json then lists Slack and ntfy alone, with no alert waiting. // alerts.json then lists Slack and ntfy alone, with no alert waiting.
@@ -1265,7 +1265,7 @@ func anAlertForEachEvent() []eventMessage {
Reason: "requests per hour over the limit of 10000", Reason: "requests per hour over the limit of 10000",
Detail: map[string]any{"mode": "observe"}, Detail: map[string]any{"mode": "observe"},
}, },
"fsn1app1/gitea: ban", "default no_entry", "fsn1app1/blog: ban", "default no_entry",
"requests per hour over the limit of 10000\nclient: 203.0.113.9\n" + "requests per hour over the limit of 10000\nclient: 203.0.113.9\n" +
"netblock: 203.0.113.0/24\ncountry: DE\nmode: observe", "netblock: 203.0.113.0/24\ncountry: DE\nmode: observe",
}, },
@@ -1275,7 +1275,7 @@ func anAlertForEachEvent() []eventMessage {
Netblock: netip.MustParsePrefix("198.51.100.7/32"), Country: "FR", Netblock: netip.MustParsePrefix("198.51.100.7/32"), Country: "FR",
Reason: "matched the rule env-file", Reason: "matched the rule env-file",
}, },
"fsn1app1/gitea: permanent_ban", "high no_entry", "fsn1app1/blog: permanent_ban", "high no_entry",
"matched the rule env-file\nclient: 198.51.100.7\n" + "matched the rule env-file\nclient: 198.51.100.7\n" +
"netblock: 198.51.100.7/32\ncountry: FR", "netblock: 198.51.100.7/32\ncountry: FR",
}, },
@@ -1286,7 +1286,7 @@ func anAlertForEachEvent() []eventMessage {
Reason: "refused by the Core Rule Set", Reason: "refused by the Core Rule Set",
Host: "app.example", Host: "app.example",
}, },
"fsn1app1/gitea: waf_block", "default shield", "fsn1app1/blog: waf_block", "default shield",
"refused by the Core Rule Set\nhost: app.example\nclient: 192.0.2.1\n" + "refused by the Core Rule Set\nhost: app.example\nclient: 192.0.2.1\n" +
"netblock: 192.0.2.1/32", "netblock: 192.0.2.1/32",
}, },
@@ -1295,7 +1295,7 @@ func anAlertForEachEvent() []eventMessage {
Event: alerts.EventAnomaly, Netblock: netip.MustParsePrefix("192.0.2.0/24"), Event: alerts.EventAnomaly, Netblock: netip.MustParsePrefix("192.0.2.0/24"),
Reason: "requests per minute over the threshold of 5000", Reason: "requests per minute over the threshold of 5000",
}, },
"fsn1app1/gitea: anomaly", "high chart_with_upwards_trend", "fsn1app1/blog: anomaly", "high chart_with_upwards_trend",
"requests per minute over the threshold of 5000\nnetblock: 192.0.2.0/24", "requests per minute over the threshold of 5000\nnetblock: 192.0.2.0/24",
}, },
{ {
@@ -1304,7 +1304,7 @@ func anAlertForEachEvent() []eventMessage {
Netblock: netip.MustParsePrefix("192.0.2.2/32"), Netblock: netip.MustParsePrefix("192.0.2.2/32"),
Reason: "listed by a DNS blocklist", Reason: "listed by a DNS blocklist",
}, },
"fsn1app1/gitea: reputation_hit", "low label", "fsn1app1/blog: reputation_hit", "low label",
"listed by a DNS blocklist\nclient: 192.0.2.2\nnetblock: 192.0.2.2/32", "listed by a DNS blocklist\nclient: 192.0.2.2\nnetblock: 192.0.2.2/32",
}, },
{ {
@@ -1312,7 +1312,7 @@ func anAlertForEachEvent() []eventMessage {
Event: alerts.EventSourceFailure, Reason: "asking GeoJS failed", Event: alerts.EventSourceFailure, Reason: "asking GeoJS failed",
Detail: map[string]any{"source": "geojs"}, Detail: map[string]any{"source": "geojs"},
}, },
"fsn1app1/gitea: source_failure", "high warning", "fsn1app1/blog: source_failure", "high warning",
"asking GeoJS failed\nsource: geojs", "asking GeoJS failed\nsource: geojs",
}, },
{ {
@@ -1323,7 +1323,7 @@ func anAlertForEachEvent() []eventMessage {
"file": "/etc/smallwebwaf/rules.d/50-app.rules", "error": "line 2: no action", "file": "/etc/smallwebwaf/rules.d/50-app.rules", "error": "line 2: no action",
}, },
}, },
"fsn1app1/gitea: file_error", "high warning", "fsn1app1/blog: file_error", "high warning",
"a rule file has an error, and the rules stay as they were\n" + "a rule file has an error, and the rules stay as they were\n" +
"file: /etc/smallwebwaf/rules.d/50-app.rules\nerror: line 2: no action", "file: /etc/smallwebwaf/rules.d/50-app.rules\nerror: line 2: no action",
}, },
+1 -1
View File
@@ -409,7 +409,7 @@ var (
errNoCertificate = errors.New("holds no PEM certificate") errNoCertificate = errors.New("holds no PEM certificate")
errNotFacility = errors.New("is not a syslog facility such as local0 or daemon") errNotFacility = errors.New("is not a syslog facility such as local0 or daemon")
errNotAppName = errors.New( errNotAppName = errors.New(
"is not 1 to 48 printable ASCII characters without a space, such as gitea") "is not 1 to 48 printable ASCII characters without a space, such as blog")
errSetTwice = errors.New("set only one of them") errSetTwice = errors.New("set only one of them")
errNotWebhookURL = errors.New( errNotWebhookURL = errors.New(
"is not an http or https URL without a user or a fragment, " + "is not an http or https URL without a user or a fragment, " +
+6 -6
View File
@@ -197,7 +197,7 @@ const (
// instance is an SWWAF_INSTANCE_NAME that is a valid app name too, and // instance is an SWWAF_INSTANCE_NAME that is a valid app name too, and
// remoteURL an SWWAF_LOG_REMOTE_URL, for the tests that send the lines. // remoteURL an SWWAF_LOG_REMOTE_URL, for the tests that send the lines.
const ( const (
instance = "fsn1app1/gitea" instance = "fsn1app1/blog"
remoteURL = "syslog+udp://192.0.2.1:514" remoteURL = "syslog+udp://192.0.2.1:514"
) )
@@ -650,11 +650,11 @@ func TestInstanceNameAndLoggedHeadersAsSet(t *testing.T) {
t.Parallel() t.Parallel()
cfg := fromEnvironment(t, environment{ cfg := fromEnvironment(t, environment{
instanceName: "fsn1app1/gitea", instanceName: "fsn1app1/blog",
logRequestHeaders: " Accept , X-Custom", logRequestHeaders: " Accept , X-Custom",
}) })
if cfg.InstanceName != "fsn1app1/gitea" || if cfg.InstanceName != "fsn1app1/blog" ||
!slices.Equal(cfg.LogRequestHeaders, []string{"accept", "x-custom"}) { !slices.Equal(cfg.LogRequestHeaders, []string{"accept", "x-custom"}) {
t.Errorf("%s is %q and %s gives %v", instanceName, cfg.InstanceName, t.Errorf("%s is %q and %s gives %v", instanceName, cfg.InstanceName,
logRequestHeaders, cfg.LogRequestHeaders) logRequestHeaders, cfg.LogRequestHeaders)
@@ -797,7 +797,7 @@ func TestInstanceNameNotAnAppNameStopsTheStartOnlyWhileSending(t *testing.T) {
want := logRemoteAppName + `: is unset, and ` + instanceName + want := logRemoteAppName + `: is unset, and ` + instanceName +
` "fsn1 app1", its default, is not 1 to 48 printable ASCII characters ` + ` "fsn1 app1", its default, is not 1 to 48 printable ASCII characters ` +
`without a space, such as gitea` `without a space, such as blog`
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want) t.Errorf("error %v, want %s", err, want)
} }
@@ -826,7 +826,7 @@ func TestAppNameSetStopsTheStartWhileSending(t *testing.T) {
}.lookupEnv) }.lookupEnv)
want := logRemoteAppName + `: "my app" is not 1 to 48 printable ASCII ` + want := logRemoteAppName + `: "my app" is not 1 to 48 printable ASCII ` +
`characters without a space, such as gitea` `characters without a space, such as blog`
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want) t.Errorf("error %v, want %s", err, want)
} }
@@ -2166,7 +2166,7 @@ func TestSettingFromFileIsCheckedAsTheSettingItself(t *testing.T) {
}.lookupEnv) }.lookupEnv)
want = logRemoteAppName + `: "my app" is not 1 to 48 printable ASCII ` + want = logRemoteAppName + `: "my app" is not 1 to 48 printable ASCII ` +
`characters without a space, such as gitea` `characters without a space, such as blog`
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want) t.Errorf("error %v, want %s", err, want)
} }
+4 -4
View File
@@ -44,7 +44,7 @@ import (
const ( const (
// started is the time a record made as a test starts gives. // started is the time a record made as a test starts gives.
started = "2000-01-01T00:00:00.000Z" started = "2000-01-01T00:00:00.000Z"
appName = "fsn1app1/gitea" appName = "fsn1app1/blog"
// local0 is the number of the default facility, and local0Info the // local0 is the number of the default facility, and local0Info the
// priority of its records. // priority of its records.
local0 = 16 local0 = 16
@@ -94,7 +94,7 @@ func TestRecordsOverTCPAreOctetCountedWithTheirFacility(t *testing.T) {
endpoint := listen(t) endpoint := listen(t)
endpointParams := params(remotelog.SchemeTCP, endpoint.Addr()) endpointParams := params(remotelog.SchemeTCP, endpoint.Addr())
endpointParams.Facility = 19 // local3 endpointParams.Facility = 19 // local3
endpointParams.AppName = "gitea" endpointParams.AppName = "blog"
sender, _, _ := run(t, endpointParams) sender, _, _ := run(t, endpointParams)
_, _ = sender.Write([]byte("first\nsecond\n")) _, _ = sender.Write([]byte("first\nsecond\n"))
@@ -103,8 +103,8 @@ func TestRecordsOverTCPAreOctetCountedWithTheirFacility(t *testing.T) {
wantCounts(t, sender, 2, 0, 0) wantCounts(t, sender, 2, 0, 0)
frames := bufio.NewReader(accept(t, endpoint)) frames := bufio.NewReader(accept(t, endpoint))
wantFrame(t, frames, record(t, "<158>", "gitea", "first")) wantFrame(t, frames, record(t, "<158>", "blog", "first"))
wantFrame(t, frames, record(t, "<158>", "gitea", "second")) wantFrame(t, frames, record(t, "<158>", "blog", "second"))
}) })
} }
-5
View File
@@ -131,11 +131,6 @@ type Line struct {
// ActionRuleBlocked, ActionWAFBlocked, or, in servewaf, ActionTooLarge // ActionRuleBlocked, ActionWAFBlocked, or, in servewaf, ActionTooLarge
// for the header limits. // for the header limits.
WouldAction string `json:"would_action,omitempty"` WouldAction string `json:"would_action,omitempty"`
// Heuristic is the name of the heuristic that refused the request or
// banned its client, in observe mode too, and HeuristicFigures are the
// figures it acted on, by name.
Heuristic string `json:"heuristic,omitempty"`
HeuristicFigures map[string]any `json:"heuristic_figures,omitempty"`
// LimitPercent and LimitPercentSetting are, for a request the rate // LimitPercent and LimitPercentSetting are, for a request the rate
// limits counted whose client a biased threshold gives a percentage of // limits counted whose client a biased threshold gives a percentage of
// the rate limits below 100, that percentage and the setting that gave // the rate limits below 100, that percentage and the setting that gave
+3 -3
View File
@@ -94,7 +94,7 @@ func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
var out bytes.Buffer var out bytes.Buffer
requestlog.NewProcessLogger(&out, "fsn1app1/gitea", slog.LevelInfo).Info("starting", requestlog.NewProcessLogger(&out, "fsn1app1/blog", slog.LevelInfo).Info("starting",
"version", "v1") "version", "v1")
var fields map[string]any var fields map[string]any
@@ -104,7 +104,7 @@ func TestProcessLinesAreMarkedProcessAndGiveTheInstance(t *testing.T) {
t.Fatalf("decode %q: %v", out.String(), err) t.Fatalf("decode %q: %v", out.String(), err)
} }
if fields["type"] != "process" || fields["instance"] != "fsn1app1/gitea" || if fields["type"] != "process" || fields["instance"] != "fsn1app1/blog" ||
fields["msg"] != "starting" || fields["level"] != "INFO" || fields["msg"] != "starting" || fields["level"] != "INFO" ||
fields["version"] != "v1" { fields["version"] != "v1" {
t.Errorf("process line %v", fields) t.Errorf("process line %v", fields)
@@ -133,7 +133,7 @@ func TestProcessLoggerWritesTheMessagesAtItsLevelOrMoreSevere(t *testing.T) {
var out bytes.Buffer var out bytes.Buffer
processLog := requestlog.NewProcessLogger(&out, "fsn1app1/gitea", level) processLog := requestlog.NewProcessLogger(&out, "fsn1app1/blog", level)
for _, at := range levels { for _, at := range levels {
processLog.Log(t.Context(), at, "message") processLog.Log(t.Context(), at, "message")
} }
-55
View File
@@ -1,55 +0,0 @@
package servewaf
// The heuristic abuseipdb. It reads SWWAF_ABUSEIPDB_KEY,
// SWWAF_ABUSEIPDB_MIN_SCORE and SWWAF_REPUTATION_ACTION.
import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// abuseIPDB notes AbuseIPDB, as noteHit does, with the score, when its score
// of the client is a hit, and refuses the request when SWWAF_REPUTATION_ACTION
// is deny, as dnsbl does for a zone. While SWWAF_ABUSEIPDB_KEY is unset it does
// nothing. A client without a score is checked in the background, by the
// request's address, if its history counts an offence, and the request does
// not wait for the answer. The score is then used for each address of the
// client. Its figures are the score and SWWAF_ABUSEIPDB_MIN_SCORE.
type abuseIPDB struct{}
func (abuseIPDB) name() string { return "abuseipdb" }
func (abuseIPDB) check(ctx context.Context, rq *request, _ time.Time) *verdict {
if rq.h.config.AbuseIPDBKey == "" {
return nil
}
client := rq.h.clientGroup(rq.client)
held, _ := rq.h.limiter.Client(client)
offender := held.History.Offences != ratelimit.Offences{}
score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender)
if !hit {
return nil
}
rq.abuseIPDBHit = true
rq.noteHit(bans.ReputationHit{Source: reputation.AbuseIPDBSource, Score: &score},
"scored by AbuseIPDB at or over SWWAF_ABUSEIPDB_MIN_SCORE")
if rq.h.config.ReputationAction != deny {
return nil
}
return &verdict{
action: requestlog.ActionDenied,
figures: map[string]any{
figureScore: score, "min_score": rq.h.config.AbuseIPDBMinScore,
},
}
}
+1 -1
View File
@@ -18,7 +18,7 @@ const (
alertWebhookURL = "SWWAF_ALERT_WEBHOOK_URL" alertWebhookURL = "SWWAF_ALERT_WEBHOOK_URL"
alertMaxPerHour = "SWWAF_ALERT_MAX_PER_HOUR" alertMaxPerHour = "SWWAF_ALERT_MAX_PER_HOUR"
// alertInstance is the instance every alert of these tests gives. // alertInstance is the instance every alert of these tests gives.
alertInstance = "fsn1app1/gitea" alertInstance = "fsn1app1/blog"
) )
func TestBanForABrokenLimitRaisesABanAlertWithItsNotes(t *testing.T) { func TestBanForABrokenLimitRaisesABanAlertWithItsNotes(t *testing.T) {
-12
View File
@@ -91,18 +91,6 @@ func TestSixteenThousandCookiesAreRefusedWellWithinTheTimeLimit(t *testing.T) {
} }
} }
func TestHeaderLimitRefusalNamesItsHeuristicCountsAndLimits(t *testing.T) {
t.Parallel()
s := startServeWAF(t, "", nil)
req := s.question(siteA, client, "/")
fillHeaders(req, headerFieldLimit+1)
do(t, req)
wantHeuristic(t, s.out.requestLine(t), "header_limits",
`{"fields": 101, "fields_limit": 100, "cookies": 0, "cookies_limit": 50}`)
}
func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) { func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
t.Parallel() t.Parallel()
+80 -24
View File
@@ -1,10 +1,7 @@
package servewaf package servewaf
// The heuristic active_ban, and the code that makes bans, which several
// heuristics share. active_ban reads SWWAF_MODE; the bans are the ban ledger's.
import ( import (
"context" "net/http"
"net/netip" "net/netip"
"time" "time"
@@ -21,15 +18,11 @@ func (rq *request) banResponse(action string) *refusal {
return &refusal{status: rq.h.config.BanResponse, action: action} return &refusal{status: rq.h.config.BanResponse, action: action}
} }
// activeBan refuses a client a ban covers: a ban on a netblock the client is // banned reports whether a ban on a netblock the client is in covers the
// in, in force at now. It notes for the log line when that ban ends. A request // request at now, and notes for the log line when that ban ends. A
// that makes the ban permanent, or in observe mode would have, raises the alert // request that makes the ban permanent, or in observe mode would have,
// for it. Its figures are the ban's netblock and when it ends. // raises the alert for it.
type activeBan struct{} func (rq *request) banned(now time.Time) bool {
func (activeBan) name() string { return "active_ban" }
func (activeBan) check(_ context.Context, rq *request, now time.Time) *verdict {
check := rq.h.ledger.Check check := rq.h.ledger.Check
if rq.h.config.Observe { if rq.h.config.Observe {
check = rq.h.ledger.Find // the ban refuses nothing, and stays as it is check = rq.h.ledger.Find // the ban refuses nothing, and stays as it is
@@ -45,22 +38,85 @@ func (activeBan) check(_ context.Context, rq *request, now time.Time) *verdict {
rq.alertBan(ban) rq.alertBan(ban)
} }
if !banned { return banned
return nil }
// limitBroken counts the request for the rate limits at now, notes the
// client's counts for the log line, and reports whether the request takes
// the client over a rate limit, as its limit percentage lowers it, which
// breaks it.
func (rq *request) limitBroken(now time.Time) bool {
counts, hit, over := rq.h.limiter.Count(rq.h.clientGroup(rq.client), now,
rq.limitPercent.percent)
rq.line.Counts = counts
if over {
rq.banForLimit(now, hit, rq.h.config.BanResponse)
} }
return &verdict{ return over
action: requestlog.ActionBanned, }
figures: map[string]any{
"netblock": ban.Netblock.String(), "ban_expires": rq.line.BanExpires, // countBytes counts the request's bytes, as countedBytes gives them, for
}, // the byte limits, once its report has come, and notes the client's byte totals
// for the log line; its requests stay there as the rate limits counted them.
// Only an allowed request has them counted, and only one the rate limits
// counted; in observe mode, not one that enforce mode would have refused. Bytes
// that take the client over a byte limit, as its limit percentage for the byte
// limits lowers it, break it; the response was passed on whole.
func (rq *request) countBytes() {
if !rq.counted || rq.line.WouldAction != "" {
return
}
now := rq.h.now()
counts, hit, over := rq.h.limiter.CountBytes(rq.h.clientGroup(rq.client), now,
rq.countedBytes(), rq.bytesPercent.percent)
rq.line.Counts.MinuteBytes = counts.MinuteBytes
rq.line.Counts.HourBytes = counts.HourBytes
rq.line.Counts.DayBytes = counts.DayBytes
if over {
rq.banForLimit(now, hit, rq.out.status)
} }
} }
// limitFigures are the figures of a heuristic that broke hit, a limit: its // countRefusal counts the request for the error burst once it has been
// window, the count and the limit. // answered, if servewaf refused it after a rule file match, a trap path or a
func limitFigures(hit ratelimit.Hit) map[string]any { // Core Rule Set match, and in observe mode if enforce mode would have: more than
return map[string]any{"window": hit.Window, "count": hit.Count, figureLimit: hit.Limit} // SWWAF_ERROR_BURST_THRESHOLD such refusals of the client within a minute
// break a limit. A client in SWWAF_ALLOW_NETS, which the checks skip, is
// not counted, and nothing is while the threshold is off.
func (rq *request) countRefusal() {
cfg := rq.h.config
if cfg.ErrorBurstThreshold == 0 {
return
}
if !rq.attack && !rq.ruleBlocked && !rq.wafBlocked {
return
}
now := rq.h.now()
hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now,
cfg.ErrorBurstThreshold)
if !over {
return
}
// What the client was sent, or in observe mode would have been.
status := rq.out.status
switch rq.line.WouldAction {
case requestlog.ActionRuleBlocked, requestlog.ActionWAFBlocked:
status = http.StatusForbidden
case requestlog.ActionBanned:
status = cfg.BanResponse
}
rq.banForLimit(now, hit, status)
} }
// countedBytes returns the request's bytes, once it has ended, as the // countedBytes returns the request's bytes, once it has ended, as the
-16
View File
@@ -228,22 +228,6 @@ func TestBannedClientIsRefusedBeforeItsCountryIsLookedUp(t *testing.T) {
} }
} }
func TestRefusalUnderABanNamesItsHeuristicNetblockAndEnd(t *testing.T) {
t.Parallel()
s, clk, _ := startWithClock(t, "", map[string]string{
rateLimitPerMinute: "1", banScopeV4Prefix: "24",
})
expires := requestlog.FormatTime(clk.Now().Add(time.Hour))
s.get(client, http.StatusOK, requestlog.ActionForward)
s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
line := s.get(otherClient, http.StatusForbidden, requestlog.ActionBanned)
wantHeuristic(t, line, "active_ban",
`{"netblock": "`+clientsNet+`", "ban_expires": "`+expires+`"}`)
}
func TestBanResponseAnswersEveryRefusalButTheSizeLimits(t *testing.T) { func TestBanResponseAnswersEveryRefusalButTheSizeLimits(t *testing.T) {
t.Parallel() t.Parallel()
-34
View File
@@ -1,34 +0,0 @@
package servewaf
// The heuristic blocklists. It reads SWWAF_BLOCKLIST_ACTION, and the blocklists
// through the reputation lists.
import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// blocklists notes the blocklists that list the client, as noteListed does,
// and refuses the request when SWWAF_BLOCKLIST_ACTION is deny. Being limit, it
// lowers the client's limits instead (see limitPercentages), and being log, it
// does nothing more. Its figure is the blocklists that list the client.
type blocklists struct{}
func (blocklists) name() string { return "blocklists" }
func (blocklists) check(_ context.Context, rq *request, _ time.Time) *verdict {
listedBy := rq.h.lists.ListedBy(rq.client)
rq.blocklisted = len(listedBy) > 0
rq.noteListed(listedBy, "listed by a blocklist")
if !rq.blocklisted || rq.h.config.BlocklistAction != deny {
return nil
}
return &verdict{
action: requestlog.ActionDenied,
figures: map[string]any{"lists": listedBy},
}
}
-43
View File
@@ -1,43 +0,0 @@
package servewaf
// The heuristic byte_limits. It reads SWWAF_BYTES_COUNT, and the byte limits
// and biased thresholds through the limiter and limitPercentages.
import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// byteLimits counts the request's bytes, as countedBytes gives them, for the
// byte limits, once its report has come, and notes the client's byte totals for
// the log line; its requests stay there as the rate limits counted them. Only
// an allowed request has them counted, and only one the rate limits counted; in
// observe mode, not one that enforce mode would have refused. Bytes that take
// the client over a byte limit, as its limit percentage for the byte limits
// lowers it, break it, which bans the client; the response was passed on whole.
// Its figures are the window, the count and the limit.
type byteLimits struct{}
func (byteLimits) name() string { return "byte_limits" }
func (byteLimits) check(_ context.Context, rq *request, now time.Time) *verdict {
if !rq.counted || rq.line.WouldAction != "" {
return nil
}
counts, hit, over := rq.h.limiter.CountBytes(rq.h.clientGroup(rq.client), now,
rq.countedBytes(), rq.bytesPercent.percent)
rq.line.Counts.MinuteBytes = counts.MinuteBytes
rq.line.Counts.HourBytes = counts.HourBytes
rq.line.Counts.DayBytes = counts.DayBytes
if !over {
return nil
}
rq.banForLimit(now, hit, rq.out.status)
return &verdict{action: requestlog.ActionBanned, figures: limitFigures(hit)}
}
-14
View File
@@ -98,20 +98,6 @@ func TestEachByteLimitBansOnceTheResponseHasEnded(t *testing.T) {
} }
} }
func TestByteLimitBanNamesItsHeuristicWindowCountAndLimit(t *testing.T) {
t.Parallel()
s, _ := startWithAnswers(t, map[string]string{bytesLimitPerMinute: byteLimit})
line, _ := s.download()
wantHeuristic(t, line, "", "null")
// Two answers of 70 bytes are over the limit of 99.
line, _ = s.download()
wantHeuristic(t, line, "byte_limits",
`{"window": "minute", "count": 140, "limit": 99}`)
}
func TestResponseOverAByteLimitByItselfIsPassedOnWhole(t *testing.T) { func TestResponseOverAByteLimitByItselfIsPassedOnWhole(t *testing.T) {
t.Parallel() t.Parallel()
+9 -19
View File
@@ -1,10 +1,6 @@
package servewaf package servewaf
// The heuristic core_rule_set. It reads SWWAF_WAF_MODE, SWWAF_WAF_EXEMPT_PATHS
// and SWWAF_WAF_ANOMALY_THRESHOLD.
import ( import (
"context"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
@@ -13,21 +9,18 @@ import (
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
) )
// coreRuleSet inspects the request with the Core Rule Set, unless // checkCoreRuleSet inspects the request with the Core Rule Set, unless
// SWWAF_WAF_MODE is off or SWWAF_WAF_EXEMPT_PATHS exempts its path, as // SWWAF_WAF_MODE is off or SWWAF_WAF_EXEMPT_PATHS exempts its path, as
// pathExempt decides, and notes the rules it matched and its score in the // pathExempt decides, and notes the rules it matched and its score in the
// log line, and the rules in the metrics. A score at or over // log line, and the rules in the metrics. A score at or over
// SWWAF_WAF_ANOMALY_THRESHOLD is a match: it raises the waf_block alert, // SWWAF_WAF_ANOMALY_THRESHOLD is a match: it raises the waf_block alert,
// and in block mode refuses the request with 403, which is an offence its // and in block mode refuses the request, which is an offence its client's
// client's history counts. Its figures are the score and the threshold. // history counts, and so returns ActionWAFBlocked. It returns "" for a
type coreRuleSet struct{} // request it does not refuse.
func (rq *request) checkCoreRuleSet() string {
func (coreRuleSet) name() string { return "core_rule_set" }
func (coreRuleSet) check(_ context.Context, rq *request, _ time.Time) *verdict {
cfg := rq.h.config cfg := rq.h.config
if cfg.WAFMode == config.WAFModeOff || pathExempt(rq.in, cfg.WAFExemptPaths) { if cfg.WAFMode == config.WAFModeOff || pathExempt(rq.in, cfg.WAFExemptPaths) {
return nil return ""
} }
start := time.Now() start := time.Now()
@@ -47,21 +40,18 @@ func (coreRuleSet) check(_ context.Context, rq *request, _ time.Time) *verdict {
threshold := cfg.WAFAnomalyThreshold threshold := cfg.WAFAnomalyThreshold
if threshold == 0 || result.Score < threshold { if threshold == 0 || result.Score < threshold {
return nil return ""
} }
rq.alertWAFBlock(result) rq.alertWAFBlock(result)
if cfg.WAFMode == config.WAFModeDetect { if cfg.WAFMode == config.WAFModeDetect {
return nil return ""
} }
rq.wafBlocked = true rq.wafBlocked = true
return &verdict{ return requestlog.ActionWAFBlocked
action: requestlog.ActionWAFBlocked,
figures: map[string]any{figureScore: result.Score, "threshold": threshold},
}
} }
// alertWAFBlock raises the waf_block alert for the request, which the Core // alertWAFBlock raises the waf_block alert for the request, which the Core
-10
View File
@@ -110,16 +110,6 @@ func TestCoreRuleSetRefusesAttacksInBlockModeAndOnlyLogsThemInDetectMode(t *test
} }
} }
func TestCoreRuleSetRefusalNamesItsHeuristicScoreAndThreshold(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(t, "", map[string]string{wafMode: block})
line := s.request(client, sqlInjection, http.StatusForbidden,
requestlog.ActionWAFBlocked)
wantHeuristic(t, line, "core_rule_set", `{"score": 5, "threshold": 5}`)
}
func TestOrdinaryRequestIsInspectedAndPassed(t *testing.T) { func TestOrdinaryRequestIsInspectedAndPassed(t *testing.T) {
t.Parallel() t.Parallel()
+8 -29
View File
@@ -1,42 +1,21 @@
package servewaf package servewaf
// The heuristic countries. It reads SWWAF_DENIED_COUNTRIES and
// SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
import ( import (
"context"
"slices" "slices"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
// countries refuses a request by the client's country as it was looked up. A // countryDenied reports whether the country lists refuse the request, by
// client without a country, or whose country cannot be found, is refused only // the client's country as it was looked up. A client without a country,
// by SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES. Its figures are the country and the // or whose country cannot be found, is refused only by
// setting that refused it. // SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
type countries struct{} func (rq *request) countryDenied() bool {
func (countries) name() string { return "countries" }
func (countries) check(_ context.Context, rq *request, _ time.Time) *verdict {
denied := rq.h.config.DeniedCountries denied := rq.h.config.DeniedCountries
allowed := rq.h.config.ExclusivelyAllowedCountries allowed := rq.h.config.ExclusivelyAllowedCountries
country := rq.line.Country country := rq.line.Country
var setting string if slices.Contains(denied, country) {
return true
switch {
case slices.Contains(denied, country):
setting = "SWWAF_DENIED_COUNTRIES"
case len(allowed) > 0 && !slices.Contains(allowed, country):
setting = "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"
default:
return nil
} }
return &verdict{ return len(allowed) > 0 && !slices.Contains(allowed, country)
action: requestlog.ActionCountryDenied,
figures: map[string]any{"country": country, "setting": setting},
}
} }
-17
View File
@@ -90,23 +90,6 @@ func TestCountryLists(t *testing.T) {
} }
} }
func TestCountryRefusalNamesItsHeuristicCountryAndSetting(t *testing.T) {
t.Parallel()
geojsURL, _ := startGeoJS(t)
s, _, _ := startWithClock(t, geojsURL, map[string]string{
lookupTimeout: "1h", deniedCountries: "kp", allowedCountries: "de",
})
line := s.get(fromKP, http.StatusForbidden, requestlog.ActionCountryDenied)
wantHeuristic(t, line, "countries",
`{"country": "KP", "setting": "SWWAF_DENIED_COUNTRIES"}`)
line = s.get(unplaced, http.StatusForbidden, requestlog.ActionCountryDenied)
wantHeuristic(t, line, "countries",
`{"country": "", "setting": "SWWAF_EXCLUSIVELY_ALLOWED_COUNTRIES"}`)
}
func TestCountryRefusalKeepsARequestFromTheApp(t *testing.T) { func TestCountryRefusalKeepsARequestFromTheApp(t *testing.T) {
t.Parallel() t.Parallel()
-39
View File
@@ -1,39 +0,0 @@
package servewaf
// The heuristic crowdsec. It reads SWWAF_CROWDSEC_LAPI_URL, and the decisions
// through the reputation lists.
import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// crowdSec refuses a request when a decision of the CrowdSec decision list on
// the client is in force at now. It then notes the list, as noteHit does, and
// bans the client until that decision ends. Its figures are the decision's
// scenario and when it ends.
type crowdSec struct{}
func (crowdSec) name() string { return "crowdsec" }
func (crowdSec) check(_ context.Context, rq *request, now time.Time) *verdict {
decision, listed := rq.h.lists.CrowdSecDecision(rq.client, now)
if !listed {
return nil
}
rq.noteHit(bans.ReputationHit{Source: rq.h.config.CrowdSecDecisionsURL},
"listed by the CrowdSec decision list")
rq.banForCrowdSec(now, decision)
return &verdict{
action: requestlog.ActionBanned,
figures: map[string]any{
"scenario": decision.Scenario,
"expires": requestlog.FormatTime(decision.Expires),
},
}
}
-15
View File
@@ -95,21 +95,6 @@ func TestClientTheCrowdSecDecisionListListsIsBannedUntilTheDecisionEnds(t *testi
float64(fetched.Unix())) float64(fetched.Unix()))
} }
func TestCrowdSecBanNamesItsHeuristicScenarioAndEnd(t *testing.T) {
t.Parallel()
s, clk, server, _ := startWithAlerts(t, map[string]string{
crowdSecURL: lapi, crowdSecKey: bouncerKey,
})
loadDecisions(t, server, clk.Now(), `[{"duration": "4h0m0s", `+
`"scenario": "crowdsecurity/ssh-bf", "scope": "Ip", "type": "ban", `+
`"value": "`+client+`"}]`)
line := s.get(client, http.StatusForbidden, requestlog.ActionBanned)
wantHeuristic(t, line, "crowdsec", `{"scenario": "crowdsecurity/ssh-bf", `+
`"expires": "`+requestlog.FormatTime(clk.Now().Add(4*time.Hour))+`"}`)
}
func TestEndedCrowdSecDecisionNoLongerBansThoughTheCopyStillHoldsIt(t *testing.T) { func TestEndedCrowdSecDecisionNoLongerBansThoughTheCopyStillHoldsIt(t *testing.T) {
t.Parallel() t.Parallel()
-29
View File
@@ -1,29 +0,0 @@
package servewaf
// The heuristic deny_nets. It reads SWWAF_DENY_NETS.
import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// denyNets refuses a client in SWWAF_DENY_NETS. Its figure is the netblock of
// the setting the client is in.
type denyNets struct{}
func (denyNets) name() string { return "deny_nets" }
func (denyNets) check(_ context.Context, rq *request, _ time.Time) *verdict {
for _, netblock := range rq.h.config.DenyNets {
if netblock.Contains(rq.client) {
return &verdict{
action: requestlog.ActionDenied,
figures: map[string]any{"netblock": netblock.String()},
}
}
}
return nil
}
-36
View File
@@ -1,36 +0,0 @@
package servewaf
// The heuristic dnsbl. It reads SWWAF_REPUTATION_ACTION, and the DNSBL zones'
// verdicts through the DNSBL checker.
import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// dnsbl notes the DNSBL zones whose verdict lists the client, as noteListed
// does, and refuses the request when SWWAF_REPUTATION_ACTION is deny. Being
// limit, it lowers the client's limits instead (see limitPercentages), and
// being log, it does nothing more. A zone without a verdict on the client is
// asked about it in the background, and the request does not wait for the
// answer. Its figure is the zones that list the client, their keys masked.
type dnsbl struct{}
func (dnsbl) name() string { return "dnsbl" }
func (dnsbl) check(ctx context.Context, rq *request, _ time.Time) *verdict {
listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client)
rq.dnsblListed = len(listedBy) > 0
rq.noteListed(listedBy, "listed by a DNSBL zone")
if !rq.dnsblListed || rq.h.config.ReputationAction != deny {
return nil
}
return &verdict{
action: requestlog.ActionDenied,
figures: map[string]any{"zones": listedBy},
}
}
@@ -106,20 +106,6 @@ func TestErrorBurstBreaksAtOneOverTheThreshold(t *testing.T) {
} }
} }
func TestRefusalLimitBanNamesItsHeuristicWindowCountAndLimit(t *testing.T) {
t.Parallel()
s, _, _ := startForErrorBurst(t, "1", nil)
line := blockRule.send(s, client)
wantHeuristic(t, line, "rule_files", `{"rule_id": "blocked", "rule_action": "block"}`)
// The second refusal is the request refusal_limit bans for, and the line
// names it instead of rule_files.
line = blockRule.send(s, client)
wantHeuristic(t, line, "refusal_limit", `{"window": "minute", "count": 2, "limit": 1}`)
}
func TestErrorBurstBanNotesHistoryAndMetrics(t *testing.T) { func TestErrorBurstBanNotesHistoryAndMetrics(t *testing.T) {
t.Parallel() t.Parallel()
-73
View File
@@ -1,73 +0,0 @@
package servewaf
// The heuristic header_limits. It reads no setting: its limits are fixed.
import (
"context"
"strings"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// The header limits: the most cookies and header fields a request may have
// when the checks reach the Core Rule Set. Real clients never reach them.
const (
maxCookies = 50
maxHeaderFields = 100
)
// headerLimits refuses a request over the header limits, with 431. Its figures
// are the request's header fields and cookies, and the limit of each.
type headerLimits struct{}
func (headerLimits) name() string { return "header_limits" }
func (headerLimits) check(_ context.Context, rq *request, _ time.Time) *verdict {
fields, cookies := rq.headerCounts()
if fields <= maxHeaderFields && cookies <= maxCookies {
return nil
}
return &verdict{
action: requestlog.ActionTooLarge,
figures: map[string]any{
"fields": fields, "fields_limit": maxHeaderFields,
"cookies": cookies, "cookies_limit": maxCookies,
},
}
}
// overHeaderLimits reports whether the request is over the header limits: it
// has more than maxHeaderFields fields or more than maxCookies cookies, as
// headerCounts counts them.
func (rq *request) overHeaderLimits() bool {
fields, cookies := rq.headerCounts()
return fields > maxHeaderFields || cookies > maxCookies
}
// headerCounts returns the request's header fields, each value of a header sent
// twice being one, and its cookies, each part of a Cookie header between
// semicolons being one unless it is empty or only spaces and tabs. The Core
// Rule Set runs each of its rules on each cookie, so that thousands of small
// cookies keep it busy for over a second. It trims only spaces and control
// characters, of which a header value can hold only tabs, so a part of Unicode
// spaces is a cookie to it.
func (rq *request) headerCounts() (int, int) {
fields, cookies := 0, 0
for _, values := range rq.in.Header {
fields += len(values)
}
for _, value := range rq.in.Header.Values("Cookie") {
for part := range strings.SplitSeq(value, ";") {
if strings.Trim(part, " \t") != "" {
cookies++
}
}
}
return fields, cookies
}
-104
View File
@@ -1,104 +0,0 @@
package servewaf
import (
"context"
"time"
)
// To add a heuristic, write one file with a type that implements heuristic,
// and add one line for it to one of the lists below. Its settings go in
// config.Config, and the comment at the top of its file names them.
// heuristic is one of the checks that decide what servewaf does with a request.
type heuristic interface {
// name is the heuristic's name, which the log line gives.
name() string
// check looks at the request at now, and returns nil to let it through.
// ctx is the request's own context.
check(ctx context.Context, rq *request, now time.Time) *verdict
}
// verdict is a heuristic refusing a request, or banning its client: the action
// the log line names, one of requestlog's Action constants, and the figures the
// heuristic acted on, by name, such as the count and the limit.
type verdict struct {
action string
figures map[string]any
}
// The names of the figures that more than one heuristic gives.
const (
figureLimit = "limit"
figureScore = "score"
)
// The lists of heuristics, each in the order its heuristics run. A client in
// SWWAF_ALLOW_NETS skips them all.
// beforeLookup are the heuristics that run before the client's AS number and
// country are looked up, so that a client they refuse is not looked up.
func beforeLookup() []heuristic {
return []heuristic{
denyNets{},
activeBan{},
}
}
// afterLookup are the heuristics that run once the lookup has been made.
func afterLookup() []heuristic {
return []heuristic{
countries{},
blocklists{},
crowdSec{},
dnsbl{},
abuseIPDB{},
rateLimits{},
trapPaths{},
ruleFiles{},
headerLimits{},
coreRuleSet{},
}
}
// afterAnswer are the heuristics that run on a request once servewaf has
// answered it, before its decision line is written. They ban the client, and do
// not change what the request got: their verdict's action is not used.
func afterAnswer() []heuristic {
return []heuristic{
refusalLimit{},
}
}
// onReport are the heuristics that run on an allowed request when rproxy's
// report of it comes, before its report line is written. Like afterAnswer, they
// only ban.
func onReport() []heuristic {
return []heuristic{
byteLimits{},
}
}
// runAfterAnswer runs the heuristics of afterAnswer on the request, at the time
// it is called.
func (rq *request) runAfterAnswer(ctx context.Context) {
rq.runHeuristics(ctx, afterAnswer(), rq.h.now())
}
// runHeuristics runs list on the request at now, in order, and returns the
// first verdict, or nil when none gives one. The log line names the heuristic
// that gave it, and its figures. ctx is the request's own context.
func (rq *request) runHeuristics(
ctx context.Context, list []heuristic, now time.Time,
) *verdict {
for _, each := range list {
v := each.check(ctx, rq, now)
if v != nil {
rq.line.Heuristic = each.name()
rq.line.HeuristicFigures = v.figures
return v
}
}
return nil
}
-12
View File
@@ -14,18 +14,6 @@ import (
// observe is the value of SWWAF_MODE for observe mode. // observe is the value of SWWAF_MODE for observe mode.
const observe = "observe" const observe = "observe"
func TestObserveModeNamesTheHeuristicThatWouldHaveRefused(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(t, "", map[string]string{
mode: observe, denyNets: client,
})
line := s.get(client, http.StatusOK, requestlog.ActionForward)
wantWouldAction(t, line, requestlog.ActionDenied)
wantHeuristic(t, line, "deny_nets", `{"netblock": "`+client+`/32"}`)
}
func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) { func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
t.Parallel() t.Parallel()
-56
View File
@@ -1,56 +0,0 @@
package servewaf
// The heuristic rate_limits. It reads SWWAF_RATE_LIMIT_EXEMPT_NETS,
// SWWAF_RATE_LIMIT_EXEMPT_PATHS and SWWAF_BAN_RESPONSE, and the rate limits and
// biased thresholds through the limiter and limitPercentages.
import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// rateLimits counts the request for the rate limits, unless
// countedByRateLimits exempts it, by the client's limit percentages, which it
// notes for the log line with the client's counts. A request that takes the
// client over a rate limit, as its limit percentage lowers it, breaks it: it is
// refused, and bans the client. A request exempt from the rate limits is exempt
// from the byte limits too. Its figures are the window, the count and the
// limit.
type rateLimits struct{}
func (rateLimits) name() string { return "rate_limits" }
func (rateLimits) check(_ context.Context, rq *request, now time.Time) *verdict {
rq.counted = rq.countedByRateLimits()
if !rq.counted {
return nil
}
rq.limitPercent, rq.bytesPercent = rq.limitPercentages()
rq.line.LimitPercent, rq.line.LimitPercentSetting = rq.limitPercent.logged()
rq.line.BytesPercent, rq.line.BytesPercentSetting = rq.bytesPercent.logged()
counts, hit, over := rq.h.limiter.Count(rq.h.clientGroup(rq.client), now,
rq.limitPercent.percent)
rq.line.Counts = counts
if !over {
return nil
}
rq.banForLimit(now, hit, rq.h.config.BanResponse)
return &verdict{action: requestlog.ActionRateLimited, figures: limitFigures(hit)}
}
// countedByRateLimits reports whether the rate limits count the request: its
// client is not in SWWAF_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt
// under SWWAF_RATE_LIMIT_EXEMPT_PATHS.
func (rq *request) countedByRateLimits() bool {
cfg := rq.h.config
return !isInside(rq.client, cfg.RateLimitExemptNets) &&
!pathExempt(rq.in, cfg.RateLimitExemptPaths)
}
-11
View File
@@ -72,17 +72,6 @@ func TestRateLimitRefusesBeforeTheApp(t *testing.T) {
} }
} }
func TestRateLimitRefusalNamesItsHeuristicWindowCountAndLimit(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(t, "", map[string]string{rateLimitPerMinute: "1"})
s.get(client, http.StatusOK, requestlog.ActionForward)
line := s.get(client, http.StatusForbidden, requestlog.ActionRateLimited)
wantHeuristic(t, line, "rate_limits", `{"window": "minute", "count": 2, "limit": 1}`)
}
func TestIPv6GroupPrefixSetsTheClientTheLimitsCount(t *testing.T) { func TestIPv6GroupPrefixSetsTheClientTheLimitsCount(t *testing.T) {
t.Parallel() t.Parallel()
-54
View File
@@ -1,54 +0,0 @@
package servewaf
// The heuristic refusal_limit. It reads SWWAF_ERROR_BURST_THRESHOLD and
// SWWAF_BAN_RESPONSE.
import (
"context"
"net/http"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
// refusalLimit counts the request once it has been answered, if servewaf
// refused it after a rule file match, a trap path or a Core Rule Set match, and
// in observe mode if enforce mode would have: more than
// SWWAF_ERROR_BURST_THRESHOLD such refusals of the client within a minute break
// a limit, which bans the client. A client in SWWAF_ALLOW_NETS, which the
// heuristics skip, is not counted, and nothing is while the threshold is off.
// Its figures are the window, the count and the limit.
type refusalLimit struct{}
func (refusalLimit) name() string { return "refusal_limit" }
func (refusalLimit) check(_ context.Context, rq *request, now time.Time) *verdict {
cfg := rq.h.config
if cfg.ErrorBurstThreshold == 0 {
return nil
}
if !rq.attack && !rq.ruleBlocked && !rq.wafBlocked {
return nil
}
hit, over := rq.h.limiter.CountRefusal(rq.h.clientGroup(rq.client), now,
cfg.ErrorBurstThreshold)
if !over {
return nil
}
// What the client was sent, or in observe mode would have been.
status := rq.out.status
switch rq.line.WouldAction {
case requestlog.ActionRuleBlocked, requestlog.ActionWAFBlocked:
status = http.StatusForbidden
case requestlog.ActionBanned:
status = cfg.BanResponse
}
rq.banForLimit(now, hit, status)
return &verdict{action: requestlog.ActionBanned, figures: limitFigures(hit)}
}
+77 -3
View File
@@ -1,17 +1,91 @@
package servewaf package servewaf
// What the reputation heuristics share: blocklists, crowdsec, dnsbl and
// abuseipdb.
import ( import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
) )
// deny is the SWWAF_BLOCKLIST_ACTION and the SWWAF_REPUTATION_ACTION that // deny is the SWWAF_BLOCKLIST_ACTION and the SWWAF_REPUTATION_ACTION that
// refuses the requests of a client a source lists. // refuses the requests of a client a source lists.
const deny = "deny" const deny = "deny"
// blocklistDenied notes the blocklists that list the client, as
// noteListed does, and reports whether SWWAF_BLOCKLIST_ACTION, being deny,
// refuses the request. Being limit, it lowers the client's limits instead
// (see limitPercentages), and being log, it does nothing more.
func (rq *request) blocklistDenied() bool {
listedBy := rq.h.lists.ListedBy(rq.client)
rq.blocklisted = len(listedBy) > 0
rq.noteListed(listedBy, "listed by a blocklist")
return rq.blocklisted && rq.h.config.BlocklistAction == deny
}
// crowdSecBanned reports whether a decision of the CrowdSec decision list
// on the client is in force at now. If one is, it notes the list, as
// noteHit does, and bans the client until that decision ends.
func (rq *request) crowdSecBanned(now time.Time) bool {
decision, listed := rq.h.lists.CrowdSecDecision(rq.client, now)
if !listed {
return false
}
rq.noteHit(bans.ReputationHit{Source: rq.h.config.CrowdSecDecisionsURL},
"listed by the CrowdSec decision list")
rq.banForCrowdSec(now, decision)
return true
}
// dnsblDenied notes the DNSBL zones whose verdict lists the client, as
// noteListed does, and reports whether SWWAF_REPUTATION_ACTION, being
// deny, refuses the request. Being limit, it lowers the client's limits
// instead (see limitPercentages), and being log, it does nothing more. A
// zone without a verdict on the client is asked about it in the
// background, and the request does not wait for the answer. ctx is the
// request's own context.
func (rq *request) dnsblDenied(ctx context.Context) bool {
listedBy := rq.h.dnsbl.ListedBy(ctx, rq.client)
rq.dnsblListed = len(listedBy) > 0
rq.noteListed(listedBy, "listed by a DNSBL zone")
return rq.dnsblListed && rq.h.config.ReputationAction == deny
}
// abuseIPDBDenied notes AbuseIPDB, as noteHit does, with the score, when
// its score of the client is a hit, and reports whether
// SWWAF_REPUTATION_ACTION, being deny, refuses the request, as dnsblDenied
// does for a zone. While SWWAF_ABUSEIPDB_KEY is unset it does nothing. A
// client without a score is checked in the background, by the request's
// address, if its history counts an offence, and the request does not
// wait for the answer. The score is then used for each address of the
// client. ctx is the request's own context.
func (rq *request) abuseIPDBDenied(ctx context.Context) bool {
if rq.h.config.AbuseIPDBKey == "" {
return false
}
client := rq.h.clientGroup(rq.client)
held, _ := rq.h.limiter.Client(client)
offender := held.History.Offences != ratelimit.Offences{}
score, hit := rq.h.abuseIPDB.Hit(ctx, client, rq.client, offender)
if !hit {
return false
}
rq.abuseIPDBHit = true
rq.noteHit(bans.ReputationHit{Source: reputation.AbuseIPDBSource, Score: &score},
"scored by AbuseIPDB at or over SWWAF_ABUSEIPDB_MIN_SCORE")
return rq.h.config.ReputationAction == deny
}
// noteListed notes each of sources, the URLs of the blocklists or the // noteListed notes each of sources, the URLs of the blocklists or the
// DNSBL zones, their keys masked, that list the client, as noteHit does, // DNSBL zones, their keys masked, that list the client, as noteHit does,
// with reason. // with reason.
-50
View File
@@ -118,56 +118,6 @@ func TestEachBlocklistActionForAListedAddressAndAListedNetblock(t *testing.T) {
} }
} }
func TestReputationRefusalNamesItsHeuristicAndWhatListedTheClient(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
heuristic string
env map[string]string
load func(*testing.T, *servewaf.Server)
figures string
}{
{
"blocklists",
map[string]string{blocklistURLs: dropURL, blocklistAction: actionDeny},
func(t *testing.T, server *servewaf.Server) {
t.Helper()
loadLists(t, server, map[string][]string{dropURL: {client}})
},
`{"lists": ["` + dropURL + `"]}`,
},
{
"dnsbl",
map[string]string{
dnsblZones: dnsblZone, dnsblResolver: noResolver,
reputationAction: actionDeny,
},
func(_ *testing.T, server *servewaf.Server) {
loadVerdicts(server, map[string][]string{client: {dnsblZone}})
},
`{"zones": ["` + dnsblZone + `"]}`,
},
{
"abuseipdb",
map[string]string{abuseIPDBKey: accountKey, reputationAction: actionDeny},
func(_ *testing.T, server *servewaf.Server) {
loadScores(server, map[string]int64{client: 80})
},
`{"score": 80, "min_score": 75}`,
},
} {
t.Run(tc.heuristic, func(t *testing.T) {
t.Parallel()
s, server, _ := startWithLookups(t, tc.env)
tc.load(t, server)
line := s.get(client, http.StatusForbidden, requestlog.ActionDenied)
wantHeuristic(t, line, tc.heuristic, tc.figures)
})
}
}
func TestBlocklistsComeAfterTheCountryListsAndSkipAllowNets(t *testing.T) { func TestBlocklistsComeAfterTheCountryListsAndSkipAllowNets(t *testing.T) {
t.Parallel() t.Parallel()
+85 -23
View File
@@ -160,20 +160,16 @@ func requestHeaders(r *http.Request, names []string) map[string]string {
} }
// check is the one place where a request can be refused. It returns nil to let // check is the one place where a request can be refused. It returns nil to let
// the request through. The heuristics of checkClient decide: a request they // the request through. The checks of checkClient decide: a request they refuse
// refuse is answered with SWWAF_BAN_RESPONSE, 403 for a block rule or the Core // is answered with SWWAF_BAN_RESPONSE, 403 for a block rule or the Core Rule Set,
// Rule Set, or 431 for the header limits. In observe mode nothing is refused, // or 431 for the header limits. In observe mode nothing is refused, and the log
// and the log line names what enforce mode would have done. ctx is the // line names what enforce mode would have done. ctx is the request's own context.
// request's own context.
func (rq *request) check(ctx context.Context) *refusal { func (rq *request) check(ctx context.Context) *refusal {
v := rq.checkClient(ctx) action := rq.checkClient(ctx)
if v == nil {
return nil
}
action := v.action
switch { switch {
case action == "":
return nil
case rq.h.config.Observe: case rq.h.config.Observe:
rq.line.WouldAction = action rq.line.WouldAction = action
@@ -187,21 +183,35 @@ func (rq *request) check(ctx context.Context) *refusal {
} }
} }
// checkClient runs the heuristics of beforeLookup on the request, then looks // checkClient runs the checks on the request's client, and returns the
// up the client's AS number and country, then runs those of afterLookup. It // action of the first that refuses the request, or "" when none does. A
// returns the first verdict, or nil when none gives one. A client in // client in SWWAF_ALLOW_NETS skips them, and is not looked up. For any
// SWWAF_ALLOW_NETS skips them all, and is not looked up. ctx is the request's // other client, SWWAF_DENY_NETS comes first, then a ban on its netblock,
// own context. // so that a client either refuses is not looked up, then the lookup of
func (rq *request) checkClient(ctx context.Context) *verdict { // its AS number and country, then the country lists, then the blocklists,
if isInside(rq.client, rq.h.config.AllowNets) { // then the CrowdSec decision list, which bans the client it lists, then
return nil // the DNSBL zones' verdicts, and then AbuseIPDB's score; a request any of
// them refuses is not counted for the rate limits. Then come the
// rate limits, unless the client is in SWWAF_RATE_LIMIT_EXEMPT_NETS or the
// request's path is exempt under SWWAF_RATE_LIMIT_EXEMPT_PATHS, so that
// every other request is counted, each of them by the client's limit
// percentages, then SWWAF_TRAP_PATHS, then the rule files, then the header
// limits, and last the Core Rule Set. A request exempt from the rate limits
// is exempt from the byte limits too. ctx is the request's own context.
func (rq *request) checkClient(ctx context.Context) string {
cfg := rq.h.config
if isInside(rq.client, cfg.AllowNets) {
return ""
} }
now := rq.h.now() now := rq.h.now()
v := rq.runHeuristics(ctx, beforeLookup(), now) if isInside(rq.client, cfg.DenyNets) {
if v != nil { return requestlog.ActionDenied
return v }
if rq.banned(now) {
return requestlog.ActionBanned
} }
rq.lookUp(ctx, now) rq.lookUp(ctx, now)
@@ -210,7 +220,59 @@ func (rq *request) checkClient(ctx context.Context) *verdict {
// again: a request counted at the time it arrived could be counted // again: a request counted at the time it arrived could be counted
// after a later one of its client's, which the rate limits would take // after a later one of its client's, which the rate limits would take
// for the clock set back. // for the clock set back.
return rq.runHeuristics(ctx, afterLookup(), rq.h.now()) now = rq.h.now()
if rq.countryDenied() {
return requestlog.ActionCountryDenied
}
if rq.blocklistDenied() {
return requestlog.ActionDenied
}
if rq.crowdSecBanned(now) {
return requestlog.ActionBanned
}
if rq.dnsblDenied(ctx) || rq.abuseIPDBDenied(ctx) {
return requestlog.ActionDenied
}
rq.counted = rq.countedByRateLimits()
if rq.counted {
rq.limitPercent, rq.bytesPercent = rq.limitPercentages()
rq.line.LimitPercent, rq.line.LimitPercentSetting = rq.limitPercent.logged()
rq.line.BytesPercent, rq.line.BytesPercentSetting = rq.bytesPercent.logged()
}
if rq.counted && rq.limitBroken(now) {
return requestlog.ActionRateLimited
}
if rq.trapPath(now) {
return requestlog.ActionBanned
}
action := rq.checkRules(now)
if action != "" {
return action
}
if rq.overHeaderLimits() {
return requestlog.ActionTooLarge
}
return rq.checkCoreRuleSet()
}
// countedByRateLimits reports whether the rate limits count the request: its
// client is not in SWWAF_RATE_LIMIT_EXEMPT_NETS, and its path is not exempt
// under SWWAF_RATE_LIMIT_EXEMPT_PATHS.
func (rq *request) countedByRateLimits() bool {
cfg := rq.h.config
return !isInside(rq.client, cfg.RateLimitExemptNets) &&
!pathExempt(rq.in, cfg.RateLimitExemptPaths)
} }
// pathExempt reports whether r is exempt under prefixes, // pathExempt reports whether r is exempt under prefixes,
+1 -21
View File
@@ -1,7 +1,6 @@
package servewaf_test package servewaf_test
import ( import (
"encoding/json"
"io" "io"
"math" "math"
"net/http" "net/http"
@@ -20,7 +19,7 @@ const (
// requestIDHeader carries the request's id. // requestIDHeader carries the request's id.
requestIDHeader = "X-Request-ID" requestIDHeader = "X-Request-ID"
// instance is the SWWAF_INSTANCE_NAME a test sets. // instance is the SWWAF_INSTANCE_NAME a test sets.
instance = "fsn1app1/gitea" instance = "fsn1app1/blog"
// ipv6Client is a client on IPv6, and ipv6Group the netblock the rate // ipv6Client is a client on IPv6, and ipv6Group the netblock the rate
// limits count it as. // limits count it as.
ipv6Client = "2001:db8::7" ipv6Client = "2001:db8::7"
@@ -273,25 +272,6 @@ func wantTimings(t *testing.T, line logLine, want ...string) {
} }
} }
// wantHeuristic checks that line names heuristic as the one that refused the
// request or banned its client, with figures, a JSON object. A heuristic of ""
// with figures "null" is a line that names none.
func wantHeuristic(t *testing.T, line logLine, heuristic, figures string) {
t.Helper()
var want map[string]any
err := json.Unmarshal([]byte(figures), &want)
if err != nil {
t.Fatalf("decode %s: %v", figures, err)
}
if line.Heuristic != heuristic || !reflect.DeepEqual(line.HeuristicFigures, want) {
t.Errorf("log line has heuristic %q with heuristic_figures %v, want %q with %s",
line.Heuristic, line.HeuristicFigures, heuristic, figures)
}
}
// microseconds is a timing in whole microseconds. // microseconds is a timing in whole microseconds.
func microseconds(milliseconds float64) int64 { func microseconds(milliseconds float64) int64 {
return int64(math.Round(milliseconds * 1000)) return int64(math.Round(milliseconds * 1000))
-55
View File
@@ -1,55 +0,0 @@
package servewaf
// The heuristic rule_files. It reads the rules of the rule files, which
// SWWAF_RULES_DIR and SWWAF_RULES_ENABLED name.
import (
"context"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// ruleFiles checks the request against the rules of the rule files at now,
// notes the ids of those it matches in the log line, and refuses the request
// when the last rule it matches refuses it: with ActionRuleBlocked for a block
// rule, and ActionBanned for a ban rule. A ban rule bans the client's netblock
// for a clear sign of attack, or in observe mode raises the alert for the ban
// it would have made. Either rule's match is noted as an offence, for the
// client's history. Its figures are the rule's id and action.
type ruleFiles struct{}
func (ruleFiles) name() string { return "rule_files" }
func (ruleFiles) check(_ context.Context, rq *request, now time.Time) *verdict {
matched := rq.h.rules.Match(rq.in)
for _, rule := range matched {
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
rq.h.metrics.RuleMatched(rule.ID, rule.Action)
}
if len(matched) == 0 {
return nil
}
// Only the last rule matched can refuse the request.
last := matched[len(matched)-1]
figures := map[string]any{"rule_id": last.ID, "rule_action": last.Action}
switch last.Action {
case rules.ActionBlock:
rq.ruleBlocked = true
return &verdict{action: requestlog.ActionRuleBlocked, figures: figures}
case rules.ActionBan:
rq.attack = true
rq.banForAttack(now, bans.Notes{RuleID: last.ID, Target: last.Target})
return &verdict{action: requestlog.ActionBanned, figures: figures}
default:
return nil
}
}
-15
View File
@@ -93,21 +93,6 @@ func TestEachRuleAction(t *testing.T) {
s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned) s.get(client, http.StatusTooManyRequests, requestlog.ActionBanned)
} }
func TestRuleRefusalNamesItsHeuristicAndTheRule(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(t, "", map[string]string{
rulesDir: writeRules(t, "noted path log ^/\n"+testRules),
})
line := s.request(client, "/blocked", http.StatusForbidden,
requestlog.ActionRuleBlocked)
wantHeuristic(t, line, "rule_files", `{"rule_id": "blocked", "rule_action": "block"}`)
line = s.request(client, "/.env", http.StatusForbidden, requestlog.ActionBanned)
wantHeuristic(t, line, "rule_files", `{"rule_id": "probe", "rule_action": "ban"}`)
}
func TestNextClearSignOfAttackAfterABanBansPermanently(t *testing.T) { func TestNextClearSignOfAttackAfterABanBansPermanently(t *testing.T) {
t.Parallel() t.Parallel()
+64
View File
@@ -0,0 +1,64 @@
package servewaf
import (
"slices"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// trapPath reports whether the request asks for a path in
// SWWAF_TRAP_PATHS: its path as a path rule sees it, without the query,
// as sent or once percent-decoded, is one of them. Such a request is a
// clear sign of attack, as a ban rule's match is: it bans the client's
// netblock, or in observe mode raises the alert for the ban it would have
// made. The ban's notes give the path as sent.
func (rq *request) trapPath(now time.Time) bool {
path := rules.Path(rq.in)
if !slices.Contains(rq.h.config.TrapPaths, path) &&
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
return false
}
rq.attack = true
rq.banForAttack(now, bans.Notes{TrapPath: path})
return true
}
// checkRules checks the request against the rules of the rule files at
// now, notes the ids of those it matches in the log line, and returns the
// action of the rule that refuses it, ActionRuleBlocked for a block rule
// and ActionBanned for a ban rule, or "" when none does. A ban rule bans
// the client's netblock for a clear sign of attack, or in observe mode
// raises the alert for the ban it would have made. Either rule's match
// is noted as an offence, for the client's history.
func (rq *request) checkRules(now time.Time) string {
matched := rq.h.rules.Match(rq.in)
for _, rule := range matched {
rq.line.RuleIDs = append(rq.line.RuleIDs, rule.ID)
rq.h.metrics.RuleMatched(rule.ID, rule.Action)
}
if len(matched) == 0 {
return ""
}
// Only the last rule matched can refuse the request.
switch last := matched[len(matched)-1]; last.Action {
case rules.ActionBlock:
rq.ruleBlocked = true
return requestlog.ActionRuleBlocked
case rules.ActionBan:
rq.attack = true
rq.banForAttack(now, bans.Notes{RuleID: last.ID, Target: last.Target})
return requestlog.ActionBanned
default:
return ""
}
}
+44 -12
View File
@@ -2,7 +2,6 @@ package servewaf
import ( import (
"bytes" "bytes"
"context"
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
@@ -53,6 +52,13 @@ const ReportPath = "/_smallwebwaf/report"
// QuestionBodyMaxBytes is the most of a visitor's body a question carries. // QuestionBodyMaxBytes is the most of a visitor's body a question carries.
const QuestionBodyMaxBytes = 64 << 10 const QuestionBodyMaxBytes = 64 << 10
// The header limits: the most cookies and header fields a request may have
// when the checks reach the Core Rule Set. Real clients never reach them.
const (
maxCookies = 50
maxHeaderFields = 100
)
const ( const (
// maxAwaitingReport is the most answered requests held until their // maxAwaitingReport is the most answered requests held until their
// reports come. Past it, the request answered longest ago is dropped. // reports come. Past it, the request answered longest ago is dropped.
@@ -159,6 +165,32 @@ func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
rq.h.limiter.OverARateLimit(rq.h.clientGroup(rq.client), now)) rq.h.limiter.OverARateLimit(rq.h.clientGroup(rq.client), now))
} }
// overHeaderLimits reports whether servewaf refuses the request for its
// headers: it has more than maxHeaderFields fields, each value of a header sent
// twice being one, or more than maxCookies cookies, each part of a Cookie header
// between semicolons being one unless it is empty or only spaces and tabs. The
// Core Rule Set runs each of its rules on each cookie, so that thousands of small
// cookies keep it busy for over a second. It trims only spaces and control
// characters, of which a header value can hold only tabs, so a part of Unicode
// spaces is a cookie to it.
func (rq *request) overHeaderLimits() bool {
fields, cookies := 0, 0
for _, values := range rq.in.Header {
fields += len(values)
}
for _, value := range rq.in.Header.Values("Cookie") {
for part := range strings.SplitSeq(value, ";") {
if strings.Trim(part, " \t") != "" {
cookies++
}
}
}
return fields > maxHeaderFields || cookies > maxCookies
}
// answerQuestion runs the checks on the visitor's request a question // answerQuestion runs the checks on the visitor's request a question
// carries, and answers rproxy what to do with it. A denied request's counting // carries, and answers rproxy what to do with it. A denied request's counting
// is finished here, and an allowed one's by its report. // is finished here, and an allowed one's by its report.
@@ -187,18 +219,18 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
if refused == nil && !underOwnPrefix { if refused == nil && !underOwnPrefix {
// In observe mode, what enforce mode would have refused counts here. // In observe mode, what enforce mode would have refused counts here.
defer rq.runAfterAnswer(r.Context()) defer rq.countRefusal()
rq.allow() rq.allow()
return return
} }
// The last deferred runs first: runAfterAnswer before addToHistory, so that // The last deferred runs first: countRefusal before addToHistory, so that a
// a broken refusal limit is in the client's history. // broken error burst is in the client's history.
defer rq.addToHistory() defer rq.addToHistory()
defer rq.countAnomalies() defer rq.countAnomalies()
defer rq.runAfterAnswer(r.Context()) defer rq.countRefusal()
if refused != nil { if refused != nil {
rq.deny(*refused) rq.deny(*refused)
@@ -380,7 +412,7 @@ func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) {
// Remove reports true to one report only, of several that name the id. A // Remove reports true to one report only, of several that name the id. A
// question still being handled is held as nil. // question still being handled is held as nil.
if found && held != nil && h.awaitingReport.Remove(report.RequestID) { if found && held != nil && h.awaitingReport.Remove(report.RequestID) {
h.finishWith(r.Context(), held, report) h.finishWith(held, report)
} }
w.WriteHeader(http.StatusNoContent) w.WriteHeader(http.StatusNoContent)
@@ -443,11 +475,11 @@ func (report *Report) checkRanges() error {
// finishWith finishes a request with what its report says, and writes the // finishWith finishes a request with what its report says, and writes the
// request's last log line, which holds its decision's fields and the report's. // request's last log line, which holds its decision's fields and the report's.
// For an allowed request it first runs the heuristics of onReport, and adds the // For an allowed request it first counts the request's bytes, and adds it to its
// request to its client's history, the anomaly counters and the metrics, with // client's history, the anomaly counters and the metrics, with the status and
// the status and bytes the report gives. A denied request's counting was // bytes the report gives. A denied request's counting was finished at its
// finished at its decision. ctx is the report's own context. // decision.
func (h *handler) finishWith(ctx context.Context, held *decision, report Report) { func (h *handler) finishWith(held *decision, report Report) {
rq := &request{ rq := &request{
decision: *held, decision: *held,
h: h, h: h,
@@ -464,7 +496,7 @@ func (h *handler) finishWith(ctx context.Context, held *decision, report Report)
if rq.allowed { if rq.allowed {
rq.line.DurationUpstreamTotal = &report.DurationUpstreamTotal rq.line.DurationUpstreamTotal = &report.DurationUpstreamTotal
rq.runHeuristics(ctx, onReport(), rq.h.now()) rq.countBytes()
rq.countAnomalies() rq.countAnomalies()
rq.addToHistory() rq.addToHistory()
-9
View File
@@ -104,15 +104,6 @@ func TestDenyNetsRefuseBeforeTheLookup(t *testing.T) {
} }
} }
func TestDenyNetsRefusalNamesItsHeuristicAndNetblock(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(t, "", map[string]string{denyNets: clientsNet})
line := s.get(client, http.StatusForbidden, requestlog.ActionDenied)
wantHeuristic(t, line, "deny_nets", `{"netblock": "`+clientsNet+`"}`)
}
func TestRequestRefusedByDenyNetsIsNotCounted(t *testing.T) { func TestRequestRefusedByDenyNetsIsNotCounted(t *testing.T) {
t.Parallel() t.Parallel()
-39
View File
@@ -1,39 +0,0 @@
package servewaf
// The heuristic trap_paths. It reads SWWAF_TRAP_PATHS.
import (
"context"
"slices"
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// trapPaths refuses a request for a path in SWWAF_TRAP_PATHS: its path as a
// path rule sees it, without the query, as sent or once percent-decoded, is one
// of them. Such a request is a clear sign of attack, as a ban rule's match is:
// it bans the client's netblock, or in observe mode raises the alert for the
// ban it would have made. The ban's notes give the path as sent, and so does
// its figure.
type trapPaths struct{}
func (trapPaths) name() string { return "trap_paths" }
func (trapPaths) check(_ context.Context, rq *request, now time.Time) *verdict {
path := rules.Path(rq.in)
if !slices.Contains(rq.h.config.TrapPaths, path) &&
!slices.Contains(rq.h.config.TrapPaths, rules.DecodeOnce(path)) {
return nil
}
rq.attack = true
rq.banForAttack(now, bans.Notes{TrapPath: path})
return &verdict{
action: requestlog.ActionBanned,
figures: map[string]any{"path": path},
}
}
-10
View File
@@ -83,16 +83,6 @@ func TestTrapPathBansAsABanRuleDoes(t *testing.T) {
} }
} }
func TestTrapPathRefusalNamesItsHeuristicAndPath(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(t, "", map[string]string{trapPaths: trapPathList})
line := s.request(client, "/wp-login.php?redirect_to=x", http.StatusForbidden,
requestlog.ActionBanned)
wantHeuristic(t, line, "trap_paths", `{"path": "/wp-login.php"}`)
}
func TestTrapPathMatchesThePathDecodedOnce(t *testing.T) { func TestTrapPathMatchesThePathDecodedOnce(t *testing.T) {
t.Parallel() t.Parallel()
+14 -14
View File
@@ -49,7 +49,7 @@ const (
adminSecret = "fedcba9876543210fedcba9876543210" adminSecret = "fedcba9876543210fedcba9876543210"
// instance is the SWWAF_INSTANCE_NAME the tests set where they look at // instance is the SWWAF_INSTANCE_NAME the tests set where they look at
// it. // it.
instance = "fsn1app1/gitea" instance = "fsn1app1/blog"
// greeting is what the tests' app answers. // greeting is what the tests' app answers.
greeting = "hello from the app" greeting = "hello from the app"
// placed is the client the tests' lookup databases place. // placed is the client the tests' lookup databases place.
@@ -341,8 +341,8 @@ func TestEveryLogLineAndMetricCarriesTheInstanceName(t *testing.T) {
// the comments. // the comments.
for line := range strings.Lines(metrics) { for line := range strings.Lines(metrics) {
if !strings.HasPrefix(line, "#") && if !strings.HasPrefix(line, "#") &&
!strings.Contains(line, `instance="fsn1app1/gitea"`) { !strings.Contains(line, `instance="fsn1app1/blog"`) {
t.Errorf("series %q, without instance=\"fsn1app1/gitea\"", line) t.Errorf("series %q, without instance=\"fsn1app1/blog\"", line)
} }
} }
} }
@@ -610,9 +610,9 @@ func TestLookupDatabaseReplacedWhileRunningTakesEffect(t *testing.T) {
rename(t, replacement, path) rename(t, replacement, path)
metrics := metricsWith(t, url+"_smallwebwaf/metrics", token, metrics := metricsWith(t, url+"_smallwebwaf/metrics", token,
`smallwebwaf_lookup_database_read_failures_total{instance="fsn1app1/gitea"} 1`) `smallwebwaf_lookup_database_read_failures_total{instance="fsn1app1/blog"} 1`)
lastRead = seriesValue(t, metrics, lastRead = seriesValue(t, metrics,
`smallwebwaf_lookup_database_last_read_timestamp_seconds{instance="fsn1app1/gitea"}`) `smallwebwaf_lookup_database_last_read_timestamp_seconds{instance="fsn1app1/blog"}`)
wantStatus(t, url, placed, http.StatusOK) wantStatus(t, url, placed, http.StatusOK)
}) })
@@ -668,7 +668,7 @@ func TestBlocklistTriesAndCopiesKeptInReputationJSONAcrossRestarts(t *testing.T)
// too. // too.
rateLimitExemptNets: placed + "," + localhost, rateLimitExemptNets: placed + "," + localhost,
} }
failures := `smallwebwaf_reputation_failures_total{instance="fsn1app1/gitea",` + failures := `smallwebwaf_reputation_failures_total{instance="fsn1app1/blog",` +
`source="` + torURL + `"} ` `source="` + torURL + `"} `
// tor.txt cannot be fetched at first, which is counted. // tor.txt cannot be fetched at first, which is counted.
@@ -909,8 +909,8 @@ func TestStalledRemoteLogEndpointHoldsUpNoRequest(t *testing.T) {
// last. // last.
metrics := metricsText(t, url+"_smallwebwaf/metrics", token) metrics := metricsText(t, url+"_smallwebwaf/metrics", token)
for _, series := range []string{ for _, series := range []string{
`smallwebwaf_remote_log_lines_sent_total{instance="fsn1app1/gitea"} 0`, `smallwebwaf_remote_log_lines_sent_total{instance="fsn1app1/blog"} 0`,
`smallwebwaf_remote_log_buffer_depth{instance="fsn1app1/gitea"} 1`, `smallwebwaf_remote_log_buffer_depth{instance="fsn1app1/blog"} 1`,
} { } {
if !strings.Contains(metrics, "\n"+series+"\n") { if !strings.Contains(metrics, "\n"+series+"\n") {
t.Errorf("no %q in the metrics:\n%s", series, metrics) t.Errorf("no %q in the metrics:\n%s", series, metrics)
@@ -918,7 +918,7 @@ func TestStalledRemoteLogEndpointHoldsUpNoRequest(t *testing.T) {
} }
const dropped = "\nsmallwebwaf_remote_log_lines_dropped_total" + const dropped = "\nsmallwebwaf_remote_log_lines_dropped_total" +
`{instance="fsn1app1/gitea"} ` `{instance="fsn1app1/blog"} `
if strings.Contains(metrics, dropped+"0\n") || !strings.Contains(metrics, dropped) { if strings.Contains(metrics, dropped+"0\n") || !strings.Contains(metrics, dropped) {
t.Errorf("no line dropped in the metrics:\n%s", metrics) t.Errorf("no line dropped in the metrics:\n%s", metrics)
} }
@@ -1011,7 +1011,7 @@ func TestBanIsAlertedAndAnAlertNotSentIsKeptAcrossARestart(t *testing.T) {
runUntilStopped(t, env, func(url string) { runUntilStopped(t, env, func(url string) {
webhook.waitFor(t, "permanent_ban", true) webhook.waitFor(t, "permanent_ban", true)
const ofWebhook = `{destination="webhook",instance="fsn1app1/gitea"}` const ofWebhook = `{destination="webhook",instance="fsn1app1/blog"}`
metrics := metricsWith(t, url+"_smallwebwaf/metrics", token, metrics := metricsWith(t, url+"_smallwebwaf/metrics", token,
"\nsmallwebwaf_alerts_sent_total"+ofWebhook+" 1\n") "\nsmallwebwaf_alerts_sent_total"+ofWebhook+" 1\n")
@@ -1062,13 +1062,13 @@ func TestBanIsAlertedToSlackAndNtfy(t *testing.T) {
slackPost := slack.firstPost(t) slackPost := slack.firstPost(t)
err := json.Unmarshal([]byte(slackPost.body), &message) err := json.Unmarshal([]byte(slackPost.body), &message)
if err != nil || !strings.HasPrefix(message.Text, "*fsn1app1/gitea: ban*\n") || if err != nil || !strings.HasPrefix(message.Text, "*fsn1app1/blog: ban*\n") ||
!strings.Contains(message.Text, "\nclient: "+client+"\n") { !strings.Contains(message.Text, "\nclient: "+client+"\n") {
t.Errorf("Slack was sent %s", slackPost.body) t.Errorf("Slack was sent %s", slackPost.body)
} }
ntfyPost := ntfy.firstPost(t) ntfyPost := ntfy.firstPost(t)
if ntfyPost.header.Get("Title") != "fsn1app1/gitea: ban" || if ntfyPost.header.Get("Title") != "fsn1app1/blog: ban" ||
ntfyPost.header.Get("Authorization") != "Bearer "+ntfyToken || ntfyPost.header.Get("Authorization") != "Bearer "+ntfyToken ||
!strings.Contains(ntfyPost.body, "\nclient: "+client+"\n") { !strings.Contains(ntfyPost.body, "\nclient: "+client+"\n") {
t.Errorf("ntfy was sent %s, with the headers %v", ntfyPost.body, t.Errorf("ntfy was sent %s, with the headers %v", ntfyPost.body,
@@ -1080,9 +1080,9 @@ func TestBanIsAlertedToSlackAndNtfy(t *testing.T) {
// test process cannot fail the test. // test process cannot fail the test.
sent := []string{ sent := []string{
"\nsmallwebwaf_alerts_sent_total{destination=\"slack\"," + "\nsmallwebwaf_alerts_sent_total{destination=\"slack\"," +
"instance=\"fsn1app1/gitea\"} 1\n", "instance=\"fsn1app1/blog\"} 1\n",
"\nsmallwebwaf_alerts_sent_total{destination=\"ntfy\"," + "\nsmallwebwaf_alerts_sent_total{destination=\"ntfy\"," +
"instance=\"fsn1app1/gitea\"} 1\n", "instance=\"fsn1app1/blog\"} 1\n",
} }
metrics := metricsWith(t, url+"_smallwebwaf/metrics", token, sent...) metrics := metricsWith(t, url+"_smallwebwaf/metrics", token, sent...)
+3 -3
View File
@@ -143,7 +143,7 @@ const filledAlertsJSON = `{
"waiting": { "waiting": {
"webhook": [ "webhook": [
{ {
"instance": "fsn1app1/gitea", "instance": "fsn1app1/blog",
"time": "2026-10-06T00:00:00Z", "time": "2026-10-06T00:00:00Z",
"event": "ban", "event": "ban",
"client": "203.0.113.9", "client": "203.0.113.9",
@@ -158,7 +158,7 @@ const filledAlertsJSON = `{
"suppressed_repeats": 0 "suppressed_repeats": 0
}, },
{ {
"instance": "fsn1app1/gitea", "instance": "fsn1app1/blog",
"time": "2026-10-06T00:00:00Z", "time": "2026-10-06T00:00:00Z",
"event": "file_error", "event": "file_error",
"client": "", "client": "",
@@ -1721,7 +1721,7 @@ func newParams(dir string) state.Params {
Events: alerts.Events(), Events: alerts.Events(),
Cooldown: 15 * time.Minute, Cooldown: 15 * time.Minute,
MaxPerHour: 2, MaxPerHour: 2,
Instance: "fsn1app1/gitea", Instance: "fsn1app1/blog",
Now: midnight, Now: midnight,
}) })
-350
View File
@@ -1,350 +0,0 @@
#!/bin/sh
# script/gitea-requests: build the image and run gitea 1.27.3 behind rproxy
# and servewaf, three containers on one network. Every setting is at its
# default but these: SWRP_UPSTREAM_URL names gitea's container, SWRP_WAF_BACKEND
# names servewaf's, and SWWAF_LOG_REQUEST_HEADERS adds the header that labels a
# request. Then send through rproxy the gitea requests that reading gitea's
# source found the Core Rule Set refuses, which docs/gitea.md records, and a
# short sample of gitea's ordinary use: signing in, pages of files and commits,
# a diff, the API, a push, clone and fetch over HTTP, and two container image
# pushes. Last, print each request servewaf logged, with what it did with it and
# the ids of the Core Rule Set's rules that matched it.
# Every request comes from one client, docker's gateway, so more than
# SWWAF_ERROR_BURST_THRESHOLD, 30, refusals in a minute would ban it, and
# the requests after would show as banned. The containers, the network,
# the images and the files the script makes are removed however it ends;
# gitea's own image stays, as the base images of a build do. It needs
# network access, and a docker daemon that pushes to a registry on
# localhost over plain HTTP, as docker does by default. script/check does
# not run this.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
# gitea 1.27.3, 2026-08-29
GITEA_IMAGE="gitea/gitea@sha256:87a67ee09d3ae0d1df5fda5dcda3e2a1f9236a45b0a59025d6e00e46adc43bef"
# Named after this run, so that runs in other clones on the same host
# never touch each other's.
NAME="$("$SCRIPT_DIR/projectname")-gitea-$$"
IMAGE="$NAME-image"
NETWORK="$NAME"
RPROXY="$NAME-rproxy"
SERVEWAF="$NAME-servewaf"
GITEA="$NAME-gitea"
# gitea's one user, who owns the repository and the container images.
PASSWORD="owner-password"
BROWSER="Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0"
# The header that labels a request in the report. rproxy makes every request's
# id itself, so a label cannot be the X-Request-ID.
LABEL_HEADER="X-Case"
# The default of SWWAF_LOG_REQUEST_HEADERS, and the header above.
LOGGED_HEADERS="accept,accept-language,accept-encoding,content-type,origin,range,x-case"
# The files the script makes, and rproxy's address once it runs, localhost and
# the port docker publishes it on: a host name, since the Core Rule Set takes a
# Host that is an IP address for a warning (920350).
WORK=""
address=""
cleanup() {
# So that a signal cannot stop it partway through.
trap '' HUP INT TERM
docker rm --force --volumes "$RPROXY" "$SERVEWAF" "$GITEA" >/dev/null 2>&1 || true
docker network rm "$NETWORK" >/dev/null 2>&1 || true
docker rmi --force "$IMAGE" >/dev/null 2>&1 || true
if [ -n "$address" ]; then
docker rmi --force "$address/owner/python3-app:1" "$address/owner/app:1" \
>/dev/null 2>&1 || true
fi
rm -rf "$WORK"
}
fail() {
echo "gitea-requests: $*" >&2
report
exit 1
}
# wait_for <what fails> <command>...: run the command every second until
# it succeeds, for at most a minute.
wait_for() {
failure="$1"
shift
tries=0
until "$@"; do
tries=$((tries + 1))
[ "$tries" -lt 60 ] || fail "$failure"
sleep 1
done
}
# healthy <container>: docker says the container's health check passes.
healthy() {
status="$(docker inspect --format '{{.State.Health.Status}}' "$1")"
[ "$status" = healthy ]
}
# logged <text>...: a line of servewaf's output holds every text, in any order.
logged() {
lines="$(docker logs "$SERVEWAF" 2>&1)"
for text in "$@"; do
lines="$(printf '%s\n' "$lines" | grep -F "$text")" || return 1
done
}
# send <label> <path> <curl option>...: send a request for path to rproxy, with
# label in LABEL_HEADER, which the report gives, and keep the body of the answer
# in $WORK/answer. The Core Rule Set reads that header too, so a label names no
# file the cases name.
send() {
label="$1"
path="$2"
shift 2
curl --silent --show-error --max-time 30 --output "$WORK/answer" \
--header "$LABEL_HEADER: $label" "$@" "http://$address$path"
}
# page <label> <path> <curl option>...: send the request as a browser
# does, with the cookies gitea has set. Its Referer is the repository's
# page, which links to the others, and never a page the Core Rule Set
# refused: its address would have the next request refused too.
page() {
label="$1"
path="$2"
shift 2
send "$label" "$path" --compressed --user-agent "$BROWSER" \
--header "Accept: text/html,application/xhtml+xml,*/*;q=0.8" \
--header "Accept-Language: en-US,en;q=0.5" \
--cookie "$WORK/cookies" --cookie-jar "$WORK/cookies" \
--referer "http://$address/owner/repo" "$@"
}
# api <label> <path> <curl option>...: send the request as a client of
# gitea's API does, as the owner.
api() {
label="$1"
path="$2"
shift 2
send "$label" "$path" --user "owner:$PASSWORD" \
--header "Accept: application/json" "$@"
}
# repo_git <git argument>...: run git as the owner, without the host's
# own git settings, so that nothing it does is kept outside $WORK.
repo_git() {
GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 git \
-c user.name=owner -c user.email=owner@example.invalid "$@"
}
# lines: forty lines of text, enough for a diff to hide some of them.
lines() {
i=1
while [ "$i" -le 40 ]; do
echo "line $i"
i=$((i + 1))
done
}
# make_repository: create the repository owner/repo, holding a file of
# each name the cases name, and push it, which is the push over HTTP of
# the sample. The second commit adds a line at the end of Runtime.java,
# so that its diff hides the lines above.
make_repository() {
status="$(api create-repository /api/v1/user/repos \
--header "Content-Type: application/json" --data '{"name":"repo"}' \
--write-out '%{http_code}')"
[ "$status" = 201 ] || fail "creating the repository was answered $status"
dir="$WORK/repo"
repo_git init --quiet --initial-branch=main "$dir"
for file in \
src/java.base/share/classes/java/lang/Runtime.java \
java.lang.String.md \
META-INF/services/org.apache.commons.logging.LogFactory \
document.write.js \
process.env.d.ts \
test/fixtures/__proto__.json \
install-script.sh \
src/backend/catalog/information_schema.sql \
.../notes.txt \
README.md; do
mkdir -p "$dir/$(dirname "$file")"
lines >"$dir/$file"
repo_git -C "$dir" add "$file"
done
repo_git -C "$dir" commit --quiet --message "Add the files"
echo "line 41" >>"$dir/src/java.base/share/classes/java/lang/Runtime.java"
repo_git -C "$dir" commit --quiet --all --message "Add a line"
repo_git -C "$dir" -c "http.extraHeader=$LABEL_HEADER: git-push" push --quiet \
"http://owner:$PASSWORD@$address/owner/repo.git" main ||
fail "the push of the repository failed"
}
# browse: sign in, then open pages as a person does. The cases are those
# of https://git.eeqj.de/sneak/smallwebwaf/issues/30.
browse() {
# Case: gitea sends a visitor who is not signed in to the sign-in
# page, with the page asked for in redirect_to, when that page needs
# signing in, as a file's edit page does.
page issue-30-case /owner/repo/_edit/main/document.write.js --location
page sign-in /user/login
status="$(page sign-in /user/login --header "Origin: http://$address" \
--data-urlencode user_name=owner --data-urlencode "password=$PASSWORD" \
--write-out '%{http_code}')"
[ "$status" = 303 ] || fail "signing in was answered $status"
page browse /
page browse /owner/repo
page browse /owner/repo/src/branch/main/README.md
page browse /owner/repo/commits/branch/main
page browse "/owner/repo/commit/$(repo_git -C "$WORK/repo" rev-parse HEAD)"
# Case: expanding the lines the diff hides, by the link the diff's
# page gives for it.
excerpt="$(sed -n 's/.*data-fetch-url="\([^"]*blob_excerpt[^"]*\)".*/\1/p' \
"$WORK/answer" | head -n 1 | sed 's/&amp;/\&/g')"
[ -n "$excerpt" ] || fail "the diff's page has no link to its hidden lines"
page issue-30-case "$excerpt"
# Cases: the commit graph of one file, and the pages of files whose
# path the Core Rule Set takes for an attack.
page issue-30-case "/owner/repo/graph?file=src/java.base/share/classes/java/lang/Runtime.java"
page issue-30-case /owner/repo/src/branch/main/META-INF/services/org.apache.commons.logging.LogFactory
page issue-30-case /owner/repo/src/branch/main/java.lang.String.md
page issue-30-case /owner/repo/src/branch/main/process.env.d.ts
page issue-30-case /owner/repo/src/branch/main/.../notes.txt
}
# call_api: the API's calls the cases name: the commits of a file, for a
# file of each name the Core Rule Set takes for an attack in the query
# parameter path, and the file-contents call, which takes its list of
# files as JSON in the query parameter body.
call_api() {
for file in \
src/java.base/share/classes/java/lang/Runtime.java \
java.lang.String.md \
document.write.js \
process.env.d.ts \
test/fixtures/__proto__.json \
install-script.sh \
src/backend/catalog/information_schema.sql; do
api issue-30-case /api/v1/repos/owner/repo/commits --get --data-urlencode "path=$file"
done
api issue-30-case /api/v1/repos/owner/repo/file-contents --get \
--data-urlencode 'body={"files":["src/java.base/share/classes/java/lang/Runtime.java"]}'
}
# push_images: push a container image to owner/python3-app, then the same
# image to owner/app. For the second, the case, docker asks gitea to take
# its layer from owner/python3-app, naming it in the query parameter
# from. docker sends no label, so the report gives its requests none. The image
# is one small file, imported rather than built.
push_images() {
echo "$NAME" >"$WORK/layer.txt"
tar -cf - -C "$WORK" layer.txt |
docker import - "$address/owner/python3-app:1" >/dev/null
docker tag "$address/owner/python3-app:1" "$address/owner/app:1"
# The login is written here rather than made with docker login, which
# would keep it in the host's credential helper when it has one.
mkdir "$WORK/docker"
printf '{"auths":{"%s":{"auth":"%s"}}}\n' "$address" \
"$(printf 'owner:%s' "$PASSWORD" | base64)" >"$WORK/docker/config.json"
for image in "$address/owner/python3-app:1" "$address/owner/app:1"; do
DOCKER_CONFIG="$WORK/docker" docker push --quiet "$image" >/dev/null ||
echo "gitea-requests: the push of $image failed" >&2
done
}
# clone_and_fetch: the clone and fetch over HTTP of the sample.
clone_and_fetch() {
repo_git -c "http.extraHeader=$LABEL_HEADER: git-clone" clone --quiet \
"http://owner:$PASSWORD@$address/owner/repo.git" "$WORK/clone" ||
fail "the clone failed"
repo_git -C "$WORK/clone" -c "http.extraHeader=$LABEL_HEADER: git-fetch" fetch --quiet ||
fail "the fetch failed"
}
# field <name> <line>: the text of the field name in line, a line of the
# request log, or nothing when it has none.
field() {
printf '%s\n' "$2" | sed -n "s/.*\"$1\":\"\([^\"]*\)\".*/\1/p"
}
# report: print a line for each request rproxy reported to servewaf: what
# servewaf did with the request, the ids of the Core Rule Set's rules that
# matched it, its label, and the request. waf_blocked is a refusal by the Core
# Rule Set, and forward a request passed on to gitea, whatever gitea answered.
report() {
docker logs "$SERVEWAF" 2>/dev/null | grep '"servewaf_line":"report"' |
sed 's/\\u0026/\&/g' >"$WORK/requests"
while IFS= read -r line; do
ids="$(printf '%s\n' "$line" | sed -n 's/.*"waf_rule_ids":\[\([0-9,]*\)\].*/\1/p')"
query="$(field query "$line")"
label="$(field x-case "$line")"
printf '%-11s %-13s %-17s %s %s%s\n' "$(field action "$line")" "${ids:--}" \
"${label:--}" "$(field method "$line")" \
"$(field path "$line")" "${query:+?$query}"
done <"$WORK/requests"
echo "gitea-requests: $(grep -c . "$WORK/requests") requests," \
"$(grep -c '"action":"waf_blocked"' "$WORK/requests") refused by the Core Rule Set"
}
main() {
cd "$ROOT"
trap cleanup EXIT
trap 'exit 1' HUP INT TERM
WORK="$(mktemp -d)"
docker build --no-cache -t "$IMAGE" .
# rproxy first, so that gitea can be given the address it is reached at,
# and gitea's image pulled before, so that gitea starts within the minute
# rproxy's health check waits for it.
docker pull --quiet "$GITEA_IMAGE" >/dev/null
docker network create "$NETWORK" >/dev/null
docker run --detach --name "$SERVEWAF" --network "$NETWORK" \
--env SMALLWEBWAF_COMMAND=servewaf \
--env SWWAF_LOG_REQUEST_HEADERS="$LOGGED_HEADERS" \
"$IMAGE" >/dev/null
wait_for "servewaf's health check did not pass" healthy "$SERVEWAF"
docker run --detach --name "$RPROXY" --network "$NETWORK" \
--publish 127.0.0.1::8080 \
--env SWRP_UPSTREAM_URL="http://$GITEA:3000" \
--env SWRP_WAF_BACKEND="http://$SERVEWAF:8080" \
"$IMAGE" >/dev/null
published="$(docker port "$RPROXY" 8080/tcp)"
address="localhost:${published##*:}"
docker run --detach --name "$GITEA" --network "$NETWORK" \
--env GITEA__security__INSTALL_LOCK=true \
--env GITEA__server__ROOT_URL="http://$address/" \
"$GITEA_IMAGE" >/dev/null
# rproxy's health check passes once gitea takes connections.
wait_for "rproxy's health check did not pass" healthy "$RPROXY"
docker exec --user git "$GITEA" gitea admin user create --admin \
--username owner --password "$PASSWORD" --email owner@example.invalid \
--must-change-password=false >/dev/null
make_repository
browse
call_api
push_images
clone_and_fetch
# The fetch's last request is the last of all; once it is logged, so
# is every other.
wait_for "servewaf logged no report of the fetch" \
logged '"x-case":"git-fetch"' '"method":"POST"' '"servewaf_line":"report"'
report
}
main "$@"
-216
View File
@@ -1,216 +0,0 @@
#!/bin/sh
# script/gitea-setup: build the image, bring up deploy/gitea under a project
# name of its own, and check it:
# - that a page load, a push and a clone sent through traefik pass through
# rproxy, and that servewaf checks them;
# - that gitea logs the visitor's address, not rproxy's;
# - that a request from a container on the shared network skips the checks;
# - that with servewaf stopped, gitea still answers through rproxy.
# It publishes no port of traefik's, since this host's ports 80 and 443 may be
# taken. It reaches traefik at its address on the shared network instead, so
# its own requests come from the network's gateway, which servewaf checks like
# any visitor. Two runs at once on one docker host cannot both start, since
# both would use the subnet compose.yaml names. The containers, the network,
# the volumes, the image and the files the script makes are removed however it
# ends; the gitea and traefik images stay, as the base images of a build do.
# It needs network access. script/check does not run this.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
# Named after this run, so that runs in other clones on the same host
# never touch each other's.
NAME="$("$SCRIPT_DIR/projectname")-gitea-setup-$$"
IMAGE="$NAME-image"
NETWORK="${NAME}_shared"
VISITOR="$NAME-visitor"
# The site's host name and gitea's image, as deploy/gitea/compose.yaml names
# them. gitea's image also runs the container on the shared network.
SITE="git.example.org"
# gitea 1.27.3, 2026-08-29
GITEA_IMAGE="gitea/gitea@sha256:87a67ee09d3ae0d1df5fda5dcda3e2a1f9236a45b0a59025d6e00e46adc43bef"
# gitea's one user, who owns the repository.
PASSWORD="owner-password"
# The files the script makes, gitea's secret key for this run, and traefik's
# and the gateway's addresses on the shared network once the setup runs.
WORK=""
secret_key=""
traefik=""
gateway=""
cleanup() {
# So that a signal cannot stop it partway through.
trap '' HUP INT TERM
docker rm --force "$VISITOR" >/dev/null 2>&1 || true
if [ -n "$WORK" ]; then
compose down --volumes --remove-orphans --timeout 5 >/dev/null 2>&1 || true
rm -rf "$WORK"
fi
docker rmi --force "$IMAGE" >/dev/null 2>&1 || true
}
# compose <argument>...: docker compose on the setup, under this run's project
# name, with the image built here, this run's secret key and no port of
# traefik's published.
compose() {
SMALLWEBWAF_IMAGE="$IMAGE" GITEA_SECRET_KEY="$secret_key" \
docker compose --project-name "$NAME" \
--file "$ROOT/deploy/gitea/compose.yaml" --file "$WORK/no-ports.yaml" "$@"
}
fail() {
echo "gitea-setup: $*; the last lines of each container's output:" >&2
compose logs --no-color --tail 30 >&2 || true
exit 1
}
# wait_for <what fails> <command>...: run the command every second until
# it succeeds, for at most a minute.
wait_for() {
failure="$1"
shift
tries=0
until "$@"; do
tries=$((tries + 1))
[ "$tries" -lt 60 ] || fail "$failure"
sleep 1
done
}
# logged <service> <text>...: a line of the service's output holds every text,
# in any order.
logged() {
lines="$(compose logs --no-color --no-log-prefix "$1" 2>&1)"
shift
for text in "$@"; do
lines="$(printf '%s\n' "$lines" | grep -F -e "$text")" || return 1
done
}
# send <path> <curl option>...: send a request for path to the site, at
# traefik's address, and print the status of the answer, 000 for none. traefik
# answers with a certificate it made itself, since the setup names no
# certificate resolver.
send() {
path="$1"
shift
curl --silent --show-error --insecure --max-time 30 --output /dev/null \
--write-out '%{http_code}' --resolve "$SITE:443:$traefik" "$@" \
"https://$SITE$path" || true
}
# routed: traefik passes a request for the site's front page on to gitea.
routed() {
[ "$(send / 2>/dev/null)" = 200 ]
}
# site_git <git argument>...: run git as the owner, without the host's own git
# settings, reaching the site at traefik's address. git gives up on a transfer
# slower than 1000 bytes a second for 30 seconds.
site_git() {
GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 git \
-c user.name=owner -c user.email=owner@example.invalid \
-c http.sslVerify=false -c "http.curloptResolve=$SITE:443:$traefik" \
-c http.lowSpeedLimit=1000 -c http.lowSpeedTime=30 "$@"
}
# address_of <service>: print the service's address on the shared network.
address_of() {
docker inspect --format "{{(index .NetworkSettings.Networks \"$NETWORK\").IPAddress}}" \
"$(compose ps --quiet "$1")"
}
# passed_through <method> <path>: rproxy passed a request from the gateway for
# path to gitea on servewaf's allow, and servewaf checked it: its report line
# gives the client's counts and the Core Rule Set's score.
passed_through() {
wait_for "rproxy logged no $1 $2 that servewaf allowed" logged rproxy \
"\"method\":\"$1\"" "\"path\":\"$2\"" "\"client_ip\":\"$gateway\"" \
'"disposition":"allow"'
wait_for "servewaf logged no check of $1 $2" logged servewaf \
'"servewaf_line":"report"' "\"method\":\"$1\"" "\"path\":\"$2\"" \
"\"client_ip\":\"$gateway\"" '"action":"forward"' '"counts":' '"waf_score":'
}
main() {
cd "$ROOT"
trap cleanup EXIT
trap 'exit 1' HUP INT TERM
WORK="$(mktemp -d)"
printf 'services:\n traefik:\n ports: !reset []\n' >"$WORK/no-ports.yaml"
docker build --no-cache -t "$IMAGE" .
secret_key="$(docker run --rm "$GITEA_IMAGE" gitea generate secret SECRET_KEY)"
# rproxy is healthy once gitea takes connections.
compose up --detach --quiet-pull --wait --wait-timeout 120 ||
fail "the setup did not start"
traefik="$(address_of traefik)"
gateway="$(docker network inspect --format '{{(index .IPAM.Config 0).Gateway}}' "$NETWORK")"
wait_for "traefik did not route the site to rproxy" routed
compose exec --user git gitea gitea admin user create --admin \
--username owner --password "$PASSWORD" --email owner@example.invalid \
--must-change-password=false >/dev/null || fail "creating gitea's user failed"
status="$(send /api/v1/user/repos --user "owner:$PASSWORD" \
--header "Content-Type: application/json" --data '{"name":"repo"}')"
[ "$status" = 201 ] || fail "creating the repository was answered $status"
status="$(send /owner/repo)"
[ "$status" = 200 ] || fail "the repository's page was answered $status"
passed_through GET /owner/repo
echo "gitea-setup: a page load passes through rproxy, and servewaf checks it"
wait_for "gitea did not log the visitor's address" logged gitea \
"GET /owner/repo for $gateway:"
echo "gitea-setup: gitea logs the visitor's address, not rproxy's"
site_git init --quiet --initial-branch=main "$WORK/repo"
echo "hello" >"$WORK/repo/README.md"
site_git -C "$WORK/repo" add README.md
site_git -C "$WORK/repo" commit --quiet --message "Add a README"
site_git -C "$WORK/repo" push --quiet \
"https://owner:$PASSWORD@$SITE/owner/repo.git" main ||
fail "the push failed"
passed_through POST /owner/repo.git/git-receive-pack
echo "gitea-setup: a push passes through rproxy, and servewaf checks it"
site_git clone --quiet "https://owner:$PASSWORD@$SITE/owner/repo.git" \
"$WORK/clone" || fail "the clone failed"
[ "$(cat "$WORK/clone/README.md")" = hello ] || fail "the clone lacks the README"
passed_through POST /owner/repo.git/git-upload-pack
echo "gitea-setup: a clone passes through rproxy, and servewaf checks it"
# A container on the shared network, as sandcastle is, asks for the site
# by its name, and so reaches traefik from its own address.
status="$(docker run --rm --name "$VISITOR" --network "$NETWORK" \
--entrypoint curl "$GITEA_IMAGE" --silent --insecure --max-time 30 \
--output /dev/null --write-out '%{http_code}' "https://$SITE/api/v1/version" ||
true)"
[ "$status" = 200 ] || fail "the request from the shared network was answered $status"
wait_for "servewaf logged no request from the shared network" logged servewaf \
'"servewaf_line":"report"' '"path":"/api/v1/version"' '"action":"forward"'
line="$(compose logs --no-color --no-log-prefix servewaf |
grep -F '"servewaf_line":"report"' | grep -F '"path":"/api/v1/version"')"
case "$line" in
*"\"client_ip\":\"$gateway\""*)
fail "the request from the shared network came from the gateway" ;;
*'"counts":'* | *'"waf_score":'*)
fail "servewaf checked the request from the shared network" ;;
esac
echo "gitea-setup: a request from the shared network skips the checks"
compose stop servewaf >/dev/null 2>&1 || fail "servewaf did not stop"
status="$(send /owner/repo)"
[ "$status" = 200 ] ||
fail "with servewaf stopped, the repository's page was answered $status"
wait_for "rproxy logged no request SWRP_DEFAULT_DISPOSITION decided" \
logged rproxy '"path":"/owner/repo"' '"default_disposition":true'
echo "gitea-setup: with servewaf stopped, gitea answers through rproxy"
}
main "$@"