Author SHA1 Message Date
sneak b5364bf282 SWW_APPS: an app's Core Rule Set fixes apply only to the hostnames that serve it (closes #220)
check / check (push) Failing after 3m37s
gitea's fixes (its parameters, its cookies and gitea.conf) now apply
only to a hostname SWW_APPS lists gitea for, or to every hostname while
it is unset. Each app's fixes are one .conf file, whose first rule skips
the rest unless Inspect sets tx.app_<name>, and one entry in apps.

A hostname in SWW_APPS that is not a DNS name, such as *.example.com,
stops the start. Unset, SWW_APPS is logged at start as null; set
empty, as "".

Model: opus-5-5
2026-11-09 02:32:38 +00:00
clawbot c66277b111 Request tags, with refusing and limits per tag (closes #175)
check / check (push) Failing after 5m8s
Each request gets tags, logged as `tags`: `asn=`, `country=`, one per `SWW_TAG_LISTS` list holding the client, `crowdsec`, `dnsbl=<zone>`, `abuseipdb` and `exploitsignature`. `SWW_TAG_DENY` refuses by tag and `SWW_TAG_LIMIT_PERCENT` lowers limits by tag. They replace `SWW_ASN_LIMIT_PERCENT`, `SWW_COUNTRY_LIMIT_PERCENT` and `SWW_DENIED_COUNTRIES`, with no aliases. A tag in those settings that nothing configured can give stops the start.

Judgement call: `SWW_ASN_LIMIT_PERCENT_URL` stays, its entries counting like `asn=` tags.
Judgement call: `SWW_TAG_LIMIT_PERCENT` also lowers byte limits, as the settings it replaces did.
Trips a reader: `sww_country_list_refusals_total` now counts only `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` refusals.

Model: opus-5-5
2026-11-09 03:30:49 +01:00
sneak f6435b1a73 Add make publish: build the image on the Docker host and push it to the registry
check / check (push) Successful in 3m23s
`script/publish` runs `script/docker` against the Docker host and pushes the image to the registry under four tags: the UTC date, the short commit hash, the branch name and `latest`. It refuses a working tree with uncommitted changes, since the hash tag names a commit.

Also carries sneak's `golines -m 77` line wrapping in three files, with no code change.

Unverified: a real build on the Docker host and a real push from this commit.
Deviation: sneak's FIXME comments are out, because the linter refuses them; they are #227, #228 and #229.

Model: opus-5-5
Co-authored-by: sneak <sneak@sneak.berlin>
2026-11-09 02:03:39 +01:00
30 changed files with 1146 additions and 279 deletions

No files matched your search

+7 -3
View File
@@ -1,11 +1,12 @@
.PHONY: bootstrap setup test lint fmt fmt-check tidy check docker hooks build \
run example-app
.PHONY: bootstrap setup test lint fmt fmt-check tidy check docker publish \
hooks build run example-app
# Makefile targets are thin shims; the implementations live in script/
# per the scripts-to-rule-them-all pattern (see the Entrypoints section
# of README.md). tidy writes go.mod and go.sum as `go mod tidy` does,
# which test checks. build and run are for working on the code by hand;
# example-app checks the image with an app built on it.
# example-app checks the image with an app built on it; publish builds
# the image on the Docker host and pushes it to the registry.
bootstrap:
@script/bootstrap
@@ -34,6 +35,9 @@ check:
docker:
@script/docker
publish:
@script/publish
hooks:
@script/install-precommit
+28 -13
View File
@@ -68,8 +68,9 @@ Not built, and not planned:
### The image
No image is published. Build it from a clone. `make docker` runs the tests and
the linter, then builds the image, tagged `smallwebwaf`.
The image is published at `git.eeqj.de/sneak/smallwebwaf`. You can also build it
from a clone. `make docker` runs the tests and the linter, then builds the
image, tagged `smallwebwaf`.
```sh
git clone https://git.eeqj.de/sneak/smallwebwaf.git
@@ -216,7 +217,7 @@ account, a key, a token or an alert destination stays off until you give one.
Every setting the code reads, with its default:
| Variable | Default | What it does |
| ---------------------------------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ---------------------------------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `SWW_COMMAND` | `rproxy` | Which command the image runs, and its health check checks: `rproxy` or `discriminator`. |
| `SWRP_WAF_BACKEND` | required | `discriminator`'s URL, such as `http://discriminator:8080`. |
| `SWRP_UPSTREAM_URL` | `http://127.0.0.1:8081` | The app's URL: a scheme, a host and an optional port, nothing more. |
@@ -258,7 +259,7 @@ Every setting the code reads, with its default:
| `SWW_ADD_LOOKUP_HEADERS` | `false` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
| `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
| `SWW_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
| `SWW_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=cn`. |
| `SWW_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. |
| `SWW_TAG_LIMIT_PERCENT` | empty | Tags whose clients get a percentage, from 1 to 99, of every request and byte limit, such as `vpn:50,asn=14061:25`; the lowest applies. |
| `SWW_ASN_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `asn=` tag's percentage. |
| `SWW_COUNTRY_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `country=` tag's percentage. |
@@ -274,6 +275,7 @@ Every setting the code reads, with its default:
| `SWW_WAF_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. |
| `SWW_WAF_BODY_LIMIT` | `off` | How much of a request body the Core Rule Set reads, a size up to `64K`; `off` reads none. |
| `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. |
| `SWW_APPS` | unset | The apps each hostname serves, such as `git.example.com:gitea;cloud.example.com:gitea`; only those apps' fixes apply there. Unset, every app's fixes apply to every hostname. |
| `SWW_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. |
| `SWW_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. |
| `SWW_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:<percent>` of every limit, or `log`. |
@@ -413,7 +415,7 @@ services:
SWW_RATE_LIMIT_PER_HOUR: "600"
SWW_RATE_LIMIT_PER_DAY: "3000"
# Hosting networks and one country get a share of those limits.
SWW_TAG_LIMIT_PERCENT: asn=14061:50,asn=16276:25,country=cn:25
SWW_TAG_LIMIT_PERCENT: asn=14061:50,asn=16276:25,country=CN:25
# Paths only scanners ask for; one request bans the client.
SWW_TRAP_PATHS: /wp-login.php,/xmlrpc.php
# Answer a refused client with 429 rather than 403.
@@ -505,10 +507,10 @@ of the request reaches the app.
`discriminator` takes the checks:
3. The static lists, `SWW_ALLOW_NETS` and `SWW_DENY_NETS`.
4. The bans.
5. The lookup of the client's AS number and country.
6. The tag lists.
3. The tag lists.
4. The static lists, `SWW_ALLOW_NETS` and `SWW_DENY_NETS`.
5. The bans.
6. The lookup of the client's AS number and country.
7. The country list.
8. The reputation sources.
9. The tag refusals of `SWW_TAG_DENY`.
@@ -572,11 +574,15 @@ client's limits by tag. See "Request tags" in [`SPEC.md`](SPEC.md).
headers of every request, and no body unless `SWW_WAF_BODY_LIMIT` is set. In
front of a code forge, bodies carry what people write and publish, which the
rules take for attacks.
- `smallwebwaf` changes the Core Rule Set in eight ways so that it lets ordinary
- `smallwebwaf` changes the Core Rule Set in six ways so that it lets ordinary
requests to a code forge through, such as git over HTTP and sign-in through
Git Credential Manager. On gitea's own paths, its searches, file names and
branch names get through too. No setting undoes the changes; "Attack
detection" in [`SPEC.md`](SPEC.md) lists them.
Git Credential Manager. No setting undoes the changes; "Attack detection" in
[`SPEC.md`](SPEC.md) lists them.
- Each supported app also has its own fixes, which let its ordinary requests
through. gitea's let its searches, file names and branch names through.
`SWW_APPS` names the apps each hostname serves. A hostname gets only its own
apps' fixes, and a hostname it does not list gets none. Unset, every app's
fixes apply to every hostname.
- `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` switches single rules off for an app's own
fields or paths.
- Rule files are plain text, one rule to a line, edited by hand and read again
@@ -682,6 +688,15 @@ on the host.
`make check`.
- `script/docker`: builds the image, whose build runs the tests and the linter
first. `make docker`.
- `script/publish`: runs `script/docker`, whose build happens on the Docker host
named by `DOCKER_HOST` (`ssh://root@fsn1app1` unless set). It then pushes the
image to `PUBLISH_IMAGE` (`git.eeqj.de/sneak/smallwebwaf` unless set). It
refuses a working tree with uncommitted changes. `make publish`. It pushes
four tags:
- the UTC date as `YYYY-MM-DD`;
- the short commit hash;
- the branch name;
- `latest`.
- `script/cibuild`: what CI runs: `script/bootstrap`, `script/check`, then the
image build.
- `script/precommit`: run by the git pre-commit hook; runs `script/check`.
+91 -65
View File
@@ -81,8 +81,8 @@ to them.
of bans is full.
- `internal/rules`: reads the rule files at start and again when they
change, and tells which rules a request matches.
- `internal/waf`: the Core Rule Set, with the eight changes listed under
"Attack detection", as Coraza directives.
- `internal/waf`: the Core Rule Set, with the six changes and the apps'
fixes listed under "Attack detection", as Coraza directives.
- `internal/lookup`: looks clients up through GeoJS or in the lookup
database.
- `internal/reputation`: fetches the lists, asks the DNS blocklists and
@@ -143,11 +143,13 @@ request log's `heuristic` field gives that name. `refusal_limit` and
- An IPv6 client is its group: the netblock of `SWW_IPV6_GROUP_PREFIX`
around its address, a /64 by default, since one abuser usually controls a
whole /64. A client is therefore one IPv4 address or one IPv6 group.
3. The static lists. A client in `SWW_ALLOW_NETS` skips steps 4 to 15, the byte
3. The tag lists (`tag_lists`; see "Request tags"). Each list that holds the
client adds its tag. A client in `SWW_ALLOW_NETS` gets no tag.
4. The static lists. A client in `SWW_ALLOW_NETS` skips steps 5 to 15, the byte
limits and the error burst. A client in `SWW_DENY_NETS` is refused
(`deny_nets`); a client in both lists is let through.
4. Bans. A client under a ban is refused (`active_ban`; see "Bans").
5. The lookup, unless `SWW_LOOKUP_SOURCE` is `off` (see "Lookup of AS number and
5. Bans. A client under a ban is refused (`active_ban`; see "Bans").
6. The lookup, unless `SWW_LOOKUP_SOURCE` is `off` (see "Lookup of AS number and
country"). The request goes on at once. It waits for the first GeoJS answer,
up to `SWW_LOOKUP_TIMEOUT`, only when one of these needs it:
- the country list;
@@ -158,8 +160,6 @@ request log's `heuristic` field gives that name. `refusal_limit` and
A request that ends before the answer comes is logged without it.
6. The tag lists (`tag_lists`; see "Request tags"). Each list that holds the
client adds its tag.
7. The country list (`countries`; see "Country list").
8. The reputation sources (see "Reputation"), in this order:
- A client on a fetched blocklist gets `SWW_BLOCKLIST_ACTION`
@@ -401,7 +401,7 @@ Conventions:
### Static lists
- `SWW_ALLOW_NETS` skips the checks step 3 names, for monitoring or the owner's
- `SWW_ALLOW_NETS` skips the checks step 4 names, for monitoring or the owner's
own networks. `SWW_RATE_LIMIT_EXEMPT_NETS` bypasses the request and byte
limits only; the error burst, attack detection and bans still apply.
- Many people can share one address, as in an office, on a mobile carrier or
@@ -480,7 +480,7 @@ service that keeps its visitors' addresses on its own host.
dropped from that table. The answer reaches the client's history and ban notes
when it comes.
- `SWW_LOOKUP_TIMEOUT` is how long a visitor's request waits for its client's
first answer, when a setting needs it (see step 5). It also bounds each
first answer, when a setting needs it (see step 6). It also bounds each
request to GeoJS. A client whose answer has not come in time counts as unknown
until it comes, and its later requests do not wait. Its address is asked about
again in the background.
@@ -534,8 +534,10 @@ the log line as `tags`, sorted, with empty lists left out (see
Each tag and where it comes from:
- `asn=<number>` and `country=<code>`, in lower case, such as `asn=64496` and
`country=de`, from the lookup. A client the lookup cannot place has neither.
- `asn=<number>` and `country=<code>`, such as `asn=64496` and `country=DE`,
from the lookup. A client the lookup cannot place has neither. A request
refused by `deny_nets` or `active_ban` has neither, because those checks come
before the lookup.
- One tag per list of `SWW_TAG_LISTS` that holds the client (`tag_lists`), the
tag being the list's name, such as `torexit` or `vpn=mullvad`. These lists are
fetched and kept like a blocklist. `discriminator` ships none of its own.
@@ -546,8 +548,9 @@ Each tag and where it comes from:
For the Core Rule Set a match is a score at or over
`SWW_WAF_ANOMALY_THRESHOLD`.
A client in `SWW_ALLOW_NETS` skips the checks and gets no tag. `SWW_TAG_DENY`
and `SWW_TAG_LIMIT_PERCENT` are read case-insensitively.
A client in `SWW_ALLOW_NETS` skips the checks and gets no tag. Every tag is in
lower case, except a country code, which is in capitals. `SWW_TAG_DENY` and
`SWW_TAG_LIMIT_PERCENT` accept tags in either case.
Two settings act on tags:
@@ -580,16 +583,16 @@ abuse comes from.
`SWW_ASN_LIMIT_PERCENT_URL`; for a country, its `country=` tag's percentage.
- `SWW_ASN_LIMIT_PERCENT_URL` names a text file of `AS:percent` lines, so one
abuse-source list can be shared by every `discriminator` in the fleet. It is
fetched and kept like a blocklist (see "Reputation"). Each entry counts as an
`asn=` tag in `SWW_TAG_LIMIT_PERCENT` does. Where both give a percentage, the
fetched and kept like a blocklist (see "Reputation"). Each entry counts like
an `asn=` tag in `SWW_TAG_LIMIT_PERCENT`. Where both give a percentage, the
lower applies.
- A client's limit percentage is the lowest of those that apply to it: its AS
number's, its country's, or a reputation hit's `limit:<percent>`. A client
without a country gets `SWW_UNKNOWN_LIMIT_PERCENT` in place of a country's.
The percentage is 100 if none applies. Clients given a lower percentage are
banned after fewer requests, first for a while and then, for repeated abuse,
permanently. The log line of each counted request, and the notes of each ban,
say which setting lowered the limit.
number's, its country's, its tags', or a reputation hit's `limit:<percent>`. A
client without a country gets `SWW_UNKNOWN_LIMIT_PERCENT` in place of a
country's. The percentage is 100 if none applies. Clients given a lower
percentage are banned after fewer requests, first for a while and then, for
repeated abuse, permanently. The log line of each counted request, and the
notes of each ban, say which setting lowered the limit.
- No budget is shared by a whole country or AS number. One abuser could use it
up and lock out everyone else there: a denial of service nobody chose, and the
one this tool exists to prevent. Refusing a whole country is left to the
@@ -635,7 +638,7 @@ files, the limits and the bans still apply to the client that sends it.
- `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` names a file of exclusions, written as
Coraza directives. An exclusion switches one rule off for some fields or paths
only, where `SWW_WAF_DISABLED_RULES` switches it off everywhere. The file
holds an app's own exclusions; the eighth change below holds gitea's.
holds an app's own exclusions; gitea's fixes below hold gitea's.
- `discriminator` loads the file after `smallwebwaf`'s own setup rules and
before the Core Rule Set's rules. The Core Rule Set loads its own
`REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf` at the same point.
@@ -681,7 +684,7 @@ files, the limits and the bans still apply to the client that sends it.
scanner from an ordinary client. In front of a code forge, registry and git
clients get `404` and `401` by design, hundreds of times a minute.
`smallwebwaf` changes the Core Rule Set 4.25.0 in eight ways, since in front of
`smallwebwaf` changes the Core Rule Set 4.25.0 in six ways, since in front of
gitea it would otherwise refuse ordinary requests. The changes hold in front of
every app, and no setting undoes them:
@@ -700,8 +703,52 @@ every app, and no setting undoes them:
and tea, which gitea registers for OAuth sign-in, ask to be sent back to
`http://127.0.0.1` on the user's own machine. The server never fetches that
address.
4. Ten query parameters are not checked against the Core Rule Set's lists of
system files (930120), shell paths (932160) and command names (932260):
4. `Referer` is not checked for a Unix command given without arguments (932340),
for Java starting a process (944110) or for a Java class name (944130).
`Referer` names the page a request comes from, on everything that page loads
and on every link followed from it. The Core Rule Set takes some page
addresses for attacks:
- a search for `env`;
- OpenJDK's `Runtime.java`, whose path holds both `runtime` and `java.`;
- a file named `java.lang.String.md`.
Every request from such a page would be refused. `Referer` keeps the other
rules, except after some gitea searches (gitea's fixes). An app that passes
`Referer` to a shell must check it itself.
5. Responses are not inspected. A raw file from a repository, such as a shell
script, looks to the response rules like source code leaking from the server.
6. The `from` parameter of a container registry's upload path,
`/v2/<image>/blobs/uploads/`, is not checked for command names (932260). When
a push reuses a layer, `from` names the image the layer comes from. An image
such as `owner/python3-app` would have that push refused.
Each supported app also has its own fixes, which let its ordinary requests
through. An app's fixes hold for the hostnames `SWW_APPS` lists the app for:
- `SWW_APPS` holds entries separated by semicolons. Each is a hostname, a colon,
then one or more apps separated by commas, for example
`git.example.com:gitea;code.example.com:gitea`.
- The request's `Host` is matched by exact name, without its port and without
regard to case. There are no wildcards.
- A hostname not listed gets no app's fixes, so every rule checks it in full.
Set empty, `SWW_APPS` lists no hostname.
- Unset, every app's fixes hold for every hostname. An app's fixes then weaken
the check of any other app that serves the same paths or fields.
- Any of these mistakes stops the start:
- an unknown app;
- a hostname that is not a DNS name, such as `*.example.com`;
- a hostname listed twice;
- an app listed twice for one hostname;
- an entry without a colon.
`gitea` is the only app supported. Each app's fixes are in one file, such as
`internal/waf/gitea.conf`, and one entry of `apps` in `internal/waf/waf.go`.
gitea's fixes are:
1. Ten query parameters are not checked against the Core Rule Set's lists of
system files (930120), shell paths (932160) and command names (932260), on
any path:
- `path`, `files`, `skip-to` and `sub_path`;
- `ref`, `sha`, `branch` and `workflow`;
- `artifactName` and `redirect_to`.
@@ -714,41 +761,18 @@ every app, and no setting undoes them:
catch reaches the app, such as the command `|cat /etc/passwd`. Path
traversal, SQL and script injection, and PHP, Java and Node.js code are
still refused there. Other parameters keep all three rules, except on the
gitea paths the eighth change names. An app that uses one of these
parameters as a file on the server, or passes it to a shell, must check the
value itself. A rule file can refuse it instead.
gitea paths the third fix names.
5. Two cookies are left out, and `Referer` keeps fewer rules:
- The Core Rule Set reads the request without the `gitea_flash` and
2. The Core Rule Set reads the request without the `gitea_flash` and
`redirect_to` cookies. gitea puts a message naming what was just done into
`gitea_flash`. It puts into `redirect_to` the page to return to after
signing in. The Core Rule Set takes file names in them, such as
`package.json`, for attacks. A browser sends such a cookie with every
request until gitea replaces it. gitea cannot replace it while those
requests are refused, so one refusal would keep the browser out of the
whole site. gitea shows the message with any script removed and returns
only to its own pages. Every other cookie is read in full. An app that
reads a cookie of either name must check it itself.
- `Referer` is not checked for a Unix command given without arguments
(932340), for Java starting a process (944110) or for a Java class name
(944130). `Referer` names the page a request comes from, on everything
that page loads and on every link followed from it. The Core Rule Set
takes some page addresses for attacks:
- a search for `env`;
- OpenJDK's `Runtime.java`, whose path holds both `runtime` and `java.`;
- a file named `java.lang.String.md`.
Every request from such a page would be refused. `Referer` keeps the
other rules, except after some gitea searches (the eighth change). An
app that passes `Referer` to a shell must check it itself.
6. Responses are not inspected. A raw file from a repository, such as a shell
script, looks to the response rules like source code leaking from the server.
7. The `from` parameter of a container registry's upload path,
`/v2/<image>/blobs/uploads/`, is not checked for command names (932260). When
a push reuses a layer, `from` names the image the layer comes from. An image
such as `owner/python3-app` would have that push refused.
8. On gitea's paths, some fields skip the rules that refused ordinary gitea
`gitea_flash`. It puts into `redirect_to` the page to return to after signing
in. The Core Rule Set takes file names in them, such as `package.json`, for
attacks. A browser sends such a cookie with every request until gitea
replaces it. gitea cannot replace it while those requests are refused, so one
refusal would keep the browser out of the whole site. gitea shows the message
with any script removed and returns only to its own pages. Every other cookie
is read in full.
3. On gitea's paths, some fields skip the rules that refused ordinary gitea
requests there. They are the fields where gitea sends a search, a file name
or a branch name. `internal/waf/gitea.conf` lists each rule and the request
it refused. The fields are:
@@ -766,8 +790,7 @@ every app, and no setting undoes them:
- packages.
A user's or an organisation's page is any path of one segment, so on
every such path, on any app, `q` skips these three rules, for example on
`/search`.
every such path `q` skips these three rules, for example on `/search`.
Some of these searches also skip the rule for Java class names (944130),
which refused `org.apache.commons.logging`.
@@ -805,11 +828,14 @@ every app, and no setting undoes them:
`actions/upload-artifact@v3` (920450).
Elsewhere these fields keep every rule. The paths are those of gitea at the
site root. An app other than gitea that serves one of them must check these
fields itself.
site root.
The parameter and cookie names in these changes are matched without regard to
case, as Coraza matches them.
On a hostname that gets gitea's fixes, an app other than gitea must check these
fields and cookies itself. Listing each hostname's own apps in `SWW_APPS` keeps
the full check for the others.
The parameter names in these changes and fixes are matched without regard to
case, and the cookie names exactly.
### Ban lengths, response and scope
@@ -1265,7 +1291,7 @@ seconds, and `SWRP_DEFAULT_DISPOSITION` then decides. So at the defaults,
and whatever its outside sources do.
GeoJS is the only outside source a question can wait on. A question waits only
when step 5 says so, and for at most `SWW_LOOKUP_TIMEOUT`, by default 800
when step 6 says so, and for at most `SWW_LOOKUP_TIMEOUT`, by default 800
milliseconds. A client without an answer by then counts as unknown. No question
waits on these, which run in the background:
+38 -5
View File
@@ -90,11 +90,11 @@ between the two (see "The image's command" below).
## Static lists
- `SWW_ALLOW_NETS` (default empty): netblocks you trust, such as your monitoring
(see step 3 in [`SPEC.md`](../SPEC.md)).
(see step 4 in [`SPEC.md`](../SPEC.md)).
- `SWW_RATE_LIMIT_EXEMPT_NETS` (default empty): netblocks whose clients are
exempt from rate limiting and the byte limits.
- `SWW_DENY_NETS` (default empty): netblocks whose clients are refused (see step
3 in [`SPEC.md`](../SPEC.md)).
4 in [`SPEC.md`](../SPEC.md)).
## Request rate limits, per client
@@ -164,7 +164,7 @@ Every request gets tags from what `discriminator` knows of its client. See
So is a URL that is also in `SWW_BLOCKLIST_URLS`, `SWW_ASN_LIMIT_PERCENT_URL`,
or the CrowdSec decision list.
- `SWW_TAG_DENY` (default empty): tags whose requests are refused, such as
`torexit,country=cn`. A tag listed twice, an `asn=` tag without a number, or a
`torexit,country=CN`. A tag listed twice, an `asn=` tag without a number, or a
`country=` tag without a country, is refused.
- `SWW_TAG_LIMIT_PERCENT` (default empty): tags that lower a client's limits, as
tag and percentage pairs, such as `vpn:50,asn=14061:25`. A percentage is from
@@ -173,6 +173,21 @@ Every request gets tags from what `discriminator` knows of its client. See
`exploitsignature`: a request gets that tag after the rate limits have counted
it.
`SWW_TAG_DENY` and `SWW_TAG_LIMIT_PERCENT` accept tags in either case. A tag
that no request can get stops the start. A request can get only these:
- a name in `SWW_TAG_LISTS`;
- an `asn=` or `country=` tag;
- a `dnsbl=` tag of a zone in `SWW_DNSBL_ZONES`, in lower case, with a key
masked as the `reputation` field shows it;
- `abuseipdb`, while `SWW_ABUSEIPDB_KEY` is set;
- `exploitsignature`, in `SWW_TAG_DENY` only, while `SWW_RULES_ENABLED` is
`true`, or while neither `SWW_WAF_MODE` nor `SWW_WAF_ANOMALY_THRESHOLD` is
`off`.
`crowdsec` stops the start in both settings, because a CrowdSec decision refuses
the request first.
## Biased thresholds
- `SWW_ASN_BYTES_PERCENT` and `SWW_COUNTRY_BYTES_PERCENT` (default empty): a
@@ -184,8 +199,8 @@ Every request gets tags from what `discriminator` knows of its client. See
lookup cannot place.
- `SWW_ASN_LIMIT_PERCENT_URL` (default unset): the `http` or `https` URL of a
text file with one AS number and percentage on each line, such as
`AS64496:50`. Each entry counts as an `asn=` tag in `SWW_TAG_LIMIT_PERCENT`
does. A URL that is also in `SWW_BLOCKLIST_URLS` is refused.
`AS64496:50`. Each entry counts like an `asn=` tag in `SWW_TAG_LIMIT_PERCENT`.
A URL that is also in `SWW_BLOCKLIST_URLS` is refused.
## Attack detection
@@ -218,6 +233,24 @@ Every request gets tags from what `discriminator` knows of its client. See
a range. So does a directive that removes or changes rules as the file loads,
such as `SecRuleRemoveById`, since none of the Core Rule Set's rules is loaded
yet. It has no `_FILE` form.
- `SWW_APPS` (default unset): the apps each hostname serves. A request gets the
fixes of its hostname's apps only (see "Attack detection" in
[`SPEC.md`](../SPEC.md)).
- Entries are separated by semicolons. Each is a hostname, a colon, then one
or more apps separated by commas, for example
`git.example.com:gitea;code.example.com:gitea`.
- `gitea` is the only app supported.
- The request's `Host` is matched by exact name, without its port and
without regard to case. There are no wildcards.
- A hostname not listed gets no app's fixes, so every rule checks it in
full. Set empty, it lists no hostname.
- Unset, every app's fixes apply to every hostname.
- Any of these mistakes stops the start:
- an unknown app;
- a hostname that is not a DNS name, such as `*.example.com`;
- a hostname listed twice;
- an app listed twice for one hostname;
- an entry without a colon.
- `SWW_ERROR_BURST_THRESHOLD` (default `30`): the most refusals of one client in
a minute (see "Attack detection" in `SPEC.md`). A count or `off`.
+272 -59
View File
@@ -29,6 +29,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// Config is the settings of `sww discriminator`. A time or size limit of
@@ -101,8 +102,8 @@ type Config struct {
// the clients they hold (SWW_TAG_LISTS). TagDeny are the tags whose
// requests are refused (SWW_TAG_DENY). TagLimitPercent gives the clients
// with a tag that percentage of every rate limit and byte limit, from 1
// to 99 (SWW_TAG_LIMIT_PERCENT). All hold tags in lower case, as requests
// carry them.
// to 99 (SWW_TAG_LIMIT_PERCENT). All hold tags as requests carry them: in
// lower case, a country code in capitals.
TagLists []TagList
TagDeny []string
TagLimitPercent map[string]int64
@@ -115,7 +116,7 @@ type Config struct {
// UnknownLimitPercent is the percentage of every limit a client without
// a country gets (SWW_UNKNOWN_LIMIT_PERCENT). ASNLimitPercentURL is
// where a file of AS:percent lines is fetched from, whose percentages
// count as asn= tags in TagLimitPercent do (SWW_ASN_LIMIT_PERCENT_URL),
// count like asn= tags in TagLimitPercent (SWW_ASN_LIMIT_PERCENT_URL),
// "" while it is unset.
ASNBytesPercent map[string]int64
CountryBytesPercent map[string]int64
@@ -165,8 +166,8 @@ type Config struct {
// BanResponse is the status a refused client is answered with, 403
// or 429, or 0 to close the connection without an answer
// (SWW_BAN_RESPONSE). It answers a banned client, a request that
// breaks a rate limit or matches a ban rule, SWW_DENY_NETS and the
// country lists.
// breaks a rate limit or matches a ban rule, SWW_DENY_NETS,
// SWW_EXCLUSIVELY_ALLOWED_COUNTRIES and SWW_TAG_DENY.
BanResponse int
// LimitBanDuration is the ban for a first broken rate limit
// (SWW_LIMIT_BAN_DURATION). A limit broken again within
@@ -231,6 +232,11 @@ type Config struct {
WAFExemptPaths []string
WAFBodyLimit int64
CoreRuleSetExclusionsFile string
// Apps are the names of the apps each hostname serves, by hostname in
// lowercase (SWW_APPS): a request gets only its hostname's apps' fixes
// to the Core Rule Set. Apps is nil while SWW_APPS is unset, and every
// request gets every app's fixes.
Apps map[string][]string
// TrapPaths are the paths a request for which is a clear sign of
// attack (SWW_TRAP_PATHS), each starting with / and without a ?.
TrapPaths []string
@@ -398,18 +404,26 @@ var (
errNotPathPrefix = errors.New(
"is not a path prefix starting with /, such as /assets/")
errNotTrapPath = errors.New(
"is not a path starting with / and without a ?, such as /wp-login.php")
"is not a path starting with / and without a ?, such as /wp-login.php",
)
errNotAppsEntry = errors.New(
"is not a hostname, : and a list of apps, such as git.example.com:gitea",
)
errNotHostname = errors.New("is not a DNS name such as git.example.com")
errNotApp = errors.New("is not an app sww has fixes for; the apps are")
errNotWAFMode = errors.New("is not off, detect or block")
errNotParanoiaLevel = errors.New("is not a paranoia level, from 1 to 4")
errNotRuleID = errors.New(
"is not the id of a Core Rule Set rule, a whole number such as 942100")
"is not the id of a Core Rule Set rule, a whole number such as 942100",
)
errSetupRuleID = errors.New(
"is from 900000 to 900999, the ids of the rules that set the Core Rule Set " +
"up and of sww's own, which cannot be switched off")
errNotBoolean = errors.New("is not true or false")
errNotLogRemoteURL = errors.New(
"is not syslog+udp, syslog+tcp or syslog+tls with a host and a port, " +
"and nothing more, such as syslog+tls://logs.example:6514")
"and nothing more, such as syslog+tls://logs.example:6514",
)
errNoCertificate = errors.New("holds no PEM certificate")
errNotFacility = errors.New("is not a syslog facility such as local0 or daemon")
errNotAppName = errors.New(
@@ -422,16 +436,21 @@ var (
"is not a header name followed by : and the header's value, " +
"such as Authorization:Bearer <token>")
errControlCharacter = errors.New(
"holds a control character, such as the carriage return of a Windows line end")
"holds a control character, such as the carriage return of a Windows line end",
)
errNotAlertEvent = errors.New(
"is not ban, permanent_ban, waf_block, anomaly, reputation_hit, " +
"source_failure or file_error")
errNotNumberOrOff = errors.New("is not a whole number above zero, such as 60, or off")
errNotNumberOrOff = errors.New(
"is not a whole number above zero, such as 60, or off",
)
errNotUTF8 = errors.New("is not valid UTF-8")
errNotASN = errors.New("is not an AS number such as AS64496")
errNotPercentItem = errors.New(
"is not a code, : and a percentage, such as AS64496:50 or cn:25")
errNotPercent = errors.New("is not a percentage, a whole number from 0 to 100")
errNotPercent = errors.New(
"is not a percentage, a whole number from 0 to 100",
)
errListedTwice = errors.New("is listed twice")
errNotListURL = errors.New(
"is not an http or https URL without a user or a fragment, " +
@@ -440,11 +459,17 @@ var (
errNotAnHourOrMore = errors.New("is not a duration of 1h or more, such as 24h")
errNotAction = errors.New(
"is not deny, limit:<percent> such as limit:25, or log")
errNotZone = errors.New("is not a DNS zone such as dnsbl.dronebl.org")
errZoneTooLong = errors.New("is longer than 189 characters, too long for the " +
"names IPv6 clients are asked about by")
errNotResolver = errors.New("is not an IP address with an optional port, " +
"such as 192.0.2.53 or [2001:db8::53]:5353")
errNotZone = errors.New(
"is not a DNS zone such as dnsbl.dronebl.org",
)
errZoneTooLong = errors.New(
"is longer than 189 characters, too long for the " +
"names IPv6 clients are asked about by",
)
errNotResolver = errors.New(
"is not an IP address with an optional port, " +
"such as 192.0.2.53 or [2001:db8::53]:5353",
)
errNotLAPIURL = errors.New(
"is not an http or https URL without a user or a fragment, " +
"such as http://172.17.0.1:8080")
@@ -464,6 +489,13 @@ var (
errExploitTag = errors.New(
"cannot lower limits: the rate limits count a request before its " +
"rules are matched")
errCrowdSecTag = errors.New(
"cannot be used: a CrowdSec decision refuses the request first")
errNoRequestGetsTag = errors.New(
"is a tag no request can get: it is not a name in SWW_TAG_LISTS, an " +
"asn= or country= tag, a dnsbl= tag of a zone in SWW_DNSBL_ZONES, " +
"abuseipdb with SWW_ABUSEIPDB_KEY set, or exploitsignature with " +
"rule files or the Core Rule Set on")
)
// FromEnvironment reads the settings with lookupEnv, normally
@@ -473,7 +505,9 @@ var (
// setting that is set but invalid is an error that names it.
//
//nolint:funlen // one line for each setting, a list that grows with them
func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
func FromEnvironment(
lookupEnv func(string) (string, bool),
) (*Config, error) {
env := &environment{lookupEnv: lookupEnv}
cfg := &Config{
ListenAddr: env.address("SWW_LISTEN_ADDR"),
@@ -543,6 +577,7 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
WAFBodyLimit: env.wafBodyLimit("SWW_WAF_BODY_LIMIT", off),
CoreRuleSetExclusionsFile: env.filePath(
"SWW_CORE_RULE_SET_EXCLUSIONS_FILE"),
Apps: env.apps("SWW_APPS"),
TrapPaths: env.trapPaths("SWW_TRAP_PATHS"),
ErrorBurstThreshold: env.count("SWW_ERROR_BURST_THRESHOLD", "30"),
LogRemoteURL: env.logRemoteURL("SWW_LOG_REMOTE_URL"),
@@ -581,6 +616,7 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
env.checkASNLimitPercentURL(cfg)
env.checkCrowdSec(cfg)
env.checkTagLists(cfg)
env.checkTags(cfg)
if env.err != nil {
return nil, env.err
@@ -672,7 +708,10 @@ func (e *environment) lookup(name string) (string, bool) {
}
if set {
e.check(name, fmt.Errorf("is set, and so is %s; %w", fileName, errSetTwice))
e.check(
name,
fmt.Errorf("is set, and so is %s; %w", fileName, errSetTwice),
)
return value, set
}
@@ -794,6 +833,25 @@ func (e *environment) pathPrefixes(name, defaultValue string) []string {
return prefixes
}
// apps reads the setting that is the apps each hostname serves. It is nil
// while the setting is unset, and logged as null, unlike a setting set
// empty.
func (e *environment) apps(name string) map[string][]string {
value, set := e.lookup(name)
if !set {
e.settings = append(e.settings, slog.Any(name, nil))
return nil
}
e.settings = append(e.settings, slog.String(name, value))
apps, err := parseApps(value)
e.check(name, err)
return apps
}
// trapPaths reads the setting that is the list of trap paths. It is empty
// by default.
func (e *environment) trapPaths(name string) []string {
@@ -955,7 +1013,10 @@ func (e *environment) zones(name string) []string {
logged[i] = MaskZoneKey(zone)
}
e.settings = append(e.settings, slog.String(name, strings.Join(logged, ",")))
e.settings = append(
e.settings,
slog.String(name, strings.Join(logged, ",")),
)
return zones
}
@@ -1024,7 +1085,7 @@ func (e *environment) checkLookupDBPath(cfg *Config) {
// limit, and the anomaly thresholds per AS number.
func (e *environment) checkCountriesAndLookups(cfg *Config) {
for _, country := range cfg.ExclusivelyAllowedCountries {
if slices.Contains(cfg.TagDeny, countryTagPrefix+strings.ToLower(country)) {
if slices.Contains(cfg.TagDeny, CountryTagPrefix+country) {
e.check("SWW_EXCLUSIVELY_ALLOWED_COUNTRIES",
fmt.Errorf("%q %w", country, errOnBothLists))
}
@@ -1098,6 +1159,56 @@ func (e *environment) checkTagLists(cfg *Config) {
}
}
// checkTags refuses a tag of SWW_TAG_DENY or SWW_TAG_LIMIT_PERCENT that would
// never apply: crowdsec, whose request is refused before tags are acted on, and
// any tag these settings cannot give a request.
func (e *environment) checkTags(cfg *Config) {
for _, tag := range cfg.TagDeny {
e.check("SWW_TAG_DENY", checkTag(cfg, tag))
}
for tag := range cfg.TagLimitPercent {
e.check("SWW_TAG_LIMIT_PERCENT", checkTag(cfg, tag))
}
}
// checkTag returns an error when tag would never apply, as checkTags says.
func checkTag(cfg *Config, tag string) error {
if tag == CrowdSecTag {
return fmt.Errorf("%q %w", tag, errCrowdSecTag)
}
if !requestCanGetTag(cfg, tag) {
return fmt.Errorf("%q %w", MaskZoneKey(tag), errNoRequestGetsTag)
}
return nil
}
// requestCanGetTag reports whether anything cfg sets up can give a request tag.
func requestCanGetTag(cfg *Config, tag string) bool {
isListName := func(list TagList) bool { return list.Name == tag }
if isLookupTag(tag) || slices.ContainsFunc(cfg.TagLists, isListName) {
return true
}
if tag == ExploitSignatureTag {
coreRuleSetOn := cfg.WAFMode != WAFModeOff && cfg.WAFAnomalyThreshold != 0
return cfg.RulesEnabled || coreRuleSetOn
}
if tag == AbuseIPDBTag {
return cfg.AbuseIPDBKey != ""
}
zone, isZoneTag := strings.CutPrefix(tag, DNSBLTagPrefix)
return isZoneTag && slices.ContainsFunc(cfg.DNSBLZones, func(each string) bool {
return strings.ToLower(MaskZoneKey(each)) == zone
})
}
// TagsNeedLookup reports whether SWW_TAG_DENY or SWW_TAG_LIMIT_PERCENT
// names an asn= or country= tag, which only the lookup can give a request.
func (c *Config) TagsNeedLookup() bool {
@@ -1118,7 +1229,7 @@ func tagLimitNeedsLookups(cfg *Config) bool {
// isLookupTag reports whether tag is an asn= or country= tag.
func isLookupTag(tag string) bool {
return strings.HasPrefix(tag, asnTagPrefix) || strings.HasPrefix(tag, countryTagPrefix)
return strings.HasPrefix(tag, ASNTagPrefix) || strings.HasPrefix(tag, CountryTagPrefix)
}
// headerNames reads a setting that is a list of header names, and
@@ -1132,7 +1243,9 @@ func (e *environment) headerNames(name, defaultValue string) []string {
// durationNotOff reads a setting that is a duration and, unlike a
// timeout, cannot be off.
func (e *environment) durationNotOff(name, defaultValue string) time.Duration {
func (e *environment) durationNotOff(
name, defaultValue string,
) time.Duration {
duration, err := ParseDurationNotOff(e.value(name, defaultValue))
e.check(name, err)
@@ -1331,7 +1444,10 @@ func (e *environment) instanceName() string {
// appName reads the setting that is the APP-NAME of the records the log
// lines are sent in, by default the instance name. Its value is checked
// when it is set, and, while lines are sent, when it is the instance name.
func (e *environment) appName(name, instanceName string, sending bool) string {
func (e *environment) appName(
name, instanceName string,
sending bool,
) string {
value, set := e.lookup(name)
if !set {
value = instanceName
@@ -1354,7 +1470,10 @@ func (e *environment) appName(name, instanceName string, sending bool) string {
// checkInstanceNameForNtfy refuses an instance name that holds a control
// character while ntfySet, SWW_ALERT_NTFY_URL being set: ntfy is sent
// the instance name in a header, which cannot hold one.
func (e *environment) checkInstanceNameForNtfy(instanceName string, ntfySet bool) {
func (e *environment) checkInstanceNameForNtfy(
instanceName string,
ntfySet bool,
) {
if ntfySet && strings.ContainsFunc(instanceName, unicode.IsControl) {
e.check("SWW_INSTANCE_NAME", fmt.Errorf(
"%q %w, and is sent to ntfy in a header while SWW_ALERT_NTFY_URL is set",
@@ -1678,7 +1797,10 @@ func parseNamedNetblocks(value string) ([]anomaly.NamedNetblock, error) {
return nil, fmt.Errorf("%q %w", name, errListedTwice)
}
named = append(named, anomaly.NamedNetblock{Name: name, Netblock: netblock})
named = append(
named,
anomaly.NamedNetblock{Name: name, Netblock: netblock},
)
}
return named, nil
@@ -1720,6 +1842,65 @@ func parseTrapPaths(value string) ([]string, error) {
return paths, nil
}
// parseApps reads the apps each hostname serves: entries separated by
// semicolons, each a hostname, a colon, and a comma-separated list of the
// names waf.AppNames returns, such as
// git.example.com:gitea;cloud.example.com:nextcloud,wordpress. Each
// hostname is a DNS name, without a wildcard. It returns them by hostname
// in lowercase, since a hostname is matched without regard to case. An
// empty value lists no hostname.
func parseApps(value string) (map[string][]string, error) {
apps := map[string][]string{}
if strings.TrimSpace(value) == "" {
return apps, nil
}
for entry := range strings.SplitSeq(value, ";") {
hostname, list, found := strings.Cut(entry, ":")
if !found {
return nil, fmt.Errorf("%q %w", entry, errNotAppsEntry)
}
hostname = strings.ToLower(strings.TrimSpace(hostname))
if hostname == "" {
return nil, fmt.Errorf("%q %w", entry, errNotAppsEntry)
}
if !isZone(hostname) {
return nil, fmt.Errorf("%q %w", hostname, errNotHostname)
}
if _, listed := apps[hostname]; listed {
return nil, fmt.Errorf("%q %w", hostname, errListedTwice)
}
names, err := parseList(list)
if err != nil {
return nil, err
}
if len(names) == 0 {
return nil, fmt.Errorf("%q %w", entry, errNotAppsEntry)
}
for i, name := range names {
if !slices.Contains(waf.AppNames(), name) {
return nil, fmt.Errorf("%q %w %s", name, errNotApp,
strings.Join(waf.AppNames(), ", "))
}
if slices.Contains(names[:i], name) {
return nil, fmt.Errorf("%q %w", name, errListedTwice)
}
}
apps[hostname] = names
}
return apps, nil
}
// parseRuleIDs reads a comma-separated list of the ids of Core Rule Set
// rules, each a whole number above zero and outside firstSetupRuleID to
// lastSetupRuleID.
@@ -1826,16 +2007,19 @@ func ParseASN(value string) (string, error) {
return "AS" + strconv.FormatUint(number, 10), nil
}
// The prefixes of the tags the lookup gives a request.
// The tags a request can get that the settings can name, as discriminator
// writes them. The prefixes are followed by a value.
const (
asnTagPrefix = "asn="
countryTagPrefix = "country="
ASNTagPrefix = "asn="
CountryTagPrefix = "country="
DNSBLTagPrefix = "dnsbl="
AbuseIPDBTag = "abuseipdb"
CrowdSecTag = "crowdsec"
// ExploitSignatureTag is the tag of a request a rule file or the Core
// Rule Set matched.
ExploitSignatureTag = "exploitsignature"
)
// ExploitSignatureTag is the tag of a request a rule file or the Core Rule
// Set matched.
const ExploitSignatureTag = "exploitsignature"
// TagList is an address list of SWW_TAG_LISTS: the tag it adds to the
// requests of the clients it holds, and the URL it is fetched from.
type TagList struct {
@@ -1876,11 +2060,13 @@ func parseTagLists(value string) ([]TagList, error) {
}
// parseTags reads a comma-separated list of tags, which parseTag reads. A
// tag listed twice is an error.
// tag listed twice is an error, which shows the tag as MaskZoneKey does.
func parseTags(value string) ([]string, error) {
items, err := parseList(value)
if err != nil {
return nil, err
// parseList's error, for an empty item, shows the whole value, keys
// included.
return nil, errEmptyItem
}
tags := make([]string, 0, len(items))
@@ -1892,7 +2078,7 @@ func parseTags(value string) ([]string, error) {
}
if slices.Contains(tags, tag) {
return nil, fmt.Errorf("%q %w", item, errListedTwice)
return nil, fmt.Errorf("%q %w", MaskZoneKey(item), errListedTwice)
}
tags = append(tags, tag)
@@ -1901,27 +2087,30 @@ func parseTags(value string) ([]string, error) {
return tags, nil
}
// parseTag reads a tag in either case, and returns it as a request carries
// it: in lower case, an asn= tag's number without leading zeros. An asn= or
// country= tag must name a number or a country, so that a mistyped one
// stops the start instead of never matching.
// parseTag reads a tag in either case, with spaces around it, and returns it
// as a request carries it: in lower case, but for a country code, which is in
// capitals, and an asn= tag's number without leading zeros. An asn= or
// country= tag must name a number or a country, so that a mistyped one stops
// the start instead of never matching.
func parseTag(value string) (string, error) {
tag := strings.ToLower(value)
tag := strings.ToLower(strings.TrimSpace(value))
if digits, isASN := strings.CutPrefix(tag, asnTagPrefix); isASN {
if digits, isASN := strings.CutPrefix(tag, ASNTagPrefix); isASN {
number, err := strconv.ParseUint(digits, 10, 32)
if err != nil {
return "", fmt.Errorf("%q %w", value, errNotASNTag)
}
return asnTagPrefix + strconv.FormatUint(number, 10), nil
return ASNTagPrefix + strconv.FormatUint(number, 10), nil
}
if code, isCountry := strings.CutPrefix(tag, countryTagPrefix); isCountry {
_, err := parseCountry(code)
if code, isCountry := strings.CutPrefix(tag, CountryTagPrefix); isCountry {
country, err := parseCountry(code)
if err != nil {
return "", err
}
return CountryTagPrefix + country, nil
}
return tag, nil
@@ -1934,7 +2123,9 @@ func parseTag(value string) (string, error) {
func parseTagPercents(value string) (map[string]int64, error) {
items, err := parseList(value)
if err != nil {
return nil, err
// parseList's error, for an empty item, shows the whole value, keys
// included.
return nil, errEmptyItem
}
percents := make(map[string]int64, len(items))
@@ -1942,7 +2133,7 @@ func parseTagPercents(value string) (map[string]int64, error) {
for _, item := range items {
colon := strings.LastIndex(item, ":")
if colon <= 0 {
return nil, fmt.Errorf("%q %w", item, errNotTagPercentItem)
return nil, fmt.Errorf("%q %w", MaskZoneKey(item), errNotTagPercentItem)
}
tag, err := parseTag(item[:colon])
@@ -1952,7 +2143,7 @@ func parseTagPercents(value string) (map[string]int64, error) {
percent, err := strconv.ParseInt(item[colon+1:], 10, 64)
if err != nil || percent < 1 || percent > 99 {
return nil, fmt.Errorf("%q %w", item[colon+1:], errNotTagPercent)
return nil, fmt.Errorf("%q %w", MaskZoneKey(item[colon+1:]), errNotTagPercent)
}
if tag == ExploitSignatureTag {
@@ -1960,7 +2151,9 @@ func parseTagPercents(value string) (map[string]int64, error) {
}
if _, listed := percents[tag]; listed {
return nil, fmt.Errorf("%q %w", item[:colon], errListedTwice)
// tag, since item[:colon] may end in a space, which MaskZoneKey
// would not mask.
return nil, fmt.Errorf("%q %w", MaskZoneKey(tag), errListedTwice)
}
percents[tag] = percent
@@ -2060,8 +2253,11 @@ func parseHeaderNames(value string) ([]string, error) {
header := strings.ToLower(item)
switch header {
case "host":
return nil, fmt.Errorf("%q %w; the request's host is the field host",
item, errHeaderTakenOut)
return nil, fmt.Errorf(
"%q %w; the request's host is the field host",
item,
errHeaderTakenOut,
)
case "transfer-encoding":
return nil, fmt.Errorf("%q %w", item, errHeaderTakenOut)
}
@@ -2098,9 +2294,11 @@ func parseUpstreamURL(value string) (*url.URL, error) {
}
onlySchemeAndHost := (upstream.Scheme == "http" || upstream.Scheme == "https") &&
upstream.Hostname() != "" && upstream.User == nil && upstream.Opaque == "" &&
upstream.Hostname() != "" && upstream.User == nil &&
upstream.Opaque == "" &&
(upstream.Path == "" || upstream.Path == "/") &&
upstream.RawQuery == "" && upstream.Fragment == ""
upstream.RawQuery == "" &&
upstream.Fragment == ""
if !onlySchemeAndHost {
return nil, fmt.Errorf("%q %w", value, errNotUpstreamURL)
}
@@ -2124,7 +2322,11 @@ func parseLogRemoteURL(value string) (*url.URL, error) {
return nil, fmt.Errorf("%q %w", value, errNotLogRemoteURL)
}
schemes := []string{remotelog.SchemeUDP, remotelog.SchemeTCP, remotelog.SchemeTLS}
schemes := []string{
remotelog.SchemeUDP,
remotelog.SchemeTCP,
remotelog.SchemeTLS,
}
port, err := strconv.ParseUint(remote.Port(), 10, 16)
onlySchemeHostAndPort := slices.Contains(schemes, remote.Scheme) &&
@@ -2187,7 +2389,9 @@ func isHTTPURL(u *url.URL) bool {
return (u.Scheme == "http" || u.Scheme == "https") && u.Hostname() != "" &&
(u.Port() == "" || (err == nil && port != 0)) &&
u.User == nil && u.Opaque == "" && u.Fragment == ""
u.User == nil &&
u.Opaque == "" &&
u.Fragment == ""
}
// isListURL reports whether value is a URL a list can be fetched from, as
@@ -2254,9 +2458,12 @@ func parseZones(value string) ([]string, error) {
for i, zone := range zones {
shown := MaskZoneKey(zone)
listedBefore := slices.ContainsFunc(zones[:i], func(earlier string) bool {
listedBefore := slices.ContainsFunc(
zones[:i],
func(earlier string) bool {
return strings.EqualFold(MaskZoneKey(earlier), shown)
})
},
)
switch {
case len(zone) > maxZoneLength:
@@ -2289,7 +2496,8 @@ func MaskZoneKey(zone string) string {
return masked + "." + rest
}
// isZone reports whether each label of zone is as parseZones takes it.
// isZone reports whether each label of zone is as parseZones takes it. It
// also checks the hostnames of SWW_APPS, which are DNS names too.
func isZone(zone string) bool {
for label := range strings.SplitSeq(zone, ".") {
badChar := strings.ContainsFunc(label, func(char rune) bool {
@@ -2343,8 +2551,13 @@ func parseWebhookHeaders(value string) (http.Header, string, error) {
name, headerValue, found := strings.Cut(item, ":")
name = strings.TrimSpace(name)
if !found || !IsHeaderName(name) || strings.ContainsAny(headerValue, "\r\n\x00") {
return nil, "", fmt.Errorf("item %d %w", i+1, errNotWebhookHeader)
if !found || !IsHeaderName(name) ||
strings.ContainsAny(headerValue, "\r\n\x00") {
return nil, "", fmt.Errorf(
"item %d %w",
i+1,
errNotWebhookHeader,
)
}
headers.Add(name, strings.TrimSpace(headerValue))
+226 -14
View File
@@ -88,6 +88,7 @@ const (
wafExemptPaths = "SWW_WAF_EXEMPT_PATHS"
wafBodyLimit = "SWW_WAF_BODY_LIMIT"
exclusionsFile = "SWW_CORE_RULE_SET_EXCLUSIONS_FILE"
apps = "SWW_APPS"
trapPaths = "SWW_TRAP_PATHS"
errorBurstThreshold = "SWW_ERROR_BURST_THRESHOLD"
logRemoteURL = "SWW_LOG_REMOTE_URL"
@@ -208,6 +209,9 @@ const off = "off"
// enabled is true, as a setting's value.
const enabled = "true"
// disabled is false, as a setting's value.
const disabled = "false"
// defaultLookupSource is the default of SWW_LOOKUP_SOURCE, and
// fileSource the source that is the lookup database.
const (
@@ -345,7 +349,7 @@ func TestValuesAsSet(t *testing.T) {
metricsToken: token,
metricsTopN: "10",
rulesDir: "/srv/waf-rules",
rulesEnabled: "false",
rulesEnabled: disabled,
})
wantSettings(t, cfg, config.Config{
@@ -578,6 +582,92 @@ func TestCoreRuleSetSettings(t *testing.T) {
}
}
func TestApps(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
env environment
want map[string][]string
}{
{environment{}, nil},
{environment{apps: ""}, map[string][]string{}},
{
environment{apps: "Git.Example.com:gitea; cloud.example.com : gitea "},
map[string][]string{
"git.example.com": {"gitea"}, "cloud.example.com": {"gitea"},
},
},
} {
cfg := fromEnvironment(t, tc.env)
if !reflect.DeepEqual(cfg.Apps, tc.want) {
t.Errorf("%v gave %#v, want %#v", tc.env, cfg.Apps, tc.want)
}
}
}
// Unset and set empty, SWW_APPS has opposite effects, so it logs differently.
func TestUnsetAndEmptyAppsAreLoggedDifferently(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
env environment
want string
}{
{environment{}, `"` + apps + `":null`},
{environment{apps: ""}, `"` + apps + `":""`},
} {
var out bytes.Buffer
slog.New(slog.NewJSONHandler(&out, nil)).
Info("starting", "settings", fromEnvironment(t, tc.env))
if !strings.Contains(out.String(), tc.want) {
t.Errorf("%v logged %s, want %s in it", tc.env, out.String(), tc.want)
}
}
}
func TestInvalidAppsStopsTheStart(t *testing.T) {
t.Parallel()
const (
notEntry = " is not a hostname, : and a list of apps, " +
"such as git.example.com:gitea"
notHostname = " is not a DNS name such as git.example.com"
)
for _, tc := range []struct{ value, want string }{
{"git.example.com:gitea;blog.example.com", `"blog.example.com"` + notEntry},
{"git.example.com:gitea;", `""` + notEntry},
{":gitea", `":gitea"` + notEntry},
{"git.example.com:", `"git.example.com:"` + notEntry},
{"*.example.com:gitea", `"*.example.com"` + notHostname},
{"git.example.com/:gitea", `"git.example.com/"` + notHostname},
{"git example.com:gitea", `"git example.com"` + notHostname},
{
"git.example.com:gitea,nextcloud",
`"nextcloud" is not an app sww has fixes for; the apps are gitea`,
},
{
"git.example.com:gitea;GIT.example.com:gitea",
`"git.example.com" is listed twice`,
},
{"git.example.com:gitea,gitea", `"gitea" is listed twice`},
{"git.example.com:gitea,", `"gitea," has an empty item in its list`},
} {
t.Run(tc.value, func(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{apps: tc.value}.lookupEnv)
want := apps + ": " + tc.want
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
})
}
}
func TestWAFBodyLimitOf64K(t *testing.T) {
t.Parallel()
@@ -1099,7 +1189,7 @@ func TestInstanceNameNotUTF8StopsTheStart(t *testing.T) {
}
}
func TestCodeOnBothCountryListsStopsTheStart(t *testing.T) {
func TestCodeInAllowedCountriesAndTagDenyStopsTheStart(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{
@@ -1107,7 +1197,7 @@ func TestCodeOnBothCountryListsStopsTheStart(t *testing.T) {
allowedCountries: "de,RU",
}.lookupEnv)
if err == nil {
t.Fatal("ru on both country lists was accepted")
t.Fatal("ru in SWW_EXCLUSIVELY_ALLOWED_COUNTRIES and SWW_TAG_DENY was accepted")
}
if !strings.HasPrefix(err.Error(), allowedCountries+": ") ||
@@ -1180,7 +1270,7 @@ func TestSettingNeedingLookupsStopsTheStartWhileTheyAreOff(t *testing.T) {
for name, value := range map[string]string{
allowedCountries: "de",
addLookupHeaders: enabled,
tagDeny: "country=kp",
tagDeny: "country=KP",
tagLimitPercent: "asn=64496:50",
asnLimitPercentURL: asnURL,
asnBytesPercent: "AS64496:50",
@@ -1210,6 +1300,7 @@ func TestSettingNeedingLookupsStopsTheStartWhileTheyAreOff(t *testing.T) {
// any other anomaly threshold, or a blocklist.
env := environment{
lookupSource: off, allowedCountries: "", asnBytesPercent: "",
tagLists: "torexit=https://example.com/tor.txt,vpn=https://example.com/vpn.txt",
tagDeny: torExitTag, tagLimitPercent: "vpn:50",
countryBytesPercent: "", unknownLimitPercent: "100", asnLimitPercentURL: "",
blocklistURLs: dropURL,
@@ -1356,8 +1447,12 @@ func TestInvalidAnomalySettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
}
}
// torExitTag is a tag these tests name.
const torExitTag = "torexit"
// torExitTag, zenTag and exploitTag are tags these tests name.
const (
torExitTag = "torexit"
zenTag = "dnsbl=zen.example"
exploitTag = "exploitsignature"
)
func TestTagSettingsAsSet(t *testing.T) {
t.Parallel()
@@ -1369,8 +1464,9 @@ func TestTagSettingsAsSet(t *testing.T) {
cfg := fromEnvironment(t, environment{
tagLists: "TorExit=" + torURL +
", vpn=mullvad=https://example.com/mullvad.txt",
tagDeny: "TorExit, Country=KP, asn=014061",
tagLimitPercent: "vpn:50, asn=014061:25, COUNTRY=cn:1, dnsbl=zen.example:99",
dnsblZones: "Zen.example",
tagDeny: "TorExit, Country=kp, asn=014061",
tagLimitPercent: "vpn=mullvad :50, asn=014061:25, COUNTRY=cn:1, dnsbl=zen.example:99",
})
wantLists := []config.TagList{
@@ -1381,19 +1477,112 @@ func TestTagSettingsAsSet(t *testing.T) {
t.Errorf("%s gave %+v, want %+v", tagLists, cfg.TagLists, wantLists)
}
wantDeny := []string{torExitTag, "country=kp", "asn=14061"}
wantDeny := []string{torExitTag, "country=KP", "asn=14061"}
if !slices.Equal(cfg.TagDeny, wantDeny) {
t.Errorf("%s gave %v, want %v", tagDeny, cfg.TagDeny, wantDeny)
}
wantPercents := map[string]int64{
"vpn": 50, "asn=14061": 25, "country=cn": 1, "dnsbl=zen.example": 99,
"vpn=mullvad": 50, "asn=14061": 25, "country=CN": 1, zenTag: 99,
}
if !maps.Equal(cfg.TagLimitPercent, wantPercents) {
t.Errorf("%s gave %v, want %v", tagLimitPercent, cfg.TagLimitPercent, wantPercents)
}
}
func TestTagThatNoRequestCanGetIsToldApartFromOneItCan(t *testing.T) {
t.Parallel()
const (
list = "torexit=https://example.com/tor.txt"
key = "abc123"
zone = key + ".xbl.dq.spamhaus.net"
masked = "********.xbl.dq.spamhaus.net"
)
for _, tc := range []struct {
name string
env environment
want string // "" is accepted
}{
{"a list's name", environment{tagLists: list, tagDeny: torExitTag}, ""},
{
"a mistyped list's name", environment{tagLists: list, tagDeny: "torexti"},
`"torexti"`,
},
{"a masked zone", environment{dnsblZones: zone, tagDeny: "dnsbl=" + masked}, ""},
{
"a zone with its key", environment{dnsblZones: zone, tagDeny: "dnsbl=" + zone},
masked,
},
{"another zone", environment{dnsblZones: zone, tagDeny: zenTag}, zenTag},
{
"abuseipdb with a key",
environment{abuseIPDBKey: "k", tagLimitPercent: "abuseipdb:50"}, "",
},
{"crowdsec, though a list has its name", environment{
tagLists: "crowdsec=https://example.com/c.txt", tagDeny: "crowdsec",
}, `"crowdsec" cannot be used`},
{exploitTag, environment{tagDeny: exploitTag}, ""},
{
exploitTag + " with rule files off",
environment{tagDeny: exploitTag, rulesEnabled: disabled}, "",
},
{
exploitTag + " with the Core Rule Set off",
environment{tagDeny: exploitTag, wafMode: off}, "",
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(tc.env.lookupEnv)
switch {
case tc.want == "" && err != nil:
t.Errorf("error %v, want none", err)
case tc.want != "" && (err == nil || !strings.Contains(err.Error(), tc.want)):
t.Errorf("error %v, want one holding %s", err, tc.want)
case err != nil && strings.Contains(err.Error(), key):
t.Errorf("error %v shows the zone's key", err)
}
})
}
}
func TestErrorOfATagSettingLeavesOutAZoneKey(t *testing.T) {
t.Parallel()
const (
tag = "dnsbl=" + spamhaus
emptyItem = "has an empty item in its list"
twice = `"` + spamhausMasked + `" is listed twice`
)
for _, tc := range []struct{ name, value, want string }{
{tagDeny, tag + "," + tag, twice},
{tagDeny, tag + ",,", emptyItem},
{
tagLimitPercent, tag,
`"` + spamhausMasked + `" is not a tag, : and a percentage, ` +
"such as vpn:50 or asn=14061:25",
},
{
tagLimitPercent, "50:" + tag,
`"` + spamhausMasked + `" is not a percentage from 1 to 99`,
},
{tagLimitPercent, tag + ":50," + tag + " :25", twice},
{tagLimitPercent, tag + ":50,,", emptyItem},
} {
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv)
want := tc.name + ": " + tc.want
if err == nil || err.Error() != want {
t.Errorf("%s=%s gave the error %v, want %s", tc.name, tc.value, err, want)
}
}
}
func TestTagListURLFetchedAsAnotherListStopsTheStart(t *testing.T) {
t.Parallel()
@@ -2084,9 +2273,12 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
{countryBytesPercent, "cn"}, {countryBytesPercent, "cn:"},
{countryBytesPercent, "cn:25:50"}, {countryBytesPercent, "china:25"},
{tagDeny, "a,,b"}, {tagDeny, "country=nk"}, {tagDeny, "asn=abc"},
{tagDeny, "asn="}, {tagDeny, "torexit,TOREXIT"},
{tagLimitPercent, "vpn"}, {tagLimitPercent, ":50"}, {tagLimitPercent, "vpn:"},
{tagLimitPercent, "vpn:0"}, {tagLimitPercent, "vpn:100"},
{tagDeny, "asn="}, {tagDeny, "torexit,TOREXIT"}, {tagDeny, "torexti"},
{tagDeny, zenTag}, {tagDeny, "abuseipdb"}, {tagDeny, "crowdsec"},
{tagLimitPercent, "torexti:50"}, {tagLimitPercent, "dnsbl=zen.example:50"},
{tagLimitPercent, "abuseipdb:50"}, {tagLimitPercent, "crowdsec:50"},
{tagLimitPercent, "vpn :50"}, {tagLimitPercent, "vpn"}, {tagLimitPercent, ":50"},
{tagLimitPercent, "vpn:"}, {tagLimitPercent, "vpn:0"}, {tagLimitPercent, "vpn:100"},
{tagLimitPercent, "vpn:1.5"}, {tagLimitPercent, "vpn:50,VPN:25"},
{tagLimitPercent, "exploitsignature:50"}, {tagLimitPercent, "asn=abc:50"},
{tagLists, torExitTag}, {tagLists, "=https://example.com/a.txt"},
@@ -2105,6 +2297,25 @@ func TestInvalidValueStopsTheStart(t *testing.T) {
{logLevel, "INFO"}, {logLevel, "warning"}, {logLevel, "trace"}, {logLevel, ""},
{rulesEnabled, "yes"}, {rulesEnabled, "True"},
})
wantExploitTagStopsTheStartWithRulesAndCoreRuleSetOff(t)
}
// wantExploitTagStopsTheStartWithRulesAndCoreRuleSetOff checks that with rule
// files off, and either setting turning off the Core Rule Set, the tag
// exploitsignature in SWW_TAG_DENY stops the start.
func wantExploitTagStopsTheStartWithRulesAndCoreRuleSetOff(t *testing.T) {
t.Helper()
for _, name := range []string{wafMode, wafAnomalyThreshold} {
_, err := config.FromEnvironment(environment{
tagDeny: exploitTag, rulesEnabled: disabled, name: off,
}.lookupEnv)
if err == nil || !strings.HasPrefix(err.Error(), tagDeny+": ") {
t.Errorf("%s=off with rule files off gave the error %v, want one naming %s",
name, err, tagDeny)
}
}
}
func TestInvalidBanOrStateValueStopsTheStart(t *testing.T) {
@@ -2359,7 +2570,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
lookupSource: defaultLookupSource,
lookupDBPath: "",
lookupTimeout: "800ms",
addLookupHeaders: "false",
addLookupHeaders: disabled,
allowedCountries: "",
tagLists: "",
tagDeny: "",
@@ -2406,6 +2617,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
wafExemptPaths: "",
wafBodyLimit: off,
exclusionsFile: "",
apps: "", // logged as null, which decodes as ""
trapPaths: "",
errorBurstThreshold: "30",
logRemoteURL: "",
+2 -1
View File
@@ -8,6 +8,7 @@ import (
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
@@ -40,7 +41,7 @@ func (abuseIPDB) check(ctx context.Context, rq *request, _ time.Time) *verdict {
}
rq.abuseIPDBHit = true
rq.addTag("abuseipdb")
rq.addTag(config.AbuseIPDBTag)
rq.noteHit(bans.ReputationHit{Source: reputation.AbuseIPDBSource, Score: &score},
"scored by AbuseIPDB at or over SWW_ABUSEIPDB_MIN_SCORE")
+6 -4
View File
@@ -37,8 +37,8 @@ const (
// The same, as SWW_TAG_LIMIT_PERCENT names fromDE's tags.
asnTagDEHalf = "asn=64496:50"
asnTagDEQuarter = "asn=64496:25"
countryTagDEHalf = "country=de:50"
countryTagDEQuarter = "country=de:25"
countryTagDEHalf = "country=DE:50"
countryTagDEQuarter = "country=DE:25"
// noCountry is in an AS of its own, AS64500, and in no country.
noCountry = "192.0.2.80"
// fourAMinute is the rate limit these tests set: half of it is 2
@@ -101,7 +101,7 @@ func TestEachBiasedThresholdLowersTheByteLimits(t *testing.T) {
setting, value, from string
}{
{tagLimitPercent, asnTagDEHalf, fromDE},
{tagLimitPercent, "Country=DE:50", fromDE},
{tagLimitPercent, "Country=de:50", fromDE},
{unknownLimitPercent, "50", unplaced},
{asnBytesPercent, "as64496:50", fromDE},
{countryBytesPercent, countryDEHalf, fromDE},
@@ -328,7 +328,9 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) {
// says, with a GeoJS that never answers.
synctest.Test(t, func(t *testing.T) {
server, out, _ := newDiscriminator(t, unansweredGeoJSURL, time.Now,
map[string]string{lookupTimeout: timeout.String(), tc.setting: tc.value})
map[string]string{
lookupTimeout: timeout.String(), tagLists: "vpn=" + vpnList, tc.setting: tc.value,
})
began := time.Now()
serveFromDE(t, server, http.MethodPost, strings.NewReader("ab"))
@@ -10,6 +10,7 @@ import (
"testing"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/discriminator"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -21,6 +22,7 @@ const (
wafDisabledRules = "SWW_WAF_DISABLED_RULES"
wafExemptPaths = "SWW_WAF_EXEMPT_PATHS"
wafBodyLimit = "SWW_WAF_BODY_LIMIT"
apps = "SWW_APPS"
block = "block"
detect = "detect"
)
@@ -282,6 +284,27 @@ func TestDisabledRulesSwitchOffWhatGiteaWouldBeRefused(t *testing.T) {
}
}
func TestGiteasFixesApplyOnlyToAHostnameThatServesGitea(t *testing.T) {
t.Parallel()
// Only gitea's fixes let a file name through in path, a parameter
// gitea sends file names in.
const fileName = "/?path=.gitignore"
s := startDiscriminator(t, "", map[string]string{
wafMode: block, apps: siteA + ":gitea",
})
wantDisposition(t, do(t, s.question(siteA, client, fileName)),
http.StatusOK, discriminator.DispositionAllow)
wantDisposition(t, do(t, s.question(siteB, client, fileName)),
http.StatusForbidden, discriminator.DispositionDeny)
lines := s.out.requestLines(t, 2)
wantLine(t, lines[0], http.StatusOK, requestlog.ActionForward)
wantLine(t, lines[1], http.StatusForbidden, requestlog.ActionWAFBlocked)
}
func TestAttackInAFormBodyIsRefusedOnlyWhileBodiesAreRead(t *testing.T) {
t.Parallel()
+2 -1
View File
@@ -8,6 +8,7 @@ import (
"time"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -26,7 +27,7 @@ func (crowdSec) check(_ context.Context, rq *request, now time.Time) *verdict {
return nil
}
rq.addTag("crowdsec")
rq.addTag(config.CrowdSecTag)
rq.noteHit(bans.ReputationHit{Source: rq.h.config.CrowdSecDecisionsURL},
"listed by the CrowdSec decision list")
rq.banForCrowdSec(now, decision)
+3 -1
View File
@@ -5,8 +5,10 @@ package discriminator
import (
"context"
"strings"
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -27,7 +29,7 @@ func (dnsbl) check(ctx context.Context, rq *request, _ time.Time) *verdict {
rq.noteListed(listedBy, "listed by a DNSBL zone")
for _, zone := range listedBy {
rq.addTag("dnsbl=" + zone)
rq.addTag(config.DNSBLTagPrefix + strings.ToLower(zone))
}
if !rq.dnsblListed || rq.h.config.ReputationAction != deny {
+1 -1
View File
@@ -43,6 +43,7 @@ const (
// country are looked up, so that a client they refuse is not looked up.
func beforeLookup() []heuristic {
return []heuristic{
tagLists{},
denyNets{},
activeBan{},
}
@@ -53,7 +54,6 @@ func beforeLookup() []heuristic {
// last, on the tag rule_files and core_rule_set can add.
func afterLookup() []heuristic {
return []heuristic{
tagLists{},
countries{},
blocklists{},
crowdSec{},
+4 -3
View File
@@ -7,6 +7,7 @@ import (
"strings"
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/lookup"
)
@@ -66,17 +67,17 @@ func asnTag(asn string) string {
return ""
}
return "asn=" + strings.TrimPrefix(asn, "AS")
return config.ASNTagPrefix + strings.TrimPrefix(asn, "AS")
}
// countryTag returns the tag of country, a code as the lookup gives it, such
// as country=de for DE, or "" for no country.
// as country=DE for DE, or "" for no country.
func countryTag(country string) string {
if country == "" {
return ""
}
return "country=" + strings.ToLower(country)
return config.CountryTagPrefix + country
}
// addLookup adds answer, an answer about a client from the lookup
+3 -1
View File
@@ -212,7 +212,8 @@ func newReputation(
// NewCoreRuleSet returns the Core Rule Set at SWW_WAF_PARANOIA_LEVEL,
// without the rules SWW_WAF_DISABLED_RULES switches off, reading bodies
// up to SWW_WAF_BODY_LIMIT, with the exclusions in
// SWW_CORE_RULE_SET_EXCLUSIONS_FILE, or nil while SWW_WAF_MODE is off. It
// SWW_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_APPS,
// or nil while SWW_WAF_MODE is off. It
// reads at most QuestionBodyMaxBytes less one byte: a question's body of
// QuestionBodyMaxBytes may be cut short, and the Core Rule Set does not read a
// JSON or XML body over its limit. An exclusions file that waf.New refuses is
@@ -227,6 +228,7 @@ func NewCoreRuleSet(cfg *config.Config) (*waf.CoreRuleSet, error) {
ParanoiaLevel: cfg.WAFParanoiaLevel, DisabledRules: cfg.WAFDisabledRules,
BodyLimit: min(cfg.WAFBodyLimit, QuestionBodyMaxBytes-1),
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
Apps: cfg.Apps,
})
if err != nil {
return nil, fmt.Errorf("SWW_CORE_RULE_SET_EXCLUSIONS_FILE: %w", err)
+1 -1
View File
@@ -71,7 +71,7 @@ const (
addLookupHeaders = "SWW_ADD_LOOKUP_HEADERS"
tagDeny = "SWW_TAG_DENY"
// denyKP is the value of SWW_TAG_DENY that refuses fromKP's country.
denyKP = "country=kp"
denyKP = "country=KP"
allowedCountries = "SWW_EXCLUSIVELY_ALLOWED_COUNTRIES"
banResponse = "SWW_BAN_RESPONSE"
limitBanDuration = "SWW_LIMIT_BAN_DURATION"
+2 -2
View File
@@ -162,8 +162,8 @@ func TestRateLimitExemptPathsAreNeitherCountedNorRefused(t *testing.T) {
// /assets/, and breaks the limit.
s.request(client, "/assets", http.StatusForbidden, requestlog.ActionRateLimited)
// A ban, SWW_DENY_NETS and the country lists still refuse a path
// under a prefix.
// A ban, SWW_DENY_NETS and SWW_TAG_DENY still refuse a path under a
// prefix.
s.request(client, "/assets/app.js", http.StatusForbidden, requestlog.ActionBanned)
s.request(denied, "/assets/app.js", http.StatusForbidden, requestlog.ActionDenied)
s.request(fromKP, "/assets/app.js",
+3 -3
View File
@@ -168,7 +168,7 @@ func TestReputationRefusalNamesItsHeuristicAndWhatListedTheClient(t *testing.T)
}
}
func TestBlocklistsComeAfterTheCountryListsAndSkipAllowNets(t *testing.T) {
func TestBlocklistsComeAfterAllowedCountriesAndSkipAllowNets(t *testing.T) {
t.Parallel()
s, server, queue := startWithLookups(t, map[string]string{
@@ -499,7 +499,7 @@ func TestReputationSourcesTagTheRequestWhateverTheirAction(t *testing.T) {
func(server *discriminator.Server) {
loadVerdicts(server, map[string][]string{fromDE: {dnsblZone}})
},
[]string{"asn=64496", "country=de", "dnsbl=" + dnsblZone},
[]string{"asn=64496", "country=DE", "dnsbl=" + dnsblZone},
},
{
abuseipdb,
@@ -507,7 +507,7 @@ func TestReputationSourcesTagTheRequestWhateverTheirAction(t *testing.T) {
func(server *discriminator.Server) {
loadScores(server, map[string]int64{fromDE: 80})
},
[]string{abuseipdb, "asn=64496", "country=de"},
[]string{abuseipdb, "asn=64496", "country=DE"},
},
} {
t.Run(tc.name, func(t *testing.T) {
+2 -4
View File
@@ -138,11 +138,9 @@ func (h *handler) startRequest(
return rq
}
// addTag adds tag, in lower case, to the request's tags, which stay sorted and
// hold each tag once.
// addTag adds tag to the request's tags, which stay sorted and hold each tag
// once.
func (rq *request) addTag(tag string) {
tag = strings.ToLower(tag)
at, found := slices.BinarySearch(rq.line.Tags, tag)
if !found {
rq.line.Tags = slices.Insert(rq.line.Tags, at, tag)
+4 -5
View File
@@ -36,9 +36,8 @@ func TestAllowNetsSkipEveryCheck(t *testing.T) {
rateLimitPerMinute: "1",
})
// Neither SWW_DENY_NETS, the country lists nor the limit of one
// request a minute refuses the client, and its country is not looked
// up.
// Neither SWW_DENY_NETS, SWW_TAG_DENY nor the limit of one request a
// minute refuses the client, and its country is not looked up.
wantAnswers(t, addr, out, []sentRequest{
{fromKP, http.StatusOK, requestlog.ActionForward},
{fromKP, http.StatusOK, requestlog.ActionForward},
@@ -147,8 +146,8 @@ func TestRateLimitExemptNetsAreNeitherCountedNorRefused(t *testing.T) {
})
// listedAddr's requests are neither refused nor counted, so the first
// request from unlistedAddr is within the limit of one a minute. The
// country lists still refuse an exempt client.
// request from unlistedAddr is within the limit of one a minute.
// SWW_TAG_DENY still refuses an exempt client.
wantAnswers(t, addr, out, []sentRequest{
{listedAddr, http.StatusOK, requestlog.ActionForward},
{listedAddr, http.StatusOK, requestlog.ActionForward},
+49 -3
View File
@@ -10,11 +10,13 @@ import (
func TestTagDenyRefusesALookupTagAndNamesTheTag(t *testing.T) {
t.Parallel()
s, _, _ := startWithLookups(t, map[string]string{tagDeny: "Country=KP"})
// The setting takes a country code in either case; the request's tag is
// in capitals.
s, _, _ := startWithLookups(t, map[string]string{tagDeny: "Country=kp"})
line := s.get(fromKP, http.StatusForbidden, requestlog.ActionDenied)
wantHeuristic(t, line, "tag_deny", `{"tag": "country=kp"}`)
wantTags(t, line, "asn=64511", "country=kp")
wantHeuristic(t, line, "tag_deny", `{"tag": "country=KP"}`)
wantTags(t, line, "asn=64511", "country=KP")
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
}
@@ -44,6 +46,22 @@ func TestTagDenyRefusesAListTagBeforeTheRateLimitsCount(t *testing.T) {
}
}
func TestTagDenyRefusesAMixedCaseZoneByItsLowerCaseTag(t *testing.T) {
t.Parallel()
const mixedCaseZone = "DNSBL.example"
// The zone's tag is in lower case, however SWW_DNSBL_ZONES writes it.
s, server, _ := startWithLookups(t, map[string]string{
dnsblZones: mixedCaseZone, dnsblResolver: noResolver,
reputationAction: actionLog, tagDeny: "dnsbl=" + dnsblZone,
})
loadVerdicts(server, map[string][]string{fromDE: {mixedCaseZone}})
wantHeuristic(t, s.get(fromDE, http.StatusForbidden, requestlog.ActionDenied),
"tag_deny", `{"tag": "dnsbl=`+dnsblZone+`"}`)
}
func TestTagDenyRefusesAnExploitSignatureOfARuleThatOnlyLogs(t *testing.T) {
t.Parallel()
@@ -73,3 +91,31 @@ func TestTagLimitPercentLowersALimitTheLowestWinning(t *testing.T) {
wantPercent(t, "limit_percent", line.LimitPercent, line.LimitPercentSetting,
"25 from "+tagLimitPercent)
}
func TestTagLimitPercentLowersAByteLimitByAListTag(t *testing.T) {
t.Parallel()
s, server, _ := startWithLookups(t, map[string]string{
bytesLimitPerMinute: twoUploads,
tagLists: "vpn=" + vpnList,
tagLimitPercent: "vpn:50",
})
loadLists(t, server, map[string][]string{vpnList: {fromDE}})
// The upload's 100 bytes are over half of 199, 99.
line := s.uploadFrom(fromDE)
if line.LimitHit != minuteBytes {
t.Errorf("log line has limit_hit %q, want %s", line.LimitHit, minuteBytes)
}
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting,
"50 from "+tagLimitPercent)
// fromKP, which the list does not hold, has the whole limit.
line = s.uploadFrom(fromKP)
if line.LimitHit != "" {
t.Errorf("log line for %s has limit_hit %q, want none", fromKP, line.LimitHit)
}
wantPercent(t, "bytes_percent", line.BytesPercent, line.BytesPercentSetting, none)
}
+30 -3
View File
@@ -43,9 +43,9 @@ func TestLookupAndTagListsGiveTheRequestItsTagsSorted(t *testing.T) {
})
wantTags(t, s.get(fromDE, http.StatusOK, requestlog.ActionForward),
"asn=64496", "country=de", "torexit", "vpn=mullvad")
"asn=64496", "country=DE", "torexit", "vpn=mullvad")
wantTags(t, s.get(fromKP, http.StatusOK, requestlog.ActionForward),
"asn=64511", "country=kp", "torexit")
"asn=64511", "country=KP", "torexit")
}
func TestRequestWithoutATagLeavesTheFieldOut(t *testing.T) {
@@ -69,5 +69,32 @@ func TestTagListsTagEvenARequestTheCountryListRefuses(t *testing.T) {
loadLists(t, server, map[string][]string{vpnList: {fromKP}})
wantTags(t, s.get(fromKP, http.StatusForbidden, requestlog.ActionCountryDenied),
"asn=64511", "country=kp", "vpn")
"asn=64511", "country=KP", "vpn")
}
func TestTagListsTagARequestRefusedBeforeTheLookup(t *testing.T) {
t.Parallel()
s, server, _ := startWithLookups(t, map[string]string{
tagLists: "vpn=" + vpnList, denyNets: fromKP, rateLimitPerMinute: "1",
})
loadLists(t, server, map[string][]string{vpnList: {fromDE, fromKP}})
line := s.get(fromKP, http.StatusForbidden, requestlog.ActionDenied)
if line.Heuristic != "deny_nets" {
t.Errorf("log line has heuristic %q, want deny_nets", line.Heuristic)
}
wantTags(t, line, "vpn")
// The second request breaks the limit of one a minute, which bans the client.
s.get(fromDE, http.StatusOK, requestlog.ActionForward)
s.get(fromDE, http.StatusForbidden, requestlog.ActionRateLimited)
line = s.get(fromDE, http.StatusForbidden, requestlog.ActionBanned)
if line.Heuristic != "active_ban" {
t.Errorf("log line has heuristic %q, want active_ban", line.Heuristic)
}
wantTags(t, line, "vpn")
}
+14 -4
View File
@@ -133,8 +133,11 @@ func (f *File) Watch(ctx context.Context) {
}
if err != nil {
f.params.ProcessLog.Error("cannot watch the lookup database for replacements",
"error", err.Error())
f.params.ProcessLog.Error(
"cannot watch the lookup database for replacements",
"error",
err.Error(),
)
return
}
@@ -194,7 +197,10 @@ func (f *File) readAgain() {
f.params.Alerts.Raise(alerts.Alert{
Event: alerts.EventFileError,
Reason: kept,
Detail: map[string]any{"file": f.params.Path, "error": err.Error()},
Detail: map[string]any{
"file": f.params.Path,
"error": err.Error(),
},
})
f.params.ProcessLog.Error(kept, "error", err.Error())
@@ -212,7 +218,11 @@ func (f *File) use(reader *maxminddb.Reader) {
f.lastRead = f.params.Now()
f.mu.Unlock()
f.params.ProcessLog.Info("read the lookup database", "file", f.params.Path)
f.params.ProcessLog.Info(
"read the lookup database",
"file",
f.params.Path,
)
}
// read reads the lookup database at path. The whole file is read into
+1 -1
View File
@@ -88,7 +88,7 @@ func (b *busiest) Collect(ch chan<- prometheus.Metric) {
}
// vecs returns the metrics: by AS number, those of requests and bytes; by
// country, the refusals by the country lists as well.
// country, the refusals by the country list as well.
func (b *busiest) vecs() []*prometheus.CounterVec {
vecs := []*prometheus.CounterVec{b.requests, b.requestBytes, b.responseBytes}
if b.refused != nil {
+11 -11
View File
@@ -115,11 +115,11 @@ type list struct {
failures int
}
// entries are what the lines of a copy say: for a blocklist, the netblocks
// it names, with the lengths among them, for the file of AS:percent lines,
// the percentage it gives each AS number, and for the CrowdSec decision
// list, the decision on each netblock that ends last, with the lengths
// among them.
// entries are what the lines of a copy say: for a blocklist or a tag list, the
// netblocks it names, with the lengths among them, for the file of AS:percent
// lines, the percentage it gives each AS number, and for the CrowdSec decision
// list, the decision on each netblock that ends last, with the lengths among
// them.
type entries struct {
netblocks map[netip.Prefix]bool
lengths []int
@@ -504,10 +504,10 @@ func (l *Lists) get(ctx context.Context, listURL string) ([]string, error) {
}
// parse reads the lines of the list at listURL, fetched at fetched: those
// of a blocklist, of the file of AS:percent lines, or of the CrowdSec
// decision list. In the first two, anything after a ; or a # on a line is
// left out, and so is a line left blank. Any other line that does not read
// is an error naming it by its number.
// of a blocklist or a tag list, of the file of AS:percent lines, or of the
// CrowdSec decision list. In all but the decision list, anything after a ; or
// a # on a line is left out, and so is a line left blank. Any other line that
// does not read is an error naming it by its number.
func (l *Lists) parse(
listURL string, lines []string, fetched time.Time,
) (entries, error) {
@@ -521,8 +521,8 @@ func (l *Lists) parse(
}
}
// parseNetblocks reads a blocklist's lines, each an address or a netblock
// as the settings take them.
// parseNetblocks reads the lines of a blocklist or a tag list, each an address
// or a netblock as the settings take them.
func parseNetblocks(lines []string) (entries, error) {
found := entries{netblocks: map[netip.Prefix]bool{}}
+23 -4
View File
@@ -19,27 +19,34 @@ const (
// ActionForward is a request discriminator allowed, which rproxy passes to
// the app.
ActionForward = "forward"
// ActionTooLarge is a request over the header limits.
ActionTooLarge = "too_large"
// ActionRateLimited is a request refused because it took its client
// over a rate limit, which bans the client.
ActionRateLimited = "rate_limited"
// ActionBanned is a request refused because a ban covers its client,
// or because it matched a ban rule, asked for a trap path or the
// CrowdSec decision list lists its client, each of which bans the
// client.
ActionBanned = "banned"
// ActionRuleBlocked is a request refused because it matched a block
// rule.
ActionRuleBlocked = "rule_blocked"
// ActionWAFBlocked is a request refused because the Core Rule Set
// scored it at or over SWW_WAF_ANOMALY_THRESHOLD.
ActionWAFBlocked = "waf_blocked"
// ActionDenied is a request refused because its client is in
// SWW_DENY_NETS, in a blocklist while SWW_BLOCKLIST_ACTION is deny,
// or listed by a DNSBL zone, or scored a hit by AbuseIPDB, while
// SWW_REPUTATION_ACTION is deny, or that carries a tag in SWW_TAG_DENY.
ActionDenied = "denied"
// ActionCountryDenied is a request refused because its client's country
// is not in SWW_EXCLUSIVELY_ALLOWED_COUNTRIES.
ActionCountryDenied = "country_denied"
@@ -258,7 +265,11 @@ func FormatTime(t time.Time) string {
// Milliseconds is d in milliseconds, to the microsecond.
func Milliseconds(d time.Duration) float64 {
return float64(d.Microseconds()) / float64(time.Millisecond/time.Microsecond)
return float64(
d.Microseconds(),
) / float64(
time.Millisecond/time.Microsecond,
)
}
// NewProcessLogger returns the logger for the process's own messages:
@@ -266,17 +277,25 @@ func Milliseconds(d time.Duration) float64 {
// as a request line's, and instanceName, SWW_INSTANCE_NAME, as instance.
// It writes only the messages at level, SWW_LOG_LEVEL, or more severe;
// the request lines Write writes are never held back.
func NewProcessLogger(w io.Writer, instanceName string, level slog.Level) *slog.Logger {
func NewProcessLogger(
w io.Writer,
instanceName string,
level slog.Level,
) *slog.Logger {
handler := slog.NewJSONHandler(w, &slog.HandlerOptions{
Level: level,
ReplaceAttr: func(groups []string, attr slog.Attr) slog.Attr {
if attr.Key == slog.TimeKey && len(groups) == 0 {
return slog.String(slog.TimeKey, FormatTime(attr.Value.Time()))
return slog.String(
slog.TimeKey,
FormatTime(attr.Value.Time()),
)
}
return attr
},
})
return slog.New(handler).With("type", "process", "instance", instanceName)
return slog.New(handler).
With("type", "process", "instance", instanceName)
}
+1 -1
View File
@@ -618,7 +618,7 @@ func TestLookupDatabaseReplacedWhileRunningTakesEffect(t *testing.T) {
instanceName: instance,
lookupSource: "file",
lookupDBPath: path,
"SWW_TAG_DENY": "country=kp",
"SWW_TAG_DENY": "country=KP",
// The requests sent until a replacement takes effect, and those
// for the metrics, must not break a rate limit, whose ban would
// refuse them too.
+52 -7
View File
@@ -1,8 +1,51 @@
# The eighth change: on gitea's paths, some fields skip the rules that refused
# ordinary gitea requests there. They are the fields where gitea sends a search,
# a file name or a branch name. Each exclusion runs with the request, before the
# rules it switches off, those of phase 1 included, so this file is loaded before
# the Core Rule Set's rules. Each comment names a request the rule refused.
# gitea's fixes: some fields skip the rules that refused ordinary gitea requests.
# They are the fields where gitea sends a search, a file name or a branch name.
# Each exclusion runs with the request, before the rules it switches off, those
# of phase 1 included, so this file is loaded before the Core Rule Set's rules.
# Each comment names a request the rule refused.
# A request whose hostname SWW_APPS does not list gitea for skips every rule
# below. Inspect sets tx.app_gitea on the others.
SecRule &TX:app_gitea "@eq 0" \
"id:900590,phase:1,pass,nolog,skipAfter:END_GITEA"
# The query parameters in which gitea sends names within a repository or its
# own records, or a page of its own site, are not checked against the lists of
# system files, shell paths and command names, on any path:
# 930120: .gitignore
# 932160: bin/docker-entrypoint
# 932260: docker-build
SecAction "id:900591,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=930120;ARGS:path,\
ctl:ruleRemoveTargetById=930120;ARGS:files,\
ctl:ruleRemoveTargetById=930120;ARGS:skip-to,\
ctl:ruleRemoveTargetById=930120;ARGS:sub_path,\
ctl:ruleRemoveTargetById=930120;ARGS:ref,\
ctl:ruleRemoveTargetById=930120;ARGS:sha,\
ctl:ruleRemoveTargetById=930120;ARGS:branch,\
ctl:ruleRemoveTargetById=930120;ARGS:workflow,\
ctl:ruleRemoveTargetById=930120;ARGS:artifactName,\
ctl:ruleRemoveTargetById=930120;ARGS:redirect_to,\
ctl:ruleRemoveTargetById=932160;ARGS:path,\
ctl:ruleRemoveTargetById=932160;ARGS:files,\
ctl:ruleRemoveTargetById=932160;ARGS:skip-to,\
ctl:ruleRemoveTargetById=932160;ARGS:sub_path,\
ctl:ruleRemoveTargetById=932160;ARGS:ref,\
ctl:ruleRemoveTargetById=932160;ARGS:sha,\
ctl:ruleRemoveTargetById=932160;ARGS:branch,\
ctl:ruleRemoveTargetById=932160;ARGS:workflow,\
ctl:ruleRemoveTargetById=932160;ARGS:artifactName,\
ctl:ruleRemoveTargetById=932160;ARGS:redirect_to,\
ctl:ruleRemoveTargetById=932260;ARGS:path,\
ctl:ruleRemoveTargetById=932260;ARGS:files,\
ctl:ruleRemoveTargetById=932260;ARGS:skip-to,\
ctl:ruleRemoveTargetById=932260;ARGS:sub_path,\
ctl:ruleRemoveTargetById=932260;ARGS:ref,\
ctl:ruleRemoveTargetById=932260;ARGS:sha,\
ctl:ruleRemoveTargetById=932260;ARGS:branch,\
ctl:ruleRemoveTargetById=932260;ARGS:workflow,\
ctl:ruleRemoveTargetById=932260;ARGS:artifactName,\
ctl:ruleRemoveTargetById=932260;ARGS:redirect_to"
# gitea's searches of code, of issues and pull requests, and of commits send the
# text searched for in q, which skips these rules there:
@@ -155,8 +198,8 @@ SecRule REQUEST_FILENAME \
# 942140: information_schema.sql
# 944110: java/lang/Runtime.java
# 944130: java.lang.String.md
# Each request skips the rules that refused it. The fourth change already has
# path and redirect_to skip 930120, 932160 and 932260 on every path.
# Each request skips the rules that refused it. Rule 900591 already has path and
# redirect_to skip 930120, 932160 and 932260 on every path.
# A file's commit graph, in file.
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/graph$" \
@@ -281,3 +324,5 @@ SecRule REQUEST_FILENAME \
"@rx ^/api/actions_pipeline/_apis/pipelines/workflows/[0-9]+/artifacts/[^/]+/upload$" \
"id:900640,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=920450;REQUEST_HEADERS_NAMES:Content-Range"
SecMarker END_GITEA
+101 -57
View File
@@ -1,9 +1,9 @@
// Package waf runs the OWASP Core Rule Set 4.25.0, through Coraza, on the
// method, the URL with its query and the headers of a request, and on its
// body while SWW_WAF_BODY_LIMIT is set, with the eight changes sww
// makes to it and the exclusions in SWW_CORE_RULE_SET_EXCLUSIONS_FILE, as
// "Attack detection" under "Settings" in SPEC.md describes them. It reads no
// response.
// body while SWW_WAF_BODY_LIMIT is set, with the six changes sww
// makes to it, the fixes for the apps SWW_APPS names, and the exclusions in
// SWW_CORE_RULE_SET_EXCLUSIONS_FILE, as "Attack detection" under "Settings"
// in SPEC.md describes them. It reads no response.
//
// sww writes only to its state directory, so Coraza is built with
// its no_fs_access tag, as the Dockerfile and script/build build it: of a
@@ -16,6 +16,8 @@ import (
"errors"
"fmt"
"io"
"maps"
"net"
"net/http"
"net/netip"
"os"
@@ -32,8 +34,8 @@ import (
// directives are the Core Rule Set as sww runs it, with the
// paranoia level for %d, bodyDirectives for the first %s while
// SWW_WAF_BODY_LIMIT is set, the exclusions file for the second, and
// giteaExclusions for the third.
// SWW_WAF_BODY_LIMIT is set, the exclusions file for the second, and the
// directives of every app in apps for the third.
// Each rule sww adds has an id from 900000 to 900999, the ids the
// Core Rule Set keeps for the rules that set it up, which
// SWW_WAF_DISABLED_RULES and the exclusions file refuse, so that no
@@ -81,44 +83,28 @@ SecRule ARGUMENTS_LIMIT_REACHED "@eq 1" "id:900300,phase:2,pass,\
# before the rules of phase 1 too.
%s
# The eighth change comes before the rules for the same reason.
# The apps' fixes come before the rules for the same reason.
%s
# The sixth: only the rules for requests are loaded, and no response is
# The fifth: only the rules for requests are loaded, and no response is
# inspected.
Include @owasp_crs/REQUEST-*.conf
# The third: redirect_uri is not checked for a URL naming an IP address or
# localhost. Coraza matches a parameter name here, and in the fourth,
# without regard to case. ARGS holds the fields of a form data or multipart
# body Coraza reads as well as the query parameters, so a field of one of
# these names is left out too.
# localhost. Coraza matches a parameter name here without regard to case.
# ARGS holds the fields of a form data or multipart body Coraza reads as
# well as the query parameters, so a field of this name is left out too.
SecRuleUpdateTargetById 931100 "!ARGS:redirect_uri"
SecRuleUpdateTargetById 934110 "!ARGS:redirect_uri"
# The fourth: the query parameters in which gitea sends names within a
# repository or its own records, or a page of its own site, are not
# checked against the lists of system files, shell paths and command
# names. Coraza takes one rule id per directive.
SecRuleUpdateTargetById 930120 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
!ARGS:sub_path|!ARGS:ref|!ARGS:sha|!ARGS:branch|!ARGS:workflow|\
!ARGS:artifactName|!ARGS:redirect_to"
SecRuleUpdateTargetById 932160 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
!ARGS:sub_path|!ARGS:ref|!ARGS:sha|!ARGS:branch|!ARGS:workflow|\
!ARGS:artifactName|!ARGS:redirect_to"
SecRuleUpdateTargetById 932260 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
!ARGS:sub_path|!ARGS:ref|!ARGS:sha|!ARGS:branch|!ARGS:workflow|\
!ARGS:artifactName|!ARGS:redirect_to"
# The fifth, for Referer: it is not checked for a Unix command without
# arguments, for Java starting a process, or for a Java class name. 944130
# refused every page opened from the page of java.lang.String.md. The
# cookies are left out in Inspect.
# The fourth: Referer is not checked for a Unix command without arguments,
# for Java starting a process, or for a Java class name. 944130 refused
# every page opened from the page of java.lang.String.md.
SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer"
SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer"
SecRuleUpdateTargetById 944130 "!REQUEST_HEADERS:Referer"
# The seventh: a container image push that reuses a layer names the image
# The sixth: a container image push that reuses a layer names the image
# it takes the layer from in the query parameter from, which is not
# checked for command names. 932260 refused the push of a layer from
# owner/python3-app.
@@ -168,18 +154,36 @@ SecRule MULTIPART_STRICT_ERROR "!@eq 0" "id:900450,phase:2,pass,\
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
`
// giteaExclusions are the eighth change: the fields of gitea's own
// requests that skip the rules that refused them there, as Coraza
// directives.
//
//go:embed gitea.conf
var giteaExclusions string
// app is an app whose ordinary requests the Core Rule Set would refuse,
// with the fixes that let them pass.
type app struct {
// directives are Coraza exclusions, loaded before the Core Rule Set's
// rules. Their first rule skips the rest unless Inspect sets the
// transaction variable app_ and the app's name, and they end with the
// marker it skips to.
directives string
// cookies are the cookies the Core Rule Set reads a request without.
cookies []string
}
// cookiesNotRead are the cookies the Core Rule Set reads a request
// without, the rest of the fifth change.
//go:embed gitea.conf
var giteaDirectives string
// apps are the apps whose fixes sww has, by the name SWW_APPS gives each.
//
//nolint:gochecknoglobals // a constant cannot be a list
var cookiesNotRead = []string{"gitea_flash", "redirect_to"}
//nolint:gochecknoglobals // a constant cannot be a map
var apps = map[string]app{
"gitea": {
directives: giteaDirectives,
cookies: []string{"gitea_flash", "redirect_to"},
},
}
// AppNames returns the names of the apps whose fixes sww has, in
// alphabetical order.
func AppNames() []string {
return slices.Sorted(maps.Keys(apps))
}
// Params are what New needs.
type Params struct {
@@ -195,6 +199,10 @@ type Params struct {
// Core Rule Set's rules (SWW_CORE_RULE_SET_EXCLUSIONS_FILE), "" while
// there is none.
ExclusionsFile string
// Apps are the names of the apps each hostname serves, by hostname in
// lowercase (SWW_APPS). A request gets the fixes of its hostname's
// apps only, or of every app while Apps is nil.
Apps map[string][]string
}
// The ids of sww's own rules, which an exclusions file may not
@@ -286,13 +294,15 @@ func checkExclusions(exclusions string) error {
type CoreRuleSet struct {
waf coraza.WAF
bodyLimit int64
apps map[string][]string
}
// New returns the Core Rule Set with the eight changes, at params'
// paranoia level, without the rules it switches off, reading request
// bodies up to params' limit, and with params' exclusions file loaded
// before the Core Rule Set's rules. An exclusions file is an error if it
// cannot be read or does not parse, or if checkExclusions refuses it.
// New returns the Core Rule Set with the six changes and the apps' fixes,
// at params' paranoia level, without the rules it switches off, reading
// request bodies up to params' limit, and with params' exclusions file
// loaded before the Core Rule Set's rules. An exclusions file is an error
// if it cannot be read or does not parse, or if checkExclusions refuses
// it.
func New(params Params) (*CoreRuleSet, error) {
body := ""
if params.BodyLimit > 0 {
@@ -315,8 +325,13 @@ func New(params Params) (*CoreRuleSet, error) {
}
}
var fixes strings.Builder
for _, name := range AppNames() {
fixes.WriteString(apps[name].directives + "\n")
}
text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions,
giteaExclusions)
fixes.String())
if len(params.DisabledRules) > 0 {
ids := make([]string, len(params.DisabledRules))
@@ -334,7 +349,7 @@ func New(params Params) (*CoreRuleSet, error) {
return nil, fmt.Errorf("load the Core Rule Set: %w", err)
}
return &CoreRuleSet{waf: waf, bodyLimit: params.BodyLimit}, nil
return &CoreRuleSet{waf: waf, bodyLimit: params.BodyLimit, apps: params.Apps}, nil
}
// Result is what the Core Rule Set found in a request.
@@ -346,11 +361,12 @@ type Result struct {
Score int
}
// Inspect runs the Core Rule Set on r, a request from client: on its
// method, its URL with the query, and its headers, the Cookie header
// without the cookies in cookiesNotRead, and on body, r's body as the
// caller has it, as readBody reads it. It returns what it found, what it
// read of body, and the error that ended the reading early, if one did.
// Inspect runs the Core Rule Set on r, a request from client, with the
// fixes of the apps appsFor names for r's hostname: on its method, its URL
// with the query, and its headers, the Cookie header without those apps'
// cookies, and on body, r's body as the caller has it, as readBody reads
// it. It returns what it found, what it read of body, and the error that
// ended the reading early, if one did.
func (c *CoreRuleSet) Inspect(
r *http.Request, client netip.Addr, body io.Reader,
) (Result, []byte, error) {
@@ -358,13 +374,24 @@ func (c *CoreRuleSet) Inspect(
// Closing would remove the files Coraza wrote, and it writes none.
defer func() { _ = tx.Close() }()
// The transaction's variables are set only through Coraza's interface
// for plugins.
state := tx.(plugintypes.TransactionState) //nolint:forcetypeassert // every one is
var cookiesNotRead []string
for _, name := range c.appsFor(r.Host) {
state.Variables().TX().Set("app_"+name, []string{"1"})
cookiesNotRead = append(cookiesNotRead, apps[name].cookies...)
}
tx.ProcessConnection(client.String(), 0, "", 0)
tx.ProcessURI(r.URL.String(), r.Method, r.Proto)
for name, values := range r.Header {
for _, value := range values {
if name == "Cookie" {
value = withoutCookiesNotRead(value)
value = withoutCookies(value, cookiesNotRead)
if value == "" {
continue // it held those cookies alone
}
@@ -457,9 +484,26 @@ func score(tx types.Transaction) int {
return n
}
// withoutCookiesNotRead returns value, a Cookie header's, without the
// cookies in cookiesNotRead.
func withoutCookiesNotRead(value string) string {
// appsFor returns the names of the apps whose fixes a request for host,
// its Host header, gets: every app while SWW_APPS is unset, and otherwise
// those SWW_APPS lists for host's name, without the port and in
// lowercase.
func (c *CoreRuleSet) appsFor(host string) []string {
if c.apps == nil {
return AppNames()
}
hostname, _, err := net.SplitHostPort(host)
if err != nil {
hostname = host // it has no port
}
return c.apps[strings.ToLower(hostname)]
}
// withoutCookies returns value, a Cookie header's, without the cookies
// named in cookiesNotRead.
func withoutCookies(value string, cookiesNotRead []string) string {
var kept []string
for cookie := range strings.SplitSeq(value, ";") {
+92 -2
View File
@@ -120,8 +120,8 @@ func atDefaults(t *testing.T) *waf.CoreRuleSet {
}
// wantChange checks that crs lets through passes, a gitea request one of
// the eight changes is for, and still finds result in refused, a request
// like it that the change is not for.
// the six changes or gitea's fixes are for, and still finds result in
// refused, a request like it that they are not for.
func wantChange(
t *testing.T, crs *waf.CoreRuleSet, passes, refused request, result waf.Result,
) {
@@ -381,6 +381,96 @@ func TestImagePushThatReusesALayerIsNotCheckedForCommandNames(t *testing.T) {
matched(930120, 932160))
}
// giteaFixed are gitea requests that only gitea's fixes let through, one
// for each kind of fix, each with what the Core Rule Set finds in it
// without them.
//
//nolint:gochecknoglobals // a constant cannot be a list
var giteaFixed = []struct {
request request
result waf.Result
}{
// A search of code, on the path of gitea's search.
{get("/explore/code?q=SELECT%20*%20FROM%20users%20WHERE"), matched(942100)},
// A file name, in a parameter gitea sends names in, on any path.
{get("/?path=.gitignore"), matched(930120)},
// A file name, in a cookie of gitea's.
{
get("/owner/repo", "Cookie: gitea_flash=success%3DFile%2Bpackage.json%2Bdeleted"),
matched(930120),
},
}
// The hostnames of the tests of SWW_APPS: the one that serves gitea, and
// one that serves another app.
const (
giteaHost = "git.example"
otherHost = "blog.example"
)
// withHost returns r sent to host.
func withHost(r request, host string) request {
return request{r.method, r.target, append([]string{"Host: " + host}, r.headers...)}
}
// servingApps returns the Core Rule Set at the defaults, with SWW_APPS
// giving apps.
func servingApps(t *testing.T, apps map[string][]string) *waf.CoreRuleSet {
t.Helper()
crs, err := waf.New(waf.Params{
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, Apps: apps,
})
if err != nil {
t.Fatalf("load the Core Rule Set: %v", err)
}
return crs
}
func TestAppsUnsetGivesEveryHostnameEveryAppsFixes(t *testing.T) {
t.Parallel()
crs := servingApps(t, nil)
for _, fixed := range giteaFixed {
for _, host := range []string{giteaHost, otherHost} {
wantResult(t, crs, withHost(fixed.request, host), waf.Result{})
}
}
}
func TestListedHostnameGetsItsAppsFixes(t *testing.T) {
t.Parallel()
crs := servingApps(t, map[string][]string{giteaHost: {"gitea"}})
// The hostname is matched without the port and without regard to case.
for _, host := range []string{giteaHost, "GIT.Example:8443"} {
for _, fixed := range giteaFixed {
wantResult(t, crs, withHost(fixed.request, host), waf.Result{})
}
}
}
func TestUnlistedHostnameGetsTheFullCheck(t *testing.T) {
t.Parallel()
for _, apps := range []map[string][]string{
{giteaHost: {"gitea"}},
// SWW_APPS set empty lists no hostname.
{},
} {
crs := servingApps(t, apps)
for _, host := range []string{otherHost, otherHost + ":8443", "git.example.org"} {
for _, fixed := range giteaFixed {
wantResult(t, crs, withHost(fixed.request, host), fixed.result)
}
}
}
}
// withExclusions returns the Core Rule Set at the defaults, with the
// exclusions file at path.
func withExclusions(path string) (*waf.CoreRuleSet, error) {
Executable
+54
View File
@@ -0,0 +1,54 @@
#!/bin/sh
# script/publish: build the image on the Docker host and push it to the
# registry under four tags: the UTC date (YYYY-MM-DD), the short commit
# hash, the branch name and latest. The build is script/docker's, so the
# tests and the linter run first. The host and the registry name are
# environment variables with defaults, for running it against another.
#
# The working tree must be clean: the hash tag names a commit, and an
# image built from uncommitted changes would carry a hash that does not
# describe it.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
: "${DOCKER_HOST:=ssh://root@fsn1app1}"
: "${PUBLISH_IMAGE:=git.eeqj.de/sneak/smallwebwaf}"
export DOCKER_HOST
main() {
cd "$ROOT"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`.
changes="$(git status --porcelain)"
if [ -n "$changes" ]; then
echo "script/publish: the working tree is not clean; commit or stash first" >&2
exit 1
fi
date="$(date -u +%Y-%m-%d)"
hash="$(git rev-parse --short HEAD)"
branch="$(git rev-parse --abbrev-ref HEAD)"
if [ "$branch" = "HEAD" ]; then
echo "script/publish: detached HEAD; check out a branch first" >&2
exit 1
fi
# A tag is [A-Za-z0-9_.-]; a branch name such as feature/x is not.
branch="$(printf '%s' "$branch" | tr -c 'A-Za-z0-9_.-' '-')"
local_tag="$("$SCRIPT_DIR/projectname")"
"$SCRIPT_DIR/docker"
for tag in "$date" "$hash" "$branch" latest; do
docker tag "$local_tag" "$PUBLISH_IMAGE:$tag"
done
for tag in "$date" "$hash" "$branch" latest; do
docker push "$PUBLISH_IMAGE:$tag"
done
echo "pushed $PUBLISH_IMAGE with tags: $date $hash $branch latest"
}
main "$@"