Author SHA1 Message Date
sneak 3fa1f4b732 Read every setting through viper (closes #211)
check / check (push) Successful in 3m47s
servewaf's and rproxy's settings are read through viper, bound to the same environment variable names as before, in place of the os.LookupEnv function Main passed down. NewViper turns AutomaticEnv and AllowEmptyEnv on, so a setting that is set but empty reads as set, as it did. A setting may also be given as a file, the _FILE form, which rproxy's settings now have too; rproxy still writes nothing to disk. The settings tests pass set into a viper instead of a lookup function.

Model: deepseek-v4-pro
2026-11-07 05:29:14 +00:00
clawbot 2e2f6f7175 Core Rule Set: gitea's ordinary requests pass by default (#208)
check / check (push) Successful in 3m29s
2026-11-07 05:21:22 +01:00
24 changed files with 1521 additions and 231 deletions

No files matched your search

+13 -9
View File
@@ -189,12 +189,12 @@ restarting the container. `rproxy` reads only the `SWRP_` settings, and
default, chosen for a service on the open internet. A setting that needs an default, chosen for a service on the open internet. A setting that needs an
account, a key, a token or an alert destination stays off until you give one. account, a key, a token or an alert destination stays off until you give one.
- Any `SWWAF_` setting can be given as a file instead, for secrets and long - Any `SWWAF_` or `SWRP_` setting can be given as a file instead, for secrets
lists: `SWWAF_ADMIN_TOKEN_FILE` names the file that holds `SWWAF_ADMIN_TOKEN`. and long lists: `SWWAF_ADMIN_TOKEN_FILE` names the file that holds
The file's contents are the value, less one newline at the end. Setting both `SWWAF_ADMIN_TOKEN`. The file's contents are the value, less one newline at
forms stops the start. `SWWAF_LOG_REMOTE_TLS_CA_FILE` already names a file, so the end. Setting both forms stops the start. `SWWAF_LOG_REMOTE_TLS_CA_FILE`
it has no `_FILE` form, and no `SWRP_` setting has one, since `rproxy` never and `SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE` already name a file, so they have no
touches disk. `_FILE` form.
- A duration is Go's syntax, with `d` for days: `90s`, `15m`, `24h`, `7d`. - A duration is Go's syntax, with `d` for days: `90s`, `15m`, `24h`, `7d`.
- A size is a number of bytes with an optional `K`, `M` or `G`, in powers of - A size is a number of bytes with an optional `K`, `M` or `G`, in powers of
1024: `512K`, `100M`, `5G`. 1024: `512K`, `100M`, `5G`.
@@ -272,6 +272,7 @@ Every setting the code reads, with its default:
| `SWWAF_WAF_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. | | `SWWAF_WAF_DISABLED_RULES` | `920340,920420,920440,920640,930130,930140` | Core Rule Set rule ids switched off; a list you give replaces the default, so repeat these six to keep them off. |
| `SWWAF_WAF_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. | | `SWWAF_WAF_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. |
| `SWWAF_WAF_BODY_LIMIT` | `off` | How much of a request body the Core Rule Set reads, a size up to `64K`; `off` reads none. | | `SWWAF_WAF_BODY_LIMIT` | `off` | How much of a request body the Core Rule Set reads, a size up to `64K`; `off` reads none. |
| `SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. |
| `SWWAF_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. | | `SWWAF_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. |
| `SWWAF_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. | | `SWWAF_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. |
| `SWWAF_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:<percent>` of every limit, or `log`. | | `SWWAF_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:<percent>` of every limit, or `log`. |
@@ -553,10 +554,13 @@ and the status for a size or time limit, such as `413` for a body too large.
headers of every request, and no body unless `SWWAF_WAF_BODY_LIMIT` is set. In headers of every request, and no body unless `SWWAF_WAF_BODY_LIMIT` is set. In
front of a code forge, bodies carry what people write and publish, which the front of a code forge, bodies carry what people write and publish, which the
rules take for attacks. rules take for attacks.
- `smallwebwaf` changes the Core Rule Set in six ways so that it lets ordinary - `smallwebwaf` changes the Core Rule Set in eight ways so that it lets ordinary
requests to a code forge through, such as git over HTTP and sign-in through requests to a code forge through, such as git over HTTP and sign-in through
Git Credential Manager. No setting undoes them; "Attack detection" in Git Credential Manager. On gitea's own paths, its searches, file names and
[`SPEC.md`](SPEC.md) lists them. branch names get through too. No setting undoes the changes; "Attack
detection" in [`SPEC.md`](SPEC.md) lists them.
- `SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE` switches single rules off for an app's
own fields or paths.
- Rule files are plain text, one rule to a line, edited by hand and read again - Rule files are plain text, one rule to a line, edited by hand and read again
while `servewaf` runs. The image ships one file of probes no real visitor while `servewaf` runs. The image ships one file of probes no real visitor
sends (see [`docs/rules.md`](docs/rules.md)). sends (see [`docs/rules.md`](docs/rules.md)).
+116 -25
View File
@@ -31,12 +31,13 @@ to them.
reputation sources, and send alerts. reputation sources, and send alerts.
- Everything is set by environment variables. The names of `servewaf`'s start - Everything is set by environment variables. The names of `servewaf`'s start
with `SWWAF_`, and those of `rproxy` with `SWRP_`. The only other inputs are with `SWWAF_`, and those of `rproxy` with `SWRP_`. The only other inputs are
`servewaf`'s files: files:
- the rule files; - `servewaf`'s rule files;
- the state files; - `servewaf`'s state files;
- a lookup database; - `servewaf`'s lookup database;
- a `SWWAF_` setting given as a file; - a `SWWAF_` or `SWRP_` setting given as a file;
- the file `SWWAF_LOG_REMOTE_TLS_CA_FILE` names. - the file `SWWAF_LOG_REMOTE_TLS_CA_FILE` names;
- the file `SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE` names.
## Non-goals ## Non-goals
@@ -81,7 +82,7 @@ to them.
of bans is full. of bans is full.
- `internal/rules`: reads the rule files at start and again when they - `internal/rules`: reads the rule files at start and again when they
change, and tells which rules a request matches. change, and tells which rules a request matches.
- `internal/waf`: the Core Rule Set, with the six changes listed under - `internal/waf`: the Core Rule Set, with the eight changes listed under
"Attack detection", as Coraza directives. "Attack detection", as Coraza directives.
- `internal/lookup`: looks clients up through GeoJS or in the lookup - `internal/lookup`: looks clients up through GeoJS or in the lookup
database. database.
@@ -321,13 +322,13 @@ Conventions:
No other setting can be switched off with `off`. No other setting can be switched off with `off`.
- A list set to an empty value is an empty list, and replaces the default. - A list set to an empty value is an empty list, and replaces the default.
- Any `SWWAF_` setting can be read from a file instead, for secrets and long - Any `SWWAF_` or `SWRP_` setting can be read from a file instead, for secrets
lists: `SWWAF_ADMIN_TOKEN_FILE` names the file that holds `SWWAF_ADMIN_TOKEN`. and long lists: `SWWAF_ADMIN_TOKEN_FILE` names the file that holds
The file's contents are the value, less one newline at the end, and are `SWWAF_ADMIN_TOKEN`. The file's contents are the value, less one newline at
checked as the value itself would be. Setting both forms stops the start, and the end, and are checked as the value itself would be. Setting both forms
so does naming a file that cannot be read. `SWWAF_LOG_REMOTE_TLS_CA_FILE` stops the start, and so does naming a file that cannot be read.
already holds a file name, so it has no `_FILE` form. No `SWRP_` setting has `SWWAF_LOG_REMOTE_TLS_CA_FILE` and `SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE`
one, since `rproxy` never touches disk. already hold a file name, so they have no `_FILE` form.
- Settings, files included, are read once at start. Changing one means - Settings, files included, are read once at start. Changing one means
restarting the container. The files `servewaf` watches while it runs are its restarting the container. The files `servewaf` watches while it runs are its
state files, its rule files and the lookup database. state files, its rule files and the lookup database.
@@ -582,6 +583,27 @@ files, the limits and the bans still apply to the client that sends it.
probes these rules caught (see "Rule files"). A list you give replaces the probes these rules caught (see "Rule files"). A list you give replaces the
default, so name these six ids in it to keep them off. default, so name these six ids in it to keep them off.
- `SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE` names a file of exclusions, written as
Coraza directives. An exclusion switches one rule off for some fields or paths
only, where `SWWAF_WAF_DISABLED_RULES` switches it off everywhere. The file
holds an app's own exclusions; the eighth change below holds gitea's.
- `servewaf` loads the file after `smallwebwaf`'s own setup rules and before
the Core Rule Set's rules. The Core Rule Set loads its own
`REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf` at the same point.
- So the file holds only `SecAction` and `SecRule` lines that exclude with
`ctl:ruleRemoveTargetById` or `ctl:ruleRemoveById`. These run with each
request, before the rules they exclude, those of phase 1 included.
- A rule that `SWWAF_WAF_DISABLED_RULES` switches off and the file names is
simply off.
- The start stops on a file that cannot be read or does not parse, whatever
`SWWAF_WAF_MODE` is.
- It stops too on a file that names a number from 900000 to 900999, the ids
of `smallwebwaf`'s own rules, alone or in a range. Comment lines are not
checked.
- It stops too on a directive that removes or changes rules as the file
loads, such as `SecRuleRemoveById`, since none of the Core Rule Set's
rules is loaded yet.
- `SWWAF_WAF_BODY_LIMIT` switches body inspection on, up to that size. It takes - `SWWAF_WAF_BODY_LIMIT` switches body inspection on, up to that size. It takes
at most `64K`, since `servewaf` gets no more of a body than that (see "The at most `64K`, since `servewaf` gets no more of a body than that (see "The
answer"). Form data and multipart bodies are read up to the size. JSON and XML answer"). Form data and multipart bodies are read up to the size. JSON and XML
@@ -610,7 +632,7 @@ files, the limits and the bans still apply to the client that sends it.
scanner from an ordinary client. In front of a code forge, registry and git scanner from an ordinary client. In front of a code forge, registry and git
clients get `404` and `401` by design, hundreds of times a minute. clients get `404` and `401` by design, hundreds of times a minute.
`smallwebwaf` changes the Core Rule Set 4.25.0 in six ways, since in front of `smallwebwaf` changes the Core Rule Set 4.25.0 in eight ways, since in front of
gitea it would otherwise refuse ordinary requests. The changes hold in front of gitea it would otherwise refuse ordinary requests. The changes hold in front of
every app, and no setting undoes them: every app, and no setting undoes them:
@@ -642,9 +664,10 @@ every app, and no setting undoes them:
or `docker-build`. In these parameters, what only these three rules would or `docker-build`. In these parameters, what only these three rules would
catch reaches the app, such as the command `|cat /etc/passwd`. Path catch reaches the app, such as the command `|cat /etc/passwd`. Path
traversal, SQL and script injection, and PHP, Java and Node.js code are traversal, SQL and script injection, and PHP, Java and Node.js code are
still refused there. Every other parameter keeps all three rules. An app still refused there. Other parameters keep all three rules, except on the
that uses one of these parameters as a file on the server, or passes it to a gitea paths the eighth change names. An app that uses one of these
shell, must check the value itself. A rule file can refuse it instead. parameters as a file on the server, or passes it to a shell, must check the
value itself. A rule file can refuse it instead.
5. Two cookies are left out, and `Referer` keeps fewer rules: 5. Two cookies are left out, and `Referer` keeps fewer rules:
- The Core Rule Set reads the request without the `gitea_flash` and - The Core Rule Set reads the request without the `gitea_flash` and
@@ -658,15 +681,83 @@ every app, and no setting undoes them:
only to its own pages. Every other cookie is read in full. An app that only to its own pages. Every other cookie is read in full. An app that
reads a cookie of either name must check it itself. reads a cookie of either name must check it itself.
- `Referer` is not checked for a Unix command given without arguments - `Referer` is not checked for a Unix command given without arguments
(932340) or for Java starting a process (944110). `Referer` names the page (932340), for Java starting a process (944110) or for a Java class name
a request comes from, on everything that page loads and on every link (944130). `Referer` names the page a request comes from, on everything
followed from it. The Core Rule Set takes some page addresses for attacks, that page loads and on every link followed from it. The Core Rule Set
such as a search for `env`, or OpenJDK's `Runtime.java`, whose path holds takes some page addresses for attacks:
both `runtime` and `java.`. All of those requests would be refused. - a search for `env`;
`Referer` keeps the other rules. An app that passes `Referer` to a shell - OpenJDK's `Runtime.java`, whose path holds both `runtime` and `java.`;
must check it itself. - a file named `java.lang.String.md`.
Every request from such a page would be refused. `Referer` keeps the
other rules, except after some gitea searches (the eighth change). An
app that passes `Referer` to a shell must check it itself.
6. Responses are not inspected. A raw file from a repository, such as a shell 6. Responses are not inspected. A raw file from a repository, such as a shell
script, looks to the response rules like source code leaking from the server. script, looks to the response rules like source code leaking from the server.
7. The `from` parameter of a container registry's upload path,
`/v2/<image>/blobs/uploads/`, is not checked for command names (932260). When
a push reuses a layer, `from` names the image the layer comes from. An image
such as `owner/python3-app` would have that push refused.
8. On gitea's paths, some fields skip the rules that refused ordinary gitea
requests there. They are the fields where gitea sends a search, a file name
or a branch name. `internal/waf/gitea.conf` lists each rule and the request
it refused. The fields are:
- `q` on gitea's searches of code, of issues and pull requests, and of
commits. A code search holds code, such as `<script>`,
`SELECT * FROM users WHERE` or `../`.
- `Referer` of a page opened from such a search, for script and SQL
injection and Java code (941110, 941120, 941160, 942100, 944100).
- `q` on gitea's searches for a name, for the names of system files and
commands (930120, 932120, 932260). Names such as `.profile` and
`python3-app` were refused. gitea searches for the names of:
- repositories;
- users and organisations;
- branches and tags;
- packages.
A user's or an organisation's page is any path of one segment, so on
every such path, on any app, `q` skips these three rules, for example on
`/search`.
Some of these searches also skip the rule for Java class names (944130),
which refused `org.apache.commons.logging`.
- The path of these requests, which names a file:
- a file's page;
- its raw file;
- its download;
- its blame page;
- its history;
- its edit page;
- its preview in the editor;
- its delete page;
- the API's call for its contents;
- a release attachment's download.
Names such as `notes.txt~`, `process.env.d.ts` and `java.lang.String.md`
were refused.
- A file name in a query parameter:
- `file` of a commit graph;
- `path` of the API's list of commits;
- `path` of a diff's hidden lines;
- `body` of the API's call for several files;
- `template` of a new issue;
- `redirect_to` of the sign-in page;
- `name` of an attachment uploaded through the API.
- A branch or tag name in a query parameter, for command names (932260).
gitea sends one:
- to delete or restore a branch;
- to start a release from a tag;
- to edit a branch protection rule;
- to list a directory's last commits.
- The `Content-Range` header of an artifact upload from
`actions/upload-artifact@v3` (920450).
Elsewhere these fields keep every rule. The paths are those of gitea at the
site root. An app other than gitea that serves one of them must check these
fields itself.
The parameter and cookie names in these changes are matched without regard to The parameter and cookie names in these changes are matched without regard to
case, as Coraza matches them. case, as Coraza matches them.
+11 -3
View File
@@ -191,6 +191,15 @@ chooses between the two (see "The image's command" below).
the Core Rule Set reads, a size of at most `64K`. A question carries no more the Core Rule Set reads, a size of at most `64K`. A question carries no more
of a body than 64 KiB, and the Core Rule Set reads at most 64 KiB less one of a body than 64 KiB, and the Core Rule Set reads at most 64 KiB less one
byte of it, whatever the value (see "The answer" in [`SPEC.md`](../SPEC.md)). byte of it, whatever the value (see "The answer" in [`SPEC.md`](../SPEC.md)).
- `SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE` (default unset): a file of Core Rule Set
exclusions, written as Coraza directives. It is loaded before the Core Rule
Set's rules, so it holds only `SecAction` and `SecRule` lines that exclude
with `ctl:ruleRemoveTargetById` or `ctl:ruleRemoveById`. A file that cannot be
read or does not parse stops the start, whatever `SWWAF_WAF_MODE` is. So does
a file that names a number from 900000 to 900999 outside a comment, alone or
in a range. So does a directive that removes or changes rules as the file
loads, such as `SecRuleRemoveById`, since none of the Core Rule Set's rules is
loaded yet. It has no `_FILE` form.
- `SWWAF_ERROR_BURST_THRESHOLD` (default `30`): the most refusals of one client - `SWWAF_ERROR_BURST_THRESHOLD` (default `30`): the most refusals of one client
in a minute (see "Attack detection" in `SPEC.md`). A count or `off`. in a minute (see "Attack detection" in `SPEC.md`). A count or `off`.
@@ -345,9 +354,8 @@ asked in the background (see "Answering in time" under "`servewaf`" in
## `rproxy` ## `rproxy`
`smallwebwaf rproxy` reads the settings below and no others. They are read once `smallwebwaf rproxy` reads the settings below and no others. They are read once
at start, and an invalid value stops the start. None can be given as a file, at start, and an invalid value stops the start. Any can be given as a file, as a
since `rproxy` never touches disk. "`rproxy`" in [`SPEC.md`](../SPEC.md) says `SWWAF_` setting can. "`rproxy`" in [`SPEC.md`](../SPEC.md) says what they do.
what they do.
A time here is a whole number of milliseconds above zero, such as `2000`. A time here is a whole number of milliseconds above zero, such as `2000`.
+10
View File
@@ -10,12 +10,14 @@ require (
github.com/maxmind/mmdbwriter v1.2.0 github.com/maxmind/mmdbwriter v1.2.0
github.com/oschwald/maxminddb-golang/v2 v2.7.0 github.com/oschwald/maxminddb-golang/v2 v2.7.0
github.com/prometheus/client_golang v1.24.1 github.com/prometheus/client_golang v1.24.1
github.com/spf13/viper v1.21.0
) )
require ( require (
github.com/beorn7/perks v1.0.1 // indirect github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/corazawaf/libinjection-go v0.3.3 // indirect github.com/corazawaf/libinjection-go v0.3.3 // indirect
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
github.com/goccy/go-json v0.10.5 // indirect github.com/goccy/go-json v0.10.5 // indirect
github.com/goccy/go-yaml v1.19.2 // indirect github.com/goccy/go-yaml v1.19.2 // indirect
github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect github.com/gotnospirit/makeplural v0.0.0-20180622080156-a5f48d94d976 // indirect
@@ -25,14 +27,22 @@ require (
github.com/kylelemons/godebug v1.1.0 // indirect github.com/kylelemons/godebug v1.1.0 // indirect
github.com/magefile/mage v1.17.0 // indirect github.com/magefile/mage v1.17.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/petar-dambovaliev/aho-corasick v0.0.0-20250424160509-463d218d4745 // indirect github.com/petar-dambovaliev/aho-corasick v0.0.0-20250424160509-463d218d4745 // indirect
github.com/prometheus/client_model v0.6.2 // indirect github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.70.1 // indirect github.com/prometheus/common v0.70.1 // indirect
github.com/prometheus/procfs v0.21.1 // indirect github.com/prometheus/procfs v0.21.1 // indirect
github.com/sagikazarmark/locafero v0.11.0 // indirect
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/tidwall/gjson v1.18.0 // indirect github.com/tidwall/gjson v1.18.0 // indirect
github.com/tidwall/match v1.1.1 // indirect github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.1 // indirect github.com/tidwall/pretty v1.2.1 // indirect
github.com/valllabh/ocsf-schema-golang v1.0.3 // indirect github.com/valllabh/ocsf-schema-golang v1.0.3 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect go4.org/netipx v0.0.0-20231129151722-fdeea329fbba // indirect
golang.org/x/net v0.58.0 // indirect golang.org/x/net v0.58.0 // indirect
golang.org/x/sync v0.23.0 // indirect golang.org/x/sync v0.23.0 // indirect
+24
View File
@@ -12,8 +12,12 @@ github.com/corazawaf/libinjection-go v0.3.3 h1:NhbXKRfRpqKzBMzv8zpCcnjyEw7BCVhBO
github.com/corazawaf/libinjection-go v0.3.3/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk= github.com/corazawaf/libinjection-go v0.3.3/go.mod h1:Ik/+w3UmTWH9yn366RgS9D95K3y7Atb5m/H/gXzzPCk=
github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0= github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0=
github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk= github.com/foxcpp/go-mockdns v1.2.0/go.mod h1:IhLeSFGed3mJIAXPH2aiRQB+kqz7oqu8ld2qVbOu7Wk=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho= github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo= github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4= github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M= github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/goccy/go-yaml v1.19.2 h1:PmFC1S6h8ljIz6gMRBopkjP1TVT7xuwrButHID66PoM= github.com/goccy/go-yaml v1.19.2 h1:PmFC1S6h8ljIz6gMRBopkjP1TVT7xuwrButHID66PoM=
@@ -34,6 +38,10 @@ github.com/kaptinlin/jsonschema v0.4.6 h1:vOSFg5tjmfkOdKg+D6Oo4fVOM/pActWu/ntkPs
github.com/kaptinlin/jsonschema v0.4.6/go.mod h1:1DUd7r5SdyB2ZnMtyB7uLv64dE3zTFTiYytDCd+AEL0= github.com/kaptinlin/jsonschema v0.4.6/go.mod h1:1DUd7r5SdyB2ZnMtyB7uLv64dE3zTFTiYytDCd+AEL0=
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk= github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/magefile/mage v1.17.0 h1:dS4tkq997Ism03akafC8509iqDjeE7TNTexI25Y7sXM= github.com/magefile/mage v1.17.0 h1:dS4tkq997Ism03akafC8509iqDjeE7TNTexI25Y7sXM=
@@ -58,8 +66,24 @@ github.com/prometheus/common v0.70.1 h1:1HvjP4D5oL3t8RsPlwxA9onvvStjtIHYE5XuuwOi
github.com/prometheus/common v0.70.1/go.mod h1:VdFUQDMZK3VLkurFUVhia6uys/0suUp86TJz5qbJRhc= github.com/prometheus/common v0.70.1/go.mod h1:VdFUQDMZK3VLkurFUVhia6uys/0suUp86TJz5qbJRhc=
github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+OJI= github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+OJI=
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY= github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/sagikazarmark/locafero v0.11.0 h1:1iurJgmM9G3PA/I+wWYIOw/5SyBtxapeHDcg+AAIFXc=
github.com/sagikazarmark/locafero v0.11.0/go.mod h1:nVIGvgyzw595SUSUE6tvCp3YYTeHs15MvlmU87WwIik=
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8/go.mod h1:3n1Cwaq1E1/1lhQhtRK2ts/ZwZEhjcQeJQ1RuC6Q/8U=
github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
github.com/spf13/viper v1.21.0/go.mod h1:P0lhsswPGWD/1lZJ9ny3fYnVqxiegrlNrEmgLjbTCAY=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE= github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg= github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
github.com/tidwall/gjson v1.18.0 h1:FIDeeyB800efLX89e5a8Y0BNH+LOngJyGrIWxG2FKQY= github.com/tidwall/gjson v1.18.0 h1:FIDeeyB800efLX89e5a8Y0BNH+LOngJyGrIWxG2FKQY=
github.com/tidwall/gjson v1.18.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk= github.com/tidwall/gjson v1.18.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA= github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
+55 -21
View File
@@ -2,9 +2,8 @@
// read. Every setting is an environment variable. The settings of `smallwebwaf // read. Every setting is an environment variable. The settings of `smallwebwaf
// servewaf` start with SWWAF_, every one of them has a default, and most can be // servewaf` start with SWWAF_, every one of them has a default, and most can be
// given as a file instead. Those of `smallwebwaf rproxy` start with SWRP_. Each // given as a file instead. Those of `smallwebwaf rproxy` start with SWRP_. Each
// of them but SWRP_WAF_BACKEND has a default, and none can be given as a file, // of them but SWRP_WAF_BACKEND has a default, and each can be given as a file
// since rproxy never touches disk. SMALLWEBWAF_COMMAND chooses which of the two // too. SMALLWEBWAF_COMMAND chooses which of the two the image runs.
// the image runs.
package config package config
import ( import (
@@ -26,6 +25,8 @@ import (
"unicode" "unicode"
"unicode/utf8" "unicode/utf8"
"github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly" "sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/remotelog" "sneak.berlin/go/smallwebwaf/internal/remotelog"
@@ -221,12 +222,15 @@ type Config struct {
// WAFExemptPaths the path prefixes it does not inspect // WAFExemptPaths the path prefixes it does not inspect
// (SWWAF_WAF_EXEMPT_PATHS), and WAFBodyLimit the most of a request body // (SWWAF_WAF_EXEMPT_PATHS), and WAFBodyLimit the most of a request body
// it reads (SWWAF_WAF_BODY_LIMIT), 0 while it is off and it reads none. // it reads (SWWAF_WAF_BODY_LIMIT), 0 while it is off and it reads none.
// CoreRuleSetExclusionsFile is the path of the file of its exclusions
// (SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE), "" while there is none.
WAFMode string WAFMode string
WAFParanoiaLevel int WAFParanoiaLevel int
WAFAnomalyThreshold int WAFAnomalyThreshold int
WAFDisabledRules []int WAFDisabledRules []int
WAFExemptPaths []string WAFExemptPaths []string
WAFBodyLimit int64 WAFBodyLimit int64
CoreRuleSetExclusionsFile string
// TrapPaths are the paths a request for which is a clear sign of // TrapPaths are the paths a request for which is a clear sign of
// attack (SWWAF_TRAP_PATHS), each starting with / and without a ?. // attack (SWWAF_TRAP_PATHS), each starting with / and without a ?.
TrapPaths []string TrapPaths []string
@@ -450,15 +454,16 @@ var (
"is in SWWAF_BLOCKLIST_URLS or is SWWAF_ASN_LIMIT_PERCENT_URL too") "is in SWWAF_BLOCKLIST_URLS or is SWWAF_ASN_LIMIT_PERCENT_URL too")
) )
// FromEnvironment reads the settings with lookupEnv, normally // FromEnvironment reads the settings through v, normally the viper
// os.LookupEnv. A setting may instead be given as a file: the variable // NewViper returns, bound to the process's environment. A setting may
// named by the setting's name with _FILE added names the file, which is // instead be given as a file: the variable named by the setting's name
// read now (see lookup). A setting that is not set takes its default. A // with _FILE added names the file, which is read now (see lookup). A
// setting that is set but invalid is an error that names it. // setting that is not set takes its default. A setting that is set but
// invalid is an error that names it.
// //
//nolint:funlen // one line for each setting, a list that grows with them //nolint:funlen // one line for each setting, a list that grows with them
func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) { func FromEnvironment(v *viper.Viper) (*Config, error) {
env := &environment{lookupEnv: lookupEnv} env := &environment{v: v, readFiles: true}
cfg := &Config{ cfg := &Config{
ListenAddr: env.address("SWWAF_LISTEN_ADDR"), ListenAddr: env.address("SWWAF_LISTEN_ADDR"),
InstanceName: env.instanceName(), InstanceName: env.instanceName(),
@@ -525,6 +530,8 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
"920340,920420,920440,920640,930130,930140"), "920340,920420,920440,920640,930130,930140"),
WAFExemptPaths: env.pathPrefixes("SWWAF_WAF_EXEMPT_PATHS", ""), WAFExemptPaths: env.pathPrefixes("SWWAF_WAF_EXEMPT_PATHS", ""),
WAFBodyLimit: env.wafBodyLimit("SWWAF_WAF_BODY_LIMIT", off), WAFBodyLimit: env.wafBodyLimit("SWWAF_WAF_BODY_LIMIT", off),
CoreRuleSetExclusionsFile: env.filePath(
"SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE"),
TrapPaths: env.trapPaths("SWWAF_TRAP_PATHS"), TrapPaths: env.trapPaths("SWWAF_TRAP_PATHS"),
ErrorBurstThreshold: env.count("SWWAF_ERROR_BURST_THRESHOLD", "30"), ErrorBurstThreshold: env.count("SWWAF_ERROR_BURST_THRESHOLD", "30"),
LogRemoteURL: env.logRemoteURL("SWWAF_LOG_REMOTE_URL"), LogRemoteURL: env.logRemoteURL("SWWAF_LOG_REMOTE_URL"),
@@ -575,8 +582,8 @@ func FromEnvironment(lookupEnv func(string) (string, bool)) (*Config, error) {
// Command reads only SMALLWEBWAF_COMMAND, the command the image runs and its // Command reads only SMALLWEBWAF_COMMAND, the command the image runs and its
// health check checks: rproxy when it is unset or empty, or servewaf. It cannot // health check checks: rproxy when it is unset or empty, or servewaf. It cannot
// be given as a file. Any other value is an error that names the setting. // be given as a file. Any other value is an error that names the setting.
func Command(lookupEnv func(string) (string, bool)) (string, error) { func Command(v *viper.Viper) (string, error) {
command, _ := lookupEnv("SMALLWEBWAF_COMMAND") command, _ := readSetting(v, "SMALLWEBWAF_COMMAND")
switch command { switch command {
case "", "rproxy": case "", "rproxy":
@@ -591,8 +598,8 @@ func Command(lookupEnv func(string) (string, bool)) (string, error) {
// ServeWAFListenAddr reads only SWWAF_LISTEN_ADDR, which may be given as a // ServeWAFListenAddr reads only SWWAF_LISTEN_ADDR, which may be given as a
// file, as FromEnvironment does. The health check in servewaf's container needs // file, as FromEnvironment does. The health check in servewaf's container needs
// no other setting, so it reads no other, nor a file that another names. // no other setting, so it reads no other, nor a file that another names.
func ServeWAFListenAddr(lookupEnv func(string) (string, bool)) (string, error) { func ServeWAFListenAddr(v *viper.Viper) (string, error) {
env := &environment{lookupEnv: lookupEnv} env := &environment{v: v, readFiles: true}
listenAddr := env.address("SWWAF_LISTEN_ADDR") listenAddr := env.address("SWWAF_LISTEN_ADDR")
return listenAddr, env.err return listenAddr, env.err
@@ -602,8 +609,8 @@ func ServeWAFListenAddr(lookupEnv func(string) (string, bool)) (string, error) {
// file, as FromEnvironment does, so that the line saying a setting is // file, as FromEnvironment does, so that the line saying a setting is
// invalid carries it too. A file that cannot be read gives the default // invalid carries it too. A file that cannot be read gives the default
// here, and FromEnvironment then stops the start over it. // here, and FromEnvironment then stops the start over it.
func InstanceName(lookupEnv func(string) (string, bool)) string { func InstanceName(v *viper.Viper) string {
env := &environment{lookupEnv: lookupEnv} env := &environment{v: v, readFiles: true}
return env.instanceName() return env.instanceName()
} }
@@ -619,13 +626,26 @@ func (c *Config) LogValue() slog.Value {
} }
// environment is where FromEnvironment reads the settings: it notes each // environment is where FromEnvironment reads the settings: it notes each
// value for the log, and keeps the first error. // value for the log, and keeps the first error. It reads every setting
// through v, the viper NewViper returns, and, unless readFiles is false,
// honours each setting's _FILE form too.
type environment struct { type environment struct {
lookupEnv func(string) (string, bool) v *viper.Viper
readFiles bool
settings []slog.Attr settings []slog.Attr
err error err error
} }
// readSetting returns a setting's value and whether it is set, reading it
// through v.
func readSetting(v *viper.Viper, name string) (string, bool) {
if !v.IsSet(name) {
return "", false
}
return v.GetString(name), true
}
// value returns a setting's value, or its default when it is not set, // value returns a setting's value, or its default when it is not set,
// and notes it for the log. // and notes it for the log.
func (e *environment) value(name, defaultValue string) string { func (e *environment) value(name, defaultValue string) string {
@@ -644,10 +664,14 @@ func (e *environment) value(name, defaultValue string) string {
// names, less one newline at their end. It notes that file's path for the // names, less one newline at their end. It notes that file's path for the
// log. Both variables set, or a file that cannot be read, is an error. // log. Both variables set, or a file that cannot be read, is an error.
func (e *environment) lookup(name string) (string, bool) { func (e *environment) lookup(name string) (string, bool) {
value, set := e.lookupEnv(name) value, set := readSetting(e.v, name)
if !e.readFiles {
return value, set
}
fileName := name + "_FILE" fileName := name + "_FILE"
path, inFile := e.lookupEnv(fileName) path, inFile := readSetting(e.v, fileName)
if !inFile { if !inFile {
return value, set return value, set
} }
@@ -1189,11 +1213,21 @@ func (e *environment) logRemoteURL(name string) *url.URL {
return remote return remote
} }
// filePath reads a setting that is the path of a file, "" while it is
// unset. Its value names a file already, so, unlike the other settings,
// it has no _FILE form. Whoever uses the file reads it.
func (e *environment) filePath(name string) string {
path, _ := readSetting(e.v, name)
e.settings = append(e.settings, slog.String(name, path))
return path
}
// certificates reads a setting that is the path of a file of PEM // certificates reads a setting that is the path of a file of PEM
// certificates. Unset or empty, it is nil. Its value names a file // certificates. Unset or empty, it is nil. Its value names a file
// already, so, unlike the other settings, it has no _FILE form. // already, so, unlike the other settings, it has no _FILE form.
func (e *environment) certificates(name string) *x509.CertPool { func (e *environment) certificates(name string) *x509.CertPool {
path, _ := e.lookupEnv(name) path, _ := readSetting(e.v, name)
e.settings = append(e.settings, slog.String(name, path)) e.settings = append(e.settings, slog.String(name, path))
if path == "" { if path == "" {
+46 -36
View File
@@ -17,6 +17,8 @@ import (
"testing" "testing"
"time" "time"
"github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/anomaly" "sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
) )
@@ -87,6 +89,7 @@ const (
wafDisabledRules = "SWWAF_WAF_DISABLED_RULES" wafDisabledRules = "SWWAF_WAF_DISABLED_RULES"
wafExemptPaths = "SWWAF_WAF_EXEMPT_PATHS" wafExemptPaths = "SWWAF_WAF_EXEMPT_PATHS"
wafBodyLimit = "SWWAF_WAF_BODY_LIMIT" wafBodyLimit = "SWWAF_WAF_BODY_LIMIT"
exclusionsFile = "SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE"
trapPaths = "SWWAF_TRAP_PATHS" trapPaths = "SWWAF_TRAP_PATHS"
errorBurstThreshold = "SWWAF_ERROR_BURST_THRESHOLD" errorBurstThreshold = "SWWAF_ERROR_BURST_THRESHOLD"
logRemoteURL = "SWWAF_LOG_REMOTE_URL" logRemoteURL = "SWWAF_LOG_REMOTE_URL"
@@ -217,18 +220,22 @@ const (
// environment is a set of environment variables, for FromEnvironment. // environment is a set of environment variables, for FromEnvironment.
type environment map[string]string type environment map[string]string
// lookupEnv reads one of the variables, as os.LookupEnv does. // viper returns a viper Set with each of the variables, as the viper
func (e environment) lookupEnv(name string) (string, bool) { // NewViper returns one bound to a process's environment.
value, ok := e[name] func (e environment) viper() *viper.Viper {
v := viper.New()
for name, value := range e {
v.Set(name, value)
}
return value, ok return v
} }
// fromEnvironment reads the settings from env, which must be valid. // fromEnvironment reads the settings from env, which must be valid.
func fromEnvironment(t *testing.T, env environment) *config.Config { func fromEnvironment(t *testing.T, env environment) *config.Config {
t.Helper() t.Helper()
cfg, err := config.FromEnvironment(env.lookupEnv) cfg, err := config.FromEnvironment(env.viper())
if err != nil { if err != nil {
t.Fatalf("settings %v: %v", env, err) t.Fatalf("settings %v: %v", env, err)
} }
@@ -455,7 +462,7 @@ func TestPathPrefixNotStartingWithSlashStopsTheStart(t *testing.T) {
t.Parallel() t.Parallel()
_, err := config.FromEnvironment( _, err := config.FromEnvironment(
environment{rateLimitExemptPaths: "/favicon.ico,assets/"}.lookupEnv) environment{rateLimitExemptPaths: "/favicon.ico,assets/"}.viper())
want := rateLimitExemptPaths + `: "assets/" is not a path prefix ` + want := rateLimitExemptPaths + `: "assets/" is not a path prefix ` +
`starting with /, such as /assets/` `starting with /, such as /assets/`
@@ -510,7 +517,7 @@ func TestInvalidTrapPathOrErrorBurstThresholdStopsTheStart(t *testing.T) {
t.Run(tc.name+"="+tc.value, func(t *testing.T) { t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel() t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv) _, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
want := tc.name + ": " + tc.want want := tc.name + ": " + tc.want
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
@@ -539,13 +546,14 @@ func TestCoreRuleSetSettings(t *testing.T) {
environment{ environment{
wafMode: config.WAFModeDetect, wafParanoiaLevel: "4", wafAnomalyThreshold: "10", wafMode: config.WAFModeDetect, wafParanoiaLevel: "4", wafAnomalyThreshold: "10",
wafDisabledRules: "942100, 920350", wafExemptPaths: "/api/, /static/", wafDisabledRules: "942100, 920350", wafExemptPaths: "/api/, /static/",
wafBodyLimit: "32K", wafBodyLimit: "32K", exclusionsFile: "/etc/smallwebwaf/crs-exclusions.conf",
}, },
config.Config{ config.Config{
WAFMode: config.WAFModeDetect, WAFParanoiaLevel: 4, WAFAnomalyThreshold: 10, WAFMode: config.WAFModeDetect, WAFParanoiaLevel: 4, WAFAnomalyThreshold: 10,
WAFDisabledRules: []int{942100, 920350}, WAFDisabledRules: []int{942100, 920350},
WAFExemptPaths: []string{"/api/", "/static/"}, WAFExemptPaths: []string{"/api/", "/static/"},
WAFBodyLimit: 32 << 10, WAFBodyLimit: 32 << 10,
CoreRuleSetExclusionsFile: "/etc/smallwebwaf/crs-exclusions.conf",
}, },
}, },
{ {
@@ -566,6 +574,7 @@ func TestCoreRuleSetSettings(t *testing.T) {
WAFAnomalyThreshold: cfg.WAFAnomalyThreshold, WAFAnomalyThreshold: cfg.WAFAnomalyThreshold,
WAFDisabledRules: cfg.WAFDisabledRules, WAFExemptPaths: cfg.WAFExemptPaths, WAFDisabledRules: cfg.WAFDisabledRules, WAFExemptPaths: cfg.WAFExemptPaths,
WAFBodyLimit: cfg.WAFBodyLimit, WAFBodyLimit: cfg.WAFBodyLimit,
CoreRuleSetExclusionsFile: cfg.CoreRuleSetExclusionsFile,
} }
if !reflect.DeepEqual(got, tc.want) { if !reflect.DeepEqual(got, tc.want) {
t.Errorf("%v gave\n%+v\nwant\n%+v", tc.env, got, tc.want) t.Errorf("%v gave\n%+v\nwant\n%+v", tc.env, got, tc.want)
@@ -636,7 +645,7 @@ func TestInvalidCoreRuleSetSettingStopsTheStart(t *testing.T) {
t.Run(tc.name+"="+tc.value, func(t *testing.T) { t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel() t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv) _, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
want := tc.name + ": " + tc.want want := tc.name + ": " + tc.want
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
@@ -761,7 +770,7 @@ func TestInvalidRemoteLogSettingStopsTheStart(t *testing.T) {
{logRemoteAppName, ""}, {logRemoteAppName, "my app"}, {logRemoteAppName, ""}, {logRemoteAppName, "my app"},
{logRemoteAppName, "gitéa"}, {logRemoteAppName, strings.Repeat("a", 49)}, {logRemoteAppName, "gitéa"}, {logRemoteAppName, strings.Repeat("a", 49)},
} { } {
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv) _, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
if err == nil || !strings.HasPrefix(err.Error(), tc.name+": ") { if err == nil || !strings.HasPrefix(err.Error(), tc.name+": ") {
t.Errorf("%s=%q: error %v, want one naming it", tc.name, tc.value, err) t.Errorf("%s=%q: error %v, want one naming it", tc.name, tc.value, err)
} }
@@ -778,7 +787,7 @@ func TestRemoteLogCAFileWithoutCertificateStopsTheStart(t *testing.T) {
t.Fatalf("write %s: %v", caFile, err) t.Fatalf("write %s: %v", caFile, err)
} }
_, err = config.FromEnvironment(environment{logRemoteTLSCAFile: caFile}.lookupEnv) _, err = config.FromEnvironment(environment{logRemoteTLSCAFile: caFile}.viper())
want := logRemoteTLSCAFile + `: "` + caFile + `" holds no PEM certificate` want := logRemoteTLSCAFile + `: "` + caFile + `" holds no PEM certificate`
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
@@ -793,7 +802,7 @@ func TestInstanceNameNotAnAppNameStopsTheStartOnlyWhileSending(t *testing.T) {
sending := environment{logRemoteURL: remoteURL, instanceName: spaced} sending := environment{logRemoteURL: remoteURL, instanceName: spaced}
_, err := config.FromEnvironment(sending.lookupEnv) _, err := config.FromEnvironment(sending.viper())
want := logRemoteAppName + `: is unset, and ` + instanceName + want := logRemoteAppName + `: is unset, and ` + instanceName +
` "fsn1 app1", its default, is not 1 to 48 printable ASCII characters ` + ` "fsn1 app1", its default, is not 1 to 48 printable ASCII characters ` +
@@ -823,7 +832,7 @@ func TestAppNameSetStopsTheStartWhileSending(t *testing.T) {
logRemoteURL: remoteURL, logRemoteURL: remoteURL,
instanceName: instance, instanceName: instance,
logRemoteAppName: "my app", logRemoteAppName: "my app",
}.lookupEnv) }.viper())
want := logRemoteAppName + `: "my app" is not 1 to 48 printable ASCII ` + want := logRemoteAppName + `: "my app" is not 1 to 48 printable ASCII ` +
`characters without a space, such as blog` `characters without a space, such as blog`
@@ -954,7 +963,7 @@ func TestWebhookHeadersAreLoggedMaskedAndNeverShown(t *testing.T) {
// An item that is not a header is named by its place, not shown. // An item that is not a header is named by its place, not shown.
_, err := config.FromEnvironment(environment{ _, err := config.FromEnvironment(environment{
alertWebhookHeaders: "X-Team:ops," + secret, alertWebhookHeaders: "X-Team:ops," + secret,
}.lookupEnv) }.viper())
want := alertWebhookHeaders + ": item 2 is not a header name followed by : " + want := alertWebhookHeaders + ": item 2 is not a header name followed by : " +
"and the header's value, such as Authorization:Bearer <token>" "and the header's value, such as Authorization:Bearer <token>"
@@ -991,7 +1000,7 @@ func TestWebhookURLIsLoggedWithoutItsPathOrQueryAndNeverShown(t *testing.T) {
"ftp://hooks.example/services/" + secret, "ftp://hooks.example/services/" + secret,
"https://hooks.example/services/%zz" + secret, "https://hooks.example/services/%zz" + secret,
} { } {
_, err := config.FromEnvironment(environment{alertWebhookURL: value}.lookupEnv) _, err := config.FromEnvironment(environment{alertWebhookURL: value}.viper())
want := alertWebhookURL + ": is not an http or https URL without a user or " + want := alertWebhookURL + ": is not an http or https URL without a user or " +
"a fragment, such as https://alerts.example/smallwebwaf" "a fragment, such as https://alerts.example/smallwebwaf"
@@ -1042,7 +1051,7 @@ func TestNtfyTokenWithAControlCharacterStopsTheStartWithoutShowingIt(t *testing.
"set": {alertNtfyToken: token + "\r"}, "set": {alertNtfyToken: token + "\r"},
"in a file": {alertNtfyToken + "_FILE": writeFile(t, token+"\r\n")}, "in a file": {alertNtfyToken + "_FILE": writeFile(t, token+"\r\n")},
} { } {
_, err := config.FromEnvironment(env.lookupEnv) _, err := config.FromEnvironment(env.viper())
want := alertNtfyToken + ": holds a control character, such as the " + want := alertNtfyToken + ": holds a control character, such as the " +
"carriage return of a Windows line end" "carriage return of a Windows line end"
@@ -1059,7 +1068,7 @@ func TestInstanceNameWithAControlCharacterStopsTheStartOnlyWithNtfySet(t *testin
_, err := config.FromEnvironment(environment{ _, err := config.FromEnvironment(environment{
instanceName: name, alertNtfyURL: "https://ntfy.example/smallwebwaf-alerts", instanceName: name, alertNtfyURL: "https://ntfy.example/smallwebwaf-alerts",
}.lookupEnv) }.viper())
want := instanceName + `: "fsn1app1\r" holds a control character, such as the ` + want := instanceName + `: "fsn1app1\r" holds a control character, such as the ` +
`carriage return of a Windows line end, and is sent to ntfy in a header ` + `carriage return of a Windows line end, and is sent to ntfy in a header ` +
@@ -1084,7 +1093,7 @@ func TestInstanceNameNotUTF8StopsTheStart(t *testing.T) {
"set": {instanceName: latin1}, "set": {instanceName: latin1},
"in a file": {instanceName + "_FILE": writeFile(t, latin1+"\n")}, "in a file": {instanceName + "_FILE": writeFile(t, latin1+"\n")},
} { } {
_, err := config.FromEnvironment(env.lookupEnv) _, err := config.FromEnvironment(env.viper())
want := instanceName + `: "caf\xe9" is not valid UTF-8` want := instanceName + `: "caf\xe9" is not valid UTF-8`
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
@@ -1099,7 +1108,7 @@ func TestCodeOnBothCountryListsStopsTheStart(t *testing.T) {
_, err := config.FromEnvironment(environment{ _, err := config.FromEnvironment(environment{
deniedCountries: "cn,ru", deniedCountries: "cn,ru",
allowedCountries: "de,RU", allowedCountries: "de,RU",
}.lookupEnv) }.viper())
if err == nil { if err == nil {
t.Fatal("ru on both country lists was accepted") t.Fatal("ru on both country lists was accepted")
} }
@@ -1161,7 +1170,7 @@ func TestLookupDBPathGoesWithTheFileSourceAlone(t *testing.T) {
lookupDBPath + ": is set while " + lookupSource + " is off; only file reads it", lookupDBPath + ": is set while " + lookupSource + " is off; only file reads it",
}, },
} { } {
_, err := config.FromEnvironment(tc.env.lookupEnv) _, err := config.FromEnvironment(tc.env.viper())
if err == nil || err.Error() != tc.want { if err == nil || err.Error() != tc.want {
t.Errorf("settings %v: error %v, want %s", tc.env, err, tc.want) t.Errorf("settings %v: error %v, want %s", tc.env, err, tc.want)
} }
@@ -1191,7 +1200,7 @@ func TestSettingNeedingLookupsStopsTheStartWhileTheyAreOff(t *testing.T) {
_, err := config.FromEnvironment(environment{ _, err := config.FromEnvironment(environment{
lookupSource: off, name: value, lookupSource: off, name: value,
}.lookupEnv) }.viper())
if err == nil || !strings.HasPrefix(err.Error(), name+": ") || if err == nil || !strings.HasPrefix(err.Error(), name+": ") ||
!strings.Contains(err.Error(), lookupSource+" is off") { !strings.Contains(err.Error(), lookupSource+" is off") {
t.Errorf("error %v, want one naming %s and %s", err, name, lookupSource) t.Errorf("error %v, want one naming %s and %s", err, name, lookupSource)
@@ -1341,7 +1350,7 @@ func TestInvalidAnomalySettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
t.Run(tc.name+"="+tc.value, func(t *testing.T) { t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel() t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv) _, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
want := tc.name + ": " + tc.want want := tc.name + ": " + tc.want
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
@@ -1420,7 +1429,7 @@ func TestInvalidBiasedThresholdStopsTheStartSayingWhatIsWrong(t *testing.T) {
t.Run(tc.name+"="+tc.value, func(t *testing.T) { t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel() t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv) _, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
want := tc.name + ": " + tc.want want := tc.name + ": " + tc.want
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
@@ -1517,7 +1526,7 @@ func TestInvalidReputationSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
t.Run(tc.name+"="+tc.value, func(t *testing.T) { t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel() t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv) _, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
want := tc.name + ": " + tc.want want := tc.name + ": " + tc.want
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
@@ -1532,7 +1541,7 @@ func TestASNLimitPercentURLThatIsABlocklistStopsTheStart(t *testing.T) {
_, err := config.FromEnvironment(environment{ _, err := config.FromEnvironment(environment{
blocklistURLs: dropURL + "," + asnURL, asnLimitPercentURL: asnURL, blocklistURLs: dropURL + "," + asnURL, asnLimitPercentURL: asnURL,
}.lookupEnv) }.viper())
want := asnLimitPercentURL + `: "` + asnURL + `" is in SWWAF_BLOCKLIST_URLS too` want := asnLimitPercentURL + `: "` + asnURL + `" is in SWWAF_BLOCKLIST_URLS too`
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
@@ -1659,7 +1668,7 @@ func TestInvalidDNSBLSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
t.Run(tc.name+"="+tc.value, func(t *testing.T) { t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel() t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv) _, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
want := tc.name + ": " + tc.want want := tc.name + ": " + tc.want
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
@@ -1727,7 +1736,7 @@ func TestDNSBLZoneKeyIsLoggedMaskedAndNeverShown(t *testing.T) {
{spamhaus + "," + otherZone, `"` + spamhausMasked + `" is listed twice`}, {spamhaus + "," + otherZone, `"` + spamhausMasked + `" is listed twice`},
{spamhaus + ",,", "has an empty item in its list"}, {spamhaus + ",,", "has an empty item in its list"},
} { } {
_, err := config.FromEnvironment(environment{dnsblZones: tc.value}.lookupEnv) _, err := config.FromEnvironment(environment{dnsblZones: tc.value}.viper())
want := dnsblZones + ": " + tc.want want := dnsblZones + ": " + tc.want
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
@@ -1780,7 +1789,7 @@ func TestInvalidAbuseIPDBSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
t.Run(tc.name+"="+tc.value, func(t *testing.T) { t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel() t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv) _, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
want := tc.name + ": " + tc.want want := tc.name + ": " + tc.want
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
@@ -1894,7 +1903,7 @@ func TestInvalidCrowdSecSettingStopsTheStartSayingWhatIsWrong(t *testing.T) {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
t.Parallel() t.Parallel()
_, err := config.FromEnvironment(tc.env.lookupEnv) _, err := config.FromEnvironment(tc.env.viper())
if err == nil || err.Error() != tc.want { if err == nil || err.Error() != tc.want {
t.Errorf("error %v, want %s", err, tc.want) t.Errorf("error %v, want %s", err, tc.want)
} }
@@ -2054,7 +2063,7 @@ func wantStartStopped(t *testing.T, invalid []struct{ name, value string }) {
t.Run(tc.name+"="+tc.value, func(t *testing.T) { t.Run(tc.name+"="+tc.value, func(t *testing.T) {
t.Parallel() t.Parallel()
_, err := config.FromEnvironment(environment{tc.name: tc.value}.lookupEnv) _, err := config.FromEnvironment(environment{tc.name: tc.value}.viper())
if err == nil { if err == nil {
t.Fatalf("%s=%q was accepted", tc.name, tc.value) t.Fatalf("%s=%q was accepted", tc.name, tc.value)
} }
@@ -2079,7 +2088,7 @@ func TestHostOrTransferEncodingStopsTheStart(t *testing.T) {
t.Run(value, func(t *testing.T) { t.Run(value, func(t *testing.T) {
t.Parallel() t.Parallel()
_, err := config.FromEnvironment(environment{logRequestHeaders: value}.lookupEnv) _, err := config.FromEnvironment(environment{logRequestHeaders: value}.viper())
if err == nil || err.Error() != logRequestHeaders+": "+want { if err == nil || err.Error() != logRequestHeaders+": "+want {
t.Errorf("error %v, want %s: %s", err, logRequestHeaders, want) t.Errorf("error %v, want %s: %s", err, logRequestHeaders, want)
} }
@@ -2096,7 +2105,7 @@ func TestShortTokenStopsTheStartWithoutShowingIt(t *testing.T) {
t.Run(name+"="+value, func(t *testing.T) { t.Run(name+"="+value, func(t *testing.T) {
t.Parallel() t.Parallel()
_, err := config.FromEnvironment(environment{name: value}.lookupEnv) _, err := config.FromEnvironment(environment{name: value}.viper())
want := name + ": is shorter than 32 characters" want := name + ": is shorter than 32 characters"
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
@@ -2152,7 +2161,7 @@ func TestSettingFromFileIsCheckedAsTheSettingItself(t *testing.T) {
_, err := config.FromEnvironment(environment{ _, err := config.FromEnvironment(environment{
bytesLimitPerMinute + "_FILE": writeFile(t, "lots\n"), bytesLimitPerMinute + "_FILE": writeFile(t, "lots\n"),
}.lookupEnv) }.viper())
want := bytesLimitPerMinute + `: "lots" is not a size such as 512K, 100M or 5G, or off` want := bytesLimitPerMinute + `: "lots" is not a size such as 512K, 100M or 5G, or off`
if err == nil || err.Error() != want { if err == nil || err.Error() != want {
@@ -2163,7 +2172,7 @@ func TestSettingFromFileIsCheckedAsTheSettingItself(t *testing.T) {
logRemoteURL: remoteURL, logRemoteURL: remoteURL,
instanceName: instance, instanceName: instance,
logRemoteAppName + "_FILE": writeFile(t, "my app\n"), logRemoteAppName + "_FILE": writeFile(t, "my app\n"),
}.lookupEnv) }.viper())
want = logRemoteAppName + `: "my app" is not 1 to 48 printable ASCII ` + want = logRemoteAppName + `: "my app" is not 1 to 48 printable ASCII ` +
`characters without a space, such as blog` `characters without a space, such as blog`
@@ -2178,7 +2187,7 @@ func TestSettingAndItsFileBothSetStopsTheStart(t *testing.T) {
_, err := config.FromEnvironment(environment{ _, err := config.FromEnvironment(environment{
metricsToken: token, metricsToken: token,
metricsToken + "_FILE": writeFile(t, token), metricsToken + "_FILE": writeFile(t, token),
}.lookupEnv) }.viper())
want := metricsToken + ": is set, and so is " + metricsToken + want := metricsToken + ": is set, and so is " + metricsToken +
"_FILE; set only one of them" "_FILE; set only one of them"
@@ -2193,7 +2202,7 @@ func TestUnreadableSettingFileStopsTheStart(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
for _, path := range []string{filepath.Join(dir, "missing"), dir} { for _, path := range []string{filepath.Join(dir, "missing"), dir} {
_, err := config.FromEnvironment(environment{metricsToken + "_FILE": path}.lookupEnv) _, err := config.FromEnvironment(environment{metricsToken + "_FILE": path}.viper())
want := metricsToken + "_FILE: cannot be read: " want := metricsToken + "_FILE: cannot be read: "
if err == nil || !strings.HasPrefix(err.Error(), want) { if err == nil || !strings.HasPrefix(err.Error(), want) {
@@ -2327,6 +2336,7 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
wafDisabledRules: defaultWAFDisabledRules, wafDisabledRules: defaultWAFDisabledRules,
wafExemptPaths: "", wafExemptPaths: "",
wafBodyLimit: off, wafBodyLimit: off,
exclusionsFile: "",
trapPaths: "", trapPaths: "",
errorBurstThreshold: "30", errorBurstThreshold: "30",
logRemoteURL: "", logRemoteURL: "",
+12 -24
View File
@@ -8,8 +8,9 @@ import (
"net/netip" "net/netip"
"net/url" "net/url"
"strconv" "strconv"
"strings"
"time" "time"
"github.com/spf13/viper"
) )
// RProxy is the settings of `smallwebwaf rproxy`. A size or time limit of zero // RProxy is the settings of `smallwebwaf rproxy`. A size or time limit of zero
@@ -58,13 +59,14 @@ var (
"is not a whole number of milliseconds above zero, such as 60000, or off") "is not a whole number of milliseconds above zero, such as 60000, or off")
) )
// RProxyFromEnvironment reads rproxy's settings with lookupEnv, normally // RProxyFromEnvironment reads rproxy's settings through v, normally the viper
// os.LookupEnv. A setting that is not set takes its default, and // NewViper returns, bound to the process's environment. A setting that is not
// SWRP_WAF_BACKEND, which has none, must be set. A setting that is set but // set takes its default, and SWRP_WAF_BACKEND, which has none, must be set. A
// invalid is an error that names it. Since rproxy never touches disk, its // setting may instead be given as a file: the variable named by the setting's
// settings have no _FILE form. // name with _FILE added names the file, which is read now. A setting that is
func RProxyFromEnvironment(lookupEnv func(string) (string, bool)) (*RProxy, error) { // set but invalid is an error that names it.
env := &environment{lookupEnv: withoutFileForms(lookupEnv)} func RProxyFromEnvironment(v *viper.Viper) (*RProxy, error) {
env := &environment{v: v, readFiles: true}
cfg := &RProxy{ cfg := &RProxy{
ListenAddr: env.address("SWRP_LISTEN_ADDR"), ListenAddr: env.address("SWRP_LISTEN_ADDR"),
UpstreamURL: env.appURL("SWRP_UPSTREAM_URL", defaultUpstreamURL), UpstreamURL: env.appURL("SWRP_UPSTREAM_URL", defaultUpstreamURL),
@@ -91,9 +93,9 @@ func RProxyFromEnvironment(lookupEnv func(string) (string, bool)) (*RProxy, erro
// SWRP_UPSTREAM_URL. The health check in rproxy's container needs no other // SWRP_UPSTREAM_URL. The health check in rproxy's container needs no other
// setting, so it reads no other. // setting, so it reads no other.
func RProxyListenAddrAndUpstreamURL( func RProxyListenAddrAndUpstreamURL(
lookupEnv func(string) (string, bool), v *viper.Viper,
) (string, *url.URL, error) { ) (string, *url.URL, error) {
env := &environment{lookupEnv: withoutFileForms(lookupEnv)} env := &environment{v: v, readFiles: true}
listenAddr := env.address("SWRP_LISTEN_ADDR") listenAddr := env.address("SWRP_LISTEN_ADDR")
upstreamURL := env.appURL("SWRP_UPSTREAM_URL", defaultUpstreamURL) upstreamURL := env.appURL("SWRP_UPSTREAM_URL", defaultUpstreamURL)
@@ -104,20 +106,6 @@ func RProxyListenAddrAndUpstreamURL(
return listenAddr, upstreamURL, nil return listenAddr, upstreamURL, nil
} }
// withoutFileForms returns lookupEnv finding no setting named with _FILE, since
// rproxy never touches disk.
func withoutFileForms(
lookupEnv func(string) (string, bool),
) func(string) (string, bool) {
return func(name string) (string, bool) {
if strings.HasSuffix(name, "_FILE") {
return "", false
}
return lookupEnv(name)
}
}
// LogValue makes an RProxy log as each setting's name with the value it was // LogValue makes an RProxy log as each setting's name with the value it was
// given, or its default. // given, or its default.
func (c *RProxy) LogValue() slog.Value { func (c *RProxy) LogValue() slog.Value {
+31 -18
View File
@@ -9,6 +9,8 @@ import (
"testing" "testing"
"time" "time"
"github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
) )
@@ -32,11 +34,12 @@ const (
// rproxyConfig reads rproxy's settings from env. // rproxyConfig reads rproxy's settings from env.
func rproxyConfig(env map[string]string) (*config.RProxy, error) { func rproxyConfig(env map[string]string) (*config.RProxy, error) {
return config.RProxyFromEnvironment(func(name string) (string, bool) { v := viper.New()
value, ok := env[name] for name, value := range env {
v.Set(name, value)
}
return value, ok return config.RProxyFromEnvironment(v)
})
} }
func TestRProxyDefaults(t *testing.T) { func TestRProxyDefaults(t *testing.T) {
@@ -177,31 +180,41 @@ func TestRProxyNeedsItsBackend(t *testing.T) {
} }
} }
func TestRProxyReadsNoFile(t *testing.T) { func TestRProxySettingReadFromFile(t *testing.T) {
t.Parallel() t.Parallel()
path := filepath.Join(t.TempDir(), "backend") path := filepath.Join(t.TempDir(), "backend")
err := os.WriteFile(path, []byte("http://from-a-file:8080\n"), 0o600) err := os.WriteFile(path, []byte(backendURL+"\n"), 0o600)
if err != nil { if err != nil {
t.Fatalf("write: %v", err) t.Fatalf("write: %v", err)
} }
_, err = rproxyConfig(map[string]string{wafBackend + "_FILE": path}) cfg, err := rproxyConfig(map[string]string{wafBackend + "_FILE": path})
if err == nil || !strings.HasPrefix(err.Error(), wafBackend+": is required") {
t.Errorf("error %v, want %s still required", err, wafBackend)
}
cfg, err := rproxyConfig(map[string]string{
wafBackend: backendURL,
rproxyTimeout + "_FILE": "/nonexistent/timeout",
})
if err != nil { if err != nil {
t.Fatalf("settings: %v, with the file left unread", err) t.Fatalf("settings: %v", err)
} }
if cfg.Timeout != 2*time.Second { if cfg.WAFBackend.String() != backendURL {
t.Errorf("timeout %v, want the default", cfg.Timeout) t.Errorf("backend %v, want %s", cfg.WAFBackend, backendURL)
}
}
func TestRProxyUnreadableSettingFileStopsTheStart(t *testing.T) {
t.Parallel()
dir := t.TempDir()
for _, path := range []string{filepath.Join(dir, "missing"), dir} {
_, err := rproxyConfig(map[string]string{
wafBackend: backendURL,
rproxyTimeout + "_FILE": path,
})
want := rproxyTimeout + "_FILE: cannot be read: "
if err == nil || !strings.HasPrefix(err.Error(), want) {
t.Errorf("%s: error %v, want one starting %s", path, err, want)
}
} }
} }
+16
View File
@@ -0,0 +1,16 @@
package config
import "github.com/spf13/viper"
// NewViper returns the viper that reads the settings: each setting's
// environment variable, bound to its own name. Every setting of servewaf and
// rproxy is read through it, and none is read with os.Getenv or os.LookupEnv
// outside it. An environment variable that is set but empty is set, as one
// that holds a value is, which viper does not take for granted.
func NewViper() *viper.Viper {
v := viper.New()
v.AutomaticEnv()
v.AllowEmptyEnv(true)
return v
}
+3 -7
View File
@@ -33,13 +33,9 @@ func announcedLength(size int) string {
func TestServerHasTheFixedLimits(t *testing.T) { func TestServerHasTheFixedLimits(t *testing.T) {
t.Parallel() t.Parallel()
cfg, err := config.RProxyFromEnvironment(func(name string) (string, bool) { cfg, err := config.RProxyFromEnvironment(viperFrom(map[string]string{
if name == "SWRP_WAF_BACKEND" { "SWRP_WAF_BACKEND": nowhere,
return nowhere, true }))
}
return "", false
})
if err != nil { if err != nil {
t.Fatalf("settings: %v", err) t.Fatalf("settings: %v", err)
} }
+14 -5
View File
@@ -18,6 +18,8 @@ import (
"testing" "testing"
"time" "time"
"github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy" "sneak.berlin/go/smallwebwaf/internal/rproxy"
@@ -341,6 +343,17 @@ func startServer(
return addr, out, server return addr, out, server
} }
// viperFrom returns a viper Set with each of settings, as the viper
// config.NewViper returns one bound to a process's environment.
func viperFrom(settings map[string]string) *viper.Viper {
v := viper.New()
for name, value := range settings {
v.Set(name, value)
}
return v
}
// startServerLogging is startServer with rproxy's request lines going to // startServerLogging is startServer with rproxy's request lines going to
// requestLog and its process lines to processLog. // requestLog and its process lines to processLog.
func startServerLogging( func startServerLogging(
@@ -354,11 +367,7 @@ func startServerLogging(
} }
maps.Copy(settings, env) maps.Copy(settings, env)
cfg, err := config.RProxyFromEnvironment(func(name string) (string, bool) { cfg, err := config.RProxyFromEnvironment(viperFrom(settings))
value, ok := settings[name]
return value, ok
})
if err != nil { if err != nil {
t.Fatalf("settings %v: %v", settings, err) t.Fatalf("settings %v: %v", settings, err)
} }
+22 -13
View File
@@ -4,6 +4,7 @@
package servewaf package servewaf
import ( import (
"fmt"
"io" "io"
"log/slog" "log/slog"
"net/http" "net/http"
@@ -60,6 +61,9 @@ type Params struct {
// LookupFile is the lookup database they are looked up in while // LookupFile is the lookup database they are looked up in while
// SWWAF_LOOKUP_SOURCE is file, and nil otherwise. // SWWAF_LOOKUP_SOURCE is file, and nil otherwise.
LookupFile *lookup.File LookupFile *lookup.File
// CoreRuleSet is what NewCoreRuleSet returns: what inspects requests
// while SWWAF_WAF_MODE is not off, and nil while it is.
CoreRuleSet *waf.CoreRuleSet
// Now tells the time by which requests are counted for the rate // Now tells the time by which requests are counted for the rate
// limits, bans are made and run out, and GeoJS's answers are kept, // limits, bans are made and run out, and GeoJS's answers are kept,
// normally time.Now in UTC, the time the state files give. // normally time.Now in UTC, the time the state files give.
@@ -144,7 +148,7 @@ func newHandler(params Params) *handler {
dnsbl: dnsbl, dnsbl: dnsbl,
abuseIPDB: abuseIPDB, abuseIPDB: abuseIPDB,
rules: params.Rules, rules: params.Rules,
coreRuleSet: newCoreRuleSet(params.Config), coreRuleSet: params.CoreRuleSet,
alerts: params.Alerts, alerts: params.Alerts,
} }
h.geojs = lookup.New(lookup.Params{ h.geojs = lookup.New(lookup.Params{
@@ -202,29 +206,34 @@ func newReputation(
return lists, dnsbl, abuseIPDB return lists, dnsbl, abuseIPDB
} }
// newCoreRuleSet returns the Core Rule Set at SWWAF_WAF_PARANOIA_LEVEL, // NewCoreRuleSet returns the Core Rule Set at SWWAF_WAF_PARANOIA_LEVEL,
// without the rules SWWAF_WAF_DISABLED_RULES switches off, reading bodies // without the rules SWWAF_WAF_DISABLED_RULES switches off, reading bodies
// up to SWWAF_WAF_BODY_LIMIT, or nil while SWWAF_WAF_MODE is off. It reads at // up to SWWAF_WAF_BODY_LIMIT, with the exclusions in
// most QuestionBodyMaxBytes less one byte: a question's body of // SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE, or nil while SWWAF_WAF_MODE is off. It
// reads at most QuestionBodyMaxBytes less one byte: a question's body of
// QuestionBodyMaxBytes may be cut short, and the Core Rule Set does not read a // QuestionBodyMaxBytes may be cut short, and the Core Rule Set does not read a
// JSON or XML body over its limit. // JSON or XML body over its limit. An exclusions file that waf.New refuses is
func newCoreRuleSet(cfg *config.Config) *waf.CoreRuleSet { // an error in every mode. While the mode is off, a set file is loaded with the
if cfg.WAFMode == config.WAFModeOff { // Core Rule Set only to check it.
return nil func NewCoreRuleSet(cfg *config.Config) (*waf.CoreRuleSet, error) {
if cfg.WAFMode == config.WAFModeOff && cfg.CoreRuleSetExclusionsFile == "" {
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
} }
coreRuleSet, err := waf.New(waf.Params{ coreRuleSet, err := waf.New(waf.Params{
ParanoiaLevel: cfg.WAFParanoiaLevel, DisabledRules: cfg.WAFDisabledRules, ParanoiaLevel: cfg.WAFParanoiaLevel, DisabledRules: cfg.WAFDisabledRules,
BodyLimit: min(cfg.WAFBodyLimit, QuestionBodyMaxBytes-1), BodyLimit: min(cfg.WAFBodyLimit, QuestionBodyMaxBytes-1),
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
}) })
if err != nil { if err != nil {
// The Core Rule Set is built in, and the settings cannot break it: return nil, fmt.Errorf("SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE: %w", err)
// the paranoia level is from 1 to 4, and the id of no rule switches
// nothing off.
panic(err)
} }
return coreRuleSet if cfg.WAFMode == config.WAFModeOff {
return nil, nil //nolint:nilnil // off is no Core Rule Set, and no error
}
return coreRuleSet, nil
} }
// handler is servewaf's handler. It holds what every request shares; what // handler is servewaf's handler. It holds what every request shares; what
+21 -10
View File
@@ -15,6 +15,8 @@ import (
"testing" "testing"
"time" "time"
"github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
@@ -347,6 +349,17 @@ func serve(t *testing.T, server *servewaf.Server) {
// at serveWAFAddr, with the settings in env on top of the defaults, and returns // at serveWAFAddr, with the settings in env on top of the defaults, and returns
// where it listens. What it writes is dropped. SWRP_TIMEOUT_MS is a minute unless // where it listens. What it writes is dropped. SWRP_TIMEOUT_MS is a minute unless
// env sets it, so that on a busy host SWRP_DEFAULT_DISPOSITION does not decide. // env sets it, so that on a busy host SWRP_DEFAULT_DISPOSITION does not decide.
// viperFrom returns a viper Set with each of settings, as the viper
// config.NewViper returns one bound to a process's environment.
func viperFrom(settings map[string]string) *viper.Viper {
v := viper.New()
for name, value := range settings {
v.Set(name, value)
}
return v
}
func startRProxy( func startRProxy(
t *testing.T, serveWAFAddr, appURL string, env map[string]string, t *testing.T, serveWAFAddr, appURL string, env map[string]string,
) string { ) string {
@@ -359,11 +372,7 @@ func startRProxy(
} }
maps.Copy(settings, env) maps.Copy(settings, env)
cfg, err := config.RProxyFromEnvironment(func(name string) (string, bool) { cfg, err := config.RProxyFromEnvironment(viperFrom(settings))
value, ok := settings[name]
return value, ok
})
if err != nil { if err != nil {
t.Fatalf("settings %v: %v", settings, err) t.Fatalf("settings %v: %v", settings, err)
} }
@@ -423,11 +432,7 @@ func newServeWAFWith(
maps.Copy(settings, env) maps.Copy(settings, env)
cfg, err := config.FromEnvironment(func(name string) (string, bool) { cfg, err := config.FromEnvironment(viperFrom(settings))
value, ok := settings[name]
return value, ok
})
if err != nil { if err != nil {
t.Fatalf("settings %v: %v", settings, err) t.Fatalf("settings %v: %v", settings, err)
} }
@@ -463,6 +468,11 @@ func newServeWAFWith(
} }
} }
coreRuleSet, err := servewaf.NewCoreRuleSet(cfg)
if err != nil {
t.Fatalf("Core Rule Set: %v", err)
}
server := build(servewaf.Params{ server := build(servewaf.Params{
Config: cfg, Config: cfg,
RequestLog: out, RequestLog: out,
@@ -470,6 +480,7 @@ func newServeWAFWith(
GeoJSURL: geojsURL, GeoJSURL: geojsURL,
AbuseIPDBURL: abuseIPDBURL, AbuseIPDBURL: abuseIPDBURL,
LookupFile: lookupFile, LookupFile: lookupFile,
CoreRuleSet: coreRuleSet,
Now: now, Now: now,
Rules: ruleFiles, Rules: ruleFiles,
Alerts: alertQueue, Alerts: alertQueue,
+10 -8
View File
@@ -10,6 +10,8 @@ import (
"net/url" "net/url"
"time" "time"
"github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/servewaf" "sneak.berlin/go/smallwebwaf/internal/servewaf"
) )
@@ -31,7 +33,7 @@ var errHealthEndpoint = errors.New("the health endpoint answered")
// args are the arguments after `healthcheck`; it takes none, and given // args are the arguments after `healthcheck`; it takes none, and given
// one it names it on stderr and returns 1 without checking anything. // one it names it on stderr and returns 1 without checking anything.
func HealthCheck( func HealthCheck(
ctx context.Context, args []string, lookupEnv func(string) (string, bool), ctx context.Context, args []string, v *viper.Viper,
stderr io.Writer, stderr io.Writer,
) int { ) int {
if len(args) > 0 { if len(args) > 0 {
@@ -44,15 +46,15 @@ func HealthCheck(
ctx, cancel := context.WithTimeout(ctx, healthCheckTimeout) ctx, cancel := context.WithTimeout(ctx, healthCheckTimeout)
defer cancel() defer cancel()
command, err := config.Command(lookupEnv) command, err := config.Command(v)
switch { switch {
case err != nil: case err != nil:
err = fmt.Errorf("invalid setting: %w", err) err = fmt.Errorf("invalid setting: %w", err)
case command == "servewaf": case command == "servewaf":
err = checkServeWAF(ctx, lookupEnv) err = checkServeWAF(ctx, v)
default: default:
err = checkRProxyAndApp(ctx, lookupEnv) err = checkRProxyAndApp(ctx, v)
} }
if err != nil { if err != nil {
@@ -65,8 +67,8 @@ func HealthCheck(
} }
// checkServeWAF asks servewaf's health endpoint. // checkServeWAF asks servewaf's health endpoint.
func checkServeWAF(ctx context.Context, lookupEnv func(string) (string, bool)) error { func checkServeWAF(ctx context.Context, v *viper.Viper) error {
listenAddr, err := config.ServeWAFListenAddr(lookupEnv) listenAddr, err := config.ServeWAFListenAddr(v)
if err != nil { if err != nil {
return fmt.Errorf("invalid setting: %w", err) return fmt.Errorf("invalid setting: %w", err)
} }
@@ -81,9 +83,9 @@ func checkServeWAF(ctx context.Context, lookupEnv func(string) (string, bool)) e
// checkRProxyAndApp asks rproxy's health endpoint, and connects to the app. // checkRProxyAndApp asks rproxy's health endpoint, and connects to the app.
func checkRProxyAndApp( func checkRProxyAndApp(
ctx context.Context, lookupEnv func(string) (string, bool), ctx context.Context, v *viper.Viper,
) error { ) error {
listenAddr, upstreamURL, err := config.RProxyListenAddrAndUpstreamURL(lookupEnv) listenAddr, upstreamURL, err := config.RProxyListenAddrAndUpstreamURL(v)
if err != nil { if err != nil {
return fmt.Errorf("invalid setting: %w", err) return fmt.Errorf("invalid setting: %w", err)
} }
+5 -12
View File
@@ -125,11 +125,12 @@ func TestHealthCheckInRProxysContainer(t *testing.T) {
wantHealthCheck(t, settings(rproxyListenAddr, "8080"), 1, wantHealthCheck(t, settings(rproxyListenAddr, "8080"), 1,
"unhealthy: invalid setting: SWRP_LISTEN_ADDR: ") "unhealthy: invalid setting: SWRP_LISTEN_ADDR: ")
// No setting of rproxy's has a form given as a file. // A setting of rproxy's can be given as a file too, here the listen
// address, which the health check reads.
wantHealthCheck(t, map[string]string{ wantHealthCheck(t, map[string]string{
rproxyListenAddr + "_FILE": writeFile(t, ":"+port+"\n"), rproxyListenAddr + "_FILE": writeFile(t, ":"+port+"\n"),
rproxyUpstreamURL: app.URL, rproxyUpstreamURL: app.URL,
}, 1, "unhealthy: ask rproxy: ") }, 0, "")
} }
func TestHealthCheckFailsWithAnyOtherCommand(t *testing.T) { func TestHealthCheckFailsWithAnyOtherCommand(t *testing.T) {
@@ -145,11 +146,7 @@ func TestHealthCheckRefusesAnArgument(t *testing.T) {
var stderr bytes.Buffer var stderr bytes.Buffer
noSettings := func(string) (string, bool) { got := smallwebwaf.HealthCheck(t.Context(), []string{"now"}, viperFrom(nil), &stderr)
return "", false
}
got := smallwebwaf.HealthCheck(t.Context(), []string{"now"}, noSettings, &stderr)
want := "smallwebwaf healthcheck: unexpected argument \"now\"\n" want := "smallwebwaf healthcheck: unexpected argument \"now\"\n"
if got != 1 || stderr.String() != want { if got != 1 || stderr.String() != want {
@@ -166,11 +163,7 @@ func wantHealthCheck(t *testing.T, env map[string]string, status int, message st
var stderr bytes.Buffer var stderr bytes.Buffer
got := smallwebwaf.HealthCheck(t.Context(), nil, func(name string) (string, bool) { got := smallwebwaf.HealthCheck(t.Context(), nil, viperFrom(env), &stderr)
value, ok := env[name]
return value, ok
}, &stderr)
wrote := stderr.String() wrote := stderr.String()
if got != status || !strings.HasPrefix(wrote, message) || if got != status || !strings.HasPrefix(wrote, message) ||
+1 -1
View File
@@ -23,7 +23,7 @@ func RunRProxy(ctx context.Context, params Params) int {
hostname, _ := os.Hostname() // "" when the host has no name to give hostname, _ := os.Hostname() // "" when the host has no name to give
processLog := requestlog.NewProcessLogger(params.Stdout, hostname, slog.LevelInfo) processLog := requestlog.NewProcessLogger(params.Stdout, hostname, slog.LevelInfo)
cfg, err := config.RProxyFromEnvironment(params.LookupEnv) cfg, err := config.RProxyFromEnvironment(params.Viper)
if err != nil { if err != nil {
processLog.Error("invalid setting", "error", err.Error()) processLog.Error("invalid setting", "error", err.Error())
+1 -5
View File
@@ -43,11 +43,7 @@ const (
func runRProxy(ctx context.Context, env map[string]string, out *output) int { func runRProxy(ctx context.Context, env map[string]string, out *output) int {
return smallwebwaf.RunRProxy(ctx, smallwebwaf.Params{ return smallwebwaf.RunRProxy(ctx, smallwebwaf.Params{
Version: testVersion, Version: testVersion,
LookupEnv: func(name string) (string, bool) { Viper: viperFrom(env),
value, ok := env[name]
return value, ok
},
Stdout: out, Stdout: out,
}) })
} }
+21 -9
View File
@@ -20,6 +20,8 @@ import (
"syscall" "syscall"
"time" "time"
"github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/lookup" "sneak.berlin/go/smallwebwaf/internal/lookup"
@@ -29,6 +31,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/servewaf" "sneak.berlin/go/smallwebwaf/internal/servewaf"
"sneak.berlin/go/smallwebwaf/internal/state" "sneak.berlin/go/smallwebwaf/internal/state"
"sneak.berlin/go/smallwebwaf/internal/waf"
) )
// shutdownTimeout is how long requests in progress may take to finish // shutdownTimeout is how long requests in progress may take to finish
@@ -58,8 +61,9 @@ commands:
type Params struct { type Params struct {
// Version is the version of the binary, set when it is built. // Version is the version of the binary, set when it is built.
Version string Version string
// LookupEnv reads an environment variable, normally os.LookupEnv. // Viper reads the settings, normally the viper config.NewViper returns
LookupEnv func(string) (string, bool) // for the process's environment.
Viper *viper.Viper
// Stdout receives the request log and the process's own messages. // Stdout receives the request log and the process's own messages.
Stdout io.Writer Stdout io.Writer
} }
@@ -78,7 +82,7 @@ func Main(version string) int {
syscall.SIGTERM, os.Interrupt) syscall.SIGTERM, os.Interrupt)
defer stop() defer stop()
params := Params{Version: version, LookupEnv: os.LookupEnv, Stdout: os.Stdout} params := Params{Version: version, Viper: config.NewViper(), Stdout: os.Stdout}
switch args[0] { switch args[0] {
case "rproxy": case "rproxy":
@@ -86,7 +90,7 @@ func Main(version string) int {
case "servewaf": case "servewaf":
return RunServeWAF(ctx, params) return RunServeWAF(ctx, params)
case "healthcheck": case "healthcheck":
return HealthCheck(ctx, args[1:], os.LookupEnv, os.Stderr) return HealthCheck(ctx, args[1:], params.Viper, os.Stderr)
default: default:
return printUsage(os.Stderr) return printUsage(os.Stderr)
} }
@@ -106,9 +110,9 @@ func RunServeWAF(ctx context.Context, params Params) int {
// Until the settings are read, the one message is an invalid setting's // Until the settings are read, the one message is an invalid setting's
// error, which every SWWAF_LOG_LEVEL lets through. // error, which every SWWAF_LOG_LEVEL lets through.
processLog := requestlog.NewProcessLogger(params.Stdout, processLog := requestlog.NewProcessLogger(params.Stdout,
config.InstanceName(params.LookupEnv), slog.LevelError) config.InstanceName(params.Viper), slog.LevelError)
cfg, err := config.FromEnvironment(params.LookupEnv) cfg, err := config.FromEnvironment(params.Viper)
if err != nil { if err != nil {
processLog.Error("invalid setting", "error", err.Error()) processLog.Error("invalid setting", "error", err.Error())
@@ -150,7 +154,14 @@ func RunServeWAF(ctx context.Context, params Params) int {
return 1 return 1
} }
server, err := newServer(cfg, stdout, processLog, now, ruleFiles, alertQueue) crs, err := servewaf.NewCoreRuleSet(cfg)
if err != nil {
processLog.Error("cannot use the Core Rule Set", "error", err.Error())
return 1
}
server, err := newServer(cfg, stdout, processLog, now, ruleFiles, alertQueue, crs)
if err != nil { if err != nil {
processLog.Error("cannot use the lookup database", "error", err.Error()) processLog.Error("cannot use the lookup database", "error", err.Error())
@@ -170,8 +181,7 @@ func RunServeWAF(ctx context.Context, params Params) int {
listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", cfg.ListenAddr) listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", cfg.ListenAddr)
if err != nil { if err != nil {
processLog.Error("cannot listen on SWWAF_LISTEN_ADDR", processLog.Error("cannot listen on SWWAF_LISTEN_ADDR", "error", err.Error())
"error", err.Error())
return 1 return 1
} }
@@ -190,6 +200,7 @@ func RunServeWAF(ctx context.Context, params Params) int {
func newServer( func newServer(
cfg *config.Config, stdout io.Writer, processLog *slog.Logger, cfg *config.Config, stdout io.Writer, processLog *slog.Logger,
now func() time.Time, ruleFiles *rules.Files, alertQueue *alerts.Queue, now func() time.Time, ruleFiles *rules.Files, alertQueue *alerts.Queue,
crs *waf.CoreRuleSet,
) (*servewaf.Server, error) { ) (*servewaf.Server, error) {
var lookupFile *lookup.File var lookupFile *lookup.File
@@ -214,6 +225,7 @@ func newServer(
GeoJSURL: lookup.URL, GeoJSURL: lookup.URL,
AbuseIPDBURL: reputation.AbuseIPDBURL, AbuseIPDBURL: reputation.AbuseIPDBURL,
LookupFile: lookupFile, LookupFile: lookupFile,
CoreRuleSet: crs,
Now: now, Now: now,
Rules: ruleFiles, Rules: ruleFiles,
Alerts: alertQueue, Alerts: alertQueue,
+70 -8
View File
@@ -19,6 +19,9 @@ import (
"testing" "testing"
"time" "time"
"github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/servewaf" "sneak.berlin/go/smallwebwaf/internal/servewaf"
"sneak.berlin/go/smallwebwaf/internal/smallwebwaf" "sneak.berlin/go/smallwebwaf/internal/smallwebwaf"
@@ -122,20 +125,29 @@ func (o *output) text() string {
return o.buf.String() return o.buf.String()
} }
// viperFrom returns a viper Set with each of settings, as the viper
// config.NewViper returns one bound to a process's environment.
func viperFrom(settings map[string]string) *viper.Viper {
v := viper.New()
for name, value := range settings {
v.Set(name, value)
}
return v
}
// run runs `smallwebwaf servewaf` with the settings in env until ctx is done, // run runs `smallwebwaf servewaf` with the settings in env until ctx is done,
// and returns its exit status. SWWAF_LOOKUP_SOURCE is off unless env sets it, // and returns its exit status. SWWAF_LOOKUP_SOURCE is off unless env sets it,
// so that no test sends GeoJS its clients' addresses. // so that no test sends GeoJS its clients' addresses.
func run(ctx context.Context, env map[string]string, out *output) int { func run(ctx context.Context, env map[string]string, out *output) int {
return smallwebwaf.RunServeWAF(ctx, smallwebwaf.Params{ v := viperFrom(env)
Version: testVersion, if _, set := env["SWWAF_LOOKUP_SOURCE"]; !set {
LookupEnv: func(name string) (string, bool) { v.Set("SWWAF_LOOKUP_SOURCE", "off")
value, ok := env[name]
if !ok && name == "SWWAF_LOOKUP_SOURCE" {
return "off", true
} }
return value, ok return smallwebwaf.RunServeWAF(ctx, smallwebwaf.Params{
}, Version: testVersion,
Viper: v,
Stdout: out, Stdout: out,
}) })
} }
@@ -551,6 +563,56 @@ func TestRulesDirThatDoesNotExistStopsTheStart(t *testing.T) {
": no such file or directory") ": no such file or directory")
} }
func TestCoreRuleSetExclusionsFileThatCannotBeReadStopsTheStart(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "crs-exclusions.conf")
out := &output{}
status := run(t.Context(), map[string]string{
listenAddr: localhost + ":0", stateDir: t.TempDir(), rulesDir: t.TempDir(),
"SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE": path,
}, out)
if status != 1 {
t.Fatalf("exit status %d, want 1", status)
}
line := out.line(t, "msg", "cannot use the Core Rule Set")
want := "SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE: read the exclusions file: open " +
path + ": no such file or directory"
if line["error"] != want {
t.Errorf("start refused with %v, want the error %q", line, want)
}
}
func TestCoreRuleSetExclusionsFileThatCannotBeUsedStopsTheStartWhileTheCoreRuleSetIsOff(
t *testing.T,
) {
t.Parallel()
broken := filepath.Join(t.TempDir(), "broken.conf")
err := os.WriteFile(broken, []byte("SecNoSuchDirective On\n"), 0o600)
if err != nil {
t.Fatal(err)
}
for _, path := range []string{filepath.Join(t.TempDir(), "missing.conf"), broken} {
out := &output{}
status := run(t.Context(), map[string]string{
listenAddr: localhost + ":0", stateDir: t.TempDir(), rulesDir: t.TempDir(),
"SWWAF_WAF_MODE": config.WAFModeOff, "SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE": path,
}, out)
if status != 1 {
t.Fatalf("%s: exit status %d, want 1", path, status)
}
out.line(t, "msg", "cannot use the Core Rule Set")
}
}
func TestLookupDatabaseReplacedWhileRunningTakesEffect(t *testing.T) { func TestLookupDatabaseReplacedWhileRunningTakesEffect(t *testing.T) {
t.Parallel() t.Parallel()
+283
View File
@@ -0,0 +1,283 @@
# The eighth change: on gitea's paths, some fields skip the rules that refused
# ordinary gitea requests there. They are the fields where gitea sends a search,
# a file name or a branch name. Each exclusion runs with the request, before the
# rules it switches off, those of phase 1 included, so this file is loaded before
# the Core Rule Set's rules. Each comment names a request the rule refused.
# gitea's searches of code, of issues and pull requests, and of commits send the
# text searched for in q, which skips these rules there:
# 930110: ../
# 930120: .gitignore
# 931100: http://127.0.0.1:8080
# 932120: Invoke-WebRequest
# 932130: $(pwd)
# 932140: if (a == b) {
# 932160: /bin/sh
# 932235: `make check`
# 932250: ls -la
# 932260: python3
# 932380: `whoami`
# 933100: <?php echo
# 933130: $_GET[
# 933135: ${VAR}
# 934100: process.env
# 934110: http://127.0.0.1:8080
# 934130: __proto__
# 941100: <script>
# 941110: <script>
# 941160: <img src=
# 941180: document.cookie
# 941390: fetch(
# 942100: SELECT * FROM users WHERE
# 942140: information_schema
# 942190: union select
# 942230: if (a == b) {
# 944100: java.lang.Runtime
# 944110: java.lang.Runtime
# 944130: java.lang.Runtime
SecRule REQUEST_FILENAME "@rx ^/(?:explore/code|issues(?:/search)?|pulls|\
[^/]+/-/code|\
[^/]+/[^/]+/(?:search|issues(?:/search)?|pulls|\
commits/(?:branch|tag|commit)/.+/search)|\
api/v1/repos/(?:issues/search|[^/]+/[^/]+/issues))$" \
"id:900600,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=930110;ARGS:q,\
ctl:ruleRemoveTargetById=930120;ARGS:q,\
ctl:ruleRemoveTargetById=931100;ARGS:q,\
ctl:ruleRemoveTargetById=932120;ARGS:q,\
ctl:ruleRemoveTargetById=932130;ARGS:q,\
ctl:ruleRemoveTargetById=932140;ARGS:q,\
ctl:ruleRemoveTargetById=932160;ARGS:q,\
ctl:ruleRemoveTargetById=932235;ARGS:q,\
ctl:ruleRemoveTargetById=932250;ARGS:q,\
ctl:ruleRemoveTargetById=932260;ARGS:q,\
ctl:ruleRemoveTargetById=932380;ARGS:q,\
ctl:ruleRemoveTargetById=933100;ARGS:q,\
ctl:ruleRemoveTargetById=933130;ARGS:q,\
ctl:ruleRemoveTargetById=933135;ARGS:q,\
ctl:ruleRemoveTargetById=934100;ARGS:q,\
ctl:ruleRemoveTargetById=934110;ARGS:q,\
ctl:ruleRemoveTargetById=934130;ARGS:q,\
ctl:ruleRemoveTargetById=941100;ARGS:q,\
ctl:ruleRemoveTargetById=941110;ARGS:q,\
ctl:ruleRemoveTargetById=941160;ARGS:q,\
ctl:ruleRemoveTargetById=941180;ARGS:q,\
ctl:ruleRemoveTargetById=941390;ARGS:q,\
ctl:ruleRemoveTargetById=942100;ARGS:q,\
ctl:ruleRemoveTargetById=942140;ARGS:q,\
ctl:ruleRemoveTargetById=942190;ARGS:q,\
ctl:ruleRemoveTargetById=942230;ARGS:q,\
ctl:ruleRemoveTargetById=944100;ARGS:q,\
ctl:ruleRemoveTargetById=944110;ARGS:q,\
ctl:ruleRemoveTargetById=944130;ARGS:q"
# A page opened from the results of one of those searches has the search's
# address in Referer, which skips these rules then:
# 941110: <script>
# 941120: onClick={
# 941160: <img src=
# 942100: LIKE '%foo%'
# 944100: java.lang.Runtime
SecRule REQUEST_HEADERS:Referer "@rx ^https?://[^/]+/(?:explore/code|\
issues(?:/search)?|pulls|[^/]+/-/code|\
[^/]+/[^/]+/(?:search|issues(?:/search)?|pulls|\
commits/(?:branch|tag|commit)/.+/search)|\
api/v1/repos/(?:issues/search|[^/]+/[^/]+/issues))\?" \
"id:900601,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=941110;REQUEST_HEADERS:Referer,\
ctl:ruleRemoveTargetById=941120;REQUEST_HEADERS:Referer,\
ctl:ruleRemoveTargetById=941160;REQUEST_HEADERS:Referer,\
ctl:ruleRemoveTargetById=942100;REQUEST_HEADERS:Referer,\
ctl:ruleRemoveTargetById=944100;REQUEST_HEADERS:Referer"
# gitea's searches of repositories, users, branches, tags and packages send a
# name in q, which skips these rules there:
# 930120: .profile
# 932120: powershell-scripts
# 932260: python3-app
# 944130: org.apache.commons.logging
SecRule REQUEST_FILENAME "@rx ^/(?:explore/repos|user/search_candidates|\
[^/]+/-/packages|[^/]+/[^/]+/(?:branches|tags)|api/v1/repos/search)$" \
"id:900602,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=930120;ARGS:q,\
ctl:ruleRemoveTargetById=932120;ARGS:q,\
ctl:ruleRemoveTargetById=932260;ARGS:q,\
ctl:ruleRemoveTargetById=944130;ARGS:q"
# gitea's other searches for a name send it in q too, which skips only these
# rules there:
# 930120: .profile
# 932120: powershell-scripts
# 932260: python3-app
# The searches:
# - the dashboard's filter of repositories;
# - the repositories on a user's or an organisation's page, whose path has
# one segment;
# - the author filter of the issue and pull request lists;
# - the lists of users and organisations, and the API's search for users.
SecRule REQUEST_FILENAME "@rx ^/(?:[^/]+|repo/search|\
explore/(?:users|organizations)|api/v1/users/search|\
[^/]+/[^/]+/(?:issues|pulls)/posters)$" \
"id:900603,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=930120;ARGS:q,\
ctl:ruleRemoveTargetById=932120;ARGS:q,\
ctl:ruleRemoveTargetById=932260;ARGS:q"
# These requests name a file in the path, which skips these rules there:
# 920500: notes.txt~
# 934100: process.env.d.ts
# 941130: docs/a b.xhtml
# 944130: java.lang.String.md
# The requests:
# - a file's page, raw file and download;
# - its blame, history, edit, preview and delete pages;
# - the API's call for its contents;
# - a release attachment's download.
SecRule REQUEST_FILENAME \
"@rx ^/(?:api/v1/repos/)?[^/]+/[^/]+/(?:src|raw|media|blame|commits|_edit|_preview|_delete|contents|releases/download)/" \
"id:900610,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=920500;REQUEST_FILENAME,\
ctl:ruleRemoveTargetById=934100;REQUEST_FILENAME,\
ctl:ruleRemoveTargetById=941130;REQUEST_FILENAME,\
ctl:ruleRemoveTargetById=944130;REQUEST_FILENAME"
# The requests below name a file in a query parameter. These rules refused
# them for the file named:
# 930120: .gitignore
# 932120: install-script.sh
# 932130: [draft].md
# 932160: bin/docker-entrypoint
# 932260: docker-compose.yml
# 934100: process.env.d.ts
# 934130: __proto__.json
# 941130: index.xhtml
# 941180: document.write.js
# 942140: information_schema.sql
# 944110: java/lang/Runtime.java
# 944130: java.lang.String.md
# Each request skips the rules that refused it. The fourth change already has
# path and redirect_to skip 930120, 932160 and 932260 on every path.
# A file's commit graph, in file.
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/graph$" \
"id:900620,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=930120;ARGS:file,\
ctl:ruleRemoveTargetById=932120;ARGS:file,\
ctl:ruleRemoveTargetById=932160;ARGS:file,\
ctl:ruleRemoveTargetById=932260;ARGS:file,\
ctl:ruleRemoveTargetById=934100;ARGS:file,\
ctl:ruleRemoveTargetById=934130;ARGS:file,\
ctl:ruleRemoveTargetById=941130;ARGS:file,\
ctl:ruleRemoveTargetById=941180;ARGS:file,\
ctl:ruleRemoveTargetById=942140;ARGS:file,\
ctl:ruleRemoveTargetById=944110;ARGS:file,\
ctl:ruleRemoveTargetById=944130;ARGS:file"
# The API's list of a file's commits, in path.
SecRule REQUEST_FILENAME "@rx ^/api/v1/repos/[^/]+/[^/]+/commits$" \
"id:900621,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=932120;ARGS:path,\
ctl:ruleRemoveTargetById=934100;ARGS:path,\
ctl:ruleRemoveTargetById=934130;ARGS:path,\
ctl:ruleRemoveTargetById=941130;ARGS:path,\
ctl:ruleRemoveTargetById=941180;ARGS:path,\
ctl:ruleRemoveTargetById=942140;ARGS:path,\
ctl:ruleRemoveTargetById=944110;ARGS:path,\
ctl:ruleRemoveTargetById=944130;ARGS:path"
# The API's call for the contents of several files, in body, as JSON.
SecRule REQUEST_FILENAME "@rx ^/api/v1/repos/[^/]+/[^/]+/file-contents$" \
"id:900622,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=930120;ARGS:body,\
ctl:ruleRemoveTargetById=932120;ARGS:body,\
ctl:ruleRemoveTargetById=932160;ARGS:body,\
ctl:ruleRemoveTargetById=934100;ARGS:body,\
ctl:ruleRemoveTargetById=934130;ARGS:body,\
ctl:ruleRemoveTargetById=941130;ARGS:body,\
ctl:ruleRemoveTargetById=941180;ARGS:body,\
ctl:ruleRemoveTargetById=942140;ARGS:body,\
ctl:ruleRemoveTargetById=944110;ARGS:body,\
ctl:ruleRemoveTargetById=944130;ARGS:body"
# An issue opened from a template, in template.
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/issues/new$" \
"id:900623,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=930120;ARGS:template,\
ctl:ruleRemoveTargetById=932120;ARGS:template,\
ctl:ruleRemoveTargetById=932130;ARGS:template,\
ctl:ruleRemoveTargetById=932160;ARGS:template,\
ctl:ruleRemoveTargetById=932260;ARGS:template,\
ctl:ruleRemoveTargetById=934100;ARGS:template,\
ctl:ruleRemoveTargetById=934130;ARGS:template,\
ctl:ruleRemoveTargetById=941130;ARGS:template,\
ctl:ruleRemoveTargetById=941180;ARGS:template,\
ctl:ruleRemoveTargetById=942140;ARGS:template,\
ctl:ruleRemoveTargetById=944110;ARGS:template,\
ctl:ruleRemoveTargetById=944130;ARGS:template"
# The sign-in page on the way to a file's edit page, in redirect_to.
SecRule REQUEST_FILENAME "@rx ^/user/login$" \
"id:900624,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=932120;ARGS:redirect_to,\
ctl:ruleRemoveTargetById=932130;ARGS:redirect_to,\
ctl:ruleRemoveTargetById=934100;ARGS:redirect_to,\
ctl:ruleRemoveTargetById=934130;ARGS:redirect_to,\
ctl:ruleRemoveTargetById=941130;ARGS:redirect_to,\
ctl:ruleRemoveTargetById=941180;ARGS:redirect_to,\
ctl:ruleRemoveTargetById=942140;ARGS:redirect_to,\
ctl:ruleRemoveTargetById=944110;ARGS:redirect_to,\
ctl:ruleRemoveTargetById=944130;ARGS:redirect_to"
# An attachment uploaded through the API to a release, an issue or a comment,
# in name.
SecRule REQUEST_FILENAME \
"@rx ^/api/v1/repos/[^/]+/[^/]+/(?:releases|issues|issues/comments)/[0-9]+/assets$" \
"id:900625,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=930120;ARGS:name,\
ctl:ruleRemoveTargetById=932120;ARGS:name,\
ctl:ruleRemoveTargetById=932260;ARGS:name,\
ctl:ruleRemoveTargetById=934100;ARGS:name,\
ctl:ruleRemoveTargetById=934130;ARGS:name,\
ctl:ruleRemoveTargetById=941130;ARGS:name,\
ctl:ruleRemoveTargetById=941180;ARGS:name,\
ctl:ruleRemoveTargetById=942140;ARGS:name,\
ctl:ruleRemoveTargetById=944130;ARGS:name"
# A diff's hidden lines, fetched when shown, in path.
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/blob_excerpt/[0-9a-f]+$" \
"id:900626,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=932120;ARGS:path,\
ctl:ruleRemoveTargetById=934100;ARGS:path,\
ctl:ruleRemoveTargetById=934130;ARGS:path,\
ctl:ruleRemoveTargetById=941130;ARGS:path,\
ctl:ruleRemoveTargetById=941180;ARGS:path,\
ctl:ruleRemoveTargetById=942140;ARGS:path,\
ctl:ruleRemoveTargetById=944110;ARGS:path,\
ctl:ruleRemoveTargetById=944130;ARGS:path"
# The requests below name a branch or tag in a query parameter, which skips
# the list of command names (932260) there. It refused names such as
# docker-build, python3-migration and feat/docker-support.
# Deleting or restoring a branch, in name.
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/branches/(?:delete|restore)$" \
"id:900630,phase:1,pass,nolog,ctl:ruleRemoveTargetById=932260;ARGS:name"
# A release started from a tag, in tag.
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/releases/new$" \
"id:900631,phase:1,pass,nolog,ctl:ruleRemoveTargetById=932260;ARGS:tag"
# A branch protection rule opened for editing, in rule_name.
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/settings/branches/edit$" \
"id:900632,phase:1,pass,nolog,ctl:ruleRemoveTargetById=932260;ARGS:rule_name"
# A directory listing's last commits, in refSubUrl.
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/lastcommit/" \
"id:900633,phase:1,pass,nolog,ctl:ruleRemoveTargetById=932260;ARGS:refSubUrl"
# An artifact upload from actions/upload-artifact@v3 sends the header
# Content-Range, which the Core Rule Set refused (920450).
SecRule REQUEST_FILENAME \
"@rx ^/api/actions_pipeline/_apis/pipelines/workflows/[0-9]+/artifacts/[^/]+/upload$" \
"id:900640,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=920450;REQUEST_HEADERS_NAMES:Content-Range"
+393
View File
@@ -0,0 +1,393 @@
# Ordinary gitea 1.27 requests that the Core Rule Set must let through at the
# defaults. Each line is a path and query, and may end with one header after a
# space. The tests of internal/waf inspect each as a GET. The Core Rule Set reads
# the query and the headers the same way for any method.
# Searches with ordinary words and symbols.
/owner/repo/search?q=package.json
/owner/repo/search?q=.gitignore
/owner/repo/search?q=docker-compose.yml
/owner/repo/search?q=python3
/owner/repo/search?q=ssh%20key
/owner/repo/search?q=feat%2Fdocker-support
/owner/repo/search?q=ls%20-la
/owner/repo/search?q=sed%20-i
/owner/repo/search?q=%2Fbin%2Fsh
/owner/repo/search?q=%60make%20check%60
/owner/repo/search?q=%60whoami%60
/owner/repo/search?q=fetch%28
/owner/repo/search?q=%24%7BVAR%7D
/owner/repo/search?q=%24%28pwd%29
/owner/repo/search?q=process.env
/owner/repo/search?q=%3Cimg%20src%3D
/owner/repo/search?q=%3Cscript%3E
/owner/repo/search?q=SELECT%20%2A%20FROM%20users%20WHERE
/owner/repo/search?q=union%20select
/owner/repo/search?q=http%3A%2F%2F127.0.0.1%3A8080
/owner/repo/search?q=..%2F
/owner/repo/search?q=%2Fetc%2Fpasswd
/owner/repo/search?q=eval%28
/owner/repo/search?q=chmod%20%2Bx
/owner/repo/search?q=echo%20%24PATH
/owner/repo/search?q=java.lang.Runtime
/owner/repo/search?q=document.cookie
/owner/repo/search?q=alert%281%29
/owner/repo/search?q=__proto__
/owner/repo/search?q=C%3A%5CWindows%5CSystem32
/owner/repo/search?q=Invoke-WebRequest
/owner/repo/search?q=if%20%28a%20%3D%3D%20b%29%20%7B
/owner/repo/search?q=sudo%20apt%20install
/owner/repo/search?q=~%2F.ssh%2Fconfig
/owner/repo/search?q=.htaccess
/owner/repo/search?q=%24_GET%5B
/owner/repo/search?q=%24_POST
/owner/repo/search?q=%24_SERVER%5B%27REQUEST_URI%27%5D
/owner/repo/search?q=%3C%3Fphp%20echo
/owner/repo/search?q=information_schema
/owner/repo/search?q=LIKE%20%27%25foo%25%27
/owner/repo/search?q=onClick%3D%7B
/explore/code?q=SELECT%20%2A%20FROM%20users%20WHERE
/explore/code?q=%3Cimg%20src%3D
/owner/-/code?q=SELECT%20%2A%20FROM%20users%20WHERE
/owner/-/code?q=%3Cimg%20src%3D
/owner/repo/issues?type=all&state=open&q=SELECT%20%2A%20FROM%20users%20WHERE
/owner/repo/issues?type=all&state=open&q=%3Cimg%20src%3D
/owner/repo/issues/search?q=SELECT%20%2A%20FROM%20users%20WHERE
/owner/repo/issues/search?q=%3Cimg%20src%3D
/owner/repo/pulls?q=SELECT%20%2A%20FROM%20users%20WHERE
/owner/repo/pulls?q=%3Cimg%20src%3D
/issues?type=your_repositories&q=SELECT%20%2A%20FROM%20users%20WHERE
/issues?type=your_repositories&q=%3Cimg%20src%3D
/issues/search?q=SELECT%20%2A%20FROM%20users%20WHERE
/issues/search?q=%3Cimg%20src%3D
/pulls?type=your_repositories&q=SELECT%20%2A%20FROM%20users%20WHERE
/pulls?type=your_repositories&q=%3Cimg%20src%3D
/owner/repo/commits/branch/main/search?q=SELECT%20%2A%20FROM%20users%20WHERE
/owner/repo/commits/branch/main/search?q=%3Cimg%20src%3D
/api/v1/repos/issues/search?q=SELECT%20%2A%20FROM%20users%20WHERE
/api/v1/repos/issues/search?q=%3Cimg%20src%3D
/api/v1/repos/owner/repo/issues?q=SELECT%20%2A%20FROM%20users%20WHERE
/api/v1/repos/owner/repo/issues?q=%3Cimg%20src%3D
# Searches for the names of repositories, users, branches, tags and packages.
/explore/repos?q=python3-app
/explore/repos?q=.profile
/explore/repos?q=powershell-scripts
/api/v1/repos/search?q=docker-build
/user/search_candidates?q=docker-ci
/owner/repo/branches?q=feat%2Fdocker-support
/owner/repo/tags?q=docker-v1.0
/owner/-/packages?q=python3-requests
/owner/-/packages?q=org.apache.commons.logging
/repo/search?q=python3-app
/repo/search?q=docker-build
/repo/search?q=powershell-scripts
/repo/search?q=.profile
/owner?tab=repositories&q=python3-app
/org?q=python3-app
/owner/repo/issues/posters?q=docker-ci
/owner/repo/pulls/posters?q=docker-ci
/explore/users?q=docker-ci
/explore/organizations?q=docker-ci
/api/v1/users/search?q=docker-ci
# Pages, raw files and edit pages of files, and the API's call for their
# contents.
/owner/repo/src/branch/main/package.json
/owner/repo/raw/branch/main/package.json
/owner/repo/_edit/main/package.json
/api/v1/repos/owner/repo/contents/package.json
/owner/repo/src/branch/main/.gitignore
/owner/repo/raw/branch/main/.gitignore
/owner/repo/_edit/main/.gitignore
/api/v1/repos/owner/repo/contents/.gitignore
/owner/repo/src/branch/main/docker-compose.yml
/owner/repo/raw/branch/main/docker-compose.yml
/owner/repo/_edit/main/docker-compose.yml
/api/v1/repos/owner/repo/contents/docker-compose.yml
/owner/repo/src/branch/main/Dockerfile
/owner/repo/raw/branch/main/Dockerfile
/owner/repo/_edit/main/Dockerfile
/api/v1/repos/owner/repo/contents/Dockerfile
/owner/repo/src/branch/main/.htaccess
/owner/repo/raw/branch/main/.htaccess
/owner/repo/_edit/main/.htaccess
/api/v1/repos/owner/repo/contents/.htaccess
/owner/repo/src/branch/main/rootfs/etc/passwd
/owner/repo/raw/branch/main/rootfs/etc/passwd
/owner/repo/_edit/main/rootfs/etc/passwd
/api/v1/repos/owner/repo/contents/rootfs/etc/passwd
/owner/repo/src/branch/main/bin/docker-entrypoint
/owner/repo/raw/branch/main/bin/docker-entrypoint
/owner/repo/_edit/main/bin/docker-entrypoint
/api/v1/repos/owner/repo/contents/bin/docker-entrypoint
/owner/repo/src/branch/main/scripts/python3-setup.sh
/owner/repo/raw/branch/main/scripts/python3-setup.sh
/owner/repo/_edit/main/scripts/python3-setup.sh
/api/v1/repos/owner/repo/contents/scripts/python3-setup.sh
/owner/repo/src/branch/main/src/java.base/share/classes/java/lang/Runtime.java
/owner/repo/raw/branch/main/src/java.base/share/classes/java/lang/Runtime.java
/owner/repo/_edit/main/src/java.base/share/classes/java/lang/Runtime.java
/api/v1/repos/owner/repo/contents/src/java.base/share/classes/java/lang/Runtime.java
/owner/repo/src/branch/main/java.lang.String.md
/owner/repo/raw/branch/main/java.lang.String.md
/owner/repo/_edit/main/java.lang.String.md
/api/v1/repos/owner/repo/contents/java.lang.String.md
/owner/repo/src/branch/main/META-INF/services/org.apache.commons.logging.LogFactory
/owner/repo/raw/branch/main/META-INF/services/org.apache.commons.logging.LogFactory
/owner/repo/_edit/main/META-INF/services/org.apache.commons.logging.LogFactory
/api/v1/repos/owner/repo/contents/META-INF/services/org.apache.commons.logging.LogFactory
/owner/repo/src/branch/main/process.env.d.ts
/owner/repo/raw/branch/main/process.env.d.ts
/owner/repo/_edit/main/process.env.d.ts
/api/v1/repos/owner/repo/contents/process.env.d.ts
/owner/repo/src/branch/main/document.write.js
/owner/repo/raw/branch/main/document.write.js
/owner/repo/_edit/main/document.write.js
/api/v1/repos/owner/repo/contents/document.write.js
/owner/repo/src/branch/main/test/fixtures/__proto__.json
/owner/repo/raw/branch/main/test/fixtures/__proto__.json
/owner/repo/_edit/main/test/fixtures/__proto__.json
/api/v1/repos/owner/repo/contents/test/fixtures/__proto__.json
/owner/repo/src/branch/main/install-script.sh
/owner/repo/raw/branch/main/install-script.sh
/owner/repo/_edit/main/install-script.sh
/api/v1/repos/owner/repo/contents/install-script.sh
/owner/repo/src/branch/main/src/backend/catalog/information_schema.sql
/owner/repo/raw/branch/main/src/backend/catalog/information_schema.sql
/owner/repo/_edit/main/src/backend/catalog/information_schema.sql
/api/v1/repos/owner/repo/contents/src/backend/catalog/information_schema.sql
/owner/repo/src/branch/main/docs/a%20b.xhtml
/owner/repo/raw/branch/main/docs/a%20b.xhtml
/owner/repo/_edit/main/docs/a%20b.xhtml
/api/v1/repos/owner/repo/contents/docs/a%20b.xhtml
/owner/repo/src/branch/main/%5Bdraft%5D.md
/owner/repo/raw/branch/main/%5Bdraft%5D.md
/owner/repo/_edit/main/%5Bdraft%5D.md
/api/v1/repos/owner/repo/contents/%5Bdraft%5D.md
/owner/repo/src/branch/main/notes.txt~
/owner/repo/raw/branch/main/notes.txt~
/owner/repo/_edit/main/notes.txt~
/api/v1/repos/owner/repo/contents/notes.txt~
# The download, blame, history, delete and preview pages of the files above
# whose names the Core Rule Set checks in the path, and their downloads as
# release attachments.
/owner/repo/media/branch/main/notes.txt~
/owner/repo/blame/branch/main/notes.txt~
/owner/repo/commits/branch/main/notes.txt~
/owner/repo/_delete/main/notes.txt~
/owner/repo/media/branch/main/process.env.d.ts
/owner/repo/blame/branch/main/process.env.d.ts
/owner/repo/commits/branch/main/process.env.d.ts
/owner/repo/_delete/main/process.env.d.ts
/owner/repo/media/branch/main/docs/a%20b.xhtml
/owner/repo/blame/branch/main/docs/a%20b.xhtml
/owner/repo/commits/branch/main/docs/a%20b.xhtml
/owner/repo/_delete/main/docs/a%20b.xhtml
/owner/repo/media/branch/main/java.lang.String.md
/owner/repo/blame/branch/main/java.lang.String.md
/owner/repo/commits/branch/main/java.lang.String.md
/owner/repo/_delete/main/java.lang.String.md
/owner/repo/_preview/main/notes.txt~
/owner/repo/_preview/main/process.env.d.ts
/owner/repo/_preview/main/java.lang.String.md
/owner/repo/releases/download/v1.0/notes.txt~
/owner/repo/releases/download/v1.0/process.env.d.ts
/owner/repo/releases/download/v1.0/java.lang.String.md
/owner/repo/releases/download/v1.0/a%20b.xhtml
# File names in the query string: the commit graph of a file, the API's lists
# of its commits and of its contents, an issue opened from a template, the
# sign-in page on the way to its edit page, and a diff's hidden lines.
/owner/repo/graph?file=package.json
/api/v1/repos/owner/repo/commits?path=package.json
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22package.json%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fpackage.json
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fpackage.json
/owner/repo/graph?file=.gitignore
/api/v1/repos/owner/repo/commits?path=.gitignore
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22.gitignore%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2F.gitignore
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2F.gitignore
/owner/repo/graph?file=docker-compose.yml
/api/v1/repos/owner/repo/commits?path=docker-compose.yml
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22docker-compose.yml%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fdocker-compose.yml
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fdocker-compose.yml
/owner/repo/graph?file=.htaccess
/api/v1/repos/owner/repo/commits?path=.htaccess
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22.htaccess%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2F.htaccess
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2F.htaccess
/owner/repo/graph?file=rootfs%2Fetc%2Fpasswd
/api/v1/repos/owner/repo/commits?path=rootfs%2Fetc%2Fpasswd
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22rootfs%2Fetc%2Fpasswd%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Frootfs%2Fetc%2Fpasswd
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Frootfs%2Fetc%2Fpasswd
/owner/repo/graph?file=bin%2Fdocker-entrypoint
/api/v1/repos/owner/repo/commits?path=bin%2Fdocker-entrypoint
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22bin%2Fdocker-entrypoint%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fbin%2Fdocker-entrypoint
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fbin%2Fdocker-entrypoint
/owner/repo/graph?file=scripts%2Fpython3-setup.sh
/api/v1/repos/owner/repo/commits?path=scripts%2Fpython3-setup.sh
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22scripts%2Fpython3-setup.sh%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fscripts%2Fpython3-setup.sh
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fscripts%2Fpython3-setup.sh
/owner/repo/graph?file=src%2Fjava.base%2Fshare%2Fclasses%2Fjava%2Flang%2FRuntime.java
/api/v1/repos/owner/repo/commits?path=src%2Fjava.base%2Fshare%2Fclasses%2Fjava%2Flang%2FRuntime.java
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22src%2Fjava.base%2Fshare%2Fclasses%2Fjava%2Flang%2FRuntime.java%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fsrc%2Fjava.base%2Fshare%2Fclasses%2Fjava%2Flang%2FRuntime.java
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fsrc%2Fjava.base%2Fshare%2Fclasses%2Fjava%2Flang%2FRuntime.java
/owner/repo/graph?file=java.lang.String.md
/api/v1/repos/owner/repo/commits?path=java.lang.String.md
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22java.lang.String.md%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fjava.lang.String.md
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fjava.lang.String.md
/owner/repo/graph?file=META-INF%2Fservices%2Forg.apache.commons.logging.LogFactory
/api/v1/repos/owner/repo/commits?path=META-INF%2Fservices%2Forg.apache.commons.logging.LogFactory
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22META-INF%2Fservices%2Forg.apache.commons.logging.LogFactory%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2FMETA-INF%2Fservices%2Forg.apache.commons.logging.LogFactory
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2FMETA-INF%2Fservices%2Forg.apache.commons.logging.LogFactory
/owner/repo/graph?file=process.env.d.ts
/api/v1/repos/owner/repo/commits?path=process.env.d.ts
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22process.env.d.ts%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fprocess.env.d.ts
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fprocess.env.d.ts
/owner/repo/graph?file=document.write.js
/api/v1/repos/owner/repo/commits?path=document.write.js
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22document.write.js%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fdocument.write.js
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fdocument.write.js
/owner/repo/graph?file=test%2Ffixtures%2F__proto__.json
/api/v1/repos/owner/repo/commits?path=test%2Ffixtures%2F__proto__.json
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22test%2Ffixtures%2F__proto__.json%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Ftest%2Ffixtures%2F__proto__.json
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Ftest%2Ffixtures%2F__proto__.json
/owner/repo/graph?file=install-script.sh
/api/v1/repos/owner/repo/commits?path=install-script.sh
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22install-script.sh%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Finstall-script.sh
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Finstall-script.sh
/owner/repo/graph?file=src%2Fbackend%2Fcatalog%2Finformation_schema.sql
/api/v1/repos/owner/repo/commits?path=src%2Fbackend%2Fcatalog%2Finformation_schema.sql
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22src%2Fbackend%2Fcatalog%2Finformation_schema.sql%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fsrc%2Fbackend%2Fcatalog%2Finformation_schema.sql
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fsrc%2Fbackend%2Fcatalog%2Finformation_schema.sql
/owner/repo/graph?file=docs%2Fa%20b.xhtml
/api/v1/repos/owner/repo/commits?path=docs%2Fa%20b.xhtml
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22docs%2Fa%20b.xhtml%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2Fdocs%2Fa%20b.xhtml
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2Fdocs%2Fa%20b.xhtml
/owner/repo/graph?file=%5Bdraft%5D.md
/api/v1/repos/owner/repo/commits?path=%5Bdraft%5D.md
/api/v1/repos/owner/repo/file-contents?body=%7B%22files%22%3A%5B%22%5Bdraft%5D.md%22%5D%7D
/owner/repo/issues/new?template=.gitea%2FISSUE_TEMPLATE%2F%5Bdraft%5D.md
/user/login?redirect_to=%2Fowner%2Frepo%2F_edit%2Fmain%2F%5Bdraft%5D.md
/owner/repo/blob_excerpt/0123456789abcdef0123456789abcdef01234567?last_left=10&last_right=10&left=20&right=20&left_hunk_size=5&right_hunk_size=5&path=process.env.d.ts&style=unified&direction=down&wiki=false
/owner/repo/blob_excerpt/0123456789abcdef0123456789abcdef01234567?last_left=10&last_right=10&left=20&right=20&left_hunk_size=5&right_hunk_size=5&path=java.lang.String.md&style=unified&direction=down&wiki=false
/owner/repo/blob_excerpt/0123456789abcdef0123456789abcdef01234567?last_left=10&last_right=10&left=20&right=20&left_hunk_size=5&right_hunk_size=5&path=install-script.sh&style=unified&direction=down&wiki=false
/owner/repo/blob_excerpt/0123456789abcdef0123456789abcdef01234567?last_left=10&last_right=10&left=20&right=20&left_hunk_size=5&right_hunk_size=5&path=information_schema.sql&style=unified&direction=down&wiki=false
/owner/repo/blob_excerpt/0123456789abcdef0123456789abcdef01234567?last_left=10&last_right=10&left=20&right=20&left_hunk_size=5&right_hunk_size=5&path=document.write.js&style=unified&direction=down&wiki=false
/owner/repo/blob_excerpt/0123456789abcdef0123456789abcdef01234567?last_left=10&last_right=10&left=20&right=20&left_hunk_size=5&right_hunk_size=5&path=__proto__.json&style=unified&direction=down&wiki=false
/owner/repo/blob_excerpt/0123456789abcdef0123456789abcdef01234567?last_left=10&last_right=10&left=20&right=20&left_hunk_size=5&right_hunk_size=5&path=src%2Fjava.base%2Fshare%2Fclasses%2Fjava%2Flang%2FRuntime.java&style=unified&direction=down&wiki=false
/owner/repo/blob_excerpt/0123456789abcdef0123456789abcdef01234567?last_left=10&last_right=10&left=20&right=20&left_hunk_size=5&right_hunk_size=5&path=index.xhtml&style=unified&direction=down&wiki=false
# File names of attachments uploaded through the API, to a release, an issue and
# a comment.
/api/v1/repos/owner/repo/releases/1/assets?name=package.json
/api/v1/repos/owner/repo/releases/1/assets?name=.gitignore
/api/v1/repos/owner/repo/releases/1/assets?name=docker-compose.yml
/api/v1/repos/owner/repo/releases/1/assets?name=.htaccess
/api/v1/repos/owner/repo/releases/1/assets?name=docker-entrypoint
/api/v1/repos/owner/repo/releases/1/assets?name=python3-setup.sh
/api/v1/repos/owner/repo/releases/1/assets?name=Runtime.java
/api/v1/repos/owner/repo/releases/1/assets?name=org.apache.commons.logging.LogFactory
/api/v1/repos/owner/repo/releases/1/assets?name=%5Bdraft%5D.md
/api/v1/repos/owner/repo/releases/1/assets?name=notes.txt~
/api/v1/repos/owner/repo/releases/1/assets?name=install-script.sh
/api/v1/repos/owner/repo/issues/1/assets?name=install-script.sh
/api/v1/repos/owner/repo/issues/comments/1/assets?name=install-script.sh
/api/v1/repos/owner/repo/releases/1/assets?name=process.env.d.ts
/api/v1/repos/owner/repo/issues/1/assets?name=process.env.d.ts
/api/v1/repos/owner/repo/issues/comments/1/assets?name=process.env.d.ts
/api/v1/repos/owner/repo/releases/1/assets?name=java.lang.String.md
/api/v1/repos/owner/repo/issues/1/assets?name=java.lang.String.md
/api/v1/repos/owner/repo/issues/comments/1/assets?name=java.lang.String.md
/api/v1/repos/owner/repo/releases/1/assets?name=information_schema.sql
/api/v1/repos/owner/repo/issues/1/assets?name=information_schema.sql
/api/v1/repos/owner/repo/issues/comments/1/assets?name=information_schema.sql
/api/v1/repos/owner/repo/releases/1/assets?name=document.write.js
/api/v1/repos/owner/repo/issues/1/assets?name=document.write.js
/api/v1/repos/owner/repo/issues/comments/1/assets?name=document.write.js
/api/v1/repos/owner/repo/releases/1/assets?name=__proto__.json
/api/v1/repos/owner/repo/issues/1/assets?name=__proto__.json
/api/v1/repos/owner/repo/issues/comments/1/assets?name=__proto__.json
/api/v1/repos/owner/repo/releases/1/assets?name=index.xhtml
/api/v1/repos/owner/repo/issues/1/assets?name=index.xhtml
/api/v1/repos/owner/repo/issues/comments/1/assets?name=index.xhtml
# An artifact upload from actions/upload-artifact@v3, which sends Content-Range.
/api/actions_pipeline/_apis/pipelines/workflows/1/artifacts/9b7c1f3a/upload?itemPath=artifact%2Freport.txt Content-Range: bytes 0-9/10
# Branch and tag names in the query string: deleting and restoring a branch,
# starting a release from a tag, a branch protection rule, a directory listing's
# last commits, and a container image push that reuses a layer.
/owner/repo/branches/delete?name=docker-build&page=1
/owner/repo/branches/restore?branch_id=1&name=docker-build&page=1
/owner/repo/releases/new?tag=docker-build
/owner/repo/settings/branches/edit?rule_name=docker-build
/owner/repo/lastcommit/main/src?refSubUrl=branch%2Fdocker-build
/v2/owner/app/blobs/uploads/?mount=sha256%3A0000&from=owner%2Fdocker-build
/owner/repo/branches/delete?name=feat%2Fdocker-support&page=1
/owner/repo/branches/restore?branch_id=1&name=feat%2Fdocker-support&page=1
/owner/repo/releases/new?tag=feat%2Fdocker-support
/owner/repo/settings/branches/edit?rule_name=feat%2Fdocker-support
/owner/repo/lastcommit/main/src?refSubUrl=branch%2Ffeat%2Fdocker-support
/v2/owner/app/blobs/uploads/?mount=sha256%3A0000&from=owner%2Ffeat%2Fdocker-support
/owner/repo/branches/delete?name=renovate%2Fdocker-build-push-action-6.x&page=1
/owner/repo/branches/restore?branch_id=1&name=renovate%2Fdocker-build-push-action-6.x&page=1
/owner/repo/releases/new?tag=renovate%2Fdocker-build-push-action-6.x
/owner/repo/settings/branches/edit?rule_name=renovate%2Fdocker-build-push-action-6.x
/owner/repo/lastcommit/main/src?refSubUrl=branch%2Frenovate%2Fdocker-build-push-action-6.x
/v2/owner/app/blobs/uploads/?mount=sha256%3A0000&from=owner%2Frenovate%2Fdocker-build-push-action-6.x
/owner/repo/branches/delete?name=python3-migration&page=1
/owner/repo/branches/restore?branch_id=1&name=python3-migration&page=1
/owner/repo/releases/new?tag=python3-migration
/owner/repo/settings/branches/edit?rule_name=python3-migration
/owner/repo/lastcommit/main/src?refSubUrl=branch%2Fpython3-migration
/v2/owner/app/blobs/uploads/?mount=sha256%3A0000&from=owner%2Fpython3-migration
/owner/repo/branches/delete?name=ansible-roles&page=1
/owner/repo/branches/restore?branch_id=1&name=ansible-roles&page=1
/owner/repo/releases/new?tag=ansible-roles
/owner/repo/settings/branches/edit?rule_name=ansible-roles
/owner/repo/lastcommit/main/src?refSubUrl=branch%2Fansible-roles
/v2/owner/app/blobs/uploads/?mount=sha256%3A0000&from=owner%2Fansible-roles
/owner/repo/branches/delete?name=whoami&page=1
/owner/repo/branches/restore?branch_id=1&name=whoami&page=1
/owner/repo/releases/new?tag=whoami
/owner/repo/settings/branches/edit?rule_name=whoami
/owner/repo/lastcommit/main/src?refSubUrl=branch%2Fwhoami
/v2/owner/app/blobs/uploads/?mount=sha256%3A0000&from=owner%2Fwhoami
# Pages a signed-in user opens.
/
/owner/repo
/owner/repo/issues
/owner/repo/pulls
/owner/repo/releases
/owner/repo/wiki
/owner/repo/activity
/owner/repo/branches
/owner/repo/tags
/owner/repo/settings
/owner/repo/actions
/owner/repo/compare/main...main
/owner/repo/commits/branch/main
/owner/repo/archive/main.zip
/notifications
/user/settings
/user/settings/applications
/explore/repos
/-/admin
+153 -15
View File
@@ -1,8 +1,9 @@
// Package waf runs the OWASP Core Rule Set 4.25.0, through Coraza, on the // Package waf runs the OWASP Core Rule Set 4.25.0, through Coraza, on the
// method, the URL with its query and the headers of a request, and on its // method, the URL with its query and the headers of a request, and on its
// body while SWWAF_WAF_BODY_LIMIT is set, with the six changes smallwebwaf // body while SWWAF_WAF_BODY_LIMIT is set, with the eight changes smallwebwaf
// makes to it, as "Attack detection" under "Settings" in SPEC.md describes // makes to it and the exclusions in SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE, as
// them. It reads no response. // "Attack detection" under "Settings" in SPEC.md describes them. It reads no
// response.
// //
// smallwebwaf writes only to its state directory, so Coraza is built with // smallwebwaf writes only to its state directory, so Coraza is built with
// its no_fs_access tag, as the Dockerfile and script/build build it: of a // its no_fs_access tag, as the Dockerfile and script/build build it: of a
@@ -11,10 +12,14 @@
package waf package waf
import ( import (
_ "embed"
"errors"
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
"net/netip" "net/netip"
"os"
"regexp"
"slices" "slices"
"strconv" "strconv"
"strings" "strings"
@@ -26,12 +31,14 @@ import (
) )
// directives are the Core Rule Set as smallwebwaf runs it, with the // directives are the Core Rule Set as smallwebwaf runs it, with the
// paranoia level for %d, and bodyDirectives for %s while // paranoia level for %d, bodyDirectives for the first %s while
// SWWAF_WAF_BODY_LIMIT is set. Each rule smallwebwaf adds has an id from // SWWAF_WAF_BODY_LIMIT is set, the exclusions file for the second, and
// 900000 to 900999, the ids the Core Rule Set keeps for the rules that set // giteaExclusions for the third.
// it up, which SWWAF_WAF_DISABLED_RULES refuses, so that no setting // Each rule smallwebwaf adds has an id from 900000 to 900999, the ids the
// switches one off. Coraza joins a line ending in \ to the next, without // Core Rule Set keeps for the rules that set it up, which
// the spaces at the start of the next. // SWWAF_WAF_DISABLED_RULES and the exclusions file refuse, so that no
// setting switches one off. Coraza joins a line ending in \ to the next,
// without the spaces at the start of the next.
const directives = ` const directives = `
# The engine only detects. smallwebwaf compares the request's anomaly # The engine only detects. smallwebwaf compares the request's anomaly
# score with SWWAF_WAF_ANOMALY_THRESHOLD itself, in block and detect mode # score with SWWAF_WAF_ANOMALY_THRESHOLD itself, in block and detect mode
@@ -69,6 +76,14 @@ SecArgumentsLimit 1000
SecRule ARGUMENTS_LIMIT_REACHED "@eq 1" "id:900300,phase:2,pass,\ SecRule ARGUMENTS_LIMIT_REACHED "@eq 1" "id:900300,phase:2,pass,\
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'" severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
# The exclusions file comes before the rules, where the Core Rule Set puts
# its own file of exclusions that run before the rules, so that they run
# before the rules of phase 1 too.
%s
# The eighth change comes before the rules for the same reason.
%s
# The sixth: only the rules for requests are loaded, and no response is # The sixth: only the rules for requests are loaded, and no response is
# inspected. # inspected.
Include @owasp_crs/REQUEST-*.conf Include @owasp_crs/REQUEST-*.conf
@@ -96,10 +111,19 @@ SecRuleUpdateTargetById 932260 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
!ARGS:artifactName|!ARGS:redirect_to" !ARGS:artifactName|!ARGS:redirect_to"
# The fifth, for Referer: it is not checked for a Unix command without # The fifth, for Referer: it is not checked for a Unix command without
# arguments, or for Java starting a process. The cookies are left out in # arguments, for Java starting a process, or for a Java class name. 944130
# Inspect. # refused every page opened from the page of java.lang.String.md. The
# cookies are left out in Inspect.
SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer" SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer"
SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer" SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer"
SecRuleUpdateTargetById 944130 "!REQUEST_HEADERS:Referer"
# The seventh: a container image push that reuses a layer names the image
# it takes the layer from in the query parameter from, which is not
# checked for command names. 932260 refused the push of a layer from
# owner/python3-app.
SecRule REQUEST_FILENAME "@rx ^/v2/.+/blobs/uploads/$" \
"id:900500,phase:1,pass,nolog,ctl:ruleRemoveTargetById=932260;ARGS:from"
` `
// bodyDirectives have the Core Rule Set read the part of a request body // bodyDirectives have the Core Rule Set read the part of a request body
@@ -144,6 +168,13 @@ SecRule MULTIPART_STRICT_ERROR "!@eq 0" "id:900450,phase:2,pass,\
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'" severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
` `
// giteaExclusions are the eighth change: the fields of gitea's own
// requests that skip the rules that refused them there, as Coraza
// directives.
//
//go:embed gitea.conf
var giteaExclusions string
// cookiesNotRead are the cookies the Core Rule Set reads a request // cookiesNotRead are the cookies the Core Rule Set reads a request
// without, the rest of the fifth change. // without, the rest of the fifth change.
// //
@@ -160,6 +191,94 @@ type Params struct {
// BodyLimit is the most of a request body the Core Rule Set reads // BodyLimit is the most of a request body the Core Rule Set reads
// (SWWAF_WAF_BODY_LIMIT), 0 while it is off and it reads none. // (SWWAF_WAF_BODY_LIMIT), 0 while it is off and it reads none.
BodyLimit int64 BodyLimit int64
// ExclusionsFile is the path of a file of exclusions loaded before the
// Core Rule Set's rules (SWWAF_CORE_RULE_SET_EXCLUSIONS_FILE), "" while
// there is none.
ExclusionsFile string
}
// The ids of smallwebwaf's own rules, which an exclusions file may not
// name.
const (
firstOwnRuleID = 900000
lastOwnRuleID = 900999
)
// numberOrRange matches a number, or a range of numbers such as
// 900000-900999. Coraza reads a number with a + before it too, so the
// end of a range may have one.
var numberOrRange = regexp.MustCompile(`(\d+)(?:-\+?(\d+))?`)
// errSmallwebwafRuleID is the error of an exclusions file that names an
// id of smallwebwaf's own rules.
var errSmallwebwafRuleID = errors.New("an id of smallwebwaf's own rules")
// errChangesNoRule is the error of an exclusions file that holds a
// directive that removes or changes rules as it loads, such as
// SecRuleRemoveById. The Core Rule Set's rules load after the file, so
// the directive finds none of them.
var errChangesNoRule = errors.New("the Core Rule Set's rules load after the file; " +
"switch a rule off with SWWAF_WAF_DISABLED_RULES, or on some requests " +
"with a ctl exclusion")
// directivesIn returns the directives in text as Coraza reads them: it
// skips blank lines and comment lines, and joins a line ending in \ to
// the next, without the spaces around either.
func directivesIn(text string) []string {
var directives []string
joined := ""
for line := range strings.Lines(text) {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
if start, found := strings.CutSuffix(line, `\`); found {
joined += start
continue
}
directives = append(directives, joined+line)
joined = ""
}
return directives
}
// checkExclusions returns an error if exclusions, the text of an
// exclusions file, names an id of smallwebwaf's own rules, alone or in a
// range, or holds a directive whose name starts with SecRuleRemove or
// SecRuleUpdate.
func checkExclusions(exclusions string) error {
for _, directive := range directivesIn(exclusions) {
for _, match := range numberOrRange.FindAllStringSubmatch(directive, -1) {
first, _ := strconv.Atoi(match[1])
last := first
if match[2] != "" {
last, _ = strconv.Atoi(match[2])
}
if first <= lastOwnRuleID && last >= firstOwnRuleID {
return fmt.Errorf("the exclusions file names %s: %w", match[0],
errSmallwebwafRuleID)
}
}
// Coraza takes the directive's name up to the first space, without
// regard to case.
name, _, _ := strings.Cut(directive, " ")
if lower := strings.ToLower(name); strings.HasPrefix(lower, "secruleremove") ||
strings.HasPrefix(lower, "secruleupdate") {
return fmt.Errorf("%s in the exclusions file changes no rule: %w", name,
errChangesNoRule)
}
}
return nil
} }
// CoreRuleSet is the Core Rule Set, ready to inspect requests. It is safe // CoreRuleSet is the Core Rule Set, ready to inspect requests. It is safe
@@ -169,16 +288,35 @@ type CoreRuleSet struct {
bodyLimit int64 bodyLimit int64
} }
// New returns the Core Rule Set with the six changes, at params' // New returns the Core Rule Set with the eight changes, at params'
// paranoia level, without the rules it switches off, and reading request // paranoia level, without the rules it switches off, reading request
// bodies up to params' limit. // bodies up to params' limit, and with params' exclusions file loaded
// before the Core Rule Set's rules. An exclusions file is an error if it
// cannot be read or does not parse, or if checkExclusions refuses it.
func New(params Params) (*CoreRuleSet, error) { func New(params Params) (*CoreRuleSet, error) {
body := "" body := ""
if params.BodyLimit > 0 { if params.BodyLimit > 0 {
body = fmt.Sprintf(bodyDirectives, params.BodyLimit) body = fmt.Sprintf(bodyDirectives, params.BodyLimit)
} }
text := fmt.Sprintf(directives, params.ParanoiaLevel, body) exclusions := ""
if params.ExclusionsFile != "" {
file, err := os.ReadFile(params.ExclusionsFile)
if err != nil {
return nil, fmt.Errorf("read the exclusions file: %w", err)
}
exclusions = string(file)
err = checkExclusions(exclusions)
if err != nil {
return nil, err
}
}
text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions,
giteaExclusions)
if len(params.DisabledRules) > 0 { if len(params.DisabledRules) > 0 {
ids := make([]string, len(params.DisabledRules)) ids := make([]string, len(params.DisabledRules))
+190 -2
View File
@@ -5,6 +5,7 @@ import (
"net/http/httptest" "net/http/httptest"
"net/netip" "net/netip"
"net/url" "net/url"
"os"
"path/filepath" "path/filepath"
"reflect" "reflect"
"strconv" "strconv"
@@ -119,7 +120,7 @@ func atDefaults(t *testing.T) *waf.CoreRuleSet {
} }
// wantChange checks that crs lets through passes, a gitea request one of // wantChange checks that crs lets through passes, a gitea request one of
// the six changes is for, and still finds result in refused, a request // the eight changes is for, and still finds result in refused, a request
// like it that the change is not for. // like it that the change is not for.
func wantChange( func wantChange(
t *testing.T, crs *waf.CoreRuleSet, passes, refused request, result waf.Result, t *testing.T, crs *waf.CoreRuleSet, passes, refused request, result waf.Result,
@@ -308,13 +309,14 @@ func TestCookiesGiteaFlashAndRedirectToAreNotRead(t *testing.T) {
matched(930120)) matched(930120))
} }
func TestRefererIsNotCheckedForACommandOrJavaStartingAProcess(t *testing.T) { func TestRefererIsNotCheckedForACommandOrJava(t *testing.T) {
t.Parallel() t.Parallel()
const ( const (
search = "https://git.example/explore/repos?q=env" search = "https://git.example/explore/repos?q=env"
runtimeJava = "https://git.example/openjdk/jdk/src/branch/master/src/" + runtimeJava = "https://git.example/openjdk/jdk/src/branch/master/src/" +
"java.base/share/classes/java/lang/Runtime.java" "java.base/share/classes/java/lang/Runtime.java"
stringMD = "https://git.example/owner/repo/src/branch/main/java.lang.String.md"
) )
crs := atDefaults(t) crs := atDefaults(t)
@@ -323,6 +325,8 @@ func TestRefererIsNotCheckedForACommandOrJavaStartingAProcess(t *testing.T) {
matched(932340)) matched(932340))
wantChange(t, crs, get("/", "Referer: "+runtimeJava), wantChange(t, crs, get("/", "Referer: "+runtimeJava),
get("/", "X-Page: "+runtimeJava), matched(944110)) get("/", "X-Page: "+runtimeJava), matched(944110))
wantChange(t, crs, get("/", "Referer: "+stringMD), get("/", "X-Page: "+stringMD),
matched(944130))
// It is still checked for script and SQL injection. // It is still checked for script and SQL injection.
wantResult(t, crs, wantResult(t, crs,
@@ -362,6 +366,190 @@ func TestEachDisabledRuleIsSwitchedOff(t *testing.T) {
wantResult(t, newCoreRuleSet(t, 1, 920350, 911100), r, waf.Result{}) wantResult(t, newCoreRuleSet(t, 1, 920350, 911100), r, waf.Result{})
} }
func TestImagePushThatReusesALayerIsNotCheckedForCommandNames(t *testing.T) {
t.Parallel()
crs := atDefaults(t)
reuse := "?mount=sha256:0000&from=owner/python3-app"
wantChange(t, crs, get("/v2/owner/app/blobs/uploads/"+reuse), get("/upload/"+reuse),
matched(932260))
// A command after the image's name is still refused.
wantResult(t, crs,
get("/v2/owner/app/blobs/uploads/?from=owner/app;cat%20/etc/passwd"),
matched(930120, 932160))
}
// withExclusions returns the Core Rule Set at the defaults, with the
// exclusions file at path.
func withExclusions(path string) (*waf.CoreRuleSet, error) {
return waf.New(waf.Params{
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, ExclusionsFile: path,
})
}
// exclusionsFile writes text to a file of exclusions and returns its path.
func exclusionsFile(t *testing.T, text string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "exclusions.conf")
err := os.WriteFile(path, []byte(text), 0o600)
if err != nil {
t.Fatal(err)
}
return path
}
// backupFileExclusion is an exclusions file that switches 920500, a rule
// of phase 1, off for the path of a backup file under /files/.
const backupFileExclusion = `SecRule REQUEST_FILENAME "@beginsWith /files/" ` +
`"id:1,phase:1,pass,nolog,ctl:ruleRemoveTargetById=920500;REQUEST_FILENAME"` + "\n"
func TestExclusionsFileIsLoadedBeforeTheRules(t *testing.T) {
t.Parallel()
crs, err := withExclusions(exclusionsFile(t, backupFileExclusion))
if err != nil {
t.Fatalf("load the exclusions: %v", err)
}
wantChange(t, crs, get("/files/notes.txt~"), get("/notes.txt~"), matched(920500))
}
func TestRuleSwitchedOffAndNamedInTheExclusionsFileIsOff(t *testing.T) {
t.Parallel()
crs, err := waf.New(waf.Params{
ParanoiaLevel: 1, DisabledRules: []int{920500},
ExclusionsFile: exclusionsFile(t, backupFileExclusion),
})
if err != nil {
t.Fatalf("load the exclusions: %v", err)
}
wantResult(t, crs, get("/notes.txt~"), waf.Result{})
}
func TestExclusionsFileMayNameAnyNumberInAComment(t *testing.T) {
t.Parallel()
_, err := withExclusions(exclusionsFile(t,
"# 942100: GET /issues/900123\n"+backupFileExclusion))
if err != nil {
t.Errorf("the exclusions did not load: %v", err)
}
}
func TestExclusionsFileThatCannotBeUsedIsAnError(t *testing.T) {
t.Parallel()
missing := filepath.Join(t.TempDir(), "missing.conf")
_, err := withExclusions(missing)
if err == nil {
t.Errorf("%s loaded", missing)
}
for _, tc := range []struct{ text, want string }{
{"SecNoSuchDirective On\n", "unknown directive"},
// smallwebwaf's own rules may not be switched off or changed, by
// id or in a range, on one line or split over several.
{"SecRuleRemoveById 899999-901000\n", "names 899999-901000"},
{
"SecAction \"id:1,phase:1,pass,nolog,ctl:ruleRemoveById=899999-901000\"\n",
"names 899999-901000",
},
{
"SecAction \"id:1,phase:1,pass,nolog,ctl:ruleRemoveById=899999-+901000\"\n",
"names 899999-+901000",
},
{
"SecAction \"id:1,phase:1,pass,nolog,ctl:ruleRemoveTargetById=900500;ARGS:from\"\n",
"names 900500",
},
{"SecRuleRemoveById 899000-\\\n 901000\n", "names 899000-901000"},
{
"SecAction \"id:1,phase:1,pass,nolog,\\\n" +
" ctl:ruleRemoveById=9005\\\n 00\"\n",
"names 900500",
},
// The Core Rule Set's rules load after the file.
{"SecRuleRemoveById 942100\n", "SecRuleRemoveById in the exclusions file"},
{
"SecRuleUpdateTargetById 942100 \"!ARGS:id\"\n",
"SecRuleUpdateTargetById in the exclusions file",
},
} {
_, err := withExclusions(exclusionsFile(t, tc.text))
if err == nil || !strings.Contains(err.Error(), tc.want) {
t.Errorf("%q gave the error %v, want one with %q", tc.text, err, tc.want)
}
}
}
// giteaRequests returns the recorded ordinary gitea requests in testdata.
// Each has the Referer of a page that names the same, as a page opened
// from it has.
func giteaRequests(t *testing.T) []request {
t.Helper()
text, err := os.ReadFile(filepath.Join("testdata", "gitea-requests.txt"))
if err != nil {
t.Fatal(err)
}
var requests []request
for line := range strings.Lines(string(text)) {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
target, header, _ := strings.Cut(line, " ")
headers := []string{"Referer: http://git.example" + target}
if header != "" {
headers = append(headers, header)
}
requests = append(requests, get(target, headers...))
}
return requests
}
func TestOrdinaryGiteaRequestsPassAndAttacksAreRefused(t *testing.T) {
t.Parallel()
const threshold = 5 // SWWAF_WAF_ANOMALY_THRESHOLD's default
crs := atDefaults(t)
for _, r := range giteaRequests(t) {
if got := inspect(t, crs, r); got.Score >= threshold {
t.Errorf("%s %q refused: %+v", r.target, r.headers[1:], got)
}
}
// In q too, off gitea's search pages.
for _, parameter := range []string{"id", "q"} {
for _, attack := range []string{
"1'%20OR%20'1'='1",
"..%2Fcustom%2Fconf%2Fapp.ini",
"%3Cscript%3Ealert(1)%3C%2Fscript%3E",
} {
target := "/owner/repo?" + parameter + "=" + attack
if got := inspect(t, crs, get(target)); got.Score < threshold {
t.Errorf("%s let through: %+v", target, got)
}
}
}
}
// bodyLimit is SWWAF_WAF_BODY_LIMIT in the tests that read bodies. // bodyLimit is SWWAF_WAF_BODY_LIMIT in the tests that read bodies.
const bodyLimit = 8 << 10 const bodyLimit = 8 << 10