Author SHA1 Message Date
sneak 9a5782de68 Name log fields, metrics and the Core Rule Set package for what they are (closes #192)
check / check (push) Failing after 17m18s
Log field, metric, action and event names now say what they count
and, for timings, the unit. Package internal/waf is internal/coreruleset.

Model: opus-5-5
2026-11-09 06:09:38 +00:00
36 changed files with 226 additions and 662 deletions

No files matched your search

+4 -9
View File
@@ -280,7 +280,6 @@ Every setting the code reads, with its default:
| `SWW_WAF_EXEMPT_PATHS` | empty | Path prefixes the Core Rule Set does not inspect. |
| `SWW_WAF_BODY_LIMIT` | `off` | How much of a request body the Core Rule Set reads, a size up to `64K`; `off` reads none. |
| `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` | unset | A file of Core Rule Set exclusions, as Coraza directives, that switch rules off for some fields or paths only. |
| `SWW_APPS` | unset | The apps each hostname serves, such as `git.example.com:gitea;cloud.example.com:gitea`; only those apps' fixes apply there. Unset, every app's fixes apply to every hostname. |
| `SWW_BLOCKLIST_URLS` | empty | Blocklists to fetch, text files of addresses and netblocks one to a line, such as `https://www.spamhaus.org/drop/drop.txt`. |
| `SWW_BLOCKLIST_REFRESH` | `24h` | How long after a list was last tried it is fetched again, `1h` or more. |
| `SWW_BLOCKLIST_ACTION` | `deny` | What a client on a blocklist gets: `deny`, `limit:<percent>` of every limit, or `log`. |
@@ -582,15 +581,11 @@ client's limits by tag. See "Request tags" in [`SPEC.md`](SPEC.md).
headers of every request, and no body unless `SWW_WAF_BODY_LIMIT` is set. In
front of a code forge, bodies carry what people write and publish, which the
rules take for attacks.
- `smallwebwaf` changes the Core Rule Set in six ways so that it lets ordinary
- `smallwebwaf` changes the Core Rule Set in eight ways so that it lets ordinary
requests to a code forge through, such as git over HTTP and sign-in through
Git Credential Manager. No setting undoes the changes; "Attack detection" in
[`SPEC.md`](SPEC.md) lists them.
- Each supported app also has its own fixes, which let its ordinary requests
through. gitea's let its searches, file names and branch names through.
`SWW_APPS` names the apps each hostname serves. A hostname gets only its own
apps' fixes, and a hostname it does not list gets none. Unset, every app's
fixes apply to every hostname.
Git Credential Manager. On gitea's own paths, its searches, file names and
branch names get through too. No setting undoes the changes; "Attack
detection" in [`SPEC.md`](SPEC.md) lists them.
- `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` switches single rules off for an app's own
fields or paths.
- Rule files are plain text, one rule to a line, edited by hand and read again
+45 -68
View File
@@ -81,8 +81,8 @@ to them.
of bans is full.
- `internal/rules`: reads the rule files at start and again when they
change, and tells which rules a request matches.
- `internal/coreruleset`: the Core Rule Set, with the six changes and the
apps' fixes listed under "Attack detection", as Coraza directives.
- `internal/coreruleset`: the Core Rule Set, with the eight changes listed
under "Attack detection", as Coraza directives.
- `internal/lookup`: looks clients up through GeoJS or in the lookup
database.
- `internal/reputation`: fetches the lists, asks the DNS blocklists and
@@ -674,7 +674,7 @@ files, the limits and the bans still apply to the client that sends it.
- `SWW_CORE_RULE_SET_EXCLUSIONS_FILE` names a file of exclusions, written as
Coraza directives. An exclusion switches one rule off for some fields or paths
only, where `SWW_WAF_DISABLED_RULES` switches it off everywhere. The file
holds an app's own exclusions; gitea's fixes below hold gitea's.
holds an app's own exclusions; the eighth change below holds gitea's.
- `discriminator` loads the file after `smallwebwaf`'s own setup rules and
before the Core Rule Set's rules. The Core Rule Set loads its own
`REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf` at the same point.
@@ -720,7 +720,7 @@ files, the limits and the bans still apply to the client that sends it.
scanner from an ordinary client. In front of a code forge, registry and git
clients get `404` and `401` by design, hundreds of times a minute.
`smallwebwaf` changes the Core Rule Set 4.25.0 in six ways, since in front of
`smallwebwaf` changes the Core Rule Set 4.25.0 in eight ways, since in front of
gitea it would otherwise refuse ordinary requests. The changes hold in front of
every app, and no setting undoes them:
@@ -739,52 +739,8 @@ every app, and no setting undoes them:
and tea, which gitea registers for OAuth sign-in, ask to be sent back to
`http://127.0.0.1` on the user's own machine. The server never fetches that
address.
4. `Referer` is not checked for a Unix command given without arguments (932340),
for Java starting a process (944110) or for a Java class name (944130).
`Referer` names the page a request comes from, on everything that page loads
and on every link followed from it. The Core Rule Set takes some page
addresses for attacks:
- a search for `env`;
- OpenJDK's `Runtime.java`, whose path holds both `runtime` and `java.`;
- a file named `java.lang.String.md`.
Every request from such a page would be refused. `Referer` keeps the other
rules, except after some gitea searches (gitea's fixes). An app that passes
`Referer` to a shell must check it itself.
5. Responses are not inspected. A raw file from a repository, such as a shell
script, looks to the response rules like source code leaking from the server.
6. The `from` parameter of a container registry's upload path,
`/v2/<image>/blobs/uploads/`, is not checked for command names (932260). When
a push reuses a layer, `from` names the image the layer comes from. An image
such as `owner/python3-app` would have that push refused.
Each supported app also has its own fixes, which let its ordinary requests
through. An app's fixes hold for the hostnames `SWW_APPS` lists the app for:
- `SWW_APPS` holds entries separated by semicolons. Each is a hostname, a colon,
then one or more apps separated by commas, for example
`git.example.com:gitea;code.example.com:gitea`.
- The request's `Host` is matched by exact name, without its port and without
regard to case. There are no wildcards.
- A hostname not listed gets no app's fixes, so every rule checks it in full.
Set empty, `SWW_APPS` lists no hostname.
- Unset, every app's fixes hold for every hostname. An app's fixes then weaken
the check of any other app that serves the same paths or fields.
- Any of these mistakes stops the start:
- an unknown app;
- a hostname that is not a DNS name, such as `*.example.com`;
- a hostname listed twice;
- an app listed twice for one hostname;
- an entry without a colon.
`gitea` is the only app supported. Each app's fixes are in one file, such as
`internal/coreruleset/gitea.conf`, and one entry of `apps` in
`internal/coreruleset/coreruleset.go`. gitea's fixes are:
1. Ten query parameters are not checked against the Core Rule Set's lists of
system files (930120), shell paths (932160) and command names (932260), on
any path:
4. Ten query parameters are not checked against the Core Rule Set's lists of
system files (930120), shell paths (932160) and command names (932260):
- `path`, `files`, `skip-to` and `sub_path`;
- `ref`, `sha`, `branch` and `workflow`;
- `artifactName` and `redirect_to`.
@@ -797,18 +753,41 @@ through. An app's fixes hold for the hostnames `SWW_APPS` lists the app for:
catch reaches the app, such as the command `|cat /etc/passwd`. Path
traversal, SQL and script injection, and PHP, Java and Node.js code are
still refused there. Other parameters keep all three rules, except on the
gitea paths the third fix names.
gitea paths the eighth change names. An app that uses one of these
parameters as a file on the server, or passes it to a shell, must check the
value itself. A rule file can refuse it instead.
2. The Core Rule Set reads the request without the `gitea_flash` and
5. Two cookies are left out, and `Referer` keeps fewer rules:
- The Core Rule Set reads the request without the `gitea_flash` and
`redirect_to` cookies. gitea puts a message naming what was just done into
`gitea_flash`. It puts into `redirect_to` the page to return to after signing
in. The Core Rule Set takes file names in them, such as `package.json`, for
attacks. A browser sends such a cookie with every request until gitea
replaces it. gitea cannot replace it while those requests are refused, so one
refusal would keep the browser out of the whole site. gitea shows the message
with any script removed and returns only to its own pages. Every other cookie
is read in full.
3. On gitea's paths, some fields skip the rules that refused ordinary gitea
`gitea_flash`. It puts into `redirect_to` the page to return to after
signing in. The Core Rule Set takes file names in them, such as
`package.json`, for attacks. A browser sends such a cookie with every
request until gitea replaces it. gitea cannot replace it while those
requests are refused, so one refusal would keep the browser out of the
whole site. gitea shows the message with any script removed and returns
only to its own pages. Every other cookie is read in full. An app that
reads a cookie of either name must check it itself.
- `Referer` is not checked for a Unix command given without arguments
(932340), for Java starting a process (944110) or for a Java class name
(944130). `Referer` names the page a request comes from, on everything
that page loads and on every link followed from it. The Core Rule Set
takes some page addresses for attacks:
- a search for `env`;
- OpenJDK's `Runtime.java`, whose path holds both `runtime` and `java.`;
- a file named `java.lang.String.md`.
Every request from such a page would be refused. `Referer` keeps the
other rules, except after some gitea searches (the eighth change). An
app that passes `Referer` to a shell must check it itself.
6. Responses are not inspected. A raw file from a repository, such as a shell
script, looks to the response rules like source code leaking from the server.
7. The `from` parameter of a container registry's upload path,
`/v2/<image>/blobs/uploads/`, is not checked for command names (932260). When
a push reuses a layer, `from` names the image the layer comes from. An image
such as `owner/python3-app` would have that push refused.
8. On gitea's paths, some fields skip the rules that refused ordinary gitea
requests there. They are the fields where gitea sends a search, a file name
or a branch name. `internal/coreruleset/gitea.conf` lists each rule and the
request it refused. The fields are:
@@ -826,7 +805,8 @@ through. An app's fixes hold for the hostnames `SWW_APPS` lists the app for:
- packages.
A user's or an organisation's page is any path of one segment, so on
every such path `q` skips these three rules, for example on `/search`.
every such path, on any app, `q` skips these three rules, for example on
`/search`.
Some of these searches also skip the rule for Java class names (944130),
which refused `org.apache.commons.logging`.
@@ -864,14 +844,11 @@ through. An app's fixes hold for the hostnames `SWW_APPS` lists the app for:
`actions/upload-artifact@v3` (920450).
Elsewhere these fields keep every rule. The paths are those of gitea at the
site root.
site root. An app other than gitea that serves one of them must check these
fields itself.
On a hostname that gets gitea's fixes, an app other than gitea must check these
fields and cookies itself. Listing each hostname's own apps in `SWW_APPS` keeps
the full check for the others.
The parameter names in these changes and fixes are matched without regard to
case, and the cookie names exactly.
The parameter and cookie names in these changes are matched without regard to
case, as Coraza matches them.
### Ban lengths, response and scope
+1 -1
View File
@@ -159,7 +159,7 @@ Every one of these lines has:
- `limit_hit`: the limit broken:
- `minute`, `hour` or `day` for a rate limit;
- `minute_bytes`, `hour_bytes` or `day_bytes` for a byte limit;
- `minute_refusals` for the error burst (`SWW_ERROR_BURST_THRESHOLD`).
- `minute_refusals`.
When the bytes counted for a request and its response break a limit, the
request has been passed on whole, so its `action` stays `forward`. Only its
-18
View File
@@ -255,24 +255,6 @@ the request first.
a range. So does a directive that removes or changes rules as the file loads,
such as `SecRuleRemoveById`, since none of the Core Rule Set's rules is loaded
yet. It has no `_FILE` form.
- `SWW_APPS` (default unset): the apps each hostname serves. A request gets the
fixes of its hostname's apps only (see "Attack detection" in
[`SPEC.md`](../SPEC.md)).
- Entries are separated by semicolons. Each is a hostname, a colon, then one
or more apps separated by commas, for example
`git.example.com:gitea;code.example.com:gitea`.
- `gitea` is the only app supported.
- The request's `Host` is matched by exact name, without its port and
without regard to case. There are no wildcards.
- A hostname not listed gets no app's fixes, so every rule checks it in
full. Set empty, it lists no hostname.
- Unset, every app's fixes apply to every hostname.
- Any of these mistakes stops the start:
- an unknown app;
- a hostname that is not a DNS name, such as `*.example.com`;
- a hostname listed twice;
- an app listed twice for one hostname;
- an entry without a colon.
- `SWW_ERROR_BURST_THRESHOLD` (default `30`): the most refusals a client may get
in a minute from a rule file, a trap path or the Core Rule Set, a count or
`off`. The next one bans the client (see "Attack detection" in `SPEC.md`).
+5 -7
View File
@@ -220,7 +220,7 @@ type State struct {
// requests to it failed, and how many alerts were dropped from its full
// queue or given up as it refused them.
type Counts struct {
Sent, SendFailures, Dropped int64
Sent, Failed, Dropped int64
}
// Queue takes the alerts raised, holds back those it must, and sends the
@@ -264,7 +264,7 @@ type destination struct {
// waiting are the alerts waiting to be sent, oldest first.
waiting []*Alert
sent, sendFailures, dropped atomic.Int64
sent, failed, dropped atomic.Int64
}
// cooldownKey is what makes an alert a repeat of another: the same event
@@ -426,9 +426,7 @@ func (q *Queue) Counts(name string) Counts {
for _, d := range q.destinations {
if d.name == name {
return Counts{
Sent: d.sent.Load(),
SendFailures: d.sendFailures.Load(),
Dropped: d.dropped.Load(),
Sent: d.sent.Load(), Failed: d.failed.Load(), Dropped: d.dropped.Load(),
}
}
}
@@ -677,7 +675,7 @@ func (d *destination) run(ctx context.Context) {
retryAt = time.Time{}
case errors.Is(err, errRefused):
d.remove(alert)
d.sendFailures.Add(1)
d.failed.Add(1)
d.dropped.Add(1)
retryDelay = 0
@@ -686,7 +684,7 @@ func (d *destination) run(ctx context.Context) {
d.processLog.Warn("gave up an alert "+d.setting+" refused",
"event", alert.Event, "error", err.Error())
case ctx.Err() == nil: // not cut off as sww stops
d.sendFailures.Add(1)
d.failed.Add(1)
retryDelay = min(max(retryDelayFactor*retryDelay, firstRetryDelay),
maxRetryDelay)
+4 -4
View File
@@ -897,7 +897,7 @@ func TestDestinationThatDoesNotAnswerHoldsUpNeitherOther(t *testing.T) {
synctest.Wait()
wantDestinationCounts(t, q, alerts.DestinationSlack,
alerts.Counts{Sent: 3, SendFailures: 1})
alerts.Counts{Sent: 3, Failed: 1})
logged := log.String()
if !strings.Contains(logged,
@@ -934,7 +934,7 @@ func TestDropsAreCountedForEachDestination(t *testing.T) {
for destination, want := range map[string]alerts.Counts{
alerts.DestinationWebhook: {Sent: all},
alerts.DestinationSlack: {Dropped: 1},
alerts.DestinationNtfy: {SendFailures: all, Dropped: all},
alerts.DestinationNtfy: {Failed: all, Dropped: all},
} {
wantDestinationCounts(t, q, destination, want)
}
@@ -1223,12 +1223,12 @@ func wantEvents(t *testing.T, webhook *standIn, want ...string) {
// requests to it that it counts as failed, the alerts it counts as held
// back, and those it counts as dropped for the webhook.
func wantCounts(
t *testing.T, q *alerts.Queue, sent, sendFailures, suppressed, dropped int64,
t *testing.T, q *alerts.Queue, sent, failed, suppressed, dropped int64,
) {
t.Helper()
wantDestinationCounts(t, q, alerts.DestinationWebhook,
alerts.Counts{Sent: sent, SendFailures: sendFailures, Dropped: dropped})
alerts.Counts{Sent: sent, Failed: failed, Dropped: dropped})
if q.Suppressed() != suppressed {
t.Errorf("%d alerts held back, want %d", q.Suppressed(), suppressed)
+1 -1
View File
@@ -391,7 +391,7 @@ func alertFor(r Request, k key, o overThreshold) alerts.Alert {
detail["asn"] = k.asn
case ScopeTotal:
counted = "the whole service"
default: // watched_net
default: // watch
counted = "the named netblock " + k.name + ", " + k.netblock.String()
detail["name"] = k.name
}
+6 -6
View File
@@ -24,9 +24,9 @@ func TestBanWithoutACauseIsAnAdmins(t *testing.T) {
func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
t.Parallel()
params := defaultParams()
params.MaxBans = 1
ledger := bans.New(params)
rules := defaultParams()
rules.MaxBans = 1
ledger := bans.New(rules)
adminsOnly := netip.MustParsePrefix("198.51.100.0/24")
both := netip.MustParsePrefix("203.0.113.1/32")
second := netip.MustParsePrefix("203.0.113.2/32")
@@ -64,9 +64,9 @@ func TestAdminsBansAreNeverDroppedAndDoNotCountTowardMaxBans(t *testing.T) {
func TestEarlierBansCountABanDroppedAfterAnAdminsBan(t *testing.T) {
t.Parallel()
params := defaultParams()
params.MaxBans = 2
ledger := bans.New(params)
rules := defaultParams()
rules.MaxBans = 2
ledger := bans.New(rules)
netblock := netip.MustParsePrefix("203.0.113.1/32")
other := netip.MustParsePrefix("203.0.113.2/32")
+12 -12
View File
@@ -80,9 +80,9 @@ func TestRepeatWindowRunsOut(t *testing.T) {
func TestFirstBanLongerThanTheMaximumIsPermanent(t *testing.T) {
t.Parallel()
params := defaultParams()
params.LimitBanDuration = params.MaxBanDuration + time.Hour
ledger := bans.New(params)
rules := defaultParams()
rules.LimitBanDuration = rules.MaxBanDuration + time.Hour
ledger := bans.New(rules)
ban, _ := ledger.BanForLimit(netip.MustParsePrefix("203.0.113.9/32"), midnight(),
bans.Notes{})
@@ -97,9 +97,9 @@ func TestLongestBanSetFarOffDoesNotOverflow(t *testing.T) {
// With bans of up to 100,000 days, the 14th ban in a row, of 3^13
// hours, is within the maximum, and three times as long would not fit
// in a time.Duration. The 15th is permanent.
params := defaultParams()
params.MaxBanDuration = 100000 * day
ledger := bans.New(params)
rules := defaultParams()
rules.MaxBanDuration = 100000 * day
ledger := bans.New(rules)
netblock := netip.MustParsePrefix("203.0.113.9/32")
now := midnight()
@@ -200,9 +200,9 @@ func TestFindCountsNothing(t *testing.T) {
func TestMaxBansDropsTheEarliestBanOfTheNetblockSeenLongestAgo(t *testing.T) {
t.Parallel()
params := defaultParams()
params.MaxBans = 3
ledger := bans.New(params)
rules := defaultParams()
rules.MaxBans = 3
ledger := bans.New(rules)
a := netip.MustParsePrefix("203.0.113.1/32")
b := netip.MustParsePrefix("203.0.113.2/32")
c := netip.MustParsePrefix("203.0.113.3/32")
@@ -239,9 +239,9 @@ func TestFullLedgerDropsTheEarlierBanOfTheNetblockBannedAgain(t *testing.T) {
// With room for one ban, the netblock's ended ban goes to make room for
// its new one, whose notes still count it.
params := defaultParams()
params.MaxBans = 1
ledger := bans.New(params)
rules := defaultParams()
rules.MaxBans = 1
ledger := bans.New(rules)
netblock := netip.MustParsePrefix("203.0.113.9/32")
first, _ := ledger.BanForLimit(netblock, midnight(), bans.Notes{})
+6 -6
View File
@@ -219,9 +219,9 @@ func TestLoadKeepsAtMostMaxBansDroppingTheEarliest(t *testing.T) {
Cause: bans.CauseLimit,
}
params := defaultParams()
params.MaxBans = 1
ledger := bans.New(params)
rules := defaultParams()
rules.MaxBans = 1
ledger := bans.New(rules)
ledger.Load([]bans.Ban{later, earlier})
held := ledger.Snapshot()
@@ -235,9 +235,9 @@ func TestLoadReplacesTheBansHeld(t *testing.T) {
// Room for three bans, so that the second load, were it added to the
// two bans held, would drop none of them to make room.
params := defaultParams()
params.MaxBans = 3
ledger := bans.New(params)
rules := defaultParams()
rules.MaxBans = 3
ledger := bans.New(rules)
kept := bans.Ban{
Netblock: netip.MustParsePrefix("2001:db8::/64"),
Start: midnight(),
+1 -92
View File
@@ -29,7 +29,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
)
@@ -236,11 +235,6 @@ type Config struct {
WAFExemptPaths []string
WAFBodyLimit int64
CoreRuleSetExclusionsFile string
// Apps are the names of the apps each hostname serves, by hostname in
// lowercase (SWW_APPS): a request gets only its hostname's apps' fixes
// to the Core Rule Set. Apps is nil while SWW_APPS is unset, and every
// request gets every app's fixes.
Apps map[string][]string
// TrapPaths are the paths a request for which is a clear sign of
// attack (SWW_TRAP_PATHS), each starting with / and without a ?.
TrapPaths []string
@@ -410,11 +404,6 @@ var (
errNotTrapPath = errors.New(
"is not a path starting with / and without a ?, such as /wp-login.php",
)
errNotAppsEntry = errors.New(
"is not a hostname, : and a list of apps, such as git.example.com:gitea",
)
errNotHostname = errors.New("is not a DNS name such as git.example.com")
errNotApp = errors.New("is not an app sww has fixes for; the apps are")
errNotWAFMode = errors.New("is not off, detect or block")
errNotParanoiaLevel = errors.New("is not a paranoia level, from 1 to 4")
errNotRuleID = errors.New(
@@ -588,7 +577,6 @@ func FromEnvironment(
WAFBodyLimit: env.wafBodyLimit("SWW_WAF_BODY_LIMIT", off),
CoreRuleSetExclusionsFile: env.filePath(
"SWW_CORE_RULE_SET_EXCLUSIONS_FILE"),
Apps: env.apps("SWW_APPS"),
TrapPaths: env.trapPaths("SWW_TRAP_PATHS"),
ErrorBurstThreshold: env.count("SWW_ERROR_BURST_THRESHOLD", "30"),
LogRemoteURL: env.logRemoteURL("SWW_LOG_REMOTE_URL"),
@@ -844,25 +832,6 @@ func (e *environment) pathPrefixes(name, defaultValue string) []string {
return prefixes
}
// apps reads the setting that is the apps each hostname serves. It is nil
// while the setting is unset, and logged as null, unlike a setting set
// empty.
func (e *environment) apps(name string) map[string][]string {
value, set := e.lookup(name)
if !set {
e.settings = append(e.settings, slog.Any(name, nil))
return nil
}
e.settings = append(e.settings, slog.String(name, value))
apps, err := parseApps(value)
e.check(name, err)
return apps
}
// trapPaths reads the setting that is the list of trap paths. It is empty
// by default.
func (e *environment) trapPaths(name string) []string {
@@ -1880,65 +1849,6 @@ func parseTrapPaths(value string) ([]string, error) {
return paths, nil
}
// parseApps reads the apps each hostname serves: entries separated by
// semicolons, each a hostname, a colon, and a comma-separated list of the
// names coreruleset.AppNames returns, such as
// git.example.com:gitea;cloud.example.com:nextcloud,wordpress. Each
// hostname is a DNS name, without a wildcard. It returns them by hostname
// in lowercase, since a hostname is matched without regard to case. An
// empty value lists no hostname.
func parseApps(value string) (map[string][]string, error) {
apps := map[string][]string{}
if strings.TrimSpace(value) == "" {
return apps, nil
}
for entry := range strings.SplitSeq(value, ";") {
hostname, list, found := strings.Cut(entry, ":")
if !found {
return nil, fmt.Errorf("%q %w", entry, errNotAppsEntry)
}
hostname = strings.ToLower(strings.TrimSpace(hostname))
if hostname == "" {
return nil, fmt.Errorf("%q %w", entry, errNotAppsEntry)
}
if !isZone(hostname) {
return nil, fmt.Errorf("%q %w", hostname, errNotHostname)
}
if _, listed := apps[hostname]; listed {
return nil, fmt.Errorf("%q %w", hostname, errListedTwice)
}
names, err := parseList(list)
if err != nil {
return nil, err
}
if len(names) == 0 {
return nil, fmt.Errorf("%q %w", entry, errNotAppsEntry)
}
for i, name := range names {
if !slices.Contains(coreruleset.AppNames(), name) {
return nil, fmt.Errorf("%q %w %s", name, errNotApp,
strings.Join(coreruleset.AppNames(), ", "))
}
if slices.Contains(names[:i], name) {
return nil, fmt.Errorf("%q %w", name, errListedTwice)
}
}
apps[hostname] = names
}
return apps, nil
}
// parseRuleIDs reads a comma-separated list of the ids of Core Rule Set
// rules, each a whole number above zero and outside firstSetupRuleID to
// lastSetupRuleID.
@@ -2545,8 +2455,7 @@ func MaskZoneKey(zone string) string {
return masked + "." + rest
}
// isZone reports whether each label of zone is as parseZones takes it. It
// also checks the hostnames of SWW_APPS, which are DNS names too.
// isZone reports whether each label of zone is as parseZones takes it.
func isZone(zone string) bool {
for label := range strings.SplitSeq(zone, ".") {
badChar := strings.ContainsFunc(label, func(char rune) bool {
-88
View File
@@ -89,7 +89,6 @@ const (
wafExemptPaths = "SWW_WAF_EXEMPT_PATHS"
wafBodyLimit = "SWW_WAF_BODY_LIMIT"
exclusionsFile = "SWW_CORE_RULE_SET_EXCLUSIONS_FILE"
apps = "SWW_APPS"
trapPaths = "SWW_TRAP_PATHS"
errorBurstThreshold = "SWW_ERROR_BURST_THRESHOLD"
logRemoteURL = "SWW_LOG_REMOTE_URL"
@@ -583,92 +582,6 @@ func TestCoreRuleSetSettings(t *testing.T) {
}
}
func TestApps(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
env environment
want map[string][]string
}{
{environment{}, nil},
{environment{apps: ""}, map[string][]string{}},
{
environment{apps: "Git.Example.com:gitea; cloud.example.com : gitea "},
map[string][]string{
"git.example.com": {"gitea"}, "cloud.example.com": {"gitea"},
},
},
} {
cfg := fromEnvironment(t, tc.env)
if !reflect.DeepEqual(cfg.Apps, tc.want) {
t.Errorf("%v gave %#v, want %#v", tc.env, cfg.Apps, tc.want)
}
}
}
// Unset and set empty, SWW_APPS has opposite effects, so it logs differently.
func TestUnsetAndEmptyAppsAreLoggedDifferently(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
env environment
want string
}{
{environment{}, `"` + apps + `":null`},
{environment{apps: ""}, `"` + apps + `":""`},
} {
var out bytes.Buffer
slog.New(slog.NewJSONHandler(&out, nil)).
Info("starting", "settings", fromEnvironment(t, tc.env))
if !strings.Contains(out.String(), tc.want) {
t.Errorf("%v logged %s, want %s in it", tc.env, out.String(), tc.want)
}
}
}
func TestInvalidAppsStopsTheStart(t *testing.T) {
t.Parallel()
const (
notEntry = " is not a hostname, : and a list of apps, " +
"such as git.example.com:gitea"
notHostname = " is not a DNS name such as git.example.com"
)
for _, tc := range []struct{ value, want string }{
{"git.example.com:gitea;blog.example.com", `"blog.example.com"` + notEntry},
{"git.example.com:gitea;", `""` + notEntry},
{":gitea", `":gitea"` + notEntry},
{"git.example.com:", `"git.example.com:"` + notEntry},
{"*.example.com:gitea", `"*.example.com"` + notHostname},
{"git.example.com/:gitea", `"git.example.com/"` + notHostname},
{"git example.com:gitea", `"git example.com"` + notHostname},
{
"git.example.com:gitea,nextcloud",
`"nextcloud" is not an app sww has fixes for; the apps are gitea`,
},
{
"git.example.com:gitea;GIT.example.com:gitea",
`"git.example.com" is listed twice`,
},
{"git.example.com:gitea,gitea", `"gitea" is listed twice`},
{"git.example.com:gitea,", `"gitea," has an empty item in its list`},
} {
t.Run(tc.value, func(t *testing.T) {
t.Parallel()
_, err := config.FromEnvironment(environment{apps: tc.value}.lookupEnv)
want := apps + ": " + tc.want
if err == nil || err.Error() != want {
t.Errorf("error %v, want %s", err, want)
}
})
}
}
func TestWAFBodyLimitOf64K(t *testing.T) {
t.Parallel()
@@ -2676,7 +2589,6 @@ func TestLogsEachSettingWithItsValue(t *testing.T) {
wafExemptPaths: "",
wafBodyLimit: off,
exclusionsFile: "",
apps: "", // logged as null, which decodes as ""
trapPaths: "",
errorBurstThreshold: "30",
logRemoteURL: "",
+60 -106
View File
@@ -1,9 +1,9 @@
// Package coreruleset runs the OWASP Core Rule Set 4.25.0, through Coraza, on the
// method, the URL with its query and the headers of a request, and on its
// body while SWW_WAF_BODY_LIMIT is set, with the six changes sww
// makes to it, the fixes for the apps SWW_APPS names, and the exclusions in
// SWW_CORE_RULE_SET_EXCLUSIONS_FILE, as "Attack detection" under "Settings"
// in SPEC.md describes them. It reads no response.
// body while SWW_WAF_BODY_LIMIT is set, with the eight changes sww
// makes to it and the exclusions in SWW_CORE_RULE_SET_EXCLUSIONS_FILE, as
// "Attack detection" under "Settings" in SPEC.md describes them. It reads no
// response.
//
// sww writes only to its state directory, so Coraza is built with
// its no_fs_access tag, as the Dockerfile and script/build build it: of a
@@ -16,8 +16,6 @@ import (
"errors"
"fmt"
"io"
"maps"
"net"
"net/http"
"net/netip"
"os"
@@ -34,8 +32,8 @@ import (
// directives are the Core Rule Set as sww runs it, with the
// paranoia level for %d, bodyDirectives for the first %s while
// SWW_WAF_BODY_LIMIT is set, the exclusions file for the second, and the
// directives of every app in apps for the third.
// SWW_WAF_BODY_LIMIT is set, the exclusions file for the second, and
// giteaExclusions for the third.
// Each rule sww adds has an id from 900000 to 900999, the ids the
// Core Rule Set keeps for the rules that set it up, which
// SWW_WAF_DISABLED_RULES and the exclusions file refuse, so that no
@@ -83,28 +81,44 @@ SecRule ARGUMENTS_LIMIT_REACHED "@eq 1" "id:900300,phase:2,pass,\
# before the rules of phase 1 too.
%s
# The apps' fixes come before the rules for the same reason.
# The eighth change comes before the rules for the same reason.
%s
# The fifth: only the rules for requests are loaded, and no response is
# The sixth: only the rules for requests are loaded, and no response is
# inspected.
Include @owasp_crs/REQUEST-*.conf
# The third: redirect_uri is not checked for a URL naming an IP address or
# localhost. Coraza matches a parameter name here without regard to case.
# ARGS holds the fields of a form data or multipart body Coraza reads as
# well as the query parameters, so a field of this name is left out too.
# localhost. Coraza matches a parameter name here, and in the fourth,
# without regard to case. ARGS holds the fields of a form data or multipart
# body Coraza reads as well as the query parameters, so a field of one of
# these names is left out too.
SecRuleUpdateTargetById 931100 "!ARGS:redirect_uri"
SecRuleUpdateTargetById 934110 "!ARGS:redirect_uri"
# The fourth: Referer is not checked for a Unix command without arguments,
# for Java starting a process, or for a Java class name. 944130 refused
# every page opened from the page of java.lang.String.md.
# The fourth: the query parameters in which gitea sends names within a
# repository or its own records, or a page of its own site, are not
# checked against the lists of system files, shell paths and command
# names. Coraza takes one rule id per directive.
SecRuleUpdateTargetById 930120 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
!ARGS:sub_path|!ARGS:ref|!ARGS:sha|!ARGS:branch|!ARGS:workflow|\
!ARGS:artifactName|!ARGS:redirect_to"
SecRuleUpdateTargetById 932160 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
!ARGS:sub_path|!ARGS:ref|!ARGS:sha|!ARGS:branch|!ARGS:workflow|\
!ARGS:artifactName|!ARGS:redirect_to"
SecRuleUpdateTargetById 932260 "!ARGS:path|!ARGS:files|!ARGS:skip-to|\
!ARGS:sub_path|!ARGS:ref|!ARGS:sha|!ARGS:branch|!ARGS:workflow|\
!ARGS:artifactName|!ARGS:redirect_to"
# The fifth, for Referer: it is not checked for a Unix command without
# arguments, for Java starting a process, or for a Java class name. 944130
# refused every page opened from the page of java.lang.String.md. The
# cookies are left out in Inspect.
SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer"
SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer"
SecRuleUpdateTargetById 944130 "!REQUEST_HEADERS:Referer"
# The sixth: a container image push that reuses a layer names the image
# The seventh: a container image push that reuses a layer names the image
# it takes the layer from in the query parameter from, which is not
# checked for command names. 932260 refused the push of a layer from
# owner/python3-app.
@@ -154,36 +168,18 @@ SecRule MULTIPART_STRICT_ERROR "!@eq 0" "id:900450,phase:2,pass,\
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
`
// app is an app whose ordinary requests the Core Rule Set would refuse,
// with the fixes that let them pass.
type app struct {
// directives are Coraza exclusions, loaded before the Core Rule Set's
// rules. Their first rule skips the rest unless Inspect sets the
// transaction variable app_ and the app's name, and they end with the
// marker it skips to.
directives string
// cookies are the cookies the Core Rule Set reads a request without.
cookies []string
}
//go:embed gitea.conf
var giteaDirectives string
// apps are the apps whose fixes sww has, by the name SWW_APPS gives each.
// giteaExclusions are the eighth change: the fields of gitea's own
// requests that skip the rules that refused them there, as Coraza
// directives.
//
//nolint:gochecknoglobals // a constant cannot be a map
var apps = map[string]app{
"gitea": {
directives: giteaDirectives,
cookies: []string{"gitea_flash", "redirect_to"},
},
}
//go:embed gitea.conf
var giteaExclusions string
// AppNames returns the names of the apps whose fixes sww has, in
// alphabetical order.
func AppNames() []string {
return slices.Sorted(maps.Keys(apps))
}
// cookiesNotRead are the cookies the Core Rule Set reads a request
// without, the rest of the fifth change.
//
//nolint:gochecknoglobals // a constant cannot be a list
var cookiesNotRead = []string{"gitea_flash", "redirect_to"}
// Params are what New needs.
type Params struct {
@@ -199,10 +195,6 @@ type Params struct {
// Core Rule Set's rules (SWW_CORE_RULE_SET_EXCLUSIONS_FILE), "" while
// there is none.
ExclusionsFile string
// Apps are the names of the apps each hostname serves, by hostname in
// lowercase (SWW_APPS). A request gets the fixes of its hostname's
// apps only, or of every app while Apps is nil.
Apps map[string][]string
}
// The ids of sww's own rules, which an exclusions file may not
@@ -292,17 +284,15 @@ func checkExclusions(exclusions string) error {
// CoreRuleSet is the Core Rule Set, ready to inspect requests. It is safe
// for concurrent use.
type CoreRuleSet struct {
engine coraza.WAF
waf coraza.WAF
bodyLimit int64
apps map[string][]string
}
// New returns the Core Rule Set with the six changes and the apps' fixes,
// at params' paranoia level, without the rules it switches off, reading
// request bodies up to params' limit, and with params' exclusions file
// loaded before the Core Rule Set's rules. An exclusions file is an error
// if it cannot be read or does not parse, or if checkExclusions refuses
// it.
// New returns the Core Rule Set with the eight changes, at params'
// paranoia level, without the rules it switches off, reading request
// bodies up to params' limit, and with params' exclusions file loaded
// before the Core Rule Set's rules. An exclusions file is an error if it
// cannot be read or does not parse, or if checkExclusions refuses it.
func New(params Params) (*CoreRuleSet, error) {
body := ""
if params.BodyLimit > 0 {
@@ -325,13 +315,8 @@ func New(params Params) (*CoreRuleSet, error) {
}
}
var fixes strings.Builder
for _, name := range AppNames() {
fixes.WriteString(apps[name].directives + "\n")
}
text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions,
fixes.String())
giteaExclusions)
if len(params.DisabledRules) > 0 {
ids := make([]string, len(params.DisabledRules))
@@ -342,16 +327,14 @@ func New(params Params) (*CoreRuleSet, error) {
text += "SecRuleRemoveById " + strings.Join(ids, " ") + "\n"
}
engine, err := coraza.NewWAF(coraza.NewWAFConfig().
waf, err := coraza.NewWAF(coraza.NewWAFConfig().
WithRootFS(coreruleset.FS).
WithDirectives(text))
if err != nil {
return nil, fmt.Errorf("load the Core Rule Set: %w", err)
}
return &CoreRuleSet{
engine: engine, bodyLimit: params.BodyLimit, apps: params.Apps,
}, nil
return &CoreRuleSet{waf: waf, bodyLimit: params.BodyLimit}, nil
}
// Result is what the Core Rule Set found in a request.
@@ -363,37 +346,25 @@ type Result struct {
Score int
}
// Inspect runs the Core Rule Set on r, a request from client, with the
// fixes of the apps appsFor names for r's hostname: on its method, its URL
// with the query, and its headers, the Cookie header without those apps'
// cookies, and on body, r's body as the caller has it, as readBody reads
// it. It returns what it found, what it read of body, and the error that
// ended the reading early, if one did.
// Inspect runs the Core Rule Set on r, a request from client: on its
// method, its URL with the query, and its headers, the Cookie header
// without the cookies in cookiesNotRead, and on body, r's body as the
// caller has it, as readBody reads it. It returns what it found, what it
// read of body, and the error that ended the reading early, if one did.
func (c *CoreRuleSet) Inspect(
r *http.Request, client netip.Addr, body io.Reader,
) (Result, []byte, error) {
tx := c.engine.NewTransaction()
tx := c.waf.NewTransaction()
// Closing would remove the files Coraza wrote, and it writes none.
defer func() { _ = tx.Close() }()
// The transaction's variables are set only through Coraza's interface
// for plugins.
state := tx.(plugintypes.TransactionState) //nolint:forcetypeassert // every one is
var cookiesNotRead []string
for _, name := range c.appsFor(r.Host) {
state.Variables().TX().Set("app_"+name, []string{"1"})
cookiesNotRead = append(cookiesNotRead, apps[name].cookies...)
}
tx.ProcessConnection(client.String(), 0, "", 0)
tx.ProcessURI(r.URL.String(), r.Method, r.Proto)
for name, values := range r.Header {
for _, value := range values {
if name == "Cookie" {
value = withoutCookies(value, cookiesNotRead)
value = withoutCookiesNotRead(value)
if value == "" {
continue // it held those cookies alone
}
@@ -486,26 +457,9 @@ func score(tx types.Transaction) int {
return n
}
// appsFor returns the names of the apps whose fixes a request for host,
// its Host header, gets: every app while SWW_APPS is unset, and otherwise
// those SWW_APPS lists for host's name, without the port and in
// lowercase.
func (c *CoreRuleSet) appsFor(host string) []string {
if c.apps == nil {
return AppNames()
}
hostname, _, err := net.SplitHostPort(host)
if err != nil {
hostname = host // it has no port
}
return c.apps[strings.ToLower(hostname)]
}
// withoutCookies returns value, a Cookie header's, without the cookies
// named in cookiesNotRead.
func withoutCookies(value string, cookiesNotRead []string) string {
// withoutCookiesNotRead returns value, a Cookie header's, without the
// cookies in cookiesNotRead.
func withoutCookiesNotRead(value string) string {
var kept []string
for cookie := range strings.SplitSeq(value, ";") {
+2 -92
View File
@@ -124,8 +124,8 @@ func atDefaults(t *testing.T) *coreruleset.CoreRuleSet {
}
// wantChange checks that crs lets through passes, a gitea request one of
// the six changes or gitea's fixes are for, and still finds result in
// refused, a request like it that they are not for.
// the eight changes is for, and still finds result in refused, a request
// like it that the change is not for.
func wantChange(
t *testing.T, crs *coreruleset.CoreRuleSet, passes, refused request,
result coreruleset.Result,
@@ -386,96 +386,6 @@ func TestImagePushThatReusesALayerIsNotCheckedForCommandNames(t *testing.T) {
matched(930120, 932160))
}
// giteaFixed are gitea requests that only gitea's fixes let through, one
// for each kind of fix, each with what the Core Rule Set finds in it
// without them.
//
//nolint:gochecknoglobals // a constant cannot be a list
var giteaFixed = []struct {
request request
result coreruleset.Result
}{
// A search of code, on the path of gitea's search.
{get("/explore/code?q=SELECT%20*%20FROM%20users%20WHERE"), matched(942100)},
// A file name, in a parameter gitea sends names in, on any path.
{get("/?path=.gitignore"), matched(930120)},
// A file name, in a cookie of gitea's.
{
get("/owner/repo", "Cookie: gitea_flash=success%3DFile%2Bpackage.json%2Bdeleted"),
matched(930120),
},
}
// The hostnames of the tests of SWW_APPS: the one that serves gitea, and
// one that serves another app.
const (
giteaHost = "git.example"
otherHost = "blog.example"
)
// withHost returns r sent to host.
func withHost(r request, host string) request {
return request{r.method, r.target, append([]string{"Host: " + host}, r.headers...)}
}
// servingApps returns the Core Rule Set at the defaults, with SWW_APPS
// giving apps.
func servingApps(t *testing.T, apps map[string][]string) *coreruleset.CoreRuleSet {
t.Helper()
crs, err := coreruleset.New(coreruleset.Params{
ParanoiaLevel: 1, DisabledRules: defaultDisabledRules, Apps: apps,
})
if err != nil {
t.Fatalf("load the Core Rule Set: %v", err)
}
return crs
}
func TestAppsUnsetGivesEveryHostnameEveryAppsFixes(t *testing.T) {
t.Parallel()
crs := servingApps(t, nil)
for _, fixed := range giteaFixed {
for _, host := range []string{giteaHost, otherHost} {
wantResult(t, crs, withHost(fixed.request, host), coreruleset.Result{})
}
}
}
func TestListedHostnameGetsItsAppsFixes(t *testing.T) {
t.Parallel()
crs := servingApps(t, map[string][]string{giteaHost: {"gitea"}})
// The hostname is matched without the port and without regard to case.
for _, host := range []string{giteaHost, "GIT.Example:8443"} {
for _, fixed := range giteaFixed {
wantResult(t, crs, withHost(fixed.request, host), coreruleset.Result{})
}
}
}
func TestUnlistedHostnameGetsTheFullCheck(t *testing.T) {
t.Parallel()
for _, apps := range []map[string][]string{
{giteaHost: {"gitea"}},
// SWW_APPS set empty lists no hostname.
{},
} {
crs := servingApps(t, apps)
for _, host := range []string{otherHost, otherHost + ":8443", "git.example.org"} {
for _, fixed := range giteaFixed {
wantResult(t, crs, withHost(fixed.request, host), fixed.result)
}
}
}
}
// withExclusions returns the Core Rule Set at the defaults, with the
// exclusions file at path.
func withExclusions(path string) (*coreruleset.CoreRuleSet, error) {
+7 -52
View File
@@ -1,51 +1,8 @@
# gitea's fixes: some fields skip the rules that refused ordinary gitea requests.
# They are the fields where gitea sends a search, a file name or a branch name.
# Each exclusion runs with the request, before the rules it switches off, those
# of phase 1 included, so this file is loaded before the Core Rule Set's rules.
# Each comment names a request the rule refused.
# A request whose hostname SWW_APPS does not list gitea for skips every rule
# below. Inspect sets tx.app_gitea on the others.
SecRule &TX:app_gitea "@eq 0" \
"id:900590,phase:1,pass,nolog,skipAfter:END_GITEA"
# The query parameters in which gitea sends names within a repository or its
# own records, or a page of its own site, are not checked against the lists of
# system files, shell paths and command names, on any path:
# 930120: .gitignore
# 932160: bin/docker-entrypoint
# 932260: docker-build
SecAction "id:900591,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=930120;ARGS:path,\
ctl:ruleRemoveTargetById=930120;ARGS:files,\
ctl:ruleRemoveTargetById=930120;ARGS:skip-to,\
ctl:ruleRemoveTargetById=930120;ARGS:sub_path,\
ctl:ruleRemoveTargetById=930120;ARGS:ref,\
ctl:ruleRemoveTargetById=930120;ARGS:sha,\
ctl:ruleRemoveTargetById=930120;ARGS:branch,\
ctl:ruleRemoveTargetById=930120;ARGS:workflow,\
ctl:ruleRemoveTargetById=930120;ARGS:artifactName,\
ctl:ruleRemoveTargetById=930120;ARGS:redirect_to,\
ctl:ruleRemoveTargetById=932160;ARGS:path,\
ctl:ruleRemoveTargetById=932160;ARGS:files,\
ctl:ruleRemoveTargetById=932160;ARGS:skip-to,\
ctl:ruleRemoveTargetById=932160;ARGS:sub_path,\
ctl:ruleRemoveTargetById=932160;ARGS:ref,\
ctl:ruleRemoveTargetById=932160;ARGS:sha,\
ctl:ruleRemoveTargetById=932160;ARGS:branch,\
ctl:ruleRemoveTargetById=932160;ARGS:workflow,\
ctl:ruleRemoveTargetById=932160;ARGS:artifactName,\
ctl:ruleRemoveTargetById=932160;ARGS:redirect_to,\
ctl:ruleRemoveTargetById=932260;ARGS:path,\
ctl:ruleRemoveTargetById=932260;ARGS:files,\
ctl:ruleRemoveTargetById=932260;ARGS:skip-to,\
ctl:ruleRemoveTargetById=932260;ARGS:sub_path,\
ctl:ruleRemoveTargetById=932260;ARGS:ref,\
ctl:ruleRemoveTargetById=932260;ARGS:sha,\
ctl:ruleRemoveTargetById=932260;ARGS:branch,\
ctl:ruleRemoveTargetById=932260;ARGS:workflow,\
ctl:ruleRemoveTargetById=932260;ARGS:artifactName,\
ctl:ruleRemoveTargetById=932260;ARGS:redirect_to"
# The eighth change: on gitea's paths, some fields skip the rules that refused
# ordinary gitea requests there. They are the fields where gitea sends a search,
# a file name or a branch name. Each exclusion runs with the request, before the
# rules it switches off, those of phase 1 included, so this file is loaded before
# the Core Rule Set's rules. Each comment names a request the rule refused.
# gitea's searches of code, of issues and pull requests, and of commits send the
# text searched for in q, which skips these rules there:
@@ -198,8 +155,8 @@ SecRule REQUEST_FILENAME \
# 942140: information_schema.sql
# 944110: java/lang/Runtime.java
# 944130: java.lang.String.md
# Each request skips the rules that refused it. Rule 900591 already has path and
# redirect_to skip 930120, 932160 and 932260 on every path.
# Each request skips the rules that refused it. The fourth change already has
# path and redirect_to skip 930120, 932160 and 932260 on every path.
# A file's commit graph, in file.
SecRule REQUEST_FILENAME "@rx ^/[^/]+/[^/]+/graph$" \
@@ -324,5 +281,3 @@ SecRule REQUEST_FILENAME \
"@rx ^/api/actions_pipeline/_apis/pipelines/workflows/[0-9]+/artifacts/[^/]+/upload$" \
"id:900640,phase:1,pass,nolog,\
ctl:ruleRemoveTargetById=920450;REQUEST_HEADERS_NAMES:Content-Range"
SecMarker END_GITEA
+7 -7
View File
@@ -40,7 +40,7 @@ func TestAdminEndpointsAreOffWhileTheTokenIsUnset(t *testing.T) {
// An empty token does not match the unset one either.
for _, authorization := range []string{adminBearer, bearer, "Bearer ", ""} {
for _, e := range adminEndpoints() {
s.ownEndpointRequest(authorization, e.method, e.path, e.body, http.StatusNotFound)
s.adminRequest(authorization, e.method, e.path, e.body, http.StatusNotFound)
}
}
@@ -67,7 +67,7 @@ func TestAdminEndpointsNeedTheAdminToken(t *testing.T) {
for _, authorization := range []string{
"", bearer, "Bearer " + strings.ToUpper(adminSecret), "Basic " + adminSecret,
} {
got := s.ownEndpointRequest(authorization, e.method, e.path, e.body,
got := s.adminRequest(authorization, e.method, e.path, e.body,
http.StatusUnauthorized)
if got.header.Get("WWW-Authenticate") != "Bearer" {
t.Errorf("%s %s with %q was answered without WWW-Authenticate: Bearer",
@@ -329,7 +329,7 @@ func TestAdminEndpointsNeedTheTokenInObserveMode(t *testing.T) {
})
for _, e := range adminEndpoints() {
s.ownEndpointRequest("", e.method, e.path, e.body, http.StatusUnauthorized)
s.adminRequest("", e.method, e.path, e.body, http.StatusUnauthorized)
}
if held := server.Ledger.Snapshot(); len(held) != 0 {
@@ -360,12 +360,12 @@ func adminEndpoints() []adminEndpoint {
func (s *sender) admin(method, path, body string, status int) answer {
s.t.Helper()
return s.ownEndpointRequest(adminBearer, method, path, body, status)
return s.adminRequest(adminBearer, method, path, body, status)
}
// ownEndpointRequest is admin with authorization as the request's Authorization
// header unless it is "".
func (s *sender) ownEndpointRequest(
// adminRequest is admin with authorization as the request's Authorization header
// unless it is "".
func (s *sender) adminRequest(
authorization, method, path, body string, status int,
) answer {
s.t.Helper()
+3 -3
View File
@@ -36,8 +36,8 @@ const (
// tooLarge is how discriminator refuses a request over the header limits.
tooLarge = http.StatusRequestHeaderFieldsTooLarge
// silentAbuseIPDBURL is where AbuseIPDB never answers: at the stand-in
// that silentGeoJSURL names too.
silentAbuseIPDBURL = "silent://abuseipdb/api/v2/check"
// that unansweredGeoJSURL names too.
silentAbuseIPDBURL = "unanswered://abuseipdb/api/v2/check"
// wholeLimit is the percentage of a rate limit that lowers it not at all.
wholeLimit = 100
// clientsDecision is the CrowdSec decision list holding client alone.
@@ -340,7 +340,7 @@ func newDiscriminatorInBubble(
settings := map[string]string{tagDeny: "country=KP"}
maps.Copy(settings, env)
server, out, _ := newDiscriminator(t, silentGeoJSURL, time.Now, settings)
server, out, _ := newDiscriminator(t, unansweredGeoJSURL, time.Now, settings)
return server, out
}
+2 -2
View File
@@ -134,9 +134,9 @@ func (rq *request) banForLimit(now time.Time, hit ratelimit.Hit, status int) {
switch hit.Kind {
case ratelimit.KindRequests:
notes.LimitPercent, notes.LimitPercentSetting = rq.rateLimitPercent.logged()
notes.LimitPercent, notes.LimitPercentSetting = rq.limitPercent.logged()
case ratelimit.KindBytes:
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged()
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesPercent.logged()
}
if rq.h.config.Observe {
+1 -1
View File
@@ -415,7 +415,7 @@ func startWithClock(
// sender sends requests to rproxy one after another, each on a connection of its
// own, and checks each one's answer and log line. discriminatorAddr is where
// discriminator's own endpoints are, which ownEndpointRequest asks. Every request
// discriminator's own endpoints are, which adminRequest asks. Every request
// discriminator gets must come through the sender, since the log lines are
// matched to them in order.
type sender struct {
+7 -7
View File
@@ -143,9 +143,9 @@ func TestBytesPercentSettingsTakeThePlaceOfTheOthersForByteLimits(t *testing.T)
for _, tc := range []struct {
name string
env map[string]string
// rateLimitPercent and bytesLimitPercent are the log line's, as
// percentText gives them, and limitHit is its limit_hit.
rateLimitPercent, bytesLimitPercent, limitHit string
// limitPercent and bytesPercent are the log line's, as percentText
// gives them, and limitHit is its limit_hit.
limitPercent, bytesPercent, limitHit string
}{
{
"lowering the byte limits alone",
@@ -183,10 +183,10 @@ func TestBytesPercentSettingsTakeThePlaceOfTheOthersForByteLimits(t *testing.T)
wantPercent(t, "rate_limit_percent",
line.RateLimitPercent, line.RateLimitPercentSetting,
tc.rateLimitPercent)
tc.limitPercent)
wantPercent(t, "bytes_limit_percent",
line.BytesLimitPercent, line.BytesLimitPercentSetting,
tc.bytesLimitPercent)
tc.bytesPercent)
})
}
}
@@ -291,7 +291,7 @@ func TestClientWithoutAnAnswerInTimeHasTheUnknownLimitPercent(t *testing.T) {
// In a synctest bubble, as TestRequestWaitsAsLongAsTheLookupTimeoutSays
// says, with a GeoJS that never answers.
synctest.Test(t, func(t *testing.T) {
server, out, _ := newDiscriminator(t, silentGeoJSURL, time.Now,
server, out, _ := newDiscriminator(t, unansweredGeoJSURL, time.Now,
map[string]string{unknownLimitPercent: "0"})
// Once the time the request waits for its answer is up, the client
@@ -335,7 +335,7 @@ func TestRequestWaitsForItsLookupWhileABiasedThresholdIsSet(t *testing.T) {
// In a synctest bubble, as TestRequestWaitsAsLongAsTheLookupTimeoutSays
// says, with a GeoJS that never answers.
synctest.Test(t, func(t *testing.T) {
server, out, _ := newDiscriminator(t, silentGeoJSURL, time.Now,
server, out, _ := newDiscriminator(t, unansweredGeoJSURL, time.Now,
map[string]string{
lookupTimeout: timeout.String(), tagLists: "vpn=" + vpnList, tc.setting: tc.value,
})
+1 -1
View File
@@ -26,7 +26,7 @@ func (byteLimits) check(_ context.Context, rq *request, now time.Time) *verdict
}
counts, hit, over := rq.h.limiter.CountBytes(rq.h.clientGroup(rq.client), now,
rq.countedBytes(), rq.bytesLimitPercent.percent)
rq.countedBytes(), rq.bytesPercent.percent)
rq.line.Counts.MinuteBytes = counts.MinuteBytes
rq.line.Counts.HourBytes = counts.HourBytes
rq.line.Counts.DayBytes = counts.DayBytes
@@ -10,7 +10,6 @@ import (
"testing"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/discriminator"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
)
@@ -22,7 +21,6 @@ const (
wafDisabledRules = "SWW_WAF_DISABLED_RULES"
wafExemptPaths = "SWW_WAF_EXEMPT_PATHS"
wafBodyLimit = "SWW_WAF_BODY_LIMIT"
apps = "SWW_APPS"
block = "block"
detect = "detect"
)
@@ -292,27 +290,6 @@ func TestDisabledRulesSwitchOffWhatGiteaWouldBeRefused(t *testing.T) {
}
}
func TestGiteasFixesApplyOnlyToAHostnameThatServesGitea(t *testing.T) {
t.Parallel()
// Only gitea's fixes let a file name through in path, a parameter
// gitea sends file names in.
const fileName = "/?path=.gitignore"
s := startDiscriminator(t, "", map[string]string{
wafMode: block, apps: siteA + ":gitea",
})
wantDisposition(t, do(t, s.question(siteA, client, fileName)),
http.StatusOK, discriminator.DispositionAllow)
wantDisposition(t, do(t, s.question(siteB, client, fileName)),
http.StatusForbidden, discriminator.DispositionDeny)
lines := s.out.requestLines(t, 2)
wantLine(t, lines[0], http.StatusOK, requestlog.ActionForward)
wantLine(t, lines[1], http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet)
}
func TestAttackInAFormBodyIsRefusedOnlyWhileBodiesAreRead(t *testing.T) {
t.Parallel()
+1 -1
View File
@@ -183,7 +183,7 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
rq.line.DiscriminatorLine = requestlog.DiscriminatorLineDecision
refused := rq.check(r.Context())
rq.checked = time.Now()
underOwnPrefix := strings.HasPrefix(r.URL.Path, ownEndpointPrefix)
underOwnPrefix := strings.HasPrefix(r.URL.Path, adminPrefix)
if refused == nil && !underOwnPrefix {
// In observe mode, what enforce mode would have refused counts here.
+1 -2
View File
@@ -21,8 +21,7 @@ func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
)
for range healthChecks {
got := s.ownEndpointRequest("", http.MethodGet, discriminator.HealthPath, "",
http.StatusOK)
got := s.adminRequest("", http.MethodGet, discriminator.HealthPath, "", http.StatusOK)
if string(got.body) != "ok\n" || got.header.Get("Content-Type") != contentType {
t.Errorf("health endpoint answered %q with Content-Type %q, want ok "+
+2 -2
View File
@@ -100,8 +100,8 @@ func TestOwnEndpointsAreNotInTheHistory(t *testing.T) {
s, _, server := startWithClock(t, "", map[string]string{metricsToken: token})
s.ownEndpointRequest("", http.MethodGet, discriminator.HealthPath, "", http.StatusOK)
s.ownEndpointRequest("", http.MethodGet, discriminator.MetricsPath, "",
s.adminRequest("", http.MethodGet, discriminator.HealthPath, "", http.StatusOK)
s.adminRequest("", http.MethodGet, discriminator.MetricsPath, "",
http.StatusUnauthorized)
s.scrape()
+7 -7
View File
@@ -385,7 +385,7 @@ func TestRequestWaitsAsLongAsTheLookupTimeoutSays(t *testing.T) {
// is never reached.
const timeout = 3 * time.Second
server, out, _ := newDiscriminator(t, silentGeoJSURL, time.Now,
server, out, _ := newDiscriminator(t, unansweredGeoJSURL, time.Now,
map[string]string{
lookupTimeout: timeout.String(),
allowedCountries: "DE",
@@ -405,25 +405,25 @@ func TestRequestWaitsAsLongAsTheLookupTimeoutSays(t *testing.T) {
})
}
// silentGeoJSURL is where a GeoJS that never answers is asked: a
// unansweredGeoJSURL is where a GeoJS that never answers is asked: a
// request to it waits, without the network, until it is abandoned.
// TestMain registers it with Go's default transport, through which GeoJS
// is asked.
const silentGeoJSURL = "silent://geojs/v1/ip/geo.json"
const unansweredGeoJSURL = "unanswered://geojs/v1/ip/geo.json"
func TestMain(m *testing.M) {
transport, _ := http.DefaultTransport.(*http.Transport)
transport.RegisterProtocol("silent", silentGeoJS{})
transport.RegisterProtocol("unanswered", unansweredGeoJS{})
transport.RegisterProtocol("abuseipdb", abuseIPDBStandIn{})
m.Run()
}
// silentGeoJS is the GeoJS at silentGeoJSURL.
type silentGeoJS struct{}
// unansweredGeoJS is the GeoJS at unansweredGeoJSURL.
type unansweredGeoJS struct{}
// RoundTrip waits until req is abandoned.
func (silentGeoJS) RoundTrip(req *http.Request) (*http.Response, error) {
func (unansweredGeoJS) RoundTrip(req *http.Request) (*http.Response, error) {
<-req.Context().Done()
return nil, req.Context().Err()
+4 -4
View File
@@ -32,7 +32,7 @@ func TestMetricsAreOffWhileTheTokenIsUnset(t *testing.T) {
// An empty token does not match the unset one either.
for _, authorization := range []string{bearer, "Bearer ", ""} {
s.ownEndpointRequest(authorization, http.MethodGet, discriminator.MetricsPath, "",
s.adminRequest(authorization, http.MethodGet, discriminator.MetricsPath, "",
http.StatusNotFound)
}
}
@@ -60,7 +60,7 @@ func TestMetricsNeedTheTokenAndOtherPathsAreNotFound(t *testing.T) {
{http.MethodGet, "/_sww/bans", bearer, http.StatusNotFound},
{http.MethodPost, discriminator.HealthPath, "", http.StatusNotFound},
} {
got := s.ownEndpointRequest(tc.authorization, tc.method, tc.path, "", tc.status)
got := s.adminRequest(tc.authorization, tc.method, tc.path, "", tc.status)
if tc.status == http.StatusUnauthorized &&
got.header.Get("WWW-Authenticate") != "Bearer" {
@@ -381,8 +381,8 @@ func scrape(t *testing.T, addr string) string {
func (s *sender) scrape() string {
s.t.Helper()
return string(s.ownEndpointRequest(bearer, http.MethodGet, discriminator.MetricsPath,
"", http.StatusOK).body)
return string(s.adminRequest(bearer, http.MethodGet, discriminator.MetricsPath, "",
http.StatusOK).body)
}
// metric returns the value of series in metrics, which are in the
+4 -6
View File
@@ -25,9 +25,9 @@ import (
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// ownEndpointPrefix starts the path of each of discriminator's own endpoints. The app
// never sees a path under it.
const ownEndpointPrefix = "/_sww/"
// adminPrefix starts the path of each of discriminator's own endpoints. The app never
// sees a path under it.
const adminPrefix = "/_sww/"
// HealthPath is the health endpoint of discriminator and of rproxy, which the
// container's health check asks.
@@ -221,8 +221,7 @@ func newReputation(
// NewCoreRuleSet returns the Core Rule Set at SWW_WAF_PARANOIA_LEVEL,
// without the rules SWW_WAF_DISABLED_RULES switches off, reading bodies
// up to SWW_WAF_BODY_LIMIT, with the exclusions in
// SWW_CORE_RULE_SET_EXCLUSIONS_FILE and the fixes of the apps in SWW_APPS,
// or nil while SWW_WAF_MODE is off. It
// SWW_CORE_RULE_SET_EXCLUSIONS_FILE, or nil while SWW_WAF_MODE is off. It
// reads at most QuestionBodyMaxBytes less one byte: a question's body of
// QuestionBodyMaxBytes may be cut short, and the Core Rule Set does not read a
// JSON or XML body over its limit. An exclusions file that coreruleset.New refuses is
@@ -237,7 +236,6 @@ func NewCoreRuleSet(cfg *config.Config) (*coreruleset.CoreRuleSet, error) {
ParanoiaLevel: cfg.WAFParanoiaLevel, DisabledRules: cfg.WAFDisabledRules,
BodyLimit: min(cfg.WAFBodyLimit, QuestionBodyMaxBytes-1),
ExclusionsFile: cfg.CoreRuleSetExclusionsFile,
Apps: cfg.Apps,
})
if err != nil {
return nil, fmt.Errorf("SWW_CORE_RULE_SET_EXCLUSIONS_FILE: %w", err)
+4 -6
View File
@@ -28,14 +28,12 @@ func (rateLimits) check(_ context.Context, rq *request, now time.Time) *verdict
return nil
}
rq.rateLimitPercent, rq.bytesLimitPercent = rq.limitPercentages()
rq.line.RateLimitPercent, rq.line.RateLimitPercentSetting =
rq.rateLimitPercent.logged()
rq.line.BytesLimitPercent, rq.line.BytesLimitPercentSetting =
rq.bytesLimitPercent.logged()
rq.limitPercent, rq.bytesPercent = rq.limitPercentages()
rq.line.RateLimitPercent, rq.line.RateLimitPercentSetting = rq.limitPercent.logged()
rq.line.BytesLimitPercent, rq.line.BytesLimitPercentSetting = rq.bytesPercent.logged()
counts, hit, over := rq.h.limiter.Count(rq.h.clientGroup(rq.client), now,
rq.rateLimitPercent.percent)
rq.limitPercent.percent)
rq.line.Counts = counts
if !over {
+5 -5
View File
@@ -256,9 +256,9 @@ func TestASNLimitPercentFileCountsAsTheSettingDoesTheLowerWinning(t *testing.T)
name string
env map[string]string
file string
// rateLimitPercent and bytesLimitPercent are the upload's, as
// percentText gives them.
rateLimitPercent, bytesLimitPercent string
// limitPercent and bytesPercent are the upload's, as percentText
// gives them.
limitPercent, bytesPercent string
}{
{"the file's alone", nil, asnDEQuarter, fromURL, fromURL},
{
@@ -293,10 +293,10 @@ func TestASNLimitPercentFileCountsAsTheSettingDoesTheLowerWinning(t *testing.T)
line := s.uploadFrom(fromDE)
wantPercent(t, "rate_limit_percent",
line.RateLimitPercent, line.RateLimitPercentSetting,
tc.rateLimitPercent)
tc.limitPercent)
wantPercent(t, "bytes_limit_percent",
line.BytesLimitPercent, line.BytesLimitPercentSetting,
tc.bytesLimitPercent)
tc.bytesPercent)
})
}
}
+3 -3
View File
@@ -48,11 +48,11 @@ type decision struct {
// so that not waiting never lowers its client's limits.
skippedLookupWait bool
// counted is true for a request the rate limits counted, whose bytes
// the byte limits count once it has ended. rateLimitPercent and
// bytesLimitPercent are then its client's limit percentages for the rate
// the byte limits count once it has ended. limitPercent and
// bytesPercent are then its client's limit percentages for the rate
// limits and for the byte limits.
counted bool
rateLimitPercent, bytesLimitPercent percentage
limitPercent, bytesPercent percentage
// attack is true for a request that matched a ban rule or asked for a
// trap path, blockedByBlockRule for one a block rule refused, and
// blockedByCoreRuleSet for one the Core Rule Set refused, each an offence
+7 -7
View File
@@ -362,7 +362,7 @@ func TestWithoutWaitTheRequestGoesOnAtOnceAndTheAnswerIsGivenWhenItComes(
mu.Unlock()
wantCountry(t, g, client, germany)
wantRequestsWithoutGeoJSAnswer(t, m, 0)
wantUnanswered(t, m, 0)
})
}
@@ -596,7 +596,7 @@ func TestClientsWithoutAnAnswerAreCounted(t *testing.T) {
// GeoJS fails, so the first client goes without an answer, and GeoJS
// is left alone for a second, which does not pass in this test.
wantCountry(t, g, clients(), "")
wantRequestsWithoutGeoJSAnswer(t, m, 1)
wantUnanswered(t, m, 1)
// Meanwhile each new client goes without one at once, while there is
// room for it among the 10,000 that may wait.
@@ -604,11 +604,11 @@ func TestClientsWithoutAnAnswerAreCounted(t *testing.T) {
wantCountry(t, g, clients(), "")
}
wantRequestsWithoutGeoJSAnswer(t, m, 10000)
wantUnanswered(t, m, 10000)
// One more, for which there is no room, goes without one too.
wantCountry(t, g, clients(), "")
wantRequestsWithoutGeoJSAnswer(t, m, 10001)
wantUnanswered(t, m, 10001)
})
}
@@ -841,9 +841,9 @@ func wantAsked(t *testing.T, geojs *standIn, i int, want ...string) {
}
}
// wantRequestsWithoutGeoJSAnswer checks how many requests m counts as having
// gone without an answer from GeoJS.
func wantRequestsWithoutGeoJSAnswer(t *testing.T, m *metrics.Metrics, want float64) {
// wantUnanswered checks how many requests m counts as having gone without
// an answer from GeoJS.
func wantUnanswered(t *testing.T, m *metrics.Metrics, want float64) {
t.Helper()
got := testutil.ToFloat64(m.RequestsWithoutGeoJSAnswer)
+1 -1
View File
@@ -359,7 +359,7 @@ func (m *Metrics) AddAlerts(queue *alerts.Queue) {
Help: "Requests to the destination that failed.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).SendFailures)
return float64(queue.Counts(name).Failed)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_suppressed_total",
+5 -5
View File
@@ -45,9 +45,9 @@ const (
idleTimeout = 120 * time.Second
)
// ownEndpointPrefix starts the path of every request for sww itself, which
// adminPrefix starts the path of every request for sww itself, which
// never reaches the app.
const ownEndpointPrefix = "/_sww/"
const adminPrefix = "/_sww/"
// allowedMethods lists the methods rproxy passes on, as an Allow header gives
// them. It answers any other method 405 itself.
@@ -211,8 +211,8 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
}
// passToApp passes the request on to the app, unless a limit or its path keeps
// it from the app. discriminator answers a path under ownEndpointPrefix itself, so
// such a request gets here only when SWRP_DEFAULT_DISPOSITION allowed it. ctx is the
// it from the app. discriminator answers a path under adminPrefix itself, so such a
// request gets here only when SWRP_DEFAULT_DISPOSITION allowed it. ctx is the
// request's own context.
func (rq *request) passToApp(ctx context.Context) {
maxBytes := rq.h.config.MaxRequestSize
@@ -220,7 +220,7 @@ func (rq *request) passToApp(ctx context.Context) {
switch {
case rq.refused.Load() != 0 || (maxBytes > 0 && rq.in.ContentLength > maxBytes):
rq.answer(http.StatusRequestEntityTooLarge)
case strings.HasPrefix(rq.in.URL.Path, ownEndpointPrefix):
case strings.HasPrefix(rq.in.URL.Path, adminPrefix):
rq.answer(http.StatusNotFound)
default:
rq.forward(ctx)
+6 -6
View File
@@ -64,7 +64,7 @@ var (
errCause = errors.New("is not limit, attack, admin or crowdsec")
errExpires = errors.New("is not after its start")
errDestination = errors.New("is not webhook, slack or ntfy")
errScope = errors.New("is not client, net, asn, total or watched_net")
errScope = errors.New("is not client, net, asn, total or watch")
errWaitingList = errors.New(`waiting is a list, but now lists the alerts by ` +
`destination: put the list under "webhook", as "waiting": {"webhook": [...]}, ` +
`or remove the file`)
@@ -858,8 +858,8 @@ func (f *alertsFile) check([]byte) error {
}
// checkAnomalyCounters refuses an anomaly counter whose scope is not
// client, net, asn, total or watched_net, most likely misspelt, and one
// without a field it needs, as missingFromCounter tells.
// client, net, asn, total or watch, most likely misspelt, and one without
// a field it needs, as missingFromCounter tells.
func checkAnomalyCounters(counters []anomaly.Counter) error {
for i, counter := range counters {
if !slices.Contains(anomaly.Scopes(), counter.Scope) {
@@ -879,9 +879,9 @@ func checkAnomalyCounters(counters []anomaly.Counter) error {
// missingFromCounter returns the first field counter, an anomaly counter,
// needs and has not, or "" when it has them all: what tells it from the
// others in its scope, without which it would never be counted again, the
// netblock of a client, net or watched_net counter, the AS number of an
// asn one and the name of a watched_net one; and the start of a window in
// which it has requests or bytes, without which they would be dropped.
// netblock of a client, net or watch counter, the AS number of an asn one
// and the name of a watch one; and the start of a window in which it has
// requests or bytes, without which they would be dropped.
func missingFromCounter(counter anomaly.Counter) string {
scope := counter.Scope
+1 -1
View File
@@ -534,7 +534,7 @@ func TestFileThatDoesNotParseStopsTheStart(t *testing.T) {
`{"version": 1, "anomaly_counters": [{"scope": "total"}, ` +
`{"scope": "nett", "netblock": "203.0.113.0/24"}]}`,
`: anomaly_counters entry 2's scope "nett" is not client, net, asn, total ` +
`or watched_net`,
`or watch`,
},
{
"a copy of a list with a line that does not read", reputationJSON,