Request tags, with refusing and limits per tag (closes #175)
check / check (push) Failing after 1m22s
check / check (push) Failing after 1m22s
Each request gets sorted tags from the lookup, new address lists, the reputation sources and the rule matches, logged as `tags`. New settings: `SWW_TAG_LISTS`, `SWW_TAG_DENY`, `SWW_TAG_LIMIT_PERCENT`. They replace `SWW_ASN_LIMIT_PERCENT`, `SWW_COUNTRY_LIMIT_PERCENT` and `SWW_DENIED_COUNTRIES`. New heuristics `tag_lists` and `tag_deny`; the latter runs before the rate limits and again after the Core Rule Set. Model: opus-5-5
This commit is contained in:
clawbot authored and 
sneak committed
1 parent
f6435b1a73
commit
de9c71a796
42 files changed
+1420
-362
No files matched your search
@@ -257,14 +257,14 @@ Every setting the code reads, with its default:
|
||||
| `SWW_LOOKUP_DB_PATH` | empty | The IPinfo Lite database file, in its `.mmdb` form, read while the source is `file`. |
|
||||
| `SWW_LOOKUP_TIMEOUT` | `800ms` | How long a request waits for its client's first GeoJS answer when a setting needs it, and how long a request to GeoJS may take. |
|
||||
| `SWW_ADD_LOOKUP_HEADERS` | `false` | `true` passes the app the client's AS number and country in `X-Client-ASN` and `X-Client-Country`. |
|
||||
| `SWW_DENIED_COUNTRIES` | empty | Countries whose clients are refused, such as `cn,ru`. |
|
||||
| `SWW_EXCLUSIVELY_ALLOWED_COUNTRIES` | empty | When set, the only countries whose clients get through; a client the lookup cannot place is refused too. |
|
||||
| `SWW_ASN_LIMIT_PERCENT` | empty | AS numbers whose clients get a percentage of every request and byte limit, such as `AS14061:50,AS16276:25`. |
|
||||
| `SWW_COUNTRY_LIMIT_PERCENT` | empty | The same by country, such as `cn:25`. |
|
||||
| `SWW_ASN_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `SWW_ASN_LIMIT_PERCENT`. |
|
||||
| `SWW_COUNTRY_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `SWW_COUNTRY_LIMIT_PERCENT`. |
|
||||
| `SWW_TAG_LISTS` | empty | Address lists that each add a tag to their clients' requests, as `name=URL` pairs, such as `torexit=https://example.com/tor.txt`. |
|
||||
| `SWW_TAG_DENY` | empty | Tags whose requests are refused, such as `torexit,country=CN`. |
|
||||
| `SWW_TAG_LIMIT_PERCENT` | empty | Tags whose clients get a percentage, from 1 to 99, of every request and byte limit, such as `vpn:50,asn=14061:25`; the lowest applies. |
|
||||
| `SWW_ASN_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the AS numbers listed, in place of their `asn=` tag's percentage. |
|
||||
| `SWW_COUNTRY_BYTES_PERCENT` | empty | A percentage of the byte limits alone for the countries listed, in place of their `country=` tag's percentage. |
|
||||
| `SWW_UNKNOWN_LIMIT_PERCENT` | `100` | The percentage of every limit a client without a country gets. |
|
||||
| `SWW_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS<number>:<percent>` lines, fetched like a blocklist; where it and `SWW_ASN_LIMIT_PERCENT` both give a percentage, the lower applies. |
|
||||
| `SWW_ASN_LIMIT_PERCENT_URL` | unset | The URL of a text file of `AS<number>:<percent>` lines, fetched like a blocklist; each line counts as an `asn=` tag in `SWW_TAG_LIMIT_PERCENT` (the lowest wins). |
|
||||
| `SWW_RULES_DIR` | `/etc/sww/rules.d` | The directory of the rule files, read at start and again when a file there changes. |
|
||||
| `SWW_RULES_ENABLED` | `true` | `false` reads no rule file. |
|
||||
| `SWW_TRAP_PATHS` | empty | Paths only scanners ask for, such as `/wp-login.php,/xmlrpc.php`; a request for one bans the client. |
|
||||
@@ -414,8 +414,7 @@ services:
|
||||
SWW_RATE_LIMIT_PER_HOUR: "600"
|
||||
SWW_RATE_LIMIT_PER_DAY: "3000"
|
||||
# Hosting networks and one country get a share of those limits.
|
||||
SWW_ASN_LIMIT_PERCENT: AS14061:50,AS16276:25
|
||||
SWW_COUNTRY_LIMIT_PERCENT: cn:25
|
||||
SWW_TAG_LIMIT_PERCENT: asn=14061:50,asn=16276:25,country=CN:25
|
||||
# Paths only scanners ask for; one request bans the client.
|
||||
SWW_TRAP_PATHS: /wp-login.php,/xmlrpc.php
|
||||
# Answer a refused client with 429 rather than 403.
|
||||
@@ -507,31 +506,34 @@ of the request reaches the app.
|
||||
|
||||
`discriminator` takes the checks:
|
||||
|
||||
3. The static lists, `SWW_ALLOW_NETS` and `SWW_DENY_NETS`.
|
||||
4. The bans.
|
||||
5. The lookup of the client's AS number and country.
|
||||
6. The country lists.
|
||||
7. The reputation sources.
|
||||
8. The client's limit percentage, from its AS number, its country or a
|
||||
reputation hit.
|
||||
9. The rate limits.
|
||||
10. The trap paths.
|
||||
11. The rule files.
|
||||
12. The header limits, and then the Core Rule Set.
|
||||
13. A path under `/_sww/` is answered `404`, so that the app never sees that
|
||||
3. The tag lists.
|
||||
4. The static lists, `SWW_ALLOW_NETS` and `SWW_DENY_NETS`.
|
||||
5. The bans.
|
||||
6. The lookup of the client's AS number and country.
|
||||
7. The country list.
|
||||
8. The reputation sources.
|
||||
9. The tag refusals of `SWW_TAG_DENY`.
|
||||
10. The client's limit percentage, from its tags, its AS number, its country or
|
||||
a reputation hit.
|
||||
11. The rate limits.
|
||||
12. The trap paths.
|
||||
13. The rule files.
|
||||
14. The header limits, and then the Core Rule Set.
|
||||
15. The tag refusals again, so that `exploitsignature` can refuse too.
|
||||
16. A path under `/_sww/` is answered `404`, so that the app never sees that
|
||||
prefix.
|
||||
|
||||
`rproxy` takes the rest, with `discriminator` counting at the end:
|
||||
|
||||
14. The request is passed to the app, streaming.
|
||||
15. After the answer, `rproxy` sends `discriminator` a report with the status
|
||||
17. The request is passed to the app, streaming.
|
||||
18. After the answer, `rproxy` sends `discriminator` a report with the status
|
||||
and the bytes each way. For an allowed request, `discriminator` counts the
|
||||
bytes toward the byte limits and adds the request to the client's history.
|
||||
|
||||
A refused visitor is answered with `SWW_BAN_RESPONSE`. A `block` rule or a Core
|
||||
Rule Set match answers `403` and bans nobody, since the Core Rule Set has false
|
||||
positives; such refusals count toward `SWW_ERROR_BURST_THRESHOLD` instead. A
|
||||
request over the header limits of step 12 is answered `431`, and bans nobody
|
||||
request over the header limits of step 14 is answered `431`, and bans nobody
|
||||
either. `rproxy` itself answers `405` to a method it does not pass on, and the
|
||||
status for a size or time limit, such as `413` for a body too large.
|
||||
|
||||
@@ -550,6 +552,21 @@ status for a size or time limit, such as `413` for a body too large.
|
||||
bans. An admin adds, keeps or lifts a ban by editing `bans.json` or through
|
||||
the ban endpoints (see [`docs/admin-endpoints.md`](docs/admin-endpoints.md)).
|
||||
|
||||
### Request tags
|
||||
|
||||
Each request gets tags from what `discriminator` knows of its client, written in
|
||||
the `tags` field of its log line:
|
||||
|
||||
- `asn=<number>` and `country=<code>`, from the lookup;
|
||||
- one tag per list of `SWW_TAG_LISTS` that holds the client, such as `torexit`
|
||||
or `vpn=mullvad`;
|
||||
- `crowdsec`, `dnsbl=<zone>` and `abuseipdb`, when that reputation source lists
|
||||
the client;
|
||||
- `exploitsignature`, when a rule file or the Core Rule Set matched the request.
|
||||
|
||||
`SWW_TAG_DENY` refuses requests by tag, and `SWW_TAG_LIMIT_PERCENT` lowers a
|
||||
client's limits by tag. See "Request tags" in [`SPEC.md`](SPEC.md).
|
||||
|
||||
### Attack detection
|
||||
|
||||
- The Core Rule Set reads the method, the URL with its query string, and the
|
||||
|
||||
Reference in new issue
Block a user