Name log fields, metrics and the Core Rule Set package for what they are (closes #192)
check / check (push) Failing after 17m18s
check / check (push) Failing after 17m18s
Log field, metric, action and event names now say what they count and, for timings, the unit. Package internal/waf is internal/coreruleset. Model: opus-5-5
This commit is contained in:
1 parent
aede434f6a
commit
9a5782de68
52 files changed
+638
-591
No files matched your search
+4
-4
@@ -9,9 +9,9 @@ back or dropped.
|
||||
- `ban`: a ban `discriminator` made.
|
||||
- `permanent_ban`: a permanent ban it made, or an existing ban made permanent by
|
||||
a request.
|
||||
- `waf_block`: the Core Rule Set scored a request at or over the threshold. The
|
||||
request was refused unless the alert's `detail` gives `mode`: `detect` or
|
||||
`observe`.
|
||||
- `core_rule_set_match`: the Core Rule Set scored a request at or over the
|
||||
threshold. The request was refused unless the alert's `detail` gives `mode`:
|
||||
`detect` or `observe`.
|
||||
- `anomaly`: a count over an anomaly threshold.
|
||||
- `reputation_hit`: a request from a client on a blocklist, the CrowdSec
|
||||
decision list or a DNS blocklist, or with an AbuseIPDB score that is a hit.
|
||||
@@ -130,7 +130,7 @@ The priority and tag by event:
|
||||
|
||||
- `ban`: priority `default`, tag `no_entry`.
|
||||
- `permanent_ban`: priority `high`, tag `no_entry`.
|
||||
- `waf_block`: priority `default`, tag `shield`.
|
||||
- `core_rule_set_match`: priority `default`, tag `shield`.
|
||||
- `anomaly`: priority `high`, tag `chart_with_upwards_trend`.
|
||||
- `reputation_hit`: priority `low`, tag `label`.
|
||||
- `source_failure` and `file_error`: priority `high`, tag `warning`.
|
||||
|
||||
+6
-5
@@ -28,7 +28,8 @@ body bytes, never headers, whatever `SWW_BYTES_COUNT` says.
|
||||
## Attack detection
|
||||
|
||||
- `sww_rule_matches_total`, by `rule_id` and `action`, and `sww_rules_loaded`;
|
||||
- `sww_waf_matches_total`, by `mode` and `rule_id`, for the rules that fired.
|
||||
- `sww_core_rule_set_matches_total`, by `mode` and `rule_id`, for the rules that
|
||||
fired.
|
||||
|
||||
## Countries and AS numbers
|
||||
|
||||
@@ -43,8 +44,8 @@ For the `SWW_METRICS_TOP_N` busiest of each, the rest as `other`:
|
||||
## Lookups
|
||||
|
||||
- `sww_geojs_requests_total` and `sww_geojs_failures_total`;
|
||||
- `sww_geojs_unanswered_total`: the requests that needed an answer and went on
|
||||
without it;
|
||||
- `sww_requests_without_geojs_answer_total`: the requests that needed an answer
|
||||
and went on without it;
|
||||
- with the file source, `sww_lookup_database_last_read_timestamp_seconds` and
|
||||
`sww_lookup_database_read_failures_total`.
|
||||
|
||||
@@ -70,8 +71,8 @@ By `file`:
|
||||
## Remote log, alerts and the process
|
||||
|
||||
- while a syslog server is set, `sww_remote_log_lines_sent_total`,
|
||||
`sww_remote_log_lines_dropped_total` and `sww_remote_log_buffer_depth`;
|
||||
- by `destination`, `sww_alerts_sent_total`, `sww_alerts_failed_total`,
|
||||
`sww_remote_log_lines_dropped_total` and `sww_remote_log_lines_waiting`;
|
||||
- by `destination`, `sww_alerts_sent_total`, `sww_alert_send_failures_total`,
|
||||
`sww_alerts_suppressed_total` and `sww_alerts_dropped_total`;
|
||||
- Go's own `go_` metrics and the process's `process_` metrics.
|
||||
|
||||
|
||||
+23
-22
@@ -26,7 +26,7 @@ These questions write no line:
|
||||
A `report` line:
|
||||
|
||||
```
|
||||
{"type":"request","discriminator_line":"report","time":"2026-10-03T12:00:00.123Z","instance":"fsn1/discriminator","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"waf_score":0,"duration_total":3.217,"duration_checks":0.041,"duration_waf":0.029,"duration_upstream_total":3.104}
|
||||
{"type":"request","discriminator_line":"report","time":"2026-10-03T12:00:00.123Z","instance":"fsn1/discriminator","client_ip":"203.0.113.9","method":"GET","scheme":"https","host":"app.example","path":"/","query":"","protocol":"HTTP/1.1","status":200,"request_bytes":0,"response_bytes":5120,"referer":"","user_agent":"curl/8.9.1","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","peer_ip":"172.18.0.2","forwarded_for":"203.0.113.9","client_group":"203.0.113.9/32","asn":"AS64496","as_name":"Example Net","country":"DE","request_headers":{"accept":"*/*"},"action":"forward","counts":{"minute":1,"hour":12,"day":40,"minute_bytes":5120,"hour_bytes":61440,"day_bytes":204800},"core_rule_set_score":0,"duration_total_ms":3.217,"duration_checks_ms":0.041,"duration_core_rule_set_ms":0.029,"duration_upstream_total_ms":3.104}
|
||||
```
|
||||
|
||||
### Every line
|
||||
@@ -37,7 +37,7 @@ Every one of these lines has:
|
||||
- the standard web log fields;
|
||||
- `request_id`, `peer_ip` and `client_group`;
|
||||
- `asn`, `as_name` and `country`;
|
||||
- `action` and `duration_total`.
|
||||
- `action` and `duration_total_ms`.
|
||||
|
||||
### Standard web log fields
|
||||
|
||||
@@ -103,10 +103,10 @@ Every one of these lines has:
|
||||
- `permanently_banned`: refused under a ban that never runs out, including a
|
||||
ban this request made or made permanent;
|
||||
- `country_denied`, `client_is_rate_limited`, `blocked_by_block_rule` and
|
||||
`waf_blocked`: refused by that check;
|
||||
`blocked_by_core_rule_set`: refused by that check;
|
||||
- `request_header_too_large`: over a header limit;
|
||||
- `admin`: answered at one of `discriminator`'s own endpoints, or `404` for
|
||||
a path under `/_sww/`.
|
||||
- `own_endpoint`: answered at one of `discriminator`'s own endpoints, or
|
||||
`404` for a path under `/_sww/`.
|
||||
- `would_action`: in `observe` mode, what `enforce` mode would have done with a
|
||||
request it would have refused. `action` then names what was done, normally
|
||||
`forward`.
|
||||
@@ -140,9 +140,10 @@ Every one of these lines has:
|
||||
`byte_limits` bans when the report comes, so only the `report` line names
|
||||
it.
|
||||
|
||||
- `limit_percent` and `limit_percent_setting`: the client's percentage of the
|
||||
rate limits and the setting that lowered it, when one did. `bytes_percent` and
|
||||
`bytes_percent_setting` do the same for the byte limits.
|
||||
- `rate_limit_percent` and `rate_limit_percent_setting`: the client's percentage
|
||||
of the rate limits and the setting that lowered it, when one did.
|
||||
`bytes_limit_percent` and `bytes_limit_percent_setting` do the same for the
|
||||
byte limits.
|
||||
- `counts`: the client's totals as rate limiting counts them, this request
|
||||
included. It is left out when rate limiting did not count the request. It
|
||||
holds:
|
||||
@@ -158,7 +159,7 @@ Every one of these lines has:
|
||||
- `limit_hit`: the limit broken:
|
||||
- `minute`, `hour` or `day` for a rate limit;
|
||||
- `minute_bytes`, `hour_bytes` or `day_bytes` for a byte limit;
|
||||
- `error_burst`.
|
||||
- `minute_refusals`.
|
||||
|
||||
When the bytes counted for a request and its response break a limit, the
|
||||
request has been passed on whole, so its `action` stays `forward`. Only its
|
||||
@@ -166,8 +167,8 @@ Every one of these lines has:
|
||||
|
||||
- `rule_ids`: the rule file rules that matched, in order, up to the one that
|
||||
refused.
|
||||
- `waf_score` and `waf_rule_ids`: the Core Rule Set's score and the rules that
|
||||
matched, for a request it inspected.
|
||||
- `core_rule_set_score` and `core_rule_set_rule_ids`: the Core Rule Set's score
|
||||
and the rules that matched, for a request it inspected.
|
||||
- `reputation`: the sources the client is on, in this order:
|
||||
- blocklist URLs;
|
||||
- the CrowdSec decision list's URL;
|
||||
@@ -189,14 +190,14 @@ Every one of these lines has:
|
||||
|
||||
All in milliseconds:
|
||||
|
||||
- `duration_total`:
|
||||
- `duration_total_ms`:
|
||||
- on the `decision` line, from `discriminator` having read the question's
|
||||
body to the line;
|
||||
- on the `report` line, from `rproxy` having read the request's headers to
|
||||
the end of the answer.
|
||||
- `duration_checks`: to the end of the checks;
|
||||
- `duration_waf`: the part of that the Core Rule Set took;
|
||||
- `duration_upstream_total`: on the `report` line of an allowed request, from
|
||||
- `duration_checks_ms`: to the end of the checks;
|
||||
- `duration_core_rule_set_ms`: the part of that the Core Rule Set took;
|
||||
- `duration_upstream_total_ms`: on the `report` line of an allowed request, from
|
||||
the hand-over to the app to the end of the answer.
|
||||
|
||||
### Requests to its own endpoints
|
||||
@@ -209,8 +210,8 @@ these meanings where they differ:
|
||||
- `scheme`: `http`.
|
||||
- `time`: when the request's headers had been read.
|
||||
- `request_bytes`: the bytes read of a ban to add, and otherwise `0`.
|
||||
- `action`: `admin`.
|
||||
- `duration_total`: from the request's headers being read to the line.
|
||||
- `action`: `own_endpoint`.
|
||||
- `duration_total_ms`: from the request's headers being read to the line.
|
||||
|
||||
## `rproxy`'s line
|
||||
|
||||
@@ -228,10 +229,10 @@ these meanings where they differ:
|
||||
a WebSocket carried from the visitor.
|
||||
- `response_bytes`: the bytes of the answer's body, with what a WebSocket
|
||||
carried from the app.
|
||||
- `duration_request`: milliseconds from the request's headers being read until
|
||||
the app had been sent the whole request, or the answer started if that was
|
||||
sooner. It includes the wait for `discriminator`.
|
||||
- `duration_response`: milliseconds from then until the end of the answer.
|
||||
- `duration_request_ms`: milliseconds from the request's headers being read
|
||||
until the app had been sent the whole request, or the answer started if that
|
||||
was sooner. It includes the wait for `discriminator`.
|
||||
- `duration_response_ms`: milliseconds from then until the end of the answer.
|
||||
- `disposition`: what was done with the request: `allow`, `deny` or `close`. It
|
||||
is `discriminator`'s answer, or the value of `SWRP_DEFAULT_DISPOSITION`. It is
|
||||
left out when:
|
||||
@@ -245,5 +246,5 @@ these meanings where they differ:
|
||||
For example:
|
||||
|
||||
```
|
||||
{"type":"request","time":"2026-10-03T12:00:00.123Z","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","client_ip":"203.0.113.9","host":"app.example","method":"GET","path":"/","query":"","status":200,"request_bytes":0,"response_bytes":5120,"duration_request":1.204,"duration_response":3.104,"disposition":"allow"}
|
||||
{"type":"request","time":"2026-10-03T12:00:00.123Z","request_id":"7Q2NHZ4KJ3VXW5YB6R3MEFTD2A","client_ip":"203.0.113.9","host":"app.example","method":"GET","path":"/","query":"","status":200,"request_bytes":0,"response_bytes":5120,"duration_request_ms":1.204,"duration_response_ms":3.104,"disposition":"allow"}
|
||||
```
|
||||
+1
-1
@@ -324,7 +324,7 @@ the request first.
|
||||
- `SWW_ALERT_NTFY_TOKEN` (default unset): an ntfy access token. A control
|
||||
character in it is refused.
|
||||
- `SWW_ALERT_EVENTS` (default
|
||||
`ban,permanent_ban,waf_block,anomaly,reputation_hit,source_failure,file_error`):
|
||||
`ban,permanent_ban,core_rule_set_match,anomaly,reputation_hit,source_failure,file_error`):
|
||||
the event types alerts are sent for, from those in [`alerts.md`](alerts.md)
|
||||
but `summary`. Any other name is refused.
|
||||
- `SWW_ALERT_COOLDOWN` (default `15m`): how long after an alert is sent its
|
||||
|
||||
+2
-2
@@ -64,8 +64,8 @@ since it was first seen:
|
||||
- body bytes of its requests, and of the responses it was sent;
|
||||
- responses by status class;
|
||||
- offences by kind: `limit`, `attack`, `blocked_by_block_rule` and
|
||||
`waf_blocked`. A file may also hold `token_refused`, which is kept, and which
|
||||
nothing adds to.
|
||||
`blocked_by_core_rule_set`. A file may also hold `token_refused`, which is
|
||||
kept, and which nothing adds to.
|
||||
|
||||
## `lookups.json`
|
||||
|
||||
|
||||
Reference in new issue
Block a user